Native Linux Microcontainers

Joblet is a micro-container runtime for running Linux jobs with: Process and filesystem isolation (PID namespace, chroot) Fine-grained CPU, memory, and IO throttling (cgroups v2) Secure job execution with mTLS and RBAC Built-in scheduler, SSE log streaming, and multi-core pinning Ideal for: Agentic AI Workloads (Untrusted code)

Project maintained by ehsaniara Hosted on GitHub Pages — Theme by mattgraham

RNX Command-Line Interface Reference

This comprehensive reference documentation provides detailed information for the RNX command-line interface, including complete command syntax, available options, configuration parameters, and practical usage examples for all supported operations.

Reference Documentation Structure

Global Options
Job Commands
- run
- list
- status
- log
- metrics
- telematics
- stop
- cancel
- delete
- delete-all
Volume Commands
Network Commands
Runtime Commands
System Commands
- version
- monitor
- nodes
- config-help
- help

Global Configuration Options

The following options are available across all RNX commands:

--config <path>    # Path to configuration file (default: searches standard locations)
--node <name>      # Node name from configuration (default: "default")
--json             # Output in JSON format
--version, -v      # Show version information for both client and server
--help, -h         # Show help for command

Configuration File Resolution

RNX resolves configuration files using the following precedence hierarchy:

./rnx-config.yml
./config/rnx-config.yml
~/.rnx/rnx-config.yml
/etc/joblet/rnx-config.yml
/opt/joblet/config/rnx-config.yml

Job Management Commands

`rnx job run`

Submits and executes a command on the target Joblet server instance.

rnx job run [parameters] <command> [arguments...]

Command Parameters

Parameter	Description	Default Value
`--max-cpu`	Maximum CPU usage percentage (0-10000)	0 (unlimited)
`--max-memory`	Maximum memory in MB	0 (unlimited)
`--max-iobps`	Maximum I/O bytes per second	0 (unlimited)
`--cpu-cores`	CPU cores to use (e.g., “0-3” or “1,3,5”)	”” (all cores)
`--gpu`	Number of GPUs to allocate to the job	0 (none)
`--gpu-memory`	Minimum GPU memory required (e.g., “8GB”, “4096MB”)	none
`--network`	Network mode: bridge, isolated, none, or custom	“bridge”
`--volume`	Volume to mount (can be specified multiple times)	none
`--upload`	Upload file or directory to workspace (auto-detects, can be specified multiple times)	none
`--runtime`	Use pre-built runtime (e.g., openjdk-21, python-3.11-ml)	none
`--env, -e`	Environment variable (KEY=VALUE, visible in logs)	none
`--secret-env, -s`	Secret environment variable (KEY=VALUE, hidden from logs)	none
`--schedule`	Schedule job execution (duration or RFC3339 time)	immediate

Examples

# Simple command
rnx job run echo "Hello, World!"

# With resource limits
rnx job run --max-cpu=50 --max-memory=512 --max-iobps=10485760 \
  python3 intensive_script.py

# CPU core binding
rnx job run --cpu-cores="0-3" stress-ng --cpu 4 --timeout 60s

# Multiple volumes
rnx job run --volume=data --volume=config \
  python3 process.py

# Environment variables (regular - visible in logs)
rnx job run --env="NODE_ENV=production" --env="PORT=8080" \
  node app.js

# Secret environment variables (hidden from logs)
rnx job run --secret-env="API_KEY=dummy_api_key_123" --secret-env="DB_PASSWORD=secret" \
  python app.py

# Mixed environment variables
rnx job run --env="DEBUG=true" --secret-env="SECRET_KEY=mysecret" \
  python app.py


# File upload
rnx job run --upload=script.py --upload=data.csv \
  python3 script.py data.csv

# Directory upload
rnx job run --upload=./project \
  npm start

# Scheduled execution
rnx job run --schedule="30min" backup.sh
rnx job run --schedule="2025-08-03T15:00:00" maintenance.sh

# Custom network
rnx job run --network=isolated ping google.com

# Using runtime
rnx job run --runtime=python-3.11-ml python -c "import torch; print(torch.__version__)"
rnx job run --runtime=openjdk-21 java -version

# GPU acceleration
rnx job run --gpu=1 python gpu_script.py
rnx job run --gpu=2 --gpu-memory=8GB python distributed_training.py
rnx job run --gpu=1 --gpu-memory=16GB --max-memory=32768 python llm_inference.py

# Complex example with GPU
rnx job run \
  --max-cpu=400 \
  --max-memory=8192 \
  --cpu-cores="0,2,4,6" \
  --gpu=1 \
  --gpu-memory=8GB \
  --network=mynet \
  --volume=persistent-data \
  --env=PYTHONPATH=/app \
  --upload=./src \
  --runtime=python-3.11-ml \
  python3 gpu_training.py --epochs=100

`rnx job list`

List all jobs on the server.

rnx job list [flags]

Flags

Flag	Description	Default
`--json`	Output in JSON format	false

Output Format

Table Format (default):

ID: Job UUID (36-character identifier)
NODE ID: Unique identifier of the Joblet node that executed the job (36-character UUID, “-“ if not assigned)
STATUS: Current job status (RUNNING, COMPLETED, FAILED, STOPPED, SCHEDULED)
START TIME: When the job started (format: YYYY-MM-DD HH:MM:SS)
COMMAND: The command being executed (truncated to 80 chars if too long)

JSON Format: Outputs a JSON array with detailed job information including all resource limits, volumes, network, and scheduling information.

Examples

# List all jobs (table format)
rnx job list

# Example output:
# UUID                                 NODE ID                              STATUS      START TIME           COMMAND
# ------------------------------------  ------------------------------------ ----------  -------------------  -------
# f47ac10b-58cc-4372-a567-0e02b2c3d479  8f94c5b2-1234-5678-9abc-def012345678 COMPLETED   2025-08-03 10:15:32  echo "Hello World"
# a1b2c3d4-e5f6-7890-abcd-ef1234567890  8f94c5b2-1234-5678-9abc-def012345678 RUNNING     2025-08-03 10:16:45  python3 script.py
# b2c3d4e5-f6a7-8901-bcde-f23456789012  -                                    FAILED      2025-08-03 10:17:20  invalid_command
# c3d4e5f6-a7b8-9012-cdef-345678901234  -                                    SCHEDULED   N/A                  backup.sh

# JSON output for scripting
rnx job list --json

# Example JSON output:
# [
#   {
#     "uuid": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
#     "status": "COMPLETED",
#     "start_time": "2025-08-03T10:15:32Z",
#     "end_time": "2025-08-03T10:15:33Z",
#     "command": "echo",
#     "args": ["Hello World"],
#     "exit_code": 0
#   },
#   {
#     "uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
#     "node_id": "8f94c5b2-1234-5678-9abc-def012345678",
#     "status": "RUNNING",
#     "start_time": "2025-08-03T10:16:45Z",
#     "command": "python3",
#     "args": ["script.py"],
#     "max_cpu": 100,
#     "max_memory": 512,
#     "cpu_cores": "0-3",
#     "scheduled_time": "2025-08-03T15:00:00Z"
#   }
# ]

# Filter with jq
rnx job list --json | jq '.[] | select(.status == "FAILED")'
rnx job list --json | jq '.[] | select(.max_memory > 1024)'

`rnx job status`

Get detailed status of a specific job.

rnx job status [flags] <job-uuid>

Job Status

Job UUIDs: 36-character UUID identifiers (e.g., “f47ac10b-58cc-4372-a567-0e02b2c3d479”)
Short UUIDs: First 8 characters are supported if they uniquely identify the job

Flags

Flag	Description	Default
`--json`	Output in JSON format	false

Examples

# Get job status (readable format)
rnx job status f47ac10b-58cc-4372-a567-0e02b2c3d479

# Get job status using short UUID
rnx job status f47ac10b

# Get status in JSON format
rnx job status --json f47ac10b-58cc-4372-a567-0e02b2c3d479

# Check multiple jobs
for uuid in f47ac10b a1b2c3d4; do rnx job status $uuid; done

# JSON output for scripting
rnx job status --json f47ac10b-58cc-4372-a567-0e02b2c3d479 | jq .status

# Example JSON output for individual job:
# {
#   "uuid": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
#   "node_id": "8f94c5b2-1234-5678-9abc-def012345678",
#   "command": "python3",
#   "args": ["process_data.py"],
#   "max_cpu": 100,
#   "cpu_cores": "0-3",
#   "max_memory": 512,
#   "max_io_bps": 0,
#   "status": "COMPLETED",
#   "start_time": "2025-08-03T10:15:32Z",
#   "end_time": "2025-08-03T10:18:45Z",
#   "exit_code": 0,
#   "scheduled_time": ""
# }

Output includes:

Job ID and command
Node ID (unique identifier of the Joblet node that executed the job)
Current status
Start/end times
Resource limits
Exit code (if completed)
Scheduling information

`rnx job log`

Stream job logs in real-time.

rnx job log <job-uuid>

Streams logs from running or completed jobs. Use Ctrl+C to stop following the log stream.

Examples

# Stream logs from a job
rnx job log f47ac10b-58cc-4372-a567-0e02b2c3d479

# Use standard Unix tools for filtering
rnx job log f47ac10b-58cc-4372-a567-0e02b2c3d479 | tail -100
rnx job log f47ac10b-58cc-4372-a567-0e02b2c3d479 | grep ERROR

# Save logs to file
rnx job log f47ac10b-58cc-4372-a567-0e02b2c3d479 > error.log

`rnx job metrics`

View resource usage metrics for a job as time-series data.

rnx job metrics <job-uuid>

Shows CPU, memory, I/O, network, and GPU metrics collected during job execution via cgroups (sampled every ~5 seconds). Metrics are stored as time-series data, allowing complete historical replay of resource usage.

For eBPF telematics events (process execution, network connections, etc.), use the separate rnx job telematics command.

Parameters

Parameter	Description
`--json`	Output in JSON format (global flag: `rnx --json`)

Behavior

Similar to rnx job log, this command streams all metrics from job start:

For completed jobs: Shows all metrics from start to finish, then exits
For running jobs: Shows all metrics from start to current, then continues streaming live until job completes or Ctrl+C

Works with both running and completed jobs. Supports short UUIDs (first 8 characters).

Metrics Collected

Category	Metrics
CPU	Usage percentage
Memory	Current usage, limit, percentage used
I/O	Read/write bytes
Network	RX/TX bytes
GPU	Utilization, memory (if GPUs allocated)

Examples

# View metrics for a completed job (shows complete history then exits)
rnx job metrics f47ac10b-58cc-4372-a567-0e02b2c3d479

# Monitor a running job using short UUID
rnx job metrics f47ac10b

# Output as JSON (one sample per line)
rnx --json job metrics f47ac10b

# Filter JSON output with jq
rnx --json job metrics f47ac10b | jq -c '{timestamp, cpu: .cpuPercent, memory: .memoryBytes}'

# Analyze metrics from a job
rnx --json job metrics f47ac10b > metrics.jsonl
cat metrics.jsonl | jq -r '[.timestamp, .cpuPercent, .memoryBytes] | @csv' > metrics.csv

Storage Location

Metrics are stored on the server as gzipped JSON Lines files:

Path: /opt/joblet/metrics/<job-uuid>/<timestamp>.jsonl.gz
Format: One JSON object per line (JSONL)
Compression: gzip (approximately 10x reduction)

You can also read metrics files directly on the server:

# Decompress and view metrics
gzip -dc /opt/joblet/metrics/<job-uuid>/*.jsonl.gz | head -10

# Parse with jq
gzip -dc /opt/joblet/metrics/<job-uuid>/*.jsonl.gz | jq -c '{timestamp, cpu: .cpu_percent}'

`rnx job telematics`

View eBPF security telematics events for a job.

rnx job telematics <job-uuid> [--types <event-types>]

Shows security-relevant events captured by eBPF tracing during job execution. These events are useful for security monitoring, debugging, and understanding job behavior.

Parameters

Parameter	Description
`--types`	Filter by event types (comma-separated): exec,connect,accept,file,mmap,mprotect,socket_data
`--json`	Output in JSON format (global flag: `rnx --json`)

Behavior

Similar to rnx job log, this command streams all telematics events from job start:

For completed jobs: Shows all events from start to finish, then exits
For running jobs: Shows all events from start to current, then continues streaming live until job completes or Ctrl+C

Works with both running and completed jobs. Supports short UUIDs (first 8 characters).

Event Types

Event Type	Display	Description
exec	EXEC	Process executions (fork/exec syscalls)
connect	CONNECT	Outgoing network connections (connect syscall)
accept	ACCEPT	Incoming network connections (accept syscall)
file	FILE	File operations (open, read, write)
mmap	MMAP	Memory mappings with executable permissions
mprotect	MPROTECT	Memory protection changes adding exec permission
socket_data	SEND/RECV	Socket data transfers (sendto/recvfrom syscalls)

Examples

# View all telematics events for a completed job
rnx job telematics f47ac10b-58cc-4372-a567-0e02b2c3d479

# Monitor a running job using short UUID
rnx job telematics f47ac10b

# Filter specific event types
rnx job telematics f47ac10b --types exec,connect

# Filter with grep for specific event types
rnx job telematics f47ac10b | grep EXEC
rnx job telematics f47ac10b | grep CONNECT
rnx job telematics f47ac10b | grep ACCEPT
rnx job telematics f47ac10b | grep MMAP

# Output as JSON (one event per line)
rnx --json job telematics f47ac10b

# Filter JSON output with jq
rnx --json job telematics f47ac10b | jq 'select(.type == "exec")'

# Analyze telematics events from a job
rnx --json job telematics f47ac10b > events.jsonl
cat events.jsonl | jq -r 'select(.type == "connect") | [.timestamp, .connect.dstAddr, .connect.dstPort] | @csv'

Use Cases

Security Monitoring: Track all processes executed and network connections made by a job
Debugging: Understand what a job is doing at the system level
Compliance: Audit job behavior for regulatory requirements
Incident Response: Investigate suspicious job activity

`rnx job stop`

Stop a running job.

rnx job stop <job-uuid>

Terminates a running job using graceful shutdown (SIGTERM) followed by force termination (SIGKILL) if necessary. The job will be marked as STOPPED and you can safely delete it afterward.

For scheduled jobs that haven’t started, use rnx job cancel instead.

Supports short UUIDs (first 8 characters) if they uniquely identify the job.

Examples

# Stop a running job (full UUID)
rnx job stop f47ac10b-58cc-4372-a567-0e02b2c3d479

# Stop using short UUID
rnx job stop f47ac10b

# Stop multiple jobs
rnx job list --json | jq -r '.[] | select(.status == "RUNNING") | .id' | xargs -I {} rnx job stop {}

`rnx job cancel`

Cancel a scheduled job before it starts executing.

rnx job cancel <job-uuid>

This command is specifically designed for jobs in SCHEDULED status and will:

Cancel the scheduled job (preventing it from executing)
Change the job status to CANCELED (not STOPPED)
Preserve the job in history for audit purposes

This provides proper cancel vs stop semantics:

rnx job stop → for RUNNING jobs (status becomes STOPPED)
rnx job cancel → for SCHEDULED jobs (status becomes CANCELED)

Note: This command only works for jobs in SCHEDULED status. For running jobs, use rnx job stop.

Examples

# Cancel a scheduled job
rnx job cancel f47ac10b-58cc-4372-a567-0e02b2c3d479

# Cancel using short UUID (first 8 characters)
rnx job cancel f47ac10b

# Cancel all scheduled jobs
rnx job list --json | jq -r '.[] | select(.status == "SCHEDULED") | .id' | xargs -I {} rnx job cancel {}

`rnx job delete`

Delete a job completely from the system.

rnx job delete <job-uuid>

Permanently removes the specified job including logs, metadata, and all associated resources. The job must be in a completed, failed, or stopped state - running jobs cannot be deleted directly and must be stopped first.

Examples

# Delete a completed job
rnx job delete f47ac10b-58cc-4372-a567-0e02b2c3d479

# Delete using short UUID (if unique)
rnx job delete f47ac10b

`rnx job delete-all`

Delete all non-running jobs from the system.

rnx job delete-all [flags]

Permanently removes all jobs that are not currently running or scheduled. Jobs in completed, failed, or stopped states will be deleted. Running and scheduled jobs are preserved and will not be affected.

Complete deletion includes:

Job records and metadata
Log files and buffers
Subscriptions and streams
Any remaining resources

Flags

--json: Output results in JSON format

Examples

# Delete all non-running jobs
rnx job delete-all

# Delete all non-running jobs with JSON output
rnx job delete-all --json

Example JSON Output:

{
  "success": true,
  "message": "Successfully deleted 3 jobs, skipped 1 running/scheduled jobs",
  "deleted_count": 3,
  "skipped_count": 1
}

Note: This operation is irreversible. Once deleted, job information and logs cannot be recovered. Only non-running jobs are affected.

Volume Commands

`rnx volume create`

Create a new volume for persistent storage.

rnx volume create <name> [flags]

Flags

Flag	Description	Default
`--size`	Volume size (e.g., 1GB, 500MB)	required
`--type`	Volume type: filesystem or memory	“filesystem”

Examples

# Create 1GB filesystem volume
rnx volume create mydata --size=1GB

# Create 512MB memory volume (tmpfs)
rnx volume create cache --size=512MB --type=memory

# Create volumes for different purposes
rnx volume create db-data --size=10GB --type=filesystem
rnx volume create temp-processing --size=2GB --type=memory

`rnx volume list`

List all volumes.

rnx volume list [flags]

Flags

Flag	Description	Default
`--json`	Output in JSON format	false

Examples

# List all volumes
rnx volume list

# JSON output
rnx volume list --json

# Check volume usage
rnx volume list --json | jq '.[] | select(.size_used > .size_total * 0.8)'

`rnx volume remove`

Remove a volume.

rnx volume remove <name>

Examples

# Remove single volume
rnx volume remove mydata

# Remove all volumes (careful!)
rnx volume list --json | jq -r '.[].name' | xargs -I {} rnx volume remove {}

Network Commands

`rnx network create`

Create a custom network.

rnx network create <name> [flags]

Flags

Flag	Description	Default
`--cidr`	Network CIDR (e.g., 10.10.0.0/24)	required

Examples

# Create basic network
rnx network create mynet --cidr=10.10.0.0/24

# Create multiple networks for different environments
rnx network create dev --cidr=10.10.0.0/24
rnx network create test --cidr=10.20.0.0/24
rnx network create prod --cidr=10.30.0.0/24

`rnx network list`

List all networks.

rnx network list [flags]

Flags

Flag	Description	Default
`--json`	Output in JSON format	false

Examples

# List all networks
rnx network list

# JSON output
rnx network list --json

`rnx network remove`

Remove a custom network. Built-in networks cannot be removed.

rnx network remove <name>

Examples

# Remove network
rnx network remove mynet

# Remove all custom networks (keep built-in networks)
rnx network list --json | jq -r '.networks[] | select(.builtin == false) | .name' | xargs -I {} rnx network remove {}

Runtime Commands

`rnx runtime list`

List all installed runtime environments on the remote server.

rnx runtime list [flags]

Flags

Flag	Description	Default
`--json`	Output in JSON format	false

Examples

# List locally installed runtimes
rnx runtime list

# JSON output for installed runtimes
rnx runtime list --json

`rnx runtime info`

Get detailed information about a specific runtime environment.

rnx runtime info <runtime-spec>

Examples

# Get runtime details
rnx runtime info python-3.11-ml
rnx runtime info openjdk-21

# JSON output for scripting
rnx runtime info python-3.11-ml --json

Output Fields

The command displays comprehensive runtime information:

Field	Description
Runtime	Runtime name
Version	Runtime version (e.g., “1.0.0”)
Description	Human-readable description
Language	Language and version (e.g., “python 3.11”)
Size	Total runtime size
Requirements	GPU requirements and supported architectures
Pre-installed Packages	Pip/npm packages included
Library Patterns	Shared library patterns copied
Environment Variables	Pre-configured environment
Build Info	When built, platform used

Example Output

Runtime: python-3.11-ml
Version: 1.0.0
Description: Python 3.11 with machine learning and data science packages

Language: python 3.11
Size: 1.3GB

Requirements:
  Architectures: amd64

Pre-installed Packages:
  - numpy==1.26.2
  - pandas==2.1.3
  - scikit-learn==1.3.2

Library Patterns:
  - libopenblas*
  - libgfortran*

Environment Variables:
  PYTHONUNBUFFERED=1
  PYTHONPATH=/usr/local/lib/python3/site-packages

Build Info:
  Built: 2025-12-22T17:43:00Z
  Platform: ubuntu-amd64

Usage:
  rnx job run --runtime=python-3.11-ml <command>

`rnx runtime build`

Build a runtime environment on the remote server from a YAML specification file.

rnx runtime build <path> [flags]

Arguments

<path> - Path to a runtime.yaml file or directory containing one

Flags

Flag	Short	Description	Default
`--force`	`-f`	Overwrite existing runtime if it already exists	false
`--verbose`	`-v`	Enable verbose output with debug logs	false

Description

The build command reads a runtime.yaml specification file, sends it to the remote joblet server, and executes the 14-phase build process on the server. Build progress is streamed back to the client in real-time.

Build Phases:

The build uses OverlayFS-based isolation to ensure the host system is never modified. System packages are installed in an isolated chroot, and only the resulting binaries/libraries are copied to the runtime directory.

Parse & Validate YAML specification
Detect platform (distro, architecture, package manager)
Check disk space
Validate package availability
Prepare directories
Run pre-install hook (if defined)
Install base language packages (in OverlayFS-isolated chroot)
Install language packages (pip/npm)
Run post-install hook (if defined)
Copy binaries from isolated overlay
Copy libraries from isolated overlay
Copy configuration
Generate runtime.yml config
Validate build

Runtime YAML Format:

See examples/ directory for sample runtime.yaml files or run rnx runtime validate to check your specification.

Examples

# Build a runtime from a YAML file
rnx runtime build ./examples/python-3.11-ml/runtime.yaml

# Build from a directory (looks for runtime.yaml inside)
rnx runtime build ./examples/python-3.11-ml/

# Build with verbose output
rnx runtime build -v ./examples/python-analytics/runtime.yaml

# Force rebuild (overwrite existing runtime)
rnx runtime build --force ./examples/python-3.11-ml/runtime.yaml

# Build with JSON output
rnx --json runtime build ./examples/python/runtime.yaml

# Validate before building
rnx runtime validate ./examples/python-3.11-ml/runtime.yaml

Example runtime.yaml

schema_version: "1.0"
name: python-3.11-ml
version: 1.0.0
description: Python 3.11 with ML packages

base:
  language: python
  version: "3.11"

pip:
  - numpy
  - pandas
  - scikit-learn

# Optional: Additional library patterns to copy to runtime
libraries:
  - libopenblas*
  - libgfortran*

environment:
  PYTHONUNBUFFERED: "1"

hooks:
  pre_install: |
    apt-get install -y libopenblas-dev || yum install -y openblas-devel

`rnx runtime test`

Test a runtime environment to verify it’s working correctly.

rnx runtime test <runtime-spec>

Examples

# Test runtime functionality
rnx runtime test python-3.11-ml
rnx runtime test openjdk:21

`rnx runtime remove`

Remove a runtime environment.

rnx runtime remove <runtime-spec>

Examples

# Remove a runtime
rnx runtime remove python-3.11-ml
rnx runtime remove openjdk-21

`rnx runtime validate`

Validate a runtime.yaml specification file comprehensively without building.

rnx runtime validate <path>

Arguments

<path> - Path to a runtime.yaml file or directory containing one

Description

Performs comprehensive server-side validation of the runtime YAML specification. This command sends the specification to the Joblet server for validation, which runs the first 4 phases of the build pipeline without actually building.

Validation checks include:

YAML syntax and schema validation
Schema version compatibility
Required fields (name, version, description, base)
Name format (lowercase, hyphens, dots, max 64 chars)
Version format (semantic versioning X.Y.Z)
Language support (python, java, node, go, rust)
Platform detection (distro, architecture, package manager)
Disk space availability
Package validation (base packages, pip, npm)
Existing runtime conflict detection (warns if runtime already exists)

This is the recommended way to check your runtime specification before building. It provides the same validation as rnx runtime build without actually installing anything.

Examples

# Validate a runtime.yaml file
rnx runtime validate ./examples/python-3.11-ml/runtime.yaml

# Validate from a directory
rnx runtime validate ./examples/openjdk-21/

# Example output:
# ✓ Runtime specification is valid
#
# Spec Information:
#   Name: python-3.11-ml
#   Version: 1.0.0
#   Description: Python 3.11 with ML packages
#   Language: python 3.11
#
# Platform:
#   Distro: ubuntu
#   Version: 22.04
#   Architecture: amd64
#   Package Manager: apt
#
# Packages:
#   System: python3.11, python3.11-dev, python3.11-venv
#   Pip: numpy, pandas, scikit-learn
#
# Warnings:
#   Runtime 'python-3.11-ml' version '1.0.0' already exists at /opt/joblet/runtimes/python-3.11-ml/1.0.0
#   Use --force with 'rnx runtime build' to overwrite

System Commands

`rnx version`

Display version information for both RNX client and Joblet server.

rnx version [flags]

Flags

Flag	Description	Default
`--json`	Output version info as JSON	false

Examples

# Show version information
rnx version

# Output:
# RNX Client:
# rnx version v4.3.3 (4c11220)
# Built: 2025-09-14T05:17:17Z
# Commit: 4c11220b6e4f98960853fa0379b5c25d2f19e33f
# Go: go1.24.0
# Platform: linux/amd64
#
# Joblet Server (default):
# joblet version v4.3.3 (4c11220)
# Built: 2025-09-14T05:18:24Z
# Commit: 4c11220b6e4f98960853fa0379b5c25d2f19e33f
# Go: go1.24.0
# Platform: linux/amd64

# Show version as JSON
rnx version --json

# Use --version flag (alternative)
rnx --version

Version Information Details

Client Version: The version of the RNX CLI tool running on your local machine
Server Version: The version of the Joblet server it’s connected to (from config)
Version Format: vMAJOR.MINOR.PATCH[+dev] where +dev indicates development builds after the tagged release
Build Information: Includes git commit hash, build date, Go version, and platform

Use Cases

Version Compatibility: Ensure client and server versions are compatible
Debugging: Identify specific builds when reporting issues
Deployment Tracking: Verify which version is deployed on production servers
Development: Track development builds with +dev suffix

`rnx monitor`

Monitor comprehensive remote joblet server metrics including CPU, memory, disk, network, processes, and volumes.

rnx monitor <subcommand> [flags]

Subcommands

status - Display comprehensive remote server status with detailed resource information
top - Show current remote server metrics in condensed format with top processes
watch - Stream real-time remote server metrics with configurable refresh intervals

Common Flags

Flag	Description	Default
`--json`	Output in UI-compatible JSON format	false
`--interval`	Update interval in seconds (watch only)	5
`--filter`	Filter metrics by type (top/watch only)	all
`--compact`	Use compact display format (watch only)	false

Available Server Metric Types (for –filter)

cpu - Server CPU usage, load averages, per-core utilization
memory - Server memory and swap usage with detailed breakdowns
disk - Server disk usage for all mount points and joblet volumes
network - Server network interface statistics with live throughput
io - Server I/O operations, throughput, and utilization
process - Server process statistics with top consumers

Monitoring Features

Enhanced Remote Server Monitoring:

Real-time server resource utilization tracking from client
Server cloud environment detection (AWS, GCP, Azure, KVM, etc.)
Remote joblet volume usage and availability monitoring
Server network throughput and packet statistics with accurate per-interface IP and MAC addresses
Server process state tracking (running, sleeping, stopped, zombie)
Server per-core CPU utilization breakdown
Joblet server version information (version, git tag, commit, build date, Go version)

Network Interface Display (v4.7.3+):

The rnx monitor status command displays accurate network interface information retrieved directly from the joblet server:

IP Addresses: Actual IP addresses assigned to each interface (not guessed)
MAC Addresses: Hardware MAC addresses for physical interfaces (not guessed)
Traffic Statistics: Real-time RX/TX rates, packet counts, and error tracking
Implementation: Data collected using Go’s net package for accuracy, no heuristics

Remote JSON Data Format:

UI-compatible JSON structure with server data for dashboards
Structured server metrics for monitoring tool integrations
Real-time server data streaming for live monitoring systems

Examples

# Comprehensive remote server status
rnx monitor status

# JSON server data for dashboards/APIs
rnx monitor status --json

# Current server metrics with top processes
rnx monitor top

# Filter specific server metrics
rnx monitor top --filter=cpu,memory

# Real-time server monitoring (5s intervals)
rnx monitor watch

# Faster server monitoring refresh rate
rnx monitor watch --interval=2

# Monitor specific server resources
rnx monitor watch --filter=disk,network

# JSON server streaming for monitoring tools
rnx monitor watch --json --interval=10

# Compact format for server monitoring
rnx monitor watch --compact

# Monitor specific joblet server node
rnx --node=production monitor status

JSON Output Structure

The --json flag produces UI-compatible output with the following structure:

{
  "hostInfo": {
    "hostname": "server-name",
    "platform": "Ubuntu 22.04.2 LTS",
    "arch": "amd64",
    "uptime": 152070,
    "cloudProvider": "AWS",
    "instanceType": "t3.medium",
    "region": "us-east-1"
  },
  "cpuInfo": {
    "cores": 8,
    "usage": 0.15,
    "loadAverage": [0.5, 0.3, 0.2],
    "perCoreUsage": [0.1, 0.2, 0.05, 0.3, ...]
  },
  "memoryInfo": {
    "total": 4100255744,
    "used": 378679296,
    "percent": 9.23,
    "swap": { "total": 0, "used": 0, "percent": 0 }
  },
  "disksInfo": {
    "disks": [
      {
        "name": "/dev/sda1",
        "mountpoint": "/",
        "filesystem": "ext4",
        "size": 19896352768,
        "used": 11143790592,
        "percent": 56.01
      },
      {
        "name": "analytics-data",
        "mountpoint": "/opt/joblet/volumes/analytics-data",
        "filesystem": "joblet-volume",
        "size": 1073741824,
        "used": 52428800,
        "percent": 4.88
      }
    ]
  },
  "networkInfo": {
    "interfaces": [...],
    "totalRxBytes": 1234567890,
    "totalTxBytes": 987654321
  },
  "processesInfo": {
    "processes": [...],
    "totalProcesses": 149
  }
}

`rnx nodes`

List configured nodes from the client configuration file.

rnx nodes [flags]

Flags

Flag	Description	Default
`--json`	Output in JSON format	false

Output Information

Node Name: Configuration name (default, production, etc.)
Address: Server connection address (host:port)
Node ID: Unique identifier of the Joblet node (if configured)
Certificate Status: Shows “***” if certificates are configured

Examples

# List all nodes with details
rnx nodes

# Example output:
# Available nodes from configuration:
#
# * default
#    Address: localhost:50051
#    Node ID: 8f94c5b2-1234-5678-9abc-def012345678
#    Cert:    ***
#    Key:     ***
#    CA:      ***
#
#  production
#    Address: prod.example.com:50051
#    Node ID: a1b2c3d4-5678-9abc-def0-123456789012
#    Cert:    ***
#    Key:     ***
#    CA:      ***

# JSON output
rnx nodes --json

# Use specific node for commands
rnx --node=production job list
rnx --node=staging job run echo "test"

`rnx config-help`

Show configuration file examples with embedded certificates.

rnx config-help

Examples

# Show configuration examples
rnx config-help

`rnx help`

Show help information.

rnx help [command]

Examples

# General help
rnx help

# Command-specific help
rnx help run
rnx help volume create

# Show configuration help
rnx help config

Advanced Usage

Scripting with RNX

#!/bin/bash
# Batch processing script

# Process files in parallel with resource limits
for file in *.csv; do
  rnx job run \
    --max-cpu=100 \
    --max-memory=1024 \
    --upload="$file" \
    python3 process.py "$file" &
done

# Wait for all jobs
wait

# Collect results
rnx job list --json | jq -r '.[] | select(.status == "COMPLETED") | .id' | \
while read job_uuid; do
  rnx job log "$job_uuid" > "result-$(echo $job_uuid | cut -c1-8).txt"
done

CI/CD Integration

# GitHub Actions example
- name: Run tests in Joblet
  run: |
    rnx job run \
      --max-cpu=400 \
      --max-memory=4096 \
      --volume=test-results \
      --upload=. \
      --env=CI=true \
      npm test

    # Check job status
    JOB_UUID=$(rnx job list --json | jq -r '.[-1].uuid')
    rnx job status $JOB_UUID

    # Get test results
    rnx job run --volume=test-results cat /volumes/test-results/report.xml

Monitoring and Alerting

# Monitor job failures
while true; do
  FAILED=$(rnx job list --json | jq '[.[] | select(.status == "FAILED")] | length')
  if [ $FAILED -gt 0 ]; then
    echo "Alert: $FAILED failed jobs detected"
    rnx job list --json | jq '.[] | select(.status == "FAILED")'
  fi
  sleep 60
done

Configuration Examples

Multi-node Configuration

version: "3.0"
nodes:
  default:
    address: "prod-server:50051"
    cert: |
      -----BEGIN CERTIFICATE-----
      ...
    key: |
      -----BEGIN PRIVATE KEY-----
      ...
    ca: |
      -----BEGIN CERTIFICATE-----
      ...

  staging:
    address: "staging-server:50051"
    cert: |
      -----BEGIN CERTIFICATE-----
      ...
    # ... rest of credentials

  viewer:
    address: "prod-server:50051"
    cert: |
      -----BEGIN CERTIFICATE-----
      # Viewer certificate with OU=viewer
      ...
    # ... rest of credentials

Usage with Different Nodes

# Production jobs
rnx --node=default run production-task.sh

# Staging tests
rnx --node=staging run test-suite.sh

# Read-only access
rnx --node=viewer list
rnx --node=viewer monitor status

Best Practices

Resource Limits: Always set appropriate resource limits for production jobs
Volumes: Use filesystem volumes for persistent data, memory volumes for temporary data
Networks: Create isolated networks for security-sensitive workloads
Monitoring: Use rnx monitor to track resource usage
Scheduling: Use ISO 8601 format for precise scheduling
Error Handling: Check exit codes and logs for job failures
Cleanup: Remove unused volumes and networks regularly

Troubleshooting

See Troubleshooting Guide for common issues and solutions.