Joblet is a micro-container runtime for running Linux jobs with: Process and filesystem isolation (PID namespace, chroot) Fine-grained CPU, memory, and IO throttling (cgroups v2) Secure job execution with mTLS and RBAC Built-in scheduler, SSE log streaming, and multi-core pinning Ideal for: Agentic AI Workloads (Untrusted code)
Complete guide to managing persistent and temporary storage volumes in Joblet.
Joblet volumes provide persistent and temporary storage for jobs, enabling:
/volumes/<name>Understanding how volumes are isolated between jobs and how sharing works is important for designing secure and correct workflows.
Each job runs in an isolated filesystem environment (chroot) and can only see volumes explicitly assigned to it. A job cannot access volumes belonging to other jobs.
Job A (assigned: volume-a) Job B (assigned: volume-b)
├── /volumes/ ├── /volumes/
│ └── volume-a/ ✓ accessible │ └── volume-b/ ✓ accessible
│ │
│ volume-b/ ✗ not visible │ volume-a/ ✗ not visible
Example:
# Create two separate volumes
rnx volume create data-for-job-a --size=1GB
rnx volume create data-for-job-b --size=1GB
# Job A can only see volume-a
rnx job run --volume=data-for-job-a ls /volumes/
# Output: data-for-job-a
# Job B can only see volume-b
rnx job run --volume=data-for-job-b ls /volumes/
# Output: data-for-job-b
# Job A cannot access Job B's volume - it simply doesn't exist in Job A's filesystem
This isolation is enforced at the kernel level using bind mounts and chroot, not application-level checks.
When multiple jobs are assigned the same volume, they share access to the same underlying storage. Both jobs see the same files and can read/write simultaneously.
Host Storage:
/opt/joblet/volumes/shared-data/data/ ← actual files on disk
│
├── bind mount ──→ Job A: /volumes/shared-data/
│
└── bind mount ──→ Job B: /volumes/shared-data/
Both jobs see identical content because they point to the same host directory.
Example:
# Create a shared volume
rnx volume create shared-data --size=1GB
# Job A writes a file
rnx job run --volume=shared-data bash -c '
echo "written by job A" > /volumes/shared-data/message.txt
'
# Job B can immediately read the file
rnx job run --volume=shared-data cat /volumes/shared-data/message.txt
# Output: written by job A
# Job B modifies the file
rnx job run --volume=shared-data bash -c '
echo "modified by job B" >> /volumes/shared-data/message.txt
'
# Job A sees the modification
rnx job run --volume=shared-data cat /volumes/shared-data/message.txt
# Output:
# written by job A
# modified by job B
Joblet does not provide automatic file locking or coordination between jobs accessing the same volume. This is intentional and consistent with how shared storage works in other systems (Docker volumes, Kubernetes PVCs, NFS).
Your responsibility when sharing volumes:
| Concern | Joblet’s Role | Your Responsibility |
|---|---|---|
| Volume mounting | Handled automatically | Assign correct volumes to jobs |
| File visibility | Enforced via isolation | Design which jobs share which volumes |
| Concurrent writes | Not managed | Coordinate access patterns |
| File locking | Not provided | Implement if needed (flock, advisory locks) |
| Data consistency | Not enforced | Use separate files or implement synchronization |
Safe patterns for concurrent access:
# Pattern 1: Sequential jobs (no conflict)
# Job A runs first, Job B runs after A completes
rnx job run --volume=pipeline-data producer.py
# Wait for completion...
rnx job run --volume=pipeline-data consumer.py
# Pattern 2: Separate directories per job
rnx job run --volume=shared-data --env=JOB_NAME=worker-1 bash -c '
mkdir -p /volumes/shared-data/$JOB_NAME
# Write only to own directory
echo "output" > /volumes/shared-data/$JOB_NAME/result.txt
'
# Pattern 3: Append-only logs (generally safe)
rnx job run --volume=logs bash -c '
echo "$(date) - Job started" >> /volumes/logs/activity.log
'
# Pattern 4: File-based locking (when needed)
rnx job run --volume=shared-data bash -c '
(
flock -x 200 # Acquire exclusive lock
# Critical section - safe to modify shared file
echo "update" >> /volumes/shared-data/shared-file.txt
) 200>/volumes/shared-data/.lock
'
Patterns to avoid:
# DANGEROUS: Multiple jobs writing to same file simultaneously
# Job A and Job B both running:
echo "data" > /volumes/shared/output.txt # Race condition!
# DANGEROUS: Read-modify-write without locking
count=$(cat /volumes/shared/counter.txt)
echo $((count + 1)) > /volumes/shared/counter.txt # Lost updates!
| Scenario | Behavior |
|---|---|
| Job A has volume X, Job B has volume Y | Fully isolated - neither can see the other’s volume |
| Job A and Job B both have volume X | Shared access - both see same files in real-time |
| Concurrent writes to same file | No automatic coordination - user must handle |
| Volume not assigned to job | Not visible in job’s /volumes/ directory |
Persistent disk-based storage that survives job restarts and system reboots.
Characteristics:
Use cases:
Temporary RAM-based storage (tmpfs) cleared when the volume is removed.
Characteristics:
Use cases:
# Create 1GB filesystem volume
rnx volume create mydata --size=1GB --type=filesystem
# Create 512MB memory volume
rnx volume create cache --size=512MB --type=memory
# Default type is filesystem
rnx volume create storage --size=5GB
# Supported size units
rnx volume create small --size=100MB # Megabytes
rnx volume create medium --size=5GB # Gigabytes
rnx volume create large --size=1TB # Terabytes
rnx volume create tiny --size=50KB # Kilobytes
# Precise sizes in bytes
rnx volume create exact --size=1073741824 # Bytes (1GB)
Volume names must:
# Valid names
rnx volume create user-data --size=1GB
rnx volume create app_cache_v2 --size=500MB
rnx volume create Dataset2024 --size=10GB
# Invalid names (will fail)
rnx volume create 123data --size=1GB # Starts with number
rnx volume create my.data --size=1GB # Contains period
rnx volume create "my data" --size=1GB # Contains space
# Mount single volume
rnx job run --volume=mydata ls -la /volumes/mydata
# Mount multiple volumes
rnx job run \
--volume=input-data \
--volume=output-data \
--volume=cache \
python3 process.py
# Volume is mounted read-write by default
rnx job run --volume=config cat /volumes/config/settings.json
# Write to volume
rnx job run --volume=results bash -c '
echo "Processing results" > /volumes/results/output.txt
date >> /volumes/results/output.txt
'
# Read from volume in separate job
rnx job run --volume=results cat /volumes/results/output.txt
# Copy files to volume
rnx job run --volume=backup --upload=data.tar.gz \
cp data.tar.gz /volumes/backup/
# Process files in volume
rnx job run --volume=dataset python3 -c "
import os
files = os.listdir('/volumes/dataset')
print(f'Found {len(files)} files')
"
All volumes are mounted under /volumes/ with their name:
# Volume 'mydata' → /volumes/mydata
# Volume 'cache' → /volumes/cache
# Volume 'ml-models' → /volumes/ml-models
# List all mounted volumes
rnx job run --volume=data1 --volume=data2 ls -la /volumes/
# List all volumes
rnx volume list
# Output format:
# NAME SIZE TYPE CREATED
# mydata 1GB filesystem 2025-08-03 10:00:00
# cache 512MB memory 2025-08-03 10:05:00
# JSON output
rnx volume list --json
Since there’s no built-in usage monitoring, use job commands to check volume usage:
# Check space usage in filesystem volume
rnx job run --volume=mydata df -h /volumes/mydata
# Detailed usage
rnx job run --volume=mydata du -sh /volumes/mydata/*
# Find large files
rnx job run --volume=logs \
find /volumes/logs -type f -size +100M -exec ls -lh {} \;
# Remove single volume
rnx volume remove mydata
# Note: Volume must not be in use by any active jobs
# If removal fails due to active jobs, stop the jobs first
# 1. Create volume for persistent storage
rnx volume create ml-checkpoints --size=50GB
# 2. Save model checkpoints during training
rnx job run \
--volume=ml-checkpoints \
--upload=train.py \
--max-cpu=800 \
--max-memory=16384 \
python3 train.py --checkpoint-dir=/volumes/ml-checkpoints
# 3. Resume training from checkpoint in separate job
rnx job run \
--volume=ml-checkpoints \
--upload=train.py \
python3 train.py --resume=/volumes/ml-checkpoints/latest.pth
# 4. Export final model (download logs to get file)
JOB_ID=$(rnx job run \
--volume=ml-checkpoints \
--json \
bash -c 'cat /volumes/ml-checkpoints/best_model.pth' | jq -r .id)
# Wait for job completion then download
sleep 5
rnx job log $JOB_ID > model.pth
# Job 1: Generate data
rnx job run --volume=shared-data python3 -c "
import json
data = {'status': 'processed', 'count': 1000}
with open('/volumes/shared-data/status.json', 'w') as f:
json.dump(data, f)
"
# Job 2: Read shared data (runs after Job 1 completes)
rnx job run --volume=shared-data python3 -c "
import json
with open('/volumes/shared-data/status.json', 'r') as f:
data = json.load(f)
print(f'Status: {data[\"status\"]}, Count: {data[\"count\"]}')
"
Joblet doesn’t have built-in backup commands, but you can implement backup workflows using job commands:
# Create backup job
BACKUP_JOB=$(rnx job run --json \
--volume=important-data \
tar -czf /work/backup.tar.gz -C /volumes/important-data . \
| jq -r .id)
# Wait for completion
sleep 5
# Download backup by getting job logs
rnx job log $BACKUP_JOB > important-data-backup.tar.gz
# Restore to new volume
rnx volume create restored-data --size=10GB
rnx job run \
--volume=restored-data \
--upload=important-data-backup.tar.gz \
tar -xzf important-data-backup.tar.gz -C /volumes/restored-data
Test volume performance using job commands:
# Test write performance
rnx job run --volume=perf-test dd \
if=/dev/zero \
of=/volumes/perf-test/test.dat \
bs=1M count=1000 \
conv=fdatasync
# Test read performance
rnx job run --volume=perf-test dd \
if=/volumes/perf-test/test.dat \
of=/dev/null \
bs=1M
# Check volume mount and filesystem type
rnx job run --volume=perf-test bash -c '
mount | grep /volumes/perf-test
df -T /volumes/perf-test
'
# Memory volumes are much faster for I/O operations
rnx job run --volume=mem-cache --max-memory=2048 python3 -c "
import time
import os
# Write test
start = time.time()
with open('/volumes/mem-cache/test.dat', 'wb') as f:
f.write(os.urandom(500 * 1024 * 1024)) # 500MB
write_time = time.time() - start
# Read test
start = time.time()
with open('/volumes/mem-cache/test.dat', 'rb') as f:
data = f.read()
read_time = time.time() - start
print(f'Write: {500/write_time:.2f} MB/s')
print(f'Read: {500/read_time:.2f} MB/s')
"
# Use memory volumes for temporary data
rnx volume create temp-work --size=2GB --type=memory
# Process large dataset with staging pattern
rnx job run \
--volume=source-data \
--volume=temp-work \
--volume=results \
bash -c '
# Copy input to fast memory volume
cp /volumes/source-data/* /volumes/temp-work/
# Process in memory
process_data.py --input=/volumes/temp-work --output=/volumes/results
# Clean up temporary files
rm /volumes/temp-work/*
'
# Regular cleanup using job scheduling
rnx job run --schedule="168h" --volume=logs bash -c '
find /volumes/logs -name "*.tmp" -mtime +7 -delete
find /volumes/logs -name "*.log" -mtime +30 -delete
'
# Start with reasonable sizes and monitor usage
rnx volume create test-vol --size=1GB
# Monitor usage regularly
rnx job run --volume=test-vol df -h /volumes/test-vol
# Create larger volume if needed (no resize capability)
rnx volume create test-vol-large --size=10GB
# Migrate data manually
rnx job run \
--volume=test-vol \
--volume=test-vol-large \
cp -r /volumes/test-vol/* /volumes/test-vol-large/
# Remove old volume after migration
rnx volume remove test-vol
# Use descriptive names with versioning
rnx volume create user-data-v1 --size=5GB
rnx volume create ml-models-2024 --size=50GB
rnx volume create cache-layer-prod --size=2GB --type=memory
# Environment-specific naming
rnx volume create dev-database --size=10GB
rnx volume create staging-uploads --size=20GB
rnx volume create prod-backups --size=100GB
# Create directory structure in jobs
rnx job run --volume=project-data bash -c '
mkdir -p /volumes/project-data/{input,output,temp,logs}
mkdir -p /volumes/project-data/archives/$(date +%Y/%m)
'
# Use subdirectories for organization
rnx job run --volume=ml-data bash -c '
mkdir -p /volumes/ml-data/{datasets,models,checkpoints,metrics}
'
# Create cleanup script
cat > cleanup.sh << 'EOF'
#!/bin/bash
# Remove old temporary files
find /volumes/temp-data -name "*.tmp" -mtime +7 -delete
# Compress old logs
find /volumes/logs -name "*.log" -mtime +30 -exec gzip {} \;
# Remove empty directories
find /volumes/data -type d -empty -delete
EOF
# Schedule regular cleanup (requires job scheduling)
rnx job run \
--schedule="168h" \
--volume=temp-data \
--volume=logs \
--volume=data \
--upload=cleanup.sh \
bash cleanup.sh
# Handle sensitive data with encryption in jobs
rnx volume create secrets --size=100MB
# Store encrypted data
rnx job run --volume=secrets --env=ENCRYPTION_KEY=xxx bash -c '
echo "sensitive data" | openssl enc -aes-256-cbc -k "$ENCRYPTION_KEY" \
> /volumes/secrets/data.enc
'
# Retrieve and decrypt
rnx job run --volume=secrets --env=ENCRYPTION_KEY=xxx bash -c '
openssl enc -aes-256-cbc -d -k "$ENCRYPTION_KEY" \
< /volumes/secrets/data.enc
'
1. Volume Creation Fails
# Error: "failed to create volume: operation not permitted"
# Solution: Check server has proper permissions
# Ensure joblet runs with necessary privileges for loop device setup
2. Volume Not Found
# Error: "volume mydata not found"
# Check volume exists
rnx volume list
# Recreate if needed
rnx volume create mydata --size=1GB
3. Out of Space
# Error: "No space left on device"
# Check volume usage
rnx job run --volume=full-vol df -h /volumes/full-vol
# Create larger volume (no resize capability)
rnx volume create full-vol-v2 --size=20GB
# Migrate data
rnx job run --volume=full-vol --volume=full-vol-v2 \
cp -r /volumes/full-vol/* /volumes/full-vol-v2/
# Remove old volume
rnx volume remove full-vol
4. Permission Denied
# Error: "Permission denied"
# Volumes are owned by job user
# Fix permissions within job
rnx job run --volume=data bash -c '
# Check current permissions
ls -la /volumes/data
# Fix if needed (be careful with chmod 777)
chmod -R 755 /volumes/data
'
5. Memory Volume Full
# Memory volumes limited by available RAM and specified size
# Check system memory and volume size
rnx job run --volume=mem-vol df -h /volumes/mem-vol
# Use smaller memory volume or switch to filesystem volume
rnx volume create cache-small --size=256MB --type=memory
6. Volume Removal Blocked
# Error: Volume is in use by active jobs
# List running jobs
rnx job list
# Stop jobs using the volume
rnx job stop <job-id>
# Then remove volume
rnx volume remove mydata
# Check volume mount status
rnx job run --volume=debug-vol mount | grep volumes
# Verify volume permissions and ownership
rnx job run --volume=debug-vol ls -la /volumes/
# Test write access
rnx job run --volume=debug-vol bash -c '
touch /volumes/debug-vol/test.txt
echo "Write test successful"
rm /volumes/debug-vol/test.txt
'
# Check filesystem type (for filesystem volumes)
rnx job run --volume=debug-vol stat -f /volumes/debug-vol
# For memory volumes, verify tmpfs mount
rnx job run --volume=mem-vol mount | grep tmpfs
# Check volume metadata (stored in volume directory)
rnx job run --volume=debug-vol bash -c '
if [ -f /volumes/debug-vol/.joblet_volume_meta.json ]; then
cat /volumes/debug-vol/.joblet_volume_meta.json
else
echo "No volume metadata found"
fi
'
# Verify volume size limits
rnx job run --volume=debug-vol df -h /volumes/debug-vol
# Create volume for PostgreSQL
rnx volume create postgres-data --size=50GB
# Run PostgreSQL with persistent storage
rnx job run \
--volume=postgres-data \
--env=POSTGRES_PASSWORD=secret \
--env=PGDATA=/volumes/postgres-data \
--network=db-network \
--runtime=postgres:latest \
postgres
# Create build cache volume (memory for speed)
rnx volume create build-cache --size=2GB --type=memory
# Use for faster builds
rnx job run \
--volume=build-cache \
--upload=./src \
--env=MAVEN_CACHE_DIR=/volumes/build-cache/maven \
--runtime=java:17 \
bash -c "
mkdir -p /volumes/build-cache/maven
mvn -Dmaven.repo.local=/volumes/build-cache/maven install
mvn -Dmaven.repo.local=/volumes/build-cache/maven package
"
# Create volumes for pipeline stages
rnx volume create raw-data --size=100GB
rnx volume create processed-data --size=50GB
rnx volume create final-results --size=10GB
# Stage 1: Ingest data
rnx job run --volume=raw-data --upload=ingest_data.sh bash ingest_data.sh
# Stage 2: Process (runs after stage 1)
rnx job run \
--volume=raw-data \
--volume=processed-data \
--upload=process_data.py \
python3 process_data.py
# Stage 3: Analysis (runs after stage 2)
rnx job run \
--volume=processed-data \
--volume=final-results \
--upload=analyze_results.py \
python3 analyze_results.py
rnx volume list for volume informationdf and duCheck the project roadmap for upcoming volume management features.