Native Linux Microcontainers

Joblet is a micro-container runtime for running Linux jobs with: Process and filesystem isolation (PID namespace, chroot) Fine-grained CPU, memory, and IO throttling (cgroups v2) Secure job execution with mTLS and RBAC Built-in scheduler, SSE log streaming, and multi-core pinning Ideal for: Agentic AI Workloads (Untrusted code)

---

Project maintained by ehsaniara Hosted on GitHub Pages — Theme by mattgraham

Service-Based Isolation Security Analysis

Executive Summary

This document provides a comprehensive security analysis of the dual chroot isolation system, examining the implementation to ensure complete compartmentalization between production jobs and runtime builds with no security leaks.

Architecture Overview

Service-Based Job Routing

The system uses automatic service-based job type detection to route jobs to appropriate isolation levels:

flowchart TD
    N1["JobService API"] --> N2["JobType: standard"]
    N2 --> N3["Production Isolation<br/>(Minimal Chroot)"]
    N4["RuntimeService API"] --> N5["JobType: runtime-build"]
    N5 --> N6["Builder Isolation<br/>(Builder Chroot)"]

Key Implementation:

• Source: /home/jay/joblet/internal/modes/isolation/isolation.go:119-136
• Detection: jobType := i.platform.Getenv("JOB_TYPE")
• Routing: isBuilderJob := jobType == "runtime-build"

Isolation Mechanisms Analysis

1. Production Jobs (JobService → standard jobs)

Isolation Method: Minimal chroot with restricted filesystem access

Implementation: jobFS.Setup() in filesystem/isolator.go

Security Boundaries:

Production Chroot: /opt/joblet/jobs/{JOB_ID}/
├── bin/          # Minimal binaries (read-only bind mount from host)
├── lib/          # Required libraries (read-only bind mount from host) 
├── usr/          # Selected /usr contents (read-only bind mount from host)
├── etc/          # Basic config files (isolated copies)
├── work/         # Job workspace (writable, isolated)
├── tmp/          # Isolated tmp: /tmp/job-{JOB_ID}/ (writable, isolated)
└── [runtime]/    # Optional runtime mount (read-only bind mount)

Key Security Features:

• Read-only host mounts: System directories mounted as read-only
• Isolated tmp: Each job gets /tmp/job-{JOB_ID}/
• Minimal attack surface: Only essential binaries available
• No package managers: Cannot install software
• No build tools: Cannot compile or modify system

2. Runtime Build Jobs (RuntimeService → runtime-build jobs)

Isolation Method: OverlayFS build environment layered over the host root

Implementation: IsolatedEnvironment.Setup() in pkg/builder/isolation.go

Security Boundaries:

The live rnx runtime build path mounts an OverlayFS in which the host root (/) is the read-only lower layer, an upper layer captures every write, and the merged view is what the build process actually sees:

OverlayFS build environment:
  lowerdir = /                # Host root, READ-ONLY (writes never reach it)
  upperdir = <build>/upper    # All writes land here, isolated from the host
  workdir  = <build>/work     # OverlayFS bookkeeping (required by overlayfs)
  merged   = <build>/merged   # Unified view the build process operates on

Because the host root is the lower layer, the build has full read access to the host OS environment for compilation but cannot modify any host file - writes are redirected into the upper layer and discarded (or promoted into the runtime tree) at the end of the build. Package managers (apt, pip, npm) work because runtime builds share host networking and have working DNS.

Key Security Features:

• Host root as read-only lower layer: The build sees the full OS but every write is redirected into the upper layer; the host filesystem is never modified.
• Writes isolated in the upper layer: Build output is captured separately; only the produced runtime tree is promoted.
• Isolated tmp: Each build gets its own temporary space.
• Same process isolation: PID, mount, IPC, UTS, and cgroup namespaces (runtime builds share host networking).

Critical Security Analysis

1. Recursion Prevention - SECURE

Problem: Builder chroot could mount /opt/joblet/jobs/ creating infinite recursion

Solution: Complete exclusion of /opt/joblet/ except controlled runtimes directory

Implementation Analysis:

// Source: filesystem/isolator.go:mountOptDirectory()
func (f *JobFilesystem) mountOptDirectory(hostOptPath, targetOptPath string) error {
    for _, entry := range optEntries {
        dirName := entry.Name()
        
        // Skip joblet directory to prevent recursion
        if dirName == "joblet" {
            log.Debug("skipping /opt/joblet to prevent recursion")
            continue  // - CRITICAL SECURITY CHECK
        }
        // ... mount other /opt contents
    }
}

// Source: filesystem/isolator.go:mountRuntimesDirectory()
func (f *JobFilesystem) mountRuntimesDirectory() error {
    hostRuntimesPath := "/opt/joblet/runtimes"
    targetRuntimesPath := filepath.Join(f.RootDir, "opt", "joblet", "runtimes")
    
    // Only mount runtimes directory - NOT the entire /opt/joblet
    if err := f.platform.Mount(hostRuntimesPath, targetRuntimesPath, "", uintptr(syscall.MS_BIND), ""); err != nil {
        return fmt.Errorf("failed to bind mount runtimes directory: %w", err)
    }
    // - PRECISE MOUNTING - only runtimes directory accessible
}

Security Guarantee: Builder jobs cannot access:

• /opt/joblet/jobs/ (where other jobs run)
• /opt/joblet/config/ (system configuration)
• /opt/joblet/logs/ (system logs)
• Any other /opt/joblet/ subdirectories

2. Filesystem Write Isolation - SECURE

Concern: Builder jobs could contaminate production job filesystems

Analysis:

• Builder writes limited to: /opt/joblet/runtimes/ and /tmp/job-{BUILD_ID}/
• Production job isolation: Each job in separate /opt/joblet/jobs/{JOB_ID}/
• No cross-job access: Build IDs and Job IDs are different UUID namespaces

Implementation Verification:

// Builder chroot creation
func (i *Isolator) CreateBuilderFilesystem(jobID string) (*JobFilesystem, error) {
    // Same directory structure as production jobs
    jobRootDir := filepath.Join(i.config.BaseDir, jobID)  // /opt/joblet/jobs/{BUILD_ID}/
    jobTmpDir := strings.Replace(i.config.TmpDir, "{JOB_ID}", jobID, -1)  // /tmp/job-{BUILD_ID}/
    
    // - BUILD_ID != JOB_ID - complete separation
}

3. Service-Based Routing Security - SECURE

Concern: Jobs could be misrouted between isolation levels

Analysis of Detection Chain:

1. Service Layer (RuntimeService vs JobService) sets JobType
2. Core Layer passes JobType to environment: JOB_TYPE=runtime-build
3. Isolation Layer reads environment: jobType := i.platform.Getenv("JOB_TYPE")
4. Routing Decision: isBuilderJob := jobType == "runtime-build"

Security Properties:

• Explicit Service Control: Only RuntimeService can set runtime-build type
• No User Override: Users cannot manually specify job type via rnx job run
• Environment Isolation: Job type passed through secure environment variables
• Fail-Safe Default: Unknown/missing job types default to production isolation

4. Process Isolation Parity - SECURE

Both job types use the same core process isolation:

Shared Security Features:

• PID Namespace: Each job is PID 1 in its own namespace
• Mount Namespace: Isolated mount table
• Cgroups: Resource limits (CPU, memory, I/O)
• Privilege drop, not UID remapping: No user namespace (CLONE_NEWUSER) is used - it is deliberately omitted because it breaks the mount operations Joblet relies on. Instead, isolation of the user context comes from dropping privileges to uid/gid 65534 (nobody) after isolation setup but before the job command executes.

Where they differ: the Network Namespace is not shared behavior - production jobs get their own network namespace (bridge networking), while runtime-build jobs share the host network stack for internet access during the build.

Implementation Verification:

// Source: filesystem/isolator.go:validateJobContext()
func (i *Isolator) validateJobContext() error {
    // Both job types must pass same safety checks
    jobID := i.platform.Getenv("JOB_ID")
    if jobID == "" {
        return fmt.Errorf("not in job context - JOB_ID not set")  // - SAFETY CHECK
    }

    if i.platform.Getpid() != 1 {
        return fmt.Errorf("not in isolated PID namespace - refusing filesystem isolation")  // - NAMESPACE VERIFICATION
    }
    return nil
}

5. Temporary Space Isolation - SECURE

Concern: Shared /tmp could leak data between job types

Analysis:

• Production jobs: /tmp/job-{JOB_ID}/
• Builder jobs: /tmp/job-{BUILD_ID}/
• Complete separation: Different UUID namespaces prevent collisions
• Automatic cleanup: Each job’s temp space removed after completion

Attack Vector Analysis

1. Container Escape Attempts

Attack: Malicious production job attempts to access builder chroot

Mitigation:

- - Read-only host mounts: Cannot modify system binaries - - No package managers: Cannot install escape tools - - Minimal attack surface: Limited binaries available - - Process isolation: Cannot see other namespaces

2. Privilege Escalation

Attack: Builder job attempts to gain root access

Mitigation:

- - Same user context: Runs as same unprivileged user as production jobs - - No setuid binaries: Host mounts are read-only - - Cgroup limits: Resource restrictions prevent DoS attacks - - Network isolation: Cannot access privileged network services

3. Data Exfiltration Between Job Types

Attack: Production job attempts to read builder artifacts

Mitigation:

- - Separate filesystem trees: No shared writable space - - Isolated temp directories: Different UUID-based paths - - Read-only runtime access: Production jobs can only read completed runtimes - - No job directory access: Cannot access other jobs’ workspaces

4. Resource Exhaustion

Attack: One job type consumes resources affecting the other

Mitigation:

- - Same cgroup limits: Both job types subject to identical resource controls - - Independent quotas: Each job gets separate CPU/memory/I/O allocation - - Process limits: Same process count restrictions - - Cleanup guarantees: Failed jobs are cleaned up automatically

Recommended Additional Hardening

While the current implementation is secure, consider these enhancements:

1. SELinux/AppArmor Integration

# Example SELinux policy for additional MAC
type joblet_production_t;
type joblet_builder_t;
type_transition initrc_t joblet_exec_t:process joblet_production_t;

2. Seccomp Syscall Filtering

// Different syscall profiles for each job type
productionSeccompProfile := &seccomp.Profile{
    AllowedSyscalls: []string{"read", "write", "open", "close"}, // minimal set
}
builderSeccompProfile := &seccomp.Profile{
    AllowedSyscalls: append(productionSeccompProfile.AllowedSyscalls, 
        "mount", "umount2"), // additional for compilation
}

3. Runtime Verification

// Verify runtime integrity before production use
func (r *RuntimeManager) VerifyRuntime(runtimePath string) error {
    // Check signatures, validate structure, scan for malware
}

Runtime Isolation Security Enhancement

Critical Issue Identified and Resolved

Previous Security Gap: Runtime installations performed in builder chroot would mount host OS paths directly to production jobs, potentially exposing the entire host filesystem.

Example of Previous Risk:

# Previous runtime.yml (INSECURE)
mounts:
  - source: "usr/lib/jvm/java-21-openjdk-amd64"  # Points to HOST filesystem
    target: "/usr/lib/jvm/java-21-openjdk-amd64"
    readonly: true

Enhanced Isolation Solution ✅

Runtime Cleanup Process: After successful runtime build, a cleanup phase creates an isolated, self-contained runtime structure:

# In builder chroot during setup:
/opt/joblet/runtimes/openjdk-21/1.0.0/
├── isolated/                    # NEW: Isolated runtime files
│   ├── usr/lib/jvm/            # Copied from host, not mounted
│   ├── usr/bin/                # Java binaries (isolated copies)
│   └── etc/ssl/certs/          # CA certificates (isolated copies)
├── runtime.yml                 # Updated with isolated paths
└── runtime.yml.original        # Backup of original config

Updated Runtime Configuration:

# New runtime.yml (SECURE)
mounts:
  - source: "isolated/usr/lib/jvm/java-21-openjdk-amd64"  # Points to ISOLATED copy
    target: "/usr/lib/jvm/java-21-openjdk-amd64"
    readonly: true

Security Benefits

1. Complete Host Isolation: Production jobs cannot access host OS paths
2. Self-Contained Runtimes: All runtime dependencies copied to isolated structure
3. Zero Host Contamination Risk: Runtime files are isolated copies, not host mounts
4. Tamper Protection: Read-only mounts of isolated runtime files
5. Auditability: Original configuration preserved for forensics

Implementation Status

- - Runtime Cleanup System: Implemented in internal/joblet/core/runtime_cleanup.go - - Setup Script Integration: Updated runtimes/openjdk-21/setup-ubuntu-amd64.sh - - Isolated Structure: Creates isolated/ directory with runtime files - - Configuration Update: Rewrites runtime.yml with isolated paths

• 🔄 Additional Runtimes: Python and Node.js runtimes need similar updates

Security Conclusion

VERDICT: - SECURE - No isolation leaks detected + Enhanced runtime isolation

The service-based dual chroot implementation provides:

1. Complete Filesystem Isolation: No shared writable space between job types
2. Precise Mount Controls: Surgical mounting prevents recursion and contamination
3. Robust Process Boundaries: Identical namespace and cgroup isolation
4. Fail-Safe Routing: Conservative defaults and explicit service-level controls
5. Defense in Depth: Multiple security layers with no single points of failure

The architecture successfully separates production workload execution from administrative runtime building while maintaining security isolation equivalent to running completely separate container environments.