Native Linux Microcontainers

Joblet is a micro-container runtime for running Linux jobs with: Process and filesystem isolation (PID namespace, chroot) Fine-grained CPU, memory, and IO throttling (cgroups v2) Secure job execution with mTLS and RBAC Built-in scheduler, SSE log streaming, and multi-core pinning Ideal for: Agentic AI Workloads (Untrusted code)

Project maintained by ehsaniara Hosted on GitHub Pages — Theme by mattgraham

Runtime Builder Guide

Complete guide for building custom runtimes for Joblet using the declarative YAML-based builder.

Overview
Quick Start
Runtime YAML Specification
Build Process
Supported Languages
Examples
Hooks
Troubleshooting

Overview

Joblet uses a declarative runtime builder that enables:

Declarative configuration - Define runtimes with simple YAML files
Remote building - Build executes on the joblet server, not locally
14-phase build process - Comprehensive build pipeline with progress streaming
Language profiles - Pre-defined packages for Python, Java, Node.js, Go, Rust
Custom hooks - Pre and post-install scripts for customization
Multi-platform support - Ubuntu, RHEL, Amazon Linux on amd64/arm64

Quick Start

1. Create a runtime.yaml

schema_version: "1.0"
name: python-3.11-ml
version: 1.0.0
description: Python 3.11 with machine learning packages

base:
  language: python
  version: "3.11"

pip:
  - numpy
  - pandas
  - scikit-learn

environment:
  PYTHONUNBUFFERED: "1"

2. Validate the specification

rnx runtime validate ./runtime.yaml

3. Build the runtime

# Build on remote server
rnx runtime build ./runtime.yaml

# Or with verbose output
rnx runtime build -v ./runtime.yaml

# Validate before building (comprehensive server-side validation)
rnx runtime validate ./runtime.yaml

4. Use the runtime

rnx job run --runtime=python-3.11-ml python -c "import numpy; print(numpy.__version__)"

Runtime YAML Specification

Complete Schema

# Required: Schema version (must be "1.0")
schema_version: "1.0"

# Required: Runtime name (lowercase, hyphens, dots, max 64 chars)
name: my-runtime

# Required: Semantic version (X.Y.Z)
version: 1.0.0

# Required: Description (max 256 chars)
description: My custom runtime environment

# Required: Base language configuration
base:
  language: python  # python | java | node | go | rust
  version: "3.11"   # Language-specific version

# Optional: Python packages to install via pip
pip:
  - numpy
  - pandas>=2.0.0
  - scikit-learn==1.3.2

# Optional: Pip options (e.g., custom index)
pip_options: "--index-url https://pypi.example.com/simple"

# Optional: Node.js packages to install via npm
npm:
  - express
  - lodash

# Optional: Additional library patterns to copy to isolated runtime
# Use this for system libraries installed via pre_install hooks
libraries:
  - libopenblas*
  - libgfortran*

# Optional: Environment variables
environment:
  PYTHONUNBUFFERED: "1"
  MY_VAR: "value"

# Optional: Runtime requirements
requirements:
  gpu: false
  cuda_version: "12.0"

# Optional: Supported platforms (default: all)
platforms:
  - ubuntu-amd64
  - ubuntu-arm64
  - rhel-amd64

# Optional: Build hooks
hooks:
  timeout: "30m"  # Hook timeout (default: 20m)
  pre_install: |
    echo "Running before package installation"
  post_install: |
    echo "Running after package installation"

Field Reference

Field	Required	Description
`schema_version`	Yes	Must be “1.0”
`name`	Yes	Runtime identifier (lowercase, hyphens, dots)
`version`	Yes	Semantic version X.Y.Z
`description`	Yes	Human-readable description (max 256 chars)
`base.language`	Yes	Base language: python, java, node, go, rust
`base.version`	Yes	Language version (e.g., “3.11”, “21”)
`pip`	No	List of Python packages
`pip_options`	No	Additional pip install options
`npm`	No	List of Node.js packages
`libraries`	No	Additional library patterns to copy (e.g., `libopenblas*`)
`environment`	No	Environment variables
`requirements`	No	GPU and memory requirements
`platforms`	No	Supported platforms
`hooks`	No	Pre/post-install scripts

Build Process

The runtime builder executes a 14-phase pipeline:

Phase	Name	Description
1	Parse & Validate	Parse YAML and validate schema
2	Detect Platform	Identify OS, architecture, package manager
3	Check Disk Space	Ensure sufficient disk space (1GB minimum)
4	Validate Packages	Verify packages are available
5	Prepare Directories	Create runtime directory structure
6	Pre-install Hook	Run pre-install script (if defined)
7	Install Base	Install base language packages
8	Install Packages	Install pip/npm packages
9	Post-install Hook	Run post-install script (if defined)
10	Copy Binaries	Copy language binaries to isolated dir
11	Copy Libraries	Copy required shared libraries
12	Copy Configuration	Copy SSL certs, resolv.conf, etc.
13	Generate Config	Generate runtime.yml for joblet
14	Validate Build	Verify build integrity

Build Output

Runtimes are installed to: /opt/joblet/runtimes/{name}/{version}/

Directory structure:

/opt/joblet/runtimes/python-3.11-ml/1.0.0/
├── runtime.yml           # Generated runtime configuration
└── isolated/             # Isolated filesystem
    ├── usr/
    │   ├── bin/          # Binaries (python3, pip, etc.)
    │   └── lib/          # Libraries
    ├── lib/              # System libraries
    ├── etc/              # Configuration files
    └── tmp/              # Temporary directory

Generated `runtime.yml`

The build generates a runtime.yml with comprehensive metadata:

name: python-3.11-ml
language: python
language_version: "3.11"
version: 1.0.0
description: Python 3.11 with machine learning packages
mounts: [...]
environment:
  PYTHONUNBUFFERED: "1"
  PATH: /usr/local/bin:/usr/bin:/bin
packages:
  - numpy==1.26.2
  - pandas==2.1.3
libraries:
  - libopenblas*
  - libgfortran*
requirements:
  architectures: [amd64]
build_info:
  built_at: "2025-12-22T17:43:00Z"
  platform: ubuntu-amd64

View runtime details:

rnx runtime info python-3.11-ml

Supported Languages

Python

base:
  language: python
  version: "3.11"  # 3.9, 3.10, 3.11, 3.12

pip:
  - numpy
  - pandas

Installed packages: python3.X, python3.X-dev, python3.X-venv, python3-pip, libssl-dev, zlib1g-dev, libffi-dev

Java

base:
  language: java
  version: "21"  # 11, 17, 21

# No additional packages needed - JVM is included

Installed packages: openjdk-X-jdk, ca-certificates

Node.js

base:
  language: node
  version: "20"  # 18, 20

npm:
  - express
  - lodash

Installed packages: nodejs, npm

Go

base:
  language: go
  version: "1.21"

Installed packages: golang, git

Rust

base:
  language: rust
  version: "1.75"

Installed packages: rustc, cargo

Examples

Python ML Runtime

schema_version: "1.0"
name: python-3.11-ml
version: 1.0.0
description: Python 3.11 with machine learning packages

base:
  language: python
  version: "3.11"

pip:
  - numpy
  - pandas
  - scikit-learn
  - matplotlib

# Copy system libraries needed by ML packages
libraries:
  - libopenblas*
  - libgfortran*
  - libgomp*

environment:
  PYTHONUNBUFFERED: "1"

hooks:
  pre_install: |
    apt-get install -y libopenblas-dev || yum install -y openblas-devel

Java 21 Runtime

schema_version: "1.0"
name: openjdk-21
version: 1.0.0
description: OpenJDK 21 runtime

base:
  language: java
  version: "21"

environment:
  JAVA_OPTS: "-Xmx512m"

Python Analytics with Custom Index

schema_version: "1.0"
name: python-analytics
version: 1.0.0
description: Python for data analytics

base:
  language: python
  version: "3.11"

pip:
  - pandas
  - matplotlib
  - seaborn

pip_options: "--index-url https://pypi.company.com/simple"

environment:
  PYTHONUNBUFFERED: "1"

Runtime with Hooks

schema_version: "1.0"
name: python-custom
version: 1.0.0
description: Python with custom setup

base:
  language: python
  version: "3.11"

pip:
  - numpy

hooks:
  timeout: "30m"
  pre_install: |
    echo "Configuring system before installation"
    # Custom pre-install commands
  post_install: |
    echo "Running post-installation tasks"
    python3.11 -c "import numpy; print('NumPy version:', numpy.__version__)"

Hooks

Pre-install Hook

Runs after base package installation but before pip/npm packages.

Use cases:

Configure system settings
Install additional system dependencies
Set up authentication for private package repositories

Post-install Hook

Runs after all packages are installed.

Use cases:

Verify installation
Compile assets
Run tests
Clean up temporary files

Hook Environment

Hooks execute with:

Root privileges (sudo)
Working directory: runtime’s isolated directory
Access to installed packages
Timeout enforcement (default: 20 minutes)

Troubleshooting

Build Fails at Package Validation

Error: package not found: python3.11-dev

Solution: The package name might differ on your platform. Check available packages:

apt-cache search python3.11

Pip Installation Fails

Error: pip install failed: Could not find a version that satisfies the requirement

Solutions:

Check package name spelling
Verify package exists on PyPI
Check network connectivity on the server
Use pip_options for custom index

Permission Denied

Error: failed to create directory: permission denied

Solution: Ensure joblet service has write access to /opt/joblet/runtimes/

Disk Space Error

Error: insufficient disk space: 500 MB available, need at least 1024 MB

Solution: Free up disk space on the server or expand the volume.

Hook Timeout

Error: pre-install hook timed out after 20m

Solution: Increase timeout in hooks configuration:

hooks:
  timeout: "60m"
  pre_install: |
    # Long-running installation

Best Practices

Version pinning - Pin package versions for reproducibility: ```yaml pip:
- numpy==1.26.2
- pandas>=2.0.0,<3.0.0 ```
Minimal dependencies - Only include packages you need
Test locally first - Validate YAML before building:
```
rnx runtime validate ./runtime.yaml
```
Use validate - Comprehensive server-side validation before building:
```
rnx runtime validate ./runtime.yaml
```
Semantic versioning - Use proper version numbers for your runtimes

Questions? Check Troubleshooting or create an issue on GitHub.