Native Linux Microcontainers

Joblet is a micro-container runtime for running Linux jobs with: Process and filesystem isolation (PID namespace, chroot) Fine-grained CPU, memory, and IO throttling (cgroups v2) Secure job execution with mTLS and RBAC Built-in scheduler, SSE log streaming, and multi-core pinning Ideal for: Agentic AI Workloads (Untrusted code)

Project maintained by ehsaniara Hosted on GitHub Pages — Theme by mattgraham

Joblet Architecture

Overview

Joblet is a high-performance job execution system with a monorepo structure containing two main services and a CLI tool.

Repository Structure

joblet/
├── cmd/
│   ├── joblet/          # Main service daemon
│   └── rnx/             # CLI client
├── internal/
│   ├── joblet/          # Main service implementation
│   ├── rnx/             # CLI implementation
│   └── proto/           # Internal protocol buffers (IPC, persist)
├── persist/             # Persistence service (sub-module)
│   ├── cmd/persist/
│   ├── internal/
│   └── go.mod           # Separate Go module
├── state/               # State persistence service (sub-module)
│   ├── cmd/state/       # State service binary
│   ├── internal/
│   │   ├── storage/     # Backend interface (Memory, DynamoDB)
│   │   └── ipc/         # IPC server
│   └── go.mod           # Separate Go module (AWS SDK dependencies)
├── api/
│   └── gen/             # Generated public API (from external joblet-proto)
├── pkg/                 # Shared packages
│   ├── config/          # Configuration management
│   ├── logger/          # Logging utilities
│   ├── security/        # TLS/mTLS support
│   └── client/          # gRPC clients
└── scripts/             # Build and deployment scripts

Components

1. Joblet (Main Service)

2. Persist (Persistence Service)

3. State (Job State Persistence Service)

4. RNX (CLI Client)

Communication Architecture

flowchart TD
    N1["RNX Client"] <-->|"gRPC :50051"| N2["Joblet Service (main)"]
    N2 -->|"Unix Socket IPC (logs/metrics)"| N3["Persist (logs/metrics)"]
    N2 -->|"Unix Socket IPC (job state)"| N4["State (job state)"]
    N3 -->|gRPC| N5["RNX Queries (historical data)"]
    N4 --> N6["Storage Backend (Memory/DynamoDB)"]

Subprocess Management

Joblet automatically spawns and supervises two subprocesses:

Both subprocesses are managed with:

Communication Protocols

  1. Public API (RNX ↔ Joblet)
    • Protocol: gRPC
    • Proto: joblet.proto (external, versioned)
    • Source: github.com/ehsaniara/joblet-proto
    • Generated: api/gen/
  2. Internal IPC (Joblet ↔ Persist)
    • Protocol: Unix Domain Socket
    • Proto: internal/proto/ipc.proto
    • Messages: Logs, Metrics
    • Socket: /opt/joblet/run/persist.sock
  3. Query API (RNX ↔ Persist)
    • Protocol: gRPC
    • Proto: internal/proto/persist.proto
    • Services: QueryLogs, QueryMetrics

Protocol Buffer Organization

External Proto (Public API)

Internal Protos (Monorepo)

Configuration

Unified Config File

Both services use: /opt/joblet/config/joblet-config.yml

# Main service config (top-level)
server:
  port: 50051
joblet:
  maxConcurrentJobs: 100
# ... other main config

# Persist service config (nested)
persist:
  server:
    grpc_socket: "/opt/joblet/run/persist-grpc.sock"  # Unix socket for gRPC queries
  ipc:
    socket: "/opt/joblet/run/persist-ipc.sock"  # Unix socket for log/metric writes
  # ... other persist config

The persist service automatically detects and reads from the persist: section.

Security

TLS/mTLS Support

Both services support TLS encryption:

Main Joblet Service

Persist Service

Shared TLS Library

Build System

Makefile Targets

Proto Generation

Proto generation uses two approaches:

  1. scripts/generate-proto.sh - External public API
    • Downloads from joblet-proto module
    • Generates api/gen/
  2. go generate ./internal/proto - Internal protos (IPC and Persist)
    • Uses local internal/proto/ files
    • Generates internal/proto/gen/ipc/ and internal/proto/gen/persist/
    • Defined via //go:generate directives in internal/proto/generate.go

Developers can regenerate protos with:

make proto              # Regenerate all protos
go generate ./internal/proto  # Regenerate internal protos only

Data Flow

Job Execution Flow

flowchart TD
    N1["RNX sends RunJob request → Joblet"] --> N2["Joblet creates job record in memory + sends to State (async IPC)"]
    N2 --> N3["State persists job metadata to backend (Memory/DynamoDB)"]
    N3 --> N4["Joblet creates isolated environment (cgroups, namespaces)"]
    N4 --> N5["Joblet executes job and streams live logs → RNX"]
    N5 --> N6["Joblet collects metrics (CPU, memory, GPU, I/O)"]
    N6 --> N7["Joblet sends logs/metrics → Persist (via IPC)"]
    N7 --> N8["Persist stores to disk (/opt/joblet/logs, /opt/joblet/metrics)"]
    N8 --> N9["Joblet updates job status + sends to State (async IPC)"]

Historical Query Flow

flowchart TD
    N1["RNX sends QueryLogs request → Persist"] --> N2["Persist reads from disk storage"]
    N2 --> N3["Persist streams results → RNX"]

State Persistence Flow

flowchart TD
    N1["Joblet job lifecycle events (create, update, complete) → State (async IPC)"] --> N2["State writes to backend (Memory/DynamoDB) with 5s timeout"]
    N2 --> N3["On joblet restart: Joblet requests job sync → State"]
    N3 --> N4["State reads from backend → returns all jobs"]
    N4 --> N5["Joblet populates in-memory cache with persisted jobs"]

Storage Layout

/opt/joblet/
├── bin/                    # Binaries
│   ├── joblet
│   ├── rnx
│   ├── persist
│   └── state               # State persistence service
├── config/                 # Configuration
│   └── joblet-config.yml   # Unified config (all services)
├── run/                    # Runtime files
│   ├── persist-ipc.sock    # IPC socket (log/metric writes)
│   ├── persist-grpc.sock   # gRPC socket (historical queries)
│   └── state-ipc.sock      # IPC socket (job state persistence)
├── jobs/                   # Job workspaces
│   └── {job-uuid}/         # Per-job directory
├── logs/                   # Historical logs
│   └── {job-uuid}/         # Per-job log files
├── metrics/                # Historical metrics
│   └── {job-uuid}/         # Per-job metric files
├── volumes/                # Named volumes
├── network/                # Network state
└── runtimes/               # Runtime environments

Deployment

Systemd Services

Single systemd service:

  1. joblet.service
    • Binary: /opt/joblet/bin/joblet
    • Config: /opt/joblet/config/joblet-config.yml
    • Automatically spawns two subprocesses:
      • persist: Log/metric persistence (persist: section)
      • state: Job state persistence (state: section)

Deployment Command

make deploy REMOTE_HOST=192.168.1.161 REMOTE_USER=jay

This builds all binaries, copies to remote server, and restarts services.

Development Workflow

1. Make Changes

# Edit code in internal/, cmd/, or persist/
vim internal/joblet/core/executor.go

2. Update Protos (if needed)

# For public API changes - update external joblet-proto repo
# For internal changes - edit internal/proto/*.proto
make proto

3. Build and Test

make clean
make all
make test

4. Deploy

make deploy

Module Structure

Main Module

Persist Module

State Module

Design Decisions

1. Why Monorepo?

2. Why Separate Persist Service?

3. Why Separate State Service?

4. Why Internal Protos?

4. Why External joblet-proto?

Performance Characteristics

Throughput

Latency

Resource Usage