Joblet is a micro-container runtime for running Linux jobs with: Process and filesystem isolation (PID namespace, chroot) Fine-grained CPU, memory, and IO throttling (cgroups v2) Secure job execution with mTLS and RBAC Built-in scheduler, SSE log streaming, and multi-core pinning Ideal for: Agentic AI Workloads (Untrusted code)
This document describes how Joblet integrates with AWS Secrets Manager to enable horizontal scaling across multiple EC2 instances while maintaining a shared Certificate Authority (CA) and client certificates.
Without Secrets Manager (Current Default):
With Secrets Manager:
┌─────────────────────────────────────────────────────────────┐
│ AWS Secrets Manager │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ joblet/ca-cert (Shared - Generated Once) │ │
│ │ joblet/ca-key (Shared - Generated Once) │ │
│ │ joblet/client-cert (Shared - Generated Once) │ │
│ │ joblet/client-key (Shared - Generated Once) │ │
│ └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
↓ ↓ ↓
┌─────────────────┼─┼─┼─────────────────┐
↓ ↓ ↓ ↓
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ EC2 Instance │ │ EC2 Instance │ │ EC2 Instance │
│ #1 │ │ #2 │ │ #3 │
├───────────────┤ ├───────────────┤ ├───────────────┤
│ Server Cert A │ │ Server Cert B │ │ Server Cert C │
│ (Local) │ │ (Local) │ │ (Local) │
└───────────────┘ └───────────────┘ └───────────────┘
↑ ↑ ↑
└─────────────────┼─────────────────┘
↓
┌───────────────┐
│ Clients │
│ (One config) │
└───────────────┘
Deploy joblet in an ASG with shared certificates:
# First instance (or launch template)
USE_SECRETS_MANAGER=true /usr/local/bin/certs_gen_with_secretsmanager.sh
# All subsequent instances automatically:
# - Retrieve shared CA/client certs from Secrets Manager
# - Generate unique server certificates
# - Work with the same client config
Replicate secrets across regions for DR:
# Primary region (us-east-1)
USE_SECRETS_MANAGER=true EC2_REGION=us-east-1 ./certs_gen_with_secretsmanager.sh
# Replicate to DR region (us-west-2)
aws secretsmanager replicate-secret-to-regions \
--secret-id joblet/ca-cert \
--add-replica-regions Region=us-west-2
# Deploy in DR region
USE_SECRETS_MANAGER=true EC2_REGION=us-west-2 ./certs_gen_with_secretsmanager.sh
Both environments share certificates:
# Blue environment
USE_SECRETS_MANAGER=true JOBLET_SERVER_ADDRESS=blue.internal ./script.sh
# Green environment (reuses CA/client)
USE_SECRETS_MANAGER=true JOBLET_SERVER_ADDRESS=green.internal ./script.sh
# Clients can connect to either with same config
┌─────────────────┐
│ ALB / NLB │
│ (TLS Passthru) │
└────────┬────────┘
│
┌───────────────────┼───────────────────┐
↓ ↓ ↓
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Server1 │ │ Server2 │ │ Server3 │
│ Cert A │ │ Cert B │ │ Cert C │
└─────────┘ └─────────┘ └─────────┘
All signed by same CA
All accept same client cert
Update your existing JobletEC2Role with Secrets Manager permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "SecretsManagerAccess",
"Effect": "Allow",
"Action": [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:CreateSecret",
"secretsmanager:UpdateSecret",
"secretsmanager:PutSecretValue",
"secretsmanager:TagResource"
],
"Resource": [
"arn:aws:secretsmanager:*:*:secret:joblet/*"
]
}
]
}
Apply the policy:
# Create policy
POLICY_ARN=$(aws iam create-policy \
--policy-name JobletSecretsManagerPolicy \
--policy-document file://secretsmanager-policy.json \
--query 'Policy.Arn' \
--output text)
# Attach to existing role
aws iam attach-role-policy \
--role-name JobletEC2Role \
--policy-arn $POLICY_ARN
Modify the EC2 user data to enable Secrets Manager:
#!/bin/bash
set -e
# Configuration
export JOBLET_VERSION="latest"
export JOBLET_SERVER_PORT="443"
export ENABLE_CLOUDWATCH="true"
# Enable Secrets Manager integration
export USE_SECRETS_MANAGER="true" # NEW!
export SECRETS_PREFIX="joblet" # Optional: custom prefix
# Download and run installation script
curl -fsSL https://raw.githubusercontent.com/ehsaniara/joblet/main/scripts/ec2-user-data.sh \
-o /tmp/joblet-install.sh
chmod +x /tmp/joblet-install.sh
/tmp/joblet-install.sh 2>&1 | tee /var/log/joblet-install.log
The first instance will:
# Verify secrets were created
aws secretsmanager list-secrets \
--filters Key=name,Values=joblet/ \
--query 'SecretList[*].[Name,Description]' \
--output table
Expected output:
----------------------------------------
| ListSecrets |
+----------------------+----------------+
| joblet/ca-cert | Joblet Root CA Certificate - shared across all instances |
| joblet/ca-key | Joblet Root CA Private Key - shared across all instances |
| joblet/client-cert | Joblet Admin Client Certificate - shared across all clients |
| joblet/client-key | Joblet Admin Client Private Key - shared across all clients |
+----------------------+----------------+
Subsequent instances will:
No additional steps needed - just use the same user data script!
Download the client config from any instance:
# From instance 1
scp ec2-user@instance1:/opt/joblet/config/rnx-config.yml ~/.rnx/
# This config works with ALL instances!
rnx --node=default job list
# Instance 1
USE_SECRETS_MANAGER=true ./certs_gen_with_secretsmanager.sh
Output:
✨ CA Certificate: Generated and stored in Secrets Manager (shared)
✨ Client Certificate: Generated and stored in Secrets Manager (shared)
🆕 Server Certificate: Generated locally (instance-specific)
# Instance 2, 3, 4...
USE_SECRETS_MANAGER=true ./certs_gen_with_secretsmanager.sh
Output:
✅ CA Certificate: Retrieved from Secrets Manager (shared)
✅ Client Certificate: Retrieved from Secrets Manager (shared)
🆕 Server Certificate: Generated locally (instance-specific)
# On each instance
sudo /usr/local/bin/certs_gen_with_secretsmanager.sh
sudo systemctl restart joblet
This regenerates only the server certificate. CA and client certs remain unchanged.
# Force regeneration on one instance
USE_SECRETS_MANAGER=true FORCE_REGENERATE=true \
/usr/local/bin/certs_gen_with_secretsmanager.sh
# This creates NEW CA and client certs in Secrets Manager
# Then restart all other instances to pick up new certs
# Finally, distribute new client config to all clients
⚠️ Warning: Rotating CA requires:
| Variable | Default | Description |
|---|---|---|
USE_SECRETS_MANAGER |
auto |
Enable Secrets Manager (true, false, auto) |
SECRETS_PREFIX |
joblet |
Prefix for secret names |
FORCE_REGENERATE |
false |
Force regenerate even if secrets exist |
EC2_REGION |
auto | AWS region (auto-detected from EC2 metadata) |
Enable explicitly:
USE_SECRETS_MANAGER=true ./certs_gen_with_secretsmanager.sh
Disable explicitly:
USE_SECRETS_MANAGER=false ./certs_gen_with_secretsmanager.sh
Auto-detect (default on EC2):
./certs_gen_with_secretsmanager.sh
# Automatically enables if running on EC2 and AWS CLI is available
Custom prefix:
USE_SECRETS_MANAGER=true SECRETS_PREFIX=my-app/joblet \
./certs_gen_with_secretsmanager.sh
Force regenerate everything:
USE_SECRETS_MANAGER=true FORCE_REGENERATE=true \
./certs_gen_with_secretsmanager.sh
Principle of Least Privilege:
{
"Effect": "Allow",
"Action": [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret"
],
"Resource": "arn:aws:secretsmanager:*:*:secret:joblet/*"
}
For read-only instances (workers that don’t create/update secrets).
Full Access (for first instance):
{
"Effect": "Allow",
"Action": [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:CreateSecret",
"secretsmanager:UpdateSecret"
],
"Resource": "arn:aws:secretsmanager:*:*:secret:joblet/*"
}
Each instance gets a unique server certificate:
Monitor secret access:
# Who accessed CA certificates?
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=ResourceName,AttributeValue=joblet/ca-cert \
--query 'Events[*].[EventTime,Username,EventName]' \
--output table
# Alert on unexpected access
aws cloudwatch put-metric-alarm \
--alarm-name JobletSecretAccess \
--metric-name SecretAccess \
--namespace AWS/SecretsManager \
--threshold 100
# Check if secrets exist
aws secretsmanager list-secrets --filters Key=name,Values=joblet/
# If empty, force regenerate
USE_SECRETS_MANAGER=true FORCE_REGENERATE=true ./certs_gen_with_secretsmanager.sh
# Check IAM role
aws sts get-caller-identity
# Test secret access
aws secretsmanager get-secret-value --secret-id joblet/ca-cert
# If permission denied, update IAM policy
# Check certificate expiration
aws secretsmanager get-secret-value \
--secret-id joblet/ca-cert \
--query SecretString \
--output text | openssl x509 -noout -dates
# Force regenerate if expired
USE_SECRETS_MANAGER=true FORCE_REGENERATE=true ./certs_gen_with_secretsmanager.sh
This is not supported. You must choose one approach:
Migration path:
USE_SECRETS_MANAGER=trueStep 1: Extract existing certs (optional)
If you want to keep your existing CA:
# On existing instance
sudo cat /opt/joblet/config/joblet-config.yml | \
yq '.security.caCert' > /tmp/ca-cert.pem
# Upload to Secrets Manager
aws secretsmanager create-secret \
--name joblet/ca-cert \
--secret-string file:///tmp/ca-cert.pem
Step 2: Enable on new instances
Launch new instances with USE_SECRETS_MANAGER=true.
Step 3: Update clients
Download new client config and distribute.
Step 1: Download certs
# Get certs from Secrets Manager
aws secretsmanager get-secret-value \
--secret-id joblet/ca-cert \
--query SecretString \
--output text > ca-cert.pem
Step 2: Launch without Secrets Manager
USE_SECRETS_MANAGER=false ./certs_gen_embedded.sh
Step 3: Delete secrets (optional)
aws secretsmanager delete-secret \
--secret-id joblet/ca-cert \
--force-delete-without-recovery
| Feature | Embedded Certs | Secrets Manager |
|---|---|---|
| Setup Complexity | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ |
| Horizontal Scaling | ❌ | ✅ |
| Load Balancer Support | ❌ | ✅ |
| One Client Config | ❌ | ✅ |
| Certificate Rotation | Manual, complex | Centralized |
| Compliance (Audit Logs) | ❌ | ✅ |
| Disaster Recovery | Manual backup | Built-in |
| Multi-Region | Manual | Replicate |
| Offline / No Internet | ✅ | ❌ |