kamal-dev

Scale your development capacity horizontally with cloud-powered devcontainer workspaces.

kamal-dev extends Kamal to deploy and manage development container workspaces to cloud infrastructure. Deploy multiple parallel development environments for AI-assisted development, remote pair programming, or horizontal scaling of development tasks.

• 🚀 One-command deployment - Deploy devcontainers from .devcontainer/devcontainer.json specifications
• ☁️ Multi-cloud support - Pluggable provider architecture (UpCloud reference implementation)
• 💰 Cost estimation - Preview cloud costs before provisioning VMs
• 🔒 Secrets injection - Secure credential management via .kamal/secrets system
• 📦 State tracking - Atomic state file operations with file locking
• 🔄 Lifecycle management - Full control: deploy, list, stop, remove workspaces
• ⚙️ Resource limits - Enforce CPU/memory constraints per container
• 🎯 Docker-native - Direct Docker command generation from devcontainer specs

Add to your application's Gemfile:

gem "kamal", "~> 2.0"
gem "kamal-dev"

Then run:

bundle install

# Run the plugin installer to set up kamal dev commands
bundle exec plugin-kamal-dev

The installer will ask which method you prefer:

Option 1 (Recommended): Patch gem executable

• Patches the global kamal executable installed with the gem
• Creates a backup (kamal.backup) before patching
• Works with kamal dev and bundle exec kamal dev
• Global installation (available in all projects)

Option 2: Create project binstub

• Creates bin/kamal in your project directory
• Local to your project only
• Use with bin/kamal dev

That's it! After installation, you can use kamal dev commands.

If you prefer not to use the installer, you can manually set up kamal-dev:

Option 1: Quick test (no installation)

Use bundle exec with the -r flag:

bundle exec ruby -rkamal-dev -S kamal dev deploy

Option 2: Manual binstub edit

Generate the binstub and edit it manually:

bundle binstubs kamal --force

Then edit bin/kamal to add this line after the bundler setup:

require "kamal-dev"  # Add this line to load kamal-dev extension

Option 3: Rails/Boot file require

If your project has a boot file (e.g., Rails config/boot.rb), add:

require "kamal-dev"

Then use: bundle exec kamal dev

1. Generate configuration template:

kamal dev init

This creates config/dev.yml with a complete template. Edit it to configure:

• Your cloud provider (currently UpCloud)
• VM size and region
• Number of workspaces
• Resource limits

2. Set up secrets (.kamal/secrets):

export UPCLOUD_USERNAME="your-username"
export UPCLOUD_PASSWORD="your-password"
export GITHUB_TOKEN="ghp_..."

3. Deploy workspaces:

# If you chose Option 1 (gem executable):
kamal dev deploy --count 3
# or: bundle exec kamal dev deploy --count 3

# If you chose Option 2 (binstub):
bin/kamal dev deploy --count 3

4. List running workspaces:

kamal dev list

# Output:
# NAME          IP            STATUS   DEPLOYED AT
# myapp-dev-1   1.2.3.4       running  2025-11-16 10:30:00 UTC
# myapp-dev-2   2.3.4.5       running  2025-11-16 10:30:15 UTC
# myapp-dev-3   3.4.5.6       running  2025-11-16 10:30:30 UTC

5. Stop/remove when done:

kamal dev stop --all     # Stop containers, keep VMs
kamal dev remove --all   # Destroy VMs, cleanup state

# Required fields
service: myapp-dev              # Service name prefix
image: .devcontainer/devcontainer.json  # Devcontainer spec or direct image

provider:
  type: upcloud                 # Cloud provider (upcloud, hetzner, aws, gcp)
  zone: us-nyc1                 # Data center location
  plan: 1xCPU-2GB               # VM size/plan

# Optional fields
secrets:                        # Secrets to inject from .kamal/secrets
  - GITHUB_TOKEN
  - DATABASE_URL

secrets_file: .kamal/secrets    # Custom secrets file path (default: .kamal/secrets)

ssh:
  key_path: ~/.ssh/id_ed25519.pub  # SSH public key (default: ~/.ssh/id_rsa.pub)

defaults:
  cpus: 2                       # Default CPU limit
  memory: 4g                    # Default memory limit
  memory_swap: 8g               # Swap limit

vms:
  count: 5                      # Number of workspaces to deploy
  spread: false                 # Colocate (false) or one per VM (true)

naming:
  pattern: "{service}-{index}"  # Container naming pattern

kamal-dev parses VS Code devcontainer.json specifications and generates Docker run commands automatically:

Supported properties:

• image - Base Docker image
• forwardPorts - Port mappings (-p 3000:3000)
• mounts - Volume mounts (-v source:target)
• containerEnv - Environment variables (-e KEY=value)
• runArgs - Docker run flags (e.g., --cpus=2)
• remoteUser - Container user (--user vscode)
• workspaceFolder - Working directory (-w /workspace)

Example devcontainer.json:

{
  "image": "ruby:3.2",
  "forwardPorts": [3000, 5432],
  "containerEnv": {
    "RAILS_ENV": "development"
  },
  "mounts": [
    "source=${localWorkspaceFolder},target=/workspace,type=bind"
  ],
  "remoteUser": "vscode",
  "workspaceFolder": "/workspace"
}

kamal-dev supports deploying complex development stacks using Docker Compose, enabling multi-service deployments (app + database + cache + workers) with custom Dockerfiles.

To build and push images, configure a container registry in config/dev.yml:

service: myapp-dev

# Registry for image building and pushing
registry:
  server: ghcr.io                    # or docker.io for Docker Hub
  username: GITHUB_USER              # ENV var name (not actual username)
  password: GITHUB_TOKEN             # ENV var name (not actual password)

provider:
  type: upcloud
  zone: us-nyc1
  plan: 2xCPU-4GB

# Reference compose file from devcontainer
image: .devcontainer/devcontainer.json  # which references compose.yaml

Then set credentials in .kamal/secrets:

export GITHUB_USER="your-github-username"
export GITHUB_TOKEN="ghp_your_personal_access_token"
export UPCLOUD_USERNAME="your-upcloud-username"
export UPCLOUD_PASSWORD="your-upcloud-password"

Supported Registries:

• GitHub Container Registry (GHCR): server: ghcr.io
• Docker Hub: server: docker.io
• Custom registries: server: registry.example.com

Build image from Dockerfile:

kamal dev build

Push image to registry:

kamal dev push

Build, push, and deploy in one command:

kamal dev deploy --count 3

Skip build or push:

kamal dev deploy --skip-build  # Use existing local image
kamal dev deploy --skip-push   # Use local image, don't push to registry

Tag strategies:

Images are automatically tagged with:

• Timestamp tag: Unix timestamp (e.g., 1700000000)
• Git SHA tag: Short commit hash (e.g., abc123f)
• Custom tag: Specify with --tag flag

Deploy full development stacks with multiple services:

Example: Rails app with PostgreSQL

.devcontainer/devcontainer.json:

{
  "dockerComposeFile": "compose.yaml",
  "service": "app",
  "workspaceFolder": "/workspace"
}

.devcontainer/compose.yaml:

services:
  app:
    build:
      context: ..
      dockerfile: .devcontainer/Dockerfile
    volumes:
      - ../:/workspace:cached
    environment:
      DATABASE_URL: postgres://postgres:postgres@postgres:5432/myapp_dev
    ports:
      - "3000:3000"
    depends_on:
      - postgres

  postgres:
    image: postgres:16
    volumes:
      - postgres_data:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: postgres

volumes:
  postgres_data:

Deployment workflow:

1. Build - Builds app service image from Dockerfile
2. Push - Pushes image to registry (e.g., ghcr.io/user/myapp-dev:abc123)
3. Transform - Replaces build: with image: reference in compose.yaml
4. Deploy - Deploys full stack to each VM via docker-compose up -d

Result: Each VM gets an isolated stack (app + postgres + volumes)

# Deploy 3 isolated stacks
kamal dev deploy --count 3

# Each VM runs:
# - myapp-dev container (your app)
# - postgres container (isolated database)
# - Named volumes for persistence

List all containers:

kamal dev list

# Output includes all services:
# NAME               IP        STATUS   DEPLOYED AT
# myapp-dev-1-app    1.2.3.4   running  2025-11-18 10:30:00
# myapp-dev-1-postgres 1.2.3.4 running  2025-11-18 10:30:00
# myapp-dev-2-app    2.3.4.5   running  2025-11-18 10:30:15
# myapp-dev-2-postgres 2.3.4.5 running  2025-11-18 10:30:15

Supported features:

• ✅ Services with build: sections (main app service)
• ✅ Services with image: references (postgres, redis, etc.)
• ✅ Build context (string or object format)
• ✅ Dockerfile path specification
• ✅ Environment variables, volumes, ports
• ✅ Service dependencies (depends_on)
• ✅ Named volumes

Limitations (Phase 1):

• ❌ Single architecture builds only (amd64)
• ❌ Advanced compose features (networks, configs, profiles)
• ❌ Shared databases across VMs (each VM gets isolated stack)

Main service detection:

• First service with build: section is treated as main app service
• Only main service image is built and pushed to registry
• Dependent services (postgres, redis) use pre-built images

Directory structure:

.devcontainer/
├── Dockerfile
├── devcontainer.json
└── compose.yaml

Dockerfile:

FROM ruby:3.2

RUN apt-get update && apt-get install -y \
  build-essential \
  libpq-dev \
  nodejs \
  yarn

WORKDIR /workspace

COPY Gemfile* ./
RUN bundle install

CMD ["bundle", "exec", "rails", "server", "-b", "0.0.0.0"]

compose.yaml:

services:
  app:
    build:
      context: ..
      dockerfile: .devcontainer/Dockerfile
    volumes:
      - ../:/workspace:cached
    environment:
      DATABASE_URL: postgres://postgres:postgres@postgres:5432/myapp_dev
      REDIS_URL: redis://redis:6379/0
    ports:
      - "3000:3000"
    depends_on:
      - postgres
      - redis

  postgres:
    image: postgres:16
    volumes:
      - postgres_data:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: postgres

  redis:
    image: redis:7-alpine
    volumes:
      - redis_data:/data

volumes:
  postgres_data:
  redis_data:

Deploy:

kamal dev deploy --count 2

# Builds app image
# Pushes to ghcr.io/user/myapp-dev:abc123
# Deploys 2 isolated stacks (each with app + postgres + redis)

Build failures:

• Check Dockerfile syntax
• Verify build context path
• Review build args and secrets
• Enable verbose mode: VERBOSE=1 kamal dev build

Push failures:

• Verify registry credentials in .kamal/secrets
• Check GHCR token has write:packages permission
• Ensure image name follows registry conventions

Deploy failures:

• Check transformed compose.yaml: .kamal/dev_transformed_compose.yaml
• Verify all service images are accessible
• Review volume mount paths
• Check for port conflicts between services

Secrets are loaded from .kamal/secrets (shell script with export statements) and injected into containers as Base64-encoded environment variables.

.kamal/secrets:

export GITHUB_TOKEN="ghp_your_token_here"
export DATABASE_URL="postgres://user:pass@host:5432/db"

In container:

# Secrets available as env vars
echo $GITHUB_TOKEN_B64 | base64 -d  # Decode if needed

Kamal-dev supports DevPod-style remote development where your code is cloned from a git repository into the container rather than mounted from your local machine. This is ideal for cloud-based development workflows.

When you configure the git: section in config/dev.yml:

1. During image build: A special entrypoint script (dev-entrypoint.sh) is injected into your Docker image
2. On container startup: The entrypoint clones your repository into the workspace folder
3. For local development: VS Code devcontainers work normally with mounted code (no git clone)

Key benefits:

• ✅ No local file mounting needed (pure cloud deployment)
• ✅ Code changes persist across container restarts
• ✅ Supports private repositories via GitHub Personal Access Token (PAT)
• ✅ Automatic credential caching for git operations (pull/push)

Step 1: Generate GitHub Personal Access Token

1. Go to GitHub Settings → Developer settings → Personal access tokens → Tokens (classic)
2. Click Generate new token (classic)
3. Give it a name: "kamal-dev deployment"
4. Select scopes:
   • ✅ repo (Full control of private repositories)
5. Click Generate token
6. Copy the token (starts with ghp_...) - you won't see it again

Step 2: Add token to secrets file

Add your token to .kamal/secrets:

export GITHUB_TOKEN="ghp_xxxxxxxxxxxxxxxxxxxx"

Important: Ensure the variable is exported so it's available to Ruby processes.

Step 3: Configure git clone in config/dev.yml

git:
  repository: https://github.com/yourorg/yourrepo.git  # HTTPS URL (not SSH)
  branch: main                                         # Branch to checkout
  workspace_folder: /workspaces/myapp                  # Where to clone code
  token: GITHUB_TOKEN                                  # Environment variable name

Step 4: Deploy

kamal dev deploy --count 2

The deployment process will:

1. Build your image with the entrypoint script injected
2. Push to registry
3. Deploy containers with git environment variables
4. On first boot, containers clone your repository

Check if code was cloned:

# SSH into VM
ssh root@<vm-ip>

# Check container logs
docker logs myapp-dev-1-app

# Should see:
# [kamal-dev] Remote deployment detected
# [kamal-dev] Cloning https://github.com/yourorg/yourrepo.git (branch: main)
# [kamal-dev] Clone complete: /workspaces/myapp

Verify git authentication is cached:

# Exec into container
docker exec -it myapp-dev-1-app bash

# Try pulling
cd /workspaces/myapp
git pull

# Should succeed without prompting for credentials

• Use HTTPS URLs: https://github.com/user/repo.git (NOT git@github.com:user/repo.git)
• Token security: The token is injected as an environment variable and used only at startup for cloning
• Credential caching: Git credentials are stored in ~/.git-credentials inside the container for future git operations
• Local development: If you use VS Code with devcontainer.json, the git clone is skipped - your local code is mounted instead
• Token scopes: For private repos, you need the repo scope. For public repos, no token is needed.

"fatal: could not read Username for 'https://github.com'"

• Verify GITHUB_TOKEN is in .kamal/secrets
• Ensure the variable is exported (export GITHUB_TOKEN=...)
• Check the token has repo scope for private repositories

"Permission denied" when cloning

• Check the token hasn't expired (GitHub tokens can have expiration dates)
• Verify the token has access to the repository (check repo permissions)
• Ensure you're using HTTPS URL, not SSH format

Code not appearing in /workspaces

• Check container logs: docker logs <container-name>
• Verify workspace_folder matches devcontainer.json workspaceFolder
• Ensure git repository URL is accessible

All commands below assume you've run bundle exec plugin-kamal-dev as described in the Installation section. If you're using an alternative setup method, adjust the commands accordingly (see Alternative Setup Methods in Installation).

Generate a configuration template.

kamal dev init

What it does:

1. Creates config/ directory if it doesn't exist
2. Copies template to config/dev.yml
3. Prompts before overwriting if file already exists
4. Displays next steps for configuration

Example output:

✅ Created config/dev.yml

Next steps:

1. Edit config/dev.yml with your cloud provider credentials
2. Create .kamal/secrets file with your secrets
3. Deploy your first workspace: kamal dev deploy --count 3

Deploy devcontainer workspaces to cloud VMs.

kamal dev deploy [OPTIONS]

Options:
  --count N         Number of containers to deploy (default: from config)
  --from PATH       Path to devcontainer.json (default: from config)
  --config PATH     Path to config file (default: config/dev.yml)

What it does:

1. Loads configuration and devcontainer spec
2. Estimates cloud costs → prompts for confirmation
3. Provisions VMs via cloud provider API
4. Bootstraps Docker on VMs (if not installed)
5. Deploys containers with injected secrets
6. Saves state to .kamal/dev_state.yml

List deployed devcontainer workspaces.

kamal dev list [OPTIONS]

Options:
  --format FORMAT   Output format: table (default), json, yaml

Example output:

NAME          IP            STATUS   DEPLOYED AT
myapp-dev-1   1.2.3.4       running  2025-11-16 10:30:00 UTC
myapp-dev-2   2.3.4.5       stopped  2025-11-16 10:30:15 UTC

Stop devcontainer(s) without destroying VMs.

kamal dev stop [NAME] [OPTIONS]

Arguments:
  NAME              Container name to stop (optional)

Options:
  --all             Stop all containers

What it does:

• Executes docker stop {container} via SSH
• Updates state file: status → "stopped"
• VMs remain running (reduces restart time)

Destroy VMs and remove container state.

kamal dev remove [NAME] [OPTIONS]

Arguments:
  NAME              Container name to remove (optional)

Options:
  --all             Remove all containers
  --force           Skip confirmation prompt

What it does:

1. Prompts for confirmation (unless --force)
2. Stops containers via docker stop
3. Destroys VMs via provider API
4. Removes entries from state file
5. Deletes state file if empty

Show detailed status of devcontainer(s).

kamal dev status [NAME] [OPTIONS]

Arguments:
  NAME              Container name to check (optional)

Options:
  --all             Show status for all containers

Problem: ProvisioningError: VM failed to reach running state

Solutions:

• Check provider API credentials in .kamal/secrets
• Verify zone/region availability
• Check account quotas (VMs, storage, IPs)
• Try smaller VM plan (e.g., 1xCPU-1GB)

Problem: Container status shows "failed"

Solutions:

• Check image name in devcontainer.json
• Verify secrets are valid (Base64 encoding issues)
• SSH to VM and check Docker logs: ssh root@{vm_ip} docker logs {container}
• Review resource limits (may be too restrictive)

Problem: "State file appears corrupted or locked"

Solutions:

# Check for lock file
ls -la .kamal/dev_state.yml.lock

# Remove stale lock (if no processes using it)
rm .kamal/dev_state.yml.lock

# Rebuild state from provider dashboard
kamal dev list --rebuild  # (future feature)

Problem: "SSH public key not found at ~/.ssh/id_rsa.pub"

Solutions:

# Configure custom SSH key in config/dev.yml
ssh:
  key_path: ~/.ssh/id_ed25519.pub

Or generate new SSH key:

ssh-keygen -t ed25519 -C "kamal-dev@example.com"

Enable verbose logging:

VERBOSE=1 kamal dev deploy --count 2

After checking out the repo:

# Install dependencies
bin/setup

# Run tests
bundle exec rspec

# Run linter
bundle exec standardrb

# Run full suite (tests + linter)
bundle exec rake

# Interactive console
bin/console

# Install locally for testing
bundle exec rake install

Integration tests provision real VMs (costs money). Set up test credentials:

# Create .kamal/secrets with UpCloud test account
export UPCLOUD_USERNAME="test-user"
export UPCLOUD_PASSWORD="test-password"

# Run integration tests
INTEGRATION_TESTS=1 bundle exec rspec

⚠️ Warning: Integration tests will provision and destroy VMs. Estimated cost: ~$0.01-0.05 per test run.

┌─────────────┐
│   CLI       │
│  Commands   │
└──────┬──────┘
       │
       ▼
┌──────────────────┐      ┌─────────────────┐
│ Provider::Base   │◄─────┤ DevConfig       │
│  (interface)     │      │ (configuration) │
└────────┬─────────┘      └─────────────────┘
         │
         ├──► Provider::Upcloud
         ├──► Provider::Hetzner  (future)
         ├──► Provider::AWS      (future)
         └──► Provider::GCP      (future)

State is tracked in .kamal/dev_state.yml with file locking to prevent corruption:

deployments:
  myapp-dev-1:
    vm_id: "00abc123-def4-5678-90ab-cdef12345678"
    vm_ip: "1.2.3.4"
    container_name: "myapp-dev-1"
    status: running
    deployed_at: "2025-11-16T14:30:00Z"

File locking:

• Uses File.flock(File::LOCK_EX) for exclusive writes
• Uses File.flock(File::LOCK_SH) for shared reads
• NFS-compatible dotlock fallback

• Hetzner Cloud provider adapter
• AWS EC2 provider adapter
• GCP Compute Engine provider adapter
• Multi-project workspace sharing
• Automatic workspace hibernation (cost optimization)
• Devcontainer features support (via devcontainer CLI)
• Web UI for workspace management

Bug reports and pull requests are welcome on GitHub at https://github.com/hyperengineering/kamal-dev.

Before submitting a PR:

1. Run full test suite: bundle exec rake
2. Ensure linter passes: bundle exec standardrb
3. Add tests for new features
4. Update CHANGELOG.md
5. Update documentation (README, YARD comments)

The gem is available as open source under the terms of the MIT License.

Built as an extension to Kamal by Basecamp.