Milvaion
A distributed job scheduling system built on .NET 11
Already on Hangfire or Quartz.NET? Keep it β add Milvaion monitoring in two lines.
What is Milvaion?
Milvaion is a distributed job scheduling system that separates the scheduler (API that decides when jobs run) from the workers (processes that execute jobs), connected via Redis and a message broker: RabbitMQ by default, or Apache Kafka if you already run it.
βββββββββββββββββββ βββββββββββββββββββ βββββββββββββββββββ
β Milvaion API β β RabbitMQ β β Workers β
β (Scheduler) ββββββββ>β (Job Queue) βββββββ>β (Executors) β
β β β β β β
β β’ REST API β β β’ Job messages β β β’ IJob classes β
β β’ Dashboard β β β’ Status queues β β β’ Retry logic β
β β’ Cron parsing β<ββββββββ β’ Log streams β<βββββββ β’ DI support β
β β’ Workflow DAG β β β β β
βββββββββββββββββββ βββββββββββββββββββ βββββββββββββββββββ
Why Milvaion?
Most job schedulers run jobs inside the same process as the scheduling logic. This works fine until:
- A long-running job blocks other jobs from executing
- A crashing job takes down the entire scheduler
- You need different hardware for different job types (e.g., GPU for ML jobs)
- You want to scale job execution independently from the API
Milvaion solves these problems by completely separating scheduling from execution.
Workers in any language
The scheduler is .NET, but workers don't have to be. Besides the .NET worker SDK, there are worker SDKs for Node.js, Go, Python, Java, PHP, Rust and Ruby. Workers written with them show up in the same dashboard, with live logs, retries, dead-lettering, cancellation, timeouts and graceful shutdown. Each worker carries a badge for its language.
| Language | SDK |
|---|---|
| .NET | Milvasoft.Milvaion.Sdk.Worker |
| Node.js | sdks/node/milvaion-worker |
| Go | sdks/go/milvaion |
| Python | sdks/python/milvaion-worker |
| Java / Kotlin | sdks/java/milvaion-worker |
| PHP | sdks/php/milvaion-worker |
| Rust | sdks/rust/milvaion-worker |
| Ruby | sdks/ruby/milvaion-worker |
Any other language can join through the worker protocol, which documents the queues and message formats. To schedule existing scripts without an SDK, use the built-in Command Worker.
Already Running Hangfire or Quartz.NET?
You don't have to migrate to adopt Milvaion.
Milvaion plugs into the scheduler you already run. Your scheduler keeps owning triggers, storage and cron expressions. Your job code doesn't change. Milvaion adds one real-time dashboard across every service, plus persisted execution history, metrics and alerting.
// Hangfire - your existing setup stays exactly as it is
builder.Services.AddMilvaionHangfireIntegration(builder.Configuration);
builder.Services.AddHangfire((sp, config) => config.UseMilvaion(sp));
// Quartz.NET - your existing setup stays exactly as it is
builder.Services.AddMilvaionQuartzIntegration(builder.Configuration);
builder.Services.AddQuartz(q => q.UseMilvaion(builder.Services));That's the whole integration.
The problem this solves
A typical .NET estate ends up with background jobs scattered across a dozen services β each with its own dashboard behind its own URL and its own auth. Nobody can answer "which jobs failed last night?" without opening every one of them, and history vanishes when the retention window rolls over.
| Before | After |
|---|---|
| One dashboard per service | One dashboard for the whole estate |
| History limited by Hangfire/Quartz retention | Full execution history in PostgreSQL |
| Logs only in each app's own sink | Real-time log streaming per execution |
| Failures noticed when someone complains | Multi-channel alerts on failure and timeout |
| No cross-service metrics | Success rate, duration and EPM per job |
The adoption path
- Add monitoring β two lines per application. No migration window, no change to job code.
- Get visibility β every Hangfire and Quartz.NET job appears in the dashboard, flagged as external.
- Migrate selectively, or never β when one job outgrows in-process execution, move just that job to a Milvaion worker. Everything else keeps running untouched.
Plenty of teams stop at step 2. That's a perfectly good outcome.
| Scheduler | Package | Status |
|---|---|---|
| Hangfire | Milvasoft.Milvaion.Sdk.Worker.Hangfire |
β Available |
| Quartz.NET | Milvasoft.Milvaion.Sdk.Worker.Quartz |
β Available |
π Full Integration Guide β
Features
Reliability
- At-least-once delivery via RabbitMQ publisher confirms and manual ACK, or Kafka offsets committed only after a job's outcome is reported
- RabbitMQ or Kafka as the message broker, chosen per deployment (Using Kafka)
- Automatic retries with exponential backoff
- Dead Letter Queue for failed jobs after max retries
- Zombie detection recovers stuck jobs
- Auto disable always failing jobs (configurable threshold)
Scheduling
- Per-job time zones - Cron expressions run at local wall clock time in any IANA zone, with predictable daylight saving behaviour
- Missed-run policy - Run once (the default), skip, or catch up on every run missed while the scheduler was down
- Job priority - Higher priority jobs are dispatched first when several are due at once
- Exclusion calendars - Skip holidays, maintenance days or weekends; skipped runs are recorded as skipped, not failed
See Scheduling Options, Time Zones and Exclusion Calendars.
Scalability
- Horizontal worker scaling - add more workers for more throughput
- Job-type routing - route specific jobs to specialized workers
- Independent scaling - scale API and workers separately
Observability
- Real-time dashboard with SignalR updates
- Execution logs - User-friendly logs stored in occurrences + technical logs to Seq
- Worker health monitoring via heartbeats
- OpenTelemetry support for metrics and tracing
Developer Experience
-
Simple
IJobinterfaces - implement one method - Full DI support - inject services into jobs
- Auto-discovery - jobs registered automatically
- Cancellation support - graceful shutdown
-
Project templates - get started quickly with
dotnet new
Workflow Engine
- DAG-based orchestration - Chain jobs into directed acyclic graphs
-
Conditional branching - Route execution through
true/falseports based on step results or statuses - Merge nodes - Wait for all parallel branches before continuing
- Data mappings - Pass output fields from upstream steps into downstream job data
- Failure strategies - Stop on first failure or continue on failure
- Cron, manual & one-time triggers - Schedule workflows with cron, trigger on demand, or run once at a given time (reschedulable)
- Retry at a specific time - A step can ask to be rerun at a given time (e.g. when a quota resets) instead of retrying with backoff
- Visual builder - Drag-and-drop DAG editor in the dashboard
Built-in Workers
- HTTP Worker - Call REST APIs on schedule
- SQL Worker - Execute database queries
- Email Worker - Send emails via SMTP
- Command Worker - Run allow-listed scripts and programs (any language) and stream their output
- Maintenance Worker - Milvaion self data warehouse cleanup and archival
Built-in workers
Five workers run out of the box with no code to write: HTTP (call REST endpoints on a schedule), SQL (run queries), Email (send over SMTP), Command (run allow-listed scripts and programs in any language) and Maintenance (Milvaion's own data cleanup and archival).
Secret References
Keep credentials out of job data. Write {{secret:smtp-password}} in a job's JSON data, a workflow step or a manual trigger, and Milvaion resolves it just before publishing the job to RabbitMQ. The database, API responses, the dashboard and the MCP tools only ever see the reference; the dashboard shows it with a lock icon. Values come from environment variables, configuration, HashiCorp Vault (KV v2) or Azure Key Vault, or your own ISecretProvider. A missing secret fails the occurrence with a clear message and nothing is published.
{ "smtpPassword": "{{secret:smtp-password}}", "auth": "Bearer {{secret:vault:apps/billing#token}}" }π Secret references guide β
External Scheduler Integration
Milvaion also monitors jobs running in Quartz.NET and Hangfire without replacing them β see Already Running Hangfire or Quartz.NET? above.
Single Sign-On (SSO)
Beyond local username/password login, Milvaion can federate authentication to an external identity provider:
- OIDC (OpenID Connect) β browser redirect SSO with providers such as Keycloak, Entra ID or Auth0.
- LDAP / Active Directory β the standard login form, with the password verified by a directory bind.
Both providers are off by default and enabled under MilvaionConfig:Authentication. The identity provider owns identity and group membership, while Milvaion owns authorization, you assign the permission set of each mirrored role inside Milvaion.
{
"MilvaionConfig": {
"Authentication": {
"Oidc": { "Enabled": true, "Authority": "https://keycloak.example.com/realms/milvaion", "ClientId": "milvaion-spa" },
"Ldap": { "Enabled": false, "Host": "ldap.example.com", "Port": 636, "UseSsl": true, "BaseDn": "dc=corp,dc=example,dc=com" }
}
}
}π SSO setup guide β
MCP Server
Point Claude Code, Cursor or GitHub Copilot at Milvaion and ask about your jobs in plain language:
{
"mcpServers": {
"milvaion": {
"type": "http",
"url": "https://milvaion.yourcompany.com/mcp",
"headers": { "X-ApiKey": "your-api-key" }
}
}
}"Which jobs failed last night and why?"
More than forty tools β reading, triggering, pausing, editing and deleting β each gated by the same permissions used everywhere else. A read-only api key gives an assistant full visibility and no ability to change anything. Milvaion is the data source here; it never calls a language model and stores no model provider keys.
π MCP Server Guide β
Jobs as Code
Keep job definitions in Git and apply them from CI, with the milvaion CLI or the Terraform provider:
# jobs.yaml
apiVersion: milvaion/v1
kind: Job
metadata:
name: nightly-invoice-report
spec:
worker: reporter-worker
jobName: InvoiceReportJob
cron: "0 0 2 * * *"
jobData: { format: pdf }milvaion diff -f jobs.yaml # what would change
milvaion apply -f jobs.yaml --prune # create, update, and delete jobs removed from the file
milvaion trigger nightly-invoice-report --followresource "milvaion_job" "invoice_report" {
name = "nightly-invoice-report"
worker_id = "reporter-worker"
job_name = "InvoiceReportJob"
cron_expression = "0 0 2 * * *"
job_data = jsonencode({ format = "pdf" })
}Both use an api key and the regular REST API. CLI binaries for Linux, macOS and Windows are attached to the cli-v* releases.
Quick Start
Prerequisites
- Docker Desktop (v20.10+) with Docker Compose
- Web browser for the dashboard
1. Start the Stack
# Clone the repository
git clone https://github.com/Milvasoft/milvaion.git
cd milvaion
# Create the local settings (passwords and signing keys)
cp .env.example .env
# Start all services
docker compose up -dTo run the stack on Kafka instead of RabbitMQ, add the Kafka Compose file:
docker compose -f docker-compose.yml -f docker-compose.kafka.yml up -d2. Access the Dashboard
Open http://localhost:5000 in your browser.
- Default username:
rootuser - Get password:
admin(which is defined in docker-compose.yml) or if not defined auto generated password :docker logs milvaion-api 2>&1 | grep -i "password"
3. Create Your First Job
curl -X POST http://localhost:5000/api/v1/jobs/job \
-H "Content-Type: application/json" \
-d '{
"displayName": "My First Job",
"workerId": "sample-worker-01",
"selectedJobName": "SampleJob",
"cronExpression": "* * * * *",
"isActive": true,
"jobData": "{\"message\": \"Hello from Milvaion!\"}"
}'π Full Quick Start Guide β
Architecture
Milvaion is organized in onion-architecture layers, Domain at the center, then Application, then Infrastructure and the API on the outside:
Solution Structure
milvaion/
βββ src/
β βββ Core/
β β βββ Milvaion.Domain/ # Entities, enums, domain logic
β β βββ Milvaion.Application/ # Use cases, DTOs, interfaces
β βββ Infrastructure/
β β βββ Milvaion.Infrastructure/ # EF Core, external services
β βββ Presentation/
β β βββ Milvaion.Api/ # REST API, controllers, dashboard
β βββ Sdk/
β β βββ Milvasoft.Milvaion.Sdk/ # Client SDK
β β βββ Milvasoft.Milvaion.Sdk.Worker/ # Worker SDK
β βββ Workers/
β β βββ HttpWorker/ # Built-in HTTP worker
β β βββ SqlWorker/ # Built-in SQL worker
β β βββ EmailWorker/ # Built-in Email worker
β β βββ CommandWorker/ # Built-in Command worker
β β βββ MilvaionMaintenanceWorker/ # Maintenance jobs
β βββ MilvaionUI/ # React dashboard
βββ sdks/
β βββ node/milvaion-worker/ # Node.js worker SDK
β βββ go/milvaion/ # Go worker SDK
β βββ python/milvaion-worker/ # Python worker SDK
β βββ java/milvaion-worker/ # Java worker SDK
β βββ php/milvaion-worker/ # PHP worker SDK
βββ tests/
β βββ Milvaion.UnitTests/
β βββ Milvaion.IntegrationTests/
βββ docs/
β βββ portaldocs/ # User documentation
β βββ githubdocs/ # Developer documentation
βββ build/ # Build scripts
Project Dependencies
Build Order: Domain β Application β Infrastructure β Api β Tests
Development Setup
Prerequisites
- .NET 11 SDK
- PostgreSQL 16
- Redis 7
- RabbitMQ 3.x
- Node.js 18+ (for UI development)
Local Development
# Clone repository
git clone https://github.com/Milvasoft/milvaion.git
cd milvaion
# Start infrastructure (PostgreSQL, Redis, RabbitMQ)
docker compose up -d
Running Tests
# Unit tests
dotnet test tests/Milvaion.UnitTests
# Integration tests (requires infrastructure)
dotnet test tests/Milvaion.IntegrationTests
# All tests with coverage
dotnet test --collect:"XPlat Code Coverage"Creating a Worker
1. Install the Template
dotnet new install Milvasoft.Templates.Milvaion2. Create a New Worker
dotnet new milvaion-console-worker -n MyCompany.MyWorker
cd MyCompany.MyWorker3. Implement a Job
using Milvasoft.Milvaion.Sdk.Worker.Abstractions;
public class MyCustomJob : IAsyncJob
{
private readonly IMyService _myService;
public MyCustomJob(IMyService myService)
{
_myService = myService;
}
public async Task ExecuteAsync(IJobContext context)
{
context.LogInformation("Starting my custom job...");
var data = JsonSerializer.Deserialize<MyJobData>(context.Job.JobData);
await _myService.ProcessAsync(data, context.CancellationToken);
context.LogInformation("Job completed successfully!");
}
}Workflows
Milvaion's workflow engine lets you chain multiple jobs into a directed acyclic graph (DAG) where the engine coordinates execution order, data passing, conditional branching, and failure handling automatically.
Key Concepts
| Concept | Description |
|---|---|
| Task Node | Dispatches a scheduled job β the primary building block |
| Condition Node | Evaluates an expression and routes to a true or false branch |
| Merge Node | Waits for all incoming branches to complete before continuing |
| Edge | Directed connection between steps defining execution order |
| Data Mapping | Passes an output field from one step as input to another |
Example: Conditional Pipeline
βββββββββββββ βββββββββββββ βββββββββββββββ
β Extract ββββββΊ β Price > 50βββTβββΊβ Send Invoiceβ
β Prices β β(Condition)β βββββββββββββββ
βββββββββββββ ββββββ¬βββββββ
β F
βββββββΌββββββ
β Log Skip β
βββββββββββββ
Condition Expressions
Conditions support status checks, JSON field comparisons, and logical operators:
# All parents completed
@status == 'Completed'
# JSON field check
$.price > 100
# Combined with AND/OR
@status == 'Completed' && $.price > 50
# Target a specific parent step
019d...83f7:@status == 'Completed'
Data Mappings
Pass results between steps using source path β target key mappings:
Step 1 (ExtractPrices) β result: { "price": 99, "item": { "name": "Widget" } }
Mapping on Step 2:
step1Id:price β amount
step1Id:item.name β title
Step 2 receives: { "amount": 99, "title": "Widget" }
Failure Handling
| Strategy | Behavior |
|---|---|
| Stop on First Failure | Workflow stops immediately; pending steps are skipped |
| Continue on Failure | Independent branches keep running; only dependent steps are skipped |
Each workflow can also configure Max Step Retries and a Timeout (auto-cancel if exceeded).
A run can be triggered once at a given time (executeAt), then moved or cancelled before it starts. A step whose job throws a retry-after error (for example "the API quota resets at 11:00") waits until that time and is rerun, without using up its retries. See Workflows.
Documentation
User Documentation (Portal Docs)
| Document | Description |
|---|---|
| Introduction | What is Milvaion, when to use it |
| Hangfire & Quartz.NET Integration | Keep your existing scheduler, add Milvaion monitoring |
| Api Keys | Credentials for CI pipelines, scripts and MCP clients |
| MCP Server | Connect Claude Code, Cursor or Copilot to Milvaion |
| Jobs as Code | Manage jobs from Git with the milvaion CLI or Terraform |
| Quick Start | Get running in under 10 minutes |
| Core Concepts | Architecture and key terms |
| Your First Worker | Create a custom worker |
| Implementing Jobs | Advanced job patterns |
| Configuration | All configuration options |
| Deployment | Docker and Kubernetes deployment |
| Kubernetes (Helm) | Install with the Helm chart |
| Reliability | Retry, DLQ, zombie detection |
| Scaling | Horizontal scaling strategies |
| Monitoring | Health checks, metrics, logging |
| Workflows | DAG-based job pipelines with conditional branching and data passing |
| Using Kafka | Run Milvaion on Apache Kafka instead of RabbitMQ |
Developer Documentation (GitHub Docs)
| Document | Description |
|---|---|
| Contributing | How to contribute |
| Architecture | Technical architecture deep-dive |
| Development | Development environment setup |
| Worker SDK | Worker SDK reference |
| Worker Protocol | Language-neutral protocol for writing workers in any language |
| Kafka Transport | Design and wire format of the optional Kafka transport |
| Node.js Worker SDK | Running Milvaion jobs in Node.js |
| Go Worker SDK | Running Milvaion jobs in Go |
| Python Worker SDK | Running Milvaion jobs in Python |
| Java Worker SDK | Running Milvaion jobs on the JVM |
| PHP Worker SDK | Running Milvaion jobs in PHP |
| Rust Worker SDK | Running Milvaion jobs in Rust |
| Ruby Worker SDK | Running Milvaion jobs in Ruby |
| MCP Server | MCP server and api key auth internals |
| milvaion CLI | CLI internals, the YAML schema table, releasing |
| Terraform Provider | Provider development, acceptance tests, GPG and registry setup |
| Security | Security policies |
Tech Stack
| Component | Technology |
|---|---|
| Backend | .NET 11, ASP.NET Core |
| Database | PostgreSQL 16, Entity Framework Core |
| Cache/Scheduling | Redis 7 |
| Message Queue | RabbitMQ 3.x, or Apache Kafka 3.x |
| Frontend | React, TypeScript, Vite |
| Real-time | SignalR |
| Logging | Serilog, Seq |
| Metrics | OpenTelemetry, Prometheus |
| Testing | xUnit, FluentAssertions, Testcontainers |
| CI/CD | GitHub Actions, Docker |
Key Libraries
- CQRS: MediatR, Milvasoft.Components.CQRS
- Data Access: Npgsql.EntityFrameworkCore.PostgreSQL, Milvasoft.DataAccess.EfCore
- Authentication: JWT Bearer, Milvasoft.Identity
- API: Asp.Versioning.Mvc, Scalar (OpenAPI)
- Validation: FluentValidation
- Messaging: RabbitMQ.Client
Contributing
We welcome contributions! Please see our Contributing Guide for details.
Quick Contribution Steps
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Make your changes
- Run tests (
dotnet test) - Commit your changes (
git commit -m 'feat: Add amazing feature.') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
Please read our Code of Conduct before contributing.
Packages
NuGet Packages
Installation
# Core SDK (for shared types)
dotnet add package Milvasoft.Milvaion.Sdk
# Worker SDK (for building workers)
dotnet add package Milvasoft.Milvaion.Sdk.Worker
# Worker Templates
dotnet new install Milvasoft.Templates.MilvaionOther Languages
These SDKs speak the same worker protocol as the .NET SDK.
# Node.js
npm install @milvasoft/milvaion-worker
# Go
go get github.com/Milvasoft/milvaion/sdks/go/milvaion
# Python
pip install milvaion-worker
# Java / Kotlin (Maven Central): com.milvasoft:milvaion-worker:1.2.0
# PHP
composer require milvasoft/milvaion-worker
# Rust
cargo add milvaion-worker
# Ruby
gem install milvaion-workerDocker
Docker Images
| Image | Description | Docker Hub |
|---|---|---|
milvasoft/milvaion-api |
Main API with scheduler and dashboard |
Quick Start with Docker
# Pull the latest image
docker pull milvasoft/milvaion-api:latest
# Run with Docker Compose, from the repository root
docker compose up -dAvailable at
- API: http://localhost:5000
- Dashboard: http://localhost:5000
- Grafana: http://localhost:3001 (admin/admin)
- Prometheus: http://localhost:9090
- RabbitMQ Management: http://localhost:15672 (guest/guest)
- Seq Logs: http://localhost:5341
Kubernetes
The Helm chart in deploy/helm/milvaion installs the API and dashboard, the built-in
workers, an optional Ingress with TLS, and, for evaluation, single-pod PostgreSQL, RabbitMQ and Redis:
helm install milvaion oci://ghcr.io/milvasoft/charts/milvaion \
--namespace milvaion --create-namespace \
--set postgresql.enabled=true --set rabbitmq.enabled=true --set redis.enabled=trueFor production, point it at your own PostgreSQL, RabbitMQ and Redis. See the chart README and the Kubernetes guide.
License
This project is licensed under the Apache 2.0 License - see the LICENSE file for details.
Support
- π Documentation
- π Issue Tracker
- π¬ Discussions
Built and maintained by Milvasoft.




