jr

I wanted to see how much of a distributed job runner I could build from scratch, without relying on external libraries. The result is a system of roughly 2,000 lines of C++ that can run jobs across multiple machines on a local network.

The scheduler manages the job queue, tracks available workers, and handles dependencies between jobs. Workers pick up tasks, execute them, and report back. The CLI lets you submit jobs, check their status, stream logs, and take workers offline when needed.

Along the way, I added features like dependency graphs, retries with backoff, placement constraints, crash recovery, live logs, and resource limits.

The 3 core functions

The same executable can run in three different modes: scheduler, worker, or CLI. Its first argument determines which one to start.

The scheduler (schedd) is the only process that maintains state. It keeps track of the job queue, registered workers, and job dependencies.

Workers connect to the scheduler and register their available capacity. When assigned a job, they execute it using fork and execvp, then send the result back. The CLI communicates with the scheduler over TCP, sending commands and displaying the responses. It doesn't maintain any state of its own.

How messages are sent

The scheduler and workers communicate using length-prefixed TCP frames. Each frame starts with a big-endian u32 containing the message length, followed by a u8 identifying the message type and the payload.

For the payload, fields are separated using 0x1f, while lists within a field use 0x1e. There are 18 message types covering the different interactions between the scheduler, workers, and CLI.

I chose to keep the protocol simple rather than introduce something like Protocol Buffers. The entire codec is around 50 lines of code, and parsing mainly comes down to splitting fields using the separator bytes.

One limitation is that field values cannot contain either separator byte. The system checks this when a job is submitted and cleans captured output before sending it.

Scheduling and failure

Dispatch walks the queue in priority order, then by submit time. Placement is best-fit: the worker with the least free CPU that can still run the job, while respecting --require labels. Smaller jobs can still be placed while a large one is waiting for a worker. --after holds a job in blocked until every dependency is done. Ids are assigned at submit time, so a job can only depend on jobs that already exist.

Workers send a heartbeat every two seconds. If a worker is quiet for six seconds, it is marked lost and its running jobs go back on the queue, up to each job's retry limit.

--limit is a hard memory cap. The amount the scheduler reserved becomes RLIMIT_AS on the child. Cancelling a job sends SIGKILL to its process group. The child calls setsid when it starts, so the signal also reaches anything that job spawned. jr drain marks a worker as draining. Tasks already running finish, new work is sent elsewhere, and undrain accepts work again.

Journal

Scheduler state is an append-only text file. S lines are submits. T lines are state changes, including the run count, exit code, and a short note. A W line stores the latest id. On startup the file is replayed and the queue is rebuilt. Jobs that were running are put back on the queue. The file is rewritten on a clean shutdown, and again about every 5,000 lines.

jr logs -f streams output over the same connection. The worker tails the task's spool file and sends chunks through the scheduler to whoever subscribed. The last 48 KB are saved with the result, so jr info can show a tail after the job finishes.

Limits

The protocol assumes a trusted network. There is no authentication or encryption, so it is meant for localhost and a private LAN. Dispatch scans queued jobs against workers, which is fine up to roughly 10,000 jobs. RLIMIT_AS measures virtual address space, so a runtime that maps a lot of memory needs a larger --mem than its resident set would suggest.