CICD Engine
Overview
A self-hosted, GitHub Actions-inspired CI/CD platform with distributed Docker runners, job scheduling, and real-time log streaming.
CICD Engine is a production-grade clone of GitHub Actions built to demonstrate how distributed CI/CD infrastructure works under the hood. It features YAML-driven workflows, matrix strategy expansion, job dependency graphs, and conditional step execution. The distributed runner system uses atomic job claiming to prevent collisions and runs isolated Docker containers for secure job execution. The React dashboard features Socket.IO-powered live log streaming and ANSI color rendering in real time.
Context
Understanding the intricate details of distributed CI/CD infrastructure is difficult because most existing solutions are either closed source or overly abstracted. It's hard to see how features like job orchestration, secret management, and real-time streaming are implemented under the hood.
Obstacles Faced
Translating incoming webhooks into a queued, parallelized job graph
Preventing distributed runner collision with atomic job claiming
Handling thousands of log lines per minute without database bottlenecks
Injecting secrets into isolated containers without exposing them to the filesystem
Engine Solutions
- Utilized BullMQ backed by Redis for priority queueing and job orchestration
- Implemented PostgreSQL SELECT FOR UPDATE SKIP LOCKED for atomic job claims
- Architected a live log pipeline bridging Docker streams to a React dashboard via Redis Pub/Sub and Socket.IO
- Stored AES-256 encrypted secrets and injected them dynamically as environment variables into runner containers
Outcomes & Impact
Demonstrated complex backend scalability and distributed concurrency models
Successfully processed matrix jobs and dependencies in isolation
Achieved sub-200ms latency for streaming terminal-colored execution logs
System Architecture
Event-driven architecture with HMAC-SHA256 verified webhooks
Distributed resource-limited Docker workers
In-memory log buffering with batched persistence
Direct artifact uploads to AWS S3 / Cloudflare R2
Interface Snapshots
Execution Lessons
"Deepened understanding of race conditions and concurrency control in distributed systems"
"Mastered container lifecycle management via Docker APIs from a Node.js context"
"Learned optimization techniques for high-frequency real-time events over WebSockets"