Architecture

stepIQ runs on a horizontally scalable architecture with stateless workers, Redis-backed queues, and PostgreSQL for persistent state.

System components

The platform consists of five main layers that work together to execute pipelines.

Astro 5 + React islands. Pipeline builder, run viewer, and dashboard.

Astro + React

REST API with JWT auth. Handles CRUD, triggers, and SSE streaming.

Node.js + Hono

PostgreSQL for state, Redis for queues and caching, S3/Minio for file storage.

PG + Redis + S3

Stateless BullMQ workers that pick jobs from Redis. N instances, auto-scaling.

BullMQ

Unified proxy layer that routes to Anthropic, OpenAI, or local models.

Multi-provider

When a pipeline run is triggered, this is the step-by-step execution flow inside a worker:

1Job picked up from Redis queue

2Load pipeline definition from pipeline_versions

3Create run record (status: running)

4For each step: interpolate vars, call model, parse output, emit SSE

5Deliver output (webhook, email, API response)

6Update run record, deduct user credits

Each layer of the stack was chosen for reliability, developer experience, and horizontal scalability.

Layer	Choice	Rationale
Frontend	Astro + React	Island architecture for fast loads, React for interactive pipeline editor
API	Node.js + Hono	Lightweight framework with excellent TypeScript support and edge compatibility
Database	PostgreSQL + Drizzle	Relational integrity for pipeline definitions, type-safe ORM with migrations
Queue	Redis + BullMQ	Reliable job processing with retries, rate limiting, and priority queues
Storage	S3 / MinIO	S3-compatible object storage for pipeline artifacts, logs, and large outputs
Auth	JWT + bcrypt	Stateless authentication with secure password hashing and API key support
Validation	Zod	Runtime schema validation for API inputs, pipeline definitions, and config