Dataset Processor.
In development
A Go application for deterministic transaction CSV processing, with durable jobs, worker recovery and exact output verification in a local Kubernetes deployment.
APIAccept & validate
CoordinatorPartition & reconcile
WorkersProcess & recover
Proposed architecture · Go / AWS / Kubernetes
Keep the job durable. Make the output verifiable.
Dataset Processor handles deterministic transaction CSV processing through version-pinned inputs, PostgreSQL-backed jobs and worker dispatch. Exact output verification keeps the result tied to the selected input version.
Prove recovery locally.
The local deployment uses Docker, Kubernetes through kind, PostgreSQL and S3/SQS-compatible services. Worker takeover and a database-pod restart have been tested in the local Kubernetes demo. AWS deployment and production throughput are not claimed.