PipelineKit

The problem

Data ingestion should resume after failure without creating duplicate records or exposing another user’s data.

Technical approach

A small transactional pipeline with checkpoints, strict validation, migrations and an authenticated paginated API. Reuse the same dataset and database rather than introducing another platform.

• Python
• FastAPI
• Pydantic
• PostgreSQL
• pytest
• Docker
• GitHub Actions

Acceptance criteria

1. Interrupt, resume and replay the same batch without changing its final record count.
2. Reject unauthorized access and invalid data.
3. Demonstrate transaction rollback and reproducible migrations.
4. Run integration checks against PostgreSQL.
5. Publish a reproducible 1000-event benchmark with environment, commands, errors and measured latency.

Build log