Repository: https://github.com/Yash170204/Containerized-DP

Containerized Self-Service Data Platform

A production-style, containerized data ingestion platform that simulates a modern self-service data platform architecture using Docker, PostgreSQL, and MinIO (S3-compatible object storage). The system ingests raw CSV files, stores them in object storage, and maintains metadata tracking in a relational database.

---

System Architecture

This diagram shows the end-to-end flow of the containerized self-service data platform.

Architecture Overview

This project follows a modular microservice-style data platform design.

High-Level Flow

Local File System
      │
      ▼
Ingestion Service (Python)
      │
      ├── Upload raw files → MinIO (Object Storage)
      │
      └── Register metadata → PostgreSQL

Components

Component	Purpose
Docker Compose	Orchestrates all services
Ingestion Service	Batch ETL service for file ingestion
PostgreSQL	Metadata registry and tracking
MinIO	S3-compatible object storage

---

Technology Stack

• Docker & Docker Compose
• Python 3.10
• PostgreSQL 15
• MinIO Object Storage
• Boto3 (AWS S3 SDK)
• Psycopg2
• Pandas

---

Folder Structure

containerized-self-service-data-platform/
│
├── docker-compose.yml
├── Dockerfile
├── requirements.txt
├── data/
│   └── raw/
│       └── sales_data.csv
├── db-init/
│   └── 01_schema.sql
└── src/
    ├── main.py
    ├── s3_client.py
    └── db_client.py

---

Features

• Automated CSV ingestion
• Object storage integration (MinIO)
• Metadata tracking in PostgreSQL
• Schema auto-initialization
• Duplicate file detection
• Container health checks
• Fully reproducible environment

---

Database Schema

Metadata Schema

metadata.file_registry

Columns

Column	Type	Description
file_id	SERIAL	Primary key
file_name	TEXT	Uploaded file name
bucket_name	TEXT	Target MinIO bucket
file_size_bytes	BIGINT	File size
status	TEXT	Processing status
ingested_at	TIMESTAMP	Ingestion timestamp

---

Docker Services Running

Docker Compose successfully builds and runs all services.

Docker Compose Startup

Running Containers

Ingestion Pipeline Execution

Below shows the ingestion service processing files and uploading data.

Object Storage (MinIO)

Raw CSV files are uploaded into MinIO using S3-compatible API.

Metadata Storage (PostgreSQL)

File ingestion metadata is stored inside PostgreSQL.

Table Schema

Metadata Records

How to Run the Project

Prerequisites

• Docker Desktop installed
• Docker Compose available

---

Step 1 - Clone Repository

git clone <your-repo-url>
cd containerized-self-service-data-platform

---

Step 2 - Start Platform

docker-compose up --build

This will automatically:

• Start PostgreSQL
• Start MinIO
• Initialize database schema
• Run ingestion job

---

Step 3 - Access MinIO Console

Open in browser:

http://localhost:9001

Login credentials (from docker-compose):

Username: minioadmin
Password: minioadmin

Navigate to:

Buckets → raw-data

Verify:

sales_data.csv

---

Step 4 - Verify Metadata in PostgreSQL

Enter PostgreSQL container:

docker exec -it platform_postgres psql -U admin -d dataplatform

Run query:

SELECT * FROM metadata.file_registry;

Expected Output:

sales_data.csv | raw-data | PENDING

---

Batch Ingestion Behavior

The ingestion service is designed as a batch job.

Behavior:

• Starts
• Scans raw directory
• Uploads files
• Registers metadata
• Exits

This design simulates real-world ETL batch pipelines.

---

Error Handling

The system handles:

• Duplicate file detection
• Missing bucket creation
• Database connection retries
• Transaction rollback on failure

---

Health Checks

Docker Compose includes:

• PostgreSQL health check
• MinIO health check

This ensures ingestion waits until dependencies are ready.

---

Screenshots

Screenshots included in repository (MinIO Console, Docker Logs, Database Output)

---

Future Enhancements

Planned improvements:

• Processed data bucket
• File lifecycle states (INGESTED, FAILED)
• Scheduled ingestion (cron/Airflow)
• Data quality validation
• REST API for metadata access
• Analytics query layer

---

Use Case

This project simulates enterprise data ingestion architecture used in:

• Data warehouses
• Lakehouse platforms
• Analytics pipelines
• Self-service BI platforms

---

Author

Yash Khurana

---

License

This project is open-source and intended for educational and portfolio demonstration purposes.