Building reliable data platforms, real-time pipelines, and AI-ready data products with Python, PySpark, Kafka, Databricks and Delta Lake.
I specialize in turning high-velocity event streams into reliable, AI-ready data products. With hands-on experience across the full data lifecycle, I design, build, and support production-grade data platforms.
I am a Data Engineer with production experience at Accenture, building configuration-driven data pipelines, streaming systems, and lakehouse platforms. I focus on the engineering discipline that makes production systems reliable — testing, CI/CD, observability, and data quality.
My work spans real-time streaming with Kafka and PySpark, lakehouse architecture with Databricks and Delta Lake, and AI-ready data foundations for RAG applications. I contribute to open-source projects and build developer tools like the DataOps Toolkit.
# Platform in a nutshell data_source = "Kafka" processing = "PySpark Structured Streaming" storage = "Delta Lake" ci_cd = "GitLab CI/CD" testing = "pytest, comprehensive coverage" platform = "AI-ready, scalable, secure"
Configuration-driven pipelines, lakehouse architecture, enterprise data products
Kafka, PySpark Structured Streaming, exactly-once processing, real-time ETL
Databricks, Delta Lake, schema enforcement, time travel, ACID transactions
Schema validation, reconciliation, testing, error handling, monitoring
CI/CD, configuration-driven code, reusable utilities, developer tooling
Vector databases, embeddings, retrieval-augmented generation, data foundations
Azure, Terraform, Event Hubs, ADLS Gen2, deployment automation
DataOps Toolkit, CLI tools, testing frameworks, documentation practices
Hands-on development and validation of configuration-driven data pipelines serving enterprise data products.
The data platform patterns I work with to build reliable, scalable systems.
Real-world solutions with measurable business impact and production-grade reliability.
Modular Python CLI toolkit for data engineering operations
Problem: Complex data environments require repeatable lineage, schema validation, quality checks and metadata analysis.
Solution: A modular Python CLI toolkit with 6 production modules, 23 tests, and comprehensive documentation.
Problem: Fixed critical bug in Dagster's AssetNode parent_keys handling affecting data pipeline reliability.
Outcome: PR #34143 submitted, regression tests added, improving reliability for thousands of users.
Problem: Build scalable real-time streaming pipelines with exactly-once delivery guarantees.
Outcome: High-throughput data processing with exactly-once delivery and Delta checkpointing.
Problem: Build multi-environment cloud infrastructure for real-time data processing.
Outcome: IaC-managed platform with exactly-once streaming and Delta Lake checkpointing.
Problem: Build a personal media platform for organizing movies, anime, series, games and music with external metadata integrations.
Outcome: Next.js application with TMDB/AniList/IGDB integrations, catalog management, collections, favorites, and responsive design.
Problem: Build intelligent document Q&A with retrieval-augmented generation.
Outcome: FastAPI + Streamlit app with vector search and configurable LLM backends.
Problem: Scale data pipeline development across enterprise business domains.
Outcome: Delivered numerous production jobs, improved deployment efficiency, maintained high availability, achieved comprehensive test coverage.
Key signals about my engineering profile and career direction.
Enterprise Data Engineering at Accenture with production pipeline ownership and live monitoring
14+ public contributions to Dagster, urllib3, axios, fastify, and other major projects
Databricks Data Engineer Associate, Microsoft DP-700 Fabric, Google Data Analytics
Streaming pipelines, lakehouse architecture, data quality, and AI-ready data platforms
Data Platform Engineering with focus on streaming, lakehouse, and AI-ready data foundations
14+ Public Pull Requests across major open-source projects. I don't just build projects — I contribute to production software used by millions.
Problem: Critical bug in AssetNode parent_keys handling affecting data pipeline reliability
Solution: Fixed mutation bug with defensive copy and added regression tests
Technology: Python, pytest
View PR →Problem: TypeScript documentation referenced outdated Fastify version
Solution: Updated docs to reference Fastify 5.x
Technology: TypeScript, Documentation
View PR →Problem: README stream example missing fs import
Solution: Added missing fs import to fix the example
Technology: JavaScript, Documentation
View PR →Problem: No way to filter findings by severity
Solution: Added --min-severity flag for severity filtering
Technology: Python, CLI
View PR →Problem: Missing secure error reporting documentation
Solution: Added secure error reporting section to docs
Technology: TypeScript, Documentation
View PR →Problem: README Node.js requirement misaligned with package.json
Solution: Aligned README requirement with actual package.json
Technology: Documentation
View PR →Core technologies organized by the engineering domains I work in.
Industry-recognized credentials that back the skills and projects displayed here.
Accenture
December 2023 - February 2024
IIT Kharagpur
Elite · 73% · July–October 2022
Verify Credential →Deep dives into data platform engineering, streaming patterns, and career learnings.
Hands-on guide to production PySpark Structured Streaming with Kafka, Delta Lake, and cloud deployment.
Read Article →How to build a retrieval-augmented generation chatbot with document ingestion, vector search, and LLM generation.
Read Article →My study plan, resources, tips, and what actually helped me pass the Databricks Data Engineer Associate exam.
Read Article →Madanapalle Institute of Technology & Science, Madanapalle, AP
2019–2023 · 7.97 CGPA · First Class Distinction
Sri Vivekananda Junior College, Chittoor, AP
2017–2019 · 9.67 CGPA
Z.P.H.S Kanipkam, Chittoor, AP
2016–2017 · 9.3 CGPA
94.17 Percentile · 2019
Quantifiable results and engineering discipline that delivers reliable data systems.