Bharath Surampudi

Data Engineer building and operating regulated payments data platforms on Snowflake, dbt, Airflow and AWS (real-time streaming, governance, observability, AI-assisted engineering).
Sydney, Australia · Permanent Resident
bharathsurampudi@gmail.com | 0410 638 861 | View on LinkedIn | View on GitHub
Download Resume (PDF) View Case Studies

Summary

Data Engineer at Australian Payments Plus (AP+), running and building the data platform behind NPP, eftpos and ConnectID. 5 years in regulated payments, including 4+ years at Mastercard. AWS Certified Data Engineer and SnowPro Core certified.

Systems I Have Built & Operated

AP+

Real-Time Payments Streaming into Snowflake

Confluent Kafka · Apache Flink · Snowflake

  • Contributed to a real-time platform streaming national payments events into Snowflake.
  • Runs alongside 200+ batch Airflow DAGs on Snowflake and dbt.
  • Event streaming via Confluent Kafka and Apache Flink.
Streaming Flink Snowflake
AP+

AI-Assisted Data Engineering

Snowflake Cortex Code · Kiro · Claude Code

  • Custom agent skills that standardise spec-driven development (requirements → design → tasks).
  • Team standards encoded into the workflow: dbt naming and required tests, Airflow DAG conventions, Snowflake user onboarding and offboarding.
  • LLM-maintained documentation for Airflow DAGs and dbt models.
Agent Skills Spec-Driven Dev LLM
Personal project

Real-Time Fraud Detection Platform View on GitHub

Kinesis · Lambda (Python) · DynamoDB · S3 · Terraform

  • Event-driven streaming pipeline for transaction scoring with latency and reliability constraints.
  • Built for at-least-once delivery with application-layer idempotency and safe retries.
  • Cold-path to S3 for auditability and offline analysis.
Streaming Idempotency NoSQL Modeling
Personal project

Medallion Lakehouse with dbt + Redshift View on GitHub

S3 · Glue (PySpark) · Redshift · dbt · Airflow · Terraform

  • Medallion-style ingestion and transformation layers: raw → curated → analytics-ready.
  • dbt tests and documentation to enforce contracts and reduce downstream defects.
  • Orchestrated and deployed with CI/CD practices appropriate for data workloads.
ELT Governance Data Quality
Mastercard

Enterprise Payment Data Ingestion & Routing

Apache NiFi · Java · Splunk · Regulated Data Flows

  • Secure ingestion and routing across global regions with operational rigor.
  • Observability based on provenance/log instrumentation to reduce MTTR.
  • Schema enforcement to protect downstream reporting and regulatory accuracy.
Reliability Observability Contracts

Architecture Case Studies

AP+
Case Study 1 — Snowflake RBAC: Proof of Concept to Production
Stack: Snowflake RBAC, PIM / just-in-time access Design Focus: least privilege, auditability, a role hierarchy that scales across projects
  • Problem: Multiple data projects holding sensitive national payments data needed one consistent, auditable least-privilege access model.
  • Key decision: Prove the design in a proof of concept before committing to a production rollout, so the role model was validated on real workloads first.
  • Implementation: Defined the Snowflake role hierarchy and rolled it out across 5 projects; moved privileged admin roles to just-in-time access through PIM.
  • Outcome: Least-privilege access live across 5 projects, with no standing admin privileges, owned end-to-end from proof of concept to production.
Case Study 2 — Event-Driven Fraud Platform View on GitHub
Stack: AWS Kinesis, Lambda (Python), DynamoDB, S3, Terraform Design Focus: delivery semantics, idempotency, hot partitions, auditability
  • Problem: Score high-velocity transactions with reliability constraints under at-least-once delivery.
  • Key decision: Apply exactly-once semantics at the application layer using idempotency keys and safe retries.
  • Data modeling: DynamoDB patterns designed to avoid hot partitions; atomic counters for velocity rules; history tracking for location signals.
  • Operational posture: Structured logging/metrics and failure modes designed for fast diagnosis and replay.
Case Study 3 — Medallion Lakehouse with dbt + Redshift (Governed Analytics) View on GitHub
Stack: S3, Glue (PySpark), Redshift, dbt, Airflow, Terraform Design Focus: data contracts, testability, reproducibility, CI/CD for data
  • Problem: Convert raw JSON events into analytics-ready dimensions/facts with controlled quality.
  • Key decision: Separate ingestion (Bronze), standardization (Silver), and business modeling (Gold) for clarity and governance.
  • Quality gates: dbt tests (schema, not-null, accepted values) and documentation as first-class artifacts.
  • Infra discipline: Terraform for reproducible environments; orchestration to support backfills safely.

How I Approach Data Engineering

Professional Experience

Data Engineer
Australian Payments Plus (AP+), Sydney Mar 2026 – Present
  • Access governance (owned end-to-end): Built the proof of concept and production solution for Snowflake RBAC, defining the role hierarchy and rolling out least-privilege access across 5 projects handling sensitive payments data.
  • Payments data operations: Run BAU for 200+ production Airflow DAGs across NPP, eftpos and ConnectID on Snowflake and dbt, resolving 5–20 production incidents a month through triage, root-cause fixes and reruns.
  • Reducing BAU toil: Built pipeline health monitoring for Airflow and GitLab CI with downstream Snowflake lineage impact tracking, and a Dynatrace audit-log pipeline (S3 → Snowpipe → Snowflake → dbt).
  • AI-assisted engineering: Developed custom agent skills for Snowflake Cortex Code and Kiro that encode team standards (dbt naming and required tests, Airflow DAG conventions, Snowflake user onboarding and offboarding) into a spec-driven development workflow, and built an LLM-maintained documentation system for DAGs and dbt models.
  • Privileged access: Implemented PIM just-in-time access for admin roles, removing standing privileges.
  • Real-time streaming: Contributed to a real-time transaction data platform streaming payment events via Confluent Kafka and Apache Flink into Snowflake.
Software Development Engineer II (Data & Payment Gateway Systems)
Mastercard, Sydney Nov 2021 – Feb 2026
  • Observability & incident reduction: Reduced MTTR for data transfer incidents by ~40% with Splunk dashboards and NiFi provenance log parsing.
  • High-volume ingestion & routing: Architected and maintained Apache NiFi + Java pipelines routing payment transaction data across regions, achieving 99.99% availability under strict SLAs.
  • Cross-region migration: Delivered pipelines moving multi-terabyte datasets securely across global regions in PCI-regulated environments.
  • Schema enforcement: Consumer-driven contract testing (Spring Cloud Contract) across 5+ producer and consumer services, protecting financial and regulatory reporting.
  • Legacy modernisation: Partnered with Principal Architects to decouple monolithic ETL into modular, cloud-aligned ELT patterns, improving throughput and maintainability.
  • Automated data quality: Built validation workflows integrated into CI/CD to keep compliance standards consistent.
Software Engineer
Neau Collective, Sydney Mar 2021 – Nov 2021
  • Automated ingestion: Python pipelines extracting and normalising Shopify and marketing data, reducing manual reporting effort by ~70%.
  • Analytics-ready datasets: Unified Sales/Marketing/Accounting data into consistent datasets for BI and reporting.

Capabilities

Streaming & Event Processing Confluent Kafka, Apache Flink, Kinesis, delivery semantics, idempotency patterns, retries, replay strategies.
Batch & Analytical Processing Snowflake (Snowpipe, Time Travel), dbt, Airflow, Spark / PySpark, Glue, SQL (window functions, optimization).
Data Modeling & Governance Dimensional modeling, medallion architecture, schema evolution, data contracts, RBAC, PIM / just-in-time access, dbt tests & docs.
Infrastructure & Reliability Terraform, CI/CD (GitLab CI, GitHub Actions), Docker, observability (Splunk, Dynatrace), production incident response.
AWS Platform S3, Glue, Redshift, Lambda, DynamoDB, IAM fundamentals for governed data systems.
Programming Python (boto3, data pipelines), Java (Spring Boot), strong engineering hygiene and testing.
AI-Assisted Engineering Agent skill development, spec-driven development, Snowflake Cortex Code, Kiro, Claude Code, LLM documentation automation.

Certifications & Credentials

Education

Master of Information Technology Verify ↗
University of New South Wales (UNSW), Sydney Feb 2019 – May 2021
Dual Specialisation: Artificial Intelligence & Database Systems
Bachelor of Technology in Computer Science
Vellore Institute of Technology, India Jun 2014 – May 2018