Himanshu Pant

Data Engineer · Tempe, AZ

I build the pipelines underneath things, and I can prove the data is right when I'm done.

1B+records migrated off a legacy platform, certified at 99.5%
86%batch runtime cut on that migration, six hours to fifty minutes
5+ yrsproduction data engineering across insurance, banking, healthcare and sports

Open to Data Engineer roleshpant.data@gmail.comLinkedInGitHub

What I do

Messy sources in, trusted data out.

Five years building batch and streaming pipelines on Spark, Databricks, Snowflake and AWS. The part I'm actually known for is the unglamorous half: reconciling sources that disagree with each other, and building the validation frameworks that let someone sign off on a number. I've been the only data engineer on a product, and I've delivered as a consultant to banking and insurance clients. Both taught me that a pipeline nobody trusts isn't finished.

Selected measurements

Before, and after.

Most of what I've shipped can be stated as a number that moved. These are the ones I'd defend in an interview.

Legacy SAS batch, migrated to Spark on Azure DatabricksFuturense · partitioning, file sizing, broadcast joins
6 hrs→50 min
PySpark pipeline missing its daily SLA, profiled rather than guessed atEXL · one 90-minute bottleneck in a 100M+ record run
baseline→−66%
Model features served from a Kafka and Iceberg feature storeEXL · continuous instead of nightly, wired into SageMaker
nightly→−58% cycle
Migration correctness, certified by a reconciliation framework I builtFuturense · every failed record investigated, none dropped silently
99.5%
Ten-plus third-party feeds that disagreed, reconciled under live loadSuper Six · cross-source validation, errors visible to users in minutes
99.9%
User churn, against a randomly assigned holdout control groupSuper Six · behavioural model at 78% accuracy
baseline→−20%
Per-query inference cost, after tracing where spend concentratedCurrent role · complexity-based model routing
baseline→−82%
Partial vendor deliveries processed as complete runs, caught and closedEXL · custom Airflow sensor replacing a stock key sensor
silent→zero
Manual executive reporting, replaced with scheduled SQL and shellKoron · $50M portfolio, 50+ sites
20 hrs/mo saved

Experience

Where I've worked.

Jan 2026 – PresentTempe, AZ

MyEdmasterCurrent

Software Engineer

Started as the capstone engineer on the platform, stayed on to build it for production. Details of the product are under NDA.

  • Leading the migration onto AWS: document storage to S3, model inference to Bedrock, the service layer onto EKS with processing workloads on ECS.
  • Built the document ingestion and embedding pipeline that processes 5,000 documents into 200K+ metadata-tagged vector chunks in Qdrant, and now scale it as the corpus grows.
  • Cut per-query inference cost 82% by tracing execution to find where spend concentrated, then routing by query complexity rather than capping usage.
  • Hardening it for production: pytest coverage, a structured logging layer across ingestion and retrieval, and answer-quality benchmarking so changes are measured before release.
Jun – Jul 2026Tempe, AZ

Arizona State University

Graduate Researcher, Power Analytics Lab

Research position under Prof. Anamitra Pal, grounding LLMs in power systems domain knowledge.

  • Built the ingestion and chunking layer over 11 NERC reliability standards and power-systems papers, writing rule-based chunkers tuned to NERC numbered-requirement structure rather than a generic splitter, producing 469 metadata-tagged chunks.
  • Benchmarked embedding models, selected bge-small-en-v1.5, and built a FAISS cosine-similarity index with top-k search and domain filtering.
  • Ran an 18-question retrieval evaluation and documented four reproducible failure modes rather than reporting only what worked.
Jul 2023 – Mar 2024Boston, MA (Remote)

EXL Services

Consultant II, Data Engineering

Delivered to banking and insurance clients, working on fraud and risk data.

  • Owned the lakehouse behind products serving 500K+ users: S3 and Apache Iceberg ingestion, a dbt transformation layer, and curated Snowflake tables analysts queried directly. Made the table layout and partitioning calls that set read throughput.
  • Cut a 100M+ record PySpark pipeline's runtime 66% on AWS Glue and EMR by profiling real execution to find a single 90-minute bottleneck, rather than guessing at it.
  • Built a Kafka and Spark Structured Streaming feature store on Iceberg feeding SageMaker, serving features continuously instead of nightly and cutting model deployment cycles 58%.
  • Rebuilt the nightly Airflow DAG orchestrating eight PySpark-on-Glue jobs across claims, policy, vehicle and driver domains, restructuring a fully sequential chain so the five independent tasks ran in parallel.
  • Replaced the stock S3 key sensor after finding it fired on the first file of a multi-file vendor delivery, so Glue processed partial batches as clean green runs. Wrote a sensor that waits for the delivery to stop changing, polling in reschedule mode so a six-hour wait never holds a worker slot.
  • Carried production ownership: Terraform with IAM and encryption controls, pytest-driven Jenkins CI/CD, CloudWatch alerting, and the on-call pager.
  • Led design and code reviews for the pod, ran daily Scrum, mentored two junior engineers.
Aug 2022 – Jul 2023Gurugram, India

Super Six Sports GamingPromoted in 6 months

Data Engineer

Only data engineer on the product. No existing pipeline, no playbook, nobody to escalate to.

  • Designed the MySQL schema and complete data model from scratch, fact and dimension tables included, then ran every implementation from requirements through production and on-call.
  • Built ingestion across 10+ third-party feeds on sub-hourly refresh and held it to 99.9% accuracy through cross-source validation, during live events where a wrong number was visible to users within minutes.
  • Adjudicated source conflicts in real time: traced discrepancies back to the originating vendor, decided which feed to trust when two diverged mid-event, and drove the fix with that vendor.
  • Shipped a production retention model on behavioural event data at 78% accuracy, cutting churn 20% against a randomly assigned holdout control group.
Oct 2021 – Jul 2022Bangalore, India

Futurense Technologies

Data Engineer

Client engagement in healthcare. Client-facing on scoping and status.

  • Co-led migration of 1B+ health insurance claims off a legacy SAS platform onto Spark on Azure Databricks, building the claims history into per-patient treatment timelines so clinicians could see where a patient actually stood.
  • Compressed batch execution from six hours to fifty minutes through partitioning scheme, file sizing and join strategy, so the new platform was measurably faster than the one it replaced rather than just newer.
  • Built the SQL validation, profiling and reconciliation framework that certified the cutover at 99.5%, with explicit exception handling so every failed record was investigated instead of quietly dropped.
  • Automated recurring Python processing against AWS Athena, removing 80% of the manual effort.
Oct 2018 – Jul 2021Gurugram, India

Koron Projects Limited

Executive Analyst, Data and Reporting

Three years inside the legacy stack.

  • Wrote PL/SQL stored procedures, packages and cursor-based routines across Oracle, MS SQL Server and MySQL, and tuned the long-running ones to stay inside their processing windows as volume grew across 50+ sites.
  • Consolidated three disconnected source systems into one reporting model spanning a $50M portfolio.
  • Replaced manual executive reporting with scheduled SQL workflows, saving 20 hours a month, and maintained 15+ operational dashboards for leadership.

Projects

Things I built on my own time.

Mostly agent and LLM systems, shipped at hackathons in 2026. I use Claude Code and Cursor daily, so this is where I keep that current.

MeetFlow

1st place · DEVHACKS 2026

An LLM pipeline that turns meeting transcripts into assigned, capacity-aware tickets. Analyses the transcript, checks team load, recommends reassignment for overloaded people, and notifies via Slack. Competed against 100+ teams.

Python · GPT-4o-mini · Taiga API · Slack · Streamlit

FairCharge

Solo build

A medical bill audit pipeline on the Claude API. Reads the bill, itemises every charge, benchmarks against CMS Medicare pricing for the patient's state, flags overcharges, and drafts the dispute letter. Built because an AI pipeline's output is only worth what you can check it against.

Claude Vision · SapBERT · ChromaDB · CMS data

TrustSeal

VillageHacks 2026

Multi-provider identity document verification against the AAMVA 2020 standard. Claude agents analyse the document independently, a local PDF417 decode acts as ground truth, and a judge agent resolves conflicts into an approve, review or reject recommendation.

Anthropic Claude · Multi-agent · PDF417 · FastAPI

Serverless Inference Platform

Academic

Multi-stage inference on Lambda, SQS and ECR with containerised PyTorch models. Asynchronous request and response queues decouple the stages, so a slow recognition step can't back-pressure ingestion when frame volume spikes.

AWS Lambda · SQS · ECR · Docker · PyTorch

Stack

What I actually use.

Listed by how much I've shipped with it, not by how it looks on a resume. Everything here I'd take a question on.

Core
Python, SQL, PySpark, shell, Java
Processing
Apache Spark, Spark Structured Streaming, Databricks, Apache Kafka, AWS Glue, EMR
Warehouse & lake
Snowflake, dbt, Apache Iceberg, S3 data lakes, dimensional modelling, partitioning and query tuning
Orchestration
Apache Airflow and AWS MWAA, Jenkins CI/CD, pytest, Git
Databases
Oracle PL/SQL, MS SQL Server, PostgreSQL, MySQL, MongoDB
Cloud & infra
AWS (S3, Lambda, SQS, ECR, Athena, RDS, SageMaker, IAM, CloudWatch), Azure, Terraform, Docker, Kubernetes
Data quality
Validation and reconciliation frameworks, Great Expectations, profiling, monitoring and alerting, incident triage
AI
LangGraph, LangChain, Anthropic Claude API, RAG and embeddings (Qdrant, pgvector), LangSmith, Claude Code and Cursor

Credentials

Certified, and one award.

AWS Certified Data Engineer

Associate · DEA-C01 · May 2026

Microsoft Certified: Fabric Analytics Engineer

Associate · DP-700 · May 2026

Databricks Certified

Generative AI & Lakehouse

SPOT Award, Exceptional Delivery

Super Six Sports Gaming

M.S. Software Engineering, AI SpecializationArizona State University · 2024–2026
B.Tech. Computer Science and EngineeringGGSIPU, Delhi · 2018

Contact

Let's talk.

Open to Data Engineer, Data Platform and Analytics Engineering roles working on large-scale pipelines, migrations and the systems that feed models. Based in Tempe, AZ and willing to relocate anywhere in the US for the right role.

hpant.data@gmail.comlinkedin.com/in/himanshupant-degithub.com/hpant5

Himanshu Pant · Tempe, AZ · 2026