Senior Data Engineer

Handshake is hiring a Senior Data Platform Engineer to design and run the infrastructure that powers dependable data movement across its career marketplace and AI products. This role combines workflow orchestration, ingestion and pipelines, streaming and warehouse systems, and technical leadership for the data platform.

Responsibilities

• Own the production operation and technical roadmap for the Airflow and Astronomer platform, including deployment, DAG packaging, scheduler and worker capacity, upgrades, observability, and incident response.
• Build streamlined paths for authoring, testing, deploying, and operating workflows, including reusable operators, CI checks, and local and staging environments, supported by clear runbooks.
• Design and operate streaming and change data capture pipelines using tools such as Pub/Sub, Dataflow or Beam, and Datastream to support analytics and product use cases.
• Make data delivery resilient to retries, duplicates, schema changes, late events, backfills, and replay, while defining measurable latency, freshness, and reliability targets.
• Strengthen the cloud foundation for data workloads using Kubernetes, Terraform, IAM, secrets, CI/CD, and cost-aware capacity management.
• Lead cross-team decisions on data contracts, interfaces, and operational ownership; mentor engineers and align application, analytics, ML, and cloud partners on scalable platform approaches.
• Participate in on-call rotation, troubleshoot production failures across systems, and convert incidents into lasting improvements.
• Build reliable data and orchestration foundations for AI workloads, including batch inference, evaluation datasets, and agent-facing data.

Requirements

• Strong software engineering skills in Python and experience building and operating production data or distributed systems.
• Deep, hands-on Airflow experience beyond DAG authoring, including scheduling and execution behavior, deployment, scaling, upgrades, monitoring, and debugging failures.
• Experience with event-driven or streaming data infrastructure, including a message broker or managed event bus and a stream processing system.
• Sound understanding of CDC, delivery guarantees, idempotency, ordering, schema evolution, and recovery or replay in production pipelines.
• Experience with cloud infrastructure and infrastructure as code, comfortable with containers, Kubernetes, CI/CD, access controls, and production observability.
• A proven record leading ambiguous infrastructure initiatives, including setting technical direction, making pragmatic architecture tradeoffs, and driving adoption across teams.
• Clear communication and experience partnering across teams while owning systems through production support.

Technologies

• Python, Airflow, Astronomer
• Pub/Sub, Dataflow, Beam, Datastream
• Datadog, BigQuery
• Kubernetes, Terraform, IAM, CI/CD, GKE, Cloud Storage, Spacelift
• dbt, Spark, Dataproc

Compensation

• USD 166,000 - 207,000 per year
• Handshake states that for cash compensation, standard ranges are set for U.S.-based roles based on function, level, and geographic location, benchmarked against similar stage growth companies. Final offer amounts depend on multiple factors, including geographic location and candidate experience and expertise, and comply with local legislation while providing transparency to candidates.

Benefits

• Equity in a fast-growing company
• 401(k) match, competitive compensation, financial coaching
• Paid parental leave, fertility benefits, parental coaching
• Medical, dental, and vision, mental health support, and a $500 wellness stipend
• $2,000 learning stipend and ongoing development
• Internet, commuting, and free lunch/gym in the SF office
• Flexible PTO, 15 holidays plus 2 flex days
• Team outings and referral bonuses

Extra Credit

• Experience with GCP services such as Pub/Sub, Dataflow, Datastream, BigQuery, GKE, and Cloud Storage.
• Experience with Astronomer, Apache Beam, Terraform, Spacelift, Datadog, dbt, Spark, or Dataproc.
• Experience supporting both batch and low-latency consumers, including product-facing data services or ML features.
• Experience supporting ML or AI workloads such as inference pipelines, reproducible evaluations, or governed data access.

Location and Employment

• Location: San Francisco, CA
• Employment type: Full time
• Location type: On-site
• Department: Engineering