Datastream Engineer Jobs

1 job found

Today

Senior Data Engineer

Handshake San Francisco, CA

San Francisco, CA

Today

$166k - $207k/yr Full time Senior Apache Airflow Application Security

Apply Now

Job Description

Handshake is hiring a Senior Data Platform Engineer to design and run the infrastructure that powers dependable data movement across its career marketplace and AI products. This role combines workflow orchestration, ingestion and pipelines, streaming and warehouse systems, and technical leadership for the data platform.

Responsibilities

Own the production operation and technical roadmap for the Airflow and Astronomer platform, including deployment, DAG packaging, scheduler and worker capacity, upgrades, observability, and incident response.
Build streamlined paths for authoring, testing, deploying, and operating workflows, including reusable operators, CI checks, and local and staging environments, supported by clear runbooks.
Design and operate streaming and change data capture pipelines using tools such as Pub/Sub, Dataflow or Beam, and Datastream to support analytics and product use cases.
Make data delivery resilient to retries, duplicates, schema changes, late events, backfills, and replay, while defining measurable latency, freshness, and reliability targets.
Strengthen the cloud foundation for data workloads using Kubernetes, Terraform, IAM, secrets, CI/CD, and cost-aware capacity management.
Lead cross-team decisions on data contracts, interfaces, and operational ownership; mentor engineers and align application, analytics, ML, and cloud partners on scalable platform approaches.
Participate in on-call rotation, troubleshoot production failures across systems, and convert incidents into lasting improvements.
Build reliable data and orchestration foundations for AI workloads, including batch inference, evaluation datasets, and agent-facing data.

Requirements

Strong software engineering skills in Python and experience building and operating production data or distributed systems.
Deep, hands-on Airflow experience beyond DAG authoring, including scheduling and execution behavior, deployment, scaling, upgrades, monitoring, and debugging failures.
Experience with event-driven or streaming data infrastructure, including a message broker or managed event bus and a stream processing system.
Sound understanding of CDC, delivery guarantees, idempotency, ordering, schema evolution, and recovery or replay in production pipelines.
Experience with cloud infrastructure and infrastructure as code, comfortable with containers, Kubernetes, CI/CD, access controls, and production observability.
A proven record leading ambiguous infrastructure initiatives, including setting technical direction, making pragmatic architecture tradeoffs, and driving adoption across teams.
Clear communication and experience partnering across teams while owning systems through production support.

Technologies

Python, Airflow, Astronomer
Pub/Sub, Dataflow, Beam, Datastream
Datadog, BigQuery
Kubernetes, Terraform, IAM, CI/CD, GKE, Cloud Storage, Spacelift
dbt, Spark, Dataproc

Compensation

USD 166,000 - 207,000 per year
Handshake states that for cash compensation, standard ranges are set for U.S.-based roles based on function, level, and geographic location, benchmarked against similar stage growth companies. Final offer amounts depend on multiple factors, including geographic location and candidate experience and expertise, and comply with local legislation while providing transparency to candidates.

Benefits

Equity in a fast-growing company
401(k) match, competitive compensation, financial coaching
Paid parental leave, fertility benefits, parental coaching
Medical, dental, and vision, mental health support, and a $500 wellness stipend
$2,000 learning stipend and ongoing development
Internet, commuting, and free lunch/gym in the SF office
Flexible PTO, 15 holidays plus 2 flex days
Team outings and referral bonuses

Extra Credit

Experience with GCP services such as Pub/Sub, Dataflow, Datastream, BigQuery, GKE, and Cloud Storage.
Experience with Astronomer, Apache Beam, Terraform, Spacelift, Datadog, dbt, Spark, or Dataproc.
Experience supporting both batch and low-latency consumers, including product-facing data services or ML features.
Experience supporting ML or AI workloads such as inference pipelines, reproducible evaluations, or governed data access.

Location and Employment

Location: San Francisco, CA
Employment type: Full time
Location type: On-site
Department: Engineering

View full job page →

Get Datastream Engineer Jobs alerts

New Datastream Engineer Jobs, daily.