EngineerJobs.io
← Back to all jobs

Job Description

Build and evolve a large-scale offline data platform that generates training datasets and orchestrates reliable ML data workflows.

Responsibilities

  • Develop big data infrastructure supporting batch and stream processing using technologies such as Flink, Spark, Ray
  • Design and operate large-scale data pipelines that produce training datasets for ML training and experimentation
  • Integrate training-data pipelines with workflow orchestration systems such as Flyte, Airflow (or similar) for reliable multi-stage training workflows
  • Increase reproducibility and observability via dataset validation, monitoring, and automated testing
  • Optimize performance and resource utilization across distributed compute systems used for data processing
  • Partner with ML engineers to support efficient large-scale experimentation and iterative model development
  • Lead architectural improvements to keep offline data pipelines scalable, reliable, and cost-efficient

Requirements

  • Experience working with distributed computing frameworks including Flink, Spark, Ray for distributed data processing
  • Experience building infrastructure for training data generation, dataset preparation, or ML feature pipelines
  • Experience optimizing big data pipelines and infrastructure for cost efficiency
  • Strong Python programming skills with experience handling large-scale distributed workloads
  • Experience with modern data infrastructure such as data lakes, warehouses, orchestration systems, and streaming platforms
  • Strong systems thinking to evaluate performance, scalability, reliability, and cost tradeoffs in distributed systems
  • Demonstrated ability to lead technical direction and influence architectural decisions across teams without formal authority

Technologies

  • Flink
  • Spark
  • Ray
  • Flyte
  • Airflow
  • Python

Location and Compensation

  • Location: Mountain View, CA (onsite)
  • Salary: USD 165,600 - 273,400 per year

Benefits

  • Comprehensive health, life, and disability insurance
  • Commute subsidy
  • Employee stock ownership
  • Competitive retirement/pension plans
  • Generous vacation and personal days
  • Support for new parents through leave and family-care programs
  • Office food snacks
  • Mental Health and Wellbeing programs and support
  • Employee Resource Groups
  • Global Employee Assistance Program
  • Training and development programs
  • Volunteering and donation matching program
  • Equity awards
  • Participation in company incentive plans (for example, annual discretionary bonuses or sales commissions)

Benefits eligibility, specific offerings, and coverage vary based on country and employment status.

Similar Jobs