EngineerJobs.io
← Back to all jobs

Job Description

An Autonomous Mobility Company is seeking a Data Engineer for Fleet Operations to design, build, and maintain data pipelines and models that power a Manufacturing Intelligence data lakehouse. This role focuses on integrating manufacturing data into the MOPS lakehouse and enabling reporting, dashboards, and AI-related applications using Databricks and AWS ingestion infrastructure.

Based in Foster City, CA, this onsite contract role supports production-grade data engineering across Delta Lake, dbt, and orchestration workflows.

Role Focus

  • Build and maintain batch and streaming ingestion pipelines into Delta Lake using Auto Loader, Lakeflow Declarative Pipelines, Lakeflow Connect, or Structured Streaming, including CDC feeds from AWS DMS and SAP SLT.
  • Design and maintain bronze, silver, and gold data layers using MOPS naming conventions and data dictionary standards.
  • Develop incremental dbt models on Databricks using MERGE, upsert, and SCD Type 2 patterns, including late or out-of-order record handling.
  • Model manufacturing data such as unit-level traceability (SFC), work orders, nonconformances, station results, and first-pass yield to support analytics, dashboards, and semantic reporting.
  • Process large-scale and semi-structured datasets with PySpark and Databricks SQL, optimizing performance and cost using clustering, partitioning, OPTIMIZE, VACUUM, file sizing, and compute configuration.
  • Maintain data quality and reliability using dbt tests, Lakeflow expectations, Lakehouse Monitoring, freshness SLAs, and automated alerts.
  • Orchestrate workloads with Databricks Workflows or Airflow, and support Redshift-to-Databricks migration through SQL conversion, parallel validation, reconciliation, and cutover.
  • Collaborate with analysts and manufacturing teams to define data requirements and document lineage, data contracts, and table definitions in Unity Catalog.
  • Build and maintain CI/CD pipelines using Databricks Asset Bundles and GitHub Actions across development, staging, and production.
  • Manage MOPS-specific Databricks configurations including Unity Catalog permissions, compute resources, SQL warehouses, service principals, secrets, and OAuth authentication, aligned with central platform governance.
  • Configure and troubleshoot AWS ingestion infrastructure including DMS, Kinesis, S3, and IAM, coordinating Terraform-based provisioning with infrastructure teams.
  • Monitor pipeline health and performance, manage Databricks Unit (DBU) costs, maintain operational documentation, and support incident response and root cause analysis.

Requirements

  • Bachelor’s degree in computer science, information systems, engineering, or a related discipline (or equivalent practical experience).
  • 6+ years of data engineering experience.
  • Strong proficiency in SQL, Python, and PySpark.
  • Hands-on production experience with Databricks, including Delta Lake, Unity Catalog permissions, Workflows, and Spark performance tuning.
  • Production experience with dbt, including incremental modeling and testing.
  • Strong understanding of dimensional modeling, medallion architecture, CDC, and incremental data loading.
  • Experience deploying data code through Git-based workflows and CI/CD tools such as GitHub Actions or Databricks Asset Bundles.
  • Working knowledge of AWS data services including S3, IAM, DMS, or Kinesis, and infrastructure-as-code tools such as Terraform.

Technologies

  • Databricks, Delta Lake, Auto Loader, Lakeflow Declarative Pipelines, Lakeflow Connect, Structured Streaming
  • AWS DMS, SAP SLT, AWS Kinesis, AWS S3, AWS IAM, Terraform
  • dbt, MERGE, upsert, Slowly Changing Dimension (SCD) Type 2
  • PySpark, Databricks SQL, OPTIMIZE, VACUUM
  • Lakehouse Monitoring, Databricks Workflows, Airflow, Redshift, Unity Catalog
  • Databricks Asset Bundles, GitHub Actions, SQL warehouses, service principals, OAuth authentication
  • Git-based workflows, SAP MES, SAP HANA, SFC

Contract Details

  • Job type: contract
  • Duration: full-time, 12-month contract
  • Pay: USD 68 per hour
  • Onsite location: Foster City, CA (local candidates)
  • Staffing: staffed via HireArt

Benefits

  • Pre-tax commuter benefits
  • Employer (HireArt) subsidized healthcare benefits (eligibility begins on the first of the month following 60 days of service)
  • Flexible Spending Account for healthcare-related costs
  • HireArt covers all costs for short- and long-term disability and life insurance
  • 401k package

Preferred Qualifications

  • Databricks Certified Data Engineer Associate or Professional certification; AWS certification such as Solutions Architect Associate is a plus
  • Experience preparing data for Databricks AI/BI dashboards, Genie, machine learning, or large language model applications
  • Experience with manufacturing or industrial data, including quality, yield, traceability, or manufacturing execution systems (MES)
  • Understanding of data governance and compliance requirements related to data pipelines

Similar Jobs