Data Engineer - Fleet Operations
Apache Spark
Aws Dms
Big Data
Bigdata
Change Data Capture
Cloud Data Engineering
Cloud Data Platform
Cloud Platform
Cloud Platforms
Data
Data Analysis
Data Analytics
Data Analytics Tools
Data Architecture
Data Build Tool
Data Engineer
Data Engineering
Data Integration
Data Integration Tools
Data Lake
Data Lakehouse
Data Management
Data Modeling
Data Operations
Data Pipeline
Data Pipelines
Data Platform
Data Processing
Data Transforms
Data Warehouse
Database
Databases
Databricks
Databricks Lakehouse
Databricks Lakehouse Platform
Dbt
Delta Lake
ETL
Informatica
Integration
Lakehouse
Spark
SQL
Streaming Data
Structured Streaming
Job Description
An Autonomous Mobility Company is seeking a Data Engineer for Fleet Operations to design, build, and maintain data pipelines and models that power a Manufacturing Intelligence data lakehouse. This role focuses on integrating manufacturing data into the MOPS lakehouse and enabling reporting, dashboards, and AI-related applications using Databricks and AWS ingestion infrastructure.
Based in Foster City, CA, this onsite contract role supports production-grade data engineering across Delta Lake, dbt, and orchestration workflows.
Role Focus
- Build and maintain batch and streaming ingestion pipelines into Delta Lake using Auto Loader, Lakeflow Declarative Pipelines, Lakeflow Connect, or Structured Streaming, including CDC feeds from AWS DMS and SAP SLT.
- Design and maintain bronze, silver, and gold data layers using MOPS naming conventions and data dictionary standards.
- Develop incremental dbt models on Databricks using MERGE, upsert, and SCD Type 2 patterns, including late or out-of-order record handling.
- Model manufacturing data such as unit-level traceability (SFC), work orders, nonconformances, station results, and first-pass yield to support analytics, dashboards, and semantic reporting.
- Process large-scale and semi-structured datasets with PySpark and Databricks SQL, optimizing performance and cost using clustering, partitioning, OPTIMIZE, VACUUM, file sizing, and compute configuration.
- Maintain data quality and reliability using dbt tests, Lakeflow expectations, Lakehouse Monitoring, freshness SLAs, and automated alerts.
- Orchestrate workloads with Databricks Workflows or Airflow, and support Redshift-to-Databricks migration through SQL conversion, parallel validation, reconciliation, and cutover.
- Collaborate with analysts and manufacturing teams to define data requirements and document lineage, data contracts, and table definitions in Unity Catalog.
- Build and maintain CI/CD pipelines using Databricks Asset Bundles and GitHub Actions across development, staging, and production.
- Manage MOPS-specific Databricks configurations including Unity Catalog permissions, compute resources, SQL warehouses, service principals, secrets, and OAuth authentication, aligned with central platform governance.
- Configure and troubleshoot AWS ingestion infrastructure including DMS, Kinesis, S3, and IAM, coordinating Terraform-based provisioning with infrastructure teams.
- Monitor pipeline health and performance, manage Databricks Unit (DBU) costs, maintain operational documentation, and support incident response and root cause analysis.
Requirements
- Bachelor’s degree in computer science, information systems, engineering, or a related discipline (or equivalent practical experience).
- 6+ years of data engineering experience.
- Strong proficiency in SQL, Python, and PySpark.
- Hands-on production experience with Databricks, including Delta Lake, Unity Catalog permissions, Workflows, and Spark performance tuning.
- Production experience with dbt, including incremental modeling and testing.
- Strong understanding of dimensional modeling, medallion architecture, CDC, and incremental data loading.
- Experience deploying data code through Git-based workflows and CI/CD tools such as GitHub Actions or Databricks Asset Bundles.
- Working knowledge of AWS data services including S3, IAM, DMS, or Kinesis, and infrastructure-as-code tools such as Terraform.
Technologies
- Databricks, Delta Lake, Auto Loader, Lakeflow Declarative Pipelines, Lakeflow Connect, Structured Streaming
- AWS DMS, SAP SLT, AWS Kinesis, AWS S3, AWS IAM, Terraform
- dbt, MERGE, upsert, Slowly Changing Dimension (SCD) Type 2
- PySpark, Databricks SQL, OPTIMIZE, VACUUM
- Lakehouse Monitoring, Databricks Workflows, Airflow, Redshift, Unity Catalog
- Databricks Asset Bundles, GitHub Actions, SQL warehouses, service principals, OAuth authentication
- Git-based workflows, SAP MES, SAP HANA, SFC
Contract Details
- Job type: contract
- Duration: full-time, 12-month contract
- Pay: USD 68 per hour
- Onsite location: Foster City, CA (local candidates)
- Staffing: staffed via HireArt
Benefits
- Pre-tax commuter benefits
- Employer (HireArt) subsidized healthcare benefits (eligibility begins on the first of the month following 60 days of service)
- Flexible Spending Account for healthcare-related costs
- HireArt covers all costs for short- and long-term disability and life insurance
- 401k package
Preferred Qualifications
- Databricks Certified Data Engineer Associate or Professional certification; AWS certification such as Solutions Architect Associate is a plus
- Experience preparing data for Databricks AI/BI dashboards, Genie, machine learning, or large language model applications
- Experience with manufacturing or industrial data, including quality, yield, traceability, or manufacturing execution systems (MES)
- Understanding of data governance and compliance requirements related to data pipelines
Similar Jobs
G