Senior Data Engineer
Python
Senior
Big Data
Bigdata
Cloud Operations
Data
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Integration
Data Integration Tools
Data Pipeline
Data Platform
Data Processing
Database
Databases
ETL
Gcp Cloud
Informatica
Information Technology (IT)
Integration
Programming
Software Development
Spark
SQL
Job Description
Own and scale Gatik’s data platform by transforming raw fleet and simulation logs into training-ready, searchable, versioned datasets for driverless operations.
Responsibilities
- Design, build, and run production-grade ETL/ELT pipelines to ingest, decode, transform, and store multi-sensor logs from autonomous trucks and simulation environments.
- Create internal tooling and services for data labeling, curation, indexing, and cataloging to help engineers locate the right drives, events, and scenarios quickly.
- Collaborate with ML researchers, autonomy engineers, and data scientists on schemas and APIs for model training, evaluation, and debugging.
- Develop and maintain feature stores, metadata systems, and dataset versioning for structured datasets (events, metrics, vehicle signals) and unstructured datasets (camera, lidar, radar).
- Integrate synthetic and simulated data with real-world logs to support hybrid training, scenario replay, and simulation workflows.
- Build scenario-mining and event-tagging pipelines that surface rare, safety-relevant situations, including unprotected turns, dock approaches, and adverse weather.
- Improve pipeline cost, latency, and traceability, ensuring full lineage from raw log to trained model and evaluation output.
- Partner with simulation, cloud platform, and safety teams to automate closed-loop testing, regression analysis, and fleet performance analytics.
- Define data quality and observability standards, and mentor engineers through design reviews and code reviews.
Requirements
- Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field, or equivalent practical experience.
- 8+ years building data-intensive software systems, ideally in autonomous driving, robotics, or large-scale ML settings.
- Strong proficiency in Python and SQL, with working familiarity with C++.
- Hands-on experience designing ETL/ELT pipelines using modern frameworks such as Apache Spark, Flyte, Union, Ray, or Airflow.
- Knowledge of cloud-native architectures on AWS (S3, EC2, EKS, Glue, Athena) or Google Cloud equivalents.
- Familiarity with sensor data (camera, lidar, radar, GPS/IMU) and serialization formats such as Protobuf, ROS/ROS 2 bags, and MCAP.
- Deep understanding of data quality, observability, and lineage in high-volume systems.
- Proven ability to build reliable, performant infrastructure for both ad-hoc exploration and repeatable production workflows.
Technologies
- Python, SQL, C++, Apache Spark, Flyte, Union, Ray, Airflow
- AWS: S3, EC2, EKS, Glue, Athena; Google Cloud
- Protobuf, ROS/ROS 2 bags, MCAP
Nice to Have
- Experience in AD/ADAS, autonomous trucking, or robotics, especially with perception, prediction, or planning datasets.
- Familiarity with ML orchestration platforms such as Kubeflow, SageMaker, or Vertex AI.
- Experience with temporal and spatial data, including geospatial indexing (H3, S2, PostGIS), HD maps, and multi-sensor time-series alignment.
- Exposure to synthetic data generation, simulation logging, or scenario replay pipelines.
- Experience with lakehouse and table formats such as Delta Lake, Apache Iceberg, or Parquet at petabyte scale.
- Familiarity with safety-critical development practices and data traceability for safety cases or regulatory requirements.
- Strong software engineering fundamentals: CI/CD, testing, code review, and service deployment best practices.
- Experience working with cross-functional, distributed teams across research and production organizations.
Location
- Santa Clara, CA (onsite)
Salary
- USD 180,000 - 260,000 per year
E-VERIFY NOTICE: By submitting your application, you acknowledge that Gatik participates in the U.S. Department of Homeland Security E-Verify program.