Sr. Data Engineer
Job Description
Arlo Technologies, Inc. is seeking a Senior Staff Data Engineer to design and scale its data platform. This onsite role in Milpitas, CA focuses on ownership of data architecture, pipelines, and data governance on Databricks, with close partnership across product, analytics, and data science to deliver trusted and secure data products.
You will help evolve how Arlo builds production-grade data workflows, from medallion-based storage patterns for both batch and streaming, to CI/CD automation, quality monitoring, and governance controls through Unity Catalog. The role also includes mentoring engineers and contributing to the data roadmap.
Responsibilities
- Define and evolve data architecture on the Databricks Lakehouse using medallion (Bronze/Silver/Gold) patterns for batch and streaming workloads.
- Design, build, and maintain scalable ETL/ELT pipelines using Python, PySpark, and SQL.
- Implement and manage Unity Catalog, including catalogs, schemas, access controls, data lineage, auditing, and secure data sharing.
- Define and enforce data governance practices across data quality, metadata management, data classification, retention, privacy, and regulatory compliance.
- Build and maintain CI/CD pipelines for data workflows using Git, GitHub Actions / Azure DevOps / Jenkins, and Databricks Asset Bundles, including automated testing and deployment.
- Implement data quality checks along with monitoring and alerting to support reliable, trusted data.
- Monitor and optimize Databricks jobs for performance, reliability, and cost.
- Mentor engineers, lead design reviews, and contribute to the data roadmap.
Requirements
- 5–6 years of experience in data engineering, including production data pipelines and architectures.
- Strong hands-on expertise in Databricks (Spark, Delta Lake, Workflows).
- Hands-on experience with Unity Catalog (access control, lineage, catalog and schema design).
- Solid understanding of data governance concepts: data quality, cataloguing, lineage, privacy, and compliance (e.g., GDPR, CCPA).
- Advanced Python (PySpark) and SQL skills.
- Solid experience with CI/CD and DataOps practices.
- Experience with data modelling and a major cloud platform (AWS, Azure, or GCP).
- Strong communication and cross-functional collaboration skills.
- Bachelor’s degree in Computer Science, Engineering, or a related field.
Technologies
- Databricks, Spark, Delta Lake, Workflows, Unity Catalog
- Python, PySpark, SQL
- Git, GitHub Actions, Azure DevOps, Jenkins, Databricks Asset Bundles
- AWS, Azure, GCP
Compensation
Salary range: USD 140,000 - 160,000 per year.
The total compensation package may include other elements, including bonus, equity, and a full range of benefits.