Databricks Data Engineer
Azure Data Engineer
Big Data
Bigdata
Cdc
Change Data Capture
Cloud Data Engineering
Cloud Data Platform
Cloud Platform
Cloud Platforms
Data
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Engineering Lead
Data Governance
Data Integration
Data Integration Tools
Data Lakehouse
Data Management
Data Pipeline
Data Pipelines
Data Platform
Data Processing
Data Security
Data Warehouse
Database
Databases
Databricks
Databricks Spark
Databricks Workflows
ETL
Event Driven Architecture
Informatica
Kafka
Lead Data Engineering
Pyspark
Scd Type 2
Spark
SQL
Streaming Data
Job Description
Robotics Technologies is seeking a Databricks Data Engineer to support the Retail Spine initiative in Phoenix, AZ. This onsite role focuses on building and optimizing scalable ETL/ELT data solutions in a fast-paced Azure environment, with hands-on work across Databricks, PySpark, Delta Lake, and Kafka streaming.
Role Overview
In this position, you will design, build, and improve end-to-end data engineering pipelines that support incremental processing and real-time or near real-time needs. The work includes new pipeline development as well as enhancements to existing workflows, including understanding and iterating on current codebases to deliver improvements efficiently.
Key Responsibilities
- Design, build, and optimize end-to-end ETL/ELT pipelines using Databricks and Apache Spark (PySpark).
- Develop and maintain scalable data solutions on Microsoft Azure leveraging Databricks, Delta Lake, and Kafka streaming.
- Build robust pipelines with Change Data Capture (CDC) to enable incremental data processing and real-time or near real-time data use cases.
- Support both new pipeline creation and enhancement of existing pipelines, including quickly understanding current codebases to deliver targeted improvements.
- Integrate and process data from diverse sources, including Flat files, XML, JSON, SQL Server and JDBC systems, Kafka topics, APIs, MQ, and Unity Catalog volumes and other enterprise platforms.
- Implement data quality frameworks, including writing technical data quality rules and quarantining error records.
- Apply canonical data modeling techniques, including SCD Type 1 and SCD Type 2 implementations.
- Ensure high levels of data reliability, performance, scalability, and monitoring across all workflows.
- Automate workflows using Databricks Workflows and CI/CD pipelines with GitHub, emphasizing reuse, efficiency, and standardization.
- Collaborate with developers, architects, and product owners, engaging with stakeholders, source teams, and downstream consumers as needed.
- Participate in Agile/Scrum ceremonies and deliver high-quality solutions within defined timelines.
Required Qualifications
- Extensive hands-on experience with Databricks, with the ability to contribute from day one.
- Proven experience building end-to-end data engineering pipelines with CDC capabilities.
- PySpark and SQL proficiency (required); Python experience is a plus.
- Experience working in Microsoft Azure cloud environments.
Technologies
- Databricks, Apache Spark, PySpark, Microsoft Azure
- Delta Lake, Kafka, Change Data Capture (CDC)
- SQL Server, JDBC, Unity Catalog
- Databricks Workflows, CI/CD, GitHub
- SCD Type 1, SCD Type 2, Agile/Scrum
- XML, JSON, Flat files, APIs, MQ, SQL
Location, Duration, and Employment Details
- Location: Phoenix, AZ (onsite)
- Duration: 9 Months + Extension (9 Months+ Extension)
- Employment Type: W-2, 1099, C2C
- Domain Exposure: Consumer Staples
- Client: To Be Discussed Later
Compensation and Work Authorization
- Salary: USD 71 - 71 per hourly
- Bill Rate: $71/hour (C2C)
- Work Authorization: US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other; USC, GC, H-1B, OPT-EAD, GC-EAD, H4-EAD
Job ID
J55801