Senior Data Engineer
Job Description
The Senior Data Engineer will build and maintain ETL pipelines in an Azure and Databricks environment, combining hands-on development in SQL and Python with a strong ability to understand and interpret data.
Role Overview
In this role, you will create pipelines daily using Azure and Databricks, with development focused on SQL and Python. Work includes implementing data ingestion within Databricks and applying Spark techniques, including Spark declarative pipelines.
Responsibilities
- Build ETL pipelines daily in an Azure and Databricks environment using SQL and Python.
- Design projects and implement them hands-on as part of the pipeline build process.
- Write Python code for pipelines within Databricks.
- Develop Spark declarative pipelines, including the ability to write Spark declarative pipeline components.
- Ingest data within Databricks.
- Demonstrate strong data understanding by reviewing a dataset and explaining what story it is telling.
- Develop and apply basic analytical skills while learning and picking up new topics quickly.
Requirements
- Hands-on experience building pipelines, designing projects, and implementing solutions.
- Good understanding of data, including the ability to review a dataset and interpret the narrative it communicates.
- Ability to learn and pick up new concepts quickly, supported by basic analytical skills.
Technologies
- Azure
- Databricks
- SQL
- Python
- Unity Catalogue
- Spark
Location and Work Schedule
Short Hills, NJ / Charlotte, NC / Hartford, CT (Hybrid)
Location supports a hybrid schedule with 2 days hybrid.
Compensation
USD 50 - 60 per hour
Benefits
- Flexible work from home options available.