EngineerJobs.io
← Back to all jobs

Job Description

The Senior Data Engineer will build and maintain ETL pipelines in an Azure and Databricks environment, combining hands-on development in SQL and Python with a strong ability to understand and interpret data.

Role Overview

In this role, you will create pipelines daily using Azure and Databricks, with development focused on SQL and Python. Work includes implementing data ingestion within Databricks and applying Spark techniques, including Spark declarative pipelines.

Responsibilities

  • Build ETL pipelines daily in an Azure and Databricks environment using SQL and Python.
  • Design projects and implement them hands-on as part of the pipeline build process.
  • Write Python code for pipelines within Databricks.
  • Develop Spark declarative pipelines, including the ability to write Spark declarative pipeline components.
  • Ingest data within Databricks.
  • Demonstrate strong data understanding by reviewing a dataset and explaining what story it is telling.
  • Develop and apply basic analytical skills while learning and picking up new topics quickly.

Requirements

  • Hands-on experience building pipelines, designing projects, and implementing solutions.
  • Good understanding of data, including the ability to review a dataset and interpret the narrative it communicates.
  • Ability to learn and pick up new concepts quickly, supported by basic analytical skills.

Technologies

  • Azure
  • Databricks
  • SQL
  • Python
  • Unity Catalogue
  • Spark

Location and Work Schedule

Short Hills, NJ / Charlotte, NC / Hartford, CT (Hybrid)

Location supports a hybrid schedule with 2 days hybrid.

Compensation

USD 50 - 60 per hour

Benefits

  • Flexible work from home options available.

Similar Jobs