Senior Data Engineer
Senior
Azure
Azure Data Factory
Azure Databricks
Big Data
Bigdata
Cloud
Cloud Platforms
Data Architecture
Data Engineer
Data Governance
Data Integration
Data Lake
Data Lakehouse
Data Pipeline
Data Platform
Data Processing
Data Security
Database
Databricks
Databricks Workflows
Delta Lake
Engineering
ETL
Event Streaming
Microsoft Azure
Spark
SQL
Job Description
Deloitte is seeking a Senior Data Engineer to work onsite in Jersey City, NJ, responsible for building and refining end-to-end data pipelines in a dynamic consulting setting. You will design, develop, and optimize data workflows using Azure Data Factory, Databricks, and PySpark, collaborating with Engagement Managers, project teams, and client architects within a project delivery framework. The position carries a salary range of USD 95,000 to 150,000 per year and requires a Bachelor's degree.
Responsibilities
- Regularly communicate with Engagement Managers (Directors), project teams, and representatives from various functional and technical groups, escalating issues as needed.
- Design, develop and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
- Write and tune PySpark and Spark SQL notebooks for large-scale data transformations.
- Architect end-to-end data solutions across development, UAT, and production environments using Unity Catalog.
- Lead design discussions with client architects and other counterparts.
- Collaborate with different teams on data contracts and schema governance.
- Lead the design and optimization of high-volume data pipelines.
- Define and enforce data engineering standards including naming conventions, partitioning strategies, cluster configurations, and Spark tuning.
- Drive performance optimization through AQE tuning, liquid clustering, broadcast joins, and shuffle partition management.
- Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies.
- Conduct root cause analysis on production incidents and implement durable fixes.
- Mentor junior and mid-level engineers through code reviews and pair programming.
- Evaluate new technologies and recommend adoption, such as DABs, DLT, Auto Loader, Serverless Compute, and event hubs.
Requirements
- Proficiency in Python, PySpark, Spark SQL, and SQL Server.
- Experience with Azure services including Azure Data Factory, ADLS Gen2, Key Vault, and Azure Monitor.
- Databricks knowledge (Delta Lake, Unity Catalog, Workflows).
- Apache Airflow experience.
- Git or Azure DevOps experience.
- Deep understanding of Spark internals (DAG optimization, spill analysis, skew handling).
- Delta Lake advanced features (time travel, deletion vectors, predictive I/O).
- Unity Catalog governance (row/column security, external locations, system tables).
- Infrastructure as code using Terraform and Azure ARM templates.
- Bachelor's degree or equivalent experience; preferably in Computer Science, Information Technology, Computer Engineering, or a related IT discipline.
- Limited immigration sponsorship may be available.
- Ability to travel 10% on average, depending on client engagements.
Technologies
- Python, PySpark, Spark SQL, SQL Server
- Azure Data Factory, ADLS Gen2, Key Vault, Azure Monitor
- Databricks, Delta Lake, Unity Catalog, Workflows
- Apache Airflow
- Git, Azure DevOps
- Deep Spark internals, DAG optimization, spill analysis, skew handling
- Delta Lake time travel, deletion vectors, predictive I/O
- Unity Catalog governance (row/column security, external locations)
- Terraform, Azure ARM templates
- DABs, DLT, Auto Loader, Serverless Compute, event hubs
Additional Requirements
Accommodation information for applicants who need assistance is available at the following Deloitte page: Deloitte assistance for disabled applicants.
Similar Jobs