Senior Data Engineer

Deloitte is seeking a Senior Data Engineer to work onsite in Jersey City, NJ, responsible for building and refining end-to-end data pipelines in a dynamic consulting setting. You will design, develop, and optimize data workflows using Azure Data Factory, Databricks, and PySpark, collaborating with Engagement Managers, project teams, and client architects within a project delivery framework. The position carries a salary range of USD 95,000 to 150,000 per year and requires a Bachelor's degree.

Responsibilities

• Regularly communicate with Engagement Managers (Directors), project teams, and representatives from various functional and technical groups, escalating issues as needed.
• Design, develop and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
• Write and tune PySpark and Spark SQL notebooks for large-scale data transformations.
• Architect end-to-end data solutions across development, UAT, and production environments using Unity Catalog.
• Lead design discussions with client architects and other counterparts.
• Collaborate with different teams on data contracts and schema governance.
• Lead the design and optimization of high-volume data pipelines.
• Define and enforce data engineering standards including naming conventions, partitioning strategies, cluster configurations, and Spark tuning.
• Drive performance optimization through AQE tuning, liquid clustering, broadcast joins, and shuffle partition management.
• Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies.
• Conduct root cause analysis on production incidents and implement durable fixes.
• Mentor junior and mid-level engineers through code reviews and pair programming.
• Evaluate new technologies and recommend adoption, such as DABs, DLT, Auto Loader, Serverless Compute, and event hubs.

Requirements

• Proficiency in Python, PySpark, Spark SQL, and SQL Server.
• Experience with Azure services including Azure Data Factory, ADLS Gen2, Key Vault, and Azure Monitor.
• Databricks knowledge (Delta Lake, Unity Catalog, Workflows).
• Apache Airflow experience.
• Git or Azure DevOps experience.
• Deep understanding of Spark internals (DAG optimization, spill analysis, skew handling).
• Delta Lake advanced features (time travel, deletion vectors, predictive I/O).
• Unity Catalog governance (row/column security, external locations, system tables).
• Infrastructure as code using Terraform and Azure ARM templates.
• Bachelor's degree or equivalent experience; preferably in Computer Science, Information Technology, Computer Engineering, or a related IT discipline.
• Limited immigration sponsorship may be available.
• Ability to travel 10% on average, depending on client engagements.

Technologies

• Python, PySpark, Spark SQL, SQL Server
• Azure Data Factory, ADLS Gen2, Key Vault, Azure Monitor
• Databricks, Delta Lake, Unity Catalog, Workflows
• Apache Airflow
• Git, Azure DevOps
• Deep Spark internals, DAG optimization, spill analysis, skew handling
• Delta Lake time travel, deletion vectors, predictive I/O
• Unity Catalog governance (row/column security, external locations)
• Terraform, Azure ARM templates
• DABs, DLT, Auto Loader, Serverless Compute, event hubs

Additional Requirements

Accommodation information for applicants who need assistance is available at the following Deloitte page: Deloitte assistance for disabled applicants.