Azure Event Hubs

Azure Event Hubs Engineer Jobs

2 jobs found

14d ago

Senior Data Engineer

firstPROPhiladelphia Philadelphia, PA

Philadelphia, PA

14d ago

$60 - $70/hr Contract Senior Analytics Automation

Apply Now

Job Description

firstPROPhiladelphia offers a hybrid contract-to-hire Senior Data Engineer role in Philadelphia. You will focus on real-time streaming data pipelines for connected medical devices, leveraging Azure Databricks, Azure services, and ML/AI data workflows. The position is commissioned at USD 60 to 70 per hour, with a path toward permanent placement through a contract-to-hire arrangement and opportunities to collaborate across data, software, and analytics teams.

Responsibilities

Design, build, and operate batch and streaming data pipelines on Azure and Databricks, with a primary focus on real-time IoT and telemetry within a Medallion architecture.
Develop ETL and ELT workflows to ingest, transform, and validate large volumes of structured and unstructured data.
Build and maintain data services, APIs, and microservices for application, analytics, and ML/AI teams.
Implement real-time streaming solutions using Azure Event Hubs, Azure Stream Analytics, and related Azure integration patterns, prioritizing cost-effective throughput, thoughtful partitioning, and reliable downstream delivery to Databricks.
Optimize production Databricks pipelines with PySpark, Spark SQL, and Delta Lake, including performance tuning, reliability improvements, and cost awareness.
Troubleshoot and resolve complex pipeline issues across Databricks, Azure, and on premise systems, conducting root cause analysis and corrective actions.
Collaborate with data analysts, software engineers, ML engineers, and business stakeholders to translate requirements into technical designs and delivery priorities.
Apply data quality, validation, and privacy-first practices, delivering reliable pipelines through established software engineering standards, documentation, testing, and CI/CD.

Requirements

Bachelor's degree in Computer Science, Mathematics, Engineering, or a related technical field plus 6+ years of professional data engineering, analytics, or warehousing experience; or Master’s degree with 3+ years of relevant experience.
5+ years designing, building, and operating big data and real-time streaming pipelines across cloud and on-prem environments.
5+ years applying DevOps and CI/CD practices to data and analytics workloads.
Production experience building data services, APIs, or microservices for downstream data consumption.
Hands-on expertise designing and delivering production-grade data pipelines on Databricks.
Proven experience with Spark optimization and tuning in Databricks, including performance analysis, partitioning, caching, shuffle optimization, and cost-aware design.
Strong experience with Azure cloud services for data engineering and streaming workloads.
Experience ensuring data quality, validation, and privacy-aware handling for regulated or sensitive data.

Technologies

Databricks, Spark, PySpark, Spark SQL, Delta Lake
Azure, Azure Event Hubs, Azure Stream Analytics
Azure Databricks, Medallion architecture

Outcomes

Onboard to the Azure Databricks environment and contribute to troubleshooting, stabilization, and optimization of existing batch and streaming pipelines.
Stand up Azure streaming ingestion for telemetry data and deliver production-ready pipelines integrated with Databricks to support API, ML, and downstream analytics within the Medallion architecture.
Design and deliver data services or consumption patterns that enable business and ML teams to access near real-time telemetry data reliably, securely, and at scale.

Work hours and travel requirements

The role requires openness to assisting in off-hours production troubleshooting as needed. The IT team operates in a hybrid environment, with a minimum of two days per week in the downtown Philadelphia office.

View full job page →

27d ago

Senior Data Engineer

Deloitte Morristown, NJ

Morristown, NJ

27d ago

$95k - $150k/yr Full time Senior Adls Gen2 Apache Airflow

Apply Now

Job Description

Deloitte is seeking a Senior Data Engineer to shape and optimize comprehensive data platforms, building scalable ETL and ELT pipelines and partnering with engagement teams and client architects to deliver robust solutions. The role is anchored on hands-on development with Azure and Databricks, conducted onsite in Morristown, New Jersey, as part of a client-facing data engineering practice.

Responsibilities

Communicate regularly with Engagement Managers (Directors), project team members, and representatives from multiple functional and technical teams, escalating issues that require leadership attention.
Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
Write and tune PySpark and Spark SQL notebooks for large-scale data transformations.
Architect end-to-end data solutions across dev, UAT, and prod environments using Unity Catalog.
Lead design discussions with client architects and counterparts to shape data solutions.
Collaborate with teams on data contracts and schema agreements.
Lead the design and optimization of high-volume data pipelines.
Define and enforce data engineering standards including naming conventions, partitioning strategies, cluster configurations, and Spark tuning.
Drive performance optimization through AQE tuning, liquid clustering, broadcast joins, and shuffle partition management.
Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies.
Conduct root cause analysis on production incidents and implement permanent fixes.
Mentor junior and mid-level engineers through code reviews and pair programming.
Evaluate new technologies and recommend adoption (examples include DABs, DLT, Auto Loader, Serverless Compute, and Azure Event Hubs).

Requirements

Python, PySpark, Spark SQL, SQL Server
Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor)
Databricks (Delta Lake, Unity Catalog, Workflows)
Apache Airflow
Git / Azure DevOps
Deep Spark internals (DAG optimization, spill analysis, skew handling)
Delta Lake advanced features (time travel, deletion vectors, predictive I/O)
Unity Catalog governance (row/column security, external locations, system tables)
IaC - Terraform, Azure ARM templates
Bachelor's degree, preferably in Computer Science, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience
Limited immigration sponsorship may be available
Ability to travel 10% on average, depending on client needs

Technologies

Python
PySpark
Spark SQL
SQL Server
Azure Data Factory (ADF)
ADLS Gen2
Key Vault
Azure Monitor
Databricks
Delta Lake
Unity Catalog
Workflows
Apache Airflow
Git
Azure DevOps
DABs
DLT
Auto Loader
Serverless Compute
Azure Event Hubs
Terraform
Azure ARM templates

Location

Morristown, NJ (onsite)

Compensation

USD 95,000 - 150,000 per year

Education

Bachelor's degree, preferably in Computer Science, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience

Accommodations

Information for applicants with a need for accommodation: accommodation information page

View full job page →

Azure Event Hubs Engineer Jobs

Senior Data Engineer

Senior Data Engineer

Get Azure Event Hubs Engineer Jobs alerts