Data Factory

Data Factory Engineer Jobs

65 jobs found

Today

Azure Data Engineer

Logic Arlington, VA

Arlington, VA

Today

$38 - $48/hr Full time Backend Developer Azure Azure Data Engineer

Apply Now

Job Description

Logic is seeking an Azure Data Engineer to support the design, development, and ongoing maintenance of data solutions for data generation, collection, and processing. This onsite role in Arlington, VA focuses on building reliable data pipelines, implementing ETL for migrating and deploying data across systems, and owning outcomes as a subject matter expert while working closely with the broader team.

You will contribute to problem-solving both within your immediate team and across multiple teams, provide technical guidance and mentorship to junior engineers, and help drive key team decisions through collaboration and SME-level expertise. The position also emphasizes continual improvement by staying current with industry trends and technologies.

Responsibilities

Design, develop, and maintain data solutions for data generation, collection, and processing.
Create and maintain data pipelines.
Ensure data quality through cross-functional collaboration to identify and address data quality issues.
Implement ETL (extract, transform, and load) processes to migrate and deploy data across systems.
Develop innovative data solutions to optimize data generation, collection, and processing.
Implement advanced ETL processes to support efficient data migration and deployment.
Provide solutions to problems for your immediate team and across multiple teams.
Stay updated with the latest industry trends and technologies to continuously improve data solutions.
Provide technical guidance and mentorship to junior team members.
Be responsible for team decisions and engage with multiple teams to contribute to key decisions.
Collaborate with and manage the team to perform expected work as an SME.

Requirements

Minimum 3+ years of work experience with one or more of the following: Databricks, Pyspark, Azure Data Factory, Azure Synapse, Delta live tables, Azure Function Apps.

Technologies

Databricks
Pyspark
Azure Data Factory
Azure Synapse
Delta live tables
Azure Function Apps

Benefits

Medical
Dental
Vision
Long-term disability coverage
401(k) plan
Paid time off
Access to benefits programs
World-class learning resources

Location: Arlington, VA (onsite)
Compensation: USD 38 - 48 per hour

View full job page →

1d ago

Data Engineer

Wize Solutions LLC Abingdon, VA

Abingdon, VA

1d ago

$70k - $90k/yr Full time Backend Developer Analytics Automation Developer

Apply Now

Job Description

This backend-focused Data Engineer role at Wize Solutions LLC supports a commercial client engaged in federal work. The position is based onsite in Abingdon, VA 24210 and focuses on building and operating SQL Server ETL pipelines and data warehouse components that power reporting workflows.

Key Responsibilities

Develop and tune SQL Server databases, including stored procedures, views, and indexes
Build and maintain ETL pipelines to move data from source systems into the data warehouse
Support and extend data warehouse models, including fact and dimension tables
Create and manage SQL Server Agent jobs, including schedules, alerts, and failure handling
Design backend data to enable dependable, high-performance Power BI dataset refresh (semantic model support)
Partner with report developers to troubleshoot refresh failures, slow queries, and data issues traced to SQL
Apply data masking to protect sensitive data in both production and non-production environments
Monitor and resolve performance issues across queries, jobs, and data loads
Document data flows, job schedules, and data models

Required Qualifications

Ability to obtain and maintain a federal security clearance (U.S. citizenship required)
3 to 5 years of hands-on SQL Server development experience
Strong T-SQL skills, including query tuning and execution plan work
Experience building ETL processes using SSIS or similar tools
Working knowledge of data warehousing and dimensional modeling
Experience creating and managing SQL Server Agent jobs
Understanding of how Power BI datasets connect to SQL Server, including Import vs. DirectQuery, refresh behavior, and gateway considerations
Experience with data masking and handling sensitive data (PII)
Clear communication with technical and non-technical stakeholders

Technologies

SQL Server, T-SQL, SQL Server Agent, ETL pipelines, SSIS, Power BI, Azure Data Factory, Azure SQL, Microsoft Fabric, Power Platform, RPA, C#/.NET

Clearance Requirements

Must be able to obtain and maintain a federal security clearance.

Preferred Qualifications

Active federal security clearance
Experience with Azure Data Factory, Azure SQL, or Microsoft Fabric
Experience on federal contracts or programs
Microsoft certifications such as DP-203, DP-600, or similar

Role Details

The work involves constructing and maintaining the backend that supports reporting. Building Power BI reports is not part of the role. However, the position requires understanding how Power BI dataset refreshes and dependencies rely on the SQL Server backend. In addition, the role supports protection of sensitive data through masking and sound security practices.

Location and Compensation

Location: Abingdon, VA 24210 (onsite)
Salary: USD 70,000 to 90,000 per year

View full job page →

2d ago

Data Engineer

Morgan, Lewis & Bockius Philadelphia, PA

Philadelphia, PA

2d ago

Full time Analytics Azure Azure Data Engineer

Apply Now

Job Description

As a Data Engineer at Morgan, Lewis & Bockius, you will help build and improve the firm’s data integration and data warehousing capabilities. This role reports to the Data Warehouse Manager and supports production systems through development, modernization, troubleshooting, and performance-focused enhancements. The position is hybrid based in Philadelphia, PA, with a mix of in-office and remote work.

What you will do

Contribute to the firm’s SQL Server/SSIS-based ETL processes and cloud data platform initiatives, including Azure and Microsoft Fabric.
Develop, support, and modernize data integration and warehousing solutions using SQL Server, SSIS, Azure SQL, Azure Data Factory (ADF), Microsoft Fabric, and Fabric Data Factory/Data Flows (FDF).
Support data warehouse and OLTP environments across data ingestion, transformation, dimensional modeling, data quality, performance optimization, and integration.
Collaborate with Business Analysts, Data Architects, Developers, DBAs, and other IT teams to define and implement data solutions.
Analyze data sources and interfaces, including dependencies, and produce technical documentation such as data flow diagrams, ERDs, source-to-target mappings, and data lineage.
Design and support integration approaches that account for transaction volumes, data availability and timeliness, regulatory and retention needs, security, privacy, and system dependencies.
Troubleshoot, diagnose, and resolve production data, database, and integration issues in coordination with other IT teams as needed.
Develop and maintain SSIS packages, Azure Data Factory pipelines, and Fabric Data Factory pipelines, along with SQL Server objects.

Qualifications

Bachelor’s degree from a four-year college or university and minimum 5 years of experience, including 3 years of SQL development.
Hands-on experience with Azure Data Factory (ADF) and Fabric Data Factory (FDF), including pipeline development, data integration, transformation, orchestration, and integration with on-premises and cloud data sources, as well as Lakehouses for FDF.
Working knowledge of Microsoft Fabric and modern cloud data warehouse architectures, including Fabric Data Warehouse, Lakehouse, OneLake, and analytics services.
Experience designing and implementing cloud-based ETL/ELT solutions using Microsoft Azure and Fabric.
Experience with Online Transaction Processing (OLTP) using Microsoft SQL Server.
Understanding of data warehouse design, including Data Marts, Multidimensional database design, Fact Tables, Star and Snow-flake, and Schemas.
Hands-on experience in SQL and stored procedure tuning, including database reorganization.
Strong verbal and written communication skills.
Self-starter who applies strategic thinking and problem solving.
Computer skills including MS Office (MS Word, PowerPoint, Excel).

Technologies

SQL Server, SSIS
Azure, Azure SQL, Azure Data Factory (ADF)
Microsoft Fabric, Fabric Data Factory (FDF), Fabric Data Warehouse
Lakehouse, OneLake, analytics services
OLTP
SQL, stored procedures
Multidimensional database design, Fact Tables, Star and Snow-flake, Schemas
MS Office (MS Word, PowerPoint, Excel)

Certifications (recommended, not required)

Microsoft Certified Solutions Expert (MCSE): Data Management and Analytics
DP-700 – Fabric Data Engineer Associate
DP-600 – Fabric Analytics Engineer Associate

For California applicants, pursuant to the California Consumer Privacy Act, the firm’s California Consumer Privacy Act Privacy Notice for Candidates explains the categories of personal information collected and the purposes for which it is used.

View full job page →

3d ago

Senior Data Engineer

Optum Eden Prairie, MN 🇺🇸 Remote

Eden Prairie, MN

3d ago

$92k - $164k/yr Full time Senior Apache Airflow Api Integration

Apply Now

Job Description

Optum is hiring a Senior Data Engineer to design and run governed data platforms and pipelines for healthcare data solutions.

Responsibilities

Lead the design and delivery of scalable, governed data engineering solutions, including technical direction, implementation planning, risk management, and production readiness for data products used by business, analytics, and AI/ML stakeholders
Develop and manage ETL/ELT pipelines to ingest, transform, and integrate data from heterogeneous sources (relational databases, APIs, file shares, and end-user-computing assets)
Create Python and PySpark solutions for distributed processing, automation, integration, and reusable transformation logic
Improve data platform performance and manage cloud consumption using SQL tuning, Spark job optimization, storage format selection, partitioning, cluster configuration, and workload scheduling
Build and optimize workflows on Azure Cloud services and Databricks, including medallion (bronze/silver/gold) layering and orchestrated pipelines
Migrate business logic from legacy and end-user-computing systems into documented, testable transformations over governed tables
Define and implement data quality practices, including validation, reconciliation, and parity testing to ensure migrated pipelines are accurate, reliable, auditable, and consistent
Establish data lineage, ownership, and cataloguing, including documented rollup and allocation rules when sources operate at different grains
Implement security and compliance controls for PHI/PII and confidential business data in accordance with HIPAA or similar regulations, including de-identification and masking for controlled sharing
Build and improve CI/CD, version control, and automated testing for data pipelines
Implement production monitoring, alerting, logging, and operational runbooks for reliability, timely issue detection, incident response, and continuous improvement
Support AI/ML and advanced analytics by provisioning trusted data, scalable infrastructure, and governed consumption patterns for data scientists, analysts, and partners
Design and publish reusable data products, data services, APIs, and consumption layers so downstream applications and analytics/AI-ML use cases consume trusted data consistently
Partner with product owners, business stakeholders, architects, data scientists, analysts, and engineering teams to translate requirements into scalable solutions and communicate technical tradeoffs
Design, develop, and deploy AI-enabled data solutions for complex business challenges and responsible AI use, leveraging AI-powered development tools to improve productivity and delivery quality
Mentor data engineers and analysts through design reviews, code reviews, and best-practice guidance

Requirements

6+ years overall experience in data engineering, data warehousing, distributed data processing, or enterprise data platform delivery
6+ years of experience with SQL, data modeling, and performance tuning for large-scale systems
3+ years with ETL/ELT frameworks, pipeline orchestration, and heterogeneous source-system integration
3+ years with Python for data processing, automation, and integration
3+ years with Databricks, Spark/PySpark, and Delta Lake
3+ years with public cloud platforms such as Azure or GCP
1+ year integrating data from heterogeneous source systems
1+ year in data quality, validation, reconciliation, observability, and production support for large-scale data pipelines
1+ year with Git, CI/CD platforms, automated testing frameworks, release management, and deployment practices for production data pipelines
1+ year with AI-powered development tools used responsibly to improve engineering productivity and delivery quality

Technologies

Python, PySpark, SQL, Databricks, Spark, Delta Lake
Azure, GCP
ETL, ELT
Git, CI/CD
HIPAA
Azure Data Factory, Airflow, dbt
Unity Catalog, Microsoft Purview
APIs

Benefits

Comprehensive benefits package
Incentive and recognition programs
Equity stock purchase
401k contribution

Work Model

Remote within the U.S.
For hires in the Minneapolis or Washington, D.C. area: required to work in the office a minimum of four days per week

Preferred Qualifications

Experience with batch and near-real-time patterns, including event-driven ingestion, streaming pipelines, or comparable technologies
Experience with Azure Data Factory or comparable orchestration tooling such as Airflow, dbt, or similar
Experience migrating legacy or end-user-computing (spreadsheet- or macro-based) business logic into governed data pipelines
Experience with data governance, lineage, and catalog tooling (Unity Catalog, Microsoft Purview, or similar)
Familiarity with AI/ML frameworks and supporting infrastructure for data science workflows
Experience working with PHI/PII and restricted client data in compliance with HIPAA or similar regulations, including de-identification and masking
Healthcare payer, actuarial, pricing, or financial-modelling domain experience

Pay

$91,700 - $163,700 per year, based on full-time employment and several factors

Application Details

Posted for a minimum of 2 business days or until a sufficient candidate pool has been collected
Posting may come down early due to volume of applicants

Additional Information

UnitedHealth Group is a drug-free workplace; candidates are required to pass a drug test before beginning employment

Location: Eden Prairie, MN (remote)

View full job page →

3d ago

Sr. Data Engineer

Carhartt Dearborn, MI 🇺🇸 Remote

Dearborn, MI

3d ago

Full time Senior Azure Azure Data Engineer

Apply Now

Job Description

Carhartt is hiring a Senior Data Engineer to design, build, and support secure, reliable, scalable enterprise data solutions. The role focuses on modern Azure data platforms and Lakehouse architecture, while also placing emphasis on technical leadership, production excellence, and building reusable, AI-ready datasets and features. This remote position is based in Dearborn, MI with limited travel for special events.

What you will deliver

Design, develop, test, deploy, and support scalable batch and streaming pipelines using Azure Databricks, Azure Data Factory, Python, PySpark, SQL, and cloud-native services.
Design and evolve Lakehouse architecture using Delta Lake and Azure Data Lake Storage, including reusable data products and medallion-oriented layers where appropriate.
Build data models across conceptual, logical, dimensional, relational, and physical levels aligned to business meaning and analytics needs.
Optimize performance and cost by tuning pipelines, SQL queries, clusters, storage layouts, and processing patterns for reliability and scalability.
Implement orchestration and operational controls such as dependency management, scheduling, monitoring, logging, alerting, error handling, recovery, lineage, and observability for production workloads.
Apply data governance and quality practices, including metadata, cataloging, validation, reconciliation, security, privacy, retention, and access controls.
Engineer trusted, well-documented datasets and features that support reporting, advanced analytics, data science, and AI/ML use cases.

Leadership and collaboration

Lead technical design discussions, establish engineering standards and reusable patterns, and conduct code and design reviews with clear documentation of architecture decisions and tradeoffs.
Mentor data engineers through pairing, technical guidance, troubleshooting, coaching, and documentation.
Partner across the organization with business stakeholders, analysts, data scientists, product owners, architects, security teams, application teams, and vendors across the solution life cycle.

Engineering practices and production support

Use software engineering practices including Git-based source control, automated testing, CI/CD, deployment automation, and infrastructure configuration.
Provide advanced production support, including root-cause analysis, corrective actions, and continuous improvement for complex data platform and pipeline incidents.

Qualifications

Bachelor’s degree in Computer Science, Information Systems, Engineering, Data Science, or a related technical field (or equivalent related experience).
Graduate degree preferred.
Minimum 8 years of professional experience in data engineering, data warehousing, business intelligence, cloud data platforms, or a related field.
Demonstrated experience with Azure Databricks and Lakehouse architecture in enterprise data solutions.
Advanced experience with data pipeline design and orchestration, including batch and streaming patterns, dependencies, scheduling, monitoring, and recovery.
Advanced SQL skills including data modeling, query optimization, performance troubleshooting, and complex analytical transformations.
Professional experience with Python and distributed data processing frameworks such as PySpark.
Experience with Azure Databricks, Delta Lake, Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), Unity Catalog, Azure DevOps, and related Microsoft Azure data platform services.
Experience optimizing cloud workloads for performance, scalability, reliability, and cost effectiveness.
Working knowledge of data governance, metadata, cataloging, lineage, data quality, security, privacy, retention, and access-control principles.
Experience preparing trusted, governed, reusable, and well-documented data products for reporting, advanced analytics, data science, machine learning, generative AI, and future AI-driven business capabilities.
Experience with Git-based development, code reviews, automated testing, CI/CD, and deployment automation.
Experience integrating data from SAP, SAP Datasphere, APIs, files, events, product and master data platforms, and third-party enterprise systems.
Demonstrated technical leadership through architecture and solution design, engineering standards and best practices, design reviews, mentoring, code reviews, reusable patterns, and technical guidance.
Excellent verbal and written communication skills for technical and business audiences.
Strong cross-functional collaboration skills with business and technical teams.
Ability to isolate complex problems, evaluate options, make practical decisions, and provide advanced production support during critical incidents.
Professional cloud or data engineering certification considered a plus.

Benefits

Comprehensive suite of medical, dental, and vision insurance
Short-term and long-term disability insurance
Life insurance
Employee assistance program
401(k) plan with employer matching contributions
Paid time off and designated paid holidays

Work setup and conditions

Normal office environment with extended periods of time sitting, standing, and typing on a computer
May require willingness to work some weekends if necessary
Remote role with no regular requirement to be on-site; travel is limited to special events
Carhartt is a tobacco free workplace

View full job page →

5d ago

Senior Analytics Engineer

NRG San Francisco, CA

San Francisco, CA

5d ago

$131k - $157k/yr Full time Senior Analytics Analytics Engineer

Apply Now

Job Description

NRG is hiring a Senior Analytics Engineer to strengthen analytics engineering and data warehousing capabilities for operational and strategic decision-making.

Responsibilities

Build relationships with analytics teams and business leaders to understand goals, initiatives, and required data solutions
Design, build, and maintain an enterprise data warehouse platform, including associated data models and data transformation pipelines
Apply data best practices and contribute innovative solutions to deliver timely, accurate, and reliable data
Support implementation and ongoing optimization of a high-performing Snowflake data warehouse platform serving thousands of users and diverse use cases
Continuously evaluate and optimize data architecture deployments to improve efficiency and reliability

Requirements

3+ years of data modeling and data warehousing development experience (dbt, SSIS, Matillion, or similar)
Expertise in SQL and cloud warehouse design principles
2+ years of professional experience with Snowflake, Redshift, or Data Factory, or BigQuery (Snowflake preferred)
2+ years experience with Git, code reviews, and CI/CD environments
Ability to communicate effectively with business leaders and engineers; strong judgment to focus on highest-impact opportunities and make sound trade-offs

Technologies

dbt, SSIS, Matillion
SQL
Snowflake, Redshift, Data Factory, BigQuery
Git, CI/CD
Python, .net, R

Nice-to-Have Experience

Experience with SSIS
Knowledge of scripting languages (Python, .net, R, etc.)

Benefits

Free daily lunch and drinks on site
Paid holidays and flexible paid time away
Employee/Friends/Family Discounts
Onsite health clinic, gym, gaming tables
Medical/dental/vision/life coverage & 24/7 Medical Hotline
401(k) with Employer Match
Employee Resource Groups

Working Conditions

Professional office environment
Routine use of standard office equipment such as computers, phones, photocopiers, filing cabinets, and fax machines

Location & Compensation

California (onsite)
USD 131,400 - 157,300 per year

Education

BA/BS or higher in Computer Science, Information Systems, Math, other technical field, or relevant work experience

View full job page →

5d ago

Data Engineering Lead

Builders Atlanta, GA

Atlanta, GA

5d ago

Full time Manager Azure Azure Data Engineer

Apply Now

Job Description

Builders is a mid-sized mutual rated A by AM Best with a reputation for reliability and fairness. In this onsite role in Atlanta, GA, you will serve as the technical owner of a mission-critical Azure Databricks lakehouse platform supporting property and casualty insurance data, with a team-first, collaborative culture recognized by Top Workplaces and “Culture Excellence” honors.

Platform impact and engineering ownership

The Data Engineering Lead drives the design, development, and modernization of the Azure Databricks lakehouse that supports policy and claims operations and downstream analytics. You will align engineering execution with data governance through Unity Catalog, establish standards for secure and reliable delivery, and lead a team of Data Engineers and junior contractors.

What you will do

Design the Databricks lakehouse architecture using Delta Lake with bronze/silver/gold medallion layers for property and casualty policy, rating, claims, billing, reinsurance, and customer data.
Plan and lead phased migration from Guidewire (PolicyCenter, ClaimCenter, BillingCenter) and legacy systems into Azure Databricks, including historical loads, incremental/CDC loads, reconciliation, and support for decommissioning.
Model core property and casualty insurance entities including accounts, policies, insured risks and exposures, coverages, endorsements, premiums, claims, reserves, payments, and recoveries.
Provide data for Actuarial and Finance use cases such as written/earned/unearned premium, loss triangles, IBNR inputs, loss ratios, and statutory reporting including Schedule P and NAIC.
Integrate third-party data sources including geospatial and weather data.
Define Unity Catalog architecture (metastore, catalogs, schemas, and environments for development, test, and production), naming conventions, permissions, row/column-level security, and lineage.
Manage pipeline orchestration and operations in Databricks: schedule and dependencies using Databricks Workflows (Lakeflow Jobs), set up retries, alerting and SLA monitoring, and establish failure handling and runbooks.
Own performance and cost optimization across Spark workloads, including tuning, partitioning and liquid clustering, Photon, and cluster and SQL warehouse sizing.
Work with IT and Platform Engineering on storage, networking, security, identity, and IaC, including CI/CD.
Set engineering standards for pipelines, testing, code review, and documentation; lead, mentor, and review the work of Data Engineers and junior contractors.
Partner with the Data Governance Lead to implement access controls, PII masking, lineage, and data quality rules.
Perform other duties as assigned.

Qualifications

Associate degree or an equivalent combination of education and experience.
10+ years of data engineering experience, including 6+ years hands-on with Databricks (PySpark, Spark SQL, Delta Lake), plus experience leading or mentoring engineering teams.
Certification as a Databricks Data Engineer Professional. Azure data or solutions architect certification preferred.
Knowledge of Unity Catalog: governance model, permissions, row/column-level security, lineage, and system tables.
Experience leading large migrations from core insurance and legacy systems into a lakehouse or data warehouse, including property and casualty insurance data and understanding of the policy and claims lifecycle and core P&C KPIs.
Data modeling knowledge including dimensional modeling, data vault, SCD patterns, and insurance data models.
Technical depth in Spark internals and performance tuning, streaming (structured streaming, auto loader), CDC patterns, MERGE, and Lakeflow/Delta Live Tables.
Expert-level SQL and strong Python skills, plus software engineering practices such as testing, modular design, and code review.
Knowledge of data infrastructure: ADLS Gen2, Azure Data Factory, Event Hubs, Key Vault, Entra ID, VNets, and Private Link.
Experience scheduling and monitoring production pipelines with Databricks Workflows and/or Azure Data Factory, including dependencies, parameterization, alerting, and monitoring with Databricks system tables.
Experience with Git-based version control (GitHub or Bitbucket), branching strategies, pull request reviews, and Databricks Git folders.
Infrastructure-as-code and CI/CD for data: Terraform, Databricks Asset Bundles, and tools such as Azure DevOps, GitHub Actions, or Bitbucket Pipelines.
Guidewire experience (Guidewire DataHub/InfoCenter, Cloud Data Access) is preferred, including supporting actuarial reserving and pricing for P&C lines of business. API and integration tools such as MuleSoft preferred.
Knowledge of insurance data standards such as ACORD is preferred.