Databricks Delta

Databricks Delta Engineer Jobs

4 jobs found

Today

Data Engineer, Finance

SHIELD AI San Diego, CA

San Diego, CA

Today

$97k - $140k/yr Full time Ai/ml Data Engineering Artificial Intelligence Big Data

Apply Now

Job Description

Shield AI is hiring a Data Engineer, Finance to create a dependable, governed data foundation that supports Finance planning, analytics, and AI-enabled decision workflows. This onsite role in San Diego focuses on reusable Databricks integrations, semantic-layer infrastructure, and production-grade data services for enterprise stakeholders.

Responsibilities

Build and operate scalable pipelines and integrations that connect enterprise data to Finance products.
Develop semantic-layer infrastructure and governed data access for analytics, planning, and AI applications.
Implement end-to-end pipeline observability, data-quality controls, and resilient operational practices.
Engineer the services and integrations that power Databricks Genie and other AI-enabled Finance workflows.
Use AI coding assistants to accelerate development while validating correctness, security, and performance.
Apply FinOps practices to manage platform spend and optimize cost without reducing service quality.
Partner with analytics engineers and enterprise teams on architecture decisions, data contracts, and sustainable ownership.

Requirements

Hands-on production data engineering experience using SQL and Python or PySpark.
Experience with cloud data platforms, orchestration, data modeling, and governed serving patterns.
Ability to design systems with observability, recoverability, security, and maintainability in mind.
Proficiency with AI coding assistants, including independent validation of generated work.
Practical experience enabling AI capabilities on enterprise data, ideally in Databricks; experience with Snowflake Cortex or comparable platforms is welcome.
Strong cross-functional collaboration skills and demonstrated production ownership.

Technologies

Databricks
SQL
Python
PySpark
Snowflake Cortex
Databricks Genie
Delta Lake

Location and Salary

Location: San Diego, CA (onsite)

Compensation: USD 97,000 - 140,000 per year

Benefits

Bonus + Benefits + Equity
Pay within range listed + Bonus + Benefits + Equity
Pay within range listed above + temporary benefits package (applicable after 60 days of employment)

Preferred Qualifications

Experience with Databricks, Delta Lake, cloud infrastructure, and deployment automation.
Experience developing AI application backends, model integrations, or LLM-based workflows.
Experience with FinOps, financial data, or enterprise-system integrations.

View full job page →

1d ago

Data Engineering Lead

Builders Atlanta, GA

Atlanta, GA

1d ago

Full time Manager Azure Azure Data Engineer

Apply Now

Job Description

Builders is a mid-sized mutual rated A by AM Best with a reputation for reliability and fairness. In this onsite role in Atlanta, GA, you will serve as the technical owner of a mission-critical Azure Databricks lakehouse platform supporting property and casualty insurance data, with a team-first, collaborative culture recognized by Top Workplaces and “Culture Excellence” honors.

Platform impact and engineering ownership

The Data Engineering Lead drives the design, development, and modernization of the Azure Databricks lakehouse that supports policy and claims operations and downstream analytics. You will align engineering execution with data governance through Unity Catalog, establish standards for secure and reliable delivery, and lead a team of Data Engineers and junior contractors.

What you will do

Design the Databricks lakehouse architecture using Delta Lake with bronze/silver/gold medallion layers for property and casualty policy, rating, claims, billing, reinsurance, and customer data.
Plan and lead phased migration from Guidewire (PolicyCenter, ClaimCenter, BillingCenter) and legacy systems into Azure Databricks, including historical loads, incremental/CDC loads, reconciliation, and support for decommissioning.
Model core property and casualty insurance entities including accounts, policies, insured risks and exposures, coverages, endorsements, premiums, claims, reserves, payments, and recoveries.
Provide data for Actuarial and Finance use cases such as written/earned/unearned premium, loss triangles, IBNR inputs, loss ratios, and statutory reporting including Schedule P and NAIC.
Integrate third-party data sources including geospatial and weather data.
Define Unity Catalog architecture (metastore, catalogs, schemas, and environments for development, test, and production), naming conventions, permissions, row/column-level security, and lineage.
Manage pipeline orchestration and operations in Databricks: schedule and dependencies using Databricks Workflows (Lakeflow Jobs), set up retries, alerting and SLA monitoring, and establish failure handling and runbooks.
Own performance and cost optimization across Spark workloads, including tuning, partitioning and liquid clustering, Photon, and cluster and SQL warehouse sizing.
Work with IT and Platform Engineering on storage, networking, security, identity, and IaC, including CI/CD.
Set engineering standards for pipelines, testing, code review, and documentation; lead, mentor, and review the work of Data Engineers and junior contractors.
Partner with the Data Governance Lead to implement access controls, PII masking, lineage, and data quality rules.
Perform other duties as assigned.

Qualifications

Associate degree or an equivalent combination of education and experience.
10+ years of data engineering experience, including 6+ years hands-on with Databricks (PySpark, Spark SQL, Delta Lake), plus experience leading or mentoring engineering teams.
Certification as a Databricks Data Engineer Professional. Azure data or solutions architect certification preferred.
Knowledge of Unity Catalog: governance model, permissions, row/column-level security, lineage, and system tables.
Experience leading large migrations from core insurance and legacy systems into a lakehouse or data warehouse, including property and casualty insurance data and understanding of the policy and claims lifecycle and core P&C KPIs.
Data modeling knowledge including dimensional modeling, data vault, SCD patterns, and insurance data models.
Technical depth in Spark internals and performance tuning, streaming (structured streaming, auto loader), CDC patterns, MERGE, and Lakeflow/Delta Live Tables.
Expert-level SQL and strong Python skills, plus software engineering practices such as testing, modular design, and code review.
Knowledge of data infrastructure: ADLS Gen2, Azure Data Factory, Event Hubs, Key Vault, Entra ID, VNets, and Private Link.
Experience scheduling and monitoring production pipelines with Databricks Workflows and/or Azure Data Factory, including dependencies, parameterization, alerting, and monitoring with Databricks system tables.
Experience with Git-based version control (GitHub or Bitbucket), branching strategies, pull request reviews, and Databricks Git folders.
Infrastructure-as-code and CI/CD for data: Terraform, Databricks Asset Bundles, and tools such as Azure DevOps, GitHub Actions, or Bitbucket Pipelines.
Guidewire experience (Guidewire DataHub/InfoCenter, Cloud Data Access) is preferred, including supporting actuarial reserving and pricing for P&C lines of business. API and integration tools such as MuleSoft preferred.
Knowledge of insurance data standards such as ACORD is preferred.

Compensation and benefits

Competitive Salary
Bonus Structure
Profit Sharing
Medical, Dental, Vision Insurance
Employer Paid Short Term Disability
Employer Paid Long Term Disability
Employer Paid Life Insurance
Voluntary Life Insurance
401K with Company Match
PTO

Equal opportunity

Builders Insurance Group is an equal opportunity employer. They welcome applicants from all walks of life and do not discriminate based on any protected status.

View full job page →

1d ago

Data Engineer

Builders Atlanta, GA

Atlanta, GA

1d ago

Full time Azure Data Engineer Azure Data Lake Big Data

Apply Now

Job Description

Design, build, and operate data pipelines that power Builders’ Azure Databricks lakehouse for property and casualty insurance.

Responsibilities

Develop pipelines for phased migration into the Azure Databricks lakehouse from Guidewire (PolicyCenter, ClaimCenter, BillingCenter) and legacy systems, including historical loads, incremental/CDC (Change Data Capture) loads, and reconciliation
Create bronze and silver layer transformations in PySpark and Spark SQL following the medallion architecture
Support data modeling for property and casualty domains: policies, insured risks and exposures, coverages, endorsements, premiums, claims, reserves, payments, and recoveries
Integrate third-party datasets, including geospatial and weather data
Schedule and run production pipelines using Databricks Workflows (Lakeflow Jobs) or Azure Data Factory, including job dependencies, parameters, retries, and alerting
Investigate and resolve pipeline failures in production
Apply Unity Catalog controls set by the Data Engineering Lead and Data Governance lead, including permissions, PII masking, tags, and lineage
Implement data quality checks, error handling, and monitoring
Tune pipeline performance and cost using partitioning, liquid clustering, and right-sized job compute
Use Git version control (GitHub or Bitbucket) with CI/CD via Databricks Asset Bundles to version, review, and deploy code across development, test, and production
Write documentation and runbooks; participate in code reviews; take long-term ownership of contractor-built pipelines
Perform other duties as assigned

Requirements

Associate degree or an equivalent combination of education and experience
Certification in Databricks engineering
3+ years in data engineering, including 1+ year hands-on with Databricks (PySpark, Spark SQL, Delta Lake)
Property and casualty insurance experience, ideally Guidewire data; experience migrating data from legacy source systems
Knowledge of Python/PySpark and SQL (Structured Query Language)
Knowledge of Unity Catalog: permissions, row/column-level security, and lineage
Experience with incremental loading patterns such as CDC, MERGE, SCD (Slowly Changing Dimension), and Auto Loader
Knowledge of data modeling, including dimensional modeling
Experience scheduling and monitoring pipelines with Databricks Workflows and/or Azure Data Factory
Experience with Git-based version control (GitHub or Bitbucket), including branching, pull request reviews, and Databricks Git folders
Experience with Azure data services such as ADLS (Azure Data Lake Storage) Gen2, Azure Data Factory, and Key Vault
CI/CD fundamentals: Databricks Asset Bundles, Azure DevOps, GitHub Actions, or Bitbucket Pipelines
Experience with streaming (Structured Streaming, Event Hubs)
Experience with API (Application Programming Interface) and integration tools such as MuleSoft is an added advantage

Technologies

Azure Databricks, Delta Lake, Unity Catalog
Lakeflow, Databricks SQL, PySpark, Spark SQL
Databricks Workflows, Azure Data Factory
Git, GitHub, Bitbucket, Databricks Asset Bundles
Python, SQL (Structured Query Language)
ADLS (Azure Data Lake Storage) Gen2, Key Vault
Structured Streaming, Event Hubs, Auto Loader
MERGE, SCD (Slowly Changing Dimension), CDC (Change Data Capture)
MuleSoft, API (Application Programming Interface)
Azure DevOps, GitHub Actions, Bitbucket Pipelines

Benefits

Competitive Salary
Bonus Structure
Profit Sharing
Medical, Dental, Vision Insurance
Employer Paid Short Term Disability
Employer Paid Long Term Disability
Employer Paid Life Insurance
Voluntary Life Insurance
401K with Company Match
PTO

Position Summary

Design, develop, and maintain data pipelines for Builders’ Azure Databricks lakehouse supporting underwriting, claims, actuarial, and finance
Build reliable, high-quality, well-tested, and well-documented data solutions using Delta Lake, Unity Catalog, Lakeflow pipelines, and Databricks SQL
Deliver phased migration pipelines from Guidewire and legacy policy, claims, and billing systems into Azure Databricks
Align solutions with Unity Catalog governance and engineering standards led by the Data Engineering Lead
Strengthen long-term sustainability by progressively assuming responsibility for contractor-developed pipelines

About Builders

Builders is a mid-sized mutual with an A rating by AM Best
Recognized for reliability and fairness in fulfilling promises to customers
Teams foster a collaborative, customer-centric environment; “Culture Excellence” honors from Top Workplaces
Supports policyholders through partnerships with knowledgeable independent agencies

Equal Opportunity: Builders is an equal opportunity employer and does not discriminate based on any protected status.

Location: Atlanta, GA (onsite) • Experience: 3+ years • Education: Associate degree

View full job page →

3d ago

Databricks Data Engineer, Lead

Booz Allen Hamilton Bethesda, MD

Bethesda, MD

3d ago

$78k - $176k/yr Full time Manager Aws Data Engineer Big Data

Apply Now

Job Description

Booz Allen Hamilton is seeking a Lead Databricks Data Engineer in Bethesda, Maryland, to support the design, build, and ongoing support of secure, scalable data pipelines. This role combines hands-on development in Databricks with technical leadership, mentorship, and platform oversight across Databricks and AWS.

Role focus

Design, build, and maintain robust data pipelines within Databricks, contributing independently while providing technical leadership and mentorship to the team
Create and optimize end-to-end data flows, including extract, transform, load (ETL) processes
Design data models and improve performance and scalability for data processing and analytics
Ensure data security and compliance throughout pipeline and platform operations
Collaborate with cross-functional teams to support client outcomes and continuous improvement
Mentor a multi-disciplinary team of analysts, data engineers, developers, and data consumers in a fast-paced, agile environment
Oversee assessment, design, building, and maintenance of scalable platforms for clients

Experience and qualifications

8+ years designing, developing, deploying, and managing ETL processes and data pipelines with comprehensive logging and robust error handling to troubleshoot and resolve issues
3+ years building, debugging, and optimizing data pipelines using Databricks notebooks, clusters, and job orchestration tools with Delta Lake, Databricks Delta, and Unity Catalog
3+ years writing Python and PySpark code for ETL scripts and custom transformations
Experience with data modeling, including choosing between dimensional vs relation data models within a medallion architecture using Bronze, Silver, and Gold
Experience with AWS services including AWS Glue, Amazon S3, and Amazon RDS
Knowledge of ETL performance optimization techniques such as partitioning, bucketing, and caching, plus troubleshooting to improve query performance
Ability to design and implement data models from schema definition through optimization for efficient queries and analytics
Public Trust
Bachelor’s degree in Computer Science, Computer Engineering, Engineering, Mathematics, Physics, Information Systems, or Information Technology

Technologies

Databricks, AWS, Databricks notebooks, clusters, job orchestration tools
Delta Lake, Databricks Delta, Unity Catalog
Python, PySpark
AWS Glue, Amazon S3, Amazon RDS

Benefits

Health, life, disability, financial, and retirement benefits
Paid leave
Professional development and tuition assistance
Work-life programs and dependent care
Recognition awards program acknowledging employees for exceptional performance and demonstration of company values
Full-time and part-time employees working at least 20 hours a week on a regular basis are eligible to participate in Booz Allen benefit programs

Additional information

Location: Bethesda, MD (onsite)
Work model: Onsite, with work primarily performed full-time at a customer facility
Compensation: projected range of $77,600.00 to $176,000.00 (annualized USD)
Applicants selected will be subject to a government investigation and may need to meet eligibility requirements of the U.S. government client
Public Trust determination is required

Identity and interview policy

Completing an identity verification process that leverages advanced biometrics and artificial intelligence to ensure authenticity and protect against identity fraud
Employees are expected to be on camera during interviews and assessments
Booz Allen reserves the right to take your picture to verify identity and prevent fraud
The use of AI or other tools to assist with responses during interviews is prohibited unless permission is explicitly provided

Nice to have

Experience with AWS cloud-native data services such as AWS Lambda or CloudWatch for serverless data processing architectures
Experience with tools such as AWS Glue Data Catalog or Collibra for managing metadata, data lineage, and cataloging data assets
Knowledge of data governance practices, data lineage, and compliance requirements (including GDPR or CCPA) supporting data quality, privacy, and security
Ability to explain complex technical concepts to a non-technical audience, including architectural tradeoffs

View full job page →

Databricks Delta Engineer Jobs

Data Engineer, Finance

Data Engineering Lead

Data Engineer

Databricks Data Engineer, Lead

Get Databricks Delta Engineer Jobs alerts