Databricks Dlt

Databricks Dlt Engineer Jobs

5 jobs found

Today

Data Engineer – Azure Databricks and Cloud Data Solutions (Safety)

Southern Company Birmingham, AL

Birmingham, AL

Today

Full time Analytics Azure Data Engineer Big Data

Apply Now

Job Description

Southern Company’s Operations Data Analytics team is seeking a Data Engineer focused on Safety data and cloud data services. The role builds and supports secure, reusable, high-performing pipelines and lakehouse solutions on Azure Databricks, aligned with enterprise expectations for quality, cost, and timeliness.

Responsibilities

Design and develop robust data pipelines to ingest, store, and process data from multiple sources for analytics and reporting.
Create and maintain logical and physical data models to support efficient querying, analysis, and visualization.
Build and maintain ETL processes that move data between systems while supporting data quality and consistency.
Implement and support lakehouse solutions using medallion architecture, including data engineering pipelines and data storage layers.
Optimize data lake storage, retrieval, performance, and resource utilization for analytical workloads.
Develop dynamic, metadata-driven frameworks and scalable data ingestion patterns.
Support streaming datasets and Databricks Delta Live Tables.
Apply lakehouse performance-tuning techniques, including partitioning, aggregations, and materialized views.
Architect, develop, implement, integrate, and support cloud technology solutions that meet business partner expectations for quality, cost, and timeliness.
Partner with developers and infrastructure teams to optimize application code, create reusable services, plan capacity, and protect application and data security.
Serve as a subject matter expert for data architecture and Databricks.
Communicate effectively and promptly with customers, team members, and management, building strong peer and business partner relationships.
Develop required project documentation, follow Change Management policies and procedures, and respond quickly to operational problems.
Build a strong understanding of business challenges and goals and exhibit Our Values.

Requirements

Bachelor’s degree required, preferably in Computer Science, Management Information Systems, Engineering, or another technology-related field.
Experience with Databricks, including notebooks, clusters, and jobs.
Experience with a major cloud platform, preferably Microsoft Azure, including services such as Cosmos DB, Storage Application Services, and Azure Data Factory.
Experience architecting data-oriented solutions using cloud data architecture platforms and patterns, preferably Azure.
Experience implementing lakehouses with medallion architecture, data engineering pipelines, and data storage layers.
Experience with dynamic metadata-driven frameworks and large-scale data ingestion patterns.
Experience developing complex logical and physical data models.
Experience optimizing cloud PaaS resources to follow best practices and reduce cost, plus lakehouse performance tuning using partitioning, aggregations, materialized views, or similar techniques.
Experience with Agile, DevOps, and CI/CD methodologies.
Familiarity with Apache Spark, Delta Lake, and MLflow, including knowledge of structured and unstructured data across databases, data warehouses, and data lakes.
Knowledge of software development technologies, methodologies, and associated tools.
Excellent root cause analysis and problem-solving skills.

Technologies

Microsoft Azure
Azure Databricks
Unity catalog
Cosmos DB
Storage Application Services
Azure Data Factory
Databricks (notebooks, clusters, jobs)
Azure data-oriented solutions
Medallion architecture
Extract, Transform, Load (ETL)
Databricks Delta Live Tables
Partitioning, aggregations, materialized views
Apache Spark
Delta Lake
MLflow
Agile, DevOps, CI/CD
Structured and unstructured data
Lakehouse

Benefits

Competitive base salary
Annual incentive awards for eligible employees
Health, welfare and retirement benefits designed to support physical, financial, and emotional/social well-being
Additional and specific details about total compensation and benefits provided during the hiring process

What Success Looks Like

Reliable, efficient data pipelines that support analytics and reporting needs.
Reusable and well-aligned solutions across on-premises and Azure technology platforms.
Consistent, high-quality data movement through maintainable ETL and lakehouse processes.
Optimized application code, cloud resources, performance, and capacity utilization.
Secure data and application solutions supported by complete project documentation and disciplined change management.
Strong working relationships and timely communication with business partners, technology teams, and management.
Fast, effective resolution of operational problems and growing team expertise in data architecture and Databricks.

What Sets You Apart (Preferred)

Hands-on experience with Microsoft Azure data services and cloud PaaS architecture and resources.
Familiarity with streaming datasets and Databricks Delta Live Tables.
Strong technical writing skills.

What We Value

Customer focus: Deliver cloud and data solutions that meet or exceed business partner expectations for quality, cost, and timeliness.
Ownership and responsiveness: Take responsibility for documentation, change management, and rapid resolution of operational problems.
Collaboration: Communicate effectively and build productive relationships with customers, peers, management, developers, infrastructure teams, and business partners.
Continuous learning: Become a subject matter expert, stay close to the business, and seek to understand the company’s challenges and goals.
Enterprise mindset: Design secure, reusable, aligned solutions and exhibit Our Values.

Location and Work Model

Birmingham, AL (onsite)

View full job page →

1d ago

Data Engineer, Finance

SHIELD AI San Diego, CA

San Diego, CA

1d ago

$97k - $140k/yr Full time Ai/ml Data Engineering Artificial Intelligence Big Data

Apply Now

Job Description

Shield AI is hiring a Data Engineer, Finance to create a dependable, governed data foundation that supports Finance planning, analytics, and AI-enabled decision workflows. This onsite role in San Diego focuses on reusable Databricks integrations, semantic-layer infrastructure, and production-grade data services for enterprise stakeholders.

Responsibilities

Build and operate scalable pipelines and integrations that connect enterprise data to Finance products.
Develop semantic-layer infrastructure and governed data access for analytics, planning, and AI applications.
Implement end-to-end pipeline observability, data-quality controls, and resilient operational practices.
Engineer the services and integrations that power Databricks Genie and other AI-enabled Finance workflows.
Use AI coding assistants to accelerate development while validating correctness, security, and performance.
Apply FinOps practices to manage platform spend and optimize cost without reducing service quality.
Partner with analytics engineers and enterprise teams on architecture decisions, data contracts, and sustainable ownership.

Requirements

Hands-on production data engineering experience using SQL and Python or PySpark.
Experience with cloud data platforms, orchestration, data modeling, and governed serving patterns.
Ability to design systems with observability, recoverability, security, and maintainability in mind.
Proficiency with AI coding assistants, including independent validation of generated work.
Practical experience enabling AI capabilities on enterprise data, ideally in Databricks; experience with Snowflake Cortex or comparable platforms is welcome.
Strong cross-functional collaboration skills and demonstrated production ownership.

Technologies

Databricks
SQL
Python
PySpark
Snowflake Cortex
Databricks Genie
Delta Lake

Location and Salary

Location: San Diego, CA (onsite)

Compensation: USD 97,000 - 140,000 per year

Benefits

Bonus + Benefits + Equity
Pay within range listed + Bonus + Benefits + Equity
Pay within range listed above + temporary benefits package (applicable after 60 days of employment)

Preferred Qualifications

Experience with Databricks, Delta Lake, cloud infrastructure, and deployment automation.
Experience developing AI application backends, model integrations, or LLM-based workflows.
Experience with FinOps, financial data, or enterprise-system integrations.

View full job page →

2d ago

Data Engineering Lead

Builders Atlanta, GA

Atlanta, GA

2d ago

Full time Manager Azure Azure Data Engineer

Apply Now

Job Description

Builders is a mid-sized mutual rated A by AM Best with a reputation for reliability and fairness. In this onsite role in Atlanta, GA, you will serve as the technical owner of a mission-critical Azure Databricks lakehouse platform supporting property and casualty insurance data, with a team-first, collaborative culture recognized by Top Workplaces and “Culture Excellence” honors.

Platform impact and engineering ownership

The Data Engineering Lead drives the design, development, and modernization of the Azure Databricks lakehouse that supports policy and claims operations and downstream analytics. You will align engineering execution with data governance through Unity Catalog, establish standards for secure and reliable delivery, and lead a team of Data Engineers and junior contractors.

What you will do

Design the Databricks lakehouse architecture using Delta Lake with bronze/silver/gold medallion layers for property and casualty policy, rating, claims, billing, reinsurance, and customer data.
Plan and lead phased migration from Guidewire (PolicyCenter, ClaimCenter, BillingCenter) and legacy systems into Azure Databricks, including historical loads, incremental/CDC loads, reconciliation, and support for decommissioning.
Model core property and casualty insurance entities including accounts, policies, insured risks and exposures, coverages, endorsements, premiums, claims, reserves, payments, and recoveries.
Provide data for Actuarial and Finance use cases such as written/earned/unearned premium, loss triangles, IBNR inputs, loss ratios, and statutory reporting including Schedule P and NAIC.
Integrate third-party data sources including geospatial and weather data.
Define Unity Catalog architecture (metastore, catalogs, schemas, and environments for development, test, and production), naming conventions, permissions, row/column-level security, and lineage.
Manage pipeline orchestration and operations in Databricks: schedule and dependencies using Databricks Workflows (Lakeflow Jobs), set up retries, alerting and SLA monitoring, and establish failure handling and runbooks.
Own performance and cost optimization across Spark workloads, including tuning, partitioning and liquid clustering, Photon, and cluster and SQL warehouse sizing.
Work with IT and Platform Engineering on storage, networking, security, identity, and IaC, including CI/CD.
Set engineering standards for pipelines, testing, code review, and documentation; lead, mentor, and review the work of Data Engineers and junior contractors.
Partner with the Data Governance Lead to implement access controls, PII masking, lineage, and data quality rules.
Perform other duties as assigned.

Qualifications

Associate degree or an equivalent combination of education and experience.
10+ years of data engineering experience, including 6+ years hands-on with Databricks (PySpark, Spark SQL, Delta Lake), plus experience leading or mentoring engineering teams.
Certification as a Databricks Data Engineer Professional. Azure data or solutions architect certification preferred.
Knowledge of Unity Catalog: governance model, permissions, row/column-level security, lineage, and system tables.
Experience leading large migrations from core insurance and legacy systems into a lakehouse or data warehouse, including property and casualty insurance data and understanding of the policy and claims lifecycle and core P&C KPIs.
Data modeling knowledge including dimensional modeling, data vault, SCD patterns, and insurance data models.
Technical depth in Spark internals and performance tuning, streaming (structured streaming, auto loader), CDC patterns, MERGE, and Lakeflow/Delta Live Tables.
Expert-level SQL and strong Python skills, plus software engineering practices such as testing, modular design, and code review.
Knowledge of data infrastructure: ADLS Gen2, Azure Data Factory, Event Hubs, Key Vault, Entra ID, VNets, and Private Link.
Experience scheduling and monitoring production pipelines with Databricks Workflows and/or Azure Data Factory, including dependencies, parameterization, alerting, and monitoring with Databricks system tables.
Experience with Git-based version control (GitHub or Bitbucket), branching strategies, pull request reviews, and Databricks Git folders.
Infrastructure-as-code and CI/CD for data: Terraform, Databricks Asset Bundles, and tools such as Azure DevOps, GitHub Actions, or Bitbucket Pipelines.
Guidewire experience (Guidewire DataHub/InfoCenter, Cloud Data Access) is preferred, including supporting actuarial reserving and pricing for P&C lines of business. API and integration tools such as MuleSoft preferred.
Knowledge of insurance data standards such as ACORD is preferred.

Compensation and benefits

Competitive Salary
Bonus Structure
Profit Sharing
Medical, Dental, Vision Insurance
Employer Paid Short Term Disability
Employer Paid Long Term Disability
Employer Paid Life Insurance
Voluntary Life Insurance
401K with Company Match
PTO

Equal opportunity

Builders Insurance Group is an equal opportunity employer. They welcome applicants from all walks of life and do not discriminate based on any protected status.

View full job page →

10d ago

Principal Analytics Engineer

Cambridge Mobile Telematics Cambridge, MA

Cambridge, MA

10d ago

$170k - $212k/yr Full time Analytics Artificial Intelligence Big Data

Apply Now

Job Description

Principal Analytics Engineer - Data Architecture is a principal individual contributor and technical leader on the Customer Analytics team.

Responsibilities

Architect relational data marts and high-scale dbt models in Databricks and Delta Lake using dimensional modeling and software engineering practices for scale, maintainability, and cost efficiency
Own analytics data products end to end: discovery and contracts, automated testing, CI/CD deployment, documentation, monitoring, support, and controlled change
Design and productize a modular semantic layer and shared data services delivering governed metrics and dimensions, metadata, approved access paths, and vector-enabled retrieval for BI, data science, and AI
Operate high-availability data and reporting products to agreed SLAs using freshness and quality alerts, lineage, observability, proactive issue detection, and urgent resolution
Drive alignment on metric definitions, governance, and domain ownership; resolve ambiguity with internal and external stakeholders
Raise team standards through reusable dbt frameworks, technical reviews, mentoring, and AI-assisted development
Complete additional tasks as they arise

Requirements

Bachelor’s degree or equivalent years of experience and/or certification in a related field
7+ years of experience in analytics engineering, data engineering, or data architecture with a bachelor’s degree, or 5+ years with a graduate degree
Expert SQL and dimensional modeling, including grain, star and snowflake schemas, conformed facts and dimensions, SCD Type 2, snapshots, event models, and incremental processing
Extensive dbt experience with modular models, macros, packages, custom tests, documentation, and source freshness; hands-on Databricks, Delta Lake, and Spark experience
Production engineering experience with Git, CI/CD, automated testing, observability, SLA support, performance tuning, data catalogs, lineage, and policy-aware access
Experience with semantic layers, embeddings, semantic search, and vector databases, plus exposure to AI/LLM tools as data consumers; proven product-oriented ownership and cross-functional influence

Technologies

SQL
dbt
Databricks
Delta Lake
Spark
Git
CI/CD
Dimensional modeling
Star schema
Snowflake schema
SCD Type 2
Event models
Incremental processing
Semantic layers
Embeddings
Semantic search
Vector databases
AI/LLM tools

Benefits

Annual performance bonus
Equity may be awarded in the form of Restricted Stock Units (RSUs)
Medical, Dental, Vision and Life Insurance
Matching 401k
Short-term & long-term disability
Parental leave
Unlimited Paid Time Off including vacation, sick days & public holidays
Flexible scheduling and work from home policy depending on role and responsibilities

Compensation

Base salary range: $169,700 to $212,100 per year (Cambridge, MA).

Additional Perks

Work on a mission with real impact: crashes prevented, injuries avoided, lives protected around the world
Join an industry leader: 65 million drivers protected, powering 140+ programs across 25 countries
Recognized innovator in mobility AI, including TIME Industry Leader in AI, a Gold Edison Award, and the Artificial Intelligence Excellence Award for AI for Social Good; Great Place to Work Certified
Be part of the team inventing the future of mobility and road safety
Move fast, own outcomes, do work that matters
High ownership, small teams, and direct access to leadership
Summer Fridays with half days for team members
Employee resource groups: Black, AAPI, LGBTQIA+, Women, Book Club, and Health & Wellness
Comprehensive wellness, education, and employee assistance programs

Diversity and Inclusion

CMT is an equal-opportunity employer committed to creating a workplace and culture where everyone feels valued, respected, and empowered
No discrimination based on race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status or disability state

Company Overview

Cambridge Mobile Telematics (CMT) is the world’s largest telematics and AI company for safer mobility
Mission: make the world’s roads and drivers safer
DriveWell Fusion® proactively identifies and reduces driving risk, leading to fewer crashes and injuries
Over 126,000 crashes prevented worldwide
Helps partners measure risk, detect crashes, provide life-saving assistance, and streamline claims
Headquartered in Cambridge, MA; offices in Budapest, Hungary; Chennai, India; Seattle, Washington; Tokyo, Japan; and Zagreb, Croatia

Location

Cambridge, MA (onsite)

View full job page →

11d ago

Data Engineer 2

Comcast Corporation Willow Grove, PA

Willow Grove, PA

11d ago

$80k - $133k/yr Full time Analytics Big Data Bigdata

Apply Now

Job Description

Design and deliver scalable data platforms and pipelines for Comcast Advertising, enabling reliable data products and self-service analytics at scale.

Responsibilities

Architect, design, and build high-performance, scalable data pipelines for large-scale data processing
Develop complex Big Data ETL workflows using modern frameworks such as Databricks Delta Live Tables (DLT)
Design and implement data lake architectures using open table formats including Delta Lake and Apache Iceberg
Define, promote, and enforce data engineering best practices to support data quality, integrity, reliability, and performance
Implement observability and monitoring across pipelines, including data cataloging, lineage, and metadata management
Participate in peer reviews, knowledge sharing, production support, incident resolution, and change management
Follow code management and version control practices across development workflows
Create high-quality documentation, including data flows, lineage diagrams, and architecture details aligned to data engineering standards
Ensure comprehensive code coverage by building unit and integration tests, using test-driven development (TDD) principles
Own critical components of large-scale batch ETL systems; communicate data models and design decisions to Product, Engineering, and Leadership stakeholders
Manage multiple priorities by balancing long-term architectural direction with immediate business needs
Build advanced semantic layers and support creation of customer-facing self-service analytics
Apply independent judgment and discretion for complex, high-impact technical decisions
Maintain regular, consistent, and punctual attendance, with flexibility to work nights, weekends, or variable schedules as required
Perform additional duties and responsibilities as assigned

Requirements

Minimum 2 years of Data Engineering experience (relevant work experience: 2-5 years)
Proficiency in Python and PySpark; solid grasp of SQL
Hands-on experience with Databricks and/or Snowflake is preferred
Data warehouse modeling paradigms
Understanding of cloud-based architectures (AWS)
Problem-solving skills
Nice to have:
- AWS Certified Developer Associate
- Databricks Certified Data Engineer Associate
- SnowPro Advanced: Data Engineer

Technologies

SQL
Python
PySpark
Apache Spark
AWS
Databricks Delta Live Tables (DLT)
Databricks
Delta Lake
Apache Iceberg
Looker
Tab
Snowflake

Location

Willow Grove, PA (onsite)

Salary

USD 79,728 - 132,880 per year
Comcast intends to offer base pay within this range, depending on job-related, non-discriminatory factors such as experience
Pay Range noted for Illinois and Massachusetts: $79,728.27 - $132,880.46
Application window is 30 days from the date job is posted, unless the number of applicants requires it to close sooner or later

Education

Bachelor's Degree
Comcast may also consider applicants with some combination of coursework and experience, or extensive related professional experience

View full job page →

Databricks Dlt Engineer Jobs

Data Engineer – Azure Databricks and Cloud Data Solutions (Safety)

Data Engineer, Finance

Data Engineering Lead

Principal Analytics Engineer

Data Engineer 2

Get Databricks Dlt Engineer Jobs alerts