Databricks Pyspark

Databricks Pyspark Engineer Jobs

21 jobs found

Today

Databricks AI Engineer

Perficient, Inc United States

United States

Today

Full time Artificial Intelligence Big Data Bigdata

Apply Now

Job Description

Perficient, Inc is hiring a Senior Databricks AI Engineer to join its AI Data Platform business unit. This United States (onsite) role focuses on delivering client applications end-to-end, from early planning through release and ongoing support. You will work with solution architects and leads to execute development activities across the full software lifecycle.

What you’ll do

Participate in all phases of the software development lifecycle, including estimating, technical design, implementation, documentation, testing, deployment, and support for client applications.
Take direction from solution architects and leads to carry out development activities.

Technology focus

This position centers on Databricks.

Work model

Location: United States (onsite).

View full job page →

2d ago

Data Engineer, Finance

SHIELD AI San Diego, CA

San Diego, CA

2d ago

$97k - $140k/yr Full time Ai/ml Data Engineering Artificial Intelligence Big Data

Apply Now

Job Description

Shield AI is hiring a Data Engineer, Finance to create a dependable, governed data foundation that supports Finance planning, analytics, and AI-enabled decision workflows. This onsite role in San Diego focuses on reusable Databricks integrations, semantic-layer infrastructure, and production-grade data services for enterprise stakeholders.

Responsibilities

Build and operate scalable pipelines and integrations that connect enterprise data to Finance products.
Develop semantic-layer infrastructure and governed data access for analytics, planning, and AI applications.
Implement end-to-end pipeline observability, data-quality controls, and resilient operational practices.
Engineer the services and integrations that power Databricks Genie and other AI-enabled Finance workflows.
Use AI coding assistants to accelerate development while validating correctness, security, and performance.
Apply FinOps practices to manage platform spend and optimize cost without reducing service quality.
Partner with analytics engineers and enterprise teams on architecture decisions, data contracts, and sustainable ownership.

Requirements

Hands-on production data engineering experience using SQL and Python or PySpark.
Experience with cloud data platforms, orchestration, data modeling, and governed serving patterns.
Ability to design systems with observability, recoverability, security, and maintainability in mind.
Proficiency with AI coding assistants, including independent validation of generated work.
Practical experience enabling AI capabilities on enterprise data, ideally in Databricks; experience with Snowflake Cortex or comparable platforms is welcome.
Strong cross-functional collaboration skills and demonstrated production ownership.

Technologies

Databricks
SQL
Python
PySpark
Snowflake Cortex
Databricks Genie
Delta Lake

Location and Salary

Location: San Diego, CA (onsite)

Compensation: USD 97,000 - 140,000 per year

Benefits

Bonus + Benefits + Equity
Pay within range listed + Bonus + Benefits + Equity
Pay within range listed above + temporary benefits package (applicable after 60 days of employment)

Preferred Qualifications

Experience with Databricks, Delta Lake, cloud infrastructure, and deployment automation.
Experience developing AI application backends, model integrations, or LLM-based workflows.
Experience with FinOps, financial data, or enterprise-system integrations.

View full job page →

3d ago

Data Engineering Lead

Builders Atlanta, GA

Atlanta, GA

3d ago

Full time Manager Azure Azure Data Engineer

Apply Now

Job Description

Builders is a mid-sized mutual rated A by AM Best with a reputation for reliability and fairness. In this onsite role in Atlanta, GA, you will serve as the technical owner of a mission-critical Azure Databricks lakehouse platform supporting property and casualty insurance data, with a team-first, collaborative culture recognized by Top Workplaces and “Culture Excellence” honors.

Platform impact and engineering ownership

The Data Engineering Lead drives the design, development, and modernization of the Azure Databricks lakehouse that supports policy and claims operations and downstream analytics. You will align engineering execution with data governance through Unity Catalog, establish standards for secure and reliable delivery, and lead a team of Data Engineers and junior contractors.

What you will do

Design the Databricks lakehouse architecture using Delta Lake with bronze/silver/gold medallion layers for property and casualty policy, rating, claims, billing, reinsurance, and customer data.
Plan and lead phased migration from Guidewire (PolicyCenter, ClaimCenter, BillingCenter) and legacy systems into Azure Databricks, including historical loads, incremental/CDC loads, reconciliation, and support for decommissioning.
Model core property and casualty insurance entities including accounts, policies, insured risks and exposures, coverages, endorsements, premiums, claims, reserves, payments, and recoveries.
Provide data for Actuarial and Finance use cases such as written/earned/unearned premium, loss triangles, IBNR inputs, loss ratios, and statutory reporting including Schedule P and NAIC.
Integrate third-party data sources including geospatial and weather data.
Define Unity Catalog architecture (metastore, catalogs, schemas, and environments for development, test, and production), naming conventions, permissions, row/column-level security, and lineage.
Manage pipeline orchestration and operations in Databricks: schedule and dependencies using Databricks Workflows (Lakeflow Jobs), set up retries, alerting and SLA monitoring, and establish failure handling and runbooks.
Own performance and cost optimization across Spark workloads, including tuning, partitioning and liquid clustering, Photon, and cluster and SQL warehouse sizing.
Work with IT and Platform Engineering on storage, networking, security, identity, and IaC, including CI/CD.
Set engineering standards for pipelines, testing, code review, and documentation; lead, mentor, and review the work of Data Engineers and junior contractors.
Partner with the Data Governance Lead to implement access controls, PII masking, lineage, and data quality rules.
Perform other duties as assigned.

Qualifications

Associate degree or an equivalent combination of education and experience.
10+ years of data engineering experience, including 6+ years hands-on with Databricks (PySpark, Spark SQL, Delta Lake), plus experience leading or mentoring engineering teams.
Certification as a Databricks Data Engineer Professional. Azure data or solutions architect certification preferred.
Knowledge of Unity Catalog: governance model, permissions, row/column-level security, lineage, and system tables.
Experience leading large migrations from core insurance and legacy systems into a lakehouse or data warehouse, including property and casualty insurance data and understanding of the policy and claims lifecycle and core P&C KPIs.
Data modeling knowledge including dimensional modeling, data vault, SCD patterns, and insurance data models.
Technical depth in Spark internals and performance tuning, streaming (structured streaming, auto loader), CDC patterns, MERGE, and Lakeflow/Delta Live Tables.
Expert-level SQL and strong Python skills, plus software engineering practices such as testing, modular design, and code review.
Knowledge of data infrastructure: ADLS Gen2, Azure Data Factory, Event Hubs, Key Vault, Entra ID, VNets, and Private Link.
Experience scheduling and monitoring production pipelines with Databricks Workflows and/or Azure Data Factory, including dependencies, parameterization, alerting, and monitoring with Databricks system tables.
Experience with Git-based version control (GitHub or Bitbucket), branching strategies, pull request reviews, and Databricks Git folders.
Infrastructure-as-code and CI/CD for data: Terraform, Databricks Asset Bundles, and tools such as Azure DevOps, GitHub Actions, or Bitbucket Pipelines.
Guidewire experience (Guidewire DataHub/InfoCenter, Cloud Data Access) is preferred, including supporting actuarial reserving and pricing for P&C lines of business. API and integration tools such as MuleSoft preferred.
Knowledge of insurance data standards such as ACORD is preferred.

Compensation and benefits

Competitive Salary
Bonus Structure
Profit Sharing
Medical, Dental, Vision Insurance
Employer Paid Short Term Disability
Employer Paid Long Term Disability
Employer Paid Life Insurance
Voluntary Life Insurance
401K with Company Match
PTO

Equal opportunity

Builders Insurance Group is an equal opportunity employer. They welcome applicants from all walks of life and do not discriminate based on any protected status.

View full job page →

3d ago

Data Engineer

Builders Atlanta, GA

Atlanta, GA

3d ago

Full time Azure Data Engineer Azure Data Lake Big Data

Apply Now

Job Description

Design, build, and operate data pipelines that power Builders’ Azure Databricks lakehouse for property and casualty insurance.

Responsibilities

Develop pipelines for phased migration into the Azure Databricks lakehouse from Guidewire (PolicyCenter, ClaimCenter, BillingCenter) and legacy systems, including historical loads, incremental/CDC (Change Data Capture) loads, and reconciliation
Create bronze and silver layer transformations in PySpark and Spark SQL following the medallion architecture
Support data modeling for property and casualty domains: policies, insured risks and exposures, coverages, endorsements, premiums, claims, reserves, payments, and recoveries
Integrate third-party datasets, including geospatial and weather data
Schedule and run production pipelines using Databricks Workflows (Lakeflow Jobs) or Azure Data Factory, including job dependencies, parameters, retries, and alerting
Investigate and resolve pipeline failures in production
Apply Unity Catalog controls set by the Data Engineering Lead and Data Governance lead, including permissions, PII masking, tags, and lineage
Implement data quality checks, error handling, and monitoring
Tune pipeline performance and cost using partitioning, liquid clustering, and right-sized job compute
Use Git version control (GitHub or Bitbucket) with CI/CD via Databricks Asset Bundles to version, review, and deploy code across development, test, and production
Write documentation and runbooks; participate in code reviews; take long-term ownership of contractor-built pipelines
Perform other duties as assigned

Requirements

Associate degree or an equivalent combination of education and experience
Certification in Databricks engineering
3+ years in data engineering, including 1+ year hands-on with Databricks (PySpark, Spark SQL, Delta Lake)
Property and casualty insurance experience, ideally Guidewire data; experience migrating data from legacy source systems
Knowledge of Python/PySpark and SQL (Structured Query Language)
Knowledge of Unity Catalog: permissions, row/column-level security, and lineage
Experience with incremental loading patterns such as CDC, MERGE, SCD (Slowly Changing Dimension), and Auto Loader
Knowledge of data modeling, including dimensional modeling
Experience scheduling and monitoring pipelines with Databricks Workflows and/or Azure Data Factory
Experience with Git-based version control (GitHub or Bitbucket), including branching, pull request reviews, and Databricks Git folders
Experience with Azure data services such as ADLS (Azure Data Lake Storage) Gen2, Azure Data Factory, and Key Vault
CI/CD fundamentals: Databricks Asset Bundles, Azure DevOps, GitHub Actions, or Bitbucket Pipelines
Experience with streaming (Structured Streaming, Event Hubs)
Experience with API (Application Programming Interface) and integration tools such as MuleSoft is an added advantage

Technologies

Azure Databricks, Delta Lake, Unity Catalog
Lakeflow, Databricks SQL, PySpark, Spark SQL
Databricks Workflows, Azure Data Factory
Git, GitHub, Bitbucket, Databricks Asset Bundles
Python, SQL (Structured Query Language)
ADLS (Azure Data Lake Storage) Gen2, Key Vault
Structured Streaming, Event Hubs, Auto Loader
MERGE, SCD (Slowly Changing Dimension), CDC (Change Data Capture)
MuleSoft, API (Application Programming Interface)
Azure DevOps, GitHub Actions, Bitbucket Pipelines

Benefits

Competitive Salary
Bonus Structure
Profit Sharing
Medical, Dental, Vision Insurance
Employer Paid Short Term Disability
Employer Paid Long Term Disability
Employer Paid Life Insurance
Voluntary Life Insurance
401K with Company Match
PTO

Position Summary

Design, develop, and maintain data pipelines for Builders’ Azure Databricks lakehouse supporting underwriting, claims, actuarial, and finance
Build reliable, high-quality, well-tested, and well-documented data solutions using Delta Lake, Unity Catalog, Lakeflow pipelines, and Databricks SQL
Deliver phased migration pipelines from Guidewire and legacy policy, claims, and billing systems into Azure Databricks
Align solutions with Unity Catalog governance and engineering standards led by the Data Engineering Lead
Strengthen long-term sustainability by progressively assuming responsibility for contractor-developed pipelines

About Builders

Builders is a mid-sized mutual with an A rating by AM Best
Recognized for reliability and fairness in fulfilling promises to customers
Teams foster a collaborative, customer-centric environment; “Culture Excellence” honors from Top Workplaces
Supports policyholders through partnerships with knowledgeable independent agencies

Equal Opportunity: Builders is an equal opportunity employer and does not discriminate based on any protected status.

Location: Atlanta, GA (onsite) • Experience: 3+ years • Education: Associate degree

View full job page →

11d ago

Sr. AI & Data Engineer

Accenture Irving, TX

Irving, TX

11d ago

$59k - $206k/yr Full time Senior Ai Engineer Artificial Intelligence

Apply Now

Job Description

Accenture is hiring a Sr. AI & Data Engineer for a hybrid role based in Irving, TX. In this position, you will work hands-on across Azure data services and Azure Databricks to architect and optimize scalable pipelines and agent-driven data experiences.

This role involves designing governed data platforms, enabling advanced analytics and AI use cases, and maintaining reliable pipeline performance. The hybrid schedule requires three days per week in the office within the specified hiring locations.

What You’ll Do

Architect, develop, and maintain scalable, low-latency data pipelines to ingest and transform diverse datasets using Azure Databricks workflows and Spark technologies.
Design and optimize Lakehouse architecture with Delta Lake tables following Bronze, Silver, and Gold layers for efficient, reliable, governed storage.
Collaborate with data scientists and business analysts to deliver clean, validated, and well-modeled datasets for advanced analytics and AI initiatives.
Design and implement autonomous agents using the Databricks Mosaic AI Agent Framework, including frameworks such as LangChain or AutoGen.
Build and optimize vector search indexes with Databricks Vector Search to support accurate context retrieval.
Process large-scale distributed datasets on the Databricks Intelligence Platform using Apache Spark.
Design and implement data models aligned to business requirements and improving overall data usability.
Continuously evaluate and improve data architecture processes to enhance efficiency and effectiveness.
Develop AI agents with retrieval, orchestration, tool invocation, evaluation harnesses, and lifecycle observability.
Monitor pipeline health and performance, and implement automated alerting and observability frameworks to enable faster troubleshooting and proactive optimization.

Minimum Qualifications

3+ years of professional experience as a Data & AI Engineer or in a related role.
3+ years of experience with Microsoft Azure cloud data services.
2+ years of experience with Databricks including PySpark, SQL, Delta Lake, and workflows.
Bachelor’s degree or equivalent (minimum 12 years work experience). If an Associate Degree is held, minimum 6 years work experience is required.

Technologies

Databricks, Azure data services, Azure Databricks, Spark, Apache Spark
Databricks Mosaic AI Agent Framework, LangChain, AutoGen
Delta Lake, Bronze, Silver, Gold layers (Lakehouse Medallion Architecture)
Databricks Vector Search
Databricks Intelligence Platform
PySpark, SQL
Git-based version control, CI/CD pipelines, Azure DevOps, GitHub Actions
Auto Loader
MLOps / LLMOps, Genie product family
Databricks Certified Data Engineer Professional (2025)
Microsoft Certified: Azure Data Engineer Associate

Benefits

Medical, dental, vision, life, and long-term disability coverage
401(k) plan
Bonus opportunities
Paid holidays
Paid time off

Preferred

Advanced Python (PySpark) and SQL for data transformation and pipeline development.
Experience with Azure Databricks including Spark, Delta Lake, Auto Loader, and Lakehouse Medallion Architecture (Bronze, Silver, Gold).
Experience with Git-based version control and CI/CD pipelines using Azure DevOps or GitHub Actions.
Experience in agent development using Mosaic AI Agent Framework.
Strong problem-solving and performance optimization skills.
Experience in MLOps / LLMOps including CI/CD for ML and LLM lifecycle.
Experience with Genie product family.
Databricks Certified Data Engineer Professional (2025).
Microsoft Certified: Azure Data Engineer Associate.

Compensation: USD 59,100 - 205,800 per year.

View full job page →

11d ago

Sr. Azure Data Engineer with Databricks

VeeRteq Solutions Inc. Philadelphia, PA

Philadelphia, PA

11d ago

Full time Azure Azure Analytics Azure Data Engineer

Apply Now

Job Description

VeeRteq Solutions Inc. is hiring a Sr. Azure Data Engineer with Databricks for a hybrid position in Philadelphia, PA. This role supports enterprise data modernization and analytics initiatives by delivering scalable cloud data pipelines and Lakehouse-based architectures on Azure and Databricks. You will help ensure data is reliable, governed, and ready for business intelligence, reporting, and advanced analytics.

What You’ll Do

Design, develop, and maintain scalable ETL/ELT pipelines for enterprise data and analytics initiatives.
Build data ingestion, transformation, and orchestration frameworks using Azure-native technologies.
Develop and optimize data processing solutions using Databricks and PySpark.
Implement and maintain Delta Lake and Lakehouse architectures to support modern data platforms.
Ensure data quality, accuracy, lineage, and reliability across all data assets.
Conduct root cause analysis and troubleshoot data-related issues.
Develop and integrate solutions leveraging: Azure Databricks, Azure Data Lake Storage (ADLS Gen2), Azure Data Factory (ADF), Azure Synapse Analytics, and Azure SQL Database.
Support data migration, modernization, and cloud transformation efforts.
Optimize performance, scalability, and cost efficiency for cloud-based data platforms.
Build and maintain logical and physical data models and create curated data layers for BI, reporting, and advanced analytics workloads.
Partner with business users and data consumers to translate requirements into scalable technical solutions.
Implement CI/CD for data engineering solutions, including automated deployment and release management processes.
Develop monitoring, logging, and observability capabilities for data pipelines.
Follow enterprise data governance and security standards.
Participate in Agile/Scrum ceremonies such as sprint planning, daily stand-ups, retrospectives, and backlog refinement.
Collaborate with Product Owners, Architects, QA teams, and business stakeholders; contribute to continuous improvement and engineering best practices.

What You Bring

Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field.
7+ years of experience in Data Engineering.
5+ years of hands-on experience with Azure data technologies.
2+ years of experience in Healthcare, Managed Care, Health Plans, or Payer organizations.
Strong expertise in: Azure Databricks, Azure Data Lake Storage (ADLS Gen2), Azure Data Factory (ADF), PySpark, Python, SQL, Delta Lake, and Data Modeling.
Extensive experience building enterprise-scale data pipelines and data integration solutions.
Strong understanding of Data Warehousing, Data Lake, and Lakehouse concepts.
Experience with large volumes of structured and unstructured data.
Experience with Git, CI/CD, and DevOps practices.
Strong analytical, troubleshooting, and problem-solving skills.

Additional Preferences

Experience with Claims Data, Membership Data, Provider Data, or Care Management Data.
Experience with Power BI or other reporting tools.
Familiarity with FHIR, HL7, or healthcare interoperability standards.
Experience with Azure DevOps and Infrastructure as Code (IaC).
Exposure to AI/ML data pipelines and advanced analytics environments.

View full job page →

12d ago

Principal Data & Analytics Engineer

SynergenX Health Holdings LLC Houston, TX

Houston, TX

12d ago

$150k+/yr Full time Senior Business Analytics Business Intelligence

Apply Now

Job Description

SynergenX Health is hiring a senior, hands-on Principal Data & Analytics Engineer to build end-to-end production data solutions using Azure and Databricks in a Houston, TX (hybrid) role.

Responsibilities

Own complex data solutions end to end: discovery, requirements gathering, architecture, development, testing, deployment, documentation, monitoring, and production support.
Build production data pipelines and analytical products using SQL, Python/PySpark, Azure Data Factory, and Databricks.
Create ETL/ELT pipelines, Databricks workflows, curated datasets, dimensional models, semantic models, and downstream integrations.
Translate business processes into reliable data logic by partnering with Operations, Finance, Pharmacy, Clinical Operations, Marketing, and other stakeholders.
Investigate difficult data discrepancies by tracing unexpected metric changes across source systems, pipelines, transformations, and reporting layers to identify root cause and implement durable fixes.
Assess downstream impact of changes across Power BI reports, KPIs, financial calculations, operational workflows, and dependent systems.
Build quality and monitoring into every solution with automated validation, reconciliation, anomaly detection, monitoring, and alerting for pipeline failures, data drift, and unexpected historical changes.
Modernize existing data solutions by refactoring legacy pipelines, removing duplicated logic, improving performance, and migrating workloads to standardized Azure and Databricks patterns.
Strengthen the analytics layer by developing and optimizing datasets and semantic models that support consistent metrics and trustworthy historical reporting.
Improve engineering practices by helping establish standards for Git, CI/CD, testing, documentation, reusable development patterns, data lineage, and deployment.
Act as a senior technical partner: mentor Data Engineers, BI developers, and analysts; lead code and design reviews; help resolve challenging technical issues.
Integrate enterprise source systems by evaluating databases, APIs, SaaS platforms, EHR and pharmacy systems, financial applications, and other sources to determine ingestion strategy.
Support Data & AI initiatives by building dependable data foundations and integrations for automation, AI/ML, and advanced analytics.

Requirements

7+ years of progressive experience in data engineering, analytics engineering, BI engineering, or a related discipline.
Proven ability to independently deliver complex data solutions from initial requirements through production.
Advanced SQL skills, including developing, optimizing, and troubleshooting complex data transformations.
Strong Python and/or PySpark development experience.
Strong hands-on experience with Databricks and Apache Spark.
Strong Microsoft Azure experience, especially Azure Data Factory and Azure data services.
Strong data modeling skills including dimensional models, star schemas, and analytical data models.
Experience building and supporting production ETL/ELT pipelines.
Experience implementing automated data quality validation, monitoring, and production alerting.
Working knowledge of Power BI architecture, semantic models, and DAX to evaluate downstream analytical impacts.
Experience with Git, CI/CD, and modern software and data engineering practices.
Ability to troubleshoot complex data discrepancies across multiple systems.
Ability to investigate source systems, understand business processes, and deliver solutions when requirements are initially ambiguous.
Ability to work directly with nontechnical stakeholders and translate business needs into technical solutions.
Strong written documentation and verbal communication skills.

Technologies

Azure Data Factory
Azure Databricks
Apache Spark
SQL
Python
PySpark
Databricks Workflows
ETL/ELT Pipelines
Microsoft Power BI
DAX
Dimensional Modeling
Star Schemas
Semantic Models
Delta Lake
Lakehouse Architecture
Microsoft Azure Data Services
Microsoft Fabric
Unity Catalog
Git
CI/CD
Azure DevOps
APIs
EHR

Benefits

Medical, Dental, and Vision Insurance
401(k) with Company Match
Paid Time Off and Paid Holidays
Company-Paid Life Insurance
Short- and Long-Term Disability
Employee Assistance Program
Professional development opportunities

Preferred Qualifications

Healthcare, pharmacy, EHR, or other regulated-data experience.
Experience working with HIPAA-regulated data.
Microsoft Fabric experience.
Experience with Databricks Lakehouse, Delta Lake, and Unity Catalog.
API integration experience.
Experience with Azure DevOps.
Experience modernizing legacy data warehouses or pipelines.
Experience integrating operational and transactional systems with analytics platforms.
Databricks Data Engineer certification.
Microsoft Fabric or Power BI certification.
Experience supporting AI/ML or automation workloads.

Work Environment

Hybrid role based in Houston, Texas (2–3 days per week onsite) for collaboration with the Data & AI team and business stakeholders.
Candidates must be able to reliably commute to the Houston office and meet onsite requirements.
As a senior individual contributor, you will remain actively involved in production development while providing technical guidance, mentoring teammates, and shaping solution design.

Employment Requirements

Must be legally authorized to work in the United States.
SynergenX Health is unable to provide current or future employment visa sponsorship for this position.
Successful completion of all pre-employment screening requirements.