Pyspark

Pyspark Engineer Jobs

152 jobs found

Today

Sr Analytics Engineer, Retail Data Products

Apple Austin, TX

Austin, TX

Today

Full time Python Analytics Apache Airflow

Apply Now

Job Description

Apple’s Retail Store Analytics team is looking for a Sr Analytics Engineer in Austin, TX to help build scalable dashboards and data products that support data-driven decisions across Retail Operations. In this role, you will connect data and AI to improve how store teams access insights, including AI-driven conversational data tools.

You will work closely with Product Management, Data Science, Engineering, and Operations to plan and deliver an extensive reporting roadmap. The focus is on creating decision support that is dynamic, intuitive, and usable by teams across Apple Retail.

What You’ll Do

Create tools that enable data-driven decision-making across Apple Retail Operations.
Partner cross-functionally to build and refine scalable dashboards and data products that delight users.
Develop a centralized hub of canonical dashboards and AI-driven conversational data tools for decision support.
Help shape the future direction of Retail Operations by integrating data and AI to delight customers and empower store employees.
Manage delivery of an extensive reporting roadmap as part of a team.

Core Qualifications

6+ years experience building analytics pipelines in a production development environment for modeling, analysis, and reporting.
6+ years coding in SQL or PySpark and Python.
6+ years building and maintaining dashboards in Tableau, PowerBI, or Looker with hundreds of users.
Ability to initiate, refine, and complete projects with minimal guidance.
Ability to think critically and collaborate cross-functionally to distill abstract requirements into clear data products for data engineering, data science, and analytics stakeholders.
Bachelor’s degree in a relevant field (Engineering, Data Science, Business) or equivalent experience.

Tools and Technologies

SQL, PySpark, Python
Tableau, PowerBI, Looker
Snowflake SQL, Airflow, dbt, Cube
LookML, Tableau viz extensions

Preferred Qualifications

Experience building analytics pipelines in production development environments, ideally with Snowflake SQL or PySpark, and tools like Airflow and dbt.
3+ years managing semantic layers in tools like Cube, dbt, Snowflake, or LookML.
3+ years developing reusable data visualization assets like Tableau viz extensions.
3+ years experience working with retail data.
Master’s degree in a relevant field (Engineering, Data Science, Business) or equivalent experience.

Location: Austin, TX (onsite)

View full job page →

1d ago

Senior Data Engineer

Kentro Fayetteville, NC

Fayetteville, NC

1d ago

Full time Senior Apache Airflow Big Data

Apply Now

Job Description

Kentro is seeking a Senior Data Engineer to support USASOC EDAT missions from Fayetteville, North Carolina in an onsite engineering environment. In this role, you will build and operate data pipelines and data products that connect mission and enterprise information into a common data environment, enabling reliable, governed datasets for analysts and decision-makers.

What you’ll do

Design, build, test, and maintain scalable batch and streaming ETL/ELT pipelines for classified and air-gapped environments.
Develop transformation logic using Python, SQL, and PySpark to ingest, clean, normalize, and enrich structured and unstructured data.
Integrate data from mission systems, operational applications, databases, APIs, and file-based sources into enterprise data platforms.
Create data models, schemas, and entity relationships that support analytics, reporting, and downstream applications.
Build and publish curated datasets and data products with clear documentation, versioning, and service-level expectations.
Implement data quality checks, validation, monitoring, and alerting to improve pipeline reliability and trust in outputs.
Establish and maintain data lineage, metadata, and cataloging aligned with data governance policies.
Apply security controls including role-based and attribute-based access, data tagging and labeling, and need-to-know enforcement.
Automate deployment and operations using CI/CD, version control, and infrastructure-as-code practices.
Troubleshoot production issues, conduct root cause analysis, and optimize pipeline performance and cost.
Support deployment of analytics and ML/AI workflows into operational systems.
Collaborate with mission owners, analysts, data scientists, security staff, and system administrators to translate requirements into engineered solutions.

Qualifications

Bachelor’s degree in computer science, data science, information systems, or a related technical field, plus 7 years of relevant experience.
8140 compliances in certification or training may substitute for a degree.
Meets DoD 8140 foundational and residential qualification requirements for the assigned work role, including approved certification, education, or training. Training options include SecurityX / CASP+, CCSP, Cloud+, CSC, GCLD, GCSA, GSEC.
7+ years of professional data engineering experience.
Strong Python and SQL development skills demonstrated through production pipelines or applications.
Experience with distributed processing such as PySpark / Apache Spark.
Experience with relational and non-relational data stores, plus data modeling and schema design.
Experience integrating multiple source systems into a centralized data platform.
Proficiency in Linux, Git, and CI/CD workflows.
Working knowledge of data governance, data lineage, and metadata management.
Active TS/SCI clearance and US citizenship.

Technologies you may work with

Python, SQL, PySpark, Apache Spark
ETL/ELT, CI/CD, infrastructure-as-code
Git, MSS (Microsoft SQL Server), T-SQL
Linux, Airflow, NiFi, Kafka
Elasticsearch/OpenSearch, graph databases, geospatial data
Docker, Kubernetes, Palantir Foundry, Databricks
JWICS, SIPRNet

Benefits

Competitive benefits package including paid time off
Healthcare benefits
Supplemental benefits
401k including an employer match
Discount perks and rewards
Education reimbursement for certifications, degrees, or professional development
Funds for activities (virtual and in-person), including happy hours, holiday events, fitness & wellness events, and annual celebrations
Charity galas/events

Preferred qualifications

Hands-on experience with Microsoft SQL Server (MSS), including T-SQL development and performance tuning
Prior work for USASOC, USSOCOM, or another special operations or Army component
Experience building and operating data platforms in classified or air-gapped environments (JWICS, SIPRNet)
Experience with a modern data platform such as Palantir Foundry, Databricks, or comparable (pipelines, ontology/object modeling, application building)
Experience with pipeline orchestration and streaming tools (e.g., Airflow, NiFi, Kafka)
Experience with Elasticsearch/OpenSearch, graph databases, or geospatial data
Experience with containers and orchestration (Docker, Kubernetes) and infrastructure as code
Experience deploying ML/AI models or analytics into operational systems
Familiarity with RMF, ATO processes, and secure software development practices
Cloud experience in classified or government cloud environments

Clearance requirement

Active TS/SCI clearance
US Citizen

View full job page →

1d ago

Fabric and Data Engineer

East West Bank Pasadena, CA

Pasadena, CA

1d ago

$30 - $34/hr Full time Analytics Application Security Automation

Apply Now

Job Description

The role focuses on designing, implementing, and supporting enterprise-grade data solutions using Microsoft Fabric and Azure. The position emphasizes secure, scalable, automated, and highly reliable data platforms that support analytics, reporting, and AI initiatives.

Responsibilities

Design, deploy, and support Microsoft Fabric environments, including workspaces and solutions.
Build and maintain Lakehouses, Warehouses, data pipelines, notebooks, and data transformation processes.
Develop scalable ETL/ELT solutions using SQL, Python, PySpark, and Spark.
Integrate Fabric with Azure services, databases, APIs, SaaS applications, and on-premises data sources.
Implement security, RBAC, data governance, lineage, and compliance controls.
Set up Git, CI/CD, and automated deployment practices across development, test, and production environments.
Monitor platform health, capacity, performance, pipelines, and production workloads.
Troubleshoot incidents, perform root-cause analysis, and optimize performance.
Create technical standards, documentation, operational procedures, and runbooks.
Coordinate with Cloud, Infrastructure, Security, Data, Application Development, and business stakeholders.

Requirements

5+ years of experience in data, cloud, platform, or database engineering.
Hands-on experience with Microsoft Fabric and Azure data services.
Strong SQL skills and experience with Python and/or PySpark.
Experience building ETL/ELT pipelines, data lakes, warehouses, and Lakehouse architectures.
Experience with Git, CI/CD, automation, and production support practices.
Working understanding of cloud security, identity and RBAC, data governance, and monitoring.
Strong troubleshooting, documentation, and communication skills.
Legal authorization to work in the United States. No visa sponsorship is offered.

Technologies

Microsoft Fabric, Azure
SQL, Python, PySpark, Spark
Lakehouses, Warehouses, notebooks
ETL/ELT, Git, CI/CD
RBAC, data governance, lineage
Azure Data Factory, OneLake
Fabric Lakehouse, Fabric Warehouse
Power BI, Delta Lake, Terraform

Preferred

Microsoft Fabric production experience and related Microsoft certifications (DP-600, DP-700).
Experience with OneLake, Fabric Lakehouse and Fabric Warehouse, Data Factory, Spark, Power BI, and Delta Lake.
Experience using Infrastructure-as-Code tools such as Terraform.
Experience supporting data platforms in financial services or another highly regulated environment.

Compensation

The base pay range for this position is USD $30.00/Hr. - USD $34.00/Hr.

Location

Pasadena, CA (onsite)

View full job page →

2d ago

Data Engineer

D2 Technologies Herndon, VA

Herndon, VA

2d ago

$100k - $150k/yr Full time Big Data Bigdata Cloud Data Engineering

Apply Now

Job Description

D2 Technologies is building enterprise-scale data capabilities and is seeking an experienced Data Engineer to design, develop, and optimize data solutions on the Databricks Lakehouse Platform. This hybrid role in Herndon, VA supports large-volume data processing through scalable ETL/ELT pipelines and performance-focused analytics engineering using Apache Spark and SQL.

What You’ll Do

Design, develop, and maintain scalable data engineering solutions using Databricks and Apache Spark
Build and optimize high-performance ETL/ELT pipelines for both structured and unstructured data
Develop, tune, and optimize complex SQL queries for data transformation and analytics
Translate business requirements into scalable technical solutions
Troubleshoot production issues and optimize data pipeline performance
Ensure data quality, integrity, security, and governance across enterprise data platforms
Collaborate with data architects, software engineers, analysts, and business stakeholders
Participate in code reviews and follow software engineering best practices
Create and maintain technical documentation for solutions, deployments, and operational support
Support production releases and provide operational support as needed

What You Bring

U.S. Citizenship is required
Top Secret (TS) Security Clearance or Public Trust Clearance preferred
Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent experience)
Hands-on experience developing and implementing data engineering solutions using Databricks
Strong, hands-on experience with the Databricks Lakehouse Platform
Strong programming experience with Apache Spark (PySpark/Spark SQL)
Expert-level SQL skills, including writing, optimizing, and troubleshooting complex queries
Strong analytical, critical thinking, and problem-solving skills
Experience designing, developing, and supporting ETL/ELT pipelines
Strong understanding of data modeling, data warehousing, and Delta Lake
Experience using Git or other version control systems
Excellent verbal and written communication skills

Additional Preferred Qualifications

Experience with Azure, AWS, or Google Cloud Platform
Knowledge of CI/CD pipelines and DevOps practices
Experience with Databricks Delta Live Tables (DLT) and Unity Catalog
Nice to have: PL/SQL

Technologies: Databricks, Databricks Lakehouse Platform, Apache Spark, PySpark, Spark SQL, SQL, ETL, ELT, Delta Lake, Git, Azure, AWS, Google Cloud Platform, Databricks Delta Live Tables (DLT), Unity Catalog, PL/SQL, CI/CD

Location: Hybrid remote in Herndon, VA 20192
Salary: $100,000 - $150,000 per year (negotiable)

Application screening: Federal Project requires US Citizenship. Are you US Citizen?

View full job page →

3d ago

Data Engineer, ADAS Fleet Analytics

Mercedes-Benz Research & Development North America, Inc. San Jose, CA

San Jose, CA

3d ago

$118k - $148k/yr Full time Adls Gen2 Application Security Artificial Intelligence

Apply Now

Job Description

Build telemetry data pipelines that translate vehicle signals into trusted KPIs, fleet dashboards, and ADAS campaign-health monitoring for engineering and management teams. On the US ADAS Data & Forensics team at Mercedes-Benz Research & Development North America, you will help shape ML-ready infrastructure for ADAS event scoring, operating at large scale on the Azure data platform with production ownership across ingestion, processing, and delivery.

Work onsite in San Jose, CA with a role focused on real-time ingestion, Bronze/Silver/Gold processing with Delta Lake, and reliable operations through CI/CD and incident response practices.

Why you will like working here

Medical, dental, and vision insurance for employees and their families
401(k) with employer match
Up to 15 company-paid holidays
Paid time off (flexible time off for salaried employees), sick time, and parental leave
Tuition assistance program
Wellness/Fitness reimbursement programs

Responsibilities

Design and maintain scalable telemetry pipelines using Event Hub, Medallion architecture, Delta Lake MERGE, schema evolution, and blue/green data deployments.
Own campaign and fleet management including enrollment rosters, ingestion-completeness monitoring, VIN reconciliation, and automated coverage alerts.
Develop fleet KPIs for engineering, operations, and management, including safety event and takeover analysis, data freshness and pipeline health, and trends and cohort comparisons.
Build ML-ready data infrastructure for SSR recordings, including video and signal pipelines, embedding stores, feature-engineering layers, and model-output integration into Gold.
Operate the Azure data platform with ADLS Gen2, Synapse Spark, Event Hub, and Container Apps; support infrastructure managed with Terraform and CI/CD.
Write automated tests, maintain documentation alongside code, participate in code reviews, and support incident response.

Requirements

Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field. Equivalent experience may be considered.
3+ years of professional experience.
2-5 years of experience in data engineering or big-data analytics with production pipeline ownership.
Strong Python, PySpark DataFrame API, and performance-tuning skills, plus SQL including window functions and aggregations.
Experience with a columnar or transactional data-lake technology such as Delta Lake, Iceberg, Hudi, or managed Parquet.
Experience with automated testing for data pipelines and Git-based development workflows.
Ability to diagnose production failures including schema conflicts, data skew, and memory issues, and communicate trade-offs to technical and non-technical stakeholders.

Strongly Preferred: Cloud data-platform experience in Azure, AWS, or GCP.

Nice to Have: ML data infrastructure, including feature stores, embedding pipelines, vector databases, or video pipelines.

Tech stack

Event Hub, Delta Lake (including Delta Lake MERGE)
PySpark, Python, SQL
ADLS Gen2, Synapse Spark, Container Apps
Terraform, CI/CD
Iceberg, Hudi, Parquet, Git
FastAPI, DuckDB, PyArrow, vector databases

Additional details

Employment type: Vollzeit (Full-time)
Start date: sofort
Compensation: USD 118,200 - 147,700 per year (range may be modified in the future)
Position number: MER00048Z4
Relocation assistance: not provided for this position
Export control: may involve access to export-controlled technology/software; Mercedes-Benz Research & Development North America may be required to obtain an export license or authorization under US law
Benefits note: Internships & Contractors excluded from Full-Time Employee benefits

View full job page →

3d ago

Lead Data Engineer

Nestlé Purina St. Louis, MO

St. Louis, MO

3d ago

$138k - $188k/yr Full time Azure Data Engineer Big Data Bigdata

Apply Now

Job Description

Nestlé Purina is seeking a Lead Data Engineer to provide technical leadership for a data product team tied to a specific data domain. In this onsite role in St. Louis, MO, you will help deliver production-grade data products by building and maintaining resilient data pipelines, aligning them to enterprise standards, and ensuring they are ready for end users and downstream applications.

Role Summary

This position serves as the technical lead for a data product team focused on a defined data domain. You will develop and maintain monitored, reliable pipelines that move, transform, and harmonize source data into data products. You will also support the integration of those data products into Purina’s enterprise data model so they can generate business value.

Key Responsibilities

Lead a data product team for a specific data domain
Develop, test, deploy, and maintain data pipelines for moving, transforming, and harmonizing source data into consumable data products
Ensure work aligns with Purina data engineering and architectural standards
Monitor pipeline performance for pipelines owned within the assigned domain
Drive adoption of data engineering best practices within the data engineering guild
Coach and mentor other data engineers
Collaborate with engineers and architects to mature the data enterprise data platform
Work with product owners and business stakeholders to shape long-term data product roadmaps
Partner with stakeholders and the data product team to define data product requirements

Required Qualifications

Bachelor’s degree from an accredited college or university
7+ years of professional experience, with 5+ years in practical, hands-on data engineering
4+ years of experience in the Azure ecosystem
4+ years of experience with Databricks and Python/PySpark

Technologies

Azure ecosystem
Databricks
Python
PySpark

Compensation and Benefits

The salary range for this position is USD 137,758 to 188,424 per year (listed as $137,758.50 to $188,424.00). Final compensation may vary based on factors including knowledge, skills and abilities, as well as geographic location, and it is not typical for an individual to be hired at the top of the range.

Bonus eligibility: Yes
401k: Company match
Healthcare: Coverage provided

Additional incentives are performance-based, and Nestlé offers a competitive total rewards package. Incentives and/or benefit packages may vary depending on the position.

Location: St. Louis, MO (onsite)

View full job page →

3d ago

Senior Data Engineer

CGI Group, Inc. Portland, OR 🇺🇸 Remote

Portland, OR

3d ago

$63k - $139k/yr Full time Senior Azure Azure Data Factory

Apply Now

Job Description

CGI is hiring a Senior Data Engineer (remote within the USA) to lead data platform testing for enterprise ETL/ELT pipelines and ensure end-to-end data quality.

Responsibilities

Design, develop, and execute test plans, test scenarios, and test cases for enterprise data platforms, ETL/ELT pipelines, and data integration solutions
Run end-to-end ETL testing and validate data accuracy, completeness, transformation logic, and reconciliation across source and target systems
Create and execute complex SQL queries for backend testing and data validation
Test large scale data processing solutions using Databricks, Apache Spark, PySpark, Python, Hive, Hadoop/HDFS, and Azure Data Factory (ADF)
Perform functional, integration, regression, system, and data quality testing across data pipelines and applications
Validate batch processing, job scheduling, dependencies, and workflow execution using Control M and Unix/Linux environments
Identify, document, track, and retest defects by coordinating with developers, data engineers, business analysts, and other stakeholders through resolution
Develop and maintain reusable test scripts and automation solutions where applicable
Participate in Agile/Scrum activities including sprint planning, refinement, daily stand ups, reviews, and retrospectives

Requirements

8+ years of experience in software quality engineering/testing with strong experience in ETL testing, data validation, integration testing, and regression testing
Strong hands-on SQL skills, including complex query development for source to target validation, reconciliation, and data quality testing
Experience testing large scale data platforms and ETL/ELT pipelines
Hands-on experience with Databricks, Apache Spark, PySpark, Python, and Azure Data Factory (ADF)
Experience with the Hadoop ecosystem, including Hadoop/HDFS and Hive
Experience with Unix/Linux and enterprise job scheduling tools such as Control M
Strong understanding of data quality concepts, data transformations, data mappings, and source to target validation
Experience with defect management and test management processes within Agile/Scrum delivery environments
Strong analytical, troubleshooting, communication, and collaboration skills

Technologies

Databricks
Apache Spark
PySpark
Python
Hive
Hadoop/HDFS
Azure Data Factory (ADF)
SQL
Control M
Unix/Linux
Agile/Scrum

Benefits

Competitive compensation including profit participation program
Comprehensive medical, dental, and vision benefits
Basic life and accidental death & dismemberment insurance
401(k) matching contributions and CGI share purchase plan
Flexibility and paid accrued vacation leave, ranging from 10 to 20 days per year based on job level, years of relevant prior experience, and years of service
10 paid holidays per year
At least 80 consecutive hours of paid sick/safe leave (with exceptions where applicable state/local law requires more)
Paid parental leave, ranging from 20 to 70 consecutive business days based on circumstances of leave and applicable laws
Bereavement leave, ranging from 1 to 7 days per year based on relationship
Paid jury duty leave, up to time summoned
Learning opportunities and tuition assistance
Wellness and Well being programs