Lead Data Engineer - Enterprise Data & Analytics

Lead data engineering role responsible for designing, prototyping, and delivering production-grade data pipelines and platform capabilities, with technical leadership and mentoring within Mayo Clinic's Enterprise Data & Analytics.

Responsibilities

• Direct the design, prototyping, and implementation of data pipeline architectures and related pipelines.
• Drive internal process improvements by automating manual workflows and enhancing data delivery.
• Redesign infrastructure to scale efficiently for growing data workloads.
• Perform root-cause analysis on internal and external data processes to uncover improvements and inform decisions.
• Apply advanced analytics to unstructured datasets to derive actionable insights.
• Understand the overall architecture, collaborate effectively, and lead technical discussions with clear communication.
• Serve as a senior individual contributor within the data or software engineering teams.
• Participate in the Technical Review Board alongside the Manager and Principal Engineer as a technical advisor.
• Act as a technical liaison between managers, software engineers, and principal engineers.
• Collaborate with software engineers to analyze, design, develop, and validate functional requirements.
• Lead hands-on technical efforts to design, build, review, and optimize production-grade data pipelines, data products, and platform capabilities.
• Contribute substantially to production codebases while establishing engineering standards, mentoring peers, and delivering scalable, resilient solutions.
• Mentor and coach engineers to foster growth and technical excellence.
• Partner with team members to explore design approaches, prototype new technologies, and assess feasibility.
• Operate within an Agile/Safe/Scrum framework to deliver high-quality software.
• Define architectural principles, select design patterns, and guide their appropriate application across the team.
• Facilitate cross-functional communication among front-end, back-end, data, and platform engineers.
• Assume a formal engineering lead role within the area of expertise and stay current with industry trends.
• Keep abreast of developments in the field to inform strategy and implementation.

Requirements

• Bachelor's degree in Computer Science, Engineering, or a related field with 6 years of experience, or an Associate degree with 8 years of experience.
• Proficiency in professional software engineering practices across the full SDLC, including coding standards, code reviews, source control, build processes, testing, and operations.
• At least 5 years of hands-on experience in data engineering, data science, or analytical modeling, with broad knowledge of related disciplines.
• Experience leading Data Engineering teams in a Continuous Integration/Continuous Delivery environment.
• Ability to build and lead highly resilient data products and oversee test automation, unit testing, data quality, monitoring, and observability.
• Minimum 5 years of experience with relational and NoSQL databases.
• Experience with cloud platforms such as Google Cloud Platform, Microsoft Azure, and Amazon Web Services.
• Proficiency with CI tools like Jenkins, GitHub Actions, or Azure Pipelines.
• Familiarity with cloud-native development and deployment practices.
• Experience with Jira, GitHub, SharePoint, or Azure Boards for work tracking and collaboration.
• Hands-on experience with advanced data processing tools such as Apache Spark, Hive, Airflow, Kafka, and cloud dataflow services (eg, GCP Dataflow).
• Background in big data, statistics, and data aspects of machine learning.
• Experience with Google BigQuery, FHIR APIs, and Vertex AI.
• Knowledge of workflow scheduling solutions like Apache Airflow and Google Composer and their relation to data systems.
• Experience using Infrastructure as Code concepts with Kubernetes and Docker in a cloud environment.

Technologies

• Python
• SQL
• Apache Spark
• Apache Flink
• Ray
• Hive
• Airflow
• Kafka
• Google Cloud Platform (GCP)
• BigQuery
• Vertex AI
• FHIR APIs
• Delta Lake
• Apache Iceberg
• Apache Hudi
• Parquet
• Avro
• ORC
• Jenkins
• GitHub Actions
• Azure Pipelines
• Jira
• SharePoint
• Azure Boards
• Kubernetes
• Docker
• Google Dataflow
• Google Composer
• AWS

Benefits

• Medical plans with multiple options
• Dental coverage through Delta Dental or a reimbursement account option
• Vision plan with nationwide network
• Pre-tax savings options including HSA and FSAs
• Competitive retirement package

Why Mayo Clinic

Mayo Clinic is recognized across more specialties than any other care provider by U.S. News & World Report. The organization prioritizes patient needs while investing in employees through competitive compensation, comprehensive benefits, continuing education, and clear advancement opportunities to support long, successful careers.

Locations

Locations include three main campuses in Phoenix/Scottsdale, Arizona; Jacksonville, Florida; Rochester, Minnesota, plus Mayo Clinic Health System campuses across the Midwest and international sites. Each Mayo Clinic location offers an environment that supports both professional growth and personal well-being.

Equal Opportunity

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender identity, sexual orientation, national origin, protected veteran status, or disability status. Mayo Clinic participates in E-Verify and may request information from new employees to verify work authorization.

Recruiter

Laura Percival