Sr Data Engineer, Data Analytics & Intelligence, NA

Vantage Data Centers Management Company LLC is seeking a Sr Data Engineer, Data Analytics & Intelligence, NA to build, operate, and scale governed data foundations that support Operations across North America. In this role, you will design Azure-based data pipelines and curated datasets, including semantic-model inputs, enabling enterprise reporting, operational intelligence, and AI-enabled use cases.

This position will be based onsite in Denver, CO in alignment with the flexible work policy, with 3 days on site required and 2 days flexible. Salary for the role is USD 130,000 - 155,000 per year, based on Colorado market data and subject to variation based on qualifications and experience.

What you will do

• Design, build, and maintain reliable, scalable data pipelines using Python and PySpark on the Microsoft Azure data platform.
• Develop and operate batch and incremental pipelines using Azure Data Factory, with Azure Data Lake Storage Gen2 as the primary data store.
• Build and maintain curated lakehouse / gold-layer datasets and semantic-model inputs for governed operational insights and AI-enabled consumption.
• Implement SQL- and Spark-based transformations to produce curated datasets supporting enterprise reporting, analytics, and downstream AI-enabled insight preparation.
• Own assigned pipelines and datasets, including monitoring, troubleshooting, performance optimization, documentation, and production support.
• Work with Azure Synapse and Microsoft Fabric / Lakehouse patterns where applicable, along with related Azure analytics services.
• Prepare structured operational data for AI-enabled use cases by documenting business rules, source lineage, data reliability constraints, known quality limitations, and data dictionary definitions.
• Support source visibility and confidence context, including Data Reliability & Trust Indicator integration where applicable.
• Contribute to ontology, taxonomy, semantic model, and data dictionary alignment across operational context, KPIs, incidents, work orders, and other enterprise domains.
• Collaborate with business analysts, operations SMEs, data stewards, IT Global, and cross-functional stakeholders to translate requirements into working data solutions.
• Apply data governance, security, access control, data classification, and engineering standards to ensure scalable, compliant solutions.
• Identify, document, and route data-quality issues to accountable owners to improve source correction rather than hiding defects downstream.
• Participate in code reviews, technical discussions, sprint planning, and platform improvement initiatives.
• Develop and maintain PySpark notebooks and jobs for ingesting, transforming, validating, and curating data.
• Create and modify Azure Data Factory pipelines for batch and incremental ingestion.
• Implement Spark transformations to write curated outputs to Azure Data Lake Storage Gen2 and/or Fabric Lakehouse patterns using established structures, naming conventions, and governance.
• Create and maintain SQL views, tables, lakehouse objects, and semantic-model inputs for analytics and operational intelligence.
• Prepare datasets for Fabric Data Agent / AI agent use cases by documenting business rules, joins, grain, quality limitations, source lineage, and operational definitions.
• Respond to pipeline failures, data validation issues, operational alerts, and data-quality escalations with root-cause analysis and remediation steps.
• Perform performance tuning of Spark jobs and SQL workloads (partitioning, filtering, incremental logic, query optimization, and resource-aware design).
• Validate data outputs with business partners and operations SMEs, and address discrepancies through documented correction paths.
• Support observability, logging, and auditability practices for data pipelines and AI-consumable datasets where applicable.
• Commit code using Git, follow branching standards, participate in pull request reviews, and support CI/CD with GitHub, Azure DevOps, or similar tools.
• Maintain documentation for pipelines, datasets, data contracts, data dictionaries, business rules, and operational runbooks as changes are made.
• Execute sprint backlog items, raising risks, dependencies, or blockers early.
• Complete additional duties as assigned by management.

Core requirements

• Bachelor’s degree in Engineering, Computer Science, Data Analytics, or a related field, or equivalent experience.
• Minimum 5–8 years of experience in data engineering, analytics engineering, or a closely related technical data role.
• Proficiency in Python and PySpark for data pipelines and transformations.
• Proficiency in SQL for querying, transformations, model validation, and data quality checks.
• Solid understanding of ETL/ELT, transformation patterns, data integration, incremental processing, and production support.
• Experience analyzing enterprise data sources to identify relationships, transformations, business rules, grain, ownership, and quality constraints.
• Experience building on Microsoft Azure, including Azure Data Factory, Azure Synapse, Azure Data Lake Storage Gen2, and Microsoft Fabric / Lakehouse patterns.
• Working knowledge of data modeling fundamentals, including fact/dimension tables and semantic models.
• Experience supporting governed data products, including metadata, lineage, issue documentation, access-control awareness, validation, and operational runbooks.
• Experience with source control and CI/CD using GitHub or Azure DevOps.
• Strong communication skills and the ability to collaborate with IT, business SMEs, and data governance partners in a fast-paced environment.
• Experience working in Agile environments and using tools such as Jira or similar.
• Travel is expected up to 10%, and may increase over time.

Technologies

• Python, PySpark, Microsoft Azure
• Azure Data Factory, Azure Data Lake Storage Gen2, Azure Synapse
• Microsoft Fabric, Fabric Lakehouse, SQL, Spark
• Git, GitHub, Azure DevOps, Jira, CI/CD
• Microsoft Fabric / Lakehouse patterns

Benefits

• Medical, dental, and vision coverage
• Life and AD&D
• Short and long-term disability coverage
• Paid time off
• Employee assistance
• Participation in a 401k program that includes company match
• Above market total compensation package
• Comprehensive suite of health and welfare, retirement, and paid leave benefits

Salary range

$130k - $155k (Colorado market-based; may vary by location). Compensation depends on qualifications, skills, competencies, and experience and may fall outside the range shown.

Additional details

• Reasonable accommodations may be made for individuals with disabilities to perform essential functions.
• Occasionally required to stand, walk, sit, use hands to handle or feel objects, reach, climb stairs, balance, stoop or kneel, talk and hear.
• Occasionally required to lift and/or move up to 25 pounds.
• This position is eligible for company benefits including medical, dental, and vision, life and AD&D, short and long-term disability, paid time off, employee assistance, 401k with company match, and other voluntary benefits.

Desired qualifications

• Experience with distributed data processing frameworks including Apache Spark.
• Experience preparing governed data products for AI-enabled use cases, including Microsoft Fabric Lakehouse, semantic models, Fabric/Data Agent patterns, and ontology or taxonomy alignment for explainable AI outputs.
• Familiarity with data observability, metadata management, lineage, data contracts, reliability indicators, and operational best practices in production environments.
• Familiarity with additional Azure services such as Azure Functions or Logic Apps.
• Experience supporting data platform enhancement, refactoring, modernization, or reusable architecture initiatives.
• Experience working with structured and unstructured operational sources such as enterprise applications, operational workflows, documents, dashboards, and knowledge assets.
• Experience working in a scaling or fast-paced organization where priorities evolve quickly and practical delivery discipline is required.