Azure Data Engineer

Skywalk Global is hiring an Azure Data Engineer for onsite support in Indianapolis, IN, focused on building reliable data products across integration, pipelines, modeling, governance, and documentation.

Responsibilities

• Assist with data technology strategy, including resolution of strategic data integration issues (data quality and stewardship, including real-time concerns)
• Provide direction and oversight for designing and implementing data solutions across multiple secure systems for integration and synthesis
• Help define and maintain data management policies and data engineering design standards
• Advise, train, and provide technical support across projects
• Participate in project activities such as requirements gathering, solution reviews, and explaining technical complexities to non-technical stakeholders
• Collaborate with solution leads, project managers, data architects, data scientists, and data analysts on solution design, architecture, and implementation
• Support extraction, transformation, and loading (ETL) from a variety of sources using data engineering tools and methods
• Design, build, and maintain batch and streaming pipelines in Azure Databricks using Py Spark, Spark SQL, and Delta Lake with a medallion (bronze/silver/gold) lakehouse architecture
• Develop, load, and tune dimensional data models and SQL workloads in Azure Synapse Analytics (dedicated and serverless SQL pools) to support reporting and dashboards
• Orchestrate and monitor data workflows using Databricks Workflows/Jobs and Synapse or Azure Data Factory pipelines, including scheduling, alerting, and failure recovery
• Apply data governance and security controls to protect sensitive health data (PHI/PII) in alignment with HIPAA and agency policy (Unity Catalog permissions, role-based access, row- and column-level security, data masking)
• Document and demonstrate solutions using documentation assets such as flowcharts, layouts, diagrams, charts, and code comments with clear code practices
• Support operations by performing systems analysis and recommending changes to policies and procedures

Requirements

• 4+ years experience (required)
• Master’s degree in data science, analytics, informatics, computer science, mathematics, engineering, or related field (or an equivalent pathway per stated substitution rules)
• Alternatives to the degree path include: bachelor’s degree plus 2+ years relevant experience, or 4+ years hands-on experience in a similar data engineering role, or relevant certifications and coursework substituting for formal requirements
• Experience developing with Databricks notebooks in PySpark and Spark SQL, including transformations, joins, aggregations, and window functions
• Knowledge of CID transactions, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance (OPTIMIZE, Z-order, VACUUM)
• Experience with medallion (bronze/silver/gold) architecture and incremental ingestion using Auto Loader, Structured Streaming, or Lakeflow/DLT
• Hands-on use of Unity Catalog for Workspace Catalogs, schemas, permissions, lineage, external locations, and storage credentials
• Ability to right-size clusters and SQL warehouses and diagnose partitioning, shuffle, skew, and caching issues via Spark UI
• Scheduling and dependency management with ADF Pipelines and Databricks Workflows (Jobs), including Git (Repos) for source control
• Designing MPP warehouse tables, including distribution (hash, round-robin, replicated), columnstore indexes, and partitioning
• Querying Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views
• Building star-schema facts and dimensions, including stored procedures and views, with SCD Type 1 and SCD Type 2
• Developing ingestion and pipeline components using COPY INTO, PolyBase, CETAS, and parameterized Synapse/ADF pipelines (copy, data flows, triggers)
• Workload management using resource classes, statistics, materialized views, result-set caching, and pool scaling/pausing
• ADLS Gen2 folder/file-format design (Parquet/Delta), Entra ID access control, managed identities, and Key Vault
• Git and CI/CD using Azure DevOps or GitHub, including branching, PRs, and deploying Synapse artifacts and Databricks Asset Bundles
• Monitoring and troubleshooting using Azure Monitor/Log Analytics, including alerting and resolution of failed jobs
• Strong written and verbal communication skills
• Familiarity with Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks

Technologies

• Azure Databricks, PySpark, Spark SQL, Delta Lake
• Medallion (bronze/silver/gold) lakehouse architecture
• Azure Synapse Analytics, SQL pools
• Databricks Workflows/Jobs, Azure Data Factory
• Unity Catalog, HIPAA; Unity Catalog permissions, role-based access, row- and column-level security, data masking
• Databricks notebooks, R
• CID transactions, MERGE upserts, schema enforcement/evolution, time travel, OPTIMIZE, Z-order, VACUUM
• Auto Loader, Structured Streaming, Lakeflow/DLT
• Spark UI
• Git (Repos)
• MPP warehouse tables; Parquet, Delta, CSV
• ADLS Gen2; OPENROWSET, external tables, views
• Star-schema, SCD Type 1, SCD Type 2, stored procedures
• COPY INTO, PolyBase, CETAS, data flows, triggers
• Materialized views, result-set caching, Azure Monitor, Log Analytics
• Entra ID, managed identities, Key Vault
• Azure DevOps, GitHub, Databricks Asset Bundles
• Microsoft lakehouse patterns, Power BI

Location / Candidate Eligibility

• Indianapolis, IN (onsite)
• Local candidates only will be considered

Job Summary

• Supports data integration, pipeline development, data modeling, governance, and documentation for the agency’s data products
• Implements the agency’s data strategy with batch and streaming pipelines in Azure, warehouse workload optimization, and enforcement of data governance standards
• Participates in requirements gathering and solution reviews; collaborates with data architects, scientists, analysts, and project managers to translate business needs into technical designs
• Supports operational processes and contributes to policy and procedure improvements; may provide mentorship or supervision based on agency needs
• Job duties may change and are not limited to the activities listed in the description