Data Engineer - Databricks / Azure Consultant
Big Data
Bigdata
Cloud Data Engineering
Cloud Data Platform
Cloud Platform
Cloud Platforms
Data
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Governance
Data Integration
Data Integration Tools
Data Lakehouse
Data Management
Data Pipeline
Data Platform
Data Processing
Data Security
Data Warehouse
Database
Databases
Databricks
Databricks Mlflow
ETL
Informatica
Information Technology (IT)
Pyspark
Software Development
Spark
SQL
Job Description
Yoh, A Day & Zimmermann Company is seeking a Databricks / Azure Consultant to design, implement, and support an enterprise Databricks data platform running on Microsoft Azure. This hybrid role centers on data engineering, integration, security, automation, and platform best practices, with collaboration across multiple engineering and cloud teams.
Responsibilities
- Design, implement, and support the design, implementation, and ongoing development of the Databricks data platform
- Partner with technology and data teams to build and enhance scalable data solutions using Databricks and Microsoft Azure
- Focus on data engineering, integration, security, automation, and platform best practices
- Collaborate across application development, infrastructure, cloud, security, architecture, and data teams
Required Skills and Experience
- Microsoft Azure
- Databricks
- Apache Spark
- CI/CD
- Hands-on experience with Databricks Workspace
- Hands-on experience with Unity Catalog
- Hands-on experience with Delta Lake
- Hands-on experience with Spark UI
- Hands-on experience with MLflow
- Strong development experience with Python, PySpark, SQL, and/or Java
- Experience developing and optimizing data processing and ETL/ELT pipelines
- Strong Microsoft Azure experience, particularly Azure Databricks
- Strong Microsoft Azure experience, particularly Azure Data Lake Storage Gen2 (ADLS Gen2)
- Strong Microsoft Azure experience, particularly Azure Data Factory
- Experience with Git
- Experience with CI/CD pipelines
- Experience implementing or supporting Databricks security
- Experience implementing or supporting Databricks access controls
- Experience implementing or supporting secrets management
- Experience implementing or supporting identity integration
- Experience implementing or supporting data governance
- Experience implementing or supporting Unity Catalog
- Experience with Databricks workflows
- Experience with job orchestration
- Experience with monitoring
- Experience with troubleshooting
- Experience with performance optimization
- Experience with production support
- Ability to collaborate across technical teams and operate within established enterprise architecture, security, change-management, and governance standards
- Knowledge of Apache Airflow
Preferred Experience
- Experience designing and delivering production-grade Databricks solutions in a large enterprise environment, particularly where security, data governance, resiliency, automation, and operational support are key requirements
Additional Experience (Desirable)
- Experience with AWS, including S3 and IAM
- Experience with OpenShift
- Experience with Kubernetes
- Experience with Terraform / Infrastructure as Code (IaC)
Location and Work Arrangement
- Jersey City, NJ (hybrid)
- Ideally 5 days onsite; 3-4 days onsite may be acceptable and discussed during interview
- Candidates local to NJ/NY area
- USC and GC only
Compensation
- Estimated hourly rate: $70.00
- Estimated hourly rate: $85.00
Benefits
- Medical, Prescription, Dental & Vision Benefits (for employees working 20+ hours per week)
- Health Savings Account (HSA) (for employees working 20+ hours per week)
- Life & Disability Insurance (for employees working 20+ hours per week)
- MetLife Voluntary Benefits
- Employee Assistance Program (EAP)
- 401K Retirement Savings Plan
- Direct Deposit & weekly epayroll
- Referral Bonus Programs
- Certification and training opportunities
Technology Focus
- Azure, Azure Databricks, Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Data Factory
- Databricks Workspace, Unity Catalog, Delta Lake, Databricks security, Databricks workflows
- Apache Spark, Spark UI, PySpark, MLflow
- ETL/ELT, CI/CD, Git
- OpenShift, Kubernetes, Terraform, Infrastructure as Code (IaC)
- AWS (S3, IAM) (desirable)
- Apache Airflow
Notes on Pay Range
Benefit eligibility is in accordance with applicable laws and client requirements. Any pay ranges displayed are estimations; actual pay is determined by an applicant's experience, technical expertise, and other qualifications as listed in the job description.