Databricks Data Engineer - Manager - Tech Cons - Open Location
Manager
Artificial Intelligence
Big Data
Bigdata
Cloud Data Engineering
Cloud Data Platform
Cloud Platform
Cloud Platforms
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Engineering Lead
Data Integration
Data Integration Tools
Data Lakehouse
Data Pipeline
Data Platform
Data Processing
Data Warehouse
Database
Databases
Databricks
ETL
Informatica
Information Technology (IT)
Lead Data Engineering
Software Development
Software Engineering
Spark
SQL
Job Description
EY seeks a Manager of Data Engineering to lead the design and delivery of complex cloud analytics solutions with an emphasis on Databricks. This role translates business requirements into technical requirements, oversees execution quality, and manages delivery across data engineering teams while partnering closely with clients.
Location
- Chicago, IL (onsite)
- Anywhere in Country (other locations)
Salary
- USD 125,500 - 261,600 per yearly
- Competitive compensation is also listed by office area, with individual salaries within published ranges based on education, experience, knowledge, skills, and geography.
Responsibilities
- Analyze business requirements and translate them into technical requirements.
- Design, build, and operate scalable data architecture and modeling solutions.
- Lead workstream delivery and ensure quality across processes.
- Engage with clients daily through working sessions, identify opportunities for additional services, and manage client relationships and expectations.
- Implement resource plans and budgets while managing engagement economics.
- Design and develop scalable data engineering solutions using Databricks on cloud platforms such as AWS, Azure, and GCP.
- Oversee architecture for complex cloud analytics solutions, ensuring alignment with business objectives and best practices.
- Drive end-to-end data pipeline development, including data ingestion, transformation, and storage using Databricks and other cloud services.
- Ensure data quality, integrity, and security across the data lifecycle, implementing data governance best practices.
- Manage and mentor data engineers, supporting a culture of innovation, collaboration, and continuous improvement.
- Communicate effectively with stakeholders, including both technical and non-technical audiences, to explain complex data concepts and project progress.
- Stay current with trends and emerging technologies in data engineering, cloud computing, and analytics.
- Provide strategic and technical leadership by aligning initiatives with organizational goals and making architectural decisions that affect outcomes.
- Use project management practices to manage multiple projects and teams, ensuring timely delivery and adherence to scope.
- Apply change management, risk management, documentation, and reporting practices to support delivery and stakeholder understanding.
Required Qualifications
- Bachelor’s degree in computer science, Engineering, or a related field (Master’s degree preferred).
- Typically 4 - 6 years of relevant experience in data engineering with a focus on cloud data solutions and analytics.
- Proven expertise in Databricks and experience with Spark for big data processing.
- Hands-on experience with AI-native data engineering practices, including generative AI and agentic workflows for accelerating pipeline development, testing, documentation, and optimization.
- Strong background in data architecture and the design of complex cloud analytics solutions.
- Experience leading and managing teams, with a focus on mentoring and developing talent.
- Strong programming skills in Python, Scala, or SQL.
- Familiarity with AI-powered IDEs and coding assistants such as Cursor, GitHub Copilot, and Databricks Assistant for day-to-day development.
- Strong problem-solving abilities and capability to work independently and collaboratively.
- Strong communication and interpersonal skills focused on client management.
Technologies
- Databricks
- AWS
- Azure
- GCP
- Spark
- Generative AI
- Agentic workflows
- Python
- Scala
- SQL
- Cursor
- GitHub Copilot
- Databricks Assistant
- ETL
- AI-powered IDEs
Benefits
- Medical and dental coverage
- Pension and 401(k) plans
- Paid time off options
Relevant Large-Scale Implementation Programs
- Enterprise Data Lake Implementation: Led the design and deployment of a cloud-based data lake for a Fortune 500 retail client, integrating data from multiple sources (including ERPs, POS systems, and e-commerce platforms) to enable advanced analytics and reporting.
- Real-Time Analytics Platform: Managed development of a real-time analytics platform using Databricks for a financial services organization, enabling real-time fraud detection and risk assessment through streaming data ingestion and processing.
- Data Warehouse Modernization: Oversaw modernization of a legacy data warehouse to a cloud-native architecture for a healthcare provider, implementing ETL processes with Databricks and improving accessibility for analytics and reporting.
Additional Information
- Travel may be required regularly based on client needs.
- Ideally, experience with advanced data analytics tools and techniques, familiarity with machine learning concepts and applications, knowledge of industry trends and best practices in data engineering, familiarity with cloud platforms and data services, knowledge of data governance and compliance standards, and experience with machine learning frameworks and tools.