Data Engineer – Azure Databricks and Cloud Data Solutions (Safety)

Southern Company’s Operations Data Analytics team is seeking a Data Engineer focused on Safety data and cloud data services. The role builds and supports secure, reusable, high-performing pipelines and lakehouse solutions on Azure Databricks, aligned with enterprise expectations for quality, cost, and timeliness.

Responsibilities

• Design and develop robust data pipelines to ingest, store, and process data from multiple sources for analytics and reporting.
• Create and maintain logical and physical data models to support efficient querying, analysis, and visualization.
• Build and maintain ETL processes that move data between systems while supporting data quality and consistency.
• Implement and support lakehouse solutions using medallion architecture, including data engineering pipelines and data storage layers.
• Optimize data lake storage, retrieval, performance, and resource utilization for analytical workloads.
• Develop dynamic, metadata-driven frameworks and scalable data ingestion patterns.
• Support streaming datasets and Databricks Delta Live Tables.
• Apply lakehouse performance-tuning techniques, including partitioning, aggregations, and materialized views.
• Architect, develop, implement, integrate, and support cloud technology solutions that meet business partner expectations for quality, cost, and timeliness.
• Partner with developers and infrastructure teams to optimize application code, create reusable services, plan capacity, and protect application and data security.
• Serve as a subject matter expert for data architecture and Databricks.
• Communicate effectively and promptly with customers, team members, and management, building strong peer and business partner relationships.
• Develop required project documentation, follow Change Management policies and procedures, and respond quickly to operational problems.
• Build a strong understanding of business challenges and goals and exhibit Our Values.

Requirements

• Bachelor’s degree required, preferably in Computer Science, Management Information Systems, Engineering, or another technology-related field.
• Experience with Databricks, including notebooks, clusters, and jobs.
• Experience with a major cloud platform, preferably Microsoft Azure, including services such as Cosmos DB, Storage Application Services, and Azure Data Factory.
• Experience architecting data-oriented solutions using cloud data architecture platforms and patterns, preferably Azure.
• Experience implementing lakehouses with medallion architecture, data engineering pipelines, and data storage layers.
• Experience with dynamic metadata-driven frameworks and large-scale data ingestion patterns.
• Experience developing complex logical and physical data models.
• Experience optimizing cloud PaaS resources to follow best practices and reduce cost, plus lakehouse performance tuning using partitioning, aggregations, materialized views, or similar techniques.
• Experience with Agile, DevOps, and CI/CD methodologies.
• Familiarity with Apache Spark, Delta Lake, and MLflow, including knowledge of structured and unstructured data across databases, data warehouses, and data lakes.
• Knowledge of software development technologies, methodologies, and associated tools.
• Excellent root cause analysis and problem-solving skills.

Technologies

• Microsoft Azure
• Azure Databricks
• Unity catalog
• Cosmos DB
• Storage Application Services
• Azure Data Factory
• Databricks (notebooks, clusters, jobs)
• Azure data-oriented solutions
• Medallion architecture
• Extract, Transform, Load (ETL)
• Databricks Delta Live Tables
• Partitioning, aggregations, materialized views
• Apache Spark
• Delta Lake
• MLflow
• Agile, DevOps, CI/CD
• Structured and unstructured data
• Lakehouse

Benefits

• Competitive base salary
• Annual incentive awards for eligible employees
• Health, welfare and retirement benefits designed to support physical, financial, and emotional/social well-being
• Additional and specific details about total compensation and benefits provided during the hiring process

What Success Looks Like

• Reliable, efficient data pipelines that support analytics and reporting needs.
• Reusable and well-aligned solutions across on-premises and Azure technology platforms.
• Consistent, high-quality data movement through maintainable ETL and lakehouse processes.
• Optimized application code, cloud resources, performance, and capacity utilization.
• Secure data and application solutions supported by complete project documentation and disciplined change management.
• Strong working relationships and timely communication with business partners, technology teams, and management.
• Fast, effective resolution of operational problems and growing team expertise in data architecture and Databricks.

What Sets You Apart (Preferred)

• Hands-on experience with Microsoft Azure data services and cloud PaaS architecture and resources.
• Familiarity with streaming datasets and Databricks Delta Live Tables.
• Strong technical writing skills.

What We Value

• Customer focus: Deliver cloud and data solutions that meet or exceed business partner expectations for quality, cost, and timeliness.
• Ownership and responsiveness: Take responsibility for documentation, change management, and rapid resolution of operational problems.
• Collaboration: Communicate effectively and build productive relationships with customers, peers, management, developers, infrastructure teams, and business partners.
• Continuous learning: Become a subject matter expert, stay close to the business, and seek to understand the company’s challenges and goals.
• Enterprise mindset: Design secure, reusable, aligned solutions and exhibit Our Values.

Location and Work Model

• Birmingham, AL (onsite)