Senior Data Engineer
Job Description
Senior Data Engineer role in the Online Data Engineer teams at The Home Depot, focused on translating business requirements into enterprise data infrastructure. You will build and maintain database pipeline architectures, create data validation methods, and develop in-house solutions that improve reliability and confidence through monitoring, continual testing, and validation. The work supports data capture, cleansing, organization, and transformation using big-data techniques, with design standards and assurance processes for data connections, flows, and storage.
Responsibilities
- Apply deep business knowledge to solution approaches, building trust and collaboration with internal customers and cross-functional teams.
- Provide general education on advanced analytics to technical and non-technical business partners.
- Use an understanding of IT needs to support delivery of business solutions and actively seek opportunities to leverage data science as a competitive advantage.
- Partner with project teams and business stakeholders to define project goals, guide prioritization, and support quality outcomes.
- Provide mentoring and coaching to more junior roles to strengthen technical competencies.
- Collaborate with managers and the team to support workload and resource distribution, and help with recruiting and hiring efforts.
- Execute tasks with high efficiency and quality by selecting and applying advanced analytical methodologies and interpreting results appropriately.
- Communicate insights and recommendations to technical and non-technical leaders, preparing progress updates, reports, and presentations.
- Clearly communicate the impacts of recommendations to drive alignment and adoption.
- Direct teams through requirements gathering, design, and development of complex applications and programs.
- Stay current on developments in advanced analytics, technical skill sets, and additional data sources.
- Support continuous improvement in predictive and prescriptive analytics by developing replicable solutions.
What You’ll Build and Maintain
- Database pipeline architectures for capturing and retrieving customer data.
- Data validation methods and data analysis tools.
- APIs to retrieve data from enterprise systems.
- In-house enterprise solutions hosted and maintained to improve reliability and confidence through monitoring and automated testing and validation.
- Design standards and assurance processes to ensure compatibility and operability across data connections, flows, and storage requirements.
- Recommendations for data operations and system or storage upgrades based on review of internal and external business and product requirements.
Required Qualifications
- At least 18 years of age.
- Must be legally permitted to work in the United States.
- 4+ years of relevant experience.
- Bachelor’s degree program or equivalent degree in a field of study related to the job.
Preferred Qualifications
- Bachelors or Masters in a quantitative field (Analytics, Computer Science, Math, Physics, Statistics, etc.) or relevant work experience.
- Demonstrated experience in predictive modeling, data mining, and data analysis.
- Experience developing and testing ETL jobs/pipelines, configuring orchestration, automated CI/CD, writing automation scripts, and supporting pipelines in production.
- Experience in Python and ability to use SQL for analytical queries and performance tuning.
- Experience defining and capturing metadata and rules associated with ETL processes.
- Experience building Batch and Streaming pipelines.
- Ability to stitch and maintain data from multiple sources.
- Experience using JavaScript and front-end frameworks (React) and QA apps (Retina, KPI Shield, Alert Goose), plus ability to produce tags for site data.
- Experience in Google BigQuery to stitch and enrich raw data from multiple sources.
- Proven ability to use PySpark, AirFlow, and DataProc to engineer and automate data flow pipelines.
- Ability to optimize pipeline run time and lower cost on slots/storage consumption.
- Ability to prioritize requests and manage a product roadmap.
- Strong verbal and written communication skills at all levels.
Location and Work Details
- Location: Atlanta, GA (onsite)
- Travel: Typically overnight travel 5% to 20% of the time.
- Management: Typically reports to Manager or Above; 0 direct reports.
- Working conditions: Comfortable indoor area; any unpleasant conditions would be infrequent and not objectionable.
Technologies
Python, JavaScript, React, ETL, ETL jobs/pipelines, BigQuery, PySpark, AirFlow, DataProc, CI/CD, SQL, Batch pipelines, Streaming pipelines