Senior Data Engineer
Job Description
CoStar Group is hiring a Senior Data Engineer to design, build, and operate dependable, scalable data pipelines and data products on a modern cloud data platform. This is a hands-on position emphasizing production reliability, performance optimization, and ongoing improvements.
Key Responsibilities
- Design, build, test, and maintain reliable, scalable data pipelines and data products.
- Develop and optimize data transformations and orchestrations using SQL, Python, dbt, Airflow, Glue, PostgreSQL, and Snowflake.
- Convert business requirements into maintainable technical designs and production-ready solutions.
- Implement automated testing, data-quality checks, monitoring, and failure handling, including recovery processes.
- Optimize large-scale data workloads for performance, scalability, and cost.
- Troubleshoot production issues and conduct root-cause analysis across pipelines and databases.
- Use Git-based development practices, including code review, CI/CD, and controlled release processes.
- Ensure data products are documented, observable, governed, and supported with appropriate metadata and lineage.
- Reduce technical debt and recurring operational issues through automation and improved design.
- Collaborate with Product Management, Application Engineering, and business stakeholders.
Required Qualifications
- 6+ years of professional experience in data engineering or a closely related discipline.
- Strong Python skills for data pipelines, automation, integrations, and supporting data applications.
- Advanced SQL experience, including work with large-scale datasets.
- Strong understanding of data modeling, database design, and data transformation patterns.
- Experience with dbt, including model development, dependency management, testing, incremental processing, and production deployment.
- Experience with Airflow, including DAG development, orchestration, dependencies, retries, failure handling, scheduling, and production operations.
- Hands-on PostgreSQL experience, including complex SQL development, database design, indexing, query optimization, and performance tuning.
- Hands-on Snowflake experience in the context of production data workloads.
- Experience with AWS data services such as S3, RDS/Aurora, Glue, MWAA, IAM, and CloudWatch.
- Experience implementing automated data validation and quality controls.
- Experience with Git, including pull requests, code review, conflict resolution, and CI/CD workflows.
- Demonstrated ability to troubleshoot complex production pipeline and data issues.
- Understanding of monitoring, failure recovery, and production support practices.
Preferred Qualifications
- Experience with Amazon Aurora PostgreSQL.
- Experience working with datasets containing hundreds of millions or billions of records.
- Experience with AWS Glue and Amazon MWAA.
- Experience with geospatial data and PostGIS.
- Experience with SQL Server.
- Experience with DataHub or another metadata, lineage, or governance platform.
- Experience with Grafana or comparable observability tooling.
- Experience with database schema migration and version-control frameworks.
- Experience working within controlled production release and change-management processes.
- Strong hands-on engineering capability and ownership.
- Ability to independently deliver complex data engineering solutions.
- Strong analytical, troubleshooting, and root-cause-analysis skills.
- Ability to work effectively with large and complex datasets.
- Strong focus on reliability, performance, scalability, and maintainability.
- Ability to balance delivery requirements with engineering quality.
- Strong written and verbal communication.
- Ability to collaborate effectively with technical and non-technical stakeholders.
Technologies
- SQL, Python, dbt, Airflow, Glue, PostgreSQL, Snowflake
- AWS S3, RDS/Aurora, MWAA, IAM, CloudWatch
- Git, CI/CD
Base Compensation
Base Compensation: $128,000 - $190,000 per year.
Benefits
- Comprehensive healthcare coverage: Medical / Vision / Dental / Prescription Drug
- Life, legal, and supplementary insurance
- Virtual and in person mental health counseling services for individuals and family
- Commuter and parking benefits
- 401(K) retirement plan with matching contributions
- Employee stock purchase plan
- Paid time off
- Tuition reimbursement
- On-site fitness center and/or reimbursed fitness center membership costs (location dependent)
- Access to CoStar Group’s Employee Resource Groups
- Complimentary gourmet coffee, tea, hot chocolate, fresh fruit, and other healthy snacks
Additional Information
- Location: Irvine, CA (onsite)
- CoStar Group is not able to provide visa sponsorship for this position.
- CoStar Group is an Equal Employment Opportunity Employer; maintains a drug-free workplace and performs pre-employment substance abuse testing.