Reference Data Engineer

The Assistant Vice President (AVP), Software Engineering role within KKR’s Operations Systems team focuses on building backend services and data pipelines that support dependable analytics and platform-wide consumption. The position emphasizes Python, distributed processing with Apache Spark, and infrastructure automation using Terraform, alongside core data engineering capabilities across ETL, modeling, and distribution.

Responsibilities

• Design, build, and take ownership of scalable backend services and data pipelines, primarily using Python.
• Develop and optimize large-scale data processing workflows with Apache Spark for batch and near-real-time scenarios.
• Lead data modeling and schema design decisions to support reliable analytics and downstream usage.
• Architect and maintain robust ETL/ELT workflows with strong data quality, lineage, and observability.
• Design and implement data distribution patterns to serve data efficiently across internal systems and consumers.
• Provision and manage cloud infrastructure using Terraform and infrastructure-as-code best practices.
• Deliver significant workstreams end-to-end, from technical design through production and ongoing operations.
• Support engineering quality through design reviews, code reviews, and established technical standards.
• Mentor and support the growth of analysts and junior engineers on the team.
• Partner with product, data, and platform leaders to shape roadmap priorities and translate requirements into scalable solutions.
• Perform root-cause analysis for complex production issues and drive continuous improvements in reliability and performance.

Requirements

• Bachelor’s degree in Computer Science, Engineering, or a related field (or equivalent practical experience).
• 4–7 years of professional software engineering experience, with a strong backend focus.
• Expert-level Python proficiency for production services and data pipelines.
• Hands-on experience with Apache Spark for distributed data processing at scale.
• Proficiency with Terraform for infrastructure-as-code and cloud resource management.
• Strong data engineering fundamentals, including data modeling, ETL/ELT design, and data distribution patterns.
• Solid command of SQL and both relational and non-relational data stores.
• Experience with Git, CI/CD pipelines, and modern software development practices.
• Proven ability to own complex workstreams and mentor other engineers.
• Strong communication skills and the ability to collaborate across technical and business teams.

Technologies

• Python
• Apache Spark
• Terraform
• ETL/ELT
• SQL
• Git
• CI/CD
• Relational data stores
• Non-relational data stores

Preferred Qualifications

• Experience with cloud platforms (AWS, Azure, or GCP) and their data services.
• Familiarity with workflow orchestration tools such as Airflow or Dagster.
• Experience with streaming technologies such as Kafka or Spark Structured Streaming.
• Exposure to containerization and orchestration using Docker and Kubernetes.
• Experience in financial services or another data-intensive, regulated industry.
• Experience with Palantir Foundry or similar data integration and analytics platforms.

Compensation and Location

New York, NY (onsite). Salary range: USD 135,000 - 170,000 per year.

Benefits

• Competitive compensation
• Comprehensive benefits
• Discretionary bonus based on factors such as individual and team performance
• Opportunities for professional growth, technical leadership, and impact