Lead Data Engineer

Capital Technology Group is seeking a Lead Data Engineer for a hybrid role in Washington, DC. The position focuses on designing, building, and maintaining scalable AWS-based data pipelines and platforms that support mission-critical analytics in federal environments with FedRAMP and NIST SP 800-53 controls.

Key Responsibilities

• Design, build, and maintain scalable data pipelines, ETL/ELT workflows, and data models using Python, Apache Spark (PySpark), SQL (PostgreSQL), and AWS Glue.
• Develop and optimize AWS-native data platforms using AWS Glue, Amazon EMR, Amazon MWAA (Apache Airflow), Amazon S3, RDS, and CloudWatch.
• Build high-performance ingestion, transformation, and orchestration workflows for structured and semi-structured data using Apache Iceberg and Parquet.
• Design and optimize analytical data platforms using Amazon Athena, Trino, Hive, OpenSearch, and enterprise data catalog technologies.
• Build AI-enabled data solutions using Amazon Bedrock, RAG pipelines, and vector search technologies including Amazon S3 Vectors and OpenSearch vector indexes.
• Develop cloud infrastructure using CloudFormation (Infrastructure as Code) and GitHub with enterprise CI/CD pipelines.
• Improve reliability, scalability, performance, and maintainability through monitoring, troubleshooting, automation, and continuous optimization.
• Support mission-critical analytics and reporting solutions in large-scale AWS-based federal data environments while implementing solutions that comply with FedRAMP and NIST SP 800-53 security controls.
• Mentor junior engineers through technical guidance, architecture discussions, and code reviews while promoting engineering best practices.
• Collaborate with cross-functional teams in an Agile environment to define requirements, deliver high-quality data solutions, and communicate technical concepts to technical and non-technical stakeholders.

Required Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related technical field.
• 15+ years of professional experience in data engineering, data architecture, or related fields.
• Strong hands-on experience with Apache Spark (PySpark) (required), Python, SQL (PostgreSQL), and dbt for large-scale data engineering, ETL/ELT development, data transformation, and data modeling.
• Experience with AWS Glue, Amazon EMR, Amazon MWAA (Apache Airflow), AWS Lambda, Amazon S3, and Amazon RDS.
• Proven ability to develop scalable data pipelines, workflow orchestration, and data integration solutions across enterprise environments.
• Experience with modern data lake technologies and formats such as Parquet and Iceberg.
• Ability to design and optimize solutions using relational and NoSQL databases.
• Experience building reliable, high-performance data platforms through performance tuning, system optimization, and enterprise-scale ETL/ELT architectures.
• Strong analytical and problem-solving skills.
• Experience working in Agile, iterative software development environments.
• Ability to quickly learn and apply new technologies and domain knowledge.
• Excellent written and verbal communication skills, including the ability to explain complex topics to diverse audiences.

Technologies

• Python, Apache Spark (PySpark), SQL (PostgreSQL), AWS Glue, Amazon EMR, Amazon MWAA (Apache Airflow), Amazon S3, RDS, CloudWatch
• Apache Iceberg, Parquet, Amazon Athena, Trino, Hive, OpenSearch, enterprise data catalog technologies
• Amazon Bedrock, RAG pipelines, Amazon S3 Vectors, OpenSearch vector indexes
• CloudFormation (Infrastructure as Code), GitHub, CI/CD pipelines
• FedRAMP, NIST SP 800-53, AWS Lambda, dbt, NoSQL databases

Benefits

• Competitive Compensation Package
• Medical, Dental, and Vision
• Life Insurance, Short/Long Term Disability
• Employee Assistance Program
• 401(k) with 4% matching
• Liberal PTO vacation policy
• Generous Annual Continuing Education
• Annual Wellness Budget
• Bonus Incentive Programs (employee referrals and performance-based rewards)
• Remote Work (hybrid roles will be specified in the job post)

Eligibility and Clearance

• Applicants MUST BE US Citizens and be able to obtain Public Trust clearance.

Salary

USD 150,000 - 200,000 per year. Final offer may vary based on experience, skills, and other factors; the stated range is not a guarantee and is subject to change.