Senior Big Data Engineer

Senior Big Data Engineer role with Systems Development and Analysis Inc (SDA), based in Arlington, VA with hybrid work options, focused on building scalable data ingestion, integration, and analytics solutions using Java, Kafka, Spark, Elasticsearch, and cloud platforms.

Responsibilities

• Design, develop, and maintain enterprise data ingestion, ETL, and data transformation solutions for batch and real-time processing.
• Build and support scalable Kafka based streaming architectures for high volume data processing and distribution.
• Develop Java based applications and services that support enterprise data integration and transformation workflows.
• Create Spark and Spark Streaming solutions to process, enrich, and analyze large datasets.
• Design and support enterprise search capabilities using Elasticsearch to enable data discovery, investigative workflows, analytics, and operational decision making.
• Analyze source and target data structures and develop source to target mapping specifications.
• Integrate data from multiple structured and semi-structured sources while ensuring data integrity, consistency, traceability, and performance.
• Develop REST APIs and system integration services that enable data exchange across enterprise platforms.
• Support data migration, modernization, and cloud transformation initiatives.
• Perform data validation, reconciliation, profiling, and quality assurance activities.
• Implement and support data quality, governance, lineage, and validation processes across ingestion and transformation pipelines.
• Optimize SQL queries, database processes, and distributed data pipelines.
• Troubleshoot production issues and improve platform reliability, scalability, and performance.
• Collaborate with architects, developers, analysts, and stakeholders to design and implement technical solutions.
• Support onboarding efforts for data consumers and integration partners.
• Evaluate emerging technologies and develop proof of concept solutions.
• Mentor junior engineers and contribute to technical leadership initiatives.

Requirements

• U.S. Citizen.
• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent experience).
• 5-10+ years of experience in Data Engineering, Software Engineering, ETL Development, or Enterprise Data Integration.
• Strong hands on Java development experience.
• Experience designing and implementing ETL and data integration solutions.
• Experience building and supporting Apache Kafka producers, consumers, and streaming applications.
• Experience developing applications using Apache Spark and/or Spark Streaming.
• Strong SQL and relational database experience, including Oracle.
• Experience creating and maintaining source to target data mappings.
• Experience with data transformation, data migration, and data validation activities.
• Experience developing REST APIs and integration services.
• Experience working with XML, JSON, CSV, and other structured or semi-structured data formats.
• Experience supporting Linux/Unix environments.
• Strong analytical, troubleshooting, and problem solving skills.

Technologies

• Java
• Kafka
• Spark
• Elasticsearch
• Oracle
• SQL
• ETL
• Data Integration
• Data Mapping
• REST APIs
• MongoDB
• AWS
• Linux
• Scala
• Spring Framework
• Spring Boot
• Apache NiFi
• AWS Glue
• Informatica
• Talend
• Hadoop
• HBase
• Delta Lake
• PostgreSQL
• DocumentDB
• Redshift
• Docker
• Kubernetes
• Azure
• Google Cloud Platform
• CI/CD Pipelines
• Spark Streaming

Benefits

• Work with modern Big Data, cloud, and enterprise integration technologies.
• Design and build large-scale data platforms supporting mission critical operations.
• Collaborate with highly skilled engineers, architects, and data professionals.
• Influence technical direction and solution architecture.
• Competitive compensation and benefits package.
• Flexible hybrid work environment.
• Opportunities for professional growth and technical leadership.

Preferred Qualifications

• Experience with Elasticsearch and enterprise search platforms.
• Experience with Hadoop ecosystem technologies.
• Experience with Scala development.
• Experience with MongoDB, DocumentDB, HBase, Delta Lake, Redshift, or other Big Data technologies.
• Experience with ETL tools such as Apache NiFi, AWS Glue, Talend, Informatica, Pentaho, or SSIS.
• Experience with event driven architectures and enterprise messaging technologies.
• Experience with Docker and Kubernetes.
• Experience supporting AWS, Azure, or Google Cloud Platform environments.
• Experience supporting enterprise data migration or modernization initiatives.
• Experience working in Agile software development environments.
• Experience supporting large-scale analytics, risk assessment, investigative, intelligence, law enforcement, or operational decision-support systems.

Technologies You’ll Work With

• Java
• Scala
• SQL
• Spring Framework
• Spring Boot
• REST APIs
• ETL Development
• Data Mapping & Transformation
• Data Migration
• Apache NiFi
• AWS Glue
• Informatica
• Talend
• Apache Kafka
• Apache Spark
• Spark Streaming
• Hadoop
• HBase
• Delta Lake
• Elasticsearch
• Oracle
• PostgreSQL
• MongoDB
• DocumentDB
• Redshift
• Relational & NoSQL Databases
• AWS
• Azure
• Google Cloud Platform
• Docker
• Kubernetes
• CI/CD Pipelines

Why Join SDA?

• Work with modern Big Data, cloud, and enterprise integration technologies.
• Design and build large-scale data platforms supporting mission critical operations.
• Collaborate with highly skilled engineers, architects, and data professionals.
• Influence technical direction and solution architecture.
• Competitive compensation and benefits package.
• Flexible hybrid work environment.
• Opportunities for professional growth and technical leadership.