Streaming Data

Streaming Data Engineer Jobs

2 jobs found

Today

Data Engineer

JANSON Fairfax, VA

Fairfax, VA

Today

$135k - $135k/yr Full time Big Data Bigdata Data

Apply Now

Job Description

JANSON is seeking a Data Engineer (Engineer IV) to support a U.S. Department of Defense (DoD) initiative. This in-person role is based in Kaiserslautern, Germany, with an emphasis on building and operating data pipelines, data warehouses/lakes, and governed, secure data integrations. You will work within established policies to enable long-term sustainability, data quality, and compliance for critical analytical and real-time use cases.

Responsibilities

Build and manage data pipelines by performing ETL from multiple sources, including DoD Advana and Army Vantage, into data warehouses or data lakes.
Clean, normalize, and enrich data during transformation to support downstream reporting and analytics.
Create automated workflows to move data across systems using tools such as Apache Airflow, AWS Glue, or Azure Data Factory.
Optimize pipelines for scalability and reliability, including real-time ingestion and processing.
Implement streaming data pipelines with technologies like Apache Kafka, Apache Flink, or Spark Streaming.
Support real-time processing and analysis for use cases such as fraud detection and IoT analytics.
Design and implement relational (for example MySQL, PostgreSQL) or NoSQL databases (for example MongoDB, Cassandra) based on project needs.
Design schemas and indexing strategies to improve performance for analytical workloads.
Build and maintain data warehouses using platforms such as Snowflake, Amazon Redshift, or Google BigQuery, and optimize storage and query performance.
Integrate and support management of data lakes in line with organizational policies; interpret policies to support the command’s long-term sustainability.
Integrate data from APIs, databases, flat files, and third-party systems; handle formats including JSON, XML, CSV, and Parquet.
Perform data migrations during system upgrades or transitions (such as on-premises to cloud).
Configure APIs to expose data for consumption by other systems or applications.
Manage data quality and governance at the 21st TSC level in accordance with USAEUR-AF policies and guidance, including checks for accuracy, completeness, and consistency.
Identify and resolve data anomalies or errors, and document data lineage, definitions, and transformations for transparency and traceability.
Implement and manage compliance and security controls, including GDPR, HIPAA, and CCPA requirements, plus encryption, access control, and auditing.
Partner with internal teams to address data quality issues affecting analyses and tooling, and work with PEOs and PMs of Systems of Record to understand and correct data issues feeding multiple platforms.

Requirements

Eighteen (18) years of recent specialized experience
Or: Major IT Certification AND fourteen (14) years of recent specialized experience
Or: Associate’s degree AND fourteen (14) years of recent specialized experience
Or: Bachelor’s degree AND ten (10) years of recent specialized experience
Or: Master’s degree AND seven (7) years of recent specialized experience
IA Technical Level II certification or higher; IAT Level II or higher (Required)
Certifications listed below:
- CCNA-Security
- CompTIA CySA+
- Global Industrial Cyber Security Professional (GICSP)
- GIAC Security Essentials (GSEC)
- CompTIA Security+ CE
- EC-Council Certified Network Defense (CND)
- (ISC)² Certified System Security Practitioner (SSCP)
Bachelor’s (Required)
recent specialized: 10 years (Required)

Key Technologies

Apache Airflow, AWS Glue, Azure Data Factory
Apache Kafka, Apache Flink, Spark Streaming
MySQL, PostgreSQL, MongoDB, Cassandra
Snowflake, Amazon Redshift, Google BigQuery
APIs; JSON, XML, CSV, Parquet
Encryption, access control, auditing; GDPR, HIPAA, CCPA

Position: Data Engineer (Engineer IV)
Location: Kaiserslautern, Germany (in person)
Clearance: Secret (Secret preferred)
Job type: Full-time
Pay: USD 135,000 per year

Application questions: Do you have a issue living in Kaiserslautern, Germany for 1-2 years? What is your salary range?

View full job page →

2d ago

GCP BigQuery Data Engineer

Virtues Irving, TX

Irving, TX

2d ago

$100k - $110k/yr Full time Analytics Big Data Bigdata

Apply Now

Job Description

Build cloud-native, scalable analytics and near-real-time ingestion pipelines across GCP BigQuery and event-driven streaming systems.

Responsibilities

Design, develop, and optimize enterprise data warehouse solutions using Google BigQuery.
Design, develop, and maintain real-time data ingestion pipelines using Apache NiFi to capture, transform, and route streaming data into Apache Kafka topics.
Build event-driven streaming architectures across Apache NiFi, Apache Kafka, and Google BigQuery for high-throughput, fault-tolerant, low-latency processing.
Configure and optimize Apache NiFi processors for extraction, transformation, routing, filtering, schema validation, error handling, retries, and reliable delivery to Kafka.
Develop streaming ingestion solutions enabling Google BigQuery to consume Kafka event streams and transform data into analytical tables and enterprise reporting datasets.
Design data models and implement ETL/ELT processes to move data from raw to curated and published layers.
Create and manage large-scale BigQuery datasets, including temporary/permanent and internal/external tables.
Optimize BigQuery workloads through query tuning, partitioning, clustering, and cost optimization techniques.
Monitor, troubleshoot, and optimize streaming workloads by tuning NiFi flows, Kafka topics/partitions, and BigQuery streaming ingestion to support high availability, data integrity, minimal latency, and cost-efficient processing.
Build scalable data lake frameworks and ingestion pipelines using GCP technologies.
Develop reporting and visualization solutions using Looker, Looker Studio (Data Studio), Connected Sheets, and other BigQuery reporting tools.
Collaborate with data architects, modelers, developers, DevOps engineers, project managers, and business stakeholders to deliver scalable enterprise analytics and continuous data platform improvements.

Requirements

7+ years of experience in Data Engineering, Data Warehousing, or Big Data platforms.
Must-have skills: Apache NiFi and Apache Kafka.
Strong hands-on experience with Google Cloud Platform (GCP), including BigQuery, Cloud Dataflow, Pub/Sub, and Google Cloud Storage (GCS).
Hands-on experience designing and supporting real-time streaming pipelines using Apache NiFi, Apache Kafka, and Google BigQuery.
Experience using BigQuery Console/Query Editor for data management, performance tuning, and SQL development.
Strong experience designing scalable data models, ETL/ELT pipelines, data lake architectures, and enterprise data warehouse solutions.
Experience with batch and streaming ingestion using GCP services.
Thorough understanding of BigQuery cost structure (storage, ingestion, and query costs) and experience implementing query optimization, partitioning, clustering, and other cost optimization techniques.
Experience managing large-scale datasets, including temporary/permanent and internal/external BigQuery tables.
Experience developing reporting and visualization solutions using Looker, Looker Studio (Data Studio), and Connected Sheets.
Excellent verbal and written communication skills, able to collaborate effectively with technical teams, business stakeholders, senior management, and executive leadership.
Experience working in Agile environments and collaborating with cross-functional teams to deliver enterprise data engineering solutions.

Technologies

Google BigQuery
Apache NiFi
Apache Kafka
Google Cloud Dataflow
Google Pub/Sub
SQL
ETL/ELT
Google Cloud Storage (GCS)
Looker
Looker Studio (Data Studio)
Connected Sheets
DevOps
CI/CD
Agile/Scrum
Python
Spark
Data Modeling
Data Warehousing
Data Lakes
Batch & Streaming Pipelines

Preferred Qualifications

Google Cloud certifications.
Experience delivering enterprise-scale analytics and cloud data solutions.

Pay

$100,000.00 - $110,000.00 per year

Work Location

Onsite in Irving, TX

View full job page →

Streaming Data Engineer Jobs

Data Engineer

GCP BigQuery Data Engineer

Get Streaming Data Engineer Jobs alerts