Stream Processing

Stream Processing Engineer Jobs

3 jobs found

2d ago

Date Engineer Lead(Big Data)

CGI Group, Inc. Pittsburgh, PA

Pittsburgh, PA

2d ago

$71k - $157k/yr Full time Big Data Bigdata Data Engineer

Apply Now

Job Description

Benefits and culture

CGI offers a benefits-forward environment that prioritizes ownership, teamwork, respect and belonging. In addition to competitive pay, you gain access to comprehensive benefits and ongoing learning opportunities that support your growth and well-being.

Competitive compensation
Comprehensive insurance options
Matching contributions through the 401(k) plan and the share purchase plan
Paid time off for vacation, holidays, and sick time
Paid parental leave
Learning opportunities and tuition assistance
Wellness and Well-Being programs

Location and role focus

This onsite role requires five days per week at client sites in Pittsburgh, PA; Cleveland, OH; Birmingham, AL; or Dallas, TX. The Lead Data Engineer will design, build and optimize real-time analytics solutions and streaming pipelines using Kafka, Flink, Spark, Hadoop and SQL.

Responsibilities

Design and implement low-latency, high-throughput streaming pipelines with Kafka, Flink, and Spark Streaming
Build and maintain event-driven architectures for real-time data ingestion and processing
Develop stateful and stateless stream processing applications using Flink and Spark Structured Streaming
Collaborate with application teams to define integration points and API specifications

Requirements

10+ years of hands-on data engineering experience
Proven track record in building streaming pipelines
Hands-on expertise with Hadoop ecosystem, Kafka, Flink, Spark Streaming and SQL
Proficiency in Scala and Python
Experience with Oracle Database, SQL, and PL/SQL
Familiarity with CI/CD tools (Git, Bitbucket, uDeploy, ServiceNow)
Strong verbal and written communication skills with client-facing capabilities
Experience in client-facing or consulting roles
Strong problem-solving and analytical thinking
Ability to manage multiple priorities in a fast-paced environment
Collaborative mindset with leadership capabilities
Experience building enterprise-scale data platforms
Exposure to cloud-based data ecosystems (AWS, Azure, or GCP)
Knowledge of data governance, security, and compliance practices
Experience mentoring junior team members

Ideal candidate profile

Thinks beyond code and understands business impact
Leads conversations and owns end-to-end solutions
Balances technical depth with strong interpersonal skills
Comfortable taking ownership of solutions from conception to delivery

Technologies

Apache Kafka
Flink
Spark Streaming
Spark Structured Streaming
Hadoop and Hadoop Ecosystem (HDFS)
SQL
Python
Scala
RESTful APIs
Oracle Database
PL/SQL
Git
Bitbucket
uDeploy
ServiceNow
AWS
Azure
GCP

Compensation

Salary: USD 70,800 - 156,700 per year

View full job page →

3d ago

Data Engineer (Ab Initio, OzoneCH, Flink)

Next Gen Software Solutions Berkeley Heights, NJ

Berkeley Heights, NJ

3d ago

$50 - $53/hr Contract Ab Initio Big Data Data Engineer

Apply Now

Job Description

The contract Data Engineer role in Berkeley Heights, NJ onsite focuses on building high-performance, event-driven data pipelines with the Ab Initio ecosystem for a large credit card processing system.

Location: Berkeley Heights, NJ (onsite) • Salary: USD 50 - 53 per hour • Type: Contract

Responsibilities

Architect and develop intricate Ab Initio continuous flow graphs, incorporating stateful joins, sliding time windows, and aggregations.
Construct event-driven data pipelines leveraging Kafka, MQ, and file streams to push updates to dashboards or downstream data stores in near real time.
Guarantee the resilience of continuous flows through checkpointing and recovery to ensure exactly-once processing semantics.
Utilize advanced Ab Initio components such as Reformat, Rollup, Join, and Partition to attain low-latency data processing.
Proactively monitor live production streams to sustain 24/7 reliability and troubleshoot data issues as they arise.
Develop ETL pipelines for both batch and real-time data ingestion and transformation.
Implement and enforce data validation, security, integrity, and compliance across big data platforms.
Diagnose and resolve performance issues in large-scale clusters.
Collaborate with data scientists, analysts, and application teams to deliver robust data solutions.
Automate workflows and improve operational efficiency through scripting and orchestration tools.

Requirements

Strong understanding of credit card processing systems.
Expertise with GDE (Graphical Development Environment), EME (Enterprise Meta Environment), Conduct>It, and Continuous Flows.
Knowledge of Kafka, message queues, and real-time stateful services.
Proficiency in Unix/Linux shell scripting, SQL, and databases such as Oracle and Teradata.
Experience in Java, Scala, Python, or Kafka is a plus.
Familiarity with Linux/Unix environments and shell scripting.
Understanding of data security, governance, and compliance standards.
Experience with cloud-based big data platforms.
Exposure to containerization with Docker and Kubernetes for big data workloads.
Knowledge of CI/CD pipelines for data engineering projects.

Technologies

Ab Initio
GDE (Graphical Development Environment)
EME (Enterprise Meta>Environment)
Conduct>It
Kafka
MQ
Oracle
Teradata
Java
Scala
Python
SQL
Linux
Unix
Shell scripting
Docker
Kubernetes
CI/CD pipelines
Cloud-based big data platforms

Equal Employment Opportunity

Next Gen Software Solutions LLC is an Equal Opportunity Employer. We are committed to fostering an inclusive and diverse workplace where all applicants and employees are treated with respect and dignity. We do not discriminate on the basis of race, color, religion, sex (including pregnancy, sexual orientation, or gender identity), national origin, age, genetic information, veteran status, or any other characteristic protected by applicable federal, state, or local laws.

View full job page →

4d ago

Senior Software Engineer

Amplitude San Francisco, CA

San Francisco, CA

4d ago

$165k - $276k/yr Full time Backend Developer Senior Clickhouse

Apply Now

Job Description

Amplitude is seeking a Senior Software Engineer to help advance the Nova OLAP engine from our San Francisco campus. In this onsite role, you’ll own critical aspects of query execution, columnar storage, and the distributed compute stack, while shaping caching strategies and cloud infrastructure to boost performance, cost-efficiency, and reliability for a large-scale analytics platform. The compensation for this position ranges from USD 165,000 to 276,000 per year.

Responsibilities

Collaborate across Nova's query execution and distributed compute layers, covering query planning, columnar storage formats, encoding and compression techniques, caching approaches, and cluster-wide resource management.
Develop new capabilities as Nova broadens support for additional warehouse-imported data types, including metrics, profiles, and dimensions.
Ensure Nova remains capable of handling high-throughput automated query workloads, anticipating AI-driven queries and designing for sustained concurrency and programmatic access at scale.
Lead initiatives to reduce infrastructure costs across compute, storage, network, and memory while maintaining or improving latency and throughput.
Profile and optimize JVM performance, focusing on garbage collection tuning, memory management, concurrency, and data layout decisions.
Build guardrails and observability to detect expensive or pathological queries before they affect system performance.
Improve the reliability posture of Nova components you own by identifying failure modes, delivering durable fixes, and enhancing productionIssue detection and response.
Participate in on-call rotations to root-cause incidents and translate one-off fixes into lasting improvements.
Contribute to capacity planning, safe rollout practices, and the operational tooling that keeps Nova healthy.
Design and execute multi-week to multi-month projects within your domain of ownership.
Contribute to technical direction through design docs, architecture discussions, and code reviews.
Pair with and learn from senior and staff engineers, then share insights with peers.
Collaborate with Product, Middleware, Data Pipeline, and other engineering teams to ensure Nova capabilities translate into customer value.

Requirements

At least 3 years of experience in backend or infrastructure engineering with exposure to distributed data systems.
Hands-on work building or extending distributed data systems such as query engines, columnar storage, large-scale data processing frameworks, streaming systems, or storage engines.
Experience reducing cost or improving performance on cloud infrastructure, including compute, storage, and network.
Strong computer science fundamentals in distributed systems (partitioning, replication, consistency, failover), data structures and algorithms, concurrency and multi-threading, and performance optimization.
Production experience with modern cloud infrastructure such as AWS (S3, DynamoDB, EC2), Kafka, Redis/ElastiCache, Kubernetes, Terraform, or strong equivalents.
Proficiency in Java, C++, or Python.
Proven ability to own and deliver significant components of complex systems and collaborate effectively with engineers and partner teams.

Technologies

Nova, AWS, S3, DynamoDB, EC2
Kafka, Redis, ElastiCache, Kubernetes, Terraform
Java, C++, Python
Druid, ClickHouse, Presto/Trino, BigQuery, Snowflake
Arrow, Parquet, ORC

Benefits

Excellent medical, dental, and vision coverage with employee premiums fully paid on select plans.
401(k) retirement plan with employer match up to 1% of eligible pay, up to $2,000 annually.
Flexible time off, paid holidays, and additional benefits.
Generous stipends for wellness, commuting, learning and development, home office needs, and more.
Parental benefits including 12 weeks of paid leave, Carrot Fertility support, and backup child care.
Mental health and wellness support with no-cost access to Modern Health coaching and therapy sessions.
Employee Stock Purchase Program (ESPP).

Who you are

You thrive inside complex distributed systems, understanding data flows, bottlenecks, and how to drive meaningful improvements.
You have hands-on experience building or extending distributed data systems such as query engines, columnar storage, streaming or batch processing frameworks, or storage engines.
You view cost, performance, and reliability as interconnected concerns and communicate tradeoffs clearly.
You articulate technical choices effectively and seek to influence the project through high-quality work and ideas.
You enjoy learning from teammates and helping others through pairing, design reviews, or explaining the rationale behind system designs.

Nice to have

Experience with OLAP or query engines such as Druid, ClickHouse, Presto/Trino, BigQuery, Snowflake, or similar.
JVM specialization including GC tuning, profiling, and memory optimization.
Familiarity with columnar formats and encodings like Arrow, Parquet, ORC, or related formats.
Knowledge of product analytics, experimentation platforms, or event-driven data systems.
Open-source contributions or published work in data infrastructure.

Our values

Humility: operate with empathy and openness to diverse perspectives.
Ownership: take initiative to tackle problems that advance the company’s success.
Growth Mindset: stay persistent, seek feedback, and help others grow.
Customer Centricity: place the customer's success at the center of decisions and actions.

Other fun facts about Amplitude

Our customers consistently rate us as the leading product analytics solution, boasting 23 consecutive quarters on G2.
Significant investments in talent and infrastructure aim to build a powerful AI analytics platform.
We support mentorship, management training, and wellness initiatives across the team.
Amplitude provides charitable giving grants and paid volunteer time off for employees.
Founded in 2012, went public via direct listing in September 2021 and trades under the ticker AMPL.
We operate globally with offices in San Francisco, New York, Vancouver, Amsterdam, London, Paris, Singapore, and Tokyo.
Our Data Monster mascot embodies our love for numbers, charts, and graphs.

View full job page →

Stream Processing Engineer Jobs

Date Engineer Lead(Big Data)

Data Engineer (Ab Initio, OzoneCH, Flink)

Senior Software Engineer

Get Job Alerts