Data Engineer - Streaming (WkStream 2 - Kafka) Job at NTT DATA (Bangalore)

Job Description

The Data Engineer - Streaming role involves designing and implementing PySpark Structured Streaming applications for data ingestion into Apache Iceberg tables. Candidates should have a strong background in Apache Kafka and PySpark, with at least 4 years of experience. Responsibilities include building ingestion pipelines, ensuring compliance with technical constraints, and conducting thorough testing. This position offers an opportunity to work in a dynamic environment with a focus on innovative data solutions.

Job Responsibility

Design and implement a PySpark Structured Streaming application that reads from Confluent Kafka topics, parses JSON and Avro payloads, applies schema mappings, and writes atomically to Iceberg tables using the Iceberg Spark runtime and foreachBatch micro-batch pattern
Ensure all functionality relies exclusively on public Apache-supported APIs — Apache Spark, Apache Kafka, and Apache Iceberg — with no unsupported Confluent connectors or proprietary sinks
Configure Kafka source parameters: bootstrap servers, consumer group IDs, offset management (startingOffsets, failOnDataLoss), checkpoint paths, and trigger intervals
Implement PII detection and Protegrity tokenization hooks within the ingestion pipeline before data lands in the Iceberg Bronze layer
Write comprehensive unit and integration tests: row count validation, schema conformance checks, Kafka offset commit verification, and data comparison against the source topic
Support PNC UAT — walk PNC engineers through the code, demonstrate no unsupported connectors are used, and address review findings

Requirements

Apache Kafka – Producer & Consumer: 4+ years of hands-on experience with Apache Kafka, including both producer and consumer development in PySpark, Java, or Scala
Deep understanding of Kafka internals: topics, partitions, consumer groups, offsets, rebalancing, and exactly-once delivery semantics
Experience with Confluent Kafka: schema registry, Avro/JSON serialisation, and Confluent Cloud or on-prem cluster configuration
Proven ability to build ingestion pipelines without relying on unsupported or third-party sink connectors — using only native Kafka consumer APIs and Spark integration
Familiarity with Kafka Connect architecture to evaluate trade-offs and articulate why application-level ingestion is preferred in constrained environments
PySpark Structured Streaming: Strong practical experience with PySpark Structured Streaming: Kafka source, file source, foreachBatch, output modes (append/update/complete), and checkpoint management
Experience tuning streaming micro-batch trigger intervals, watermarking, and late data handling for production workloads
Hands-on experience writing streaming data directly to Apache Iceberg tables using the Iceberg Spark runtime
Ability to implement robust error handling: dead-letter queues, parse error isolation, and recovery from checkpoint failures
Data Engineering & Iceberg: Working knowledge of Apache Iceberg: catalog configuration, schema definition, append writes, and partition strategy for event and log data
Familiarity with S3-compatible object storage as an Iceberg warehouse destination
Understanding of medallion architecture — ability to correctly land streaming data in the Bronze layer with appropriate schema governance

NTT DATA - All Job Offers

Select Country

Data Engineer - Streaming (WkStream 2 - Kafka)

Job Description

Job Responsibility

Requirements

Looking for more opportunities?

Data Engineer - Streaming (WkStream 2 - Kafka)

Grad Pharmacist

Grad Pharmacist

Pharmacy Intern

Pharmacy Intern

Grad Pharmacist

Sales Support Administrator

Grad Pharmacist

Bar Manager

Our AI answers in your language