CrawlJobs Logo
NTT DATA Logo NTT DATA · IT - Software Development

Data Engineer - Security (Kafka Experience)

India, Remote · Job Posted April 27, 2026
Apply Position
Job Link Share

Job Description

The Data Engineer - Security role focuses on designing and operating large-scale event-streaming platforms using Kafka. The ideal candidate will have strong expertise in data ingestion, AWS data lakes, and proficiency in Python and PySpark. This position offers the opportunity to work in a collaborative environment with a focus on innovation and client success.

Job Responsibility

  • Designing and operating large-scale event-streaming platforms using Kafka
  • API-first data ingestion
  • Building/operating S3-based lakes
  • Designing and optimizing Glue jobs using PySpark/DynamicFrames
  • Writing clean, parameterized, idempotent DAGs
  • Building ELT models in Snowflake

Requirements

  • Kafka-Strong expertise in Kafka (4-5 years), with hands-on experience designing and operating large-scale, highly available event-streaming platforms, including partitioning strategies, consumer group optimization, schema management, and performance tuning
  • API-first data ingestion. Strong hands-on pulling data from REST/GraphQL APIs with auth (OAuth2, API keys), pagination, rate limits, retries/backoff, and webhooks
  • strong Python skills to normalize/enrich data and land it cleanly into S3 (schema, partitioning, Parquet)
  • AWS data lake, end to end. Comfortable building/operating S3-based lakes with layered zones (raw → harmonized → conformed → modeled), Glue Data Catalog, IAM/Secrets Manager, VPC endpoints, encryption, lifecycle/versioning, and cost/perf best practices (file sizing, compaction)
  • AWS Glue + PySpark expert. Designs and optimizes Glue jobs using PySpark/DynamicFrames, bookmarks for incremental loads, dependency packaging, robust error handling, logging/metrics, and unit tests
  • knows how to tune jobs for scale and cost
  • Airflow orchestration. Writes clean, parameterized, idempotent DAGs (sensors, SLAs, retries, alerts), manages dependencies across pipelines, and uses Git-based CI/CD to promote changes safely
  • Snowflake proficiency. Builds ELT models (staging/ODS/marts), tunes performance (warehouse sizing, clustering, micro-partitions, caching), uses Streams/Tasks/Snowpipe for CDC, and follows solid RBAC and data governance practices
NTT DATA - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Data Engineer - Security (Kafka Experience)

8 matching positions

Data Engineer (Kafka)

Altamira is seeking a Data Engineer to design, build, and operate high-performan...
Location
Location
United States , Dayton, OH
Salary
Salary:
Not provided
altamiracorp.com Logo
Altamira Technologies
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Active TS/SCI clearance
  • Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience)
  • Experience in data engineering, distributed systems, or backend engineering roles
  • Hands-on experience with Apache Kafka in production environments
  • Experience building and supporting real-time data pipelines
  • Strong proficiency in Java, Python, Scala, or similar programming languages
  • Experience working in AWS or hybrid cloud environments
  • Strong Linux systems administration and troubleshooting skills
  • Ability to work effectively in secure, mission-focused environments
Job Responsibility
Job Responsibility
  • Design, deploy, and operate Apache Kafka clusters in classified and hybrid environments
  • Build and maintain reliable, scalable, and secure data streaming pipelines
  • Develop and optimize producers, consumers, and stream processing applications
  • Configure and manage topics, partitions, replication, and retention policies
  • Monitor, tune, and troubleshoot Kafka performance, availability, and latency
  • Integrate streaming platforms with databases, storage systems, and analytics tools
  • Implement data governance, retention, and access control policies
  • Automate deployment and management of streaming infrastructure
  • Collaborate with platform, infrastructure, and application teams to support data requirements
  • Support system accreditation, compliance, and security requirements
  • Fulltime
Read More
Arrow Right

Data Security Backend Engineer - Java

Engineer the future of global finance. At Citi, our Tech team doesn’t just suppo...
Location
Location
United Kingdom , Belfast
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proven relevant experience in software development, with a strong focus on Java and microservices
  • Demonstrated experience working in a large, complex, and/or global environment, preferably within Financial Services
  • Proven ability to manage technical projects or significant project components effectively
  • Consistently demonstrates clear and concise written and verbal communication skills, especially in technical contexts
  • Comprehensive understanding of software design principles, development methodologies, and analytics tools to identify best practices
  • Demonstrated analytic and diagnostic skills for troubleshooting and problem-solving
  • Ability to work effectively in a matrix environment and partner with virtual teams
  • Ability to work independently, prioritize tasks, and take ownership of various parts of a project or initiative from conception to deployment
  • Ability to work under pressure and manage tight deadlines or unexpected changes in expectations or requirements in a fast-paced development environment
  • Proven track record of contributing to operational process change and improvement through technical solutions
Job Responsibility
Job Responsibility
  • Develop, deploy, and maintain microservice applications primarily using Java and Spring Boot, and occasionally Python, focusing on automation and cost reduction initiatives
  • Contribute to the technical design, development, and testing of new features and functionalities within the identity innovation space
  • Collaborate with business and development teams to translate engineering requirements into technical specifications and assist in defining application and system processes
  • Ensure the seamless integration of microservices with existing systems and cloud infrastructure
  • Implement solutions and processes in accordance with audit initiatives and requirements, consulting with Business Information Security Officers (BISOs) and Technology Information Security Officers (TISOs) as needed
  • Exhibit an in-depth understanding of Java development best practices, microservices architecture, and cloud-native principles
  • Participate in code reviews, mentor junior team members, and contribute to continuous improvement of development processes
  • Appropriately assess risk when making technical decisions, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients, and assets, by driving compliance with applicable laws, rules, and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct, and business practices, and escalating, managing, and reporting control issues with transparency
What we offer
What we offer
  • 27 days annual leave (plus bank holidays)
  • A discretional annual performance related bonus
  • Private Medical Care & Life Insurance
  • Employee Assistance Program
  • Pension Plan
  • Paid Parental Leave
  • Special discounts for employees, family, and friends
  • Access to an array of learning and development resources
  • Fulltime
Read More
Arrow Right

Senior Data Engineer - Data Platform

We are looking for a Senior Data Engineer - Data Platform to join our Data & AI ...
Location
Location
France , Paris
Salary
Salary:
Not provided
doctolib.fr Logo
Doctolib
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • More than 7 years of experience as Site Reliability Engineer, Data Ops, Data Platform Engineer or in a similar role, with a proven track record of building and maintaining complex data infrastructures
  • Strong proficiency in data engineering and infrastructure tools and technologies, such as stream and events processing (Kafka, PubSub, Firehose) and Kubernetes
  • Expertise in programming languages like Python
  • Familiar with cloud infrastructure and services, preferably AWS, Azure, or GCP, and have experience with infrastructure-as-code tools such as Terraform
  • Excellent problem-solving skills with a focus on identifying and resolving data infrastructure bottlenecks and performance issues
Job Responsibility
Job Responsibility
  • Design and implement a scalable and reliable data infrastructure that supports the collection, processing, storage, and analysis of large-scale datasets while pushing security and privacy best practices
  • Build and maintain data pipelines that efficiently extract, transform, and load data from various sources into our data warehouse
  • Implement automation and orchestration tools to streamline infrastructure provisioning, data workflows, reduce manual effort, and improve operational efficiency
  • Monitor data platform for performance and reliability, identify and troubleshoot issues, and implement proactive solutions to ensure data quality and availability
  • Streamline and monitor platform costs, identify optimizations and saving opportunities while collaborating with data engineers, data scientists, and other stakeholders
What we offer
What we offer
  • Free comprehensive health insurance for you and your children
  • Parent Care Program: receive one additional month of leave on top of the legal parental leave
  • Free mental health and coaching services through our partner Moka.care
  • For caregivers and workers with disabilities, a package including an adaptation of the remote policy, extra days off for medical reasons, and psychological support
  • Work from EU countries and the UK for up to 10 days per year, thanks to our flexibility days policy
  • Up to 14 days of RTT
  • A subsidy from the work council to refund part of the membership to a sport club or a creative class
  • Lunch voucher with Swile card
  • Fulltime
Read More
Arrow Right

Data Engineer Big Data Technology

The Applications Development Intermediate Programmer Analyst is an intermediate ...
Location
Location
Singapore , Singapore
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 4-6 years of hands-on experience in Hadoop, Scala, Java, Spark, Hive, Kafka, Impala, Unix Scripting and other Big data frameworks
  • 4+ years of experience with relational SQL and NoSQL databases: Oracle, MongoDB, HBase
  • Strong proficiency in Python and Spark Java with knowledge of core spark concepts (RDDs, Dataframes, Spark Streaming, etc) and Scala and SQL
  • Data Integration, Migration & Large Scale ETL experience (Common ETL platforms such as PySpark/DataStage/)
  • 4-6 years of relevant experience in the Financial Service industry
  • Experienced in working with large and multiple datasets and data warehouses
  • Experience building and optimizing ‘big data’ data pipelines, architectures, and datasets
  • Strong analytic skills and experience working with unstructured datasets
  • Ability to effectively use complex analytical, interpretive, and problem-solving techniques
  • Experience with Confluent Kafka, Redhat JBPM, CI/CD build pipelines and toolchain – Git, BitBucket, Jira
Job Responsibility
Job Responsibility
  • Consult with users, clients, and other technology groups on issues, and recommend programming solutions, install, and support customer exposure systems
  • Apply fundamental knowledge of programming languages for design specifications
  • Analyze applications to identify vulnerabilities and security issues, as well as conduct testing and debugging
  • Serve as advisor or coach to new or lower level analysts
  • Identify problems, analyze information, and make evaluative judgements to recommend and implement solutions
  • Resolve issues by identifying and selecting solutions through the applications of acquired technical experience and guided by precedents
  • Has the ability to operate with a limited level of direct supervision
  • Can exercise independence of judgement and autonomy
  • Acts as SME to senior stakeholders and /or other team members
  • Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency
  • Fulltime
Read More
Arrow Right

Software engineer 2 / Senior Software engineer - Azure Data

Microsoft's Azure Data engineering team is leading the transformation of analyti...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 3+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
  • OR equivalent experience
  • Experience with the Azure stack including Storage, Compute, Networking, Fabric, Purview, Synapse, AKS, DevOps, Data Factory, or Power BI
  • Experience with big data technologies such as Spark, Kafka, Hadoop, or HBase
  • Experience building data lake or data engineering products, tools, or pipelines
  • Familiarity with container-based architectures (Docker, Kubernetes)
  • Ability to debug complex distributed systems on Linux and/or Windows platforms
Job Responsibility
Job Responsibility
  • Write extensible, maintainable code in C#, Java, Scala, or Python for Fabric Materialized Lake View services and HDInsight components
  • Use AI tools and coding best practices across the development lifecycle
  • Design data refresh, scheduling, and query optimisation features with minimal supervision
  • Review code from teammates for correctness, test coverage, security risks, and adherence to team standards
  • Coach junior engineers through code reviews
  • Debug complex issues in distributed systems running on Azure, Linux, and Windows
  • Run live site operations on a rotational, on-call basis
  • Integrate logging and instrumentation to gather telemetry on system health, performance, reliability, and security
  • Work with product managers, technical leads, and partners across geographies to define customer requirements for Materialized Lake View features
  • Fulltime
Read More
Arrow Right
New

Data Engineer

We are looking for a Data Engineer to support and enhance critical data operatio...
Location
Location
United States , Greenville
Salary
Salary:
Not provided
https://www.roberthalf.com Logo
Robert Half
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience working in data engineering or data platform operations roles
  • Strong hands-on knowledge of Python for scripting, automation, and operational support
  • Experience managing or supporting Apache Kafka environments and related streaming data workflows
  • Familiarity with Snowflake administration, performance tuning, and resource management
  • Solid understanding of ETL processes, including data ingestion, transformation, and delivery concepts
  • Experience working with AWS services in support of modern data infrastructure
  • Knowledge of Terraform or similar infrastructure-as-code tools for environment management and automation
Job Responsibility
Job Responsibility
  • Oversee the health and performance of data pipelines that run across Snowflake, Kafka, and connected platforms
  • Investigate operational issues affecting data ingestion, transformation, or downstream delivery and drive timely resolution
  • Maintain stable batch and streaming processes by improving resiliency, uptime, and overall execution efficiency
  • Administer Snowflake resources, including warehouses, databases, permissions, and usage optimization
  • Manage Kafka infrastructure by tuning clusters, topics, partitions, and consumer group behavior for reliable throughput
  • Create and maintain automated solutions for deployment, monitoring, failure recovery, and routine workflow support
  • Develop operational scripts and utilities using Python, Bash, and related tools to reduce manual effort and improve consistency
  • Contribute to CI/CD practices that strengthen the release and maintenance process for data infrastructure
  • Partner with engineering and analytics teams to improve pipeline design, data performance, and delivery accuracy
  • Support data governance, security, compliance, and data quality standards through validation checks and alerting frameworks
What we offer
What we offer
  • medical
  • vision
  • dental
  • life and disability insurance
  • 401(k) plan
  • Fulltime
Read More
Arrow Right

Data Engineer

We are looking for an experienced Data Engineer to join a team delivering modern...
Location
Location
United States , Poughkeepsie
Salary
Salary:
Not provided
https://www.roberthalf.com Logo
Robert Half
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Hands-on experience building data engineering solutions with Databricks and Apache Spark
  • Strong programming ability in Python, including development of ETL and data transformation workflows
  • Knowledge of lakehouse and big data technologies such as Delta Lake, Apache Hadoop, and Apache Kafka
  • Experience working with Azure Data Lake Storage Gen2 or comparable cloud-based data storage platforms
  • Ability to optimize distributed data processing jobs and troubleshoot performance issues in Spark environments
  • Familiarity with data governance, data quality, and security practices for enterprise data platforms
  • Comfortable working independently and collaborating with cross-functional teams in an agile delivery model
  • Proven ability to analyze technical problems, break them into manageable components, and implement effective solutions.
Job Responsibility
Job Responsibility
  • Create and support scalable data pipelines in Databricks using Spark technologies such as PySpark or Scala to process and deliver high-quality data
  • Develop lakehouse architectures on Azure Data Lake Storage Gen2 and ensure strong integration with Databricks for efficient data management
  • Establish and monitor data quality controls and governance practices within the platform using validation methods and Delta Lake capabilities
  • Investigate pipeline and application inefficiencies, then implement tuning strategies to improve Spark and Databricks performance
  • Work closely with analysts and other stakeholders to translate business data needs into refined, analytics-ready datasets
  • Automate ingestion, transformation, testing, and release processes, including integration with CI/CD workflows where appropriate
  • Provide guidance to less experienced engineers by sharing best practices for Databricks development, optimization, and support
  • Maintain clear technical documentation for notebooks, workflows, data models, configurations, and operational procedures
  • Protect data assets by applying security controls and compliance standards across the Databricks environment
  • Contribute to design sessions, solve complex data issues, and uphold change management and data integrity standards while delivering large assignments on schedule
What we offer
What we offer
  • Medical, vision, dental, and life and disability insurance
  • enrollment in company 401(k) plan
  • Fulltime
Read More
Arrow Right

Azure Data Engineer

We are currently seeking a Azure Data Engineer to join our team in Bangalore, Ka...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
nttdata.com Logo
NTT DATA
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 6+ years of experience supporting Software Engineering, Data Engineering, or Data Analytics projects
  • 5+ years of demonstrated production experience in core data platforms such as Databricks and Azure related data services
  • 5+ years of expertise in SQL and Python (Spark) as data manipulation languages
  • 5+ years of hands-on knowledge of Cloud and Distributed Data Storage, including ADLS
  • 5+ years of Data integration technologies, encompassing Spark, Kafka, eventing/streaming, Azure DataFactory, Azure Synapse, and Databricks
  • Showcase professional written and verbal communication skills to effectively convey complex technical concepts
  • Undergraduate or Graduate degree preferred
Job Responsibility
Job Responsibility
  • Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical stack
  • Provide thought leadership by recommending the most appropriate technologies and solutions for a given use case, covering the entire spectrum from the application layer to infrastructure
  • Demonstrate proficiency in coding skills, utilizing languages such as Python, Java, and Scala to efficiently move solutions into production while prioritizing performance, security, scalability, and robust data integrations
  • Generate comprehensive solution documentation, including sequence diagrams, class hierarchies, logical system views, etc.
  • Adhere to Agile practices throughout the solution development process
  • Design, build, and deploy databases, data pipelines and data stores to support organizational requirements
Read More
Arrow Right