Data Engineer

Principal Data Engineer

We are on the lookout for a Principal Data Engineer to help define and lead the ...

Location

United Kingdom

Salary:

Not provided

Dotdigital

Expiration Date

Until further notice

Requirements

Extensive experience delivering python-based projects in the data engineering space
Extensive experience working with SQL and NoSQL database technologies (e.g. SQL Server, MongoDB & Cassandra)
Proven experience with modern data warehousing and large-scale data processing tools (e.g. Snowflake, DBT, BiqQuery, Clickhouse)
Hands on experience with data orchestration tools like Airflow, Dagster or Prefect
Experience using cloud environments (e.g. Azure, AWS, GCP) to process, store and surface large scale data
Experience using Kafka or similar event-based architectures e.g. (Pub/Sub via AWS SQS, Azure EventHubs, AWS Kinesis)
Strong grasp of data architecture and data modelling principles for both OLAP and OLTP workloads
Capable in the wider software development lifecycle in terms of agile ways of working and continuous integration/deployment of data solutions
Experience as a lead or Principal Engineer on large-scale data initiative or product builds
Demonstrated ability to architect data systems and data structures for high volume, high throughput systems

Job Responsibility

Lead the design and implementation of scalable, secure and resilient data systems across streaming, batch and real-time use cases
Architect data pipelines, model and storage solutions that power analytical and product use cases
using primarily Python and SQL via orchestration tooling that run workloads in the cloud
Leverage AI to automate both data processing and engineering processes
Assure and drive best practices relating to data infrastructure, governance, security and observability
Work with technologists across multiple teams to deliver coherent features and data outcomes
Support the data team to help adopt data engineering principles
Identify, validate and promote new tools and technologies that improve the performance and stability of data services

What we offer

Parental leave
Medical benefits
Paid sick leave
Dotdigital day
Share reward
Wellbeing reward
Wellbeing Days
Loyalty reward

Fulltime

Principal Data Engineer

Atlassian is looking for a Principal Data Engineer to join our Data Engineering ...

Location

United States , San Francisco; Seattle; Austin

Salary:

168700.00 - 271100.00 USD / Year

Atlassian

Expiration Date

Until further notice

Requirements

12+ years of experience in a Data Engineer role as an individual contributor
At least 2 years of experience as a tech lead for a Data Engineering team
Engineer with a track record of driving and delivering large (multi-person or multi-team) and complex efforts
Great communicator and maintain many of the essential cross-team and cross-functional relationships necessary for the team's success
Experience with building streaming pipelines with a micro-services architecture for low-latency analytics
Experience working with varied forms of data infrastructure, including relational databases (e.g. SQL), Spark, and column stores (e.g. Redshift)
Experience building scalable data pipelines using Spark using Airflow scheduler/executor framework or similar scheduling tools
Experience working in a technical environment with the latest technologies like AWS data services (Redshift, Athena, EMR) or similar Apache projects (Spark, Flink, Hive, or Kafka)
Understanding of Data Engineering tools/frameworks and standards to improve the productivity and quality of output for Data Engineers across the team
Industry experience working with large-scale, high-performance data processing systems (batch and streaming) with a "Streaming First" mindset to drive Atlassian's business growth and improve the product experience

Job Responsibility

Own the technical evolution of the data engineering capabilities and be responsible for ensuring solutions are being delivered incrementally, meeting outcomes, and promptly escalating risks and issues
Establish a deep understanding of how things work in data engineering, use this to direct and coordinate the technical aspects of work across data engineering, and systematically improve productivity across the teams
Maintain a high bar for operational data quality and proactively address performance, scale, complexity and security considerations
Drive complex decisions that can impact the work in data engineering
Set the technical direction and balance customer and business needs with long-term maintainability & scale
Understand and define the problem space, and architect solutions
Coordinate a team of engineers towards implementing them, unblocking them along the way if necessary
Lead a team of data engineers through mentoring and coaching, work closely with the engineering manager, and provide consistent feedback to help them manage and grow the team
Work with close counterparts in other departments as part of a multi-functional team, and build this culture in your team

What we offer

Health coverage
Paid volunteer days
Wellness resources

Fulltime

Senior Data Engineer

We are looking for a highly skilled Senior Data Engineer to join our team on a l...

Location

United States , Dallas

Salary:

Not provided

Robert Half

Expiration Date

Until further notice

Requirements

Bachelor's degree in Computer Science, Engineering, or a related discipline
At least 7 years of experience in data engineering
Strong background in designing and managing data pipelines
Proficiency in tools such as Apache Kafka, Airflow, NiFi, Databricks, Spark, Hadoop, Flink, and Amazon S3
Expertise in programming languages like Python, Scala, or Java for data processing and automation
Strong knowledge of both relational and NoSQL databases
Experience with Kubernetes-based data engineering and hybrid cloud environments
Familiarity with data modeling principles, governance frameworks, and quality assurance processes
Excellent problem-solving, analytical, and communication skills

Job Responsibility

Design and implement robust data pipelines and architectures to support data-driven decision-making
Develop and maintain scalable data pipelines using tools like Apache Airflow, NiFi, and Databricks
Implement and manage real-time data streaming solutions utilizing Apache Kafka and Flink
Optimize and oversee data storage systems with technologies such as Hadoop and Amazon S3
Establish and enforce data governance, quality, and security protocols
Manage complex workflows and processes across hybrid and multi-cloud environments
Work with diverse data formats, including Parquet and Avro
Troubleshoot and fine-tune distributed data systems
Mentor and guide engineers at the beginning of their careers

What we offer

Medical, vision, dental, and life and disability insurance
401(k) plan
Free online training

Fulltime

Principal Data Engineer

Atlassian is looking for a Principal Data Engineer to join our Data Engineering ...

Location

United States , San Francisco

Salary:

168700.00 - 271100.00 USD / Year

Atlassian

Expiration Date

Until further notice

Requirements

You have 12+ years of experience in a Data Engineer role as an individual contributor
You have at least 2 years of experience as a tech lead for a Data Engineering team
You are an engineer with a track record of driving and delivering large (multi-person or multi-team) and complex efforts
You are a great communicator and maintain many of the essential cross-team and cross-functional relationships necessary for the team's success
Experience with building streaming pipelines with a micro-services architecture for low-latency analytics
Experience working with varied forms of data infrastructure, including relational databases (e.g. SQL), Spark, and column stores (e.g. Redshift)
Experience building scalable data pipelines using Spark using Airflow scheduler/executor framework or similar scheduling tools
Experience working in a technical environment with the latest technologies like AWS data services (Redshift, Athena, EMR) or similar Apache projects (Spark, Flink, Hive, or Kafka)
Understanding of Data Engineering tools/frameworks and standards to improve the productivity and quality of output for Data Engineers across the team
Industry experience working with large-scale, high-performance data processing systems (batch and streaming) with a 'Streaming First' mindset to drive Atlassian's business growth and improve the product experience

Job Responsibility

Own the technical evolution of the data engineering capabilities and be responsible for ensuring solutions are being delivered incrementally, meeting outcomes, and promptly escalating risks and issues
Establish a deep understanding of how things work in data engineering, use this to direct and coordinate the technical aspects of work across data engineering, and systematically improve productivity across the teams
Maintain a high bar for operational data quality and proactively address performance, scale, complexity and security considerations
Drive complex decisions that can impact the work in data engineering. Set the technical direction and balance customer and business needs with long-term maintainability & scale
Understand and define the problem space, and architect solutions. Coordinate a team of engineers towards implementing them, unblocking them along the way if necessary
Lead a team of data engineers through mentoring and coaching, work closely with the engineering manager, and provide consistent feedback to help them manage and grow the team
Work with close counterparts in other departments as part of a multi-functional team, and build this culture in your team

What we offer

health coverage
paid volunteer days
wellness resources

Fulltime

Senior Data Engineer

Senior Data Engineer position at Checkr, building the data platform to power saf...

Location

United States , San Francisco

Salary:

162000.00 - 190000.00 USD / Year

Checkr

Expiration Date

Until further notice

Requirements

7+ years of development experience in the field of data engineering
5+ years writing PySpark
Experience building large-scale (100s of Terabytes and Petabytes) data processing pipelines - batch and stream
Experience with ETL/ELT, stream and batch processing of data at scale
Strong proficiency in PySpark and Python
Expertise in understanding of database systems, data modeling, relational databases, NoSQL (such as MongoDB)
Experience with big data technologies such as Kafka, Spark, Iceberg, Datalake and AWS stack (EKS, EMR, Serverless, Glue, Athena, S3, etc.)
Knowledge of security best practices and data privacy concerns
Strong problem-solving skills and attention to detail

Job Responsibility

Create and maintain data pipelines and foundational datasets to support product/business needs
Design and build database architectures with massive and complex data, balancing with computational load and cost
Develop audits for data quality at scale, implementing alerting as necessary
Create scalable dashboards and reports to support business objectives and enable data-driven decision-making
Troubleshoot and resolve complex issues in production environments
Work closely with product managers and other stakeholders to define and implement new features

What we offer

Learning and development reimbursement allowance
Competitive compensation and opportunity for professional and personal advancement
100% medical, dental, and vision coverage for employees and dependents
Additional vacation benefits of 5 extra days and flexibility to take time off
Reimbursement for work from home equipment
Lunch four times a week
Commuter stipend
Abundance of snacks and beverages

Fulltime

Data Governance Engineer

The role focuses on deploying and managing enterprise-scale Data Governance prac...

Location

India , Bangalore

Salary:

Not provided

Hewlett Packard Enterprise

Expiration Date

Until further notice

Requirements

7+ years of Data Governance and Data Engineering experience, with significant exposure to enabling Data availability, data discovery, quality & reliability, with appropriate security & access controls in enterprise-scale ecosystem
First level university degree
Experience working with Data governance & metadata management tools (Collibra, Databricks Unity Catalog, Atlan, etc.)
Subject matter expertise of consent management concepts and tools
Demonstrated knowledge of research methodology and the ability to manage complex data requests
Excellent analytical thinking, technical analysis, and data manipulation skills
Proven track record of development of SQL SSIS packages with ETL flow
Experience with AI application deployment governance a plus
Technologies such as MS SQL Server, Databricks, Hadoop, SAP S4/HANA
Experience with SQL databases and building SSIS packages

Job Responsibility

Drive the design and development of Data Dictionary, Lineage, Data Quality, Security & Access Control for Business-relevant data subjects & reports across business domains
Engage with the business users community to enable ease of Data Discovery and build trust in the data through Data Quality & Reliability monitoring with key metrics & SLAs defined
Supports the development and sustaining of Data subjects in the Database layer to enable BI dashboards and AI solutions
Drives the engagement and alignment with the HPE IT/CDO team on Governance initiatives, including partnering with functional teams across the business
Test, validate and assure the quality of complex AI-powered product features
Partner with a highly motivated and talented set of colleagues
Be a motivated, self-starter who can operate with minimal handholding
Collaborate across teams and time zones, demonstrating flexibility and accountability.

What we offer

Comprehensive suite of benefits supporting physical, financial and emotional wellbeing
Specific programs to help achieve career goals
Comprehensive inclusion and flexibility to manage work and personal needs.

Fulltime

Data Engineering Lead

Data Engineering Lead a strategic professional who stays abreast of developments...

Location

India , Pune

Salary:

Not provided

Citi

Expiration Date

Until further notice

Requirements

10-15 years of hands-on experience in Hadoop, Scala, Java, Spark, Hive, Kafka, Impala, Unix Scripting and other Big data frameworks
4+ years of experience with relational SQL and NoSQL databases: Oracle, MongoDB, HBase
Strong proficiency in Python and Spark Java with knowledge of core spark concepts (RDDs, Dataframes, Spark Streaming, etc) and Scala and SQL
Data Integration, Migration & Large Scale ETL experience (Common ETL platforms such as PySpark/DataStage/AbInitio etc.) - ETL design & build, handling, reconciliation and normalization
Data Modeling experience (OLAP, OLTP, Logical/Physical Modeling, Normalization, knowledge on performance tuning)
Experienced in working with large and multiple datasets and data warehouses
Experience building and optimizing ‘big data’ data pipelines, architectures, and datasets
Strong analytic skills and experience working with unstructured datasets
Ability to effectively use complex analytical, interpretive, and problem-solving techniques
Experience with Confluent Kafka, Redhat JBPM, CI/CD build pipelines and toolchain – Git, BitBucket, Jira

Job Responsibility

Strategic Leadership: Define and execute the data engineering roadmap for Global Wealth Data, aligning with overall business objectives and technology strategy
Team Management: Lead, mentor, and develop a high-performing, globally distributed team of data engineers, fostering a culture of collaboration, innovation, and continuous improvement
Architecture and Design: Oversee the design and implementation of robust and scalable data pipelines, data warehouses, and data lakes, ensuring data quality, integrity, and availability for global wealth data
Technology Selection and Implementation: Evaluate and select appropriate technologies and tools for data engineering, staying abreast of industry best practices and emerging trends specific to wealth management data
Performance Optimization: Continuously monitor and optimize data pipelines and infrastructure for performance, scalability, and cost-effectiveness, ensuring optimal access to global wealth data
Collaboration: Partner with business stakeholders, data scientists, portfolio managers, and other technology teams to understand data needs and deliver effective solutions that support investment strategies and client reporting
Data Governance: Implement and enforce data governance policies and procedures to ensure data quality, security, and compliance with relevant regulations, particularly around sensitive financial data

Fulltime

Location: 100% remote; Years’ Experience: 10+ years professional experience; Edu...

Location

United States

Salary:

Not provided

Sparibis

Expiration Date

Until further notice

Requirements

10+ years of IT experience focusing on enterprise data architecture and management
Experience with Databricks, Structured Streaming, Delta Lake concepts, and Delta Live Tables required
Experience with ETL and ELT tools such as SSIS, Pentaho, and/or Data Migration Services
Advanced level SQL experience (Joins, Aggregation, Windowing functions, Common Table Expressions, RDBMS schema design, Postgres performance optimization)
Must be able to obtain a Public Trust security clearance
Bachelor degree required
Experience in Conceptual/Logical/Physical Data Modeling & expertise in Relational and Dimensional Data Modeling
Additional experience with Spark, Spark SQL, Spark DataFrames and DataSets, and PySpark
Data Lake concepts such as time travel and schema evolution and optimization
Experience leading and architecting enterprise-wide initiatives specifically system integration, data migration, transformation, data warehouse build, data mart build, and data lakes implementation / support

Job Responsibility

Plan, create, and maintain data architectures, ensuring alignment with business requirements
Obtain data, formulate dataset processes, and store optimized data
Identify problems and inefficiencies and apply solutions
Determine tasks where manual participation can be eliminated with automation
Identify and optimize data bottlenecks, leveraging automation where possible
Create and manage data lifecycle policies (retention, backups/restore, etc)
In-depth knowledge for creating, maintaining, and managing ETL/ELT pipelines
Create, maintain, and manage data transformations
Maintain/update documentation
Create, maintain, and manage data pipeline schedules

Data Engineer - Security (Kafka Experience)

NTT DATA

Location:
India , Remote

Category:
IT - Software Development

Contract Type:
Not provided

Salary:

Job Description:

Job Responsibility:

Requirements:

Additional Information:

Job Posted:
April 27, 2026

Looking for more opportunities? Search for other job offers that match your skills and interests.

Similar Jobs for Data Engineer - Security (Kafka Experience)

Principal Data Engineer

Principal Data Engineer

Senior Data Engineer

Principal Data Engineer

Senior Data Engineer

Data Governance Engineer

Data Engineering Lead

Data Engineer

Data Engineer - Security (Kafka Experience)

NTT DATA

Location:India , Remote

Category:IT - Software Development

Contract Type:Not provided

Salary:

Job Description:

Job Responsibility:

Requirements:

Additional Information:

Job Posted:April 27, 2026

Looking for more opportunities? Search for other job offers that match your skills and interests.

Similar Jobs for Data Engineer - Security (Kafka Experience)

Principal Data Engineer

Principal Data Engineer

Senior Data Engineer

Principal Data Engineer

Senior Data Engineer

Data Governance Engineer

Data Engineering Lead

Data Engineer

Location:
India , Remote

Category:
IT - Software Development

Contract Type:
Not provided

Job Posted:
April 27, 2026