Data Engineering Lead Job at 3Pillar Global

Data Engineering Lead

Role: Data Engineering Lead. Location: New York, NY. FTE ONLY. Must Have Technic...

Location

United States , New York

Salary:

138000.00 USD / Year

Realign

Expiration Date

Until further notice

Requirements

AWS Data Engineering Services (EMR/Glue, Redshift, Aurora, S3, Lambda)
Spark
Python
Collibra
Snowflake/Databricks
Tableau
4–15 years of professional data engineering experience
Strong Python, SQL, and Spark (PySpark) skills, and/or Kafka
Snowflake (Snowpipe, Tasks, Streams) as a complementary warehouse
Databricks (Delta formats, workflows, cataloging) or equivalent Spark platforms

Job Responsibility

Ingest and model data from APIs, files/SFTP, and relational sources
implement layered architectures (raw/clean/serving) using PySpark/SQL and dbt, Python
Design and operate pipelines with Prefect (or Airflow), including scheduling, retries, parameterization, SLAs, and well documented runbooks
Build on cloud data platforms, leveraging S3/ADLS/GCS for storage and a Spark platform (e.g., Databricks or equivalent) for compute
manage jobs, secrets, and access
Publish governed data services and manage their lifecycle with Azure API Management (APIM) authentication/authorization, policies, versioning, quotas, and monitoring
Enforce data quality and governance through data contracts, validations/tests, lineage, observability, and proactive alerting
Optimize performance and cost via partitioning, clustering, query tuning, job sizing, and workload management
Uphold security and compliance (e.g., PII handling, encryption, masking) in line with firm standards
Collaborate with stakeholders (analytics, AI engineering, and business teams) to translate requirements into reliable, production ready datasets

Fulltime

Data Engineering Lead

We’re looking for a Lead Data Engineer who is ready to both lead and build. This...

Location

Salary:

Not provided

5CA

Expiration Date

Until further notice

Requirements

Located in Albania, Bosnia and Herzegovina, Bulgaria, Greece, Hungary, Kosovo, North Macedonia, Poland, Romania, Serbia, South Africa, or Tunisia
Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, or a related field
Minimum 7 years’ experience in data engineering
At least 2 years of leadership experience managing or mentoring engineers
Proven experience in designing, building, and maintaining scalable ETL/ELT pipelines and data platforms
Proficiency in SQL, Python, and data modelling techniques
Experience with cloud-based data architectures (e.g., Azure, AWS, or GCP)
Strong understanding of data governance, quality frameworks, and security best practices
Experience integrating data from multiple sources, including APIs and third-party systems
Familiarity with modern data tools and technologies (e.g., Databricks, Snowflake, Apache Spark, Airflow, Microsoft Fabric)

Job Responsibility

Lead the strategy, design, and implementation of scalable, high-performance data pipelines, platforms, and architectures
Manage the Data Engineering team’s backlog, priorities, and capacity to ensure timely delivery of high-quality outputs
Collaborate with data scientists, analysts, and product teams to understand data requirements and deliver fit-for-purpose solutions
Oversee and contribute to the development of ETL/ELT processes, data models, and data integration frameworks
Ensure data quality, consistency, and governance across all systems and datasets
Monitor and optimise data infrastructure for performance, scalability, and cost efficiency
Define and enforce engineering best practices, coding standards, and security guidelines
Evaluate and adopt new technologies, tools, and frameworks that can improve data engineering capabilities
Provide technical leadership and mentorship to data engineers, fostering a high-performance culture
Troubleshoot complex data issues and drive root cause analysis and remediation

What we offer

A position at a fast-paced international company with ambitious gaming, e-commerce, and tech clients
A diverse and inclusive culture with people from 80+ countries, speaking 25+ languages - where we celebrate everyone's uniqueness
Innovative digital tools, and continuous opportunities for learning and development
Various learning and career development initiatives throughout the year
Fun employee engagement activities and participation in 5CA employee-led communities such as 5CA Connect, Pride, 5CA Gamers, Women of 5CA, to name a few

Fulltime

Data Engineering Lead

The next step of your career starts here, where you can bring your own unique mi...

Location

India , Bangalore

Salary:

Not provided

Metyis

Expiration Date

Until further notice

Requirements

7-10 years of professional hands-on experience
A broad practice in multiple software engineering fields
Experience on managing & leading data engineering / warehousing projects
Must have experience on Python, SQL and Distributed programming languages, preferable Spark
Experience working on Cloud, Azure is a plus
Experience working setting up data lakes using entire Big Data and DWH ecosystem
Experience on data workflows and ETL, Apache Airflow is a plus
Comfortable with Unix OS type systems, Bash and Linux
Basic knowledge of webservices and APIs
Basic knowledge of Containers and Docker as platform

Job Responsibility

Independently lead & manage execution of data engineering projects
Engineer complete technical solutions to solve concrete business challenges in the areas of digital marketing, eCommerce, Business Intelligence and self-service analytics
Collect functional and non-functional requirements, consider technical environments, business constraints and enterprise organizations
Support our clients in executing their Big Data strategies by designing and building operational data platforms: ETL pipelines, data anonymization pipelines, data lakes, near real-time streaming data hubs, web services, training and scoring machine learning models
Troubleshoot and quality check work done by team members
Collaborate closely with partners, strategy consultants and data scientists in a flat and agile organization where personal initiative is highly valued
Share data engineering knowledge by giving technical trainings
Communicate and interact with clients at the executive level
Guide and mentor team members

What we offer

Interact with C-level at our clients on regular basis to drive their business towards impactful change
Lead your team in creating new business solutions
Seize opportunities at the client and at Metyis in our entrepreneurial environment
Become part of a fast-growing international and diverse team

Data Engineering Lead

The next step of your career starts here, where you can bring your own unique mi...

Location

India , Gurgaon

Salary:

Not provided

Metyis

Expiration Date

Until further notice

Requirements

7-10 years of professional hands-on experience
A broad practice in multiple software engineering fields
Experience on managing & leading data engineering / warehousing projects
Must have experience on Python, SQL and Distributed programming languages, preferable Spark
Experience working on Cloud, Azure is a plus
Experience working setting up data lakes using entire Big Data and DWH ecosystem
Experience on data workflows and ETL, Apache Airflow is a plus
Comfortable with Unix OS type systems, Bash and Linux
Basic knowledge of webservices and APIs
Basic knowledge of Containers and Docker as platform

Job Responsibility

Independently lead & manage execution of data engineering projects
Engineer complete technical solutions to solve concrete business challenges in the areas of digital marketing, eCommerce, Business Intelligence and self-service analytics
Collect functional and non-functional requirements, consider technical environments, business constraints and enterprise organizations
Support our clients in executing their Big Data strategies by designing and building operational data platforms: ETL pipelines, data anonymization pipelines, data lakes, near real-time streaming data hubs, web services, training and scoring machine learning models
Troubleshoot and quality check work done by team members
Collaborate closely with partners, strategy consultants and data scientists in a flat and agile organization where personal initiative is highly valued
Share data engineering knowledge by giving technical trainings
Communicate and interact with clients at the executive level
Guide and mentor team members

What we offer

Interact with C-level at our clients on regular basis to drive their business towards impactful change
Lead your team in creating new business solutions
Seize opportunities at the client and at Metyis in our entrepreneurial environment
Become part of a fast-growing international and diverse team

Data Engineering Lead

We are looking for a highly skilled Data Engineering Manager to join our Data En...

Location

United States

Salary:

215000.00 - 240000.00 USD / Year

YipitData

Expiration Date

Until further notice

Requirements

6+ years of professional experience in Data Engineering or equivalent technical roles (e.g., data architecture, big data development, or ETL engineering)
2+ years of managerial experience, including mentoring, team leadership, and supporting delivery
Strong expertise in SQL and distributed data systems
Proficiency with PySpark and Databricks for processing and scaling large datasets
Hands-on experience with Airflow for pipeline orchestration (Dagster/dbt a plus)
Proven track record of delivering in fast-paced, deadline-driven environments with minimal oversight
Strong problem-solving skills and ability to translate business needs into scalable technical solutions
Excellent communication and collaboration skills with both technical and non-technical stakeholders

Job Responsibility

Own the design, build, and optimization of end-to-end data pipelines that power our vendor universe
Establish and enforce best practices in data modeling, orchestration, and system reliability
Collaborate with product, engineering, and business stakeholders to translate requirements into robust, scalable data solutions
Work extensively with Databricks and Airflow for large-scale data processing and orchestration
Troubleshoot and resolve complex pipeline issues to ensure reliability and performance
Contribute to the team’s technical strategy, helping drive improvements in scalability, performance, and efficiency
Lead, mentor, and support engineers through challenges, code reviews, and project execution

What we offer

flexible work hours
flexible vacation
a generous 401K match
parental leave
team events
wellness budget
learning reimbursement
equity

Fulltime

Data Engineering Lead

Data Engineering Lead (Contract) to take technical ownership of an enterprise Az...

Location

United Kingdom , Hampshire

Salary:

Not provided

Eutopia

Expiration Date

Until further notice

Requirements

hands-on experience
technical ownership of an enterprise Azure data platform
building, leading, and solving real production challenges
experience with Azure Databricks
experience with SQL data models
experience with Power BI
experience with Terraform (Infrastructure as Code)
experience with CI/CD using GitHub Actions
responsible for stabilising, optimising, and evolving a business-critical platform

Job Responsibility

take technical ownership of an enterprise Azure data platform
building, leading, and solving real production challenges from day one
take ownership of a live Azure-based data platform
responsible for stabilising, optimising, and evolving it

Data Engineering Lead

The Engineering Lead Analyst is a senior level position responsible for leading ...

Location

Singapore , Singapore

Salary:

Not provided

Citi

Expiration Date

Until further notice

Requirements

10-15 years of hands-on experience in Hadoop, Scala, Java, Spark, Hive, Kafka, Impala, Unix Scripting and other Big data frameworks
4+ years of experience with relational SQL and NoSQL databases: Oracle, MongoDB, HBase
Strong proficiency in Python and Spark Java with knowledge of core spark concepts (RDDs, Dataframes, Spark Streaming, etc) and Scala and SQL
Data Integration, Migration & Large Scale ETL experience (Common ETL platforms such as PySpark/DataStage/AbInitio etc.) - ETL design & build, handling, reconciliation and normalization
Data Modeling experience (OLAP, OLTP, Logical/Physical Modeling, Normalization, knowledge on performance tuning)
Experienced in working with large and multiple datasets and data warehouses
Experience building and optimizing ‘big data’ data pipelines, architectures, and datasets
Strong analytic skills and experience working with unstructured datasets
Ability to effectively use complex analytical, interpretive, and problem-solving techniques
Experience with Confluent Kafka, Redhat JBPM, CI/CD build pipelines and toolchain – Git, BitBucket, Jira

Job Responsibility

Define and execute the data engineering roadmap for Global Wealth Data, aligning with overall business objectives and technology strategy
Lead, mentor, and develop a high-performing, globally distributed team of data engineers, fostering a culture of collaboration, innovation, and continuous improvement
Oversee the design and implementation of robust and scalable data pipelines, data warehouses, and data lakes, ensuring data quality, integrity, and availability for global wealth data
Evaluate and select appropriate technologies and tools for data engineering, staying abreast of industry best practices and emerging trends specific to wealth management data
Continuously monitor and optimize data pipelines and infrastructure for performance, scalability, and cost-effectiveness
Partner with business stakeholders, data scientists, portfolio managers, and other technology teams to understand data needs and deliver effective solutions
Implement and enforce data governance policies and procedures to ensure data quality, security, and compliance with relevant regulations

What we offer

Equal opportunity employer commitment
Accessibility and accommodation support
Global workforce benefits

Fulltime

Data Engineering Lead

Job description

Location

United States , Mechanicsville

Salary:

Not provided

Yotta Tech Ports

Expiration Date

Until further notice

Requirements

Bachelor’s degree in Computer Science, Engineering, or a related field
advanced degree preferred
10+ years of experience in data engineering, with a proven track record of leadership and technical expertise in managing complex data projects
Proficiency in programming languages such as Python, Java, or Scala, as well as expertise in SQL and relational databases (e.g., PostgreSQL, MySQL)
Strong understanding of distributed computing, cloud technologies (e.g., AWS), and big data frameworks (e.g., Hadoop, Spark)
Experience with data architecture design, data modeling, and optimization techniques
Excellent communication, collaboration, and leadership skills, with the ability to effectively manage remote teams and engage with onshore stakeholders
Proven ability to adapt to evolving project requirements and effectively prioritize tasks in a fast-paced environment

Job Responsibility

Lead and manage an offshore team of data engineers, providing strategic guidance, mentorship, and support to ensure the successful delivery of projects and the development of team members
Collaborate closely with onshore stakeholders to understand project requirements, allocate resources efficiently, and ensure alignment with client expectations and project timelines
Drive the technical design, implementation, and optimization of data pipelines, ETL processes, and data warehouses, ensuring scalability, performance, and reliability
Define and enforce engineering best practices, coding standards, and data quality standards to maintain high-quality deliverables and mitigate project risks
Stay abreast of emerging technologies and industry trends in data engineering, and provide recommendations for tooling, process improvements, and skill development
Assume a data architect role as needed, leading the design and implementation of data architecture solutions, data modeling, and optimization strategies
Demonstrate proficiency in AWS services such as: Expertise in cloud data services, including AWS services like Amazon Redshift, Amazon EMR, and AWS Glue, to design and implement scalable data solutions
Experience with cloud infrastructure services such as AWS EC2, AWS S3, to optimize data processing and storage
Knowledge of cloud security best practices, IAM roles, and encryption mechanisms to ensure data privacy and compliance
Proficiency in managing or implementing cloud data warehouse solutions, including data modeling, schema design, performance tuning, and optimization techniques

Fulltime

Select Country

Data Engineering Lead

Job Description

Requirements

Looking for more opportunities?

Data Engineering Lead

Data Engineering Lead

Data Engineering Lead

Data Engineering Lead

Data Engineering Lead

Data Engineering Lead

Data Engineering Lead

Data Engineering Lead

Data Engineering Lead

Our AI answers in your language