Lead Data Engineer (Platform) Job at Valtech

Lead Data Engineer (Platform)

As a Lead Data Engineer (Platform), you are passionate about engineering excelle...

Location

North Macedonia

Salary:

Not provided

Valtech

Expiration Date

Until further notice

Requirements

Strong proficiency in Python for building frameworks, automation tooling and reusable components
Hands-on experience with Databricks (including notebooks, workflows, jobs and Unity Catalog)
Strong SQL skills and experience with distributed processing frameworks such as Apache Spark
Deep experience with dbt Core, including project structure, models, tests, macros and deployment at scale
Proven experience designing and maintaining CI/CD pipelines (e.g. GitHub Actions, Azure DevOps or GitLab CI)
Experience with data engineering platform design, including scalable pipeline and workflow architectures
Strong understanding of software engineering principles (DRY, SOLID, modular design)
Experience working with version control systems and modern Git workflows
Experience with cloud platforms (preferably AWS) and infrastructure-as-code concepts (e.g. Terraform)
Experience implementing automated testing strategies (unit, integration, data quality)

Job Responsibility

Design, build, and maintain scalable data engineering frameworks and platform utilities used across engineering teams
Develop reusable patterns, templates, and abstractions to standardise and accelerate delivery
Define and evolve platform architecture decisions, ensuring scalability, maintainability and consistency
Design and implement CI/CD pipelines and automation frameworks to improve engineering velocity
Define and enforce engineering standards for testing, code quality, deployment and documentation
Identify and eliminate manual or repetitive processes through automation and tooling improvements
Integrate AI-assisted development tools into engineering workflows to improve productivity
Develop and maintain AI engineering assets such as coding guidelines, prompt frameworks and reusable agent configurations
Lead the development and operational support of core data transformation frameworks (including dbt Core at enterprise scale)
Investigate and resolve framework-level issues, including deployment failures, dependency conflicts and production incidents

What we offer

Private health insurance
Education program
Wellbeing program
Free beverages
Events
Competitive conditions
Challenging projects
Cool colleagues
Honest feedback

Fulltime

Lead Data Engineer (Platform)

As a Lead Data Engineer (Platform), you are passionate about engineering excelle...

Location

Bulgaria , Sofia

Salary:

Not provided

Valtech

Expiration Date

Until further notice

Requirements

Strong proficiency in Python for building frameworks, automation tooling and reusable components
Hands-on experience with Databricks (including notebooks, workflows, jobs and Unity Catalog)
Strong SQL skills and experience with distributed processing frameworks such as Apache Spark
Deep experience with dbt Core, including project structure, models, tests, macros and deployment at scale
Proven experience designing and maintaining CI/CD pipelines (e.g. GitHub Actions, Azure DevOps or GitLab CI)
Experience with data engineering platform design, including scalable pipeline and workflow architectures
Strong understanding of software engineering principles (DRY, SOLID, modular design)
Experience working with version control systems and modern Git workflows
Experience with cloud platforms (preferably AWS) and infrastructure-as-code concepts (e.g. Terraform)
Experience implementing automated testing strategies (unit, integration, data quality)

Job Responsibility

Design, build, and maintain scalable data engineering frameworks and platform utilities used across engineering teams
Develop reusable patterns, templates, and abstractions to standardise and accelerate delivery
Define and evolve platform architecture decisions, ensuring scalability, maintainability and consistency
Design and implement CI/CD pipelines and automation frameworks to improve engineering velocity
Define and enforce engineering standards for testing, code quality, deployment and documentation
Identify and eliminate manual or repetitive processes through automation and tooling improvements
Integrate AI-assisted development tools into engineering workflows to improve productivity
Develop and maintain AI engineering assets such as coding guidelines, prompt frameworks and reusable agent configurations
Lead the development and operational support of core data transformation frameworks (including dbt Core at enterprise scale)
Investigate and resolve framework-level issues, including deployment failures, dependency conflicts and production incidents

What we offer

Flexibility, with hybrid work options and 25 vacation days for a healthy work-life balance
Co-subsidized transportation & Multisport cards
Premium health insurance for fast and easy access to top healthcare services
Training policy for technical and other skills-related events, courses and certifications
Personal career development roadmap guided by performance evaluations
Self-care program offering psychological consultations & discussions for you and the team
Cozy office space designed for comfort and productivity
Exciting team events and company gatherings

Fulltime

Lead Data Engineer (Platform)

As a Lead Data Engineer (Platform), you are passionate about engineering excelle...

Location

Portugal , Lisbon

Salary:

Not provided

Valtech

Expiration Date

Until further notice

Requirements

Strong proficiency in Python for building frameworks, automation tooling and reusable components
Hands-on experience with Databricks (including notebooks, workflows, jobs and Unity Catalog)
Strong SQL skills and experience with distributed processing frameworks such as Apache Spark
Deep experience with dbt Core, including project structure, models, tests, macros and deployment at scale
Proven experience designing and maintaining CI/CD pipelines (e.g. GitHub Actions, Azure DevOps or GitLab CI)
Experience with data engineering platform design, including scalable pipeline and workflow architectures
Strong understanding of software engineering principles (DRY, SOLID, modular design)
Experience working with version control systems and modern Git workflows
Experience with cloud platforms (preferably AWS) and infrastructure-as-code concepts (e.g. Terraform)
Experience implementing automated testing strategies (unit, integration, data quality)

Job Responsibility

Design, build, and maintain scalable data engineering frameworks and platform utilities used across engineering teams
Develop reusable patterns, templates, and abstractions to standardise and accelerate delivery
Define and evolve platform architecture decisions, ensuring scalability, maintainability and consistency
Design and implement CI/CD pipelines and automation frameworks to improve engineering velocity
Define and enforce engineering standards for testing, code quality, deployment and documentation
Identify and eliminate manual or repetitive processes through automation and tooling improvements
Integrate AI-assisted development tools into engineering workflows to improve productivity
Develop and maintain AI engineering assets such as coding guidelines, prompt frameworks and reusable agent configurations
Lead the development and operational support of core data transformation frameworks (including dbt Core at enterprise scale)
Investigate and resolve framework-level issues, including deployment failures, dependency conflicts and production incidents

What we offer

Flexibility, with remote and hybrid work options (country-dependent)
Career advancement, with international mobility and professional development programs
Learning and development, with access to cutting-edge tools, training and industry experts

Fulltime

Senior AWS Data Engineer / Data Platform Engineer

We are seeking a highly experienced Senior AWS Data Engineer to design, build, a...

Location

United Arab Emirates , Dubai

Salary:

Not provided

NorthBay

Expiration Date

Until further notice

Requirements

8+ years of experience in data engineering and data platform development
Strong hands-on experience with: AWS Glue
Amazon EMR (Spark)
AWS Lambda
Apache Airflow (MWAA)
Amazon EC2
Amazon CloudWatch
Amazon Redshift
Amazon DynamoDB
AWS DataZone

Job Responsibility

Design, develop, and optimize scalable data pipelines using AWS native services
Lead the implementation of batch and near-real-time data processing solutions
Architect and manage data ingestion, transformation, and storage layers
Build and maintain ETL/ELT workflows using AWS Glue and Apache Spark on EMR
Orchestrate complex data workflows using Apache Airflow (MWAA)
Develop and manage serverless data processing using AWS Lambda
Design and optimize data warehouses using Amazon Redshift
Implement and manage NoSQL data models using Amazon DynamoDB
Utilize AWS DataZone for data governance, cataloging, and access management
Monitor, log, and troubleshoot data pipelines using Amazon CloudWatch

Fulltime

Lead Data Platform Engineer

Build the foundation of data-driven transformation. At Riverflex, we design and ...

Location

Netherlands , Amsterdam

Salary:

Not provided

Riverflex

Expiration Date

Until further notice

Requirements

6+ years of experience in data engineering or data infrastructure roles
Strong proficiency in Python, SQL, and distributed data frameworks (e.g., Spark, Kafka, Airflow)
Proven experience building data platforms in a cloud-native environment (Azure preferred)
Deep understanding of data lakehouse architectures and modern data warehousing (e.g., Databricks)
Hands-on experience with DevOps practices for data (CI/CD, IaC, Terraform, Docker)
Strong understanding of data governance, privacy, and compliance frameworks
Comfortable working in fast-paced, cross-functional Agile teams
Excellent communication and stakeholder management skills

Job Responsibility

Architect and maintain end-to-end data platforms across cloud environments (Azure, Databricks)
Design and implement secure, scalable, and automated data pipelines
Collaborate with analytics, product, and engineering teams to translate business needs into data solutions
Own data platform reliability, performance, monitoring, and incident response
Implement and champion best practices in data modelling, governance, testing, and documentation
Mentor junior engineers and help shape a high-performing data engineering team
Evaluate and integrate new technologies to improve data stack performance, scalability, and cost-efficiency
Build reusable tooling, components, and internal frameworks for data pipeline development
Ensure compliance with data privacy, protection, and security regulations (e.g. GDPR)

What we offer

25 days off per year plus closure between Christmas and New Year's
Flexible remote work from abroad options for up to 6 weeks per year
Learning & Development budget, including full access to Udemy courses
Classpass membership to support well-being
Latest tech & tools, including home office budget and professional software subscriptions
Equity share scheme to give long-term team members ownership in Riverflex
Annual company trips to celebrate successes together

Senior Lead Data Engineer (Enterprise Platform Technology) (Java, Python, Scala, AWS)

Do you love building and pioneering in the technology space? Do you enjoy solvin...

Location

United States , McLean; New York

Salary:

229900.00 - 286200.00 USD / Year

Capital One

Expiration Date

Until further notice

Requirements

Bachelor's Degree
At least 6 years of experience in application development (Internship experience does not apply)
At least 2 years of experience in big data technologies
At least 1 year experience with cloud computing (AWS, Microsoft Azure, Google Cloud)

Job Responsibility

Collaborate with and across Agile teams to design, develop, test, implement, and support technical solutions in full-stack development tools and technologies
Lead a team of developers with deep experience in machine learning, distributed microservices, and full stack systems
Utilize programming languages like Java, Scala, Python and Open Source RDBMS and NoSQL databases and Cloud based data warehousing services such as Redshift and Snowflake
Share your passion for staying on top of tech trends, experimenting with and learning new technologies, participating in internal & external technology communities, and mentoring other members of the engineering community
Collaborate with digital product managers, and deliver robust cloud-based solutions that drive powerful experiences to help millions of Americans achieve financial empowerment
Perform unit tests and conduct reviews with other team members to make sure your code is rigorously designed, elegantly coded, and effectively tuned for performance

What we offer

performance based incentive compensation, which may include cash bonus(es) and/or long term incentives (LTI)
comprehensive, competitive, and inclusive set of health, financial and other benefits

Fulltime

Lead Data Engineer - Data Pipelines

Economic Intelligence is a Mastercard Services program within Economic & Busines...

Location

Czechia , Prague

Salary:

Not provided

Mastercard

Expiration Date

Until further notice

Requirements

Strong professional experience with Python building production-grade systems
Strong SQL skills, including writing readable and well-tuned queries for large-scale datasets
Experience leading design and delivery of data pipelines and/or data platform components, with attention to scalability, reliability, and maintainability
Familiarity with Databricks (or similar Spark-based data platforms)
Comfortable using LLM‑based coding tools, with the discipline to validate, test, and take full ownership of the generated code
Strong foundation in object-oriented software design and engineering best practices
Comfortable with Linux command line and production troubleshooting
Clear communicator with strong written and verbal English
Ownership mindset: motivation, creativity, self-direction, and ability to thrive in small, collaborative teams while influencing across boundaries
Passion for analytical/quantitative problem solving, quality, and continuous improvement of both platform and team processes

Job Responsibility

Lead technical execution to deliver end-to-end capabilities from data processing through publishing and customer consumption
Design, build, and evolve scalable, maintainable data pipelines that deliver insights from economic data
Drive performance, reliability, observability, and readability in the data platform codebase
Own and unify the data layer across multiple products, establishing shared patterns, reusable components, and clear interfaces
Translate business needs into durable technical designs
Tackle technical debt strategically
Set engineering direction and standards for pipeline development and cross-team integration
Write, test, and review code
Provide technical leadership through mentorship, rigorous code reviews, and support for other engineers’ growth and delivery
Continuously innovate, evaluating new approaches, tools, and technologies

Fulltime

Lead Data Engineer - Data Pipelines

Economic Intelligence is a Mastercard Services program within Economic & Busines...

Location

Czechia , Prague

Salary:

Not provided

Mastercard

Expiration Date

Until further notice

Requirements

Strong professional experience with Python building production-grade systems
Strong SQL skills, including writing readable and well-tuned queries for large-scale datasets
Experience leading design and delivery of data pipelines and/or data platform components, with attention to scalability, reliability, and maintainability
Familiarity with Databricks (or similar Spark-based data platforms)
Comfortable using LLM‑based coding tools
Strong foundation in object-oriented software design and engineering best practices
Comfortable with Linux command line and production troubleshooting
Clear communicator with strong written and verbal English
Ownership mindset: motivation, creativity, self-direction, and ability to thrive in small, collaborative teams while influencing across boundaries
Passion for analytical/quantitative problem solving, quality, and continuous improvement of both platform and team processes

Job Responsibility

Lead technical execution to deliver end-to-end capabilities from data processing through publishing and customer consumption
Design, build, and evolve scalable, maintainable data pipelines that deliver insights from economic data
Drive performance, reliability, observability, and readability in the data platform codebase
Own and unify the data layer across multiple products, establishing shared patterns, reusable components, and clear interfaces
Translate business needs into durable technical designs
Tackle technical debt strategically
Set engineering direction and standards for pipeline development and cross-team integration
Write, test, and review code
Provide technical leadership through mentorship, rigorous code reviews, and support for other engineers’ growth and delivery
Continuously innovate, evaluating new approaches, tools, and technologies

Fulltime

Select Country

Lead Data Engineer (Platform)

Job Description

Job Responsibility

Requirements

Nice to have

What we offer

Looking for more opportunities?

Lead Data Engineer (Platform)

Lead Data Engineer (Platform)

Lead Data Engineer (Platform)

Lead Data Engineer (Platform)

Senior AWS Data Engineer / Data Platform Engineer

Lead Data Platform Engineer

Senior Lead Data Engineer (Enterprise Platform Technology) (Java, Python, Scala, AWS)

Lead Data Engineer - Data Pipelines

Lead Data Engineer - Data Pipelines

Our AI answers in your language