This list contains only the countries for which job offers have been published in the selected language (e.g., in the French version, only job offers written in French are displayed, and in the English version, only those in English).
We are seeking an experienced Data Engineering Specialist with strong hands-on expertise in Databricks on GCP, Python-based data engineering, and Spark processing. The individual will design, build, and optimise large-scale data pipelines across the GCP ecosystem, applying robust engineering practices, data quality frameworks, and cost-optimised solutions.
Job Responsibility:
Design and build data pipelines on GCP using Databricks (Delta Lake and Unity Catalog) for orchestration, Dataproc for Spark execution, supporting both ETL/ELT and feature engineering workloads
Engineer declarative, modular, and reusable pipelines in Python, following configuration-as-code principles and CI/CD practices including Git-based promotion, testing, and deployment
Implement and maintain data quality and observability practices using validation frameworks, logging, metrics, and alerts
Optimise pipeline performance, reliability, and cost through techniques such as cluster sizing, auto-termination, Z-ordering, caching, and partitioning strategies
Apply robust error handling, parameterisation, and triggers within Cloud Data Fusion pipelines
Ensure operational excellence by maintaining monitoring, performance tuning, and continuous improvements across data products and workloads
Requirements:
Strong expertise in Databricks on GCP including Delta Lake, notebooks/jobs, Unity Catalog, and cluster policies
Experienced in Cloud Data Fusion design, including pipeline management, error handling, and orchestration
Skilled in Dataproc Spark with experience building PySpark jobs, configuring ephemeral clusters, and handling initialisation actions
Proficient in Python for data engineering including packaging, unit testing, type hints, and linting
Strong SQL skills, specifically with BigQuery including performance tuning, partitioning, and clustering
Familiar with GCP services such as Cloud Storage, Pub/Sub, and Cloud Composer/Airflow
Holds a qualification such as B.E., B.Tech, BCA, MCA, BSc, or MSc in Computer Science or a related field
What we offer:
The opportunity to build and scale data solutions using leading GCP and Databricks technologies
Exposure to enterprise-level CI/CD, observability, and configuration-as-code practices
A collaborative environment where innovation, continuous learning, and technical excellence are encouraged
The chance to contribute to high-impact global data platforms