This list contains only the countries for which job offers have been published in the selected language (e.g., in the French version, only job offers written in French are displayed, and in the English version, only those in English).
We are looking for a highly skilled Data Engineering Manager to join our Data Engineering team and play a central role in building and scaling our vendor universe—the database of companies we track and analyze. Reporting directly to the Senior Director of Data Engineering, this role demands technical leadership, independence, and the ability to deliver under tight deadlines in a fast-paced environment. As a Data Engineering Manager, you will design and maintain large-scale data pipelines, define best practices for ETL, and integrate emerging technologies to keep our platform on the cutting edge. You’ll work cross-functionally to deliver reliable, efficient, and scalable data solutions while contributing hands-on expertise in Databricks, Airflow, PySpark, and SQL.
Job Responsibility:
Own the design, build, and optimization of end-to-end data pipelines that power our vendor universe
Establish and enforce best practices in data modeling, orchestration, and system reliability
Collaborate with product, engineering, and business stakeholders to translate requirements into robust, scalable data solutions
Work extensively with Databricks and Airflow for large-scale data processing and orchestration
Troubleshoot and resolve complex pipeline issues to ensure reliability and performance
Contribute to the team’s technical strategy, helping drive improvements in scalability, performance, and efficiency
Lead, mentor, and support engineers through challenges, code reviews, and project execution
Requirements:
6+ years of professional experience in Data Engineering or equivalent technical roles (e.g., data architecture, big data development, or ETL engineering)
2+ years of managerial experience, including mentoring, team leadership, and supporting delivery
Strong expertise in SQL and distributed data systems
Proficiency with PySpark and Databricks for processing and scaling large datasets
Hands-on experience with Airflow for pipeline orchestration (Dagster/dbt a plus)
Proven track record of delivering in fast-paced, deadline-driven environments with minimal oversight
Strong problem-solving skills and ability to translate business needs into scalable technical solutions
Excellent communication and collaboration skills with both technical and non-technical stakeholders
Nice to have:
Experience leveraging AI/ML models, vector search, or Elasticsearch to enhance data pipelines
Familiarity with Django or similar web frameworks to support data-driven applications