CrawlJobs Logo
C
Codvo AI · IT - Software Development

Data & Databricks Practice Lead

· Job Posted June 03, 2026
Apply Position
Job Link Share

Job Description

We are looking for a dynamic and visionary Practice Lead to build and scale our Data & AI capabilities with a strong focus on Databricks, Apache Spark, Iceberg, Azure Data Fabric, Amazon EMR, and Redshift. This role is pivotal in driving Lakehouse architecture adoption, cloud-native modernization, and ML engineering across industries such as Oil & Gas, Medtech, Retail CPG, and Fintech. You will lead a cross-functional team of architects, engineers, and ML specialists to deliver high-impact solutions that combine technical excellence, cost efficiency, and governance maturity.

Job Responsibility

  • Define and execute the strategic roadmap for the Data & AI practice, aligning with evolving trends in Lakehouse, cloud-native platforms, and AI engineering
  • Build reusable solution accelerators (e.g., Iceberg ingestion templates, Spark ETL frameworks, MLflow pipelines)
  • Establish and manage strategic partnerships with Databricks, AWS, Azure, and other ecosystem players
  • Represent the practice in industry forums, client workshops, and thought leadership initiatives
  • Architect scalable Lakehouse solutions using Delta Lake or Iceberg, optimized for performance and cost
  • Lead the design of unified batch + streaming pipelines using Spark Structured Streaming and Scala
  • Oversee multi-cloud data platform modernization using Azure Data Fabric, EMR, and Redshift
  • Guide implementation of data governance frameworks using Unity Catalog, Azure Purview, and Iceberg metadata
  • Mentor and grow a high-performing team of data engineers, ML engineers, and platform specialists
  • Define career paths, training plans, and certification goals aligned with Databricks, AWS, and Azure
  • Foster a culture of innovation, experimentation, and continuous improvement
  • Engage with clients to understand business challenges and translate them into scalable data solutions
  • Lead solutioning, proposal development, and delivery for strategic engagements
  • Ensure delivery excellence through performance optimization (Photon, EMR Autoscaling) and cost engineering

Requirements

  • 10+ years of experience in data engineering, architecture, and platform modernization
  • 3+ years of hands-on experience with Databricks, Apache Spark, Iceberg, and Scala
  • Deep expertise in cloud-native data platforms (Azure Data Fabric, Amazon EMR, Redshift)
  • Proven track record in building ML pipelines, implementing MLOps (MLflow), and integrating Redshift ML
  • Strong understanding of data governance, observability, and schema evolution
  • Excellent leadership, communication, and stakeholder management skills

Nice to have

  • Certifications in Databricks, Azure Data Engineering, or AWS Big Data
  • Experience in vertical-specific data solutions (e.g., predictive maintenance in Oil & Gas, real-time analytics in Retail)
  • Familiarity with data mesh architectures, time travel, and audit compliance frameworks
Codvo AI - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Data & Databricks Practice Lead

8 matching positions

Data Engineering Lead - Big Data, Snowflake, DataBricks-Vice President

Job Overview: Data Engineering Lead a strategic professional who stays abreast o...
Location
Location
India , Chennai
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 10-15 years of hands-on experience in Hadoop, Scala, Java, Spark, Hive, Kafka, Impala, Unix Scripting and other Big data frameworks
  • 6+ years of experience with relational SQL and NoSQL databases: Oracle, MongoDB, HBase
  • Strong proficiency in Python and Spark Java with knowledge of core spark concepts (RDDs, Dataframes, Spark Streaming, etc) and Scala and SQL
  • Data Integration, Migration & Large Scale ETL experience (Common ETL platforms such as PySpark/DataStage/AbInitio etc.) - ETL design & build, handling, reconciliation and normalization
  • Data Modeling experience (OLAP, OLTP, Logical/Physical Modeling, Normalization, knowledge on performance tuning)
  • Experienced in working with large and multiple datasets and data warehouses
  • Experience building and optimizing ‘big data’ data pipelines, architectures, and datasets.
  • Strong analytic skills and experience working with unstructured datasets
  • Ability to effectively use complex analytical, interpretive, and problem-solving techniques
  • Experience with Confluent Kafka, Redhat JBPM, CI/CD build pipelines and toolchain – Git, BitBucket, Jira
Job Responsibility
Job Responsibility
  • Strategic Leadership: Define and execute the data engineering roadmap for Global Wealth Data, aligning with overall business objectives and technology strategy. This includes understanding the data needs of portfolio managers, investment advisors, and other stakeholders in the wealth management ecosystem.
  • Team Management: Lead, mentor, and develop a high-performing, globally distributed team of data engineers, fostering a culture of collaboration, innovation, and continuous improvement.
  • Architecture and Design: Oversee the design and implementation of robust and scalable data pipelines, data warehouses, and data lakes, ensuring data quality, integrity, and availability for global wealth data. This includes designing solutions for handling large volumes of structured and unstructured data from various sources.
  • Technology Selection and Implementation: Evaluate and select appropriate technologies and tools for data engineering, staying abreast of industry best practices and emerging trends specific to wealth management data.
  • Performance Optimization: Continuously monitor and optimize data pipelines and infrastructure for performance, scalability, and cost-effectiveness, ensuring optimal access to global wealth data.
  • Collaboration: Partner with business stakeholders, data scientists, portfolio managers, and other technology teams to understand data needs and deliver effective solutions that support investment strategies and client reporting.
  • Data Governance: Implement and enforce data governance policies and procedures to ensure data quality, security, and compliance with relevant regulations, particularly around sensitive financial data.
  • Fulltime
Read More
Arrow Right

Lead Data Engineer - Data Pipelines

Economic Intelligence is a Mastercard Services program within Economic & Busines...
Location
Location
Czechia , Prague
Salary
Salary:
Not provided
mastercard.com Logo
Mastercard
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong professional experience with Python building production-grade systems
  • Strong SQL skills, including writing readable and well-tuned queries for large-scale datasets
  • Experience leading design and delivery of data pipelines and/or data platform components, with attention to scalability, reliability, and maintainability
  • Familiarity with Databricks (or similar Spark-based data platforms)
  • Comfortable using LLM‑based coding tools, with the discipline to validate, test, and take full ownership of the generated code
  • Strong foundation in object-oriented software design and engineering best practices
  • Comfortable with Linux command line and production troubleshooting
  • Clear communicator with strong written and verbal English
  • Ownership mindset: motivation, creativity, self-direction, and ability to thrive in small, collaborative teams while influencing across boundaries
  • Passion for analytical/quantitative problem solving, quality, and continuous improvement of both platform and team processes
Job Responsibility
Job Responsibility
  • Lead technical execution to deliver end-to-end capabilities from data processing through publishing and customer consumption
  • Design, build, and evolve scalable, maintainable data pipelines that deliver insights from economic data
  • Drive performance, reliability, observability, and readability in the data platform codebase
  • Own and unify the data layer across multiple products, establishing shared patterns, reusable components, and clear interfaces
  • Translate business needs into durable technical designs
  • Tackle technical debt strategically
  • Set engineering direction and standards for pipeline development and cross-team integration
  • Write, test, and review code
  • Provide technical leadership through mentorship, rigorous code reviews, and support for other engineers’ growth and delivery
  • Continuously innovate, evaluating new approaches, tools, and technologies
  • Fulltime
Read More
Arrow Right

Lead Data Engineer - Data Pipelines

Economic Intelligence is a Mastercard Services program within Economic & Busines...
Location
Location
Czechia , Prague
Salary
Salary:
Not provided
mastercard.com Logo
Mastercard
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong professional experience with Python building production-grade systems
  • Strong SQL skills, including writing readable and well-tuned queries for large-scale datasets
  • Experience leading design and delivery of data pipelines and/or data platform components, with attention to scalability, reliability, and maintainability
  • Familiarity with Databricks (or similar Spark-based data platforms)
  • Comfortable using LLM‑based coding tools
  • Strong foundation in object-oriented software design and engineering best practices
  • Comfortable with Linux command line and production troubleshooting
  • Clear communicator with strong written and verbal English
  • Ownership mindset: motivation, creativity, self-direction, and ability to thrive in small, collaborative teams while influencing across boundaries
  • Passion for analytical/quantitative problem solving, quality, and continuous improvement of both platform and team processes
Job Responsibility
Job Responsibility
  • Lead technical execution to deliver end-to-end capabilities from data processing through publishing and customer consumption
  • Design, build, and evolve scalable, maintainable data pipelines that deliver insights from economic data
  • Drive performance, reliability, observability, and readability in the data platform codebase
  • Own and unify the data layer across multiple products, establishing shared patterns, reusable components, and clear interfaces
  • Translate business needs into durable technical designs
  • Tackle technical debt strategically
  • Set engineering direction and standards for pipeline development and cross-team integration
  • Write, test, and review code
  • Provide technical leadership through mentorship, rigorous code reviews, and support for other engineers’ growth and delivery
  • Continuously innovate, evaluating new approaches, tools, and technologies
  • Fulltime
Read More
Arrow Right

Data Engineer (Big Data, Cloud - AWS, Databricks) - Assistant Vice President

Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Scala, Spark/Pyspark is must, Hadoop ( BIG Data ), + AWS,Databricks
  • 8 to 11 years’ experience implementing data-intensive solutions using agile methodologies
  • Experience of relational databases and using SQL for data querying, transformation and manipulation
  • Experience of modelling data for analytical consumers
  • Ability to automate and streamline the build, test and deployment of data pipelines
  • Experience in cloud native technologies and patterns
  • A passion for learning new technologies, and a desire for personal growth, through self-study, formal classes, or on-the-job training
  • Excellent communication and problem-solving skills
  • An inclination to mentor
  • an ability to lead and deliver medium sized components independently
Job Responsibility
Job Responsibility
  • Developing and supporting scalable, extensible, and highly available data solutions
  • Deliver on critical business priorities while ensuring alignment with the wider architectural vision
  • Identify and help address potential risks in the data supply chain
  • Follow and contribute to technical standards
  • Design and develop analytical data models
  • Fulltime
Read More
Arrow Right

Lead Data Engineer

Lead Data Engineer | Databricks | Azure | PySpark. You are an experienced Data E...
Location
Location
Netherlands , Amsterdam
Salary
Salary:
Not provided
levy-professionals.com Logo
Levy Professionals
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Around 10 years of experience in Data Engineering
  • Extensive hands-on experience with Databricks, PySpark, and Python
  • Strong experience with Azure Data Factory, Azure DevOps, CI/CD practices
  • Automated testing frameworks
  • Software engineering best practices
  • Ability to mentor engineers
Job Responsibility
Job Responsibility
  • Design, build, and optimise scalable data pipelines using PySpark and Databricks
  • Develop and maintain high-quality Python code
  • Contribute to automated testing frameworks using pytest
  • Support and enhance CI/CD pipelines within Azure DevOps
  • Collaborate with cross-functional teams
  • Mentor junior and medior Data Engineers
  • Fulltime
Read More
Arrow Right

Senior Technical Lead - Data & AI (Oil & Gas)

At Codvo, we are committed to building scalable, future-ready data platforms tha...
Location
Location
India , Pune
Salary
Salary:
Not provided
Codvo AI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience: 12+ years in technical roles, with at least 4+ years in a technical leadership or principal engineer capacity, managing cross-functional engineering teams
  • Cloud Expertise: Expert-level, hands-on knowledge of Microsoft Azure as a primary platform
  • Data & AI Expertise: Proven mastery of Databricks and Apache Spark (PySpark)
  • Extensive experience designing and building large-scale ETL/ELT data pipelines
  • Strong understanding of AI/ML workflows and data requirements for industrial AI use cases
  • DevOps & IaC Expertise: Deep proficiency with Infrastructure as Code (Terraform, ARM templates)
  • Hands-on mastery of Kubernetes and containerization (Docker)
  • Proven experience building and managing complex CI/CD pipelines (Azure DevOps, Jenkins)
  • Cloud Security: Strong, practical knowledge of cloud network security (VNETs, NSGs, Firewalls) and identity management (Azure Key Vault, IAM)
  • Industry Knowledge: Experience in the Oil & Gas sector or a similar heavy-industry/critical infrastructure environment
Job Responsibility
Job Responsibility
  • Technical Leadership & Mentorship: Lead, mentor, and provide technical direction to a multi-disciplinary team of engineers (Data, DevOps, Security). Foster a culture of high technical standards, collaboration, and continuous improvement
  • Detailed Solution Design: Partner with the Solutions Architect and Project Manager to decompose high-level architecture into detailed technical designs, actionable tasks, and implementation plans for the engineering team
  • Hands-on Development: Act as the lead engineer, personally designing and building the most complex components of the solution, including data pipelines, IaC modules, and CI/CD frameworks
  • Cross-Functional Integration: Drive the seamless integration of diverse systems, including OT data from historians (OSIsoft PI), data platforms (Databricks), cloud infrastructure (Azure), and CI/CD pipelines (Azure DevOps)
  • Technical Governance & Quality: Enforce technical best practices across all domains. Conduct rigorous code reviews, design reviews, and security assessments. Ensure all solutions adhere to security-by-design principles, CI/CD automation standards, and data governance policies
  • Stakeholder Collaboration: Translate complex technical concepts and risks to the Project Manager and business stakeholders. Manage technical dependencies and help the PM with accurate effort estimation and milestone tracking
  • Problem Solving: Serve as the primary escalation point for the team's most complex technical challenges, leading root cause analysis and implementing robust, long-term solutions
  • Fulltime
Read More
Arrow Right

Lead Data Engineer (Python, AWS, SQL, GenAI) (Enterprise Platforms Technology)

Do you love building and pioneering in the technology space? Do you enjoy solvin...
Location
Location
United States , McLean, Virginia; Chicago, Illinois
Salary
Salary:
179400.00 - 225100.00 USD / Year
capitalone.com Logo
Capital One
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s Degree
  • At least 4 years of experience in application development (Internship experience does not apply)
  • At least 2 years of experience in big data technologies
  • At least 1 year experience with cloud computing (AWS, Microsoft Azure, Google Cloud)
Job Responsibility
Job Responsibility
  • Collaborate with and across Agile teams to design, develop, test, implement, and support technical solutions using data movement tools and technologies
  • Work, as a lead developer, with a team of developers with deep experience in data movement, distributed computing, and full stack systems
  • Utilize programming languages like Python, SQL and Open Source RDBMS and NoSQL databases and Cloud based data warehousing services such as Snowflake, Databricks
  • Optimize information system for end-user and downstream application consumers by using sound data design practices
  • Share your passion for staying on top of tech trends, experimenting with and learning new technologies, participating in internal & external technology communities, and mentoring other members of the engineering community
  • Collaborate with digital product managers, and deliver robust cloud-based solutions that drive powerful experiences to help millions of Americans achieve financial empowerment
  • Perform unit tests and conduct reviews with other team members to make sure your code is rigorously designed, elegantly coded, and effectively tuned for performance
What we offer
What we offer
  • Performance based incentive compensation, which may include cash bonus(es) and/or long term incentives (LTI)
  • Comprehensive, competitive, and inclusive set of health, financial and other benefits
  • Fulltime
Read More
Arrow Right

Senior Data Lead

At Barclays you will spearhead the transformation of our data landscape, driving...
Location
Location
India , Pune
Salary
Salary:
Not provided
barclays.co.uk Logo
Barclays
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proficiency in designing and implementing scalable data platforms on AWS using services such as S3, Glue (ETL Jobs, Data Catalog), Redshift, Athena, and Iceberg with strong hands-on experience with Databricks for building notebooks, ELT pipelines, and working with distributed data processing frameworks such as Apache Spark
  • Expertise in migrating large-scale SQL Server workloads to AWS, including defining source-to-target mappings and ensuring performance and data integrity
  • Proficiency in Python and SQL for developing data pipelines, transformations, and analytical workflows and Experience in building and orchestrating ETL/ELT pipelines using AWS Glue, DBT, and tools such as Airflow/Astronomer
  • Ability to define data contracts, design Iceberg table schemas, and implement partitioning, metadata management, and Glue Catalog integration
  • Strong understanding of data governance, including implementation of Lake Formation policies for row- and column-level security
  • Experience embedding MNPI controls, including data classification, access restrictions, and working with Compliance/CISO stakeholders
  • Hands-on experience with CI/CD pipelines, DevOps practices, and Agile delivery methodologies
  • Design and optimize data lake and warehouse solutions for structured and unstructured data across AWS platforms
Job Responsibility
Job Responsibility
  • At Barclays you will spearhead the transformation of our data landscape, driving innovation and excellence across enterprise data platforms
  • As a Senior Data Lead, you will leverage cutting-edge AWS technologies and Databricks to modernize and migrate SQL Server workloads into scalable, secure, and compliant cloud-native solutions
  • With a primary focus on AWS services including S3, Glue, Redshift, Athena, and Iceberg, alongside Python, SQL, and orchestration tools like Airflow, you will design and deliver robust data pipelines, establish strong data governance frameworks including MNPI controls, and enable high-quality, insight-driven decision-making
What we offer
What we offer
  • Competitive holiday allowance
  • Life assurance
  • Private medical care
  • Pension contribution
  • Fulltime
Read More
Arrow Right