CrawlJobs Logo

Principal Engineer – Data Platforms & MLOps (Databricks)

India, Pune · Job Posted June 03, 2026
Apply Position
Job Link Share

Job Description

We are looking for a hands-on Principal Engineer with deep expertise in Databricks to design, build, and scale enterprise-grade data platforms and MLOps pipelines. You will be the technical authority on how enterprises adopt and maximize Databricks — from ingestion to governance to machine learning deployment — and a mentor who raises the bar for engineering excellence.

Job Responsibility

  • Platform Architecture: Design and implement end-to-end data architectures on Databricks Lakehouse, covering ingestion, transformation, storage, and analytics
  • Pipelines & Workflows: Build and optimize ETL/ELT pipelines with Delta Live Tables, Spark Structured Streaming, and workflow orchestration
  • Governance & Security: Implement Unity Catalog, fine-grained access controls, and compliance frameworks across enterprise data estates
  • MLOps at Scale: Operationalize ML models using MLflow, Model Registry, and CI/CD pipelines integrated with cloud DevOps tools
  • Performance & Cost Optimization: Tune Databricks clusters, jobs, and workflows for scale, speed, and efficiency across multi-cloud deployments
  • Client Advisory: Work closely with enterprise stakeholders to provide best practices, reference architectures, and accelerators tailored to their use cases
  • Mentorship & Standards: Guide engineers in Databricks best practices, enforce coding standards, and lead design/code reviews

Requirements

  • 8+ years in large-scale data engineering / platform engineering, with 3+ years hands-on Databricks experience
  • Deep expertise in: Databricks Lakehouse Platform (Delta Lake, Delta Live Tables, Databricks SQL)
  • Governance & Security with Unity Catalog
  • MLOps with MLflow and model lifecycle management
  • Strong programming skills in PySpark, SQL, Python
  • experience with Scala a plus
  • Hands-on with cloud integration (AWS, Azure, or GCP) and DevOps pipelines (Terraform, GitHub Actions, Azure DevOps, etc.)
  • Proven track record of building and scaling Databricks workloads in production for enterprise clients

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Principal Engineer – Data Platforms & MLOps (Databricks)

8 matching positions

Principal Consulting AI / Data Engineer

As a Principal Consulting AI / Data Engineer, you will design, build, and optimi...
Location
Location
Australia , Sydney
Salary
Salary:
Not provided
dyflex.com.au Logo
DyFlex Solutions
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proven expertise in delivering enterprise-grade data engineering and AI solutions in production environments
  • Strong proficiency in Python and SQL, plus experience with Spark, Airflow, dbt, Kafka, or Flink
  • Experience with cloud platforms (AWS, Azure, or GCP) and Databricks
  • Ability to confidently communicate and present at C-suite level, simplifying technical concepts into business impact
  • Track record of engaging senior executives and influencing strategic decisions
  • Strong consulting and stakeholder management skills with client-facing experience
  • Background in MLOps, ML pipelines, or AI solution delivery highly regarded
  • Degree in Computer Science, Engineering, Data Science, Mathematics, or a related field
Job Responsibility
Job Responsibility
  • Design, build, and maintain scalable data and AI solutions using Databricks, cloud platforms, and modern frameworks
  • Lead solution architecture discussions with clients, ensuring alignment of technical delivery with business strategy
  • Present to and influence executive-level stakeholders, including boards, C-suite, and senior directors
  • Translate highly technical solutions into clear business value propositions for non-technical audiences
  • Mentor and guide teams of engineers and consultants to deliver high-quality solutions
  • Champion best practices across data engineering, MLOps, and cloud delivery
  • Build DyFlex’s reputation as a trusted partner in Data & AI through thought leadership and client advocacy
What we offer
What we offer
  • Work with SAP’s latest technologies on cloud as S/4HANA, BTP and Joule, plus Databricks, ML/AI tools and cloud platforms
  • A flexible and supportive work environment including work from home
  • Competitive remuneration and benefits including novated lease, birthday leave, salary packaging, wellbeing programme, additional purchased leave, and company-provided laptop
  • Comprehensive training budget and paid certifications (Databricks, SAP, cloud platforms)
  • Structured career advancement pathways with opportunities to lead large-scale client programs
  • Exposure to diverse industries and client environments, including executive-level engagement
  • Fulltime
Read More
Arrow Right

Principal Data Engineer

Our Principal Data Engineers are responsible for leading and delivering strategi...
Location
Location
United Kingdom , Bristol; London; Manchester; Swansea
Salary
Salary:
100000.00 - 115000.00 GBP / Year
madetech.com Logo
Made Tech
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Understanding of the issues and challenges that the public sector faces in delivering services that make the best use of data and digital capabilities, transforming legacy infrastructure, and taking an innovative and user-centric approach
  • Ability to innovate and take learnings from the commercial sector, other countries and advances in technology and apply them to UK Public Sector challenges to create tangible solutions for our clients
  • Experience building trusted advisor relationships with senior client stakeholders within the public sector.
  • Experience of building and leading high performing, consulting teams and creating the leveraged engagements to provide a cost-effective, profitable, successful client-facing delivery
  • Leadership of bids and solution shaping to produce compelling proposals that help Made Tech win new business and grow the industry
  • Experience of managing third-party partnerships and suppliers (in conjunction with Made Tech colleagues) to provide a consolidated and seamless delivery team to clients.
  • Experience in delivering complex and difficult engagements that span multiple capabilities for user-facing digital and data services in the public sector
  • Experience in identifying opportunities based on client needs and developing targeted solutions to progress the development of the opportunity
  • Experience of working with sales professionals and commercial responsibility for strategic organisational goals.
  • Experience working directly with customers and users within a technology consultancy
Job Responsibility
Job Responsibility
  • Collaborate with clients to understand their needs, provide solution advice in your role as a trusted advisor and shape solutions that leverage Made Tech's wider capabilities and credentials
  • Assess project performance as a part of the billable delivery team, Quality Assure (QA) the deliverables and outcomes, and ensure client satisfaction. Coach and mentor team members as well as providing direction to enable them to achieve their engagement outcomes and to develop their careers
  • Act as a Technical Authority of the Data & AI capability to provide oversight and ensure alignment with internal and industry best practices. Ensure engagement experience is captured and used to improve standards and contribute to Made Tech knowledge
  • Participate in business development activities, including bids and pre-sales within the account, industry and practice. Coach team members on their contributions and oversee the relevant technical aspects of the proposal submission
  • Undertake people management responsibilities, including performance reviews and professional development of your engagement and practice colleagues
  • Serve as a thought leader within Made Tech, our account engagements and the wider public sector and represent the company at industry events
What we offer
What we offer
  • 30 days of paid annual leave + bank holidays
  • Flexible Parental Leave
  • Remote Working
  • Paid counselling as well as financial and legal advice
  • Flexible benefit platform which includes a Smart Tech scheme, Cycle to work scheme, and an individual benefits allowance which you can invest in a Health care cash plan or Pension plan
  • Optional social and wellbeing calendar of events
  • Fulltime
Read More
Arrow Right

Principal Engineer I – Senior Azure Databricks Administrator

Software Resources has an immediate, direct hire job opportunity for a Principal...
Location
Location
United States , Phoenix
Salary
Salary:
Not provided
softwareresources.com Logo
Software Resources
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years of related experience in data analytics administration and development
  • 4+ years of Databricks related experience
  • Bachelor’s degree in related field required
  • Advanced proven experience in Azure Databricks (Workspace management, Clusters, Jobs, Unity Catalog, Delta Lake, User access management, Rest APIs and SDKs)
  • Knowledge of MLFlow & MLOps
  • Deep understanding of Azure infrastructure and data services, including Azure Data Lake, Azure Data Factory, Azure SQL, Azure Synapse Analytics, Azure Key Vault, Azure Monitor, networking
  • Experience with CI/CD pipelines (Azure DevOps preferred)
  • Strong programming skills in SQL, Python, and/or PySpark
  • Advanced proven experience in leading cross-functional teams and managing multiple projects simultaneously
  • Advanced ability to see the big picture and align projects with organizational goals
Job Responsibility
Job Responsibility
  • Responsible for delivery and operations of technologies and platforms required to model, transform, analyze, report, visualize data
  • Provide SME expertise in designing, building, optimizing, streamlining and automating the Azure Databricks platform
  • Partner with ML engineers, data scientists, data analysts, and enterprise architects to provide frameworks, set standards, enforce best practices, train and enable users
  • Develop technical skills of one or more junior team-members
  • Take assignments that can be worked on individually without supervision, and manage work effort from concept to completion
  • Design, build, optimize, automate and maintain the Azure Databricks platform, ensuring scalability, security, governance and performance
  • Design, implement and manage Azure Databricks workspaces, clusters, jobs, access management
  • Design, implement and manage policies, monitoring and observability
  • Implement data analytics principles aimed at business enablement, reliability practices and sound recovery procedures
  • Ensure compliance with IT policies, procedures, and industry standards
What we offer
What we offer
  • Competitive salaries
  • An ownership stake in the company
  • Medical and dental insurance
  • Time off
  • A great 401k matching program
  • Tuition assistance program
  • An employee volunteer program
  • A wellness program
  • Fulltime
Read More
Arrow Right

Principal Machine Learning Engineer

As a Principal Machine Learning Engineer, you will lead the architecture and dev...
Location
Location
India , Hyderabad
Salary
Salary:
Not provided
amgen.com Logo
Amgen
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree in computer science, Engineering, Data Science, or a related field with 12 to 17 years of total experience
  • 8+ years of experience in software engineering, machine learning engineering, or ML infrastructure
  • Strong experience building production ML systems or ML platforms
  • Hands-on experience with MLOps frameworks and tools such as MLflow / Equivalent - Model lifecycle management frameworks
  • Strong programming experience in Python and modern software engineering practices such as API Driven Architecture and Event based systems
  • Experience designing scalable distributed systems or cloud-native architectures
  • Experience deploying and operating machine learning models in production environments
  • Solid understanding of modern ML workflows including training, evaluation, deployment, monitoring, and retraining
Job Responsibility
Job Responsibility
  • Architect and build a scalable ML platform for training, deployment, and lifecycle management of ML, LLM, and Generative AI models
  • Lead development of infrastructure that supports production hosting of complex AI systems, including large-scale inference workloads
  • Design developer-friendly abstractions and automation that make it easy for researchers to build and deploy models within the Amgen ecosystem
  • Implement and evolve MLOps capabilities including experiment tracking, model versioning, CI/CD for ML, monitoring, and reproducibility using tools such as Databricks and MLflow
  • Build platform capabilities supporting Generative AI and emerging Agentic AI systems
  • Serve as the technical leader for a team of engineers, guiding architecture, design reviews, and engineering best practices
  • Partner with AI researchers, data scientists, and platform teams to translate cutting-edge AI research into reliable production systems
  • Evaluate and adopt emerging technologies across the modern AI stack including foundation models, vector databases, agent frameworks, and model serving systems
  • Champion AI-native engineering practices, leveraging tools like GitHub Copilot, Codex, and AI-assisted development workflows
  • Contribute to the broader strategy and evolution of the Enterprise AI Platforms ecosystem
Read More
Arrow Right

Principal Ai Architect

At Teradata, we believe that people thrive when empowered with better informatio...
Location
Location
United States , California
Salary
Salary:
217300.00 - 325900.00 USD / Year
teradata.com Logo
Teradata
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 10+ years of software engineering experience, including 3+ years in a senior architect or principal engineer role with platform-wide technical scope
  • Demonstrated expertise designing AI/ML platforms or developer tools: model serving infrastructure, feature stores, experiment tracking, MLOps pipelines, or AI agent development environments
  • Deep understanding of LLM integration patterns: RAG architectures, fine-tuning pipelines, evaluation frameworks, and agent tool-calling interfaces
  • Experience with enterprise data platforms (Teradata Vantage, Snowflake, Databricks, or equivalent) at sufficient depth to architect against their APIs, security models, and performance characteristics
  • Experience building developer-facing platforms — SDKs, APIs, or IDEs — that external developers adopt and extend
  • Familiarity with open-source AI development tools: MLflow, Weights & Biases, Hugging Face, LangChain, LangGraph, or comparable
  • Understanding of enterprise AI governance requirements: model lineage, data access controls, audit logging, and responsible AI guardrails
  • Experience with cloud-native architecture (AWS, Azure, GCP) and containerized ML workloads (Kubernetes, Docker)
  • Strong cross-functional influence: you can drive alignment across engineering, product, and customer-facing teams without formal authority
  • A portfolio of architectural decisions — RFCs, design docs, or open-source work — that demonstrates your approach
Job Responsibility
Job Responsibility
  • Define the technical architecture of Teradata's end-to-end AI development environment — the platform where data scientists, ML engineers, and AI developers build, test, deploy, and monitor AI and agentic applications on top of Vantage
  • Set the architectural direction for how AI Studio integrates with Teradata Vantage's query engine, model registry, feature store, and agent harness
  • Establish the patterns for how enterprise customers build trustworthy AI workflows — from data preparation through model deployment to agent-driven automation
  • Ensure that AI Studio is the most capable, governed, and scalable AI development environment in the market
  • Ship architectural decisions that other engineers can build on with confidence
  • Drive customer adoption of AI Studio at scale
What we offer
What we offer
  • Healthcare
  • life and disability insurance plans
  • 401(k)-retirement savings plan
  • time-off programs
  • Fulltime
Read More
Arrow Right

Principal MLOps Architect

We are seeking a highly experienced and hands-on Principal AI/ML Architect & App...
Location
Location
Salary
Salary:
Not provided
Intellias
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Master's or Ph.D. in Computer Science, Data Science, Machine Learning, or a related field
  • 10+ years of experience in AI/ML, data science, or distributed systems engineering.
  • Proven experience designing and deploying production-grade AI solutions at enterprise scale.
  • Strong background in both research and industrial AI environments.
  • Experience leading global or distributed technical teams.
  • Demonstrated success delivering AI transformation initiatives.
  • Large Language Models (LLMs)
  • Generative AI systems
  • NLP / NLU
  • Apache Spark
Job Responsibility
Job Responsibility
  • Lead the design and implementation of AI/ML solutions across multiple business domains.
  • Drive enterprise adoption of Large Language Models (LLMs), Generative AI, NLP/NLU, and advanced analytics solutions.
  • Define AI architecture standards, MLOps best practices, and scalable deployment strategies.
  • Evaluate emerging AI technologies and identify opportunities for innovation and operational impact.
  • Translate research initiatives into production-ready AI solutions.
  • Architect scalable distributed data-processing systems capable of handling large-scale datasets and real-time pipelines.
  • Design and optimize cloud-native AI platforms using modern data engineering frameworks.
  • Lead cloud migration and modernization initiatives from on-premises environments to Azure and/or AWS.
  • Implement efficient data pipelines leveraging Spark, Delta Lake, Databricks, Kubernetes, and containerized environments.
  • Ensure reliability, scalability, observability, and cost-efficiency of AI infrastructure.
  • Fulltime
Read More
Arrow Right

Principal AI Safety Engineer for Autonomous Vehicles Technical Lead

The AV Safety Strategy and Assessment team is seeking an AI Safety Technical Lea...
Location
Location
United States
Salary
Salary:
250600.00 - 384600.00 USD / Year
gm.com Logo
General Motors
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree in Computer Science, Electrical Engineering, Mathematics, Physics, or a related field
  • or equivalent practical experience
  • 10+ years of experience in AI/ML, engineering or a related field
  • 5+ years in autonomous vehicles, robotics or related field
  • Experience in Machine Learning & AI: Extensive experience in building large-scale models with significant focus on E2E validation. Experience using Large Language Models (LLMs), Generative AI, RAG, Deep learning, Reinforcement Learning, Natural Language Processing (NLP), SVM, XGBoost, Random Forest, Decision Trees, Clustering
  • AI Standards and Evolving Regulations: Understanding of ISO/PAS 8800, NIST AI Risk Management Framework, EU AI Act (2024-2027), other applicable industry standards and best practices for autonomous vehicles, aerospace and/or robotics.
  • Programming & Frameworks : Python, R, Java, PySpark, PyTorch, TensorFlow, Scikit-learn, LangChain, SQL
  • Cloud & Big Data Platforms: ( Preferred Microsoft Azure - Data Lake, Machine Learning, Databricks)
  • Deployment & MLOps: MLflow, Model Monitoring & Versioning, Docker & Kubernetes, GitHub, Jira
  • Data Analysis & Visualization : Tableau, PowerBI, Pandas, NumPy
Job Responsibility
Job Responsibility
  • Lead the development of AI safety strategies for ADS and establish safety engineering guidance and sufficiency criteria.
  • Actively engage with partners and seek input, provide technical expertise to inform leadership decision-making, and take ownership of technical projects
  • Define GM’s strategy for AI safety standards, engage externally to influence evolving standards, and contribute to internal and external thought leadership that strengthens GM’s position in the autonomous vehicle ecosystem.
  • Support regulatory rulemaking and policy responses related to AI safety-critical systems.
  • Establish an assurance plan and process to evaluate AI-related safety case evidence and verify that sufficiency criteria are met.
  • Provide AI expertise and safety guidance across Global Product Safety, Systems, and Certification activities.
  • Identify and drive opportunities to improve the efficiency and quality of safety work through the application of AI methodologies.
  • Mentor and develop team members, fostering a culture of technical excellence and continuous learning.
What we offer
What we offer
  • medical
  • dental
  • vision
  • Health Savings Account
  • Flexible Spending Accounts
  • retirement savings plan
  • sickness and accident benefits
  • life insurance
  • paid vacation & holidays
  • tuition assistance programs
  • Fulltime
Read More
Arrow Right
New

Principal AI Architect

As the Principal AI Architect for Teradata AI Studio, you will define the techni...
Location
Location
India , Bengaluru, Karnataka, India | Hyderabad, Telangana, India
Salary
Salary:
Not provided
teradata.com Logo
Teradata
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 10+ years of software engineering experience, including 3+ years in a senior architect or principal engineer role with platform-wide technical scope
  • Demonstrated expertise designing AI/ML platforms or developer tools: model serving infrastructure, feature stores, experiment tracking, MLOps pipelines, or AI agent development environments
  • Deep understanding of LLM integration patterns: RAG architectures, fine-tuning pipelines, evaluation frameworks, and agent tool-calling interfaces
  • Experience with enterprise data platforms (Teradata Vantage, Snowflake, Databricks, or equivalent) at sufficient depth to architect against their APIs, security models, and performance characteristics
Job Responsibility
Job Responsibility
  • Define the technical architecture of Teradata's end-to-end AI development environment
  • Set the architectural direction for how AI Studio integrates with Teradata Vantage's query engine, model registry, feature store, and agent harness
  • Establish the patterns for how enterprise customers build trustworthy AI workflows
  • Ensure that AI Studio is the most capable, governed, and scalable AI development environment in the market
What we offer
What we offer
  • Flexible work model
  • People-first culture
  • Focus on well-being
  • Commitment to fostering an inclusive environment
  • Fulltime
Read More
Arrow Right