CrawlJobs Logo
Optiver Logo Optiver · IT - Software Development

Databricks Platform Engineer

United States, Chicago 150000.00 - 225000.00 USD / Year · Job Posted January 29, 2026
Apply Position
Job Link Share

Job Description

Optiver is seeking a Databricks Platform Engineer to manage, and operate Optiver’s deployment of the Databricks platform. This role focuses on infrastructure, automation, and operational excellence, enabling teams to use Databricks effectively while maintaining strong security, consistency, and governance. As markets grow more data-driven and competitive, our ability to ingest, model, and deliver high-quality data at scale is fundamental to maintaining our trading edge. This role will offer ownership of a critical internal platform in a high-impact role with influence over platform standards and direction.

Job Responsibility

  • Design, deploy, and operate Databricks workspaces and environments
  • Manage workspace configuration, cluster policies, access controls, and platform settings
  • Establish and enforce standards for reliability, security, and cost governance
  • Build and maintain infrastructure using Terraform
  • Create repeatable, version-controlled deployment patterns across environments
  • Automate platform operations and maintenance using Python and shell scripting
  • Design and maintain CI/CD systems for infrastructure and platform configuration
  • Implement automated validation, testing, and promotion of configuration changes
  • Support Git-based workflows and environment lifecycle management
  • Troubleshoot Platform level issues
  • Help manage system configuration, networking, and access controls
  • Improve observability, logging, and operational tooling
  • Implement role-based access control, secrets management, and security best practices
  • Ensure platform compliance with internal security and operational standards
  • Participate in incident response, root cause analysis, and reliability improvements

Requirements

  • 2+ years experience administering Databricks environments
  • Strong experience with Terraform and infrastructure-as-code practices
  • Proficiency in Python for automation and tooling
  • Solid understanding of CI/CD principles and tooling
  • Experience with Git and collaborative development workflows
  • Proficiency in using Linux CLI
  • Understanding of cloud infrastructure concepts (networking, IAM, compute, storage)
  • Familiarity with monitoring, alerting, and operational best practices

Nice to have

  • Experience with AWS infrastructure using Terraform is beneficial
  • Ideally experience operating shared developer platforms or internal tooling as well as with multi-environment or multi-account setups
  • Knowledge of cloud cost management and governance is nice to have

What we offer

  • The opportunity to work alongside best-in-class professionals from over 40 different countries
  • Highly competitive compensation package
  • Global profit-sharing pool and performance-based bonus structure
  • 401(k) match up to 50%
  • Comprehensive health, mental, dental, vision, disability, and life coverage
  • 25 paid vacation days alongside market holidays
  • Extensive office perks, including breakfast, lunch and snacks, regular social events, clubs, sporting leagues and more
Optiver - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Databricks Platform Engineer

8 matching positions

Databricks Platform Engineer

Bright Vision Technologies is looking for a skilled Databricks Platform Engineer...
Location
Location
United States , Bridgewater
Salary
Salary:
Not provided
bvteck.com Logo
Bright Vision Technologies
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Databricks Lakehouse Platform
  • Apache Spark
  • Delta Lake
  • Databricks Workflows
  • Databricks SQL
  • Cloud Platforms (AWS / Azure / GCP)
  • Data Engineering Pipelines
  • ETL/ELT
  • Python
  • Scala
What we offer
What we offer
  • H-1B sponsorship for the 2026 quota
  • Equal employment opportunity
  • Inclusive work environment
  • Fulltime
Read More
Arrow Right

Staff Engineer I – Azure Databricks Engineer

Location
Location
United States , Irving
Salary
Salary:
Not provided
softwareresources.com Logo
Software Resources
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of related experience in data analytics administration and development
  • 2+ years of Databricks related experience
  • Bachelor's degree in related field required
  • Hands-on experience in Azure Databricks (Workspace management, Clusters, Jobs, Unity Catalog, Delta Lake, User access management, Rest APIs and SDKs)
  • Understanding of Azure infrastructure and data services, including Azure Data Lake, Azure Data Factory, Azure SQL, Azure Synapse Analytics, Azure Key Vault, Azure Monitor, networking
  • Experience with CI/CD pipelines (Azure DevOps preferred)
  • Strong programming skills in SQL, Python, and/or PySpark
  • Proven experience in leading cross-functional teams and managing multiple projects simultaneously
  • Intermediate to advanced ability to see the big picture and align projects with organizational goals
  • Capable of leading and motivating cross-functional teams
Job Responsibility
Job Responsibility
  • Delivery and operations of technologies and platforms required to model, transform, analyze, report, visualize data
  • Building, optimizing, streamlining and automating the Azure Databricks platform to enable analytical workloads like Machine learning (ML), Model Development, Data Insights, Data Science etc
  • Partner with ML engineers, data scientists, data analysts, and enterprise architects to enforce best practices, train and enable users using the Azure Databricks platform
  • Take assignments that can be worked on individually without supervision and manage work effort from concept to completion
  • Build, optimize and maintain the Azure Databricks platform, ensuring scalability, security, governance and performance
  • Implement and manage Azure Databricks workspaces, clusters, jobs, access management
  • Implement and manage policies, monitoring and observability for the Azure Databricks platform
  • Ensure compliance with IT policies, procedures, and industry standards, including reviewing and refining IT control enhancements
  • Work closely with business and analytics teams to ensure reliable and governed data access for data needed for analytics
  • Troubleshoot platform issues, optimize performance, and ensure uptime for critical Databricks services
What we offer
What we offer
  • Competitive salaries
  • An ownership stake in the company
  • Medical and dental insurance
  • Time off
  • A great 401k matching program
  • Tuition assistance program
  • An employee volunteer program
  • A wellness program
Read More
Arrow Right

Staff Engineer I – Azure Databricks Engineer

Software Resources has an immediate, direct hire job opportunity for a Staff Eng...
Location
Location
United States , Irving, Dallas
Salary
Salary:
Not provided
softwareresources.com Logo
Software Resources
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of related experience in data analytics administration and development
  • 2+ years of Databricks related experience
  • Bachelor’s degree in related field required
  • Hands-on experience in Azure Databricks (Workspace management, Clusters, Jobs, Unity Catalog, Delta Lake, User access management, Rest APIs and SDKs)
  • Understanding of Azure infrastructure and data services, including Azure Data Lake, Azure Data Factory, Azure SQL, Azure Synapse Analytics, Azure Key Vault, Azure Monitor, networking
  • Experience with CI/CD pipelines (Azure DevOps preferred)
  • Strong programming skills in SQL, Python, and/or PySpark
  • Proven experience in leading cross-functional teams and managing multiple projects simultaneously
  • Intermediate to advanced ability to see the big picture and align projects with organizational goals. Capable of leading and motivating cross-functional teams. Expertise in resolving conflicts and addressing challenges as well as skilled at identifying and mitigating risks at the project level
  • Good communication skills, excellent teamwork experience
Job Responsibility
Job Responsibility
  • Responsible for delivery and operations of technologies and platforms required to model, transform, analyze, report, visualize data
  • Work in a mid-level role with proficiency needed in building, optimizing, streamlining and automating the Azure Databricks platform to enable analytical workloads like Machine learning (ML), Model Development, Data Insights, Data Science etc
  • Partner with ML engineers, data scientists, data analysts, and enterprise architects to enforce best practices, train and enable users using the Azure Databricks platform
  • Take assignments that can be worked on individually without supervision and manage work effort from concept to completion
  • Build, optimize and maintain the Azure Databricks platform, ensuring scalability, security, governance and performance
  • Implement and manage Azure Databricks workspaces, clusters, jobs, access management
  • Implement and manage policies, monitoring and observability for the Azure Databricks platform
  • Ensure compliance with IT policies, procedures, and industry standards, including reviewing and refining IT control enhancements
  • Work closely with business and analytics teams to ensure reliable and governed data access for data needed for analytics
  • Troubleshoot platform issues, optimize performance, and ensure uptime for critical Databricks services
What we offer
What we offer
  • Competitive salaries
  • An ownership stake in the company
  • Medical and dental insurance
  • Time off
  • A great 401k matching program
  • Tuition assistance program
  • An employee volunteer program
  • A wellness program
  • Opportunity to bolster your business knowledge, learning the ins and outs of how successful companies operate and manage their finances, giving you invaluable hands-on experience to help grow your career
  • Fulltime
Read More
Arrow Right
New

Databricks Engineer

Design, develop, and maintain scalable data pipelines, ETL/ELT processes, and da...
Location
Location
Salary
Salary:
Not provided
myticas.com Logo
Myticas Consulting
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Databricks, PySpark, Delta Lake
  • Azure Databricks, Azure Data Lake Storage (ADLS Gen2)
  • Python, SQL, Spark
  • Snowflake, business intelligence tools
  • Azure Data Factory (ADF), Azure Functions
  • Databricks Unity Catalog
  • Azure DevOps, GitHub Actions, Infrastructure-as-Code (Terraform)
  • Spark, Kafka, Event Hubs
  • Airflow, dbt
Job Responsibility
Job Responsibility
  • Design, develop, and maintain scalable data pipelines, ETL/ELT processes, and data integration solutions using Databricks, PySpark, and Delta Lake
  • Build and optimize cloud-native lakehouse architectures leveraging Azure Databricks, Azure Data Lake Storage (ADLS Gen2), and modern data engineering best practices
  • Develop robust data ingestion frameworks supporting structured, semi-structured, and unstructured data sources
  • Create high-performance data transformation workflows using Python, SQL, Spark, and Databricks notebooks
  • Design, implement, and optimize data models, schemas, partitioning strategies, and storage structures for large-scale analytics workloads
  • Integrate Databricks environments with downstream platforms including Snowflake, business intelligence tools, APIs, enterprise applications, and reporting solutions
  • Utilize Azure Data Factory (ADF), Azure Functions, and Azure ecosystem services to support end-to-end data platform operations
  • Implement and manage Databricks Unity Catalog, data governance frameworks, security controls, data lineage, and access management processes
  • Monitor, troubleshoot, tune, and optimize data pipeline performance, reliability, scalability, and operational efficiency
  • Develop and support CI/CD pipelines using Azure DevOps, GitHub Actions, Infrastructure-as-Code (Terraform), and modern DevOps practices
Read More
Arrow Right

Senior Data Engineer – Databricks & SaaS Integrations

We are seeking a Senior Data Engineer with strong Databricks and backend integra...
Location
Location
India , Pune
Salary
Salary:
Not provided
Codvo AI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong experience with Databricks, Apache Spark, and data pipelines
  • Backend development experience (Python, Java, or Scala)
  • Experience integrating data platforms with SaaS application backends
  • Strong knowledge of REST APIs and event-driven architectures
  • Experience working with AWS or Azure cloud platforms
  • Understanding of enterprise security (authN, authZ, RBAC)
  • Experience with spec-driven development and coding agents
Job Responsibility
Job Responsibility
  • Design and implement integrations between Databricks workflows and SaaS application backends
  • Enable batch and streaming data flows, event triggers, and API-based communication
  • Build backend services such as REST APIs, webhooks, and data interfaces
  • Implement data quality, validation, monitoring, and observability mechanisms
  • Design secure and auditable pipelines including authN, authZ, and RBAC
  • Troubleshoot pipeline failures, performance bottlenecks, and data inconsistencies
  • Collaborate with application, platform, and data engineering teams
  • Fulltime
Read More
Arrow Right

Senior Databricks Engineer

We are looking for an experienced Senior Databricks Engineer to support enterpri...
Location
Location
United States , Cincinnati
Salary
Salary:
Not provided
https://www.roberthalf.com Logo
Robert Half
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • At least 5 years of experience in data engineering working with large-scale or distributed data processing platforms
  • Minimum 3 years of hands-on experience building solutions in Databricks within production environments
  • Strong coding ability in Python, especially PySpark, with additional SQL proficiency for data transformation and analysis
  • Solid knowledge of Apache Spark concepts, including execution behavior, performance tuning, and resource management
  • Practical experience with Delta Lake features such as schema evolution, versioned data management, and reliable transactional processing
  • Familiarity with cloud-based data ecosystems and orchestration tools used to schedule, monitor, and manage data workloads
  • Experience using version control and CI/CD practices to support repeatable and controlled data engineering deployments
  • ServiceNow knowledge is highly valued, especially for roles involving data extraction and integration into downstream platforms
Job Responsibility
Job Responsibility
  • Develop and maintain robust data pipelines in Databricks using Apache Spark to support enterprise reporting and analytical use cases
  • Extract and transform data from ServiceNow, then create and manage structured tables that supply ARDoc as part of a larger architecture initiative
  • Design layered data models using Delta Lake practices to improve scalability, data usability, and long-term maintainability
  • Configure and manage Databricks jobs, workflows, and compute resources to improve processing efficiency and platform stability
  • Investigate performance issues across Spark workloads and implement tuning strategies that improve speed, reliability, and cost control
  • Establish data governance practices through cataloging, lineage visibility, and secure access controls within the Databricks environment
  • Work closely with data scientists, analysts, and platform teams to deploy production-ready data assets and support advanced analytics needs
  • Provide technical guidance to other engineers by promoting engineering standards, reviewing designs, and sharing platform best practices
What we offer
What we offer
  • medical
  • vision
  • dental
  • life and disability insurance
  • 401(k) plan
Read More
Arrow Right

Principal Data Genai Platform Engineer - Senior Vice President

Location
Location
India , Chennai
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 12+ years of relevant experience in enterprise application development, data engineering, or AI platform engineering, with a strong track record of leadership in regulated environments
  • 8+ years of experience leading multi-team Agile organizations (20+ engineers), including managing distributed and hybrid AI-assisted teams
  • Advanced expertise in Python, PySpark, and Databricks ecosystem for large-scale data processing and ELT/ETL pipelines
  • Proven experience architecting and implementing enterprise AI/GenAI platforms, including agentic AI frameworks, LLM integrations, and prompt engineering
  • Hands-on experience with AI-assisted development tools such as Devin.AI and GitHub Copilot and integrating them into engineering workflows
  • Strong experience with microservices architecture, APIs, and cloud-native deployment (Kubernetes/OpenShift)
  • Strong experience with event-driven architectures and streaming platforms (Kafka)
  • Deep understanding of data architecture, data mesh, data federation, and regulatory data requirements
  • Exceptional leadership, communication, stakeholder management, and decision-making capabilities
  • Experience with cloud platforms (AWS, Azure, GCP, Databricks) and modern data ecosystems
Job Responsibility
Job Responsibility
  • Lead multiple agile scrum teams comprising ~15+ engineers, including hybrid teams of human engineers and AI-assisted development (Devin.AI, Copilot), ensuring delivery excellence and alignment with business priorities
  • Define and execute the enterprise strategy for Python engineering, AI agent platforms, and full-stack data applications, aligned with Retail and Wealth Risk objectives
  • Serve as the senior architect and technical authority for enterprise-scale AI agents, data engineering pipelines, and microservices-based applications, ensuring scalability, resilience, and security
  • Drive the adoption and operationalization of AI Product Development Lifecycle (AI PDLC), including model governance, evaluation, deployment, monitoring, and compliance with Model Risk Management (MRM)
  • Lead development of high-volume data pipelines and data federation layers using PySpark, Databricks, Kafka, and Data Mesh architecture to support regulatory reporting (CCAR, FDIC) and risk analytics
  • Architect and oversee GenAI agent ecosystems using LLMs (Google ADK, Gemini/Flash), implementing Human-in-the-Loop (HITL) frameworks to ensure explainability, auditability, and compliance
  • Drive AI-augmented software development lifecycle, integrating tools such as Devin.AI, GitHub Copilot, and MCP platforms through advanced prompt engineering and governance guardrails
  • Lead microservices and cloud-native architecture using FastAPI/Spring Boot, Kubernetes/OpenShift, and CI/CD pipelines, ensuring high availability and performance
  • Drive engineering efficiency and standardization by reusing and repurposing enterprise-level frameworks, platforms, and tools, reducing duplication and accelerating delivery across teams
  • Ensure all engineering solutions incorporate data governance and non-functional requirements, including Data Quality (DQ), data lineage, data tracing, and auditability, aligned with enterprise governance processes and regulatory expectations
  • Fulltime
Read More
Arrow Right

Senior / Staff Full-Stack Data Engineer – Databricks

At Codvo, we are committed to building scalable, future-ready data platforms tha...
Location
Location
India , Pune
Salary
Salary:
Not provided
Codvo AI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong hands-on experience with Databricks, Apache Spark, and Delta Lake
  • Proven experience building and operating production-grade data pipelines
  • Experience operationalizing machine learning models and inference pipelines
  • Strong understanding of data reliability, observability, and monitoring practices
  • Experience with CI/CD, DevOps, and MLOps workflows
  • Experience working with cloud platforms (AWS or Azure)
  • Familiarity with Unity Catalog and enterprise data governance concepts
  • Experience with spec-driven development and coding agents
Job Responsibility
Job Responsibility
  • Design, build, and maintain ETL/ELT pipelines on Databricks using Spark, Delta Lake, and Databricks Workflows
  • Build and operate batch and real-time data pipelines for ingestion, transformation, and orchestration
  • Operationalize machine learning inference pipelines authored by data scientists (batch and real-time)
  • Ensure consistency between model training and inference environments
  • Implement data quality checks, validation rules, monitoring, alerting, and automated recovery
  • Collaborate with data scientists to productionize models and optimize inference performance and cost
  • Implement CI/CD, DevOps, and MLOps best practices for data pipelines and ML workflows
  • Optimize compute, storage, and job configurations for performance and cost efficiency
  • Implement and manage enterprise data governance using Unity Catalog (schemas, lineage, ownership, documentation)
  • Work with Databricks infrastructure and platform configurations
  • Fulltime
Read More
Arrow Right