CrawlJobs Logo
Mastercard Logo Mastercard · IT - Software Development

Senior Data Engineer (DATABRICKS)

Ireland, Dublin 18 · Job Posted February 01, 2026
Apply Position
Job Link Share

Job Description

As a Senior Databricks Administrator, you will be responsible for the setup, configuration, administration, and optimization of the Databricks Platform on AWS. This role will play a critical part in managing secure, scalable, and high-performing Databricks environments, with a strong focus on governance, user access management, cost optimization, and platform operations. You will collaborate closely with engineering, infrastructure, and compliance teams to ensure that the Databricks platform meets enterprise data and regulatory requirements.

Job Responsibility

  • Databricks Platform Administration (AWS) Provision, configure, and manage Databricks Workspaces across multiple environments (Dev, UAT, Prod)
  • Define and enforce cluster policies, compute tagging, and usage controls to optimize cost and resource utilization
  • Administer Unity Catalog, including creation and management of catalogs, schemas, access control, and data lineage
  • Manage SCIM integrations for user/group provisioning with IdPs like Okta or Azure/AWS SSO
  • Configure and maintain CI/CD pipelines for notebook deployment and workflow promotion using GitHub, GitLab, or similar tools
  • Security, Governance & Compliance Implement and enforce role-based access controls (RBAC) and fine-grained data permissions using Unity Catalog and Lake Formation
  • Ensure auditability and lineage tracking for compliance with Open Banking, GDPR, and PSD2 regulations
  • Configure and manage token scopes, secrets management, and credential passthrough for secure access to underlying data
  • Work with InfoSec and compliance teams to ensure platform security aligns with regulatory frameworks
  • Monitoring, Support & Troubleshooting Monitor workspace performance, cluster health, job execution, and user activity using Databricks native tools, CloudWatch, and third-party integrations
  • Provide L2/L3 support for Databricks usage issues, including notebook errors, job failures, and workspace anomalies
  • Maintain operational runbooks, automation scripts, and alerting mechanisms for platform health and governance events
  • Automation & Best Practices Build and manage Terraform/IaC scripts for environment provisioning and infrastructure consistency
  • Define naming conventions, resource tagging standards, and workspace governance guidelines
  • Promote standardization of reusable notebook templates, shared cluster configurations, and workflow orchestrations
  • Drive internal knowledge sharing, onboarding, and platform enablement sessions

Requirements

  • 6+ years of experience in Databricks administration on AWS or multi-cloud environments
  • Deep understanding of Databricks workspace architecture, Unity Catalog, and cluster configuration best practices
  • Strong experience in managing IAM policies, SCIM integration, and access provisioning workflows
  • Hands-on experience with monitoring, cost optimization, and governance of large-scale Databricks deployments
  • Hands-on experience with infrastructure-as-code (Terraform) and CI/CD pipelines
  • Experience with ETL orchestration and collaboration with engineering teams (Databricks Jobs, Workflows, Airflow)

Nice to have

  • Understanding of Delta Lake, Lakehouse architecture, and data governance patterns
  • Experience in AWS Glue and database technologies SQL (Amazon Aurora) and NoSQL (MongoDB/DynamoDB)
  • Exposure to regulated industries such as banking, fintech, or healthcare
  • Experience with incident response, audit logging, and security remediation
Mastercard - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Senior Data Engineer (DATABRICKS)

8 matching positions

Senior Databricks Data Engineer

At LeverX, we have had the privilege of delivering over 1,500 projects for vario...
Location
Location
Georgia
Salary
Salary:
Not provided
leverx.com Logo
LeverX
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years in Data Engineering
  • Strong experience with Databricks: Spark (PySpark + SQL), Delta Lake internals, Jobs, Workflows, SQL Warehouses, Unity Catalog
  • Excellent SQL skills (analytical queries, performance tuning)
  • Strong Python for data engineering
  • Experience with dbt (models, tests, exposures, CI/CD)
  • Experience with orchestration tools (Airflow, Dagster)
  • Experience designing scalable lakehouse architectures
  • Understanding of cost optimization in Databricks (clusters, warehouses, storage layout)
  • English B2+
Job Responsibility
Job Responsibility
  • Design and build scalable data pipelines on Databricks using PySpark, Databricks SQL, and Delta Lake
  • Develop, optimize, and troubleshoot Spark workloads for performance and reliability
  • Manage Delta Lake tables, including Z-Ordering, compaction, schema evolution, retention, and file layout optimization
  • Configure and administer Databricks Jobs, clusters, SQL Warehouses, and workflows
  • Own data governance using Unity Catalog, including permissions, lineage, and catalog/schema management
  • Implement data transformations and medallion architecture (bronze–silver–gold) using dbt
  • Build CI/CD pipelines and ensure data quality, observability, SLAs, auditing, and monitoring
  • Collaborate with and mentor engineers, partner with analytics and ML teams, and drive continuous platform improvements
What we offer
What we offer
  • Projects in different domains: healthcare, manufacturing, e-commerce, fintech, etc.
  • Projects for every taste: Startup products, enterprise solutions, research & development initiatives, and projects at the crossroads of SAP and the latest web technologies
  • Global clients based in Europe and the US, including Fortune 500 companies
  • Employment security: We hire for our team, not just a specific project. If your project ends, we will find you a new one
  • Healthy work atmosphere: On average, our employees stay with the company for 4+ years
  • Market-based compensation and regular performance reviews
  • Internal expert communities and courses
  • Perks to support your growth and well-being
Read More
Arrow Right

Senior Data Engineer – Databricks & SaaS Integrations

We are seeking a Senior Data Engineer with strong Databricks and backend integra...
Location
Location
India , Pune
Salary
Salary:
Not provided
Codvo AI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong experience with Databricks, Apache Spark, and data pipelines
  • Backend development experience (Python, Java, or Scala)
  • Experience integrating data platforms with SaaS application backends
  • Strong knowledge of REST APIs and event-driven architectures
  • Experience working with AWS or Azure cloud platforms
  • Understanding of enterprise security (authN, authZ, RBAC)
  • Experience with spec-driven development and coding agents
Job Responsibility
Job Responsibility
  • Design and implement integrations between Databricks workflows and SaaS application backends
  • Enable batch and streaming data flows, event triggers, and API-based communication
  • Build backend services such as REST APIs, webhooks, and data interfaces
  • Implement data quality, validation, monitoring, and observability mechanisms
  • Design secure and auditable pipelines including authN, authZ, and RBAC
  • Troubleshoot pipeline failures, performance bottlenecks, and data inconsistencies
  • Collaborate with application, platform, and data engineering teams
  • Fulltime
Read More
Arrow Right

Senior Data Engineer with Databricks

At KMD, we build digital solutions used by millions of people around the world. ...
Location
Location
Poland , Warsaw
Salary
Salary:
160.00 - 180.00 PLN / Hour
kmd.net Logo
KMD Poland
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of commercial experience in implementing, developing, or maintaining data load systems (ETL/ELT)
  • Strong programming skills in Python, with a deep understanding of data-related challenges
  • Hands-on experience with Apache Spark and Databricks
  • Familiar with MSSQL databases
  • Experience working with distributed systems on a cloud platform
  • Worked on large-scale systems and understands performance optimization
  • Comfortable with Git and CI/CD practices, and can design or implement deployment processes for data pipelines
  • Proactive, solution-oriented, and has a strong can-do attitude
  • Communicates fluently in English and Polish, both written and spoken
  • Team player with excellent collaboration and communication skills
Job Responsibility
Job Responsibility
  • Develop and maintain data delivery pipelines for a leading IT solution in the energy market, leveraging Apache Spark, Databricks, Delta Lake, and Python
  • Have end-to-end responsibility for the full lifecycle of features you develop
  • Design technical solutions for business requirements from the product roadmap
  • Ensure optimal performance
  • Refactor existing code and enhance system architecture to improve maintainability and scalability
  • Design and evolve the test automation strategy, including technology stack and solution architecture
  • Prepare reviews, participate in retrospectives, estimate user stories, and refine features ensuring their readiness for development
What we offer
What we offer
  • KMD (an NEC company) is committed to providing equal opportunities. Hence, we invite all qualified interested applicants to apply for career opportunities. At KMD all aspects of employment and cooperation including the decision to hire/cooperate with will be based on merit, competence, performance, and business needs without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, or any other status protected under local anti-discrimination legislation
Read More
Arrow Right

Senior Azure Data Engineer with Databricks

Location
Location
Poland
Salary
Salary:
Not provided
dcg.pl Logo
DCG Sp. z o. o.
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 3+ years’ experience with Azure Data Factory and Databricks
  • 5+ years’ experience with data engineering or backend/fullstack software development
  • Strong SQL skills
  • Python scripting proficiency
  • Experience with data transformation tools - Databricks and Spark
  • Experience in structuring and modelling data in both relational and non-relational forms
  • Experience with CI/CD tooling
  • Working knowledge of Git
  • English level: B2, C1
Job Responsibility
Job Responsibility
  • Being responsible for at-scale infrastructure design, build and deployment with a focus on distributed systems
  • Building and maintaining architecture patterns for data processing, workflow definitions, and system to system integrations using Big Data and Cloud technologies
  • Evaluating and translating technical design to workable technical solutions/code and technical specifications at par with industry standards
  • Driving creation of re-usable artifacts
  • Establishing scalable, efficient, automated processes for data analysis, data model development, validation, and implementation
  • Working closely with analysts/data scientists to understand impact to the downstream data models
  • Writing efficient and well-organized software to ship products in an iterative, continual release environment
  • Contributing and promoting good software engineering practices across the team
  • Communicating clearly and effectively to technical and non-technical audiences
  • Defining data retention policies
What we offer
What we offer
  • Private medical care
  • Co-financing for the sports card
  • Constant support of dedicated consultant
  • Employee referral program
Read More
Arrow Right

Senior / Staff Full-Stack Data Engineer – Databricks

At Codvo, we are committed to building scalable, future-ready data platforms tha...
Location
Location
India , Pune
Salary
Salary:
Not provided
Codvo AI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong hands-on experience with Databricks, Apache Spark, and Delta Lake
  • Proven experience building and operating production-grade data pipelines
  • Experience operationalizing machine learning models and inference pipelines
  • Strong understanding of data reliability, observability, and monitoring practices
  • Experience with CI/CD, DevOps, and MLOps workflows
  • Experience working with cloud platforms (AWS or Azure)
  • Familiarity with Unity Catalog and enterprise data governance concepts
  • Experience with spec-driven development and coding agents
Job Responsibility
Job Responsibility
  • Design, build, and maintain ETL/ELT pipelines on Databricks using Spark, Delta Lake, and Databricks Workflows
  • Build and operate batch and real-time data pipelines for ingestion, transformation, and orchestration
  • Operationalize machine learning inference pipelines authored by data scientists (batch and real-time)
  • Ensure consistency between model training and inference environments
  • Implement data quality checks, validation rules, monitoring, alerting, and automated recovery
  • Collaborate with data scientists to productionize models and optimize inference performance and cost
  • Implement CI/CD, DevOps, and MLOps best practices for data pipelines and ML workflows
  • Optimize compute, storage, and job configurations for performance and cost efficiency
  • Implement and manage enterprise data governance using Unity Catalog (schemas, lineage, ownership, documentation)
  • Work with Databricks infrastructure and platform configurations
  • Fulltime
Read More
Arrow Right

Senior Staff Data Engineer- Data Platform

At Marktplaats, data is at the heart of everything we do, but Intelligence is wh...
Location
Location
Netherlands , Amsterdam
Salary
Salary:
Not provided
adevinta.com Logo
Adevinta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 10+ years of hands-on experience in Software Development or Data Engineering
  • at least 5+ years specifically focused on building Data Platforms
  • deep understanding of how Platform infra supports Analytics workloads
  • proven experience evolving complex platforms from legacy patterns to modern, cloud-native solutions
  • deep knowledge of Spark internals, JVM tuning, and performance optimization for high-scale batch and streaming datasets
  • deep expertise in Unity Catalog, Delta Lake internals, and optimizing high-volume workloads
  • strict software engineering discipline (CI/CD, Testing, OOP) applied to data pipelines
  • understanding of microservices architecture
  • understanding the needs of Analytics/DWH teams (Data Modeling, dbt)
  • strong background in building automated pipelines using Terraform/Terragrunt and ensuring system observability
Job Responsibility
Job Responsibility
  • Lead the evolution of our Data Platform and architect the "Data Exchange" strategy
  • define robust patterns for API-based ingestion, Event-Driven Architectures (Kafka), and Reverse ETL
  • ensure architectures are optimized for cost and performance on AWS
  • act as a catalyst for technical evolution
  • constantly scan the horizon for next-generation technologies
  • lead the implementation of new paradigms
  • design the strategy for Unity Catalog implementation and Data Contracts
  • champion FinOps, automating cost controls for our highest-volume workloads
  • build the underlying infrastructure that allows Analytics/DWH teams to run efficient transformations
  • elevate the technical bar of the team, mentoring Staff and Senior engineers
What we offer
What we offer
  • An attractive Base Salary
  • Participation in our Short Term Incentive plan (annual bonus)
  • Work From Anywhere: Enjoy up to 20 days a year of working from anywhere
  • A 24/7 Employee Assistance Program for you and your family
  • a collaborative environment with an opportunity to explore your potential and grow
  • a range of locally relevant benefits
  • Fulltime
Read More
Arrow Right

Senior Data Entitlements Engineer - Market Data Platform

Join us as a Senior Data Entitlements Engineer - Market Data Platform at Barclay...
Location
Location
Czechia , Prague
Salary
Salary:
Not provided
barclays.co.uk Logo
Barclays
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Expert-level experience designing and implementing entitlement and license governance frameworks for data platforms, including defining policies, rules, and workflows that control access to sensitive market data at scale
  • Expertise and hands-on experience with entitlements engines and access-control models (RBAC, ABAC, ...), with a proven track record of building or configuring fine-grained permission structures in production environments
  • Demonstrated experience implementing and managing entitlements across multiple analytics and data platforms — specifically AWS (IAM, Lake Formation), Snowflake, Databricks, and/or KDB — understanding platform-specific security models and how to harmonise them under a unified governance layer
  • Strong AWS cloud experience, including identity federation, and security services relevant to data access and governance
Job Responsibility
Job Responsibility
  • Build and maintenance of data architectures pipelines that enable the transfer and processing of durable, complete and consistent data
  • Design and implementation of data warehoused and data lakes that manage the appropriate data volumes and velocity and adhere to the required security measures
  • Development of processing and analysis algorithms fit for the intended data complexity and volumes
  • Collaboration with data scientist to build and deploy machine learning models
What we offer
What we offer
  • Competitive holiday allowance
  • Life assurance
  • Private medical care
  • Pension contribution
  • Fulltime
Read More
Arrow Right

Data Engineer - Big Data, Python, Databricks

The Applications Development Intermediate Programmer Analyst is an intermediate ...
Location
Location
India , Chennai
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 4 - 6 years of relevant experience in the Financial Service industry
  • Intermediate level experience in Applications Development role
  • Proficient in Big DataTechnologies - Cloudera, Hive, Python, Java/PySpark, Data Bricks
  • Proficient in Data analysis and data modelling
  • Good understanding of ETL concepts
  • Consistently demonstrates clear and concise written and verbal communication
  • Demonstrated problem-solving and decision-making skills
  • Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
  • Bachelor's degree/University degree or equivalent experience
Job Responsibility
Job Responsibility
  • Utilize knowledge of applications development procedures and concepts, and basic knowledge of other technical areas to identify and define necessary system enhancements, including using script tools and analyzing/interpreting code
  • Consult with users, clients, and other technology groups on issues, and recommend programming solutions, install, and support customer exposure systems
  • Apply fundamental knowledge of programming languages for design specifications
  • Analyze applications to identify vulnerabilities and security issues, as well as conduct testing and debugging
  • Serve as advisor or coach to new or lower level analysts
  • Identify problems, analyze information, and make evaluative judgements to recommend and implement solutions
  • Resolve issues by identifying and selecting solutions through the applications of acquired technical experience and guided by precedents
  • Has the ability to operate with a limited level of direct supervision
  • Can exercise independence of judgement and autonomy
  • Acts as SME to senior stakeholders and /or other team members
  • Fulltime
Read More
Arrow Right