CrawlJobs Logo
NTT DATA Logo NTT DATA · IT - Software Development

Data Engineer Sr (Databricks)

India, Bangalore · Job Posted June 10, 2026
Apply Position
Job Link Share

Job Responsibility

  • Develop Databricks notebooks, jobs, and workflows to replicate and enhance DB2/Guidewire-based pipelines and transformations
  • Implement Delta Lake tables and patterns (bronze/silver/gold, ACID, time travel, schema evolution) for migrated data
  • Integrate Databricks with AWS/S3 or Azure ADLS, ADF/Synapse, Key Vault, and Snowflake as required
  • Optimize Databricks clusters, jobs, and queries for performance and cost
  • Implement incremental loads, CDC patterns, and batch schedules for large datasets
  • Collaborate with Snowflake and dbt teams to ensure consistent data models and data contracts
  • Participate in data validation and reconciliation between DB2 400 / Guiderwire and Databricks outputs
  • Follow coding standards, version control, and CI/CD practices using Git/Azure DevOps
  • Provide defect fixes and support during SIT/UAT and post go-live stabilization

Requirements

  • 8+ years of experience in databricks with understanding complex legacy data models and getting that data into the cloud: Extracting DB2/AS400: Experience with Change Data Capture (CDC) or scheduled batch extractions from DB2 into cloud storage. Involves working through JDBC connections, mapping table dependencies, and re-platforming legacy SQL to distributed computing standards
  • Handling Guidewire Data: Integrating with Guidewire Cloud Data Access (CDA) or InsuranceSuite to replicate complex P&C (Property & Casualty) insurance schemas. Senior engineers parse these highly normalized operational databases and transform them into analytical-friendly schemas in the cloud
  • 2. Architecture & Pipeline Development: The core of the experience involves transitioning these legacy, row-based stores into a scalable Medallion Architecture (Bronze, Silver, Gold layers)
  • Delta Lake Optimization: Using Databricks and Apache Spark to build ETL/ELT data pipelines with ACID transactions. Senior engineers handle schema evolution, upserts, and slowly changing dimensions (SCD Type 2)
  • Business Logic Refactoring: Translating rigid legacy procedural code (e.g., RPG/COBOL background logic, stored procedures) into scalable distributed patterns (PySpark, Spark SQL, and Scala)
  • 3. Data Governance & Observability: A senior engineer is expected to govern vast amounts of incoming and generated data across the enterprise
  • Unity Catalog: Implementing strict data governance, lineage tracing, and table-level security
  • Data Quality: Automating data validation frameworks to ensure a seamless transition from legacy to modern systems without data loss or corruption
  • 4. Integration with the Databricks Platform Ecosystem: Moving beyond basic storage to utilizing the full power of the Databricks Data Intelligence Platform
  • Serverless Compute: Managing Databricks serverless resources, ensuring optimal cluster sizing, and reducing compute costs
  • Streaming and Batch Workflows: Building event-driven pipelines using features like Databricks Auto Loader to ingest flat files and streaming records directly into Delta tables
NTT DATA - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Data Engineer Sr (Databricks)

8 matching positions

Sr. Data Engineer – Clinical Data Foundation

The Sr. Data Engineer is responsible for designing, building, maintaining, analy...
Location
Location
India , Hyderabad
Salary
Salary:
Not provided
amgen.com Logo
Amgen
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Master’s /Bachelor’s degree with 9-12 years of experience in Computer Science, IT or related field
  • Hands on experience with big data technologies and platforms, such as Databricks, Apache Spark (PySpark, SparkSQL), workflow orchestration, performance tuning on big data processing
  • Hands on experience with various Python/R packages for data analysis, feature engineering and machine learning model training
  • Proficiency in data analysis tools (eg. SQL) and experience with data visualization tools
  • Excellent problem-solving skills and the ability to work with large, complex datasets
  • Strong understanding of data governance frameworks, tools, and best practices.
  • Knowledge of data protection regulations and compliance requirements (e.g., GDPR, CCPA)
Job Responsibility
Job Responsibility
  • Design, develop, and maintain data solutions for data generation, collection, and processing
  • Be a key team member that assists in design and development of the data pipeline
  • Create data pipelines and ensure data quality by implementing ETL processes to migrate and deploy data across systems
  • Contribute to the design, development, and implementation of data pipelines, ETL/ELT processes, and data integration solutions
  • Take ownership of data pipeline projects from inception to deployment, manage scope, timelines, and risks
  • Collaborate with cross-functional teams to understand data requirements and design solutions that meet business needs
  • Develop and maintain data models, data dictionaries, and other documentation to ensure data accuracy and consistency
  • Implement data security and privacy measures to protect sensitive data
  • Leverage cloud platforms (AWS preferred) to build scalable and efficient data solutions
  • Collaborate with Data Architects, Business SMEs, and Data Scientists to design and develop end-to-end data pipelines to meet fast paced business needs across geographic regions
Read More
Arrow Right
New

Sr Data Engineer

Designing, building, maintaining, analyzing, and interpreting data to provide ac...
Location
Location
India , Hyderabad
Salary
Salary:
Not provided
amgen.com Logo
Amgen
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s / Master’s degree and 8 to 13 years of Computer Science, IT or related field experience
  • Hands-on experience with big data technologies and platforms, such as Databricks, Apache Spark (PySpark, SparkSQL), workflow orchestration, performance tuning on big data processing
  • Proficiency in data analysis tools (e.g. SQL) and experience with data visualization tools
  • Excellent problem-solving skills and the ability to work with large, complex datasets
  • Strong understanding of data governance frameworks, tools, and best practices
  • Excellent critical-thinking and problem-solving skills
  • Strong communication and collaboration skills
  • Demonstrated awareness of how to function in a team setting
  • Demonstrated presentation skills
Job Responsibility
Job Responsibility
  • Design, develop, and maintain data solutions for data generation, collection, and processing
  • Be a key team member that assists in design and development of the data pipeline
  • Create data pipelines and ensure data quality by implementing ETL processes to migrate and deploy data across systems
  • Contribute to the design, development, and implementation of data pipelines, ETL/ELT processes, and data integration solutions
  • Take ownership of data pipeline projects from inception to deployment, manage scope, timelines, and risks
  • Collaborate with cross-functional teams to understand data requirements and design solutions that meet business needs
  • Develop and maintain data models, data dictionaries, and other documentation to ensure data accuracy and consistency
  • Implement data security and privacy measures to protect sensitive data
  • Leverage cloud platforms (AWS preferred) to build scalable and efficient data solutions
  • Collaborate and communicate effectively with product teams
  • Fulltime
Read More
Arrow Right

Sr Data Engineer

We’re seeking a Data Engineer with strong Databricks and Azure experience.
Location
Location
United States , Jacksonville
Salary
Salary:
Not provided
https://www.roberthalf.com Logo
Robert Half
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Databricks + Spark
  • Azure Data Factory or similar
  • SQL and Python
Job Responsibility
Job Responsibility
  • Build and maintain data pipelines
  • Work with large-scale data processing frameworks
  • Optimize data flows and storage solutions
What we offer
What we offer
  • Medical, vision, dental, and life and disability insurance
  • Company 401(k) plan
  • Free online training
Read More
Arrow Right

Sr Engineer, Data

This role is essential for designing and developing data architectures across on...
Location
Location
United States , Bothell
Salary
Salary:
105100.00 - 189600.00 USD / Year
https://www.t-mobile.com Logo
T-Mobile
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree plus 5 years of related work experience OR Advanced degree with 3 years of related experience (Required)
  • Acceptable areas of study include Computer Engineering, Computer Science, a related subject area (Required)
  • 4-7 years Developing cloud solutions using data series
  • experience with cloud platforms (Amazon Web Services, Azure, or Google Cloud) (Required)
  • 4-7 years Hands-on development using and migrating data to cloud platforms (Required)
  • 4-7 years Experience in SQL, NoSQL, and/or relational database design and development (Required)
  • 4-7 years Advanced knowledge and experience in building complex data pipelines with Python, Experience in languages such as SQL, DAX Python, Java, Scala, and/or Go (Required)
  • Cloud Computing (Required)
  • Collaboration (Required)
  • Data Analysis (Required)
Job Responsibility
Job Responsibility
  • Develop data engineering solutions that enable data pipelines, visualization, and analytical tools to support business requirements
  • Design and develop data architectures across on-premise, cloud, and hybrid platforms to ensure scalable data infrastructure
  • Architect, build, and maintain reliable data pipelines that ingest, transform, and deliver data from diverse customer and business data sources into enterprise data lake, warehouse, and analytics platforms
  • Design reusable data models and standardized schemas, including Customer 360 and customer behavioral data structures, to support consistent metric definitions and trusted downstream use by analytics, marketing, and AI/ML teams
  • Perform data wrangling, exploration, and discovery of heterogeneous data to generate new business insights
  • Embed data quality practices into pipeline development, including automated checks for completeness, accuracy, timeliness, and governance alignment
  • Build and curate self-service data products, metadata, and documentation that improve data discoverability and enable approved users to access trusted datasets efficiently
  • Partner with analytics, marketing, AI/ML, and business stakeholders to understand data needs, define data contracts, and support scalable customer data use cases
  • Contribute to team knowledge sharing and drive the advancement of new data engineering capabilities
  • Mentor team members to build and enhance their data engineering skillsets and professional growth
What we offer
What we offer
  • Medical, dental and vision insurance
  • flexible spending account
  • 401(k)
  • employee stock grants
  • employee stock purchase plan
  • paid time off
  • up to 12 paid holidays
  • paid parental and family leave
  • family building benefits
  • back-up care
  • Fulltime
Read More
Arrow Right

Sr. Data Engineer

We are looking for an accomplished Sr. Data Engineer to join a team building res...
Location
Location
United States , Dallas
Salary
Salary:
Not provided
https://www.roberthalf.com Logo
Robert Half
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree in Computer Science, Engineering, or a related technical discipline
  • At least 7 years of experience in data engineering with a strong record of designing and supporting production data pipelines
  • Advanced hands-on knowledge of Kafka, Airflow, NiFi, Databricks, Spark, Hadoop, Flink, and Amazon S3
  • Proficiency in Python plus working ability in Scala or Java for automation, transformation, and data processing tasks
  • Solid SQL skills and experience working with both relational and NoSQL database technologies
  • Background supporting data engineering solutions in on-premises and Kubernetes-based environments
  • Understanding of data modeling, governance practices, and data quality controls
  • Strong analytical, troubleshooting, and communication skills
  • experience in hybrid or multi-cloud settings is preferred
Job Responsibility
Job Responsibility
  • Build and enhance robust data pipelines that support ingestion, transformation, and delivery using platforms such as Airflow, NiFi, Databricks, and Spark
  • Create streaming and event-driven data solutions with Kafka and Flink to enable timely processing of high-volume data flows
  • Architect and refine storage patterns across Hadoop and Amazon S3 with attention to scalability, performance, and cost control
  • Establish monitoring, validation, and governance practices that strengthen data quality, security, and operational reliability
  • Coordinate complex workflow orchestration across hybrid and multi-cloud environments, including enterprise data processing operations where needed
  • Work with structured and semi-structured file types such as Parquet and Avro to improve usability and interoperability across systems
  • Diagnose bottlenecks in distributed processing environments and implement improvements that increase efficiency and stability
  • Provide technical guidance to less experienced engineers and contribute to a strong engineering culture centered on quality and continuous improvement
What we offer
What we offer
  • medical, vision, dental, and life and disability insurance
  • 401(k) plan
  • Fulltime
Read More
Arrow Right

Sr. Data Engineer III

Piper Companies is seeking a Sr. Data Engineer III to join an enterprise technol...
Location
Location
United States
Salary
Salary:
130000.00 - 150000.00 USD / Year
pipercompanies.com Logo
Piper Companies
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of data engineering or related experience
  • Strong expertise in SQL, Python, dbt, and or C#
  • Experience with Azure Data Factory or similar orchestration tools
  • Hands-on experience with Microsoft Azure and Fabric
  • Experience with Databricks or Snowflake is acceptable
  • Strong understanding of data modeling and ETL concepts
  • Experience with Git and version control best practices
  • Ability to work independently and solve complex problems
  • Background working in small or mid-sized environments with leadership experience
  • Must be authorized to work in the United States
Job Responsibility
Job Responsibility
  • Design and maintain scalable data pipelines and ETL workflows
  • Develop and optimize advanced SQL queries for performance and reliability
  • Leverage Azure, Fabric, Spark, and Python to automate complex data processes
  • Build and support REST and SOAP APIs for system integrations
  • Monitor and troubleshoot data pipelines to ensure high availability
  • Identify and resolve bottlenecks in data workflows
  • Lead engineering efforts supporting AI and machine learning initiatives
  • Collaborate with stakeholders and cross-functional teams
  • Conduct code reviews and contribute to technical documentation
  • Provide Tier 3 support for reporting and integrated systems
What we offer
What we offer
  • medical
  • dental
  • vision
  • 401k
  • PTO
  • sick leave as required by law
  • Fulltime
Read More
Arrow Right

Sr. Data Engineer - Python Developer

Seeking a hands-on Senior Data Engineer (ETL / Python Developer) to support the ...
Location
Location
United States , Springfield
Salary
Salary:
52.00 - 54.00 USD / Hour
myticas.com Logo
Myticas Consulting
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of data engineering experience with a focus on enterprise data warehousing
  • 5+ years of hands-on ETL development using Informatica PowerCenter, Azure Data Factory, or similar tools
  • 5+ years of Python development for data engineering and automation
  • 3+ years of experience with Spark-based processing frameworks (Databricks or equivalent)
  • Strong SQL expertise and experience with relational databases (such as Teradata, Snowflake, Oracle, SQL Server)
  • Experience with source control and DevOps practices (Azure DevOps, GitHub, CI/CD)
  • Bachelor's degree or higher in Computer Science, Engineering, Analytics, or a related field
  • Strong analytical, problem-solving, and troubleshooting skills
Job Responsibility
Job Responsibility
  • Design, develop, and maintain enterprise ETL pipelines using Azure Data Factory (ADF), Informatica PowerCenter, and Python-based frameworks
  • Build and optimize scalable data processing solutions using Python, Spark, and Databricks
  • Support Medicaid analytics and federal reporting initiatives (e.g., T-MSIS, PERM, MARS, Quality of Care)
  • Develop robust data validation, reconciliation, and audit-traceable data pipelines
  • Write and optimize SQL and stored procedures across relational platforms such as Snowflake, Oracle, and SQL Server
  • Participate in cloud migration and modernization initiatives within Azure-based architectures
  • Collaborate with analysts, QA, and reporting teams to ensure data quality, accuracy, and timeliness
  • Follow data engineering best practices for performance, reliability, reusability, and security
  • Support production operations, incident resolution, and root-cause analysis
  • Participate in code reviews, source control, and CI/CD processes using Azure DevOps and GitHub
  • Fulltime
Read More
Arrow Right

Sr Data Engineer

The Senior Data Engineer role involves leading data architecture and modernizati...
Location
Location
India , Chennai
Salary
Salary:
Not provided
nttdata.com Logo
NTT DATA
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong expertise in Azure and Databricks
  • Focus on data modeling and integration
  • Azure
  • Databricks
  • Data Modeling
  • Team Leadership
  • Client Interviews
  • SQL
Job Responsibility
Job Responsibility
  • Engage heavily with business users across North America and Europe, facilitating workshops and data discovery sessions
  • Drive consensus on business rules, data definitions, and data sources, especially where regional processes differ
  • Serve as the architectural thought leader enabling teams to transition from manual, inconsistent processes to standardized, modernized workflows
  • Partner closely with business analysts, data analysts, product owners, and engineering teams across multiple geographies
  • Architect a unified master stitched data model to replace downstream reliance on Varicent for data assembly
  • Lead the re‑architecture of compensation data processing—including internal and external compensation flows—into a scalable, cloud‑native Azure environment
  • Define patterns, frameworks, and integration strategies across Azure services (Data Factory, Databricks, Data Lake, SQL, etc.)
  • Evaluate and evolve the use of rules engines/ODM/Drools to externalize and modernize embedded business logic currently locked in application code
  • Guide decisions to shift logic and data ownership into enterprise‑owned systems rather than third‑party tools
  • Analyze current‑state processes (38 in NA, 9 in Europe) and identify opportunities for re‑engineering, automation, and consolidation
Read More
Arrow Right