CrawlJobs Logo
Citi Logo Citi · IT - Software Development

Senior Databricks & Apache Spark Developer - Vice President

India, Pune · Job Posted May 27, 2026
Apply Position
Job Link Share

Job Description

We are looking for a highly skilled Senior Databricks Engineer to contribute to the engineering, modernization, and continuous evolution of data processing platform on Databricks on AWS. While supporting the transition from the legacy Cloudera Hadoop platform to Databricks on AWS, this role will continue to play a key part in enhancing performance, simplifying pipelines, and delivering new capabilities on the Databricks platform over the long term. The ideal candidate is a strong hands‑on Spark engineer with solid design experience, capable of contributing to architectural decisions while leading complex implementation and optimization efforts.

Job Responsibility

  • Platform Engineering & Modernization: Refactor and modernize existing Spark pipelines to Databricks native architectures
  • Eliminate legacy Hadoop dependencies and adopt cloud native AWS patterns
  • Enhance and extend existing processing logic using optimized Spark (JavaSpark / PySpark) on Databricks
  • Databricks Native Development: Build and optimize solutions using Databricks features, including Delta Lake, Databricks Workflows for orchestration and Auto scaling and job clusters
  • Design & Solution Engineering: Contribute to low and mid level architecture and design
  • Translate high level architecture into detailed technical designs
  • Define data models, pipeline patterns, and reusable components
  • Ensure solutions are scalable, maintainable, and production ready
  • Performance Optimization & Simplification: Analyze, improve Spark job performance and simplify complex or over engineered pipelines into standardized, efficient patterns
  • Engineering Standards & Best Practices: Follow and contribute to Databricks and Spark engineering standards
  • Write clean, modular, and testable code
  • Contribute to shared frameworks, reusable libraries, and quality standards
  • Collaboration & Stakeholder Engagement: Work closely with senior architects, platform teams, and DevOps engineers
  • Provide technical inputs, troubleshooting support, and implementation guidance
  • Participate in design discussions and technical decision making
  • Testing & Quality Assurance: Develop unit, integration, and data validation tests
  • Support production releases and post deployment validation

Requirements

  • 10+ years in data engineering or distributed systems
  • Strong expertise in Apache Spark (JavaSpark / PySpark), Databricks on AWS, and Delta Lake
  • Experience with AWS services and large‑scale distributed data processing
  • Experience modernizing or refactoring legacy data platforms into cloud‑based architectures
  • Strong background in Spark performance tuning and large‑scale batch optimization
  • Ability to translate architecture into implementable designs
  • Understanding of data modeling and pipeline orchestration patterns
  • Strong problem‑solving mindset for complex distributed systems
  • Comfortable working in time‑bound, high‑impact environments
  • Proactive, accountable, and collaborative
  • Clear communication skills across global teams
  • Bachelor’s degree/University degree or equivalent experience
Citi - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Senior Databricks & Apache Spark Developer - Vice President

8 matching positions

Pyspark Big Data Senior Developer - Vice President

We are building an A-team of highly skilled and autonomous engineers, and we are...
Location
Location
Canada , Mississauga
Salary
Salary:
120800.00 - 170800.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 6+ years of extensive, hands-on experience as a Senior Big Data Developer, with a strong emphasis on PySpark and the Apache Spark ecosystem, operating as a player/coach
  • Expert proficiency in Python, with a proven track record of developing robust, scalable, and high-performance PySpark applications for large-scale data processing
  • Deep understanding and extensive hands-on experience with Apache Spark (Spark Core, Spark SQL, Spark Streaming) and its ecosystem
  • Experience with distributed computing frameworks such as Hadoop (HDFS, YARN)
  • Expert proficiency in SQL and extensive experience with data warehousing concepts and technologies (e.g., Hive, Snowflake, Redshift, Databricks SQL)
  • Proven experience with various data storage formats (e.g., Parquet, ORC, Avro) and data lake solutions (e.g., Delta Lake, Iceberg)
  • Experience with NoSQL databases (e.g., MongoDB, Cassandra, HBase) is a significant plus
  • Strong experience with Apache Kafka for building real-time data pipelines and event-driven architectures
  • Demonstrated experience with big data services on major cloud platforms (e.g., AWS EMR/Glue/Redshift, Azure Databricks/Data Factory/Synapse, GCP Dataflow/Dataproc/BigQuery) is highly desirable
  • Proven effectiveness with AI coding tools (e.g., Claude Code, Codex, Antigravity) is a mandatory requirement
Job Responsibility
Job Responsibility
  • Operate end-to-end in the design, development, and implementation of robust big data solutions, ensuring optimal performance, scalability, data quality, and security
  • Collaborate closely within small, co-located squads (4-7 person teams), fostering high communication and low coordination overhead, to translate complex business requirements into technical specifications for big data processing and analytical solutions
  • Act as a player/coach within the team, mentoring junior members and leading by example in the development of efficient and innovative big data architectures
  • Design, develop, and optimize large-scale data pipelines using PySpark for data ingestion, transformation, and aggregation, always with an eye towards efficiency and domain relevance
  • Implement and manage real-time data streaming and event-driven architectures using technologies like Apache Kafka
  • Design and implement sophisticated data warehousing solutions and dimensional models for efficient data storage and retrieval, ensuring alignment with business needs
  • Work with various distributed data storage technologies, including distributed file systems (e.g., HDFS, S3) and NoSQL databases (e.g., MongoDB, Cassandra), selecting the right tool for the right problem
  • Implement efficient data processing and storage strategies to optimize the performance and scalability of big data applications, with a strong focus on the 'why' behind the technology choices
  • Champion best practices in software development, including rigorous code reviews, implementing comprehensive testing, and supporting continuous integration and continuous deployment (CI/CD) pipelines
  • Demonstrate high autonomy and agency in driving projects forward, making informed decisions, and proactively identifying areas for improvement
  • Fulltime
Read More
Arrow Right

Databricks AWS Engineer - Vice President

We are looking for a highly skilled Senior Databricks Engineer to contribute to ...
Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 10+ years in data engineering or distributed systems
  • Strong expertise in Apache Spark (JavaSpark / PySpark), Databricks on AWS, and Delta Lake
  • Experience with AWS services and large‑scale distributed data processing
  • Experience modernizing or refactoring legacy data platforms into cloud‑based architectures
  • Strong background in Spark performance tuning and large‑scale batch optimization
  • Ability to translate architecture into implementable designs
  • Understanding of data modeling and pipeline orchestration patterns
  • Strong problem‑solving mindset for complex distributed systems
  • Comfortable working in time‑bound, high‑impact environments
  • Proactive, accountable, and collaborative
Job Responsibility
Job Responsibility
  • Refactor and modernize existing Spark pipelines to Databricks native architectures
  • Eliminate legacy Hadoop dependencies and adopt cloud native AWS patterns
  • Enhance and extend existing processing logic using optimized Spark (JavaSpark / PySpark) on Databricks
  • Build and optimize solutions using Databricks features, including Delta Lake, Databricks Workflows for orchestration and Auto scaling and job clusters
  • Contribute to low and mid level architecture and design
  • Translate high level architecture into detailed technical designs
  • Define data models, pipeline patterns, and reusable components
  • Ensure solutions are scalable, maintainable, and production ready
  • Analyze, improve Spark job performance and simplify complex or over engineered pipelines into standardized, efficient patterns
  • Follow and contribute to Databricks and Spark engineering standards
  • Fulltime
Read More
Arrow Right

Senior Data Engineer (ETL/Big Data/Python) - Assistant Vice President

CITI Bank Enterprise Analytical Services organization is seeking a highly skille...
Location
Location
India , Chennai
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in Computer Science, Information Technology, or a related field
  • 8+ years of experience in data engineering, software development, or a similar role, with at least 2 years in a lead capacity
  • Proven experience with ETL pipeline design and development
  • Expert proficiency in Python and Spark
  • Strong experience with advanced scripting
  • Deep knowledge of relational databases (Oracle) and NoSQL databases (MongoDB), Snowflake
  • Solid understanding of data modeling, data warehousing, and performance tuning
  • Experience with CI/CD pipelines and tools (e.g., Jenkins, GitLab CI, GitHub Actions)
  • Experience with monitoring, logging, and alerting tools (e.g., Prometheus, Grafana, ELK Stack, Splunk, Datadog)
  • Excellent problem-solving and analytical skills, with a keen eye for detail
Job Responsibility
Job Responsibility
  • Design, develop, and implement ETL based reporting framework, focusing on data collection, transformation, and presentation
  • Utilize a robust technology stack including Databricks, Spark, Hive, Ozone, and Hadoop to process large volumes of data efficiently
  • Write and optimize complex ETL jobs using PySpark and advanced Python scripting
  • Design and manage data storage in various databases, including Oracle, MongoDB and Snowflake for flexible data models
  • Develop and maintain automated job schedules using Autosys, Apache Airflow for seamless data pipeline execution
  • Ability to develop visualization reporting dashboards
  • Able to leverage enterprise approved productivity tools like Copilot, etc., in daily analysis & development tasks
  • Act as the technical subject matter expert (SME) for the reporting framework, providing guidance and mentorship to the development team
  • Lead requirements gathering discussions with business stakeholders and product owners to understand reporting needs and translate them into technical solutions
  • Coordinate closely with the QA team to ensure thorough testing and data validation, maintaining high standards of data accuracy and integrity
  • Fulltime
Read More
Arrow Right

Senior Data Software Engineer (Python & PySpark) - Vice President

The Senior Data Software Engineer is a senior level position responsible for est...
Location
Location
Singapore , Singapore
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field
  • 7+ years of experience in data engineering, with a strong focus on Python and big data technologies
  • Proven expertise in designing and implementing large-scale data processing solutions using PySpark
  • Extensive experience with distributed computing frameworks like Apache Spark
  • Strong understanding of data warehousing concepts, dimensional modeling, and ETL/ELT principles
  • Proficiency in SQL and experience with various relational and NoSQL databases
  • Experience with cloud platforms (AWS, Azure, GCP) and their data services (e.g., S3, ADLS, Google Cloud Storage, Redshift, Snowflake, BigQuery, Databricks)
  • Familiarity with workflow orchestration tools (e.g., Apache Airflow, Azure Data Factory, AWS Step Functions)
  • Experience with version control systems (e.g., Git)
  • Excellent problem-solving, analytical, and communication skills.
Job Responsibility
Job Responsibility
  • Partner with multiple management teams to ensure appropriate integration of functions to meet goals as well as identify and define necessary system enhancements to deploy new products and process improvements
  • Resolve variety of high impact problems/projects through in-depth evaluation of complex business processes, system processes, and industry standards
  • Provide expertise in area and advanced knowledge of applications programming and ensure application design adheres to the overall architecture blueprint
  • Utilize advanced knowledge of system flow and develop standards for coding, testing, debugging, and implementation
  • Develop comprehensive knowledge of how areas of business, such as architecture and infrastructure, integrate to accomplish business goals
  • Provide in-depth analysis with interpretive thinking to define issues and develop innovative solutions
  • Serve as advisor or coach to mid-level developers and analysts, allocating work as necessary
  • Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency.
  • Fulltime
Read More
Arrow Right

Vice President, Big Data Scala Engineer

We are seeking an experienced and highly skilled Vice President, Big Data Scala ...
Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's or Master's degree in Computer Science, Engineering, or a related field
  • 12+ years of progressive experience in software development, with at least 5+ years focusing on big data technologies
  • 3+ years of experience in a leadership or senior architectural role
  • Extensive hands-on experience with Scala for big data processing
  • Demonstrated expertise with Apache Spark (Spark Core, Spark SQL, Spark Streaming)
  • Strong experience with distributed systems and big data ecosystems (e.g., Hadoop, Kafka, Cassandra, HBase, Delta Lake, Snowflake, Databricks)
  • Proficiency with cloud platforms (AWS, Azure, GCP) and their big data services (e.g., EMR, Redshift, Glue, DataProc, BigQuery)
  • Experience with containerization technologies (Docker, Kubernetes) and CI/CD pipelines
  • Solid understanding of data warehousing concepts, ETL/ELT processes, and data modeling
  • Familiarity with functional programming paradigms in Scala
Job Responsibility
Job Responsibility
  • Lead the architecture, design, and development of high-performance, scalable, and reliable big data processing systems using Scala and Apache Spark
  • Drive technical vision and strategy for big data initiatives
  • Evaluate and recommend new technologies and tools
  • Design, develop, and optimize data pipelines for ingestion, transformation, and storage of massive datasets
  • Implement robust and efficient data processing jobs using Scala and Spark (batch and streaming)
  • Ensure data quality, integrity, and security
  • Promote and enforce best practices in coding, testing, and deployment
  • Mentor and guide a team of talented big data engineers
  • Conduct code reviews, provide constructive feedback
  • Participate in the recruitment and hiring
What we offer
What we offer
  • Opportunity to work on cutting-edge big data technologies and impactful projects
  • A collaborative and innovative work environment
  • Competitive compensation and benefits package
  • Opportunities for professional growth and career advancement
  • Fulltime
Read More
Arrow Right

Security Officer

Anabas is a dynamic national Facilities Management company, dedicated to deliver...
Location
Location
United Kingdom , Swindon
Salary
Salary:
13.45 GBP / Hour
anabas.co.uk Logo
Anabas
Expiration Date
July 30, 2026
Flip Icon
Requirements
Requirements
  • Previous experience in a corporate security role (ideally in a front-facing environment)
  • Strong communication and interpersonal skills
  • IT literacy is an essential element as will need to work on door access software and CCTV
  • The ability to remain calm under pressure and act decisively during incidents
  • A team-oriented approach and commitment to providing excellent service
  • Valid SIA licence (required)
  • BS7858 vetting level will be required
Job Responsibility
Job Responsibility
  • Patrol and safeguard the entire premises, ensuring the safety of staff, visitors, contractors, and company assets
  • Respond swiftly and effectively to any incidents, emergencies, or security breaches
  • Assist in the coordination and safe evacuation of personnel during emergencies
  • Promote and maintain a secure and welcoming environment for all building users
  • Support Health & Safety objectives and assist with first aid or in aiding trained First Aider's
  • Collaborate closely with colleagues and site teams to uphold the highest security standards
  • Working in a control room environment is an essential part of the role
What we offer
What we offer
  • 28 days holiday (including Bank Holidays)
  • Life Insurance (1x annual salary)
  • Employee Assistance Programme (EAP)
  • Recognition & Reward scheme
  • Cycle to Work scheme
  • Recommend a Friend scheme
  • Regular company events
  • Training and development opportunities for career growth
  • Fulltime
Read More
Arrow Right

Warehouse Operative

This is a fast packed role where you could be involved in a wide variety of task...
Location
Location
United Kingdom , Birmingham
Salary
Salary:
Not provided
jobs.360resourcing.co.uk Logo
360 Resourcing Solutions
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Great timekeeping skills
  • An enthusiastic and adaptable approach to carrying out duties
  • Be able to work in a fast paced environment
  • Teamwork to ensure targets and deadlines are met
  • Flexible approach to working hours
Job Responsibility
Job Responsibility
  • Replenishing stock
  • Processing bulk sales through the checkout
  • Picking and packing customer orders
  • Assisting drivers with deliveries
  • Ensuring our customers receive the best possible service at all times
What we offer
What we offer
  • Competitive hourly rate
  • Pension
  • 22 days annual leave
  • Onsite parking
  • Fulltime
Read More
Arrow Right

Senior Product Designer

We’re hiring a Senior Product Designer to create intuitive, high-impact user exp...
Location
Location
United States , Austin
Salary
Salary:
Not provided
https://www.roberthalf.com Logo
Robert Half
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 4+ years of product design experience, ideally in SaaS or data-driven products
  • Strong portfolio showcasing end-to-end UX/UI design for complex applications
  • Expertise in Figma, prototyping, and component-based design systems
  • Experience designing data-heavy interfaces (dashboards, analytics, or workflows)
  • Strong collaboration and communication skills across technical and non-technical teams
  • Comfort working in agile environments and managing multiple priorities
  • Ability to operate independently, drive decisions, and iterate quickly
  • Understanding of accessibility, responsive design, and modern UX best practices
Job Responsibility
Job Responsibility
  • Design end-to-end product experiences, from discovery through high-fidelity execution
  • Translate complex data and technical workflows into intuitive dashboards and interfaces
  • Collaborate cross-functionally in agile environments (PM, Engineering, QA)
  • Build and evolve design systems and reusable UI components
  • Prototype and iterate quickly using Figma and AI-assisted tools
  • Partner with engineering to ensure high-quality, accurate implementation
  • Gather user feedback and continuously refine designs based on insights and metrics
  • Balance speed and quality while making decisions in ambiguous, fast-moving environments
What we offer
What we offer
  • medical
  • vision
  • dental
  • life and disability insurance
  • 401(k) plan
  • Fulltime
Read More
Arrow Right