CrawlJobs Logo

Data Engineer - Python AND Kafka AND (Hadoop OR HDFS OR Hive) AND Snowflake AND apache AND (iceberg

India, Bangalore Employment contract · Job Posted April 24, 2026
Apply Position
Job Link Share

Job Description

The Data Engineer will play a crucial role in migrating data from on-prem DataLake to AWS LakeHouse. This position requires a minimum of 3-5 years of experience in data engineering, with strong skills in Python and SQL. The candidate will engage with stakeholders to ensure data integrity and will be responsible for translating legacy data patterns for compatibility with modern tools. A Bachelor's or Master's degree in a relevant field is required.

Job Responsibility

  • Perform end-to-end datastore migration from on-prem DataLake to AWS hosted LakeHouse
  • Pipeline Migration - Refactoring and migrating extraction logic and job scheduling from legacy frameworks to the new Lakehouse environment
  • Data Transfer - Executing the physical migration of underlying datasets while ensuring data integrity
  • Stakeholder Engagement - Acting as a technical liaison to internal clients, facilitating handoff and sign-off conversations with data owners to ensure migrated assets meet business requirements
  • Consumption Pattern Migration - Translating and optimizing legacy SQL and Spark-based consumption patterns for compatibility with Snowflake and Iceberg
  • Usage analysis to understand usage patterns and deliver required data products
  • Data Reconciliation and Quality - Work with reconciliation frameworks to build confidence that migrated data is functionally equivalent to that already used within production flows

Requirements

  • Bachelor's or Master's degree in Computer Science, Applied Mathematics, Engineering, or a related quantitative field
  • Minimum of 3-5 years of professional hands-on-keyboard coding experience in a collaborative, team-based environment
  • Ability to troubleshoot SQL and basic scripting experience
  • Professional proficiency in Python or Java
  • Deep familiarity with the full Software Development Life Cycle (SDLC) and CI/CD best practices
  • K8s deployment experience
  • Sophisticated understanding of Temporal Data Modeling, Schema Management, Performance Optimization, and Architectural Theory
  • Experience with Kafka, ANSI SQL, FTP, Apache Spark
  • Experience with JSON, Avro, Parquet
  • Experience with Hadoop (HDFS/Hive), Snowflake, Apache Iceberg, Sybase IQ

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Data Engineer - Python AND Kafka AND (Hadoop OR HDFS OR Hive) AND Snowflake AND apache AND (iceberg

8 matching positions

Data Engineer - Python AND Kafka AND (Hadoop OR HDFS OR Hive) AND Snowflake AND apache AND (iceberg

The Data Engineer role at NTT DATA requires a Bachelor’s or Master’s degree in C...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
nttdata.com Logo
NTT DATA
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s or Master’s degree in Computer Science, Applied Mathematics, Engineering, or a related quantitative field
  • 5–7 years of professional hands-on coding experience in collaborative, team-based environments
  • strong troubleshooting skills in SQL and scripting
  • proficiency in Python or Java
  • deep familiarity with SDLC, CI/CD best practices, and Kubernetes deployment
  • expertise in temporal data modeling (e.g., SCD Type 2)
  • schema management with a focus on schema evolution (Iceberg Apache)
  • performance optimization through data partitioning and clustering
  • architectural theory involving normalization/denormalization and natural vs. surrogate keys
  • experience with Python
Job Responsibility
Job Responsibility
  • Designing and implementing data solutions
  • optimizing performance
  • collaborating within a team
  • Fulltime
Read More
Arrow Right

Pyspark Big Data Senior Developer - Vice President

We are building an A-team of highly skilled and autonomous engineers, and we are...
Location
Location
Canada , Mississauga
Salary
Salary:
120800.00 - 170800.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 6+ years of extensive, hands-on experience as a Senior Big Data Developer, with a strong emphasis on PySpark and the Apache Spark ecosystem, operating as a player/coach
  • Expert proficiency in Python, with a proven track record of developing robust, scalable, and high-performance PySpark applications for large-scale data processing
  • Deep understanding and extensive hands-on experience with Apache Spark (Spark Core, Spark SQL, Spark Streaming) and its ecosystem
  • Experience with distributed computing frameworks such as Hadoop (HDFS, YARN)
  • Expert proficiency in SQL and extensive experience with data warehousing concepts and technologies (e.g., Hive, Snowflake, Redshift, Databricks SQL)
  • Proven experience with various data storage formats (e.g., Parquet, ORC, Avro) and data lake solutions (e.g., Delta Lake, Iceberg)
  • Experience with NoSQL databases (e.g., MongoDB, Cassandra, HBase) is a significant plus
  • Strong experience with Apache Kafka for building real-time data pipelines and event-driven architectures
  • Demonstrated experience with big data services on major cloud platforms (e.g., AWS EMR/Glue/Redshift, Azure Databricks/Data Factory/Synapse, GCP Dataflow/Dataproc/BigQuery) is highly desirable
  • Proven effectiveness with AI coding tools (e.g., Claude Code, Codex, Antigravity) is a mandatory requirement
Job Responsibility
Job Responsibility
  • Operate end-to-end in the design, development, and implementation of robust big data solutions, ensuring optimal performance, scalability, data quality, and security
  • Collaborate closely within small, co-located squads (4-7 person teams), fostering high communication and low coordination overhead, to translate complex business requirements into technical specifications for big data processing and analytical solutions
  • Act as a player/coach within the team, mentoring junior members and leading by example in the development of efficient and innovative big data architectures
  • Design, develop, and optimize large-scale data pipelines using PySpark for data ingestion, transformation, and aggregation, always with an eye towards efficiency and domain relevance
  • Implement and manage real-time data streaming and event-driven architectures using technologies like Apache Kafka
  • Design and implement sophisticated data warehousing solutions and dimensional models for efficient data storage and retrieval, ensuring alignment with business needs
  • Work with various distributed data storage technologies, including distributed file systems (e.g., HDFS, S3) and NoSQL databases (e.g., MongoDB, Cassandra), selecting the right tool for the right problem
  • Implement efficient data processing and storage strategies to optimize the performance and scalability of big data applications, with a strong focus on the 'why' behind the technology choices
  • Champion best practices in software development, including rigorous code reviews, implementing comprehensive testing, and supporting continuous integration and continuous deployment (CI/CD) pipelines
  • Demonstrate high autonomy and agency in driving projects forward, making informed decisions, and proactively identifying areas for improvement
  • Fulltime
Read More
Arrow Right
New

VP Java Full Stack Tech Lead

The Java Full Stack Applications Development Technology Lead is a senior level p...
Location
Location
United States , Tampa
Salary
Salary:
113840.00 - 170760.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
July 09, 2026
Flip Icon
Requirements
Requirements
  • 5-8 years of hands-on experience in Java, including proficiency with modern frameworks such as Spring Boot, Quarkus, Micronaut, or Vert.x
  • Significant experience with JavaScript, Angular, Ext's, and data visualization tools like Tableau
  • Strong command of Oracle SQL for database management and querying
  • Demonstrable experience in cloud-native development and orchestrating containerized applications using Docker, Kubernetes, OpenShift, and Serverless technologies
  • Proven background in designing and implementing Service-Oriented and Microservices architectures, including REST and GraphQL APIs
  • Exposure to and practical experience with Continuous Integration and Continuous Delivery (CI/CD) pipelines (e.g., Harness, CircleCI, Cloudbees Jenkins)
  • Experience working within agile and iterative software delivery environments
  • Bachelor's degree/University degree or equivalent experience
  • Master’s degree preferred
Job Responsibility
Job Responsibility
  • Partner with multiple management teams to ensure appropriate integration of functions to meet goals as well as identify and define necessary system enhancements to deploy new products and process improvements
  • Resolve variety of high impact problems/projects through in-depth evaluation of complex business processes, system processes, and industry standards
  • Provide expertise in area and advanced knowledge of applications programming and ensure application design adheres to the overall architecture blueprint
  • Utilize advanced knowledge of system flow and develop standards for coding, testing, debugging, and implementation
  • Develop comprehensive knowledge of how areas of business, such as architecture and infrastructure, integrate to accomplish business goals
  • Provide in-depth analysis with interpretive thinking to define issues and develop innovative solutions
  • Serve as advisor or coach to mid-level developers and analysts, allocating work as necessary
  • Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency
What we offer
What we offer
  • medical, dental & vision coverage
  • 401(k)
  • life, accident, and disability insurance
  • wellness programs
  • paid time off packages, including planned time off (vacation), unplanned time off (sick leave), and paid holidays
  • Fulltime
!
Read More
Arrow Right
New

Social Care Leader

TTM Healthcare is partnering with a residential mainstream children’s service to...
Location
Location
Ireland , Tipperary Town
Salary
Salary:
42000.00 - 46000.00 EUR / Year
ttmhealthcare.ie Logo
TTM Healthcare
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Level 7 or 8 in Social Care or Similar field
  • CORU registration is desirable but not essential
  • A minimum of 2 years experience in Social Care
  • Car and full manual license is desirable but not essential for every role
  • Full Flexibility to work a residential roster
Job Responsibility
Job Responsibility
  • Be the team leader of the shift in which they are working
  • Work dynamically and collaboratively with the Service Manager/Deputy Manager to support the delivery of tailored Care Plans
  • Lead shifts and champion best practice to ensure a consistent high standard of care aligned with the service’s mission and values
What we offer
What we offer
  • Health Insurance
  • Pension Scheme
  • Career Progression Opportunities
  • Supportive & Reflective Team Environment
  • Additional Day off for your birthday on top of your annual leave
  • Fulltime
Read More
Arrow Right
New

Heavy Equipment Operator Lead

Provide direct support to the Fleet Operations Foremen, Supervisor or Traverse O...
Location
Location
Antarctica , McMurdo Station
Salary
Salary:
Not provided
amentum.com Logo
Amentum
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • High School diploma or GED
  • Four years of experience as a heavy equipment operator
  • SPoT positions require a minimum of 1 previous season with MCM Fleet Operations or South Pole Traverse
  • Valid Driver's License issued in the United States
  • Must be able to obtain and maintain facility credentials/authorization
  • US Citizenship is required for facility credentials/authorization at this work location
Job Responsibility
Job Responsibility
  • Provide direct support to the Fleet Operations Foremen, Supervisor or Traverse Operations Manager ensuring efficient and productive operations within the department
  • Leads small teams under the guidance of a Foreman or the Supervisor to accomplish specific projects and tasks
  • Helping to coordinate daily tasking for the Operators and direct projects such as loading cargo, fuel bladder filling, and leading by example as they teach Operators Standard Operating Procedures
  • Safely and proficiently operate and/or over-see the operation of light vehicles such as pickups, heavy equipment such as bulldozers, articulating loaders, excavators, graders, backhoes, fork-lifts, tele-handlers, snow blasters, tracked agricultural tractors, piston bullys, dump-trucks, snow moving and grooming equipment
  • Operate autonomously to complete specific tasks and projects in support of Fleet Operations
  • Effectively lead small groups of people to achieve departmental goals on a daily, weekly, and sometimes longer-term basis under the guidance of a Foreman or the Supervisor
  • Coordinate with other HEO Leads and Foremen within Fleet Operations to ensure department-wide goals and daily operations are met and department wide resources are used efficiently
  • Perform daily equipment checks and ensure that all safety and maintenance issues are reported to the Vehicle Maintenance Facility (VMF) daily via proper reporting methods and channels
  • Tracks equipment hours daily and notifies Fleet Operations Foreman/Traverse Shop Foreman any time equipment is within 25 hours of its PM due hours
  • Ensure all team members are properly trained and evaluated in the safe use of any piece of equipment prior to allowing that team member to use the equipment unsupervised
  • Fulltime
Read More
Arrow Right
New

Commercial Roofing Sales Executive

A well established, multi division commercial and residential roofing company se...
Location
Location
United States , Los Angeles
Salary
Salary:
Not provided
careermovement.com Logo
Career Movement
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Established, transferable book of business in commercial roofing
  • Proven track record of self-generated commercial sales volume
  • Existing relationships with property managers, HOAs, general contractors, asset managers, or building owners across the LA, Orange County, and San Diego corridor
  • Comfortable owning a territory as a hunter and closer
Job Responsibility
Job Responsibility
  • Prospect, hunt, and close commercial roofing deals across HOA, multifamily, commercial building, and general contractor segments
  • Conduct assessments, build estimates, and sell them through to a signed contract
  • Act as the point of contact between the customer, production, and leadership through the sales cycle
  • Leverage both your own relationships and company-provided leads to build pipeline
  • Grow the commercial book within the Southern California territory
Read More
Arrow Right
New

Multi-Task Attendant

TTM Healthcare Solutions is recruiting Multi-Task Attendant for agency work in T...
Location
Location
Ireland , Kilkenny
Salary
Salary:
Not provided
ttmhealthcare.ie Logo
TTM Healthcare
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • QQI Level 5 Qualification in Healthcare Support or an equivalent qualification
  • Vaccinations including Hepatitis B, MMR, Tuberculosis & Varicella
  • Resident in Ireland and hold one of the following: EU passport or GNIB card with Stamp 1G, Stamp 2, Stamp 4, Stamp 4D, Stamp 5, or 4 EUFam
  • Willing to undergo Garda Vetting
  • International Police Clearance (if lived outside of Ireland for more than 6 months after the age of 16)
Job Responsibility
Job Responsibility
  • Providing personal care, including washing and dressing
  • Support patients with mobility and safe movement
  • Report any incidents or notable observations to supervising staff
  • Offer companionship and emotional support to patients
  • Perform domestic tasks such as cleaning, laundry, and catering assistance
  • Carry out additional duties as needed to maintain a high standard or care and service
What we offer
What we offer
  • Access HSE shifts nationwide
  • Competitive pay rates
  • Flexible schedules
  • Dedicated support
  • Weekly payroll
  • TTM employee assistance programme
  • Exclusive discounts via TTM Perks at Work
  • Refer a friend bonuses (T&Cs apply)
Read More
Arrow Right
New

Banamex IT Business Analyst

Location
Location
Mexico , Ciudad De Mexico
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5-8 years of relevant experience
  • Experience in data analysis with intermediate/advanced Microsoft Office Suite skills
  • Proven interpersonal, data analysis, diplomatic, management and prioritization skills
  • Consistently demonstrate clear and concise written and verbal communication
  • Proven ability to manage multiple activities and build/develop working relationships
  • Proven self-motivation to take initiative and master new tasks quickly
  • Demonstrated ability to work under pressure to meet tight deadlines and approach work methodically with attention to detail
  • Bachelor's degree/University degree or equivalent experience
  • Más de 5 años en Tecnología o áreas relacionadas (mandatorio)
  • Más de 3 años trabajando en la Industria Bancaria, idealmente, o bien, 5 años en la Industria de Consumo Masivo o Retail
Job Responsibility
Job Responsibility
  • Formulate and define systems scope and objectives for complex projects and foster communication between business leaders and IT
  • Consult with users and clients to solve complex system issues/problems through in-depth evaluation of business processes, systems and industry standards and recommends solutions
  • Support system change processes from requirements through implementation and provide input based on analysis of information
  • Consult with business clients to determine system functional specifications and provides user and operational support
  • Identify and communicate risks and impacts, considering business implications of the application of technology to the current business environment
  • Act as advisor or coach to new or lower level analysts and work as a team to achieve business objectives, performing other duties and functions as assigned
  • Has the ability to operate with a limited level of direct supervision
  • Can exercise independence of judgement and autonomy
  • Acts as SME to senior stakeholders and /or other team members
  • Fulltime
Read More
Arrow Right