CrawlJobs Logo

Data Engineer - Python AND Kafka AND (Hadoop OR HDFS OR Hive) AND Snowflake AND apache AND (iceberg

India, Bangalore Employment contract · Job Posted April 24, 2026
Apply Position
Job Link Share

Job Description

The Data Engineer will play a crucial role in migrating data from on-prem DataLake to AWS LakeHouse. This position requires a minimum of 3-5 years of experience in data engineering, with strong skills in Python and SQL. The candidate will engage with stakeholders to ensure data integrity and will be responsible for translating legacy data patterns for compatibility with modern tools. A Bachelor's or Master's degree in a relevant field is required.

Job Responsibility

  • Perform end-to-end datastore migration from on-prem DataLake to AWS hosted LakeHouse
  • Pipeline Migration - Refactoring and migrating extraction logic and job scheduling from legacy frameworks to the new Lakehouse environment
  • Data Transfer - Executing the physical migration of underlying datasets while ensuring data integrity
  • Stakeholder Engagement - Acting as a technical liaison to internal clients, facilitating handoff and sign-off conversations with data owners to ensure migrated assets meet business requirements
  • Consumption Pattern Migration - Translating and optimizing legacy SQL and Spark-based consumption patterns for compatibility with Snowflake and Iceberg
  • Usage analysis to understand usage patterns and deliver required data products
  • Data Reconciliation and Quality - Work with reconciliation frameworks to build confidence that migrated data is functionally equivalent to that already used within production flows

Requirements

  • Bachelor's or Master's degree in Computer Science, Applied Mathematics, Engineering, or a related quantitative field
  • Minimum of 3-5 years of professional hands-on-keyboard coding experience in a collaborative, team-based environment
  • Ability to troubleshoot SQL and basic scripting experience
  • Professional proficiency in Python or Java
  • Deep familiarity with the full Software Development Life Cycle (SDLC) and CI/CD best practices
  • K8s deployment experience
  • Sophisticated understanding of Temporal Data Modeling, Schema Management, Performance Optimization, and Architectural Theory
  • Experience with Kafka, ANSI SQL, FTP, Apache Spark
  • Experience with JSON, Avro, Parquet
  • Experience with Hadoop (HDFS/Hive), Snowflake, Apache Iceberg, Sybase IQ

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Data Engineer - Python AND Kafka AND (Hadoop OR HDFS OR Hive) AND Snowflake AND apache AND (iceberg

8 matching positions

Data Engineer - Python AND Kafka AND (Hadoop OR HDFS OR Hive) AND Snowflake AND apache AND (iceberg

The Data Engineer role at NTT DATA requires a Bachelor’s or Master’s degree in C...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
nttdata.com Logo
NTT DATA
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s or Master’s degree in Computer Science, Applied Mathematics, Engineering, or a related quantitative field
  • 5–7 years of professional hands-on coding experience in collaborative, team-based environments
  • strong troubleshooting skills in SQL and scripting
  • proficiency in Python or Java
  • deep familiarity with SDLC, CI/CD best practices, and Kubernetes deployment
  • expertise in temporal data modeling (e.g., SCD Type 2)
  • schema management with a focus on schema evolution (Iceberg Apache)
  • performance optimization through data partitioning and clustering
  • architectural theory involving normalization/denormalization and natural vs. surrogate keys
  • experience with Python
Job Responsibility
Job Responsibility
  • Designing and implementing data solutions
  • optimizing performance
  • collaborating within a team
  • Fulltime
Read More
Arrow Right

Pyspark Big Data Senior Developer - Vice President

We are building an A-team of highly skilled and autonomous engineers, and we are...
Location
Location
Canada , Mississauga
Salary
Salary:
120800.00 - 170800.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 6+ years of extensive, hands-on experience as a Senior Big Data Developer, with a strong emphasis on PySpark and the Apache Spark ecosystem, operating as a player/coach
  • Expert proficiency in Python, with a proven track record of developing robust, scalable, and high-performance PySpark applications for large-scale data processing
  • Deep understanding and extensive hands-on experience with Apache Spark (Spark Core, Spark SQL, Spark Streaming) and its ecosystem
  • Experience with distributed computing frameworks such as Hadoop (HDFS, YARN)
  • Expert proficiency in SQL and extensive experience with data warehousing concepts and technologies (e.g., Hive, Snowflake, Redshift, Databricks SQL)
  • Proven experience with various data storage formats (e.g., Parquet, ORC, Avro) and data lake solutions (e.g., Delta Lake, Iceberg)
  • Experience with NoSQL databases (e.g., MongoDB, Cassandra, HBase) is a significant plus
  • Strong experience with Apache Kafka for building real-time data pipelines and event-driven architectures
  • Demonstrated experience with big data services on major cloud platforms (e.g., AWS EMR/Glue/Redshift, Azure Databricks/Data Factory/Synapse, GCP Dataflow/Dataproc/BigQuery) is highly desirable
  • Proven effectiveness with AI coding tools (e.g., Claude Code, Codex, Antigravity) is a mandatory requirement
Job Responsibility
Job Responsibility
  • Operate end-to-end in the design, development, and implementation of robust big data solutions, ensuring optimal performance, scalability, data quality, and security
  • Collaborate closely within small, co-located squads (4-7 person teams), fostering high communication and low coordination overhead, to translate complex business requirements into technical specifications for big data processing and analytical solutions
  • Act as a player/coach within the team, mentoring junior members and leading by example in the development of efficient and innovative big data architectures
  • Design, develop, and optimize large-scale data pipelines using PySpark for data ingestion, transformation, and aggregation, always with an eye towards efficiency and domain relevance
  • Implement and manage real-time data streaming and event-driven architectures using technologies like Apache Kafka
  • Design and implement sophisticated data warehousing solutions and dimensional models for efficient data storage and retrieval, ensuring alignment with business needs
  • Work with various distributed data storage technologies, including distributed file systems (e.g., HDFS, S3) and NoSQL databases (e.g., MongoDB, Cassandra), selecting the right tool for the right problem
  • Implement efficient data processing and storage strategies to optimize the performance and scalability of big data applications, with a strong focus on the 'why' behind the technology choices
  • Champion best practices in software development, including rigorous code reviews, implementing comprehensive testing, and supporting continuous integration and continuous deployment (CI/CD) pipelines
  • Demonstrate high autonomy and agency in driving projects forward, making informed decisions, and proactively identifying areas for improvement
  • Fulltime
Read More
Arrow Right
New

IT Training Lead

The IT Training Lead will drive technology learning and user adoption across the...
Location
Location
United States , Delray Beach
Salary
Salary:
Not provided
https://www.roberthalf.com Logo
Robert Half
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience in IT training, instructional design, technical enablement, or learning and development
  • Strong knowledge of Microsoft 365
  • Excellent communication, facilitation, and content development skills
  • Ability to translate technical concepts into practical, user-friendly training.
Job Responsibility
Job Responsibility
  • Design, develop, and deliver IT training programs in instructor-led, virtual, and self-paced formats
  • Take lead in the Microsoft Copilot and AI training strategy, including onboarding, advanced use cases, responsible AI usage, and ongoing enablement
  • Partner with IT leadership to support new technology rollouts, system upgrades, and digital transformation initiatives
  • Create and maintain training content, including videos, guides, tutorials, and job aids
  • Identify skill gaps and develop targeted learning solutions to improve adoption and productivity
  • Gather feedback and measure training effectiveness to continuously improve programs.
Read More
Arrow Right
New

K Kitchen Representative

The position includes, but is not limited to, the following essential job duties...
Location
Location
United States , New Albany
Salary
Salary:
Not provided
https://www.circlek.com Logo
Circle K
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Excellent communication skills
  • Team player who can work well with others or independently
  • Acts with integrity
  • keeps commitments
  • Contagious positive attitude
  • Focuses on achieving results while having fun
  • Frequently bend, twist at waist, kneel, squat, stand, and walk
  • Occasionally climb and descend ladders
  • Tolerate extreme cold and hot temperatures and work in and around fryers, ovens, grills, coolers, freezers, sharp objects, and loud noises
  • Reach, grasp, and manipulate objects with hands for entire shift, including reaching for objects overhead
Job Responsibility
Job Responsibility
  • Provides excellent guest service in a fast and friendly manner
  • Maintains a clean restaurant environment by cleaning and performing general housekeeping duties
  • Prepares and serves food items in accordance with all Brand, Company, and health department regulations
  • Ensures product quality, food safety, and operational standards are met
  • Keeps accurate cash, sales, and inventory control records
  • Follows all government laws and safety codes
  • Completes reports on all incidents following our 5-minute rule policy
  • Lives our Company values: One Team, Do the Right Thing, Takes Ownership, Play to Win
What we offer
What we offer
  • Medical, Dental, Vision, Term Life and AD&D plans
  • Flexible spending and health savings accounts (FT)
  • Vacation paid time off
  • Company holidays paid at time and a half
  • Matching 401(k)
  • Tuition Reimbursement
  • Stock Purchase Plan
  • Employee Discount Program
  • Discount Meal Benefit
  • Wellness Plan
Read More
Arrow Right
New

K Kitchen Representative

Location
Location
United States , Decatur
Salary
Salary:
Not provided
https://www.circlek.com Logo
Circle K
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Excellent communication skills
  • Team player who can work well with others or independently
  • Acts with integrity
  • keeps commitments
  • Contagious positive attitude
  • Focuses on achieving results while having fun
  • Frequently bend, twist at waist, kneel, squat, stand, and walk
  • Occasionally climb and descend ladders
  • Tolerate extreme cold and hot temperatures and work in and around fryers, ovens, grills, coolers, freezers, sharp objects, and loud noises
  • Reach, grasp, and manipulate objects with hands for entire shift, including reaching for objects overhead
Job Responsibility
Job Responsibility
  • Provides excellent guest service in a fast and friendly manner
  • Maintains a clean restaurant environment by cleaning and performing general housekeeping duties
  • Prepares and serves food items in accordance with all Brand, Company, and health department regulations
  • Ensures product quality, food safety, and operational standards are met
  • Keeps accurate cash, sales, and inventory control records
  • Follows all government laws and safety codes
  • Completes reports on all incidents following our 5-minute rule policy
  • Lives our Company values: One Team, Do the Right Thing, Takes Ownership, Play to Win
What we offer
What we offer
  • Medical, Dental, Vision, Term Life and AD&D plans
  • Flexible spending and health savings accounts (FT)
  • Vacation paid time off
  • Company holidays paid at time and a half
  • Matching 401(k)
  • Tuition Reimbursement
  • Stock Purchase Plan
  • Employee Discount Program
  • Discount Meal Benefit
  • Wellness Plan
Read More
Arrow Right
New

Restaurant Assistant Manager

This position assists the Restaurant Manager (RM) with daily operations of the r...
Location
Location
United States , Holly Springs
Salary
Salary:
Not provided
https://www.circlek.com Logo
Circle K
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Full time required
  • availability during all hours of operation and at least one hour pre-opening and post-closing required
  • Valid state Driver's License required
  • Excellent communication skills
  • Motivates, coaches, and leads team members
  • Acts with integrity
  • keeps commitments
  • Contagious positive attitude
  • Focuses on achieving results while having fun
  • Ability to gain control during stressful situations
Job Responsibility
Job Responsibility
  • Assists the Restaurant Manager with daily operations of the restaurant and supervises the team in their absence
  • Leads and coaches Restaurant Team Members and partners with the management team to maintain the Company and Brand operational standards
  • Provides excellent guest service in a fast and friendly manner
  • coaches and corrects team
  • Conducts second interviews for team members and shift leads
  • Maintains a clean restaurant environment by cleaning and performing general housekeeping duties
  • Assigns shift duties to team members and follows up to ensure completion
  • Directs team and ensures all food items are prepared and served in accordance with all Brand, Company, and health department regulations
  • Coaches team members to follow guidelines for food preparation and production management
  • Cascades relevant information to team members and assists with new product training
What we offer
What we offer
  • Unlimited tip pooling
  • Medical, Dental, Vision, Term Life and AD&D plans
  • Flexible spending and health savings accounts
  • Short-Term Disability
  • Vacation paid time off
  • Company holidays paid at time and a half
  • Matching 401(k)
  • Tuition Reimbursement
  • Stock Purchase Plan
  • Employee Discount Program
  • Fulltime
Read More
Arrow Right
New

Plant Operator - Crushing and Screen

Are you an experienced and ticketed Machine Operator looking for stable, high-ho...
Location
Location
Australia , Petrie
Salary
Salary:
42.00 - 52.00 AUD / Hour
https://www.randstad.com Logo
Randstad
Expiration Date
July 09, 2026
Flip Icon
Requirements
Requirements
  • Proven Experience working in a quarry, concrete recycling, or heavy industrial yard
  • Current tickets for Front-End Loader (LL) and Excavator (LE)
  • Truck License: Heavy Rigid (HR) or higher is highly regarded
  • Reliability with strong work ethic and punctuality
  • Own reliable vehicle and current driver's license
Job Responsibility
Job Responsibility
  • Safe and efficient operation of heavy machinery in a fast-paced recycling and quarry environment
  • Operating Front-End Loaders
  • Operating Excavators utilized as material handlers
  • Operating Moxy (Articulated Dump Trucks) and other yard machinery as required
  • Assisting with daily machinery pre-starts, basic maintenance, and ensuring the yard runs smoothly
  • Adhering strictly to site health and safety protocols
What we offer
What we offer
  • Top Rates: $42.00 to $52.00 per hour + overtime penalties
  • Big Hours: Consistent 40 to 55-hour work weeks
  • Career Progression: Pathway from casual to permanent full-time employment within 3-6 months
  • Local Work: Convenient Brisbane Northside location (Petrie)
  • Immediate Start
  • Fulltime
Read More
Arrow Right

Graduate Student Instructors

Location
Location
United States , Ann Arbor
Salary
Salary:
Not provided
umich.edu Logo
University of Michigan
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Enrolled in good standing as a graduate student at the University of Michigan
  • Available to teach the scheduled section times
  • Have previously taken at least 1 philosophy course (or equivalent course) in this general area as an undergraduate or graduate student, and have demonstrated expertise and interest in subject area
  • Experience with Canvas
  • If the language of instruction at a student's undergraduate institution was not English, they must be evaluated by the English Language Institute (ELI) for English proficiency and either pass the GSI-OET or have this test waived by the ELI before they can be eligible for a GSI appointment in LS&A.
Job Responsibility
Job Responsibility
  • Attend all lectures and exams for your preferred course
  • Run discussion sections as scheduled
  • Hold at least 2 office hours each week
  • Grade assignments, per lead faculty instruction
  • Meet weekly with the lead instructor, and respond promptly to emails
  • Additional items listed in fraction calculation form
  • Parttime
Read More
Arrow Right