CrawlJobs Logo

Data Engineer with Pyspark Kafka Snowflake

India, Chennai · Job Posted July 13, 2026
Apply Position
Job Link Share

Job Description

The Engineering Analyst 2 is an intermediate level position responsible for a variety of engineering activities including the design, acquisition and development of hardware, software and network infrastructure in coordination with the Technology team. The overall objective of this role is to ensure quality standards are being met within existing and planned frameworks.

Job Responsibility

  • Perform system and application monitoring, capacity planning and systems tests to ensure products meet performance requirements
  • Evaluate technologies, develop prototypes, contribute to design issues, and implement solutions
  • Work with various internal and external teams to identify and resolve problems
  • Consult with end users and clients to identify and correct systems problems or propose solutions
  • Assist in the development of software and systems tools used by integration teams to create end user packages
  • Provide support for operating systems and in-house applications, including third party applications, as needed
  • Perform coding, analysis, testing or other appropriate functions in order to identify problems and propose solutions
  • Adhere to Citi technology standards, audit requirements and corporate compliance issues and requirements
  • Apply knowledge of engineering procedures and concepts and basic knowledge of other technical areas to day to day activities
  • Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency

Requirements

  • 2-5 years of relevant experience in an Engineering role
  • Experience working in Financial Services or a large complex and/or global environment
  • Project Management experience
  • Consistently demonstrates clear and concise written and verbal communication
  • Comprehensive knowledge of design metrics, analytics tools, benchmarking activities and related reporting to identify best practices
  • Demonstrated analytic/diagnostic skills
  • Ability to work in a matrix environment and partner with virtual teams
  • Ability to work independently, multi-task, and take ownership of various parts of a project or initiative
  • Ability to work under pressure and manage to tight deadlines or unexpected changes in expectations or requirements
  • Proven track record of operational process change and improvement
  • Bachelor’s degree/University degree or equivalent experience

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Data Engineer with Pyspark Kafka Snowflake

8 matching positions

Data Engineer - Security (Kafka Experience)

The Data Engineer - Security role focuses on designing and operating large-scale...
Location
Location
India , Remote
Salary
Salary:
Not provided
nttdata.com Logo
NTT DATA
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Kafka-Strong expertise in Kafka (4-5 years), with hands-on experience designing and operating large-scale, highly available event-streaming platforms, including partitioning strategies, consumer group optimization, schema management, and performance tuning
  • API-first data ingestion. Strong hands-on pulling data from REST/GraphQL APIs with auth (OAuth2, API keys), pagination, rate limits, retries/backoff, and webhooks
  • strong Python skills to normalize/enrich data and land it cleanly into S3 (schema, partitioning, Parquet)
  • AWS data lake, end to end. Comfortable building/operating S3-based lakes with layered zones (raw → harmonized → conformed → modeled), Glue Data Catalog, IAM/Secrets Manager, VPC endpoints, encryption, lifecycle/versioning, and cost/perf best practices (file sizing, compaction)
  • AWS Glue + PySpark expert. Designs and optimizes Glue jobs using PySpark/DynamicFrames, bookmarks for incremental loads, dependency packaging, robust error handling, logging/metrics, and unit tests
  • knows how to tune jobs for scale and cost
  • Airflow orchestration. Writes clean, parameterized, idempotent DAGs (sensors, SLAs, retries, alerts), manages dependencies across pipelines, and uses Git-based CI/CD to promote changes safely
  • Snowflake proficiency. Builds ELT models (staging/ODS/marts), tunes performance (warehouse sizing, clustering, micro-partitions, caching), uses Streams/Tasks/Snowpipe for CDC, and follows solid RBAC and data governance practices
Job Responsibility
Job Responsibility
  • Designing and operating large-scale event-streaming platforms using Kafka
  • API-first data ingestion
  • Building/operating S3-based lakes
  • Designing and optimizing Glue jobs using PySpark/DynamicFrames
  • Writing clean, parameterized, idempotent DAGs
  • Building ELT models in Snowflake
  • Fulltime
Read More
Arrow Right

Data Engineer

Collinson is the global, privately-owned company dedicated to helping the world ...
Location
Location
South Africa , Cape Town
Salary
Salary:
Not provided
collinsongroup.com Logo
Collinson
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Extensive experience leading AWS and cloud data platform transformations
  • Proven track record of delivering large-scale data and analytical solutions in a cloud environment
  • Hands-on experience with end-to-end data pipeline implementation on AWS, including data preparation, extraction, transformation & loading, normalization, aggregation, warehousing, data lakes, and data governance
  • Expertise in developing Data Warehouses
  • In-depth understanding of modern data architecture such as Data Lake, Data Warehouse, Lakehouse, and Data Mesh
  • Strong knowledge of data architecture and data modelling practices
  • Cost-effective management of data pipelines
  • Familiarity with CI/CD driven data pipeline and infrastructure
  • Agile delivery approach using Scrum and Kanban methodologies
  • Ability to scope, estimate, and deliver committed work within deadlines, both independently and as part of an agile team
Job Responsibility
Job Responsibility
  • Design, develop, and maintain data pipelines for collecting, transforming, and loading data into various data stores
  • Build and maintain data warehousing and data lake solutions
  • Develop and deploy data models that support various business requirements
  • Write efficient and scalable code in languages such as Python, Scala, or Java
  • Lead the design of data solutions with quality, automation, and performance in mind
  • Own the data pipelines feeding into the Data Platform ensuring they are reliable and scalable
  • Ensure data is available in a fit-for-purpose and timely manner for business and analytics consumption
  • Work with Data Governance team to ensure solutions are compliant with regulations such as GDPR and CISO policies and data quality is baked-in to pipelines
  • Maintain and optimise existing data pipelines to improve performance and quality, minimising impacts to business
  • Collaborate with cross-functional teams to understand data requirements and provide support for data-driven initiatives
  • Fulltime
Read More
Arrow Right

Senior Data Engineer

As Senior Data Engineer you will be responsible for designing, building, maintai...
Location
Location
Portugal , Lisbon
Salary
Salary:
35571.65 - 48126.35 EUR / Year
amgen.com Logo
Amgen
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree and 3 to 5 years of Computer Science, IT or related field experience
  • Proficiency in Python, PySpark, and Scala for data processing and ETL (Extract, Transform, Load) workflows, with hands-on experience in using Databricks for building ETL pipelines and handling big data processing
  • Experience with data warehousing platforms such as Amazon Redshift, or Snowflake
  • Strong knowledge of SQL and experience with relational (e.g., PostgreSQL, MySQL) databases
  • Familiarity with big data frameworks like Apache Hadoop, Spark, and Kafka for handling large datasets
  • Experienced with software engineering best-practices, including but not limited to version control (GitLab, Subversion, etc.), CI/CD (Jenkins, GITLab etc.), automated unit testing, and Dev Ops
Job Responsibility
Job Responsibility
  • Design, develop, and maintain data solutions for data generation, collection, and processing
  • Be a key team member that assists in design and development of the data pipeline
  • Create data pipelines and ensure data quality by implementing ETL processes to migrate and deploy data across systems
  • Contribute to the design, development, and implementation of data pipelines, ETL/ELT processes, and data integration solutions
  • Take ownership of data pipeline projects from inception to deployment, manage scope, timelines, and risks
  • Collaborate with cross-functional teams to understand data requirements and design solutions that meet business needs
  • Develop and maintain data models, data dictionaries, and other documentation to ensure data accuracy and consistency
  • Implement data security and privacy measures to protect sensitive data
  • Leverage cloud platforms (AWS preferred) to build scalable and efficient data solutions
  • Collaborate and communicate effectively with product teams
What we offer
What we offer
  • Work That Matters – Build tech that accelerates scientific breakthroughs and helps patients worldwide
  • Modern Tech Stack – Cloud-first, automation-focused, AI-powered
  • Global Scale, Agile Mindset – Collaborate across continents while working in nimble, high-impact teams
  • Continuous Learning – Access to certifications, trainings, mentorship, and career mobility
  • AMGEN Total Rewards Plan – Comprehensive benefits in healthcare, finance, and well-being
  • Flexibility – Hybrid work model with time split between our Lisbon office and remote work
  • Fulltime
Read More
Arrow Right

ETL Data Engineer

We are currently seeking a ETL Data Engineer - HYBRID to join our team in Halifa...
Location
Location
Canada , Halifax
Salary
Salary:
Not provided
nttdata.com Logo
NTT DATA
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 6+ years of overall experience in data engineering, data warehousing, or cloud data platform development
  • 6+ Years of Strong hands-on experience in: SQL (advanced), Python / Spark / Scala, ETL/ELT frameworks, Data modeling and warehousing concepts
  • 6+ Years of Experience with at least one major cloud data platform: Snowflake or Databricks
  • 6+ Years of Experience building enterprise-scale data pipelines and ingestion frameworks
  • 6+ Years of Languages & Processing: SQL, Python, PySpark, Scala
  • 6+ Years of Cloud & Warehousing: AWS, Azure, Snowflake, Databricks, Redshift, Synapse
  • 6+ Years of Data Engineering Tools: Airflow, dbt, Kafka, Informatica, Talend
  • 6+ Years of DevOps & Automation: Git, CI/CD, Terraform, Docker, Kubernetes
  • 6+ Years of Data Governance: Lineage, Metadata, Data Quality, Data Reconciliation
  • Bachelors in Computer Science or equivalent work experience
Job Responsibility
Job Responsibility
  • Cloud Data Warehousing & Engineering
  • Data Pipeline Development
  • Data Modeling & Warehousing
  • Cloud Platform Engineering
  • Data Quality, Governance & Security
  • Performance Optimization
  • DevOps & Automation
  • Fulltime
Read More
Arrow Right

Senior Data Engineer

RemoteStar is looking to hire a Remote Senior Data Engineer (ETL Data Modeling) ...
Location
Location
India , Bengaluru
Salary
Salary:
Not provided
Remotestar
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Must have total 5+ yrs. of IT experience and 3+ years' experience in data Integration, ETL/ETL development, and database design or Data Warehouse design
  • Broad expertise and experience with distributed systems, streaming systems, and data engineering tools, such as Kubernetes, Kafka, Airflow, Dagster, etc.
  • Experience in data transformation, ETL/ELT tool and technologies such as AWS Glue, DBTetc for transforming structured/semi structured and unstructured datasets
  • Experience in ingesting and integrating data from APIs/JDBC/CDC sources
  • Deep knowledge of Python, SQL, relational/ non-relational database design, and master data strategies
  • Experience defining, architecting, and rolling out data products, including ownership of data products through their entire lifecycle
  • Deep understanding of Star and Snowflake dimensional modeling
  • Experience with relational databases, including SQL queries, database definition, and schema design
  • Experience with data warehouses, distributed data platforms, and data lakes
  • Strong proficiency in SQL and at least one programming language (e.g., Python,Scala, JS)
Job Responsibility
Job Responsibility
  • Implementing ETL/ELT pipelines within and outside of a data warehouse using Python, Pyspark and Snowflakes Snow SQL
  • Support Redshift DWH to Snowflake Migration
  • Design, implement, and support data warehouse/data lake infrastructure using AWS big data stack, Python, Redshift, Snowflake, Glue/lake formation, EMR/Spark/Scala etc.
  • Work with data analysts to scale value-creating capabilities, including data integrations and transformations, model features, and statistical and machine learning models
  • Work with Product Managers, Finance, Service Engineering Teams and Sales Teams on a day-to-day basis to support their new analytics requirements
  • Implement data quality and data governance measures and execute data profiling and data validation procedures
  • Implement and uphold data governance practices to maintain data quality, integrity, and security throughout the data lifecycle
  • Leverage open-source technologies to build robust and cost-effective data solutions
  • Develop and maintain streaming pipelines using technologies like Apache Kafka etc.
What we offer
What we offer
  • Dynamic working environment in an extremely fast-growing company
  • Work in an international environment
  • Work in a pleasant environment with very little hierarchy
  • Intellectually challenging, play a massive role in client’s success and scalability
  • Flexible working hours
  • Fulltime
Read More
Arrow Right

Quality Automation Engineer - Data

We are looking for a Data Quality Engineer to design and implement robust data q...
Location
Location
India
Salary
Salary:
Not provided
Codvo AI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s or Master’s in Computer Science, Data Engineering, or related field
  • 5–8 years of experience in data-related roles
  • Strong problem-solving and analytical skills
  • Clear communication and documentation abilities
  • Experience working in cross-functional teams
  • Databricks (PySpark, Delta Lake, SQL)
  • Snowflake
  • Azure Data Factory
  • Informatica
  • Teradata
Job Responsibility
Job Responsibility
  • Develop and maintain data quality checks and validation frameworks
  • Collaborate with data engineers and analysts to identify and resolve data issues
  • Monitor data pipelines for anomalies and inconsistencies
  • Implement data profiling and cleansing processes
  • Ensure adherence to data governance and compliance standards
  • Automate data quality reporting and alerting mechanisms
  • Contribute to continuous improvement of data quality practices
  • Fulltime
Read More
Arrow Right

Fullstack Data Engineer

We are looking for a highly skilled Full Stack Data Engineer with expertise in d...
Location
Location
India
Salary
Salary:
Not provided
Codvo AI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong in PySpark, Delta Lake, Databricks SQL
  • Experience with Databricks Workflows, Unity Catalog, and Delta Live Tables
  • Experience in snowflake, data engineering technologies like ETL, ELT
  • Python (mandatory), SQL (expert)
  • Exposure to Java/Scala (for Spark jobs)
  • Knowledge of APIs, microservices (FastAPI/Flask), or basic front-end (React/Angular) is a plus
  • Proficiency with at least one: Azure Databricks, AWS Databricks, or GCP Databricks
  • Knowledge of cloud storage (ADLS, S3, GCS), IAM, networking
  • Git, CI/CD tools (GitHub Actions, Azure DevOps, Jenkins)
  • Containerization (Docker, Kubernetes is a plus)
Job Responsibility
Job Responsibility
  • Design and develop ETL/ELT pipelines on platforms like Databricks (PySpark, Delta Lake, SQL), Informatica, Teradata, Snowflake
  • Architect data models (batch and streaming) for analytics, ML, and reporting
  • Optimize performance of large-scale distributed data processing jobs
  • Implement CI/CD pipelines for Databricks workflows using GitHub Actions, Azure DevOps, or similar
  • Build and maintain APIs, dashboards, or applications that consume processed data (full-stack aspect)
  • Collaborate with data scientists, analysts, and business stakeholders to deliver solutions
  • Ensure data quality, lineage, governance, and security compliance
  • Deploy solutions across cloud environments (Azure, AWS, or GCP)
  • Fulltime
Read More
Arrow Right

Data Engineer - DB2

Wissen Technology is hiring for Data Engineer - DB2 About Wissen Technology: At...
Location
Location
India , Bangalore South
Salary
Salary:
Not provided
votredircom.fr Logo
Wissen
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's/master's in computer science, IT, or related field
  • 3-5 years in data engineering, with a strong emphasis on data-intensive environments
  • Proficiency in Python, Java (core for data apps), SQL/NoSQL, PySpark/Spark, Kafka for streaming
  • Experience with big data tools, cloud data services, and ETL frameworks (Airflow, dbt)
Job Responsibility
Job Responsibility
  • Design, develop, and optimize ETL/ELT pipelines for handling massive datasets using Python, Java & Spark for batch and real-time streaming
  • Build and maintain data architectures including data lakes, warehouses (e.g., Snowflake, DB2)
  • Implement data modelling (star/snowflake schemas), ensure data quality, lineage tracking, and reliability engineering practices
  • Collaborate with data scientists, analysts, and stakeholders to translate requirements into robust data solutions while optimizing cost and performance
  • Deploy and monitor pipelines on cloud platforms (AWS/GCP/Azure) with CI/CD
  • Fulltime
Read More
Arrow Right