Pyspark Developer, Citi

Pyspark Developer

Citi

Location:
India, Chennai

Category:
IT - Software Development

Contract Type:
Employment contract

Salary:

Not provided

Save Job

Apply Position

Job Description:

We are looking for a highly skilled PySpark Developer with deep expertise in Distributed data processing. The ideal candidate will be responsible for optimizing Spark Jobs and ensuring efficient data processing in a Big Data platform. This role requires a strong understanding of Spark performance tuning, distributed computing, and Big Data architecture.

Job Responsibility:

Analyze and comprehend existing data ingestion and reconciliation frameworks
Develop and implement PySpark programs to process large datasets in Hive tables and Big Data platforms
Perform complex transformations including reconciliation and advanced data manipulations
Fine-tune Spark jobs for performance optimization, ensuring efficient data processing at scale
Work closely with Data Engineers, Architects, and Analysts to understand data reconciliation requirements
Collaborate with cross-functional teams to improve data ingestion, transformation, and validation workflows.

Requirements:

Extensive hands-on experience with Python, PySpark, and PyMongo for efficient data processing across distributed and columnar databases
Expertise in Spark Optimization techniques, and ability to debug Spark performance issues and optimize resource utilization
Proficiency in Python and Spark DataFrame API, and strong experience in complex data transformations using PySpark
Experience working with large-scale distributed data processing, and solid understanding of Big Data architecture and distributed computing frameworks
Strong problem-solving and analytical skills
Experience with CI/CD for data pipelines
Experience with SnowFlake for data processing and integration
Bachelor’s degree/University degree or equivalent experience in Computer science
Master’s degree preferred
8+ years of relevant experience in Apps Development or systems analysis.

Additional Information:

Job Posted:
May 08, 2025

Employment Type:

Fulltime

Work Type:

On-site work

View All Jobs In This Company

Job Link Share:

Welcome to CrawlJobs.com –
Your Global Job Discovery Platform

At CrawlJobs.com, we simplify finding your next career opportunity by bringing job listings directly to you from all corners of the web. Using cutting-edge AI and web-crawling technologies, we gather and curate job offers from various sources across the globe, ensuring you have access to the most up-to-date job listings in one place.