CrawlJobs Logo

Pyspark Module Lead

https://www.soprasteria.com Logo

Sopra Steria

Location Icon

Location:
India , Noida

Category Icon
Category:
IT - Software Development

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

Not provided
Save Job
Save Icon
Apply Position

Job Description:

We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. As a Data Engineer, you will collaborate closely with our Data Scientists to develop and deploy machine learning models. Responsibilities include working with PySpark, AWS EMR, and S3 for data processing, designing machine learning pipelines, optimizing pipelines for performance, and managing ETL workflows using Streamsets.

Job Responsibility:

  • Work in tandem with Data Scientists to design, develop, and implement machine learning pipelines
  • Utilize PySpark for data processing, transformation, and preparation for model training
  • Leverage AWS EMR and S3 for scalable and efficient data storage and processing
  • Implement and manage ETL workflows using Streamsets for data ingestion and transformation
  • Design and construct pipelines to deliver high-quality training and inference datasets
  • Collaborate with cross-functional teams to ensure smooth deployment and real-time/near real-time inferencing capabilities
  • Optimize and fine-tune pipelines for performance, scalability, and reliability
  • Ensure IAM policies and permissions are appropriately configured for secure data access and management
  • Implement Spark architecture and optimize Spark jobs for scalable data processing

Requirements:

  • Proficiency in Advanced SQL (Window functions), Spark Architecture, Pyspark or Scala with Spark, Hadoop
  • Proven expertise in designing and deploying data pipelines
  • Strong problem-solving skills and ability to work effectively in a collaborative team environment
  • Excellent communication skills and ability to translate technical concepts to non-technical stakeholders

Nice to have:

  • Hands-on experience with Airflow, S3, and Stream sets or similar ETL tools
  • Understanding of real-time or near real-time inferencing architectures
  • Basic knowledge on Kafka, AWS IAM, AWS EMR and Snowflake
What we offer:
  • All positions are open to people with disabilities
  • Commitment to fighting against all forms of discrimination
  • Inclusive and respectful work environment

Additional Information:

Job Posted:
April 26, 2025

Icon
Sopra Steria - All Job Offers
Job Link Share:
Copy Link Icon
Facebook Icon Twitter Icon LinkedIn Icon

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Pyspark Module Lead

Pyspark Module Lead

Sopra Steria is seeking a PySpark Module Lead based in Bengaluru, India. The pos...
Location
Location
India , Bengaluru
Salary
Salary:
Not provided
https://www.soprasteria.com Logo
Sopra Steria
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Python
  • PySpark
  • SQL
  • JavaScript
  • Proficient with Skywise platform and tools e.g. Contour, Code-Workbook, Code, Slate and Ontology
Job Responsibility
Job Responsibility
  • Designing and building solutions for Airbus on foundry
  • Data engineering and integration
  • Data analysis and visualization
  • Application development within Skywise Framework
  • Working closely with Product Owners and core Airbus business functions to understand processes, capture requirements, and solve business problems
  • Full delivery lifecycle for designing, implementing, testing, documenting, and supporting applications
  • CI/CD pipeline development and automated deployments
  • Troubleshooting application issues
  • Providing support on developed tools
What we offer
What we offer
  • Inclusive and respectful work environment
  • Open to people with disabilities
  • Fulltime
Read More
Arrow Right

PySpark Module Lead

PySpark Module Lead position at Sopra Steria, a major Tech player in Europe. The...
Location
Location
India , Bengaluru
Salary
Salary:
Not provided
https://www.soprasteria.com Logo
Sopra Steria
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Good experience in applying best coding practices
  • Good understanding and ability to develop simple to medium applications
  • Strong learnability and analytical skills
  • Strong team player with excellent communication skills
  • Mandatory skills: AWS, PySpark, PL SQL, Java
  • AWS Skills are on S3, Lambda, Cloudwatch, Dynamo DB, SQS, SNS, CloudFormation, Stack, Serverless
  • Total Experience Expected: 04-06 years
  • B.Tech, MCA or equivalent degree
What we offer
What we offer
  • Committed to fighting against all forms of discrimination
  • Fosters a work environment that is inclusive and respectful of all differences
  • All positions are open to people with disabilities
Read More
Arrow Right

Senior Software Engineer

The Senior Software Engineer will lead a team of data engineers to design, build...
Location
Location
United States , San Diego
Salary
Salary:
Not provided
gomillenniumsoft.com Logo
MillenniumSoft Inc
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 10+ years of experience in developing data solutions
  • Proven experience in designing and implementing large-scale data pipelines
  • Hands-on expertise with AWS cloud technologies and distributed systems
  • Strong programming skills in PySpark, Python, and SQL
  • Data pipeline development using AWS tools (Step Functions, Airflow, etc.)
  • Distributed systems (S3, ADLS), and modern file/table formats (Parquet, Hudi)
  • ETL tools, orchestration technologies, and database design
  • Knowledge of regulatory standards like SOX, PCI, and HIPAA
  • Bachelor’s or Master’s in Computer Science, Information Systems, or related fields
  • AWS certification preferred
Job Responsibility
Job Responsibility
  • Design and implement scalable data pipelines for structured and unstructured data
  • Lead module architecture design for optimal performance, reliability, and maintainability
  • Mentor team members and ensure alignment with business goals
  • Collaborate with cross-functional teams to enable real-time and batch data processing
  • Ensure adherence to data security, privacy, and governance standards
  • Create and maintain technical documentation for architectures and decisions
  • Conduct code reviews and establish best practices
  • Develop monitoring systems for data pipeline reliability
  • Drive adoption of new technologies and workflows
  • Fulltime
Read More
Arrow Right
New

Primary Teaching Assistant

Are you passionate about shaping young minds? As a Teaching Assistant in primary...
Location
Location
United Kingdom , Caterham
Salary
Salary:
Not provided
https://www.randstad.com Logo
Randstad
Expiration Date
May 22, 2026
Flip Icon
Requirements
Requirements
  • communication
  • teaching assistant experience
  • passionate
  • positive
  • flexible
  • hold two work references
  • UK work eligibility
Job Responsibility
Job Responsibility
  • providing crucial support to the class teacher
  • working with a child one-on-one or in small groups
  • preparing the classroom and aiding in the learning process
What we offer
What we offer
  • Convenient Mobility: Ideal for drivers due to diverse school locations
  • Versatile Exposure: Work across various schools for diverse experiences
  • Weekly Pay: Enjoy hassle-free weekly payments with fantastic daily rates
  • Effortless Hours: Submit your hours seamlessly through our time sheet app
  • Rewards Await: Earn £300 through our Refer a Friend scheme
  • Dedicated Support: Benefit from a consultant deeply connected with supported schools
  • Personalised On boarding: Receive tailored assistance for compliance
  • Compliance Assistance: Support for DBS on the update service or help to obtain one
  • career
  • work-life
  • Fulltime
Read More
Arrow Right
New

Primary Teaching Assistant

Are you passionate about shaping young minds? As a Teaching Assistant in primary...
Location
Location
United Kingdom , Caterham
Salary
Salary:
Not provided
https://www.randstad.com Logo
Randstad
Expiration Date
May 21, 2026
Flip Icon
Requirements
Requirements
  • Non Teaching
  • communication
  • teaching assistant experience
Job Responsibility
Job Responsibility
  • Providing crucial support to the class teacher
  • Working with a child one-on-one or in small groups
  • Preparing the classroom
  • Aiding in the learning process
What we offer
What we offer
  • Convenient Mobility: Ideal for drivers due to diverse school locations
  • Versatile Exposure: Work across various schools for diverse experiences
  • Weekly Pay: Enjoy hassle-free weekly payments with fantastic daily rates
  • Effortless Hours: Submit your hours seamlessly through our time sheet app
  • Rewards Await: Earn £300 through our Refer a Friend scheme
  • Dedicated Support: Benefit from a consultant deeply connected with supported schools
  • Personalised On boarding: Receive tailored assistance for compliance
  • Essential Traits: Be passionate, positive, flexible, and hold two work references
  • Compliance Assistance: Support for DBS on the update service or help to obtain one, and UK work eligibility
  • Fulltime
Read More
Arrow Right
New

Long term supply teacher

Transform lives through education as a caring and dedicated teacher, making a la...
Location
Location
United Kingdom , Caterham
Salary
Salary:
Not provided
https://www.randstad.com Logo
Randstad
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • QTS/PGCE qualification required
  • comprehensive working history and two recent working references
  • DBS on the update service or readiness to obtain one
  • eligibility to work in the UK
  • passion and positivity for teaching
  • good communication skills
  • key stage 1 experience
Job Responsibility
Job Responsibility
  • Full teaching responsibilities within your placement
  • marking, planning, and teaching throughout extended periods
  • payment to scale for placements surpassing 6 weeks
  • full-time availability dedicated to teaching
What we offer
What we offer
  • Refer a friend bonus scheme
  • weekly pay with competitive daily rates
  • simple hour submissions through user-friendly app
  • £300 Refer a Friend bonus
  • dedicated consultant
  • personalised assistance for compliance and seamless onboarding
  • career
  • work-life
  • Fulltime
Read More
Arrow Right
New

Supply Sen Teaching Assistant

Are you passionate about shaping young minds? As a Teaching Assistant in primary...
Location
Location
United Kingdom , Caterham
Salary
Salary:
Not provided
https://www.randstad.com Logo
Randstad
Expiration Date
May 21, 2026
Flip Icon
Requirements
Requirements
  • Non Teaching
  • 121 teaching experience
  • communication
  • SEN experience
  • teaching assistant experience
What we offer
What we offer
  • Career
  • Work-Life balance
  • Refer a friend bonus scheme
  • Dedicated support
  • Personalised onboarding
  • Compliance assistance
  • Fulltime
Read More
Arrow Right
New

Primary Teaching Assistant

Are you passionate about shaping young minds? As a Teaching Assistant in primary...
Location
Location
United Kingdom , Caterham
Salary
Salary:
Not provided
https://www.randstad.com Logo
Randstad
Expiration Date
May 21, 2026
Flip Icon
Requirements
Requirements
  • communication
  • teaching assistant experience
  • Non Teaching
  • passionate
  • positive
  • flexible
  • hold two work references
  • UK work eligibility
Job Responsibility
Job Responsibility
  • providing crucial support to the class teacher
  • working with a child one-on-one or in small groups
  • preparing the classroom and aiding in the learning process
What we offer
What we offer
  • Refer a friend bonus scheme
  • Convenient Mobility
  • Versatile Exposure
  • Weekly Pay
  • Effortless Hours
  • Rewards Await £300 through Refer a Friend scheme
  • Dedicated Support
  • Personalised On boarding
  • Compliance Assistance
  • training and education
  • Fulltime
Read More
Arrow Right