CrawlJobs Logo

Pyspark Module Lead

https://www.soprasteria.com Logo

Sopra Steria

Location Icon

Location:
India , Noida

Category Icon
Category:
IT - Software Development

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

Not provided
Save Job
Save Icon
Apply Position

Job Description:

We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. As a Data Engineer, you will collaborate closely with our Data Scientists to develop and deploy machine learning models. Responsibilities include working with PySpark, AWS EMR, and S3 for data processing, designing machine learning pipelines, optimizing pipelines for performance, and managing ETL workflows using Streamsets.

Job Responsibility:

  • Work in tandem with Data Scientists to design, develop, and implement machine learning pipelines
  • Utilize PySpark for data processing, transformation, and preparation for model training
  • Leverage AWS EMR and S3 for scalable and efficient data storage and processing
  • Implement and manage ETL workflows using Streamsets for data ingestion and transformation
  • Design and construct pipelines to deliver high-quality training and inference datasets
  • Collaborate with cross-functional teams to ensure smooth deployment and real-time/near real-time inferencing capabilities
  • Optimize and fine-tune pipelines for performance, scalability, and reliability
  • Ensure IAM policies and permissions are appropriately configured for secure data access and management
  • Implement Spark architecture and optimize Spark jobs for scalable data processing

Requirements:

  • Proficiency in Advanced SQL (Window functions), Spark Architecture, Pyspark or Scala with Spark, Hadoop
  • Proven expertise in designing and deploying data pipelines
  • Strong problem-solving skills and ability to work effectively in a collaborative team environment
  • Excellent communication skills and ability to translate technical concepts to non-technical stakeholders

Nice to have:

  • Hands-on experience with Airflow, S3, and Stream sets or similar ETL tools
  • Understanding of real-time or near real-time inferencing architectures
  • Basic knowledge on Kafka, AWS IAM, AWS EMR and Snowflake
What we offer:
  • All positions are open to people with disabilities
  • Commitment to fighting against all forms of discrimination
  • Inclusive and respectful work environment

Additional Information:

Job Posted:
April 26, 2025

Icon
Sopra Steria - All Job Offers
Job Link Share:
Copy Link Icon
Facebook Icon Twitter Icon LinkedIn Icon

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Pyspark Module Lead

Pyspark Module Lead

Sopra Steria is seeking a PySpark Module Lead based in Bengaluru, India. The pos...
Location
Location
India , Bengaluru
Salary
Salary:
Not provided
https://www.soprasteria.com Logo
Sopra Steria
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Python
  • PySpark
  • SQL
  • JavaScript
  • Proficient with Skywise platform and tools e.g. Contour, Code-Workbook, Code, Slate and Ontology
Job Responsibility
Job Responsibility
  • Designing and building solutions for Airbus on foundry
  • Data engineering and integration
  • Data analysis and visualization
  • Application development within Skywise Framework
  • Working closely with Product Owners and core Airbus business functions to understand processes, capture requirements, and solve business problems
  • Full delivery lifecycle for designing, implementing, testing, documenting, and supporting applications
  • CI/CD pipeline development and automated deployments
  • Troubleshooting application issues
  • Providing support on developed tools
What we offer
What we offer
  • Inclusive and respectful work environment
  • Open to people with disabilities
  • Fulltime
Read More
Arrow Right

PySpark Module Lead

PySpark Module Lead position at Sopra Steria, a major Tech player in Europe. The...
Location
Location
India , Bengaluru
Salary
Salary:
Not provided
https://www.soprasteria.com Logo
Sopra Steria
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Good experience in applying best coding practices
  • Good understanding and ability to develop simple to medium applications
  • Strong learnability and analytical skills
  • Strong team player with excellent communication skills
  • Mandatory skills: AWS, PySpark, PL SQL, Java
  • AWS Skills are on S3, Lambda, Cloudwatch, Dynamo DB, SQS, SNS, CloudFormation, Stack, Serverless
  • Total Experience Expected: 04-06 years
  • B.Tech, MCA or equivalent degree
What we offer
What we offer
  • Committed to fighting against all forms of discrimination
  • Fosters a work environment that is inclusive and respectful of all differences
  • All positions are open to people with disabilities
Read More
Arrow Right

Senior Software Engineer

The Senior Software Engineer will lead a team of data engineers to design, build...
Location
Location
United States , San Diego
Salary
Salary:
Not provided
gomillenniumsoft.com Logo
MillenniumSoft Inc
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 10+ years of experience in developing data solutions
  • Proven experience in designing and implementing large-scale data pipelines
  • Hands-on expertise with AWS cloud technologies and distributed systems
  • Strong programming skills in PySpark, Python, and SQL
  • Data pipeline development using AWS tools (Step Functions, Airflow, etc.)
  • Distributed systems (S3, ADLS), and modern file/table formats (Parquet, Hudi)
  • ETL tools, orchestration technologies, and database design
  • Knowledge of regulatory standards like SOX, PCI, and HIPAA
  • Bachelor’s or Master’s in Computer Science, Information Systems, or related fields
  • AWS certification preferred
Job Responsibility
Job Responsibility
  • Design and implement scalable data pipelines for structured and unstructured data
  • Lead module architecture design for optimal performance, reliability, and maintainability
  • Mentor team members and ensure alignment with business goals
  • Collaborate with cross-functional teams to enable real-time and batch data processing
  • Ensure adherence to data security, privacy, and governance standards
  • Create and maintain technical documentation for architectures and decisions
  • Conduct code reviews and establish best practices
  • Develop monitoring systems for data pipeline reliability
  • Drive adoption of new technologies and workflows
  • Fulltime
Read More
Arrow Right
New

Regional finance associate

We are seeking a high-potential Finance Associate to be groomed for future manag...
Location
Location
Malaysia , Kuala Lumpur
Salary
Salary:
6000.00 - 9000.00 MYR / Month
https://www.randstad.com Logo
Randstad
Expiration Date
May 02, 2026
Flip Icon
Requirements
Requirements
  • Degree in Accounting, Business Administration, or an equivalent professional qualification
  • 3 – 5 years of accounting experience
  • External audit exposure in fund accounting, private equity, or fund management (preference)
  • Strong foundation in Singapore Financial Reporting Standards (FRS) and IRAS regulations (significant advantage)
  • Proficiency in Microsoft Office
  • Experience with NetSuite or similar ERP systems (bonus)
  • Proactive problem-solver with a passion for process improvement
  • Ability to prioritize and perform under tight timelines with minimal supervision
  • Excellent communication skills and high attention to detail
Job Responsibility
Job Responsibility
  • Independently manage the full set of accounts for assigned entities, encompassing monthly closing, reconciliations, and supporting schedules
  • Execute core accounting functions including cash management, payment operations, and payroll processing
  • Determine Net Asset Values (NAVs), calculate fund performance, and ensure the accuracy of fund accounting records
  • Prepare and analyze financial and operational reports to facilitate management decision-making
  • Lead the coordination of annual external audits across multiple entities and funds, ensuring all schedules and documentation are complete and accurate
  • Prepare and submit quarterly GST filings and manage local tax matters with relevant authorities
  • Assist in the preparation of statutory forms, reports, and surveys required by the Monetary Authority of Singapore (MAS)
  • Ensure compliance with statutory regulations and internal financial controls
  • Manage relationships with professional service providers, including fund administrators, banks, tax agents, corporate secretaries, and auditors
  • Support the distribution of returns to existing investors, including the drafting of necessary documentation
Read More
Arrow Right
New

Assistant Front Office Manager

Assists the Front Office Manager in administering front office functions and sup...
Location
Location
United States , Maui
Salary
Salary:
72000.00 - 95000.00 USD / Year
https://www.marriott.com Logo
Marriott Bonvoy
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • High school diploma or GED
  • 2 years experience in the guest services, front desk, or related professional area
  • OR 2-year degree from an accredited university in Hotel and Restaurant Management, Hospitality, Business Administration, or related major
  • no work experience required
Job Responsibility
Job Responsibility
  • Assists the Front Office Manager in administering front office functions and supervising staff on a daily basis
  • Utilizes interpersonal and communication skills to lead, influence, and encourage others
  • advocates sound financial/business decision making
  • demonstrates honesty/integrity
  • leads by example
  • Encourages and builds mutual trust, respect, and cooperation among team members
  • Supervises and manages employees
  • Managing all day-to-day operations
  • Understanding employee positions well enough to perform duties in employees' absence
  • Ensures employee recognition is taking place on all shifts
  • Fulltime
Read More
Arrow Right
New

VP Sales - Insurtech

An innovative and fast-growing insurtech operating at the forefront of digital t...
Location
Location
United Kingdom , London
Salary
Salary:
120000.00 GBP / Year
calibrecandidates.com Logo
Calibre Candidates
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proven track record of SaaS sales success within Insurance and London Market clients
  • Strong focus within Property & Casualty (P&C) insurance
  • Entrepreneurial mindset with a proactive, self-starting approach
  • Demonstrated ability to build and leverage a strong industry network
  • Credible communicator with the ability to engage C-suite stakeholders
  • Commercially astute with a strategic approach to business development
Job Responsibility
Job Responsibility
  • Drive new business sales across London Market (insurers, brokers, MGAs)
  • Build and execute a strategic sales plan aligned with company growth targets
  • Develop and maintain a strong network of senior industry contacts
  • Identify and convert new opportunities within Property & Casualty (P&C) lines
  • Collaborate with product and leadership teams to refine market proposition
  • Represent the business at industry events, conferences, and regular client meetings
  • Lead complex sales cycles from origination through to close
  • Fulltime
Read More
Arrow Right
New

Finance Manager

You will be working for a dynamic and growing property development company commi...
Location
Location
Malaysia , Kuala Lumpur
Salary
Salary:
10000.00 - 13000.00 MYR / Month
https://www.randstad.com Logo
Randstad
Expiration Date
May 03, 2026
Flip Icon
Requirements
Requirements
  • Bachelor’s degree in Finance, Accounting, or related field
  • Minimum 5 years experience in finance or accounting roles
  • leadership experience preferred
  • Strong knowledge of Malaysian financial regulations and tax laws
Job Responsibility
Job Responsibility
  • Prepare and analyze financial statements, budgets, forecasts, and reports
  • Manage accounting operations including month-end and year-end closing
  • Ensure compliance with statutory requirements, financial regulations, and tax filings
  • Lead financial planning and analysis to support decision making
What we offer
What we offer
  • Competitive salary and performance bonus
  • Medical and healthcare benefits
  • EPF & SOCSO contributions
  • Learning and development opportunities
  • Supportive and collaborative work environment
Read More
Arrow Right
New

Regional Senior Accountant

The Regional Accountant will play a pivotal role in our Malaysia-based Southeast...
Location
Location
Malaysia , Kuala Lumpur
Salary
Salary:
8000.00 - 12000.00 MYR / Month
https://www.randstad.com Logo
Randstad
Expiration Date
May 02, 2026
Flip Icon
Requirements
Requirements
  • Degree in Accounting or Finance
  • professional certification (e.g., ACCA, CPA, CIMA, or MIA) is highly preferred
  • Mandatory foundational experience (3+ years) in a Big 4 Audit Firm (PwC, EY, KPMG, or Deloitte)
  • Proven track record in a commercial accounting or regional reporting role within an MNC
  • Hands-on experience with major ERP systems (e.g., SAP or Oracle) and advanced Microsoft Excel skills
  • Strong analytical mindset, excellent communication skills for stakeholder management, and the ability to work independently in a fast-paced environment
Job Responsibility
Job Responsibility
  • Oversee the month-end and year-end closing processes for assigned SEA entities, ensuring accuracy and timeliness
  • Prepare and consolidate regional financial statements in accordance with IFRS/MFRS and global accounting policies
  • Perform in-depth variance analysis on regional P&L and Balance Sheet movements to provide management with actionable insights
  • Manage intercompany reconciliations and settlements across various regional jurisdictions
  • Ensure all regional entities remain compliant with local statutory laws, tax regulations, and internal control frameworks
  • Coordinate and manage statutory audits by acting as the primary liaison for external auditors across the region
  • Support regional tax compliance, including the preparation of tax computations and documentation for Transfer Pricing and Withholding Tax
  • Drive the implementation of internal control improvements to mitigate regional operational risks
  • Monitor regional cash flow performance and prepare accurate cash forecasts for liquidity planning
  • Oversee Accounts Receivable (AR) and Accounts Payable (AP) functions to optimize net working capital
Read More
Arrow Right
Jobs by Category
Art and Entertainment Jobs Banking Jobs Call Center Jobs Construction Jobs Consulting Jobs Customer Service Jobs Education Jobs Energy Jobs Finance Jobs Gastronomy Jobs Health and Beauty Jobs Hospitality and Tourism Jobs Human Resources Jobs Insurance Jobs IT - Administration Jobs IT - Software Development Jobs Legal Jobs Logistics Jobs Manufacturing Jobs Marketing, Digital Marketing, SEO, SEM Jobs Media Jobs Nursing Jobs Office Administration Jobs Pharmacy Jobs Physical Work Jobs Public Relations Jobs Public Sector Jobs Purchasing Jobs Quality Control Jobs Real Estate Jobs Research and Development Jobs Sales Jobs
Job Positions Jobs by Country
United States Jobs United Kingdom Jobs India Jobs Canada Jobs Australia Jobs Germany Jobs Ireland Jobs Poland Jobs
Company
Blog About Us Contact
Follow Us
Facebook
X (Twitter)
LinkedIn
Instagram
CrawlJobs Group
ITFlashcards AllDevBlogs
© 2026 CrawlJobs Ltd. All Rights Reserved
Terms & Conditions Privacy Policy