CrawlJobs Logo
Realign Logo Realign · IT - Software Development

Python/Pyspark Engineer

United States, Jersey City 115000.00 USD / Year · Job Posted March 21, 2026
Apply Position
Job Link Share

Job Responsibility

  • Design and develop scalable Python/PySpark ingestion and transformation pipelines
  • Implement schema evolution logic, validation frameworks, and resilient error-handling mechanisms
  • Optimize Spark jobs for performance, cost efficiency, and production readiness
  • Integrate all jobs into automated CI/CD pipelines, ensuring versioning and release governance
  • Work closely with Ops teams to ensure proper monitoring, logging, and operational supportability
  • Participate in Agile ceremonies, sprint planning, code reviews, and demo sessions

Requirements

  • Strong proficiency in Python, packaging, dependency management, and virtual environments
  • Hands-on experience with PySpark, including Spark performance tuning (partitioning, caching, broadcast joins, memory optimization)
  • Expertise in data ingestion (batch/stream), schema management, and robust error-handling/retry logic
  • Solid unit and integration testing practices, including data quality validations
  • Experience with CI/CD pipelines (Azure DevOps/Jenkins), Git branching strategies, and artifact versioning
  • Working experience with Cloudera/Hadoop (HDFS, Spark, Hive/Impala) and Databricks (Delta Lake, clusters, jobs, notebooks)
  • Knowledge of observability techniques: structured logging, metrics, tracing, and debugging in distributed systems
  • Secure coding practices including secrets management, PII protection, and compliance-aware development
  • Strong documentation discipline for frameworks, reusable components, and best-practice patterns
  • Effective collaboration with Cloud Architects and Data Ops to ensure stable and supportable pipelines
  • Clear communication of technical ideas and solution approaches
  • Comfort working in Agile environments with iterative development and frequent releases
Realign - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Python/Pyspark Engineer

8 matching positions

Python/Pyspark Engineer

Location
Location
Slovakia , Bratislava
Salary
Salary:
Not provided
signifytechnology.com Logo
Signify Technology
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • min. 4-year demonstrable project experience in the field of software Python engineering
  • language SQL for searching and manipulating data
  • framework PySpark or equivalent for creating and optimizing complex data pipelines
  • Scrum/Agile development methodologies
  • working in a global distributed team in a multicultural environment
  • ability to clearly express technical topics to a non-technical audience
  • active knowledge of English at a communicative level (min. B2-C1)
  • min. Bachelor's or equivalent degree in computer science, data science or a similar discipline
Job Responsibility
Job Responsibility
  • Development of a modern Lakehouse architecture based on Azure Datalake using Python and the PySpark framework for implementing business services in the field of insurance
  • implementation of business functions that will allow you to run accounting processes and generate data to meet reporting requirements
  • designing, developing, automating and supporting backend applications that combine data elements from multiple domains and systems
  • cooperation with: other engineers, analysts, product owners and stakeholders to deliver value-added solutions that meet business needs and expectations
  • team lead engineer to create a target architecture for products within the team's scope
  • design of data transformation and data flow services and active participation in coding
  • presentation and communication of ideas and proposals to various stakeholders for the purpose of evaluation and brainstorming
  • implementation of software engineering practices to ensure the quality, performance and sustainability of applications
  • performing peer code reviews
Read More
Arrow Right
New

Principal Data Platform Engineer Vice President

We are seeking an exceptionally skilled and motivated Principal Data Platform En...
Location
Location
United States , Irving; Jacksonville
Salary
Salary:
125760.00 - 188640.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 6+ years of relevant experience in Apps Development or systems analysis role with extensive Python and/or big data expertise
  • Python/Pyspark Mastery
  • Big Data Technologies & Platforms: Extensive experience (8+ years) architecting, designing, implementing, and managing solutions within distributed data processing platforms, specifically with the Hadoop ecosystem (preferably Cloudera distributions). Proficient in leveraging key big data components such as distributed file systems (e.g., HDFS), data warehousing solutions (e.g., Hive), data transformation frameworks (e.g., Pig), and data ingestion tools (e.g., Sqoop), alongside hands-on experience with NoSQL databases (preferably MongoDB)
  • ETL Architecture & Development: Proven ability to architect, design, and implement highly scalable data pipelines. Extensive experience leveraging industry-standard ETL tools and frameworks for efficient data extraction, transformation, and loading into various relational databases and data warehouses, coupled with a strategic vision and planning experience for migrating to cloud-native, serverless ETL solutions
  • Data Architecture & Strategic Modeling: Understanding of advanced data modeling principles and practical experience in data warehouse design and development, ensuring data integrity, scalability, security, and optimal performance
  • AI-Powered Development: Ability to leverage advanced AI tools, such as Devin, for efficient code refactoring, optimization, and identifying potential code improvements, thereby enhancing code quality and developer productivity
  • DevOps, Version Control & Containerization: CI/CD pipelines, Git, Docker and Kubernetes
  • Extensive experience system analysis and in programming of software applications
  • Experience in managing and implementing successful projects
  • Subject Matter Expert (SME) in at least one area of Applications Development
Job Responsibility
Job Responsibility
  • Partner with multiple management teams to ensure appropriate integration of functions to meet goals as well as identify and define necessary system enhancements to deploy new products and process improvements
  • Resolve variety of high impact problems/projects through in-depth evaluation of complex business processes, system processes, and industry standards
  • Provide expertise in area and advanced knowledge of applications programming and ensure application design adheres to the overall architecture blueprint
  • Utilize advanced knowledge of system flow and develop standards for coding, testing, debugging, and implementation
  • Develop comprehensive knowledge of how areas of business, such as architecture and infrastructure, integrate to accomplish business goals
  • Provide in-depth analysis with interpretive thinking to define issues and develop innovative solutions
  • Serve as advisor or coach to mid-level developers and analysts, allocating work as necessary
  • Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency
What we offer
What we offer
  • medical, dental & vision coverage
  • 401(k)
  • life, accident, and disability insurance
  • wellness programs
  • paid time off packages, including planned time off (vacation), unplanned time off (sick leave), and paid holidays
  • Fulltime
Read More
Arrow Right

Epss Systems & Data Management Intern

The Enhanced Powerline Safety Settings Program Management Office (EPSS PMO) is a...
Location
Location
United States , Oakland
Salary
Salary:
20.88 - 35.37 USD / Hour
pge.com Logo
PG&E Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Qualified candidates are pursuing a Bachelor’s or Master’s degree in Engineering, Data Science or related field at an accredited University
  • Students must be continuing their education towards their degree during and/or after the internship
  • Must have reliable transportation to and from work location
Job Responsibility
Job Responsibility
  • Supporting senior staff with: Python/pyspark coding & debugging
  • Data pipeline building & maintenance
  • Front-end application features & enhancements
  • Reporting on-site to company headquarters 2-3 days per week
  • Travel to the field may be required
  • frequency depends on the assignment and is typically less than 10%
Read More
Arrow Right

Data Migration Engineer

Location
Location
United Kingdom , London
Salary
Salary:
Not provided
nttdata.com Logo
NTT DATA
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience in data engineering or data migration delivery
  • Strong focus on testing, validation, and data quality assurance
  • Ability to work across data pipelines and transformation workflows
  • Good analytical and problem-solving skills
  • Effective communication and teamwork skills
  • Willingness to learn and develop in data migration and cloud technologies
  • Hands-on experience with: AWS cloud services, especially AWS Glue, Python / PySpark, SQL querying and validation
  • YAML configuration (desirable)
  • Experience testing or supporting: ETL/ELT pipelines, Data migration processes
  • Familiarity with: Data lake / Lakehouse concepts (e.g., Apache Iceberg), Distributed processing frameworks (e.g., Spark)
Job Responsibility
Job Responsibility
  • Support delivery within data migration programmes, contributing to key workstreams
  • Collaborate with architects, engineers, and stakeholders to implement migration solutions
  • Assist in planning and executing data migration tasks and deliverables
  • Build and maintain data migration pipelines from legacy data warehouses to AWS-based platforms
  • Develop ETL/ELT pipelines using: AWS Glue, Python / PySpark, SQL, YAML configurations
  • Support execution of bulk data migrations and incremental/delta loads
  • Assist with pipeline repointing and migration to cloud environments
  • Test ETL/ELT data pipelines on AWS services, including AWS Glue and Apache Iceberg
  • Support validation of data pipeline migrations to AWS Data Lakehouse architectures
  • Test pipelines using: Python/PySpark transformations, SQL-based validation logic, YAML-driven configurations
What we offer
What we offer
  • Range of tailored benefits that support your physical, emotional, and financial wellbeing
  • Continuous growth and development opportunities
  • Flexible work options
  • Fulltime
Read More
Arrow Right

.Net Full Stack Engineer

We believe in the power of ingenuity to build a positive human future. As strate...
Location
Location
United States , Boston
Salary
Salary:
155000.00 - 180000.00 USD / Year
paconsulting.com Logo
PA Consulting
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8-10 years of industry working experience
  • Experience or coursework developing full-stack web applications with ASP.NET Framework (.NET 7 preferred)
  • Basic understanding of Entity Framework Core and relational databases (e.g., Azure SQL Database, MSSQL)
  • Familiarity with building and deploying web applications on cloud platforms (AWS, GCP, Azure)
  • Exposure to YAML-based CI/CD pipelines or similar tools (e.g., Azure DevOps, GitHub)
  • Interest in or exposure to .NET Blazor for frontend development
  • Some familiarity with Azure services like Azure Web Apps or Functions
  • Eagerness to learn Python/PySpark and deepen cloud deployment expertise
Job Responsibility
Job Responsibility
  • develop clean .NET code
  • contribute to full-stack applications
  • gain exposure to cloud-based deployment and CI/CD processes
What we offer
What we offer
  • Group medical insurance
  • Health Savings Account with company match
  • Teladoc and informed Nurse line resources
  • Long term care plan
  • Group dental insurance
  • Vision plan
  • 401(k) Savings Plan with company profit sharing contribution
  • Commuter and Parking tax-savings benefit
  • 15 days paid vacation days with the opportunity to buy five additional days
  • 10 paid Holidays plus 10 paid sick days
  • Fulltime
Read More
Arrow Right

Senior Data Engineer

We are looking for a hands-on Data Engineer who is passionate about solving busi...
Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 6+ Years Experience of implementing data-intensive solutions using agile methodologies
  • Code contributing member of Agile teams, working to deliver sprint goals
  • Write clean, efficient, and maintainable code that meets the highest standards of quality
  • Very strong in coding Python/Pyspark, UNIX shell scripting
  • Experience in cloud native technologies and patterns
  • Ability to automate and streamline the build, test and deployment of data pipelines
  • ETL: Hands on experience of building data pipelines. Proficiency in data integration platforms such as Apache Spark
  • Experienced in writing Pyspark code to handle large data set ,perform data transformation , familiarity with Pyspark integration with other Apache Spark component ,such as Spark SQL , Understanding of Pyspark optimization techniques
  • Strong proficiency in working with relational databases and using SQL for data querying, transformation, and manipulation
  • Big Data:Exposure to 'big data' platforms such as Hadoop, Hive or Iceberg for data storage and processing
Job Responsibility
Job Responsibility
  • We are looking for a hands-on Data Engineer who is passionate about solving business problems through innovation and engineering practices
  • As a Data Engineer, the candidate will leverage deep technical knowledge and will apply knowledge of data architecture standards, data warehousing, data structures, and business intelligence to drive the creation of high-quality data products for data driven decision making
  • Fulltime
Read More
Arrow Right

ETL Engineer

The Applications Development Intermediate Programmer Analyst is an intermediate ...
Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 2-5 years of relevant experience in the Financial Service industry
  • Intermediate level experience in Applications Development role
  • Consistently demonstrates clear and concise written and verbal communication
  • Demonstrated problem-solving and decision-making skills
  • Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
  • Bachelor's degree/University degree or equivalent experience
  • 5+ years of relevant experience as a data engineer
  • Data Pipelining & ETL: Expertise in designing and implementing scalable data pipelines using Python/Pyspark, with experience in ETL tools (e.g., Microsoft BI/SSIS, IBM DataStage etc.)
  • Database Management: Advanced proficiency (3+ years) in RDBMS (T-SQL/PL/SQL) and NoSQL (e.g., MongoDB)
  • Big Data Platforms: Extensive experience (3+ years) with Hadoop ecosystem technologies (e.g., HDFS, Hive, Pig, Sqoop), preferably Cloudera
Job Responsibility
Job Responsibility
  • Utilize knowledge of applications development procedures and concepts, and basic knowledge of other technical areas to identify and define necessary system enhancements, including using script tools and analyzing/interpreting code
  • Consult with users, clients, and other technology groups on issues, and recommend programming solutions, install, and support customer exposure systems
  • Apply fundamental knowledge of programming languages for design specifications
  • Analyze applications to identify vulnerabilities and security issues, as well as conduct testing and debugging
  • Serve as advisor or coach to new or lower level analysts
  • Identify problems, analyze information, and make evaluative judgements to recommend and implement solutions
  • Resolve issues by identifying and selecting solutions through the applications of acquired technical experience and guided by precedents
  • Has the ability to operate with a limited level of direct supervision
  • Can exercise independence of judgement and autonomy
  • Acts as SME to senior stakeholders and /or other team members
  • Fulltime
Read More
Arrow Right

Data & Analytics Engineer

As a Data & Analytics Engineer at Dynavox Group, you’ll step into a broad and dy...
Location
Location
Sweden , Stockholm
Salary
Salary:
Not provided
Tobii Dynavox
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree or equivalent in Computer Science, Engineering (Industrial or Mechanical) or similar fields with skills in statistics and reporting
  • Experience in BI roles such as BI Specialist, Analyst, Consultant, or Data Engineer
  • Coding skills in SQL, Python/PySpark with interest in business processes and curiosity to explore programming languages or platforms as our solutions develop
  • Communication, collaboration and interpersonal skills to work closely with external and internal stakeholders across various levels and functions
  • Proficiency in both spoken and written communication in English
  • Proficiency in Swedish is valued
Job Responsibility
Job Responsibility
  • Work to improve and streamline BI data warehouse maintenance, enhancements and daily operations
  • Translate business needs into technical solutions through hands-on development around architecture, coding and integration to contribute to a scalable, and stable BI architecture
  • Build and optimize data models, semantic layers, and ETL pipelines
  • Be part of the BI data warehouse maintenance, enhancement and daily operations
  • Help develop capabilities within Azure technologies such as AI, Fabric, Event Hubs, Stream Analytics and Synapse pipelines
  • Work across the full data value chain from integrating data sources using connectivity tools and APIs, to transforming and aligning data, and delivering insights via reports, dashboards, and analytical models
  • Contribute to the improvement of Power BI dashboards and analytics solutions using the Microsoft BI stack
  • Collaborate with colleagues to support global data initiatives and participate in projects focused on customer usage data and event-based analytics
What we offer
What we offer
  • Purpose-Driven Work
  • Yes, and... Flexibility
  • Growth and Development
  • Inclusive and Supportive Culture
  • A Global Leader with Heart
  • Fulltime
Read More
Arrow Right