CrawlJobs Logo
Realign Logo Realign · IT - Software Development

Pyspark Engineer

United States, Irving 135000.00 USD / Year · Job Posted May 04, 2026
Apply Position
Job Link Share

Job Description

Job Title: Pyspark Engineer Location: Irving, TX (ONSITE) F2F Interview Full Time ONLY Job Description Must Have Technical/Functional Skills PySpark Developer with 5-10 years’ experience in data engineering practice. Responsible for designing, developing and maintaining scalable data pipelines, optimizing data workflows and ensuring the integrity and availability of data for business intelligence

Job Responsibility

  • Designing, developing and maintaining scalable data pipelines
  • optimizing data workflows
  • ensuring the integrity and availability of data for business intelligence
  • onsite role with strong experience in Apache Spark framework
  • collaborate with the stakeholders and analysts to understand data requirement and deliver robust, creative and innovative solutions
  • perform root-cause analysis and troubleshoot errors on data pipelines, evaluating data quality issues, and implementing corrective fixes

Requirements

  • PySpark Developer with 5-10 years’ experience in data engineering practice
  • strong experience in Apache Spark framework including good understanding of core concepts, performance optimization and industry best practices
  • proficient in PySpark with hands-on coding experience and ability to implement complex business level transformations
  • collaborate with the stakeholders and analysts to understand data requirement and deliver robust, creative and innovative solutions
  • familiarity with unit testing, object-oriented programming (OOPS) concepts and interpreting test results
  • proficient to write complex and efficient SQL queries to extract the business-critical insights from large-scale data
  • experience with scheduling of the transformation jobs as per business requirement
  • perform root-cause analysis and troubleshoot errors on data pipelines, evaluating data quality issues, and implementing corrective fixes
Realign - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Pyspark Engineer

8 matching positions

Pyspark Engineer

Job Title: Pyspark Engineer Location: Irving, TX (ONSITE) F2F Interview Full Ti...
Location
Location
United States , Irving
Salary
Salary:
135000.00 USD / Year
realign-llc.com Logo
Realign
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • PySpark Developer with 5-10 years’ experience in data engineering practice
  • Strong experience in Apache Spark framework including good understanding of core concepts, performance optimization and industry best practices
  • Proficient in PySpark with hands-on coding experience and ability to implement complex business level transformations
  • Familiarity with unit testing, object-oriented programming (OOPS) concepts and interpreting test results
  • Proficient to write complex and efficient SQL queries to extract the business-critical insights from large-scale data
  • Experience with scheduling of the transformation jobs as per business requirement
  • Perform root-cause analysis and troubleshoot errors on data pipelines, evaluating data quality issues, and implementing corrective fixes
Job Responsibility
Job Responsibility
  • Designing, developing and maintaining scalable data pipelines
  • Optimizing data workflows and ensuring the integrity and availability of data for business intelligence
  • Collaborate with the stakeholders and analysts to understand data requirement and deliver robust, creative and innovative solutions
  • Fulltime
Read More
Arrow Right

Senior Python Pyspark Engineer

The Applications Development Senior Programmer Analyst is an intermediate level ...
Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8 - 10 years of relevant experience
  • Experience in systems analysis and programming of software applications
  • Experience in managing and implementing successful projects
  • Working knowledge of consulting/project management techniques/methods
  • Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
  • Programming Languages:Python, PySpark
  • Data Lake Table Format: Apache Iceberg
  • Data Orchestration:Apache Airflow
  • Data Visualization: Tableau
  • Big Data Processing: Apache Spark
Job Responsibility
Job Responsibility
  • Conduct tasks related to feasibility studies, time and cost estimates, IT planning, risk technology, applications development, model development, and establish and implement new or revised applications systems and programs to meet specific business needs or user areas
  • Monitor and control all phases of development process and analysis, design, construction, testing, and implementation as well as provide user and operational support on applications to business users
  • Utilize in-depth specialty knowledge of applications development to analyze complex problems/issues, provide evaluation of business process, system process, and industry standards, and make evaluative judgement
  • Recommend and develop security measures in post implementation analysis of business usage to ensure successful system design and functionality
  • Consult with users/clients and other technology groups on issues, recommend advanced programming solutions, and install and assist customer exposure systems
  • Ensure essential procedures are followed and help define operating standards and processes
  • Serve as advisor or coach to new or lower level analysts
  • Has the ability to operate with a limited level of direct supervision.
  • Can exercise independence of judgement and autonomy.
  • Acts as SME to senior stakeholders and /or other team members.
  • Fulltime
Read More
Arrow Right

Python PySpark Engineer

Location
Location
Poland , Wroclaw; Bialystok; Cracow; Gdansk; Lodz; Szczecin; Warsaw
Salary
Salary:
100.00 - 160.00 PLN / Hour
spyro-soft.com Logo
Spyrosoft
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Excellent knowledge of PySpark / Python
  • Great knowledge of ETL/ELT processes
  • Experience with working with data lake systems (preferably Palantir Foundry) for data ingestions
  • Practice with creating documentation on the Confluence platform
  • Ability to use ticketing systems such as JIRA and/or Azure DevOps
  • Familiarity with Snowflake infrastructure as an advance
  • Ability to work in an agile BI team (DevOps) and to share skills and experience
  • Fluency in English
Job Responsibility
Job Responsibility
  • You will play a key role in migrating Building ETL/ELT processes in the Client’s Palantir Foundry infrastructure under the Data Sphere Program, establishing Foundry as the primary Data Lake platform for the Healthcare Commercial
  • Fulltime
Read More
Arrow Right

Data Engineer - Pyspark

This is a data engineer position - a programmer responsible for the design, deve...
Location
Location
India , Chennai
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5-8 Years of experience in working in data eco systems
  • 4-5 years of hands-on experience in Hadoop, Scala, Java, Spark, Hive, Kafka, Impala, Unix Scripting and other Big data frameworks
  • 3+ years of experience with relational SQL and NoSQL databases: Oracle, MongoDB, HBase
  • Strong proficiency in Python and Spark Java with knowledge of core spark concepts (RDDs, Dataframes, Spark Streaming, etc) and Scala and SQL
  • Data Integration, Migration & Large Scale ETL experience (Common ETL platforms such as PySpark/DataStage/AbInitio etc.) - ETL design & build, handling, reconciliation and normalization
  • Data Modeling experience (OLAP, OLTP, Logical/Physical Modeling, Normalization, knowledge on performance tuning)
  • Experienced in working with large and multiple datasets and data warehouses
  • Experience building and optimizing ‘big data’ data pipelines, architectures, and datasets
  • Strong analytic skills and experience working with unstructured datasets
  • Ability to effectively use complex analytical, interpretive, and problem-solving techniques
Job Responsibility
Job Responsibility
  • Ensuring high quality software development, with complete documentation and traceability
  • Develop and optimize scalable Spark Java-based data pipelines for processing and analyzing large scale financial data
  • Design and implement distributed computing solutions for risk modeling, pricing and regulatory compliance
  • Ensure efficient data storage and retrieval using Big Data
  • Implement best practices for spark performance tuning including partition, caching and memory management
  • Maintain high code quality through testing, CI/CD pipelines and version control (Git, Jenkins)
  • Work on batch processing frameworks for Market risk analytics
  • Promoting unit/functional testing and code inspection processes
  • Work with business stakeholders and Business Analysts to understand the requirements
  • Work with other data scientists to understand and interpret complex datasets
  • Fulltime
Read More
Arrow Right

Data Engineer - PySpark

You will be responsible for supporting the successful delivery of Location Strat...
Location
Location
India , Bengaluru
Salary
Salary:
Not provided
barclays.co.uk Logo
Barclays
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Hands on experience in pyspark and strong knowledge on Dataframes, RDD and SparkSQL
  • Hands on Experience in developing, testing and maintaining applications on AWS Cloud
  • Strong hold on AWS Data Analytics Technology Stack (Glue, S3, Lambda, Lake formation, Athena)
  • Design and implement scalable and efficient data transformation/storage solutions using Snowflake
  • Experience in Data ingestion to Snowflake for different storage format such Parquet, Iceberg, JSON, CSV etc
  • Experience in using DBT (Data Build Tool) with snowflake for ELT pipeline development
  • Experience in Writing advanced SQL and PL SQL programs
  • Hands On Experience for building reusable components using Snowflake and AWS Tools/Technology
  • Should have worked at least on two major project implementations
Job Responsibility
Job Responsibility
  • Build and maintenance of data architectures pipelines that enable the transfer and processing of durable, complete and consistent data
  • Design and implementation of data warehoused and data lakes that manage the appropriate data volumes and velocity and adhere to the required security measures
  • Development of processing and analysis algorithms fit for the intended data complexity and volumes
  • Collaboration with data scientist to build and deploy machine learning models
What we offer
What we offer
  • Competitive holiday allowance
  • Life assurance
  • Private medical care
  • Pension contribution
  • Fulltime
Read More
Arrow Right

Data Engineer - PySpark

Join us as a Data Engineer - PySpark responsible for supporting the successful d...
Location
Location
India , Pune
Salary
Salary:
Not provided
barclays.co.uk Logo
Barclays
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Hands on experience in pyspark and strong knowledge on Dataframes, RDD and SparkSQL
  • Hands on Experience in developing, testing and maintaining applications on AWS Cloud
  • Strong hold on AWS Data Analytics Technology Stack (Glue, S3, Lambda, Lake formation, Athena)
  • Design and implement scalable and efficient data transformation/storage solutions using Snowflake
  • Experience in Data ingestion to Snowflake for different storage format such Parquet, Iceberg, JSON, CSV etc
  • Experience in using DBT (Data Build Tool) with snowflake for ELT pipeline development
  • Experience in Writing advanced SQL and PL SQL programs
  • Hands On Experience for building reusable components using Snowflake and AWS Tools/Technology
  • Should have worked at least on two major project implementations
  • Exposure to data governance or lineage tools such as Immuta and Alation is added advantage
Job Responsibility
Job Responsibility
  • Build and maintenance of data architectures pipelines that enable the transfer and processing of durable, complete and consistent data
  • Design and implementation of data warehoused and data lakes that manage the appropriate data volumes and velocity and adhere to the required security measures
  • Development of processing and analysis algorithms fit for the intended data complexity and volumes
  • Collaboration with data scientist to build and deploy machine learning models
What we offer
What we offer
  • Competitive holiday allowance
  • Life assurance
  • Private medical care
  • Pension contribution
  • Fulltime
Read More
Arrow Right

Pyspark Data Engineer

The Pyspark Data Engineer is responsible for participation in the establishment ...
Location
Location
Canada , Mississauga
Salary
Salary:
79320.00 - 110680.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 2-5 years of relevant experience in the Financial Service industry
  • Intermediate level experience in Applications Development role
  • Consistently demonstrates clear and concise written and verbal communication
  • Demonstrated problem-solving and decision-making skills
  • Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
  • Big Data Infrastructure: Develop and manage large-scale data processing systems using frameworks like Apache Spark, Hadoop, and Kafka
  • Proficiency in Python programming
  • Expertise in data processing frameworks such as Apache Spark, Hadoop
  • Expertise in Data Lakehouse technologies (Apache Iceberg, Trino, Deltalake)
  • Expertise in SQL and database technologies (e.g., Oracle, PostgreSQL, etc.)
Job Responsibility
Job Responsibility
  • Utilize knowledge of applications development procedures and concepts, and basic knowledge of other technical areas to identify and define necessary system enhancements, including using script tools and analyzing/interpreting code
  • Consult with users, clients, and other technology groups on issues, and recommend programming solutions, install, and support customer exposure systems
  • Apply fundamental knowledge of programming languages for design specifications
  • Analyze applications to identify vulnerabilities and security issues, as well as conduct testing and debugging
  • Serve as advisor or coach to new or lower level analysts
  • Identify problems, analyze information, and make evaluative judgements to recommend and implement solutions
  • Resolve issues by identifying and selecting solutions through the applications of acquired technical experience and guided by precedents
  • Has the ability to operate with a limited level of direct supervision
  • Can exercise independence of judgement and autonomy
  • Acts as SME to senior stakeholders and /or other team members
  • Fulltime
Read More
Arrow Right

Senior Data Software Engineer (Python & PySpark) - Vice President

The Senior Data Software Engineer is a senior level position responsible for est...
Location
Location
Singapore , Singapore
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field
  • 7+ years of experience in data engineering, with a strong focus on Python and big data technologies
  • Proven expertise in designing and implementing large-scale data processing solutions using PySpark
  • Extensive experience with distributed computing frameworks like Apache Spark
  • Strong understanding of data warehousing concepts, dimensional modeling, and ETL/ELT principles
  • Proficiency in SQL and experience with various relational and NoSQL databases
  • Experience with cloud platforms (AWS, Azure, GCP) and their data services (e.g., S3, ADLS, Google Cloud Storage, Redshift, Snowflake, BigQuery, Databricks)
  • Familiarity with workflow orchestration tools (e.g., Apache Airflow, Azure Data Factory, AWS Step Functions)
  • Experience with version control systems (e.g., Git)
  • Excellent problem-solving, analytical, and communication skills.
Job Responsibility
Job Responsibility
  • Partner with multiple management teams to ensure appropriate integration of functions to meet goals as well as identify and define necessary system enhancements to deploy new products and process improvements
  • Resolve variety of high impact problems/projects through in-depth evaluation of complex business processes, system processes, and industry standards
  • Provide expertise in area and advanced knowledge of applications programming and ensure application design adheres to the overall architecture blueprint
  • Utilize advanced knowledge of system flow and develop standards for coding, testing, debugging, and implementation
  • Develop comprehensive knowledge of how areas of business, such as architecture and infrastructure, integrate to accomplish business goals
  • Provide in-depth analysis with interpretive thinking to define issues and develop innovative solutions
  • Serve as advisor or coach to mid-level developers and analysts, allocating work as necessary
  • Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency.
  • Fulltime
Read More
Arrow Right