Pyspark Engineer Job at Realign (Irving)

Pyspark Engineer

Job Title: Pyspark Engineer Location: Irving, TX (ONSITE) F2F Interview Full Tim...

Location

United States , Irving

Salary:

135000.00 USD / Year

Realign

Expiration Date

Until further notice

Requirements

PySpark Developer with 5-10 years’ experience in data engineering practice
strong experience in Apache Spark framework including good understanding of core concepts, performance optimization and industry best practices
proficient in PySpark with hands-on coding experience and ability to implement complex business level transformations
collaborate with the stakeholders and analysts to understand data requirement and deliver robust, creative and innovative solutions
familiarity with unit testing, object-oriented programming (OOPS) concepts and interpreting test results
proficient to write complex and efficient SQL queries to extract the business-critical insights from large-scale data
experience with scheduling of the transformation jobs as per business requirement
perform root-cause analysis and troubleshoot errors on data pipelines, evaluating data quality issues, and implementing corrective fixes

Job Responsibility

Designing, developing and maintaining scalable data pipelines
optimizing data workflows
ensuring the integrity and availability of data for business intelligence
onsite role with strong experience in Apache Spark framework
collaborate with the stakeholders and analysts to understand data requirement and deliver robust, creative and innovative solutions
perform root-cause analysis and troubleshoot errors on data pipelines, evaluating data quality issues, and implementing corrective fixes

Fulltime

Senior Python Pyspark Engineer

The Applications Development Senior Programmer Analyst is an intermediate level ...

Location

India , Pune

Salary:

Not provided

Citi

Expiration Date

Until further notice

Requirements

8 - 10 years of relevant experience
Experience in systems analysis and programming of software applications
Experience in managing and implementing successful projects
Working knowledge of consulting/project management techniques/methods
Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
Programming Languages:Python, PySpark
Data Lake Table Format: Apache Iceberg
Data Orchestration:Apache Airflow
Data Visualization: Tableau
Big Data Processing: Apache Spark

Job Responsibility

Conduct tasks related to feasibility studies, time and cost estimates, IT planning, risk technology, applications development, model development, and establish and implement new or revised applications systems and programs to meet specific business needs or user areas
Monitor and control all phases of development process and analysis, design, construction, testing, and implementation as well as provide user and operational support on applications to business users
Utilize in-depth specialty knowledge of applications development to analyze complex problems/issues, provide evaluation of business process, system process, and industry standards, and make evaluative judgement
Recommend and develop security measures in post implementation analysis of business usage to ensure successful system design and functionality
Consult with users/clients and other technology groups on issues, recommend advanced programming solutions, and install and assist customer exposure systems
Ensure essential procedures are followed and help define operating standards and processes
Serve as advisor or coach to new or lower level analysts
Has the ability to operate with a limited level of direct supervision.
Can exercise independence of judgement and autonomy.
Acts as SME to senior stakeholders and /or other team members.

Fulltime

Python PySpark Engineer

Location

Poland , Wroclaw; Bialystok; Cracow; Gdansk; Lodz; Szczecin; Warsaw

Salary:

100.00 - 160.00 PLN / Hour

Spyrosoft

Expiration Date

Until further notice

Requirements

Excellent knowledge of PySpark / Python
Great knowledge of ETL/ELT processes
Experience with working with data lake systems (preferably Palantir Foundry) for data ingestions
Practice with creating documentation on the Confluence platform
Ability to use ticketing systems such as JIRA and/or Azure DevOps
Familiarity with Snowflake infrastructure as an advance
Ability to work in an agile BI team (DevOps) and to share skills and experience
Fluency in English

Job Responsibility

You will play a key role in migrating Building ETL/ELT processes in the Client’s Palantir Foundry infrastructure under the Data Sphere Program, establishing Foundry as the primary Data Lake platform for the Healthcare Commercial

Fulltime

Data Engineer - Pyspark

This is a data engineer position - a programmer responsible for the design, deve...

Location

India , Chennai

Salary:

Not provided

Citi

Expiration Date

Until further notice

Requirements

5-8 Years of experience in working in data eco systems
4-5 years of hands-on experience in Hadoop, Scala, Java, Spark, Hive, Kafka, Impala, Unix Scripting and other Big data frameworks
3+ years of experience with relational SQL and NoSQL databases: Oracle, MongoDB, HBase
Strong proficiency in Python and Spark Java with knowledge of core spark concepts (RDDs, Dataframes, Spark Streaming, etc) and Scala and SQL
Data Integration, Migration & Large Scale ETL experience (Common ETL platforms such as PySpark/DataStage/AbInitio etc.) - ETL design & build, handling, reconciliation and normalization
Data Modeling experience (OLAP, OLTP, Logical/Physical Modeling, Normalization, knowledge on performance tuning)
Experienced in working with large and multiple datasets and data warehouses
Experience building and optimizing ‘big data’ data pipelines, architectures, and datasets
Strong analytic skills and experience working with unstructured datasets
Ability to effectively use complex analytical, interpretive, and problem-solving techniques

Job Responsibility

Ensuring high quality software development, with complete documentation and traceability
Develop and optimize scalable Spark Java-based data pipelines for processing and analyzing large scale financial data
Design and implement distributed computing solutions for risk modeling, pricing and regulatory compliance
Ensure efficient data storage and retrieval using Big Data
Implement best practices for spark performance tuning including partition, caching and memory management
Maintain high code quality through testing, CI/CD pipelines and version control (Git, Jenkins)
Work on batch processing frameworks for Market risk analytics
Promoting unit/functional testing and code inspection processes
Work with business stakeholders and Business Analysts to understand the requirements
Work with other data scientists to understand and interpret complex datasets

Fulltime

Data Engineer - PySpark

You will be responsible for supporting the successful delivery of Location Strat...

Location

India , Bengaluru

Salary:

Not provided

Barclays

Expiration Date

Until further notice

Requirements

Hands on experience in pyspark and strong knowledge on Dataframes, RDD and SparkSQL
Hands on Experience in developing, testing and maintaining applications on AWS Cloud
Strong hold on AWS Data Analytics Technology Stack (Glue, S3, Lambda, Lake formation, Athena)
Design and implement scalable and efficient data transformation/storage solutions using Snowflake
Experience in Data ingestion to Snowflake for different storage format such Parquet, Iceberg, JSON, CSV etc
Experience in using DBT (Data Build Tool) with snowflake for ELT pipeline development
Experience in Writing advanced SQL and PL SQL programs
Hands On Experience for building reusable components using Snowflake and AWS Tools/Technology
Should have worked at least on two major project implementations

Job Responsibility

Build and maintenance of data architectures pipelines that enable the transfer and processing of durable, complete and consistent data
Design and implementation of data warehoused and data lakes that manage the appropriate data volumes and velocity and adhere to the required security measures
Development of processing and analysis algorithms fit for the intended data complexity and volumes
Collaboration with data scientist to build and deploy machine learning models

What we offer

Competitive holiday allowance
Life assurance
Private medical care
Pension contribution

Fulltime

Data Engineer - PySpark

Join us as a Data Engineer - PySpark responsible for supporting the successful d...

Location

India , Pune

Salary:

Not provided

Barclays

Expiration Date

Until further notice

Requirements

Hands on experience in pyspark and strong knowledge on Dataframes, RDD and SparkSQL
Hands on Experience in developing, testing and maintaining applications on AWS Cloud
Strong hold on AWS Data Analytics Technology Stack (Glue, S3, Lambda, Lake formation, Athena)
Design and implement scalable and efficient data transformation/storage solutions using Snowflake
Experience in Data ingestion to Snowflake for different storage format such Parquet, Iceberg, JSON, CSV etc
Experience in using DBT (Data Build Tool) with snowflake for ELT pipeline development
Experience in Writing advanced SQL and PL SQL programs
Hands On Experience for building reusable components using Snowflake and AWS Tools/Technology
Should have worked at least on two major project implementations
Exposure to data governance or lineage tools such as Immuta and Alation is added advantage

Job Responsibility

Build and maintenance of data architectures pipelines that enable the transfer and processing of durable, complete and consistent data
Design and implementation of data warehoused and data lakes that manage the appropriate data volumes and velocity and adhere to the required security measures
Development of processing and analysis algorithms fit for the intended data complexity and volumes
Collaboration with data scientist to build and deploy machine learning models

What we offer

Competitive holiday allowance
Life assurance
Private medical care
Pension contribution

Fulltime

New

Data Engineer (AWS & PySpark)

We are looking for a hands-on Data Engineer with strong expertise in AWS data se...

Location

India , Bangalore South

Salary:

Not provided

Wissen

Expiration Date

Until further notice

Requirements

Strong Python programming skills
Hands-on PySpark development experience
AWS EMR
AWS Athena
AWS Glue
SQL
Data Warehousing Concepts
ETL Development
Data Pipeline Design

Job Responsibility

Design, develop, and maintain scalable data pipelines using PySpark and AWS services
Build ETL workflows using AWS Glue and EMR
Develop data ingestion, transformation, and processing frameworks
Optimize large-scale data processing jobs and improve performance
Write efficient SQL queries for analytics and reporting requirements
Work with structured and semi-structured datasets in cloud environments
Collaborate with data analysts, architects, and business stakeholders
Ensure data quality, reliability, and operational excellence

Fulltime

Pyspark Data Engineer

The Pyspark Data Engineer is responsible for participation in the establishment ...

Location

Canada , Mississauga

Salary:

79320.00 - 110680.00 USD / Year

Citi

Expiration Date

Until further notice

Requirements

2-5 years of relevant experience in the Financial Service industry
Intermediate level experience in Applications Development role
Consistently demonstrates clear and concise written and verbal communication
Demonstrated problem-solving and decision-making skills
Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
Big Data Infrastructure: Develop and manage large-scale data processing systems using frameworks like Apache Spark, Hadoop, and Kafka
Proficiency in Python programming
Expertise in data processing frameworks such as Apache Spark, Hadoop
Expertise in Data Lakehouse technologies (Apache Iceberg, Trino, Deltalake)
Expertise in SQL and database technologies (e.g., Oracle, PostgreSQL, etc.)

Job Responsibility

Utilize knowledge of applications development procedures and concepts, and basic knowledge of other technical areas to identify and define necessary system enhancements, including using script tools and analyzing/interpreting code
Consult with users, clients, and other technology groups on issues, and recommend programming solutions, install, and support customer exposure systems
Apply fundamental knowledge of programming languages for design specifications
Analyze applications to identify vulnerabilities and security issues, as well as conduct testing and debugging
Serve as advisor or coach to new or lower level analysts
Identify problems, analyze information, and make evaluative judgements to recommend and implement solutions
Resolve issues by identifying and selecting solutions through the applications of acquired technical experience and guided by precedents
Has the ability to operate with a limited level of direct supervision
Can exercise independence of judgement and autonomy
Acts as SME to senior stakeholders and /or other team members

Fulltime

Select Country

Pyspark Engineer

Job Description

Job Responsibility

Requirements

Looking for more opportunities?

Pyspark Engineer

Pyspark Engineer

Senior Python Pyspark Engineer

Python PySpark Engineer

Data Engineer - Pyspark

Data Engineer - PySpark

Data Engineer - PySpark

Data Engineer (AWS & PySpark)

Pyspark Data Engineer

Our AI answers in your language