Data Engineer - PySpark Job at Barclays (Bengaluru)

Data Engineer - Pyspark

This is a data engineer position - a programmer responsible for the design, deve...

Location

India , Chennai

Salary:

Not provided

Citi

Expiration Date

Until further notice

Requirements

5-8 Years of experience in working in data eco systems
4-5 years of hands-on experience in Hadoop, Scala, Java, Spark, Hive, Kafka, Impala, Unix Scripting and other Big data frameworks
3+ years of experience with relational SQL and NoSQL databases: Oracle, MongoDB, HBase
Strong proficiency in Python and Spark Java with knowledge of core spark concepts (RDDs, Dataframes, Spark Streaming, etc) and Scala and SQL
Data Integration, Migration & Large Scale ETL experience (Common ETL platforms such as PySpark/DataStage/AbInitio etc.) - ETL design & build, handling, reconciliation and normalization
Data Modeling experience (OLAP, OLTP, Logical/Physical Modeling, Normalization, knowledge on performance tuning)
Experienced in working with large and multiple datasets and data warehouses
Experience building and optimizing ‘big data’ data pipelines, architectures, and datasets
Strong analytic skills and experience working with unstructured datasets
Ability to effectively use complex analytical, interpretive, and problem-solving techniques

Job Responsibility

Ensuring high quality software development, with complete documentation and traceability
Develop and optimize scalable Spark Java-based data pipelines for processing and analyzing large scale financial data
Design and implement distributed computing solutions for risk modeling, pricing and regulatory compliance
Ensure efficient data storage and retrieval using Big Data
Implement best practices for spark performance tuning including partition, caching and memory management
Maintain high code quality through testing, CI/CD pipelines and version control (Git, Jenkins)
Work on batch processing frameworks for Market risk analytics
Promoting unit/functional testing and code inspection processes
Work with business stakeholders and Business Analysts to understand the requirements
Work with other data scientists to understand and interpret complex datasets

Fulltime

Data Engineer - PySpark

Join us as a Data Engineer - PySpark responsible for supporting the successful d...

Location

India , Pune

Salary:

Not provided

Barclays

Expiration Date

Until further notice

Requirements

Hands on experience in pyspark and strong knowledge on Dataframes, RDD and SparkSQL
Hands on Experience in developing, testing and maintaining applications on AWS Cloud
Strong hold on AWS Data Analytics Technology Stack (Glue, S3, Lambda, Lake formation, Athena)
Design and implement scalable and efficient data transformation/storage solutions using Snowflake
Experience in Data ingestion to Snowflake for different storage format such Parquet, Iceberg, JSON, CSV etc
Experience in using DBT (Data Build Tool) with snowflake for ELT pipeline development
Experience in Writing advanced SQL and PL SQL programs
Hands On Experience for building reusable components using Snowflake and AWS Tools/Technology
Should have worked at least on two major project implementations
Exposure to data governance or lineage tools such as Immuta and Alation is added advantage

Job Responsibility

Build and maintenance of data architectures pipelines that enable the transfer and processing of durable, complete and consistent data
Design and implementation of data warehoused and data lakes that manage the appropriate data volumes and velocity and adhere to the required security measures
Development of processing and analysis algorithms fit for the intended data complexity and volumes
Collaboration with data scientist to build and deploy machine learning models

What we offer

Competitive holiday allowance
Life assurance
Private medical care
Pension contribution

Fulltime

Pyspark Data Engineer

The Pyspark Data Engineer is responsible for participation in the establishment ...

Location

Canada , Mississauga

Salary:

79320.00 - 110680.00 USD / Year

Citi

Expiration Date

Until further notice

Requirements

2-5 years of relevant experience in the Financial Service industry
Intermediate level experience in Applications Development role
Consistently demonstrates clear and concise written and verbal communication
Demonstrated problem-solving and decision-making skills
Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
Big Data Infrastructure: Develop and manage large-scale data processing systems using frameworks like Apache Spark, Hadoop, and Kafka
Proficiency in Python programming
Expertise in data processing frameworks such as Apache Spark, Hadoop
Expertise in Data Lakehouse technologies (Apache Iceberg, Trino, Deltalake)
Expertise in SQL and database technologies (e.g., Oracle, PostgreSQL, etc.)

Job Responsibility

Utilize knowledge of applications development procedures and concepts, and basic knowledge of other technical areas to identify and define necessary system enhancements, including using script tools and analyzing/interpreting code
Consult with users, clients, and other technology groups on issues, and recommend programming solutions, install, and support customer exposure systems
Apply fundamental knowledge of programming languages for design specifications
Analyze applications to identify vulnerabilities and security issues, as well as conduct testing and debugging
Serve as advisor or coach to new or lower level analysts
Identify problems, analyze information, and make evaluative judgements to recommend and implement solutions
Resolve issues by identifying and selecting solutions through the applications of acquired technical experience and guided by precedents
Has the ability to operate with a limited level of direct supervision
Can exercise independence of judgement and autonomy
Acts as SME to senior stakeholders and /or other team members

Fulltime

Senior PySpark Data Engineer

We are seeking a highly skilled and experienced Senior PySpark Data Engineer to ...

Location

India , Pune

Salary:

Not provided

Citi

Expiration Date

Until further notice

Requirements

6+ years of professional relevant experience in a data engineering role
Extensive hands-on experience with PySpark and advanced Python programming skills
Proven experience with Big Data ecosystems, including Cloudera and/or DataBricks
Hands-on experience with distributed query engines like Starburst (Trino/Presto)
Proficient in designing and managing complex workflows using scheduling tools, particularly Apache Airflow
Strong expertise in SQL and experience with relational and non-relational databases
Solid understanding of data warehousing concepts, ETL/ELT processes, and data modeling techniques
Experience working in a Linux/Unix environment
GIT HUB, CI/CD Pipeline
Bachelor’s degree/University degree or equivalent experience

Job Responsibility

Design, develop, and maintain robust, scalable, and high-performance data pipelines using PySpark
Develop, schedule, and monitor complex data workflows using orchestration tools like Apache Airflow
Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver high-quality data solutions
Optimize and tune Spark jobs for performance and efficiency
Implement data quality checks and ensure data integrity across all data pipelines
Design and implement data models for optimal storage and retrieval
Mentor junior data engineers and promote best practices in data engineering
Ensure compliance with data governance and security policies
Troubleshoot and resolve data-related issues in a timely manner

Fulltime

Senior Pyspark Data Engineer

The Applications Development Senior Programmer Analyst is an intermediate level ...

Location

Canada , Mississauga

Salary:

94300.00 - 141500.00 USD / Year

Citi

Expiration Date

Until further notice

Requirements

5-8 years of relevant experience
Experience in systems analysis and programming of software applications
Experience in managing and implementing successful projects
Working knowledge of consulting/project management techniques/methods
Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
Big Data Infrastructure: Develop and manage large-scale data processing systems using frameworks like Apache Spark, Hadoop, and Kafka
Proficiency in Python programming
Strong expertise in data processing frameworks such as Apache Spark, Hadoop
Expertise in Data Lakehouse technologies (Apache Iceberg, Trino, Deltalake)
Expertise in SQL and database technologies (e.g., Oracle, PostgreSQL, etc.)

Job Responsibility

Conduct tasks related to feasibility studies, time and cost estimates, IT planning, risk technology, applications development, model development, and establish and implement new or revised applications systems and programs to meet specific business needs or user areas
Monitor and control all phases of development process and analysis, design, construction, testing, and implementation as well as provide user and operational support on applications to business users
Utilize in-depth specialty knowledge of applications development to analyze complex problems/issues, provide evaluation of business process, system process, and industry standards, and make evaluative judgement
Recommend and develop security measures in post implementation analysis of business usage to ensure successful system design and functionality
Consult with users/clients and other technology groups on issues, recommend advanced programming solutions, and install and assist customer exposure systems
Ensure essential procedures are followed and help define operating standards and processes
Serve as advisor or coach to new or lower level analysts
Has the ability to operate with a limited level of direct supervision
Can exercise independence of judgement and autonomy
Acts as SME to senior stakeholders and /or other team members

Fulltime

Senior Pyspark Data Engineer

The Applications Development Senior Programmer Analyst is an intermediate level ...

Location

Canada , Mississauga

Salary:

94300.00 - 141500.00 USD / Year

Citi

Expiration Date

Until further notice

Requirements

5-8 years of relevant experience
Experience in systems analysis and programming of software applications
Experience in managing and implementing successful projects
Working knowledge of consulting/project management techniques/methods
Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
Big Data Infrastructure: Develop and manage large-scale data processing systems using frameworks like Apache Spark, Hadoop, and Kafka
Proficiency in Python programming
Strong expertise in data processing frameworks such as Apache Spark, Hadoop
Expertise in Data Lakehouse technologies (Apache Iceberg, Trino, Deltalake)
Expertise in SQL and database technologies (e.g., Oracle, PostgreSQL, etc.)

Job Responsibility

Conduct tasks related to feasibility studies, time and cost estimates, IT planning, risk technology, applications development, model development, and establish and implement new or revised applications systems and programs to meet specific business needs or user areas
Monitor and control all phases of development process and analysis, design, construction, testing, and implementation as well as provide user and operational support on applications to business users
Utilize in-depth specialty knowledge of applications development to analyze complex problems/issues, provide evaluation of business process, system process, and industry standards, and make evaluative judgement
Recommend and develop security measures in post implementation analysis of business usage to ensure successful system design and functionality
Consult with users/clients and other technology groups on issues, recommend advanced programming solutions, and install and assist customer exposure systems
Ensure essential procedures are followed and help define operating standards and processes
Serve as advisor or coach to new or lower level analysts
Has the ability to operate with a limited level of direct supervision
Can exercise independence of judgement and autonomy
Acts as SME to senior stakeholders and /or other team members

Fulltime

Pyspark Data Engineer

We are seeking a highly motivated and intuitive Python Developer to join our dyn...

Location

India , Chennai

Salary:

Not provided

Citi

Expiration Date

Until further notice

Requirements

4-7 years of relevant experience in the Financial Service industry
Strong Proficiency in Python: Excellent command of Python programming, including object-oriented principles, data structures, and algorithms
PySpark Experience: Demonstrated experience with PySpark for big data processing and analysis
Database Expertise: Proven experience working with relational databases, specifically Oracle, and connecting applications using JDBC
SQL Mastery: Advanced SQL querying skills for complex data extraction, manipulation, and optimization
Big Data Handling: Experience in working with and processing large datasets efficiently
Data Streaming: Familiarity with data streaming concepts and technologies (e.g., Kafka, Spark Streaming) for processing continuous data flows
Data Analysis Libraries: Proficient in using data analysis libraries such as Pandas for data manipulation and exploration
Software Engineering Principles: Solid understanding of software engineering best practices, including version control (Git), testing, and code review
Problem-Solving: Intuitive problem-solver with a self-starter mindset and the ability to work independently and as part of a team

Job Responsibility

Develop, test, and deploy high-quality Python code for data migration, data profiling, and data processing
Design and implement scalable solutions for working with large and complex datasets, ensuring data integrity and performance
Utilize PySpark for distributed data processing and analytics on large-scale data platforms
Develop and optimize SQL queries for various database systems, including Oracle, to extract, transform, and load data efficiently
Integrate Python applications with JDBC-compliant databases (e.g., Oracle) for seamless data interaction
Implement data streaming solutions to process real-time or near real-time data efficiently
Perform in-depth data analysis using Python libraries, especially Pandas, to understand data characteristics, identify anomalies, and support profiling efforts
Collaborate with data architects, data engineers, and business stakeholders to understand requirements and translate them into technical specifications
Contribute to the design and architecture of data solutions, ensuring best practices in data management and engineering
Troubleshoot and resolve technical issues related to data pipelines, performance, and data quality

Fulltime

Data Engineer - AWS, PySpark

You will be responsible for supporting the successful delivery of Location Strat...

Location

India , Bengaluru; Pune

Salary:

Not provided

Barclays

Expiration Date

Until further notice

Requirements

Hands on experience in pyspark and strong knowledge on Dataframes, RDD and SparkSQL
Hands on Experience in developing, testing and maintaining applications on AWS Cloud
Strong hold on AWS Data Analytics Technology Stack (Glue, S3, Lambda, Lake formation, Athena)
Design and implement scalable and efficient data transformation/storage solutions using Snowflake
Experience in Data ingestion to Snowflake for different storage format such Parquet, Iceberg, JSON, CSV etc
Experience in using DBT (Data Build Tool) with snowflake for ELT pipeline development
Experience in Writing advanced SQL and PL SQL programs
Hands On Experience for building reusable components using Snowflake and AWS Tools/Technology
Should have worked at least on two major project implementations
Exposure to data governance or lineage tools such as Immuta and Alation is added advantage

Job Responsibility

Build and maintenance of data architectures pipelines that enable the transfer and processing of durable, complete and consistent data
Design and implementation of data warehoused and data lakes that manage the appropriate data volumes and velocity and adhere to the required security measures
Development of processing and analysis algorithms fit for the intended data complexity and volumes
Collaboration with data scientist to build and deploy machine learning models

What we offer

Competitive holiday allowance
Life assurance
Private medical care
Pension contribution

Fulltime

Select Country

Data Engineer - PySpark

Job Description

Job Responsibility

Requirements

Nice to have

What we offer

Looking for more opportunities?

Data Engineer - PySpark

Data Engineer - Pyspark

Data Engineer - PySpark

Pyspark Data Engineer

Senior PySpark Data Engineer

Senior Pyspark Data Engineer

Senior Pyspark Data Engineer

Pyspark Data Engineer

Data Engineer - AWS, PySpark

Our AI answers in your language