Big Data ETL Developer Job at Realign (Mississauga)

Big Data Developer

The Applications Development Intermediate Programmer Analyst is an intermediate ...

Location

India , Pune

Salary:

Not provided

Citi

Expiration Date

Until further notice

Requirements

Minimum of 5 years of overall IT experience, with at least 3 years of hands-on experience in Big Data technologies
Proven experience working with large-scale, high-volume datasets in distributed environments
Strong proficiency in Hadoop ecosystem tools, including: HDFS (Hadoop Distributed File System) for data storage
Hive for data querying and warehousing
Sqoop for data ingestion from relational databases
Advanced knowledge of Apache Spark, including: Spark Core, Spark SQL, and Spark Streaming (preferred)
Performance tuning and optimization techniques (e.g., partitioning, caching, memory management)
Solid programming skills in Python and PySpark for data processing and pipeline development
Strong command of SQL for complex queries, data transformations, and performance tuning
Hands-on experience in data sourcing, ingestion, and extraction from multiple structured and unstructured data sources

Job Responsibility

Participation in the establishment and implementation of new or revised application systems and programs in coordination with the Technology team
Contribute to applications systems analysis and programming activities
Design, develop, and optimize large-scale data processing systems
Work closely with cross-functional teams to build efficient data pipelines, perform data analysis, and support business-critical financial solutions

Fulltime

Big Data Developer

The Big Data Developer is a senior level position responsible for establishing a...

Location

Canada , Mississauga

Salary:

120800.00 - 170800.00 USD / Year

Citi

Expiration Date

Until further notice

Requirements

6+ years of relevant experience in Big Data/Application Development or systems analysis roles, including building and operating production-grade data pipelines on Hadoop/Spark
Extensive experience in system analysis and in programming of big data applications and data platforms
Proven experience designing and managing Hadoop-based architectures, including cluster configuration, resource management (YARN), and ecosystem integration
Strong understanding and hands-on expertise with the Hadoop ecosystem: HDFS, YARN, MapReduce, Hive, HBase, and Spark
Strong hands-on and architectural knowledge of Python, PySpark, Unix/Linux, and SQL
Experience with data modeling, ETL processes, and data warehousing concepts and implementation
Experience implementing data security and governance (e.g., RBAC, encryption, data quality, data lineage, catalog)
Exposure to AI/ML lifecycle management, MLOps, and GenAI solution patterns and integration points
Experience with major cloud platforms—AWS, Azure, Google Cloud—and related big data services (e.g., EMR, HDInsight, Dataproc, Databricks)
Subject Matter Expert (SME) in at least one area of Big Data/Application Development (e.g., Spark performance tuning, Hive optimization, HBase administration, data security)

Job Responsibility

Partner with multiple management teams to ensure appropriate integration of functions to meet goals, and to identify and define necessary platform and system enhancements to deploy new data products and process improvements
Design and implement scalable and efficient Hadoop architecture solutions encompassing core ecosystem components, including HDFS, YARN, MapReduce, Hive, HBase, and Spark
Collaborate with data engineers, data scientists, and analytics stakeholders to understand data requirements and deliver robust, reliable pipelines and analytical datasets
Develop Spark/PySpark solutions to support near real-time data ingestion, analytics, and reporting, ensuring high performance and reliability
Optimize Hadoop and Spark clusters for performance and resource utilization, including capacity planning, tuning, and job orchestration best practices
Maintain and monitor Hadoop infrastructure to ensure high availability, reliability, and observability
implement proactive alerting, logging, and issue resolution
Implement and enforce data security and governance policies (e.g., access controls, encryption, data quality, lineage, and cataloging) across big data platforms
Troubleshoot and resolve issues across the Hadoop ecosystem (jobs, services, resource management), driving root-cause analysis and permanent fixes
Provide expertise in the area and advanced knowledge of applications programming, ensuring application and data solution design adheres to the overall architecture blueprint and cloud reference patterns

Fulltime

Lead Big Data Developer

We are looking for an experienced Lead Big Data Developer with strong expertise ...

Location

India , Haveli

Salary:

Not provided

Wissen

Expiration Date

Until further notice

Requirements

Bachelor’s degree in computer science, Information Technology, or a related field
7 – 9 years of experience in data engineering, with a focus on big data technologies
Strong experience with AWS services, particularly EMR, S3, Redshift, Lambda, and Glue
Proficiency in programming languages Java
Experience with big data frameworks and tools such as Hadoop, Spark, Hive, and Pig
Solid understanding of data modelling, ETL processes, and data warehousing concepts
Experience with SQL and NoSQL databases
Familiarity with CI/CD pipelines and version control systems (e.g., Git)
Strong problem-solving skills and the ability to work independently and collaboratively in a team environment

Job Responsibility

Design, develop, and maintain data pipelines on AWS EMR (Elastic MapReduce) to support data processing and analytics
Implement data ingestion processes from various sources including APIs, databases, and flat files
Optimize and tune big data workflows for performance and scalability
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions
Manage and monitor EMR clusters, ensuring high availability and reliability
Develop ETL (Extract, Transform, Load) processes to cleanse, transform, and store data in data lakes and data warehouses
Implement data security best practices to ensure data is protected and compliant with relevant regulations
Create and maintain technical documentation related to data pipelines, workflows, and infrastructure
Troubleshoot and resolve issues related to data processing and EMR cluster performance

Fulltime

Senior Big Data Developer

The Applications Development Senior Programmer Analyst is an intermediate level ...

Location

India , Pune

Salary:

Not provided

Citi

Expiration Date

Until further notice

Requirements

8-14 years of relevant experience
Experience in systems analysis and programming of software applications
Experience in managing and implementing successful projects
Working knowledge of consulting/project management techniques/methods
Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
Bachelor’s degree/University degree or equivalent experience
Strong Object-Oriented Programming (OOP) concepts
proficient in Python (specifically for PySpark)
Extensive experience with Apache Spark (PySpark), Hadoop, and related components like Hive and Sqoop
skilled in writing shell scripts

Job Responsibility

Conduct tasks related to feasibility studies, time and cost estimates, IT planning, risk technology, applications development, model development, and establish and implement new or revised applications systems and programs to meet specific business needs or user areas
Monitor and control all phases of development process and analysis, design, construction, testing, and implementation as well as provide user and operational support on applications to business users
Utilize in-depth specialty knowledge of applications development to analyze complex problems/issues, provide evaluation of business process, system process, and industry standards, and make evaluative judgement
Recommend and develop security measures in post implementation analysis of business usage to ensure successful system design and functionality
Consult with users/clients and other technology groups on issues, recommend advanced programming solutions, and install and assist customer exposure systems
Ensure essential procedures are followed and help define operating standards and processes
Serve as advisor or coach to new or lower level analysts
Has the ability to operate with a limited level of direct supervision
Can exercise independence of judgement and autonomy
Acts as SME to senior stakeholders and /or other team members

Fulltime

Senior Python Big Data Developer

The Applications Development Senior Programmer Analyst is an intermediate level ...

Location

India , Pune

Salary:

Not provided

Citi

Expiration Date

Until further notice

Requirements

7 - 12 years of relevant experience
Experience in systems analysis and programming of software applications
Experience in managing and implementing successful projects
Working knowledge of consulting/project management techniques/methods
Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
Bachelor’s degree/University degree or equivalent experience
Strong expertise in Big Data technologies (Spark, Hadoop, Hive, Impala, Kafka, Scala, Cloudera)
Design, develop, and maintain robust and scalable data pipelines using Python, SQL, PySpark, and streaming technologies like Kafka
Strong SQL and NoSQL experience (Oracle, MongoDB, PostgreSQL) for data extraction, reconciliation, and transformation
Proficiency in Python and Shell scripting for data processing and automation

Job Responsibility

Conduct tasks related to feasibility studies, time and cost estimates, IT planning, risk technology, applications development, model development, and establish and implement new or revised applications systems and programs to meet specific business needs or user areas
Monitor and control all phases of development process and analysis, design, construction, testing, and implementation as well as provide user and operational support on applications to business users
Utilize in-depth specialty knowledge of applications development to analyze complex problems/issues, provide evaluation of business process, system process, and industry standards, and make evaluative judgement
Recommend and develop security measures in post implementation analysis of business usage to ensure successful system design and functionality
Consult with users/clients and other technology groups on issues, recommend advanced programming solutions, and install and assist customer exposure systems
Ensure essential procedures are followed and help define operating standards and processes
Serve as advisor or coach to new or lower level analysts
Has the ability to operate with a limited level of direct supervision
Can exercise independence of judgement and autonomy
Acts as SME to senior stakeholders and /or other team members

Fulltime

Fullstack Big Data Developer Application Development Technical Lead Analyst Vice President

Discover your future at Citi. Working at Citi is far more than just a job. A car...

Location

Canada , Mississauga

Salary:

120800.00 - 170800.00 USD / Year

Citi

Expiration Date

Until further notice

Requirements

6+ years of Application development experience
6+ years of experience in full stack development, with a focus on Bigdata and Python/Scala
6+ years experience with big data technologies such as Python, Pyspark, Hadoop, Kafka, etc.
Experience with Core Java/J2EE Application with complete command over OOPs and Design Patterns
Commendable in Data Structures and Algorithms
Worked on Core Application Development of complex size encompassing all areas of Java/J2EE
Thorough knowledge and hands on experience in following technologies Hadoop, Map Reduce Framework, Spark, YARN, Sqoop, Pig , Hue, Unix, Java, Sqoop, Impala, Cassandra on Mesos
Should have implemented or part complex project execution in Big Data Spark eco system, where processing volumes of data thorough understanding of distributed processing and integrated applications
Exposure to ETL and BI tools
Work in an agile environment following through the best practices of agile Scrum

Job Responsibility

Partner with multiple management teams to ensure appropriate integration of functions to meet goals as well as identify and define necessary system enhancements to deploy new products and process improvements
Resolve variety of high impact problems/projects through in-depth evaluation of complex business processes, system processes, and industry standards
Provide expertise in area and advanced knowledge of applications programming and ensure application design adheres to the overall architecture blueprint
Utilize advanced knowledge of system flow and develop standards for coding, testing, debugging, and implementation
Develop comprehensive knowledge of how areas of business, such as architecture and infrastructure, integrate to accomplish business goals
Provide in-depth analysis with interpretive thinking to define issues and develop innovative solutions
Serve as advisor or coach to mid-level developers and analysts, allocating work as necessary
Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency
Design, develop, and maintain scalable and robust architecture for the project using Java/Python/Scala and other full stack technologies
Manage big data technologies such as python, pyspark to ensure seamless data integration, storage, and analysis

Fulltime

Data Engineer (Big Data, Cloud - AWS, Databricks) - Assistant Vice President

Location

India , Pune

Salary:

Not provided

Citi

Expiration Date

Until further notice

Requirements

Scala, Spark/Pyspark is must, Hadoop ( BIG Data ), + AWS,Databricks
8 to 11 years’ experience implementing data-intensive solutions using agile methodologies
Experience of relational databases and using SQL for data querying, transformation and manipulation
Experience of modelling data for analytical consumers
Ability to automate and streamline the build, test and deployment of data pipelines
Experience in cloud native technologies and patterns
A passion for learning new technologies, and a desire for personal growth, through self-study, formal classes, or on-the-job training
Excellent communication and problem-solving skills
An inclination to mentor
an ability to lead and deliver medium sized components independently

Job Responsibility

Developing and supporting scalable, extensible, and highly available data solutions
Deliver on critical business priorities while ensuring alignment with the wider architectural vision
Identify and help address potential risks in the data supply chain
Follow and contribute to technical standards
Design and develop analytical data models

Fulltime

Data Engineer - Big Data, Python, Databricks

The Applications Development Intermediate Programmer Analyst is an intermediate ...

Location

India , Chennai

Salary:

Not provided

Citi

Expiration Date

Until further notice

Requirements

4 - 6 years of relevant experience in the Financial Service industry
Intermediate level experience in Applications Development role
Proficient in Big DataTechnologies - Cloudera, Hive, Python, Java/PySpark, Data Bricks
Proficient in Data analysis and data modelling
Good understanding of ETL concepts
Consistently demonstrates clear and concise written and verbal communication
Demonstrated problem-solving and decision-making skills
Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
Bachelor's degree/University degree or equivalent experience

Job Responsibility

Utilize knowledge of applications development procedures and concepts, and basic knowledge of other technical areas to identify and define necessary system enhancements, including using script tools and analyzing/interpreting code
Consult with users, clients, and other technology groups on issues, and recommend programming solutions, install, and support customer exposure systems
Apply fundamental knowledge of programming languages for design specifications
Analyze applications to identify vulnerabilities and security issues, as well as conduct testing and debugging
Serve as advisor or coach to new or lower level analysts
Identify problems, analyze information, and make evaluative judgements to recommend and implement solutions
Resolve issues by identifying and selecting solutions through the applications of acquired technical experience and guided by precedents
Has the ability to operate with a limited level of direct supervision
Can exercise independence of judgement and autonomy
Acts as SME to senior stakeholders and /or other team members

Fulltime

Select Country

Big Data ETL Developer

Job Description

Job Responsibility

Requirements

Nice to have

Looking for more opportunities?