Test Data Architect, Citi

Citi

Location:
India, Chennai

Category:
IT - Software Development

Contract Type:
Employment contract

Salary:

Not provided

Save Job

Apply Position

Job Description:

This job posting is for a senior-level manager who specializes in automating the movement and transformation of data (ETL) within a banking environment. The role focuses on developing robust PySpark-based data processing solutions, building testing frameworks for ETL jobs, and leveraging containerization and orchestration platforms for scalable workloads. Key responsibilities include building ETL pipelines, managing large datasets, cloud integration, and team management.

Job Responsibility:

Build Data Pipelines: Create programs using PySpark to extract data, clean and transform it, and load it into target systems
Testing and Validation: Develop automated tests to ensure the data pipelines are working correctly and the data is accurate
Containerization and Orchestration: Package these data pipelines into containers using Docker and manage their execution using orchestration tools like AWS EKS
Cloud Integration: Work with various cloud services for data storage, processing, and scheduling
Test Data Management - Oversee test data strategies and environment simulations for scalable, reliable automation
Build and maintain ETL validation and testing scripts that run on Red Hat OpenShift containers
Work with Hive, HDFS, and Oracle data sources to extract, transform, and load large-scale datasets
Develop Dockerfiles and create container images for PySpark jobs
Deploy and orchestrate ETL jobs using AWS EKS and integrate them into workflows
Leverage AWS services such as S3, Lambda, and Airflow for data ingestion, event-driven processing, and orchestration
Design and develop PySpark-based ETL pipelines on Cloudera Hadoop platform
Create reusable frameworks, libraries, and templates to accelerate automation and testing of ETL jobs
Participate in code reviews, CI/CD pipelines, and maintain best practices in Spark and cloud-native development
Ensures tooling can be run in CICD providing real-time on-demand test execution shortening the feedback loop to fully support Handsfree execution
Regression, Integration, Sanity testing – provide solutions and ensures timely completion
Team Management: Lead a team of automation professionals
Strategy and Planning: Define the overall strategy for automating data processes and testing
Tooling and Innovation: Research and implement new automation tools and techniques, including AI and machine learning/low-code solutions
Collaboration: Work closely with other teams and partners to ensure smooth data operations and meet regulatory requirements
Reporting and Metrics: Track key performance indicators related to automation and report on progress to leadership.

Requirements:

12-15 years of experience on automation testing across UI, Data analytics and BI reports in the Financial Service industry especially with knowledge of regulatory compliance and risk management
Extensive knowledge on developing and maintaining automation frameworks, AI/ML related solutions
Detailed knowledge data flows in relational database and Bigdata
Selenium BDD Cucumber using Java, Python
Strong experience with PySpark for batch and stream processing deploying PySpark workloads to AWS EKS
Proficiency in working on Cloudera Hadoop ecosystem (HDFS, Hive, YARN)
Hands-on experience with ETL automation and validation framework
Strong knowledge of Oracle SQL and HiveQL
Familiarity with Red Hat OpenShift for container-based deployments
Proficient in creating Dockerfiles and managing container lifecycle
Solid understanding of AWS services like S3, Lambda, EKS, Airflow, and IAM
Experience with Airflow DAGs to orchestrate ETL jobs
Familiarity with CI/CD tools (e.g., Jenkins, GitLab CI)
Scripting knowledge in Bash, Python, and YAML
Version Control: GIT, Bitbucket, GitHub
Experience on automating BI reports e.g., Tableau dashboards and views validation
Hands-on experience in Python for developing utilities for Data Analysis using Pandas, NumPy etc.
Experience with mobile testing using perfecto, API Testing-SoapUI, Postman/Rest Assured, SAS Tools will be added advantage
Strong problem-solving and debugging skills
Excellent communication and collaboration abilities to lead and mentor a large techno-functional team across different geographical locations
Strong Acumen and great presentation skills
Able to work in an Agile environment and deliver results independently.

Nice to have:

Experience with mobile testing using perfecto, API Testing-SoapUI, Postman/Rest Assured, SAS Tools
Strong Acumen and great presentation skills.

Additional Information:

Job Posted:
June 18, 2025

Employment Type:

Fulltime

Work Type:

On-site work

View All Jobs In This Company

Job Link Share:

Test Data Architect