CrawlJobs Logo

Hadoop PySpark, Python, Apache Kafka

realign-llc.com Logo

Realign

Location Icon

Location:
United States , Charlotte ▼

Category Icon
Category:
IT - Software Development

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

160000.00 USD / Year
Save Job
Save Icon
Apply Position

Job Responsibility:

  • Architectural Leadership: Define end-to-end architecture for data platforms, streaming systems, and web applications
  • Ensure alignment with enterprise standards, security, and compliance requirements
  • Evaluate emerging technologies and recommend adoption strategies
  • Data Engineering: Design and implement data ingestion, transformation, and processing pipelines using Hadoop, PySpark, and related tools
  • Optimize ETL workflows for large-scale datasets and real-time streaming
  • Integrate Apache Kafka for event-driven architectures and messaging
  • Application Development: Build and maintain backend services using Python and microservices architecture
  • Develop responsive, dynamic front-end applications using Angular
  • Implement RESTful APIs and ensure seamless integration between components
  • Collaboration & Leadership: Work closely with product owners, business analysts, and DevOps teams
  • Mentor junior developers and data engineers
  • Participate in agile ceremonies, code reviews, and design discussions

Requirements:

  • Primary Skill: Hadoop ecosystem (HDFS, Hive, Spark), PySpark, Python, Apache Kafka
  • Secondary: UI – Angular
  • Experience: Minimum 9 years
  • Technical Expertise: Strong experience with Hadoop ecosystem (HDFS, Hive, Spark)
  • Proficiency in PySpark for distributed data processing
  • Advanced programming skills in Python
  • Hands-on experience with Apache Kafka for real-time streaming
  • Frontend development using Angular (TypeScript, HTML, CSS)
  • Architectural Skills: Expertise in designing scalable, secure, and high-performance systems
  • Familiarity with microservices, API design, and cloud-native architectures
  • Additional Skills: Knowledge of CI/CD pipelines, containerization (Docker/Kubernetes)
  • Exposure to cloud platforms (AWS, Azure, GCP)
  • Education: Bachelor’s or Master’s degree in Computer Science, Engineering, or related field
  • Experience: 9+ years in software development, with at least 4 + years in architecture and Big Data technologies
  • Soft Skills: Strong analytical and problem-solving abilities
  • Excellent communication and leadership skills
  • Ability to thrive in a fast-paced, agile environment

Nice to have:

  • BFSI domain experience or large-scale enterprise systems
  • Understanding of data governance, security, and compliance standards

Additional Information:

Job Posted:
March 21, 2026

Employment Type:
Fulltime
Icon
Realign - All Job Offers
Job Link Share:
Copy Link Icon
Facebook Icon Twitter Icon LinkedIn Icon

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Hadoop PySpark, Python, Apache Kafka

Senior Big Data Engineer

The Big Data Engineer is a senior level position responsible for establishing an...
Location
Location
Canada , Mississauga
Salary
Salary:
94300.00 - 141500.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ Years of Experience in Big Data Engineering (PySpark)
  • Data Pipeline Development: Design, build, and maintain scalable ETL/ELT pipelines to ingest, transform, and load data from multiple sources
  • Big Data Infrastructure: Develop and manage large-scale data processing systems using frameworks like Apache Spark, Hadoop, and Kafka
  • Proficiency in programming languages like Python, or Scala
  • Strong expertise in data processing frameworks such as Apache Spark, Hadoop
  • Expertise in Data Lakehouse technologies (Apache Iceberg, Apache Hudi, Trino)
  • Experience with cloud data platforms like AWS (Glue, EMR, Redshift), Azure (Synapse), or GCP (BigQuery)
  • Expertise in SQL and database technologies (e.g., Oracle, PostgreSQL, etc.)
  • Experience with data orchestration tools like Apache Airflow or Prefect
  • Familiarity with containerization (Docker, Kubernetes) is a plus
Job Responsibility
Job Responsibility
  • Partner with multiple management teams to ensure appropriate integration of functions to meet goals as well as identify and define necessary system enhancements to deploy new products and process improvements
  • Resolve variety of high impact problems/projects through in-depth evaluation of complex business processes, system processes, and industry standards
  • Provide expertise in area and advanced knowledge of applications programming and ensure application design adheres to the overall architecture blueprint
  • Utilize advanced knowledge of system flow and develop standards for coding, testing, debugging, and implementation
  • Develop comprehensive knowledge of how areas of business, such as architecture and infrastructure, integrate to accomplish business goals
  • Provide in-depth analysis with interpretive thinking to define issues and develop innovative solutions
  • Serve as advisor or coach to mid-level developers and analysts, allocating work as necessary
  • Appropriately assess risk when business decisions are made, demonstrating consideration for the firm's reputation and safeguarding Citigroup, its clients and assets
  • Fulltime
Read More
Arrow Right

Senior Big Data Engineer

The Big Data Engineer is a senior level position responsible for establishing an...
Location
Location
Canada , Mississauga
Salary
Salary:
94300.00 - 141500.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ Years of Experience in Big Data Engineering (PySpark)
  • Data Pipeline Development: Design, build, and maintain scalable ETL/ELT pipelines to ingest, transform, and load data from multiple sources
  • Big Data Infrastructure: Develop and manage large-scale data processing systems using frameworks like Apache Spark, Hadoop, and Kafka
  • Proficiency in programming languages like Python, or Scala
  • Strong expertise in data processing frameworks such as Apache Spark, Hadoop
  • Expertise in Data Lakehouse technologies (Apache Iceberg, Apache Hudi, Trino)
  • Experience with cloud data platforms like AWS (Glue, EMR, Redshift), Azure (Synapse), or GCP (BigQuery)
  • Expertise in SQL and database technologies (e.g., Oracle, PostgreSQL, etc.)
  • Experience with data orchestration tools like Apache Airflow or Prefect
  • Familiarity with containerization (Docker, Kubernetes) is a plus
Job Responsibility
Job Responsibility
  • Partner with multiple management teams to ensure appropriate integration of functions to meet goals as well as identify and define necessary system enhancements to deploy new products and process improvements
  • Resolve variety of high impact problems/projects through in-depth evaluation of complex business processes, system processes, and industry standards
  • Provide expertise in area and advanced knowledge of applications programming and ensure application design adheres to the overall architecture blueprint
  • Utilize advanced knowledge of system flow and develop standards for coding, testing, debugging, and implementation
  • Develop comprehensive knowledge of how areas of business, such as architecture and infrastructure, integrate to accomplish business goals
  • Provide in-depth analysis with interpretive thinking to define issues and develop innovative solutions
  • Serve as advisor or coach to mid-level developers and analysts, allocating work as necessary
  • Appropriately assess risk when business decisions are made, demonstrating consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency
What we offer
What we offer
  • Well-being support
  • Growth opportunities
  • Work-life balance support
  • Fulltime
Read More
Arrow Right

Hadoop PySpark, Python, Apache Kafka

Architectural Leadership: Define end-to-end architecture for data platforms, str...
Location
Location
United States , Charlotte, NC / New York, NY/ Dallas, TX / Jersey City, NJ
Salary
Salary:
160000.00 USD / Year
realign-llc.com Logo
Realign
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Minimum 9 years experience
  • Strong experience with Hadoop ecosystem (HDFS, Hive, Spark)
  • Proficiency in PySpark for distributed data processing
  • Advanced programming skills in Python
  • Hands-on experience with Apache Kafka for real-time streaming
  • Frontend development using Angular (TypeScript, HTML, CSS)
  • Expertise in designing scalable, secure, and high-performance systems
  • Familiarity with microservices, API design, and cloud-native architectures
  • Knowledge of CI/CD pipelines, containerization (Docker/Kubernetes)
  • Exposure to cloud platforms (AWS, Azure, GCP)
Job Responsibility
Job Responsibility
  • Define end-to-end architecture for data platforms, streaming systems, and web applications
  • Ensure alignment with enterprise standards, security, and compliance requirements
  • Evaluate emerging technologies and recommend adoption strategies
  • Design and implement data ingestion, transformation, and processing pipelines using Hadoop, PySpark, and related tools
  • Optimize ETL workflows for large-scale datasets and real-time streaming
  • Integrate Apache Kafka for event-driven architectures and messaging
  • Build and maintain backend services using Python and microservices architecture
  • Develop responsive, dynamic front-end applications using Angular
  • Implement RESTful APIs and ensure seamless integration between components
  • Work closely with product owners, business analysts, and DevOps teams
  • Fulltime
Read More
Arrow Right

Hadoop PySpark, Python, Apache Kafka

Architectural Leadership, Data Engineering, Application Development, Collaborati...
Location
Location
United States , Charlotte, NC / New York, NY/ Dallas, TX / Jersey City, NJ
Salary
Salary:
160000.00 USD / Year
realign-llc.com Logo
Realign
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Minimum 9 years experience
  • Strong experience with Hadoop ecosystem (HDFS, Hive, Spark)
  • Proficiency in PySpark for distributed data processing
  • Advanced programming skills in Python
  • Hands-on experience with Apache Kafka for real-time streaming
  • Frontend development using Angular (TypeScript, HTML, CSS)
  • Expertise in designing scalable, secure, and high-performance systems
  • Familiarity with microservices, API design, and cloud-native architectures
  • Knowledge of CI/CD pipelines, containerization (Docker/Kubernetes)
  • Exposure to cloud platforms (AWS, Azure, GCP)
Job Responsibility
Job Responsibility
  • Define end-to-end architecture for data platforms, streaming systems, and web applications
  • Ensure alignment with enterprise standards, security, and compliance requirements
  • Evaluate emerging technologies and recommend adoption strategies
  • Design and implement data ingestion, transformation, and processing pipelines using Hadoop, PySpark, and related tools
  • Optimize ETL workflows for large-scale datasets and real-time streaming
  • Integrate Apache Kafka for event-driven architectures and messaging
  • Build and maintain backend services using Python and microservices architecture
  • Develop responsive, dynamic front-end applications using Angular
  • Implement RESTful APIs and ensure seamless integration between components
  • Work closely with product owners, business analysts, and DevOps teams
  • Fulltime
Read More
Arrow Right

Hadoop PySpark, Python, Apache Kafka

Role: Hadoop PySpark, Python, Apache Kafka. FTE only. Architectural Leadership, ...
Location
Location
United States , Charlotte, NC / New York, NY / Dallas, TX / Jersey City, NJ
Salary
Salary:
160000.00 USD / Year
realign-llc.com Logo
Realign
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Minimum 9 years experience in software development
  • Strong experience with Hadoop ecosystem (HDFS, Hive, Spark)
  • Proficiency in PySpark for distributed data processing
  • Advanced programming skills in Python
  • Hands-on experience with Apache Kafka for real-time streaming
  • Frontend development using Angular (TypeScript, HTML, CSS)
  • Expertise in designing scalable, secure, and high-performance systems
  • Familiarity with microservices, API design, and cloud-native architectures
  • Knowledge of CI/CD pipelines, containerization (Docker/Kubernetes)
  • Exposure to cloud platforms (AWS, Azure, GCP)
Job Responsibility
Job Responsibility
  • Define end-to-end architecture for data platforms, streaming systems, and web applications
  • Ensure alignment with enterprise standards, security, and compliance requirements
  • Evaluate emerging technologies and recommend adoption strategies
  • Design and implement data ingestion, transformation, and processing pipelines using Hadoop, PySpark, and related tools
  • Optimize ETL workflows for large-scale datasets and real-time streaming
  • Integrate Apache Kafka for event-driven architectures and messaging
  • Build and maintain backend services using Python and microservices architecture
  • Develop responsive, dynamic front-end applications using Angular
  • Implement RESTful APIs and ensure seamless integration between components
  • Work closely with product owners, business analysts, and DevOps teams
  • Fulltime
Read More
Arrow Right

Big Data Engineering Lead

The Senior Big Data engineering lead will play a pivotal role in designing, impl...
Location
Location
India , Chennai
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's or Master’s degree in Computer Science, Information Technology, or related field
  • Atleast 10 -12 years overall software development experience on majorly working with handling application with large scale data volumes from ingestion, persistence and retrieval
  • Deep understanding of big data technologies, including Hadoop, Spark, Kafka, Flink, NoSQL databases, etc.
  • Experience with Bigdata technologies Developer Hadoop, Apache Spark, Python, PySpark
  • Strong programming skills in languages such as Java, Scala, or Python
  • Excellent problem-solving skills with a knack for innovative solutions
  • Strong communication and leadership abilities
  • Proven ability to manage multiple projects simultaneously and deliver results
Job Responsibility
Job Responsibility
  • Lead the design and development of a robust and scalable big data architecture handling exponential data growth while maintaining high availability and resilience
  • Design complex data transformation processes using Spark and other big data technologies using Java, Pyspark or Scala
  • Design and implement data pipelines that ensure data quality, integrity, and availability
  • Collaborate with cross-functional teams to understand business needs and translate them into technical requirements
  • Evaluate and select technologies that improve data efficiency, scalability, and performance
  • Oversee the deployment and management of big data tools and frameworks such as Hadoop, Spark, Kafka, and others
  • Provide technical guidance and mentorship to the development team and junior architects
  • Continuously assess and integrate emerging technologies and methodologies to enhance data processing capabilities
  • Optimize big data frameworks, such as Hadoop, Spark, for performance improvements and reduced processing time across distributed systems
  • Implement data governance frameworks to ensure data accuracy, consistency, and privacy across the organization, leveraging metadata management and data lineage tracking
  • Fulltime
Read More
Arrow Right
New

Senior Pyspark Data Engineer

The Applications Development Senior Programmer Analyst is an intermediate level ...
Location
Location
Canada , Mississauga
Salary
Salary:
94300.00 - 141500.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5-8 years of relevant experience
  • Experience in systems analysis and programming of software applications
  • Experience in managing and implementing successful projects
  • Working knowledge of consulting/project management techniques/methods
  • Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
  • Big Data Infrastructure: Develop and manage large-scale data processing systems using frameworks like Apache Spark, Hadoop, and Kafka
  • Proficiency in Python programming
  • Strong expertise in data processing frameworks such as Apache Spark, Hadoop
  • Expertise in Data Lakehouse technologies (Apache Iceberg, Trino, Deltalake)
  • Expertise in SQL and database technologies (e.g., Oracle, PostgreSQL, etc.)
Job Responsibility
Job Responsibility
  • Conduct tasks related to feasibility studies, time and cost estimates, IT planning, risk technology, applications development, model development, and establish and implement new or revised applications systems and programs to meet specific business needs or user areas
  • Monitor and control all phases of development process and analysis, design, construction, testing, and implementation as well as provide user and operational support on applications to business users
  • Utilize in-depth specialty knowledge of applications development to analyze complex problems/issues, provide evaluation of business process, system process, and industry standards, and make evaluative judgement
  • Recommend and develop security measures in post implementation analysis of business usage to ensure successful system design and functionality
  • Consult with users/clients and other technology groups on issues, recommend advanced programming solutions, and install and assist customer exposure systems
  • Ensure essential procedures are followed and help define operating standards and processes
  • Serve as advisor or coach to new or lower level analysts
  • Has the ability to operate with a limited level of direct supervision
  • Can exercise independence of judgement and autonomy
  • Acts as SME to senior stakeholders and /or other team members
  • Fulltime
Read More
Arrow Right
New

Senior Pyspark Data Engineer

The Applications Development Senior Programmer Analyst is an intermediate level ...
Location
Location
Canada , Mississauga
Salary
Salary:
94300.00 - 141500.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5-8 years of relevant experience
  • Experience in systems analysis and programming of software applications
  • Experience in managing and implementing successful projects
  • Working knowledge of consulting/project management techniques/methods
  • Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
  • Big Data Infrastructure: Develop and manage large-scale data processing systems using frameworks like Apache Spark, Hadoop, and Kafka
  • Proficiency in Python programming
  • Strong expertise in data processing frameworks such as Apache Spark, Hadoop
  • Expertise in Data Lakehouse technologies (Apache Iceberg, Trino, Deltalake)
  • Expertise in SQL and database technologies (e.g., Oracle, PostgreSQL, etc.)
Job Responsibility
Job Responsibility
  • Conduct tasks related to feasibility studies, time and cost estimates, IT planning, risk technology, applications development, model development, and establish and implement new or revised applications systems and programs to meet specific business needs or user areas
  • Monitor and control all phases of development process and analysis, design, construction, testing, and implementation as well as provide user and operational support on applications to business users
  • Utilize in-depth specialty knowledge of applications development to analyze complex problems/issues, provide evaluation of business process, system process, and industry standards, and make evaluative judgement
  • Recommend and develop security measures in post implementation analysis of business usage to ensure successful system design and functionality
  • Consult with users/clients and other technology groups on issues, recommend advanced programming solutions, and install and assist customer exposure systems
  • Ensure essential procedures are followed and help define operating standards and processes
  • Serve as advisor or coach to new or lower level analysts
  • Has the ability to operate with a limited level of direct supervision
  • Can exercise independence of judgement and autonomy
  • Acts as SME to senior stakeholders and /or other team members
  • Fulltime
Read More
Arrow Right