CrawlJobs Logo

Big Data Lead Developer

Canada, Mississauga Employment contract · Job Posted November 14, 2025
Apply Position
Job Link Share

Job Description

We are seeking a highly skilled and experienced Big Data Lead Developer to establish and implement new or revised application systems and programs, focusing on designing, developing, and maintaining robust big data applications and pipelines, while providing technical leadership and mentorship to engineers.

Job Responsibility

  • Lead and mentor a team of big data engineers, fostering a collaborative and high-performing environment
  • Provide technical guidance, code reviews, and support for professional development
  • Design and implement scalable and robust big data architectures and pipelines to handle large volumes of data from various sources
  • Evaluate and select appropriate big data technologies and tools based on project requirements and industry best practices
  • Implement and integrate these technologies into our existing infrastructure
  • Develop and optimize data processing and analysis workflows using technologies such as Spark, Hadoop, Hive, and other relevant tools
  • Implement data quality checks and ensure adherence to data governance policies and procedures
  • Continuously monitor and optimize the performance of big data systems and pipelines to ensure efficient data processing and retrieval
  • Collaborate effectively with cross-functional teams, including data scientists, business analysts, and product managers, to understand their data needs and deliver impactful solutions
  • Stay up to date with the latest advancements in big data technologies and explore new tools and techniques to improve our data infrastructure

Requirements

  • 6+ years of relevant experience in Big Data application development or systems analysis role
  • Experience in leading and mentoring big data engineering teams
  • Strong understanding of big data concepts, architectures, and technologies (e.g., Hadoop, PySpark, Hive, Kafka, NoSQL databases)
  • Proficiency in programming languages such as Java, Scala, or Python
  • Excellent problem-solving and analytical skills
  • Strong presentation, communication and interpersonal skills
  • Experience with data warehousing and business intelligence tools
  • Experience with data visualization and reporting
  • Knowledge of cloud-based big data platforms (e.g., AWS EMR, Azure HDInsight, Google Cloud Dataproc)
  • Proficiency in Unix/Linux environments
  • Certifications in relevant big data technologies is an advantage
  • Ability to adjust priorities quickly as circumstances dictate
  • Demonstrated leadership and project management skills
  • Consistently demonstrates clear and concise written and verbal communication

Nice to have

  • Certifications in relevant big data technologies
  • Knowledge of cloud-based big data platforms

What we offer

Global benefits designed to support your well-being, growth, and work-life balance

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Big Data Lead Developer

8 matching positions

Lead Big Data Developer

We are looking for an experienced Lead Big Data Developer with strong expertise ...
Location
Location
India , Haveli
Salary
Salary:
Not provided
votredircom.fr Logo
Wissen
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree in computer science, Information Technology, or a related field
  • 7 – 9 years of experience in data engineering, with a focus on big data technologies
  • Strong experience with AWS services, particularly EMR, S3, Redshift, Lambda, and Glue
  • Proficiency in programming languages Java
  • Experience with big data frameworks and tools such as Hadoop, Spark, Hive, and Pig
  • Solid understanding of data modelling, ETL processes, and data warehousing concepts
  • Experience with SQL and NoSQL databases
  • Familiarity with CI/CD pipelines and version control systems (e.g., Git)
  • Strong problem-solving skills and the ability to work independently and collaboratively in a team environment
Job Responsibility
Job Responsibility
  • Design, develop, and maintain data pipelines on AWS EMR (Elastic MapReduce) to support data processing and analytics
  • Implement data ingestion processes from various sources including APIs, databases, and flat files
  • Optimize and tune big data workflows for performance and scalability
  • Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions
  • Manage and monitor EMR clusters, ensuring high availability and reliability
  • Develop ETL (Extract, Transform, Load) processes to cleanse, transform, and store data in data lakes and data warehouses
  • Implement data security best practices to ensure data is protected and compliant with relevant regulations
  • Create and maintain technical documentation related to data pipelines, workflows, and infrastructure
  • Troubleshoot and resolve issues related to data processing and EMR cluster performance
  • Fulltime
Read More
Arrow Right

Fullstack Big Data Developer Application Development Technical Lead Analyst Vice President

Discover your future at Citi. Working at Citi is far more than just a job. A car...
Location
Location
Canada , Mississauga
Salary
Salary:
120800.00 - 170800.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 6+ years of Application development experience
  • 6+ years of experience in full stack development, with a focus on Bigdata and Python/Scala
  • 6+ years experience with big data technologies such as Python, Pyspark, Hadoop, Kafka, etc.
  • Experience with Core Java/J2EE Application with complete command over OOPs and Design Patterns
  • Commendable in Data Structures and Algorithms
  • Worked on Core Application Development of complex size encompassing all areas of Java/J2EE
  • Thorough knowledge and hands on experience in following technologies Hadoop, Map Reduce Framework, Spark, YARN, Sqoop, Pig , Hue, Unix, Java, Sqoop, Impala, Cassandra on Mesos
  • Should have implemented or part complex project execution in Big Data Spark eco system, where processing volumes of data thorough understanding of distributed processing and integrated applications
  • Exposure to ETL and BI tools
  • Work in an agile environment following through the best practices of agile Scrum
Job Responsibility
Job Responsibility
  • Partner with multiple management teams to ensure appropriate integration of functions to meet goals as well as identify and define necessary system enhancements to deploy new products and process improvements
  • Resolve variety of high impact problems/projects through in-depth evaluation of complex business processes, system processes, and industry standards
  • Provide expertise in area and advanced knowledge of applications programming and ensure application design adheres to the overall architecture blueprint
  • Utilize advanced knowledge of system flow and develop standards for coding, testing, debugging, and implementation
  • Develop comprehensive knowledge of how areas of business, such as architecture and infrastructure, integrate to accomplish business goals
  • Provide in-depth analysis with interpretive thinking to define issues and develop innovative solutions
  • Serve as advisor or coach to mid-level developers and analysts, allocating work as necessary
  • Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency
  • Design, develop, and maintain scalable and robust architecture for the project using Java/Python/Scala and other full stack technologies
  • Manage big data technologies such as python, pyspark to ensure seamless data integration, storage, and analysis
  • Fulltime
Read More
Arrow Right

Lead Big Data Spark Engineer

We are seeking an experienced and highly skilled Big Data Engineer to lead the d...
Location
Location
Canada , Mississauga
Salary
Salary:
120800.00 - 170800.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's or Master's degree in Computer Science, Engineering, or a related field
  • 8+ years of progressive experience in software development
  • 5+ years focusing on big data technologies
  • 3+ years of experience in a leadership or senior architectural role
  • Extensive hands-on experience with Scala for big data processing
  • Demonstrated expertise with Apache Spark (Spark Core, Spark SQL, Spark Streaming)
  • Strong experience with distributed systems and big data ecosystems (e.g., Hadoop, Kafka, Cassandra, HBase, Delta Lake, Snowflake, Databricks)
  • Proficiency with cloud platforms (AWS, Azure, GCP) and their big data services
  • Experience with containerization technologies (Docker, Kubernetes) and CI/CD pipelines
  • Solid understanding of data warehousing concepts, ETL/ELT processes, and data modeling
Job Responsibility
Job Responsibility
  • Lead the architecture, design, and development of high-performance, scalable, and reliable big data processing systems using Scala and Apache Spark
  • Drive technical vision and strategy for big data initiatives
  • Evaluate and recommend new technologies and tools
  • Design, develop, and optimize data pipelines for ingestion, transformation, and storage of massive datasets
  • Implement robust and efficient data processing jobs using Scala and Spark (batch and streaming)
  • Ensure data quality, integrity, and security
  • Promote and enforce best practices in coding, testing, and deployment
  • Mentor and guide a team of talented big data engineers
  • Conduct code reviews, provide constructive feedback
  • Participate in the recruitment and hiring
  • Fulltime
Read More
Arrow Right

Lead Java Big Data Engineer Vice President

At Citi, we are at the forefront of financial technology, driven by a belief in ...
Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 10+ years of progressive experience in professional software engineering, with at least 3 years in a technical leadership or architect role
  • Proven track record of designing and building complex, high-performance, scalable server-side applications using Java
  • Deep, hands-on experience with the Big Data ecosystem, including mastery of Apache Spark, Hadoop (HDFS), and real-time data streaming with Kafka
  • Extensive experience with relational databases, data modeling, and data warehousing concepts
  • Demonstrated experience leading and mentoring technical teams and successfully delivering complex, large-scale data projects from concept to production
  • Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field
Job Responsibility
Job Responsibility
  • Define the end-to-end architectural vision and technical roadmap for migrating from Sybase IQ to a modern Big Data platform, ensuring solutions are scalable, resilient, and secure
  • Lead the design, development, and deployment of robust, large-scale data processing pipelines using technologies like Apache Spark, Kafka, and distributed data stores
  • Develop and execute a comprehensive, phased strategy for migrating petabytes of historical and transactional data from Sybase IQ, ensuring data integrity, minimal downtime, and zero business disruption
  • Oversee the design and development of Java-based microservices that interact with the new data platform, ensuring seamless integration with the broader Oasys application ecosystem
  • Lead, inspire, and mentor a high-performing team of Java and Big Data engineers. Foster a culture of engineering excellence, innovation, and accountability
  • Partner with global business leaders, product owners, and other senior technology managers to define requirements, manage expectations, and deliver solutions that drive significant business value
  • Remain deeply technical and contribute to coding, design, and architectural decisions, leading by example
  • Fulltime
Read More
Arrow Right

Pyspark Big Data Senior Developer - Vice President

We are building an A-team of highly skilled and autonomous engineers, and we are...
Location
Location
Canada , Mississauga
Salary
Salary:
120800.00 - 170800.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 6+ years of extensive, hands-on experience as a Senior Big Data Developer, with a strong emphasis on PySpark and the Apache Spark ecosystem, operating as a player/coach
  • Expert proficiency in Python, with a proven track record of developing robust, scalable, and high-performance PySpark applications for large-scale data processing
  • Deep understanding and extensive hands-on experience with Apache Spark (Spark Core, Spark SQL, Spark Streaming) and its ecosystem
  • Experience with distributed computing frameworks such as Hadoop (HDFS, YARN)
  • Expert proficiency in SQL and extensive experience with data warehousing concepts and technologies (e.g., Hive, Snowflake, Redshift, Databricks SQL)
  • Proven experience with various data storage formats (e.g., Parquet, ORC, Avro) and data lake solutions (e.g., Delta Lake, Iceberg)
  • Experience with NoSQL databases (e.g., MongoDB, Cassandra, HBase) is a significant plus
  • Strong experience with Apache Kafka for building real-time data pipelines and event-driven architectures
  • Demonstrated experience with big data services on major cloud platforms (e.g., AWS EMR/Glue/Redshift, Azure Databricks/Data Factory/Synapse, GCP Dataflow/Dataproc/BigQuery) is highly desirable
  • Proven effectiveness with AI coding tools (e.g., Claude Code, Codex, Antigravity) is a mandatory requirement
Job Responsibility
Job Responsibility
  • Operate end-to-end in the design, development, and implementation of robust big data solutions, ensuring optimal performance, scalability, data quality, and security
  • Collaborate closely within small, co-located squads (4-7 person teams), fostering high communication and low coordination overhead, to translate complex business requirements into technical specifications for big data processing and analytical solutions
  • Act as a player/coach within the team, mentoring junior members and leading by example in the development of efficient and innovative big data architectures
  • Design, develop, and optimize large-scale data pipelines using PySpark for data ingestion, transformation, and aggregation, always with an eye towards efficiency and domain relevance
  • Implement and manage real-time data streaming and event-driven architectures using technologies like Apache Kafka
  • Design and implement sophisticated data warehousing solutions and dimensional models for efficient data storage and retrieval, ensuring alignment with business needs
  • Work with various distributed data storage technologies, including distributed file systems (e.g., HDFS, S3) and NoSQL databases (e.g., MongoDB, Cassandra), selecting the right tool for the right problem
  • Implement efficient data processing and storage strategies to optimize the performance and scalability of big data applications, with a strong focus on the 'why' behind the technology choices
  • Champion best practices in software development, including rigorous code reviews, implementing comprehensive testing, and supporting continuous integration and continuous deployment (CI/CD) pipelines
  • Demonstrate high autonomy and agency in driving projects forward, making informed decisions, and proactively identifying areas for improvement
  • Fulltime
Read More
Arrow Right

Big Data Engineering Lead

The Senior Big Data engineering lead will play a pivotal role in designing, impl...
Location
Location
India , Chennai
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's or Master’s degree in Computer Science, Information Technology, or related field
  • Atleast 10 -12 years overall software development experience on majorly working with handling application with large scale data volumes from ingestion, persistence and retrieval
  • Deep understanding of big data technologies, including Hadoop, Spark, Kafka, Flink, NoSQL databases, etc.
  • Experience with Bigdata technologies Developer Hadoop, Apache Spark, Python, PySpark
  • Strong programming skills in languages such as Java, Scala, or Python
  • Excellent problem-solving skills with a knack for innovative solutions
  • Strong communication and leadership abilities
  • Proven ability to manage multiple projects simultaneously and deliver results
Job Responsibility
Job Responsibility
  • Lead the design and development of a robust and scalable big data architecture handling exponential data growth while maintaining high availability and resilience
  • Design complex data transformation processes using Spark and other big data technologies using Java, Pyspark or Scala
  • Design and implement data pipelines that ensure data quality, integrity, and availability
  • Collaborate with cross-functional teams to understand business needs and translate them into technical requirements
  • Evaluate and select technologies that improve data efficiency, scalability, and performance
  • Oversee the deployment and management of big data tools and frameworks such as Hadoop, Spark, Kafka, and others
  • Provide technical guidance and mentorship to the development team and junior architects
  • Continuously assess and integrate emerging technologies and methodologies to enhance data processing capabilities
  • Optimize big data frameworks, such as Hadoop, Spark, for performance improvements and reduced processing time across distributed systems
  • Implement data governance frameworks to ensure data accuracy, consistency, and privacy across the organization, leveraging metadata management and data lineage tracking
  • Fulltime
Read More
Arrow Right

Big Data / PySpark Engineering Lead - Vice President

The Applications Development Technology Lead Analyst is a senior level position ...
Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Highly experienced and skilled technical lead with 12+years of experience with software building and platform engineering
  • Experience in Data Engineering, focused on Big Data ecosystems
  • Knowledge in Hadoop, YARN, Hive, Impala, Spark, and Spark SQL with extensive high volume of data processing pipeline development
  • Programming Expert level and hand on experience in Python
  • Familiarity with data formats like Avro, Parquet, CSV, JSON
  • Hands-on experience in writing SQL queries
  • Highly experienced with Unix based operating systems and shell scripting
  • Experience with source code management tools such as Bitbucket, Git etc
  • Big Data Tech Proficiency and hands-on in Hadoop, Spark, Hive, Kafka, and NoSQL databases (MongoDB, HBase)
  • Experience working with query engines like Trino, Presto, Starburst
Job Responsibility
Job Responsibility
  • Design and implement scalable, fault-tolerant batch and real-time data processing pipelines
  • Develop robust data models and schema designs optimized for both performance and storage efficiency
  • Evaluate and integrate emerging tools and frameworks (e.g., Spark, Flink, Kafka) into the existing stack
  • Provide in-depth analysis with interpretive thinking to define issues and develop innovative solutions
  • Develop comprehensive knowledge of how areas of business, such as architecture and infrastructure, integrate to accomplish business goals
  • Legacy Systems Decommissioning: Lead the strategic migration of data and logic from legacy platforms (e.g. on-premises SQL Servers) to a modern Data Lakehouse environment
  • ETL/ELT Transformation: Re-engineer existing stored procedures and complex legacy ETL jobs into scalable, distributed processing frameworks using Spark (Python) and Starburst/Trino
  • Validation & Parity Testing: Design and implement automated frameworks for Data Parity Testing to ensure 100% accuracy and consistency between legacy outputs and new big data results
  • Schema Evolution: Map and transform rigid, legacy relational schemas into flexible, high-performance formats optimized for the cloud (e.g., Parquet, Avro, or Iceberg)
  • Phased Cutover Management: Orchestrate a phased migration strategy (Parallel Run, Shadow Execution) to ensure zero downtime for downstream business applications and reporting tools
  • Fulltime
Read More
Arrow Right

Lead Technology Analyst - Big Data

Are you passionate about IT, experienced in software development techniques and ...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
airbus.com Logo
Airbus
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Engineering graduate/post-graduate
  • 7 - 9 years of experience in developing, delivering AWS based solutions
  • Full stack developer with capabilities in Backend Java and Frontend technologies
  • Have maintained DevOps pipelines for applications and have worked on Fast APIs
  • Knowledge on some of the Spark & Spark-SQL with Python
  • Experience with writing Javascript, HTML, CSS
  • Knowledge on some of the following products/tools: Git, Confluence, VersionOne etc
  • Proven track record working in Agile Scrum and/or Kanban projects
  • Knowledge of continuous integration frameworks and some of the following products/tools: VersionOne, Klaxoon, etc.
  • Willing to work in CET / IST timezone based on project need
Job Responsibility
Job Responsibility
  • Designing solutions to meet functional and non-functional requirements at scale for business needs
  • Involvement in the full delivery lifecycle – responsible for designing, implementing, testing, documenting, supporting and operating Foundry based applications
  • Implementing new tools and enhancing the existing tools & data products built on top of the foundry platform
  • Refactor and debug pipelines
  • Participate in knowledge sharing activities to further enhance the knowledge base for the project
  • This job requires the constant awareness of the compliance risks we face in day-to-day responsibilities
  • Continuous commitment to act with integrity with each other, with your communities, business partners and suppliers is the foundation of your success and sustainable growth
  • The commitment to integrity is supported by your adherence to all internal policies and procedures that govern business activities
  • Compliance with these policies will also protect Airbus reputation and brand, some of our most strategic and important assets
  • Fulltime
Read More
Arrow Right