Software Development Engineer - Big Data Job at Skyhigh Security (Bangalore)

Big Data / Scala / Python Engineering Lead

The Applications Development Technology Lead Analyst is a senior level position ...

Location

India , Chennai

Salary:

Not provided

Citi

Expiration Date

Until further notice

Requirements

At least two years (Over all 10+ hands on Data Engineering experience) of experience building and leading highly complex, technical data engineering teams
Lead data engineering team, from sourcing to closing
Drive strategic vision for the team and product
Experience managing an data focused product, ML platform
Hands on experience relevant experience in design, develop, and optimize scalable distributed data processing pipelines using Apache Spark and Scala
Experience managing, hiring and coaching software engineering teams
Experience with large-scale distributed web services and the processes around testing, monitoring, and SLAs to ensure high product quality
7 to 10+ years of hands-on experience in big data development, focusing on Apache Spark, Scala, and distributed systems
Proficiency in Functional Programming: High proficiency in Scala-based functional programming for developing robust and efficient data processing pipelines
Proficiency in Big Data Technologies: Strong experience with Apache Spark, Hadoop ecosystem tools such as Hive, HDFS, and YARN

Job Responsibility

Partner with multiple management teams to ensure appropriate integration of functions to meet goals as well as identify and define necessary system enhancements to deploy new products and process improvements
Resolve variety of high impact problems/projects through in-depth evaluation of complex business processes, system processes, and industry standards
Provide expertise in area and advanced knowledge of applications programming and ensure application design adheres to the overall architecture blueprint
Utilize advanced knowledge of system flow and develop standards for coding, testing, debugging, and implementation
Provide in-depth analysis with interpretive thinking to define issues and develop innovative solutions
Serve as advisor or coach to mid-level developers and analysts, allocating work as necessary

Fulltime

Big Data Engineering Lead

The Senior Big Data engineering lead will play a pivotal role in designing, impl...

Location

India , Chennai

Salary:

Not provided

Citi

Expiration Date

Until further notice

Requirements

Bachelor's or Master’s degree in Computer Science, Information Technology, or related field
Atleast 10 -12 years overall software development experience on majorly working with handling application with large scale data volumes from ingestion, persistence and retrieval
Deep understanding of big data technologies, including Hadoop, Spark, Kafka, Flink, NoSQL databases, etc.
Experience with Bigdata technologies Developer Hadoop, Apache Spark, Python, PySpark
Strong programming skills in languages such as Java, Scala, or Python
Excellent problem-solving skills with a knack for innovative solutions
Strong communication and leadership abilities
Proven ability to manage multiple projects simultaneously and deliver results

Job Responsibility

Lead the design and development of a robust and scalable big data architecture handling exponential data growth while maintaining high availability and resilience
Design complex data transformation processes using Spark and other big data technologies using Java, Pyspark or Scala
Design and implement data pipelines that ensure data quality, integrity, and availability
Collaborate with cross-functional teams to understand business needs and translate them into technical requirements
Evaluate and select technologies that improve data efficiency, scalability, and performance
Oversee the deployment and management of big data tools and frameworks such as Hadoop, Spark, Kafka, and others
Provide technical guidance and mentorship to the development team and junior architects
Continuously assess and integrate emerging technologies and methodologies to enhance data processing capabilities
Optimize big data frameworks, such as Hadoop, Spark, for performance improvements and reduced processing time across distributed systems
Implement data governance frameworks to ensure data accuracy, consistency, and privacy across the organization, leveraging metadata management and data lineage tracking

Fulltime

Software Engineer - Data Engineering

Akuna Capital is a leading proprietary trading firm specializing in options mark...

Location

United States , Chicago

Salary:

130000.00 USD / Year

AKUNA CAPITAL

Expiration Date

Until further notice

Requirements

BS/MS/PhD in Computer Science, Engineering, Physics, Math, or equivalent technical field
5+ years of professional experience developing software applications
Java/Scala experience required
Highly motivated and willing to take ownership of high-impact projects upon arrival
Prior hands-on experience with data platforms and technologies such as Delta Lake, Spark, Kubernetes, Kafka, ClickHouse, and/or Presto/Trino
Experience building large-scale batch and streaming pipelines with strict SLA and data quality requirements
Must possess excellent communication, analytical, and problem-solving skills
Recent hands-on experience with AWS Cloud development, deployment and monitoring necessary
Demonstrated experience working on an Agile team employing software engineering best practices, such as GitOps and CI/CD, to deliver complex software projects
The ability to react quickly and accurately to rapidly changing market conditions, including the ability to quickly and accurately respond and/or solve math and coding problems are essential functions of the role

Job Responsibility

Work within a growing Data Engineering division supporting the strategic role of data at Akuna
Drive the ongoing design and expansion of our data platform across a wide variety of data sources, supporting an array of streaming, operational and research workflows
Work closely with Trading, Quant, Technology & Business Operations teams throughout the firm to identify how data is produced and consumed, helping to define and deliver high impact projects
Build and deploy batch and streaming pipelines to collect and transform our rapidly growing Big Data set within our hybrid cloud architecture utilizing Kubernetes/EKS, Kafka/MSK and Databricks/Spark
Mentor junior engineers in software and data engineering best practices
Produce clean, well-tested, and documented code with a clear design to support mission critical applications
Build automated data validation test suites that ensure that data is processed and published in accordance with well-defined Service Level Agreements (SLA’s) pertaining to data quality, data availability and data correctness
Challenge the status quo and help push our organization forward, as we grow beyond the limits of our current tech stack

What we offer

Discretionary performance bonus
Comprehensive benefits package that may encompass employer-paid medical, dental, vision, retirement contributions, paid time off, and other benefits

Fulltime

Senior Software Engineer - Data Integration & JVM Ecosystem

The Connectors team is the bridge between ClickHouse and the entire data ecosyst...

Location

Germany

Salary:

Not provided

ClickHouse

Expiration Date

Until further notice

Requirements

6+ years of software development experience focusing on building and delivering high-quality, data-intensive solutions
Proven experience with the internals of at least one of the following technologies: Apache Spark, Apache Flink, Kafka Connect, or Apache Beam
Experience developing or extending connectors, sinks, or sources for at least one big data processing framework such as Apache Spark, Flink, Beam, or Kafka Connect
Strong understanding of database fundamentals: SQL, data modeling, query optimization, and familiarity with OLAP/analytical databases
A track record of building scalable data integration systems (beyond simple ETL jobs)
Strong proficiency in Java and the JVM ecosystem, including deep knowledge of memory management, garbage collection tuning, and performance profiling
Solid experience with concurrent programming in Java, including threads, executors, and reactive or asynchronous patterns
Outstanding written and verbal communication skills to collaborate effectively within the team and across engineering functions
Understanding of JDBC, network protocols (TCP/IP, HTTP), and techniques for optimizing data throughput over the wire
Passion for open-source development

Job Responsibility

Own and maintain critical parts of ClickHouse's Data engineering ecosystem
Own the full lifecycle of data framework integrations - from the core database driver to SDKs and connectors
Build the foundation that thousands of Data engineers rely on for their most critical data workloads
Collaborate closely with the open-source community, internal teams, and enterprise users to ensure our JVM integrations set the standard for performance, reliability, and developer experience

What we offer

Flexible work environment - ClickHouse is a globally distributed company and remote-friendly
Healthcare - Employer contributions towards your healthcare
Equity in the company - Every new team member who joins our company receives stock options
Time off - Flexible time off in the US, generous entitlement in other countries
A $500 Home office setup if you’re a remote employee
Global Gatherings – opportunities to engage with colleagues at company-wide offsites

Senior Software Engineer - Data Integration & JVM Ecosystem

The Connectors team is the bridge between ClickHouse and the entire data ecosyst...

Location

United Kingdom

Salary:

Not provided

ClickHouse

Expiration Date

Until further notice

Requirements

6+ years of software development experience focusing on building and delivering high-quality, data-intensive solutions
Proven experience with the internals of at least one of the following technologies: Apache Spark, Apache Flink, Kafka Connect, or Apache Beam
Experience developing or extending connectors, sinks, or sources for at least one big data processing framework such as Apache Spark, Flink, Beam, or Kafka Connect
Strong understanding of database fundamentals: SQL, data modeling, query optimization, and familiarity with OLAP/analytical databases
A track record of building scalable data integration systems (beyond simple ETL jobs)
Strong proficiency in Java and the JVM ecosystem, including deep knowledge of memory management, garbage collection tuning, and performance profiling
Solid experience with concurrent programming in Java, including threads, executors, and reactive or asynchronous patterns
Outstanding written and verbal communication skills to collaborate effectively within the team and across engineering functions
Understanding of JDBC, network protocols (TCP/IP, HTTP), and techniques for optimizing data throughput over the wire
Passion for open-source development

Job Responsibility

Serve as a core contributor, owning and maintaining critical parts of ClickHouse's Data engineering ecosystem
Own the full lifecycle of data framework integrations - from the core database driver that handles billions of records per second, to SDKs and connectors that make ClickHouse feel native in JVM-based applications
Build the foundation that thousands of Data engineers rely on for their most critical data workloads
Collaborate closely with the open-source community, internal teams, and enterprise users to ensure our JVM integrations set the standard for performance, reliability, and developer experience

What we offer

Flexible work environment - ClickHouse is a globally distributed company and remote-friendly. We currently operate in 20 countries
Healthcare - Employer contributions towards your healthcare
Equity in the company - Every new team member who joins our company receives stock options
Time off - Flexible time off in the US, generous entitlement in other countries
A $500 Home office setup if you’re a remote employee
Global Gatherings – We believe in the power of in-person connection and offer opportunities to engage with colleagues at company-wide offsites

Fulltime

Software Engineer – Data

We are seeking a skilled Software Engineer with expertise in data engineering an...

Location

Ireland , Dublin 1

Salary:

65000.00 EUR / Year

Solas IT Recruitment

Expiration Date

Until further notice

Requirements

Strong experience in data engineering with Azure or AWS
Hands-on expertise with Databricks, Delta Lake, Data Factory, and NoSQL databases
Proficiency in Python, Scala, or SQL for data workflows
Solid understanding of ETL/ELT patterns and big data processing

Job Responsibility

Develop and maintain scalable ETL/ELT pipelines for data transformation
Manage Data Lakes and big data frameworks, ensuring performance and security
Utilize Azure tools such as Data Factory, Databricks, and Delta Lake
Collaborate with analytics teams to deliver curated datasets
Optimize NoSQL databases and leverage Apache Spark for big data processing

What we offer

pension
healthcare
dental
25 days annual leave
Hybrid working model for flexibility
Opportunities for professional development and cutting-edge projects

Fulltime

Senior Principal Data Platform Software Engineer

We’re looking for a Sr Principal Data Platform Software Engineer (P70) to be a k...

Location

Salary:

239400.00 - 312550.00 USD / Year

Atlassian

Expiration Date

Until further notice

Requirements

15+ years in Data Engineering, Software Engineering, or related roles, with substantial exposure to big data ecosystems
Demonstrated experience building and operating data platforms or large‑scale data services in production
Proven track record of building services from the ground up (requirements → design → implementation → deployment → ongoing ownership)
Hands‑on experience with AWS, GCP (e.g., compute, storage, data, and streaming services) and cloud‑native architectures
Practical experience with big data technologies, such as Databricks, Apache Spark, AWS EMR, Apache Flink, or StarRocks
Strong programming skills in one or more of: Kotlin, Scala, Java, Python
Experience leading cross‑team technical initiatives and influencing senior stakeholders
Experience mentoring Staff/Principal engineers and lifting the technical bar for a team or org
Bachelor’s or Master’s degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience

Job Responsibility

Design, develop and own delivery of high quality big data and analytical platform solutions aiming to solve Atlassian’s needs to support millions of users with optimal cost, minimal latency and maximum reliability
Improve and operate large‑scale distributed data systems in the cloud (primarily AWS, with increasing integration with GCP and Kubernetes‑based microservices)
Drive the evolution of our high-performance analytical databases and its integrations with products, cloud infrastructures (AWS and GCP) and isolated cloud environments
Help define and uplift engineering and operational standards for petabyte scale data platforms, with sub‑second analytic queries and multi‑region availability (coding guidelines, code review practices, observability, incident response, SLIs/SLOs)
Partner across multiple product and platform teams (including Analytics, Marketplace/Ecosystem, Core Data Platform, ML Platform, Search, and Oasis/FedRAMP) to deliver company‑wide initiatives that depend on reliable, high‑quality data
Act as a technical mentor and multiplier, raising the bar on design quality, code quality, and operational excellence across the broader team
Design and implement self‑healing, resilient data platforms with strong observability, fault tolerance, and recovery characteristics
Own the long‑term architecture and technical direction of Atlassian’s product data platform with projects that are directly tied to Atlassian’s company-level OKRs
Be accountable for the reliability, cost efficiency, and strategic direction of Atlassian’s product analytical data platform
Partner with executives and influence senior leaders to align engineering efforts with Atlassian’s long-term business objectives

What we offer

health and wellbeing resources
paid volunteer days

Fulltime

Big Data Spark Development Vice President

We are seeking a Software/Application Dev Lead Analyst for Global Data Platform ...

Location

India , Chennai

Salary:

Not provided

Citi

Expiration Date

Until further notice

Requirements

Bachelor's or master's degree in computer science, Information Technology, or equivalent
Minimum 15 years of work experience
At least 8 years of experience of leading Enterprise Bigdata Development and Data Science (Analytical) teams
Experience developing Applications for High volume instanced payments and/or AML industry is a significant plus
Experience working in Banking technology is preferred
Very strong Engineer excellence knowledge and experience like Data patterns, Pipelines, DQs, Recons, Governance, Scalable application traits and cloud portable coding
Project management, interpersonal, and communication skills, with high attention to details
Must-Have: Experience in building enterprise-grade transaction management and AML system with focus on recovery, stability, reliability, scalability and performance
Must-Have: Strong hand-on seasoned engineer/lead in Bigdata data store application building using enterprise SDLC/Agile methodology, Data modelling/Architect, supporting Operational and production systems
Must-Have: Strong technical skills and hand-on in Spark/Scala, Kafka, Hadoop, Couchbase, HBASE and Snowflake

Job Responsibility

Manage one or more Applications Development teams to accomplish established goals as well as conduct personnel duties for team (e.g. performance evaluations, hiring and disciplinary actions)
Utilize in-depth knowledge and skills across multiple Applications Development areas to provide technical oversight across systems and applications
Review and analyze proposed technical solutions for projects
Contribute to formulation of strategies for applications development and other functional areas
Develop comprehensive knowledge of how areas of business integrate to accomplish business goals
Provide evaluative judgment based on analysis of factual data in complicated and unique situations
Impact the Applications Development area through monitoring delivery of end results, participate in budget management, and handling day-to-day staff management issues, including resource management and allocation of work within the team/project
Ensure essential procedures are followed and contribute to defining standards negotiating with external parties when necessary
Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency, as well as effectively supervise the activity of others and create accountability with those who fail to maintain these standards

Fulltime

Software Development Engineer - Big Data

Skyhigh Security

Location:
India , Bangalore

Category:
IT - Software Development

Contract Type:
Not provided

Salary:

Job Description:

Job Responsibility:

Requirements:

Nice to have:

Additional Information:

Job Posted:
February 20, 2026

Looking for more opportunities? Search for other job offers that match your skills and interests.

Similar Jobs for Software Development Engineer - Big Data