CrawlJobs Logo

Software Development Engineer - Big Data

India, Bangalore · Job Posted February 20, 2026
Apply Position
Job Link Share

Job Description

At Skyhigh Security, we are building ground-breaking technology to help enterprises enable and accelerate the safe adoption of cloud services. SSE products help the world’s largest organizations unleash the power of the cloud by providing real-time protection for enterprise data and users across all cloud services. The Data Analytics team of our cloud service BU is looking for a capable, enthusiastic Big Data Engineer who will be a creative, innovative and results-oriented person willing to go the extra mile in a fast-paced environment. Take ownership of major big data components/services and all backend aspects of the software life cycle in a SaaS environment. Data Analytics team manages big data pipelines and machine learning systems pertaining to our Skyhigh Security Cloud. We are responsible for analysing more than 40 terabytes of data per day, and we inspect close to billion activities of users in real time for threat protection and monitoring.

Job Responsibility

  • Build, test, automate, maintenance, and production support of big data pipelines and Hadoop ecosystem
  • Recognise the big picture and take initiative to solve the problem
  • Being aware of current big data technology trends & factoring this into current design and implementation
  • Identifies, recommends, coordinates, deliver timely knowledge to the globally distributed teams regarding technologies, processes, and tools
  • Proactively identify and communicate roadblocks
  • Provide technical expertise (architecture, design, development, code reviews, use of modern static analysis tools, unit testing & system integration, automated testing, etc.)

Requirements

  • Bachelor's degree in Computer Science or equivalent degree
  • Master's degree is a plus
  • Overall 1 to 4 years of experience
  • Good experience in Python and Java
  • Experience in building of big data pipelines and Hadoop ecosystem
  • Good exposure to test frameworks like JUnit, Test NG, Cucumber and mocking frameworks
  • Develop, Test and implement automated big data pipelines using Python/Java
  • Experience in any Automation Framework
  • Exposure to Agile development, TDD, and Lean development

Nice to have

  • Hands on experience on Robot Framework will be a plus
  • Experience with AWS CloudFormation, Cloudwatch, SQS, Lambda is a plus
  • Hands-on experience in Hadoop and Spark will be plus

What we offer

  • Retirement Plans
  • Medical, Dental and Vision Coverage
  • Paid Time Off
  • Paid Parental Leave
  • Support for Community Involvement

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Software Development Engineer - Big Data

8 matching positions

Senior Software Development Engineer, Big Data

This role will be responsible for developing and supporting data platform infras...
Location
Location
Mexico , Mexico City
Salary
Salary:
Not provided
Zillow
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience in software development with one or more programming languages such as Python, Java, C++, etc.
  • Proficiency in developing and supporting services with cloud infrastructure such as AWS
  • Experience with and ability to debug/troubleshoot container orchestration platforms like Kubernetes
  • Experience with data infrastructure systems like Databricks, Spark, Hive, Airflow, Trino, Kafka, and Flink
  • Experience in setting up and optimizing data pipelines
  • Experience in developing scalable processes for supporting multiple user groups and teams
  • Familiarity and Experience with running on-call operational responsibilities for data platform systems
Job Responsibility
Job Responsibility
  • Developing and supporting data platform infrastructure
  • Collaborating to provide Data Platform Environments that empower internal Zillow Group customers to deliver high-quality data solutions
What we offer
What we offer
  • Competitive base salary and benefits
  • Equity awards based on factors such as experience, performance and location
  • Fulltime
Read More
Arrow Right

Senior Software Development Engineer - Big Data

Zillow’s Comprehensive Customer Data team is part of the Data Engineering organi...
Location
Location
Mexico , Mexico City
Salary
Salary:
Not provided
Zillow
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • A degree (BS+) in Computer Science or a related discipline
  • 5+ years of experience with building and maintaining data-intensive applications
  • Experience developing complex software systems scaling to millions of users with production quality deployment, monitoring and reliability
  • Experience with modern data technologies (Spark, Flink, Kafka, Airflow)
  • Strong background in Java or Python
  • Excellent interpersonal skills and passionate about collaborating across organizational boundaries
Job Responsibility
Job Responsibility
  • Design, implement, and take ownership of mission-critical data pipelines and data products
  • Collaborate with various internal teams (platform, product, other stakeholders) to scope and deliver tools ensuring ease of system use
  • Carefully review specifications, designs, pull requests and provide constructive/helpful feedback
  • Write high-quality code, refactor and optimize code for better scalability, performance, and readability
  • Integrate new frameworks, libraries, technologies, and components into codebase
  • Proactively log/collect and review metrics, monitor, alert and address issues quickly
  • Ensure the consistency and maintainability of existing applications adhering to standards/procedures for implementing technical solutions
  • Execute production/support tasks according to a documented schedule that meets or exceeds customer expectations
  • Support and mentor teams through design, implementation and deployment of solutions using relevant technologies
  • Mentor junior engineers
What we offer
What we offer
  • competitive base salary and benefits
  • equity awards based on factors such as experience, performance and location
  • Fulltime
Read More
Arrow Right

Senior Software Development Engineer, Big Data

The Zillow Group’s Housing Trends Data Engineering team brings innovative analys...
Location
Location
United States
Salary
Salary:
152900.00 - 257100.00 USD / Year
Zillow
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong proficiency in Python programming
  • Hands-on experience with big data ecosystems, architectures and modern data platforms, including AWS cloud data services and Lakehouse/Lakebase platforms
  • Hands-on experience with Databricks is a significant advantage
  • Proven ability to design, build, and orchestrate batch and real-time data pipelines, processing large-scale datasets from diverse sources
  • Practical experience with cutting-edge data engineering technologies such as SparkSQL and Spark Streaming, PostgreSQL, EKS, and Kubernetes
  • Familiarity with Agile/DevOps software development processes and tools such as GitLab and CI/CD
  • A good understanding of data science concepts, fundamental machine learning algorithms/libraries, and development lifecycle is a must-to-have
  • Familiarity with LLM and Agentic AI technologies and engineering patterns is a big advantage
  • Strong ability in performing root cause analysis to address operational issues and identify opportunities for improvement
  • Exceptional communication skills to drive effective customer engagement, collaborate with cross-functional teams, and clearly articulate technical concepts to technical and non-technical stakeholders
Job Responsibility
Job Responsibility
  • Build data systems to link raw data and concrete Housing Trends insights
  • Build and extend our internal forecasting model development framework
  • Develop and maintain scalable data products and deliver pipelines built for speed, accuracy, and consistency that will scale as our stakeholders’ need
  • Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability and performance, etc
  • Collaborate with data scientists to understand their data requirements and build systems/tools that enable efficient model training and experimentation
  • Implement feature stores and ETL pipelines to streamline feature engineering and model development
  • Build and maintain infrastructure for deploying machine learning models into production environments
  • Be a trusted partner for our stakeholders, such as Forecasting, Zestimates, and the ML teams within the AI organization to assist them for data investigation/publication, ML model development and infrastructure support
What we offer
What we offer
  • Eligible for equity awards based on factors such as experience, performance and location
  • Fulltime
Read More
Arrow Right

Software engineer 2 / Senior Software engineer - Azure Data

Microsoft's Azure Data engineering team is leading the transformation of analyti...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 3+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
  • OR equivalent experience
  • Experience with the Azure stack including Storage, Compute, Networking, Fabric, Purview, Synapse, AKS, DevOps, Data Factory, or Power BI
  • Experience with big data technologies such as Spark, Kafka, Hadoop, or HBase
  • Experience building data lake or data engineering products, tools, or pipelines
  • Familiarity with container-based architectures (Docker, Kubernetes)
  • Ability to debug complex distributed systems on Linux and/or Windows platforms
Job Responsibility
Job Responsibility
  • Write extensible, maintainable code in C#, Java, Scala, or Python for Fabric Materialized Lake View services and HDInsight components
  • Use AI tools and coding best practices across the development lifecycle
  • Design data refresh, scheduling, and query optimisation features with minimal supervision
  • Review code from teammates for correctness, test coverage, security risks, and adherence to team standards
  • Coach junior engineers through code reviews
  • Debug complex issues in distributed systems running on Azure, Linux, and Windows
  • Run live site operations on a rotational, on-call basis
  • Integrate logging and instrumentation to gather telemetry on system health, performance, reliability, and security
  • Work with product managers, technical leads, and partners across geographies to define customer requirements for Materialized Lake View features
  • Fulltime
Read More
Arrow Right

Senior Software Engineer- Big Data & Java

Location
Location
Canada , Mississauga
Salary
Salary:
Not provided
pointclickcare.com Logo
PointClickCare
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience with data pipelines processing large amounts of data in Real Time or Near Real Time
  • Experience with Lakehouse architecture, Azure ADLS Gen2, Apache Hudi
  • Experience developing Java/Spring Boot microservices
  • Experience with Agentic AI components
  • Legally authorized to work in Canada
  • Fulltime
Read More
Arrow Right

Senior Software Engineer- Big Data & Java

Location
Location
United States
Salary
Salary:
Not provided
pointclickcare.com Logo
PointClickCare
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Do you have experience with data pipelines processing large amounts of data in Real Time or Near Real Time?
  • Do you have experience with Lakehouse architecture, Azure ADLS Gen2, Apache Hudi?
  • How many years of experience developing Java/Spring Boot microservices?
  • Do you have experience with Agentic AI components?
  • Are you legally authorized to work in the US for our company?
  • Fulltime
Read More
Arrow Right

Data Engineer (Big Data, Python, Databricks) - Assistant Vice President

The Applications Development Senior Programmer Analyst is an intermediate level ...
Location
Location
India , Chennai, Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5-8 years of relevant handson experience in Big Data technologies like Cloudera, Python, HQL, Java/PySpark
  • Knowledge on Machine Learning, AI would be added advantage
  • Experience in systems analysis, data analysis and programming of software applications
  • Experience in managing and implementing successful projects
  • Working knowledge of consulting/project management techniques/methods
  • Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
  • Bachelor’s degree/University degree or equivalent experience
Job Responsibility
Job Responsibility
  • Conduct tasks related to feasibility studies, time and cost estimates, IT planning, risk technology, applications development, model development, and establish and implement new or revised applications systems and programs to meet specific business needs or user areas
  • Monitor and control all phases of development process and analysis, design, construction, testing, and implementation as well as provide user and operational support on applications to business users
  • Utilize in-depth specialty knowledge of applications development to analyze complex problems/issues, provide evaluation of business process, system process, and industry standards, and make evaluative judgement
  • Recommend and develop security measures in post implementation analysis of business usage to ensure successful system design and functionality
  • Consult with users/clients and other technology groups on issues, recommend advanced programming solutions, and install and assist customer exposure systems
  • Ensure essential procedures are followed and help define operating standards and processes
  • Serve as advisor or coach to new or lower level analysts
  • Has the ability to operate with a limited level of direct supervision
  • Can exercise independence of judgement and autonomy
  • Acts as SME to senior stakeholders and /or other team members
  • Fulltime
Read More
Arrow Right

Software Engineer III (Data Engineer)

We are seeking a motivated and detail-oriented Data Engineer to join our data en...
Location
Location
United States , Irvine
Salary
Salary:
96500.00 - 138061.00 USD / Year
haeaus.com Logo
Hyundai AutoEver America
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree in Computer Science, Information Technology, or related field
  • 10+ years of experience in data warehouse and MDM applications
  • Extensive knowledge of SQL and relational databases
  • Strong programming skills in Python and PL/SQL
  • 10+ years of experience working with Informatica
  • Knowledge of Informatica IDMC and PowerCenter is a must
  • Knowledge of big data technologies such as Apache Spark, Hadoop, or equivalent cloud-based services
  • Strong understanding of data governance, security, and quality practices in cloud environments
Job Responsibility
Job Responsibility
  • Design, build, and maintain scalable data pipelines and ETL processes to transform and integrate data from multiple sources
  • Develop workflow automation using Apache Airflow and manage production batches with performance tuning
  • Create and optimize data models, schemas, and database structures for scalability and efficiency
  • Implement best practices in data warehouse and MDM applications, ensuring data governance, security, and quality
  • Collaborate with data scientists, analysts, and business stakeholders to translate requirements into technical solutions
  • Develop scripts and automation tools to streamline data processing and pipeline operations
  • Implement automated monitoring, alerting systems, and maintain documentation for pipelines and data models
  • Create reports and visualizations to communicate pipeline performance and data insights
What we offer
What we offer
  • comprehensive medical/dental coverage
  • generous PTO
  • education assistance
  • annual merit increase eligibility
  • Fulltime
Read More
Arrow Right