CrawlJobs Logo

Sr. Software Engineer, Big Data

haeaus.com Logo

Hyundai AutoEver America

Location Icon

Location:
United States , Irvine

Category Icon

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

103170.00 - 158873.00 USD / Year

Job Description:

Design, build, and maintain scalable data platforms, pipelines, and data lakes to enable large-scale data processing and analysis. Collaborate with data scientists and stakeholders to ensure proper data collection, storage, and security, troubleshoot infrastructure issues, and optimize systems for scalability and efficiency.

Job Responsibility:

  • Design, build, and maintain scalable data platforms, pipelines, and data lakes to enable large-scale data processing and analysis
  • Collaborate with data scientists and stakeholders to ensure proper data collection, storage, and security, troubleshoot infrastructure issues, and optimize systems for scalability and efficiency
  • Build and maintain scalable data pipelines and robust data models for AI/ML from structured and unstructured sources
  • Develop Big Data pipelines using orchestration tools (Airflow/Oozie) and implement access management, monitoring, and self-service ETL/analytics solutions
  • Write data pipelines using Spark, Python, and Scala
  • develop frameworks/utilities in Python
  • and follow DevOps best practices
  • Advanced SQL skills with ability to query and transform large structured/unstructured datasets using Spark/PySpark, Spark SQL/Hive, Hive/NoSQL
  • hands-on experience with On-Prem Big Data platforms, distributed frameworks like YARN, and proficiency in building data pipelines using Spark, Python, and Scala
  • Diagnose software issues, optimize performance, and support BI tools (Tableau, Power BI, MicroStrategy) for Big Data
  • Lead and guide team members, adapt to new technologies, and work independently or collaboratively to achieve goals

Requirements:

  • Bachelor’s degree in computer science or related field (or equivalent)
  • 10+ years in IT application support
  • 7+ years as a Big Data Engineer
  • Big Data Expertise: Hadoop, Spark, Spark Streaming, Trino, Flink, Hive, Pig, Kafka, NoSQL (MongoDB, Cassandra, HBase)
  • Programming & Modeling: Proficiency in Java, Python, Scala, SQL
  • strong data modeling and ETL experience
  • Cloud & Security: Familiarity with AWS/Azure/GCP
  • knowledge of data security principles and implementation
  • Performance & Analytics: Experience with performance tuning, data warehousing, and ML/analytics integration in Big Data systems
What we offer:
  • comprehensive medical/dental coverage
  • generous PTO
  • education assistance
  • annual merit increase eligibility

Additional Information:

Job Posted:
January 13, 2026

Employment Type:
Fulltime
Work Type:
On-site work
Job Link Share:

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Sr. Software Engineer, Big Data

Sr Application Data Engineer

The Applications Development Senior Programmer Analyst will contribute to applic...
Location
Location
India , Pune; Chennai
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5-8 years of relevant experience
  • experience in systems analysis and programming of software applications
  • experience in managing and implementing successful projects
  • working knowledge of consulting/project management techniques/methods
  • ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
  • 10+ years of application/software development/maintenance
  • banking domain experience
  • 8+ years of experience on Big Data Technologies like Apache Spark, Hive, Hadoop
  • proficiency in ETL technologies like Abinitio, Data stage, Informatica
  • strong technical knowledge of Apache Spark, Hive, SQL, Hadoop ecosystem, UNIX/Python Scripting, Oracle/DB2
Job Responsibility
Job Responsibility
  • conduct feasibility studies, time and cost estimates, IT planning, risk technology, applications development, model development, and system implementation
  • monitor and control all phases of the development process, including analysis, design, construction, testing, and implementation
  • provide user and operational support on applications to business users
  • analyze complex problems and make evaluative judgments
  • recommend and develop security measures post-implementation
  • consult with users/clients on issues and recommend advanced programming solutions
  • install and assist customer exposure systems
  • define operating standards and processes
  • serve as advisor or coach to new or lower-level analysts
  • exercise judgment and autonomy
  • Fulltime
Read More
Arrow Right

Sr. Software Engineer - Data Assets

6sense is on a mission to revolutionize how B2B organizations create revenue by ...
Location
Location
United States , San Francisco
Salary
Salary:
Not provided
https://6sense.com Logo
6sense
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • BS/MS degree in Computer Science or related technical field or equivalent practical experience
  • Experience in the Big Data technologies (Hadoop, M/R, Hive, Spark, Flink, Metastore, Presto, Kafka etc.)
  • Experience with performing data analysis, data ingestion and data integration
  • Experience with ETL (Extraction, Transformation & Loading) and architecting data systems
  • Experience with schema design, data modeling and SQL queries
  • Passionate and self-motivated about technologies in the Big Data area
Job Responsibility
Job Responsibility
  • Design and build data transformations efficiently and reliably for different purposes (e.g. reporting, growth analysis, multi-dimensional analysis)
  • Design and implement reliable, scalable, robust and extensible big data systems that support core products and business
  • Establish solid design and best engineering practice for engineers as well as non-technical people
What we offer
What we offer
  • generous health insurance coverage
  • life, and disability insurance
  • a 401K employer matching program
  • paid holidays
  • self-care days
  • paid time off (PTO)
  • stock options
  • paid parental leave
  • generous paid time-off
  • quarterly self-care days off
  • Fulltime
Read More
Arrow Right

Sr. Software Engineer

You are a Sr. Software Engineer, who is passionate about writing beautiful code ...
Location
Location
United States , San Francisco
Salary
Salary:
172599.00 - 253145.00 USD / Year
https://6sense.com Logo
6sense
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • BS/MS in Computer Science, or related fields
  • 5+ years of relevant technical experience
  • Solid foundation in computer science, with strong competencies in data structures, algorithms, and software design
  • Experience with software design and development, with knowledge of UNIX/Linux
  • Strong coding skills and hands on experience in Java on Spring Boot, Python on Django and OO framework
  • Knowledge in SQL or Non-SQL, direct work experience on MySQL and/or Hive
  • Working experience of SingleStore(memsql) DB is a huge plus
  • Work with teams and being able to multi-task on multiple products and projects
  • Excellent communication skills, both written and oral
  • Experience with test-driven development, continuous integration, and continuous deployment processes
Job Responsibility
Job Responsibility
  • Develop backend & data engineering framework for 6sense’s revenue ai, marketing and analytics products
  • Work on complex problems related to scalability, performance & big data
  • Write performant REST APIs for both internal and external consumption
  • Build robust high-volume, large data set production applications
  • Collaborate with Product and Architect to solve complex problems
  • Support QA and DevOps teams with test frameworks and automation
  • Mentors junior engineers and provides technical guidance to the team
What we offer
What we offer
  • Health coverage
  • Paid parental leave
  • Generous paid time-off and holidays
  • Quarterly self-care days off
  • Stock options
  • 401K employer matching program
  • Life insurance
  • Disability insurance
  • Fulltime
Read More
Arrow Right

Director of Engineering

We are seeking a highly experienced and strategic Director of Engineering to tak...
Location
Location
India , Chennai
Salary
Salary:
Not provided
arrcus.com Logo
Arrcus
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • BS/MS/PhD in Computer Engineering/Computer Science or equivalent degree
  • Excellent communication, presentation, and interpersonal skills
  • 5+ years of experience leading and managing distributed engineering teams involved in creating complex software products
  • 10+ years of relevant experience in managing very senior technical talent in some of the following areas: Networking protocols such as OSPF, BGP, ISIS, MPLS, BFD, MLAG, EVPN, VxLAN, SR-MPLS, SRv6, L3VPN
  • Test Harness like Robot framework, Jinja2
  • Familiarity network merchant silicon chipsets and whitebox platforms
  • Software development of Network Data Path (Linux, virtual and ASIC)
  • Virtualization technologies like SR-IOV, Intel DPDK, FD.io, NSX, OVS
  • High Availability, ISSU, Linux networking
  • Debian Build/Packaging, Linux Kernel, Kernel Networking Stack
Job Responsibility
Job Responsibility
  • Work with customer and product teams to understand and prioritise new requirements
  • Develop a holistic understanding of individual employee skill sets and drive resource allocation for customer requirements
  • Help drive the recruiting process both in terms of attracting new talent as well as defining and streamlining the recruitment process
  • Continuous Process Improvement: Solicit feedback, drive discussion and implement process and workflow improvements
  • Provide technical guidance and mitigation for engineering projects
  • Build 1:1 rapport with engineers, help identify and fulfil personal aspirations by aligning with larger team goals
  • Strong ability to plan, execute and deliver multiple projects across worldwide sites
  • Experience with rapidly growing engineering organizations in all aspects of people, resources, tools, and more
What we offer
What we offer
  • Generous compensation packages including equity
  • Medical Insurance
  • Parental Leave
  • Fulltime
Read More
Arrow Right
New

Sr Data Engineer

As a Data Engineer, you will be responsible for designing, building, maintaining...
Location
Location
India , Hyderabad
Salary
Salary:
Not provided
amgen.com Logo
Amgen
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Doctorate degree / Master's degree / Bachelor's degree and 8 to 13 years of Computer Science, IT or related field experience
  • Hands on experience with big data technologies and platforms, such as Databricks, Apache Spark (PySpark, SparkSQL),Snowflake, workflow orchestration, performance tuning on big data processing
  • Experienced with software engineering best-practices, including but not limited to version control (Git, Subversion, etc.), CI/CD (Jenkins, Maven etc.), automated unit testing, and Dev Ops
  • Proficient in SQL, Python for extracting, transforming, and analyzing complex datasets from relational data stores
  • Proficient in Python with strong experience in ETL tools such as Apache Spark and various data processing packages, supporting scalable data workflows and machine learning pipeline development.
  • Strong understanding of data modeling, data warehousing, and data integration concepts
  • Proven ability to optimize query performance on big data platforms
  • Knowledge on Data visualization and analytics tools like Spotfire, PowerBI
Job Responsibility
Job Responsibility
  • Design, develop, and maintain data solutions for data generation, collection, and processing
  • Be a key team member that assists in design and development of the data pipeline
  • Create data pipelines and ensure data quality by implementing ETL processes to migrate and deploy data across systems
  • Contribute to the design, development, and implementation of data pipelines, ETL/ELT processes, and data integration solutions
  • Develop and maintain data models, data dictionaries, and other documentation to ensure data accuracy and consistency
  • Implement data security and privacy measures to protect sensitive data
  • Leverage cloud platforms (AWS preferred) to build scalable and efficient data solutions
  • Collaborate and communicate effectively with product teams
  • Collaborate with Data Architects, Business SMEs, and Data Scientists to design and develop end-to-end data pipelines to meet fast paced business needs across geographic regions
  • Identify and resolve complex data-related challenges
What we offer
What we offer
  • Competitive and comprehensive Total Rewards Plans that are aligned with local industry standards
Read More
Arrow Right

Sr. Analyst, Informatics

At CVS Health, we’re building a world of health around every consumer and surrou...
Location
Location
United States , Woonsocket, RI
Salary
Salary:
104645.00 USD / Year
https://www.cvshealth.com/ Logo
CVS Health
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Master’s degree (or foreign equivalent) in Computer Science, Computer Engineering, Data Science, Statistics, Analytics, Business Analytics, or a related field and two (2) years of experience in the job offered or related occupation
  • Requires one (1) year of experience in each of the following: CI/CD, Jenkins, GIT, or DevOps
  • Java, Python, or R
  • Agile methodologies or SAFe Software Development Principles
  • ATG Commerce or Spring Boot
  • JIRA, Rally, or Confluence
  • Spark, Airflow, Kafka, Hbase, Pig, MySQL, or NoSQL
  • Objected-oriented JavaScript
  • Relational database concepts
  • SQL for querying and manipulating data
Job Responsibility
Job Responsibility
  • Apply statistical techniques and health informatics principles to enhance healthcare data analytics solutions and develop trend reporting for CVS Health business units
  • analyze business requirements and interpret statistical data for analytics initiatives in support of business objectives
  • develop, build and manage large-scale data structures, pipelines and efficient Extract/Load/Transform (ETL) workflows to organize, collect and standardize data to generate insights and addresses reporting needs
  • write ETL (Extract/Transform/Load) processes, design database systems, and develop tools for real-time and offline analytic processing that improve existing systems and expand capabilities
  • identity data trends and factors impacting research, and determine appropriate statistical methods based on user needs
  • utilize data-oriented programming languages and visualization software to explore, analyze, and interpret large volumes of data and effectively communicate insights to inform business decisions
  • compare modeling techniques using statistical performance metrics and recommend improvements
  • structure data analysis to drive analytics insights and help data scientists determine the best solutions
  • visualize, interpret, and report data findings to internal clients and leadership
  • collaborate with teams and stakeholders to understand requirements and provide usable information to develop analytics solutions using industry leading tools, technologies and best practices
What we offer
What we offer
  • The Company offers a full range of medical, dental, and vision benefits
  • Eligible employees may enroll in the Company’s 401(k) retirement savings plan, and an Employee Stock Purchase Plan is also available for eligible employees
  • The Company provides a fully-paid term life insurance plan to eligible employees, and short-term and long term disability benefits
  • CVS Health also offers numerous well-being programs, education assistance, free development courses, a CVS store discount, and discount programs with participating partners
  • As for time off, Company employees enjoy Paid Time Off (“PTO”) or vacation pay, as well as paid holidays throughout the calendar year
  • Fulltime
Read More
Arrow Right

Sr Big Data Engineer - Oozie and Pig (GCP)

We are seeking a Senior Big Data Engineer with deep expertise in distributed sys...
Location
Location
United States
Salary
Salary:
116100.00 - 198440.00 USD / Year
rackspace.com Logo
Rackspace
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in Computer Science, software engineering or related field of study
  • Experience with managed cloud services and understanding of cloud-based batch processing systems
  • Proficiency in Oozie, Airflow, Map Reduce, Java
  • Strong programming skills with Java (specifically Spark), Python, Pig, and SQL
  • Expertise in public cloud services, particularly in GCP
  • Proficiency in the Apache Hadoop ecosystem with Oozie, Pig, Hive, Map Reduce
  • Familiarity with BigTable and Redis
  • Experienced in Infrastructure and Applied DevOps principles in daily work
  • Utilize tools for continuous integration and continuous deployment (CI/CD), and Infrastructure as Code (IaC) like Terraform
  • Proven experience in engineering batch processing systems at scale
Job Responsibility
Job Responsibility
  • Design and develop scalable batch processing systems using technologies like Hadoop, Oozie, Pig, Hive, MapReduce, and HBase, with hands-on coding in Java or Python (Java is a must)
  • Must be able to lead Jira Epics
  • Write clean, efficient, and production-ready code with a strong focus on data structures and algorithmic problem-solving applied to real-world data engineering tasks
  • Develop, manage, and optimize complex data workflows within the Apache Hadoop ecosystem, with a strong focus on Oozie orchestration and job scheduling
  • Leverage Google Cloud Platform (GCP) tools such as Dataproc, GCS, and Composer to build scalable and cloud-native big data solutions
  • Implement DevOps and automation best practices, including CI/CD pipelines, infrastructure as code (IaC), and performance tuning across distributed systems
  • Collaborate with cross-functional teams to ensure data pipeline reliability, code quality, and operational excellence in a remote-first environment
  • Fulltime
Read More
Arrow Right

Sr Software Engineer - Python

As a Sr. Software Engineer, you serve as a specialist in the engineering team th...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
blueyonder.com Logo
Blue Yonder
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree in computer science is required, Masters is preferred
  • 4+ years of software engineering experience building production software
  • Experience in Frontend technologies, JavaScript, TypeScript, React
  • Good working knowledge of Kubernetes and other virtualized execution technologies
  • 1+ years of experience working on at least one cloud environment, GCP preferred
  • 4+ years of Python programming experience with excellent understanding of Object-Oriented Design & Patterns
  • 3+ years of experience in building REST APIs
  • 1+ Working Experience on Kafka and its integration with Cloud Services
  • 3+ years of Linux scripting experience
  • 1+ years working with traditional and new relational SQL DBMS
Job Responsibility
Job Responsibility
  • Design, architect, implement and help operate the Machine Learning platform
  • Develop and gain insight in the application architecture
  • Distill an abstract architecture into concrete design and influence the implementation
  • Observing inefficiencies, both in cost and reliability, of existing processes
  • Researching alternative solutions using custom or existing open source technologies
  • Designing replacement processes and components
  • Implementing processes, extending and configuring open source components
  • Work with the ML DevOps and Support teams to operate ML platform
  • Helping implement DevOps best practices of in-house and open source components
  • Ensuring smooth operation via monitoring and alerting facilities
  • Fulltime
Read More
Arrow Right