Bigdata Support Lead Engineer

• Lead day to day operation and support for Cloudera Hadoop ecosystem components (HDFS, YARN, Hive, Impala, Spark, HBase, etc)
• Troubleshoot issues related to data ingestion, job failures, performance degradation and service unavailability
• Monitor cluster health using Cloudera Manager and respond to alerts, logs, and metrics
• Collaborate with engineering teams to analyze root causes and implement preventive measures
• Collaborate patching, service restarts, failovers and rolling restarts for cluster maintenance
• Assist in user onboarding, access control and issues in accessing the cluster services
• Contribute to documentation for knowledge base
• Work on data recovery, replication, and backup support tasks
• Responsible for moving all legacy workloads to cloud platform
• Ability to research and assess open-source technologies, public cloud tech stack (AWS/GCP) components to recommend and integrate into the design and implementation
• Be the technical expert and mentor other team members on Big Data and Cloud Tech stacks
• Define needs around maintainability, testability, performance, security, quality and usability for data platform
• Drive implementation, consistent patterns, reusable components, and coding standards for data engineering processes
• Tune Big data applications on Hadoop and non-Hadoop platforms for optimal performance
• Evaluate new IT developments and evolving business requirements and recommend appropriate systems alternatives and/or enhancements to current systems
• Applies in-depth understanding of how data analytics collectively integrate within the sub-function
• Produces detailed analysis of issues where the best course of action is not evident
• Supervise day-to-day staff management issues, including resource management, work allocation, mentoring/coaching
• Appropriately assess risk when business decisions are made

• 10+ years of total IT experience
• 5+ years of experience in supporting Hadoop (Cloudera)/big data technologies
• 5+ years of experience in public cloud infrastructure (AWS or GCP)
• Experience with Kubernetes and cloud-native technologies
• Experience with all aspects of DevOps (source control, continuous integration, deployments, etc.)
• Advanced knowledge of the Hadoop ecosystem and Big Data technologies
• Hands-on experience with the Hadoop eco-system (HDFS, MapReduce, Hive, Pig, Impala, Spark, Kafka, Kudu, Solr)
• Knowledge of troubleshooting techniques for Hive, Spark, YARN, Kafka, and HDFS
• Advanced Linux system administration and scripting skills (Shell, Python)
• Experience on designing and developing Data Pipelines for Data Ingestion or Transformation using Spark with Java or Scala or Python
• Expert level building pipelines using Apache Spark
• Familiarity with core provider services from AWS, Azure or GCP
• Hands-on experience with Python/Pyspark/Scala
• System level understanding - Data structures, algorithms, distributed storage & compute
• Possess team management experience and have led a team of data/platform engineers and analysts
• Bachelor's degree/University degree or equivalent experience

Experience in Snowflake or Delta lake