Senior Big Data Engineer

• Design and develop scalable data pipelines and solutions using Python and PySpark
• Utilize big data technologies such as Hadoop, Spark, Kafka, or similar tools for processing and analyzing large datasets
• Develop and maintain ETL processes to extract, transform, and load data into data lakes or warehouses
• Collaborate with data engineers and scientists to implement machine learning models and algorithms
• Optimize and tune data processing workflows for performance and efficiency
• Implement data governance and security measures to ensure data integrity and privacy
• Create and maintain documentation for data pipelines, workflows, and processes
• Provide technical leadership and mentorship to junior team members

• Proficiency in Python programming for data manipulation and analysis
• Experience with PySpark for processing large-scale data
• Strong understanding and practical experience with big data technologies such as Hadoop, Spark, Kafka, etc.
• Knowledge of designing and implementing ETL processes for data integration
• Ability to work with large datasets, perform data cleansing, transformations, and aggregations
• Familiarity with machine learning concepts and experience implementing ML models
• Understanding of data governance principles and experience implementing data security measures
• Ability to create clear and concise documentation for data pipelines and processes
• Strong teamwork and collaboration skills to work with cross-functional teams
• Analytical and problem-solving skills to optimize data workflows and processes
• For senior roles, the ability to provide technical leadership, mentorship, and guidance to junior team members
• Knowledge of SQL for querying and manipulating data in databases

Experience with cloud platforms like AWS, Azure, or Google Cloud is a plus