Senior Data Engineer

• Develop and maintain scalable, efficient data pipelines within Databricks, continuously evolving them as requirements and technologies change
• Build and manage an enterprise data model within Databricks
• Integrate new data sources into the platform using batch and streaming processes, adhering to SLAs
• Create and maintain documentation for data pipelines and associated systems, following security and monitoring protocols
• Ensure data quality and reliability processes are effective, maintaining trust in the data
• Be comfortable with taking ownership of complex data engineering projects and develop appropriate solutions in accordance with business requirements
• Able to work closely with stakeholders and managing their requirements
• Actively coach and mentor others in the team and foster a culture of innovation and peer review within the team to ensure best practice

• Extensive experience of Python preferred, including advanced concepts like decorators, protocols, functools, context managers, and comprehensions
• Strong understanding of SQL, database design, and data architecture
• Experience with Databricks and/or Spark
• Knowledgeable in data governance, data cataloguing, data quality principles, and related tools
• Skilled in data extraction, joining, and aggregation tasks, especially with big data and real-time data using Spark
• Capable of performing data cleansing operations to prepare data for analysis, including transforming data into useful formats
• Understand data storage concepts and logical data structures, such as data warehousing
• Able to write repeatable, production-quality code for data pipelines, utilizing templating and parameterization where needed
• Can make data pipeline design recommendations based on business requirements
• Open to new ways of working and new technologies
• Self-motivated with the ability to set goals and take initiative
• Driven to troubleshoot, deconstruct problems, and build effective solutions
• Experience of Git / Version control
• Experience working with larger, legacy codebases
• Understanding of unit and integration testing
• Understanding and experience with CI/CD and general software development best practices
• A strong attention to detail and a curiosity about the data you will be working with
• A strong understanding of Linux based tooling and concepts
• Knowledge and experience of Amazon Web Services is essential

Experience with data migration is a plus