Job Description:
Lead the design and implementation of scalable, secure, and high-performing data solutions using Azure Databricks, Delta Lake, and Delta Live Tables. Own the complete architecture process, including requirements gathering, solution design, documentation, technical reviews and implement advanced data solutions on the Databricks platform. Define best practices for cloud data architecture, data modeling, ELT/ETL pipelines, workspace setup, cluster management, repos, and job orchestration. Coordinate and communicate with onshore and offshore teams, including end-users, data engineers, reporting specialists, and business analysts. Ensure solution compliance with data privacy, security, and governance standards. Conduct performance tuning and optimization of Databricks clusters. Ensure data quality, lineage, and observability across all pipelines. Monitor and troubleshoot data pipelines to ensure data quality and reliability. Lead the integration of Databricks with other data platforms and tools. Create processes and workflows to support data solutions documents, and lead solution reviews and audits for quality. Stay updated with industry best practices, Databricks and Azure developments, and introduce improvements to the technical stack.
