Senior Data Engineer Job at City and County of San Francisco (San Francisco)

Job Description

DataSF is seeking a Senior Data Engineer with 3+ years of experience to join our growing team. Reporting to the Principal Data Engineer, you will be instrumental in designing, building, and maintaining the City's data infrastructure, enabling robust data pipelines and reliable data access for analytical and operational needs. This is an exciting position for someone eager to apply advanced data engineering techniques to complex urban challenges, contributing directly to San Francisco's commitment to efficient, equitable, and ethical service delivery.

Job Responsibility

Platform Administration: Manage our central Snowflake data warehouse, including access control, security policies, resource monitoring, performance tuning, and cost optimization
Pipeline Development: Build and maintain scalable and resilient pipelines to ingest and structure data from diverse sources
Infrastructure as Code (IaC): Use Terraform to define, deploy, and manage data infrastructure, ensuring our pipelines are reproducible, version-controlled, and production-ready
Best Practices & Innovation: Champion and implement best practices for documentation, data modeling, warehouse architecture, SQL optimization, and testing
Collaboration: Work closely with data scientists, analysts, product managers, software engineers, and nontechnical stakeholders in diverse domains to understand data requirements and build solutions that meet their needs
Monitoring & Support: Proactively monitor the health of the data platform and pipelines, troubleshoot issues, and ensure high standards of data quality and availability

Requirements

An associate degree in computer science, computer engineering, information systems, or a closely related field from an accredited college or university OR its equivalent in terms of total course credits/units
Three (3) years of experience analyzing, installing, configuring, enhancing, and/or maintaining the components of an enterprise network
Additional experience as described above may be substituted for the required degree on a year-for-year basis (up to a maximum of two (2) years)
Completion of the 1010 Information Systems Trainee Program may be substituted for the required degree

Nice to have

Hands-on experience administering and developing on managed cloud data platforms such as Snowflake, BigQuery, or Databricks
Demonstrated expertise in writing advanced, performant SQL, and using tools like dbt for SQL-based data transformation and modeling
Strong programming skills in Python (with libraries like pandas, PySpark) for data processing and automation
Proficiency with an Infrastructure as Code tool, with a preference for Terraform
Experience building and deploying data pipelines using orchestration tools like Azure Data Factory, Airflow, Dagster, or similar technologies
Deep understanding of data warehousing concepts, data modeling, and modern ELT principles
Understanding of data governance, data security, and data privacy principles
Experience with real-time data streaming technologies (e.g., Kafka, Kinesis, Snowpipe)
Experience deploying and managing data pipelines for machine learning models
Strong problem-solving skills with ability to design practical and effective data solutions
Excellent verbal and written communication skills, including the ability to explain technical concepts to non-technical stakeholders
A collaborative mindset with enthusiasm to work across diverse, cross-functional teams
Commitment to equity, transparency, and ethical data use
Passion for public service and using data to improve government services
Empathy for San Francisco's diverse communities and a drive to make data and services and more accessible for SF residents
Interest or experience in public sector data or social impact work

What we offer

Permanent Exempt status
Hybrid work schedule
Equal Employment Opportunity
Veterans Preference
ADA Accommodation
Employee Benefits Overview
Conviction History
Disaster Service Worker
Right to Work

City and County of San Francisco - All Job Offers

Select Country

Senior Data Engineer

Job Description

Job Responsibility

Requirements

Nice to have

What we offer

Looking for more opportunities?