You don't need a data service,
you just need an object store and some JSON files

a lazy approach to APIs

Gordon Inggs

PyConZA 2024

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away

Antoine de Saint-Exupéry, Airman's Odyssey

Larry Wall's three virues of programmers:

1. Laziness
2. Impatience
3. Hubris

Talk Overview

• Who is this guy?
• Why did we build lazy APIs in the first place?
• How does this work, and what is actually going on here?
• Why is this a good and/or bad idea?
• Taking it further

So, who is this guy?

$ whoami

• Native Capetonian (Simontonian, really)
• Education:
  • UCT:
    • Undergraduate in Electrical and Computer Engineering
    • Masters in Electrical Engineering
  • Imperial College London:
    • PhD in Computer Engineering
• Work Experience:
  • 2016 - mid-2018: Amazon Web Services ("The Cloud")
  • August 2018: City
• City Work:
  • PPO: Data Science (2018 - 2021)
  • PPO: Data Engineer (2021 - )
• Can be found on Mastodon: @gord1i@fosstodon.org

$ groups ginggs

• Data Science Branch -> Policy and Strategy Department -> Future Planning and Resilience Directorate
• Centralised, support function -> advanced analytics.
• Project Examples:
  • Data mining resident complaints
  • Covid-19 Fatalities Management Analytics
  • Building Change Detection using high resolution satellite imagery
• Benefit the residents of Cape Town through data led decision making

• Team Composition:
  • 4 x Data Scientists
  • 5 x Data Engineers
  • 2 X Data Analyst
  • 1 x Product Manager
  • 1 x Project Administrator
  • 1 x Manager
• Ways of Working:
  • Agile-ish process -> prioritise relationships + iteration
  • Cross functional project teams
  • Open Source stack

Why did we do this in the first place?

• (Literal) Dark days of 2021
• Data App with a polling architecture
  • No infrastructure to run a web service
  • No human capacity to run a web service

• Solution: Call those weird people in Data Science!

How could this work?

• Object Stores are the simplest possible ways to share files (objects)
• a HTTP GET on a path gives you a file
• The other key bit is Media Types - application/json, text/plain, etc.

Something important that Gordon learnt

What is actually going on here?

1. Decoupling data provisioning from serving:
   1. Data prep can be async
   2. Use modern web infrastructure for serving
2. Applying lessons from the web world to data

More Complicated - Service Alerts Pipeline

Why this be a good idea?

It's simple!

• Putting files in a bucket
• One line of code

It scales!

It's cheap!

• September 2023 - lots of loadshedding
• 70k unique users per day
• 100k unique requests per day

• ...

• Grand total of R14 for the month

Why might this be a bad idea?

• Limited to GET, PUT, HEAD, etc.
• Combinatorial Explosion!

Taking it even further...

Write an OpenAPI spec

• Postel's Robustness principle in action!
• clients in any language

Using the (non-AI) generated Python Client:

import cct_service_alerts

with cct_service_alerts.ApiClient() as api_client:
    api_instance = cct_service_alerts.V1Api(api_client)
    unplanned_alerts = api_instance.v1_coct_service_alerts_current_unplanned_json_get()

Have a Public Repo

Thanks!