Where the data actually goes.
Not a stage of the spine. Client data has its own path: pulled on a schedule into a fenced landing zone, shaped by thin transforms, served through cache tables the portal reads, with a freshness meter beside every source. The acceptance standard: every displayed number names its source pull, its freshness and its client boundary.
Sources, ingest, landing, shape, serve.
- The platformsAds, GA4, Search Console, Meta, Clarity
- AirbyteSelf-hosted · every new source enters here
- Ops-machine workersThe proven set · no new ones
- raw_* schemasOwn role, no CREATE, never exposed
- ClickHouse · plannedNo second database until volume earns one
- TransformsFour chains today · one per board
- Sync meterLast pull, rows, freshness
- Cache tablesWhat the portal actually reads
- The HubRow-level security is the whole story
- 01
Sources
Where the data is made. Ads, GA4, Search Console, Meta, Clarity.
- 02
Ingest
Airbyte, self-hosted, on a schedule. Every new source enters here. The proven legacy workers remain for the few sources it doesn't cover, and no new ones are written.
- 03
Landing
Raw schemas with their own role, no CREATE, never exposed through the public API. Proven by a negative-access probe before a source carries real data.
- 04
Shape
Thin SQL transforms, one chain per board. Four chains today: analytics daily, search daily, search query, page-speed mart.
- 05
Serve
Small cache tables the portal reads, a sync meter per source (last pull, rows, freshness), and row-level security as the whole story.
Three rules the numbers live under.
Freshness is a node, not a hope
A per-source meter records last pull, what it brought and how old it is, so stale numbers show as stale. Built for covered sources; not every number meets the standard yet, and a probe enumerates which still fail it.
A cost that isn't known is left blank
Never shown as zero. A usage counter once turned an empty string into 0 and summed unrecorded rows. That's a closed bug and a standing rule.
No warehouse until volume earns one
A second database is a recorded no for now. One shared clean layer per source inside the existing database.
Connections are Nango's job. Calls are the broker's.
A connection is an OAuth grant or an API key for one platform, filed under one organisation. Nango owns the consent screen, the token and the refresh lifecycle, so the client never sees the plumbing: Connect Google in Hub, the provider's consent screen, done. The broker then makes every outbound call under a scoped, short-lived grant, so a token Nango holds is still never in an agent's hands. Read-only analytics keys live with Airbyte, where they're used, because they're low stakes and revocable in minutes.
More docs
Architecture
The topology: four zones wired around one record, where agents run, and the four ways a worker is started.
Security
Row-level security as the whole authorisation story, the credential broker, the ten-check gate chain, and what has actually been made to fire.
The stack
Next.js, TypeScript, Supabase, Nango, Airbyte, n8n and Vercel: what each does, what's being rebuilt, and what a stranger needs to run it.
Data plane
How client data reaches Hub: Airbyte into a fenced landing zone, thin transforms, a freshness stamp on every number.
Automations & skills
The split: one repo per organisation, a birth certificate for every automation, skills whose home is git.
Status
The capability matrix, five labels, no hedging. What you could use this afternoon versus what needs a switch or a sitting.
Self-hosting
The clone manifest and the transfer sequence: what you stand up, what moves, what never leaves our custody.