263 lines
14 KiB
Markdown
263 lines
14 KiB
Markdown
# Concord discovery: why no community ever reaches `subscribe`
|
||
|
||
Audit + fix plan. Read alongside `docs/concord-usage.md` and
|
||
`docs/concord-simplification-plan.md`.
|
||
|
||
## Symptom
|
||
|
||
`crates/community/src/lib.rs::subscribe` is never called, so no wrap is ever
|
||
subscribed to and the sidebar stays empty. `community load: 0 state document(s)
|
||
found` is the only clue.
|
||
|
||
## Root cause
|
||
|
||
`CommunityRegistry::load` only ever reads the **local database**. Nothing in the
|
||
discovery path touches a relay.
|
||
|
||
```
|
||
community::init
|
||
└─ SignerChanged → load
|
||
└─ sync::load
|
||
├─ store::load_states(client) → client.database().query(..) // local only
|
||
└─ load_list(client, ..) → client.database().query(..) // local only, .limit(1)
|
||
→ track([])
|
||
→ sync_subscriptions: `for community in self.communities` runs zero times
|
||
→ subscribe never called
|
||
→ no relay is ever queried
|
||
→ the database never fills
|
||
→ load stays empty forever
|
||
```
|
||
|
||
The loop is self-reinforcing: the local database is populated *by* the
|
||
subscriptions that the empty load prevents. That is why an account which belongs
|
||
to several communities in another client still shows nothing — a fresh install
|
||
has no `concord/*` state document, and coop has no way to ask for one.
|
||
|
||
Confirmed by inspection:
|
||
|
||
| Location | What it does |
|
||
| --- | --- |
|
||
| `crates/community/src/lib.rs:167-191` | `load` → `sync::load`, then `track(states)` |
|
||
| `crates/community/src/sync.rs:125-137` | `load` = `store::load_states` + `load_list` |
|
||
| `crates/concord/src/store.rs:314-341` | `load_states` queries `client.database()` only |
|
||
| `crates/community/src/sync.rs:139-153` | `load_list` queries `client.database()` only, `.limit(1)` |
|
||
| `crates/community/src/lib.rs:233-276` | `sync_subscriptions` skips everything when `communities` is empty |
|
||
|
||
`subscribe` itself is correct. Do not debug it.
|
||
|
||
## What the protocol actually says
|
||
|
||
Read from the spec (`concord-protocol/concord`, the submodule referenced by
|
||
accordion.chat): `02.md` §8 and `examples.md` §6.2.
|
||
|
||
A member's memberships live in the **Community List**, on relays:
|
||
|
||
- **Kind `33302`**, addressable, NIP-44-encrypted to self, signed by the
|
||
member's real key, one event per **fragment** with `d` = the fragment index in
|
||
decimal (`"0"`, `"1"`, …). `13302` is explicitly **retired** ("the
|
||
single-event Community List, superseded by `33302` once it outgrew one event —
|
||
a replaceable kind cannot fragment", `02.md:314`).
|
||
- Every 32-byte value at **any depth** is unpadded base64url, not hex. This is
|
||
section-scoped: CORD-05 invite fields stay hex (`examples.md` §6.3).
|
||
- Join material is the membership subset — `owner, owner_salt, community_root,
|
||
root_epoch, control_pk, channels, relays, name`, plus `control_root` when
|
||
held. It is the *only* durable home of a member's keys.
|
||
- The two snapshots solve opposite problems: `seed` is the earliest epoch held
|
||
(backfill anchor), `current` the latest ("so a fresh device reconstructs the
|
||
Community instantly with no epoch-by-epoch walk"). `seed` is omitted when
|
||
equal to `current`; embedded snapshots omit `community_id` (inherited).
|
||
- A client holds the complete List when it holds a fragment at every index below
|
||
`frags`; it unions fragments and merges, so a partial read is safe.
|
||
|
||
Two consequences for coop:
|
||
|
||
1. **The state document is a coop invention.** `store::{save_state, load_state,
|
||
load_states}` write kind `30078` with `d = concord/<id>`, signed by a
|
||
per-process `LOCAL_KEYS`, and never leave the machine. No equivalent exists
|
||
anywhere in the spec. It is a local cache and must never be treated as the
|
||
discovery source.
|
||
2. **Discovery is: fetch my `33302` from relays → materialize a community from
|
||
`current` join material → subscribe to its planes → fold.** The fold produces
|
||
the authoritative state; the List only supplies the keys to start.
|
||
|
||
## Divergences (coop vs spec)
|
||
|
||
| # | Spec | coop today |
|
||
| --- | --- | --- |
|
||
| 1 | kind `33302`, addressable | `cord02::list::KIND_COMMUNITY_LIST = 13302` (retired) |
|
||
| 2 | one event per fragment, `d` = index, `frags` declared | no `frags`, single event, `d` unused, `load_list` `.limit(1)` |
|
||
| 3 | 32-byte values unpadded base64url at any depth | hex: `JoinMaterial.owner`/`control_root` (`PublicKey`/`String`), `CommunityId` serde, `ChannelGrant.key` |
|
||
| 4 | `seed` omitted when equal to `current`; embedded snapshot omits `community_id`; `seed`'s cosmetic fields rewritten from `current` | both snapshots always serialized verbatim; `community_id` always present |
|
||
| 5 | fetch from relays | local database only |
|
||
| 6 | materialize `CommunityState` from join material | no such path; only `CommunityState::from_genesis` |
|
||
| 7 | publish the List on create/join (read-modify-write) | `build_list_event` is referenced only by tests and docs |
|
||
| 8 | private channel keys ride in join material | `ChannelKeyRef` has no key field |
|
||
|
||
Divergences 1–4 meant that even if the fetch existed, coop could neither read
|
||
what accordion wrote nor write something accordion could read. **Phases A and B
|
||
are done**, so 1–4 and 6 are resolved; 5, 7 and 8 remain (8 only in that private
|
||
planes are still not subscribed).
|
||
|
||
## Plan
|
||
|
||
Ordered so each phase is independently reviewable and testable. Nothing here
|
||
touches the frozen HKDF derivations or `cord01` envelope semantics.
|
||
|
||
### Phase A — make the List interoperable (pure, no I/O) — DONE
|
||
|
||
`crates/concord/src/cords/cord02/list.rs`
|
||
|
||
1. `KIND_COMMUNITY_LIST` → `33302`; add `frags: u64` to `CommunityList` and
|
||
`is_complete(&self, frags) -> bool`.
|
||
2. Add a base64url codec for the §8 value set and apply it to every 32-byte
|
||
field at every depth. Because `JoinMaterial` currently types `owner` and
|
||
`control_pk` as `PublicKey` (nostr's hex serde), this needs either wire
|
||
newtypes or `serialize_with`/`deserialize_with` helpers. Keep it local to the
|
||
List: `cord05` stays hex.
|
||
3. Implement the two §8 MUSTs: omit `community_id` on an embedded snapshot,
|
||
omit `seed` when it byte-equals `current`, and rewrite `seed`'s cosmetic
|
||
fields (`name`, `relays`, each channel's `name`) from `current` on every
|
||
serialization.
|
||
4. `build_list_event`/`parse_list_event` take the fragment index and emit/read
|
||
the `d` tag.
|
||
|
||
Tests: round-trip the `examples.md` §6.2 payload verbatim; `merge` convergence
|
||
for two devices and mixed-age fragments; `frags` disagreement resolves to the
|
||
larger value; a repack does not shed unknown fields.
|
||
|
||
**As built.** The §8 rules live behind private wire structs (`WireList`,
|
||
`WireEntry`, `WireSnapshot`, `WireChannel`), so a writer re-encodes on every
|
||
serialization while the public types keep their internal hex/`PublicKey`
|
||
spellings and `cord05` stays hex. Three deviations from the sketch above:
|
||
|
||
- `is_complete` takes the set of fragment indices a client holds, not a count:
|
||
a count is wrong when the indices are sparse.
|
||
- The reader tolerates non-zero base64url trailing bits. The spec's own §6.2
|
||
example has five such values, so a strict decoder rejects the worked example;
|
||
the writer still emits the canonical spelling.
|
||
- The third omission MUST was implemented too: an entry whose `added_at` does
|
||
not outrun its tombstone is not written. It is a serialization rule exactly
|
||
like the other two, so it belongs here rather than in Phase D.
|
||
|
||
`parse_list_event` validates the `d` tag but returns just the `CommunityList`;
|
||
`fragment_index(event)` reads the index, which keeps `sync.rs` untouched until
|
||
Phase C. `MAX_MEMBERSHIPS = 50` is kept for now as a stopgap (see risks): §8 has
|
||
no membership limit, and Phase D's fragmentation is what removes the cap.
|
||
|
||
### Phase B — materialize a community from join material (pure) — DONE
|
||
|
||
`crates/concord/src/store.rs`, `crates/concord/src/cords/cord02/list.rs`
|
||
|
||
1. `CommunityState::from_join_material(material: &JoinMaterial, added_at_ms:
|
||
u64) -> Result<Self>`: identity/owner/salt/root/root_epoch from the material;
|
||
`control_pks = { root_epoch → control_pk }`; `relays` parsed; `channels` from
|
||
the grants; `control_root` when present; `heads` empty (the first control
|
||
fold fills them); `banned` empty; `dissolved` false.
|
||
2. Carry the private channel key: add `key: Option<[u8; 32]>` to
|
||
`ChannelKeyRef` (or a parallel map) so a grant's `key` has a home. Without
|
||
this, a private channel is silently read-only-until-rekey.
|
||
|
||
Tests: a material with and without `control_root`; a private grant's key
|
||
survives; `from_join_material` then `planes()` yields the control `control_pk`
|
||
plus the guestbook and public channels, i.e. a subscription filter that
|
||
addresses real planes.
|
||
|
||
**As built.** `from_join_material` does not verify `community_id` against
|
||
`owner`/`owner_salt`: the List is signed by the member's own key and encrypted
|
||
to self, and the invite path already validates that binding in
|
||
`CommunityInvite::validate`. `private` on a materialized channel is simply
|
||
`key.is_some()` — the spec's `channels` carry only the Private Channel keys a
|
||
member was granted, so a grant with no key is a public channel. Nothing else
|
||
changed: `from_genesis` and `apply_fold` construct every channel with
|
||
`key: None`, and `planes()` still skips private channels, whose address derives
|
||
from the granted key rather than the `community_root`. Carrying the key is what
|
||
makes subscribing to them possible later; it is not needed to fix discovery.
|
||
|
||
Two tests. In `concord`, `from_join_material` (with and without `control_root`,
|
||
a granted key surviving, a public grant staying keyless). In `community`,
|
||
`planes()` plus `subscription_filter` over a state built field-by-field (control
|
||
+ guestbook + public channel addressed, private skipped) — `JoinMaterial` and
|
||
`ChannelGrant` cannot be constructed from `community` because their `extra`
|
||
field's type is crate-private, so the materialization and the plane derivation
|
||
are each proved where they live.
|
||
|
||
### Phase C — fetch the List from relays, then load
|
||
|
||
`crates/community/src/sync.rs`
|
||
|
||
1. `load` becomes:
|
||
- resolve where to ask: the account's NIP-65 write relays (kind `10002`) plus
|
||
the pool's connected relays. If only the app's bootstrap relays are queried,
|
||
a List published by another client (e.g. accordion on `relay.damus.io` /
|
||
`nos.lol`) will simply not be found.
|
||
- `client.fetch_events(Filter::new().kind(33302).author(self_pk))` — one
|
||
filter returns every fragment. Fetched events are persisted by the client
|
||
(`nostr-sdk/src/relay/inner.rs:1291`), so the database read stays valid.
|
||
- merge fragments → `CommunityList`.
|
||
- for each entry whose `is_live(&id)`: if a state document exists, keep its
|
||
`heads` (the fold's authority) and refresh relays/keys from `current`;
|
||
otherwise `from_join_material(..)`.
|
||
- `store::save_state` each result so the next `load` is warm.
|
||
2. `load_list` keeps reading `client.database()` — after the fetch it is
|
||
populated. It must stop using `.limit(1)`.
|
||
3. Drop the `states.retain(..)` shape: the List is now the *source* of states,
|
||
not just a filter over local ones. A local state whose membership is
|
||
tombstoned is still dropped, but a List entry with no local state now
|
||
produces one.
|
||
|
||
Tests (no network, `nostr-memory`): a `33302` fragment written by the account is
|
||
discovered with **no** state document present; a tombstoned id is dropped; a
|
||
missing fragment leaves the rest usable. A `nostr_sdk::local_relay::LocalRelay`
|
||
(in-process relay, public in this pinned revision) can drive the real
|
||
fetch/subscribe path end to end.
|
||
|
||
### Phase D — publish
|
||
|
||
`crates/community/src/sync.rs`, `crates/concord/src/store.rs`
|
||
|
||
1. `create` appends to the List and publishes the fragment read-modify-write per
|
||
§8, targeting the metadata's relays.
|
||
2. `create` publishes the genesis wraps to those relays. Today it only
|
||
`client.database().save_event(wrap)`s, so a created community is invisible to
|
||
every other account.
|
||
3. Leave uses a tombstone; a repack requires the complete List and is a
|
||
non-goal until memberships outgrow one fragment.
|
||
|
||
### Phase E — verify live
|
||
|
||
`RUST_LOG=info cargo run -p coop`, sign in with the accordion account that
|
||
already belongs to communities. Expect `community {id}: subscribing to ..` and
|
||
rows in the sidebar. This is the first time the path can be exercised at all.
|
||
|
||
## Validation per phase
|
||
|
||
- `cargo test -p concord` (A, B), `cargo test -p community` (B, C, D).
|
||
- `cargo clippy --workspace --all-targets`, `cargo fmt --all -- --check`.
|
||
- A is provable against the spec's worked example, so it needs no relay.
|
||
- C is provable with `nostr-memory` + `LocalRelay`, so it needs no network.
|
||
- E is the only step that needs real relays.
|
||
|
||
## Risks and open decisions
|
||
|
||
- **Base64url is case-significant and coop's ids are hex everywhere else.**
|
||
Confine the codec to `cord02::list`; any normalisation that case-folds will
|
||
silently corrupt §8 values. **Resolved in Phase A**: the codec is private to
|
||
`list.rs` and never case-folds.
|
||
- **`MAX_MEMBERSHIPS = 50` is not in the spec.** §8 has no membership limit; its
|
||
only bound is the 65,536-byte *encoded event*. `fits()` still measures the
|
||
NIP-44 plaintext, which understates that by roughly a third, so the count cap is
|
||
kept as a conservative stopgap until Phase D measures the built event and
|
||
fragments on write.
|
||
- **Relay selection for the fetch is the difference between finding the account's
|
||
List and not.** NIP-65 write relays + pool, or a user-visible relay setting?
|
||
- **Private channels stay unsubscribed until `planes()` derives their address
|
||
from the granted key** (Phase B gave `ChannelKeyRef` a home for it, but the
|
||
discovery fix does not need it). Public discovery works regardless.
|
||
- **Two writers, one key.** Once coop publishes `33302`, an account used from
|
||
both accordion and coop has both clients writing the List. §8's
|
||
read-modify-write is what keeps that from losing memberships — it is not
|
||
optional.
|
||
- **`store::save_state` signs with a per-process random key.** Harmless while it
|
||
stays local, but it means the state document can never be published or
|
||
compared; if a future phase wants it on the wire, it needs the account signer.
|