reya/coop
1
0
Fork 0
Code Issues Pull Requests 1 Releases 3 Activity

refactor

Browse Source
This commit is contained in:
reya
2026-09-19 08:39:25 +07:00
parent 0328d35945
commit 1cf5ecba51
9 changed files with 951 additions and 71 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/concord-community-discovery-plan.md
+241
View File
@@ -0,0 +1,241 @@
# Concord discovery: why no community ever reaches `subscribe`
Audit + fix plan. Read alongside `docs/concord-usage.md` and
`docs/concord-simplification-plan.md`.
## Symptom
`crates/community/src/lib.rs::subscribe` is never called, so no wrap is ever
subscribed to and the sidebar stays empty. `community load: 0 state document(s)
found` is the only clue.
## Root cause
`CommunityRegistry::load` only ever reads the **local database**. Nothing in the
discovery path touches a relay.
```
community::init
└─ SignerChanged → load
└─ sync::load
├─ store::load_states(client) → client.database().query(..) // local only
└─ load_list(client, ..) → client.database().query(..) // local only, .limit(1)
→ track([])
→ sync_subscriptions: `for community in self.communities` runs zero times
→ subscribe never called
→ no relay is ever queried
→ the database never fills
→ load stays empty forever
```
The loop is self-reinforcing: the local database is populated *by* the
subscriptions that the empty load prevents. That is why an account which belongs
to several communities in another client still shows nothing — a fresh install
has no `concord/*` state document, and coop has no way to ask for one.
Confirmed by inspection:
| Location | What it does |
| --- | --- |
| `crates/community/src/lib.rs:167-191` | `load` → `sync::load`, then `track(states)` |
| `crates/community/src/sync.rs:125-137` | `load` = `store::load_states` + `load_list` |
| `crates/concord/src/store.rs:314-341` | `load_states` queries `client.database()` only |
| `crates/community/src/sync.rs:139-153` | `load_list` queries `client.database()` only, `.limit(1)` |
| `crates/community/src/lib.rs:233-276` | `sync_subscriptions` skips everything when `communities` is empty |
`subscribe` itself is correct. Do not debug it.
## What the protocol actually says
Read from the spec (`concord-protocol/concord`, the submodule referenced by
accordion.chat): `02.md` §8 and `examples.md` §6.2.
A member's memberships live in the **Community List**, on relays:
- **Kind `33302`**, addressable, NIP-44-encrypted to self, signed by the
member's real key, one event per **fragment** with `d` = the fragment index in
decimal (`"0"`, `"1"`, …). `13302` is explicitly **retired** ("the
single-event Community List, superseded by `33302` once it outgrew one event —
a replaceable kind cannot fragment", `02.md:314`).
- Every 32-byte value at **any depth** is unpadded base64url, not hex. This is
section-scoped: CORD-05 invite fields stay hex (`examples.md` §6.3).
- Join material is the membership subset — `owner, owner_salt, community_root,
root_epoch, control_pk, channels, relays, name`, plus `control_root` when
held. It is the *only* durable home of a member's keys.
- The two snapshots solve opposite problems: `seed` is the earliest epoch held
(backfill anchor), `current` the latest ("so a fresh device reconstructs the
Community instantly with no epoch-by-epoch walk"). `seed` is omitted when
equal to `current`; embedded snapshots omit `community_id` (inherited).
- A client holds the complete List when it holds a fragment at every index below
`frags`; it unions fragments and merges, so a partial read is safe.
Two consequences for coop:
1. **The state document is a coop invention.** `store::{save_state, load_state,
load_states}` write kind `30078` with `d = concord/<id>`, signed by a
per-process `LOCAL_KEYS`, and never leave the machine. No equivalent exists
anywhere in the spec. It is a local cache and must never be treated as the
discovery source.
2. **Discovery is: fetch my `33302` from relays → materialize a community from
`current` join material → subscribe to its planes → fold.** The fold produces
the authoritative state; the List only supplies the keys to start.
## Divergences (coop vs spec)
| # | Spec | coop today |
| --- | --- | --- |
| 1 | kind `33302`, addressable | `cord02::list::KIND_COMMUNITY_LIST = 13302` (retired) |
| 2 | one event per fragment, `d` = index, `frags` declared | no `frags`, single event, `d` unused, `load_list` `.limit(1)` |
| 3 | 32-byte values unpadded base64url at any depth | hex: `JoinMaterial.owner`/`control_root` (`PublicKey`/`String`), `CommunityId` serde, `ChannelGrant.key` |
| 4 | `seed` omitted when equal to `current`; embedded snapshot omits `community_id`; `seed`'s cosmetic fields rewritten from `current` | both snapshots always serialized verbatim; `community_id` always present |
| 5 | fetch from relays | local database only |
| 6 | materialize `CommunityState` from join material | no such path; only `CommunityState::from_genesis` |
| 7 | publish the List on create/join (read-modify-write) | `build_list_event` is referenced only by tests and docs |
| 8 | private channel keys ride in join material | `ChannelKeyRef` has no key field |
Divergences 1–4 meant that even if the fetch existed, coop could neither read
what accordion wrote nor write something accordion could read. **Phase A is
done**, so 1–4 are resolved; 5–8 remain.
## Plan
Ordered so each phase is independently reviewable and testable. Nothing here
touches the frozen HKDF derivations or `cord01` envelope semantics.
### Phase A — make the List interoperable (pure, no I/O) — DONE
`crates/concord/src/cords/cord02/list.rs`
1. `KIND_COMMUNITY_LIST` → `33302`; add `frags: u64` to `CommunityList` and
`is_complete(&self, frags) -> bool`.
2. Add a base64url codec for the §8 value set and apply it to every 32-byte
field at every depth. Because `JoinMaterial` currently types `owner` and
`control_pk` as `PublicKey` (nostr's hex serde), this needs either wire
newtypes or `serialize_with`/`deserialize_with` helpers. Keep it local to the
List: `cord05` stays hex.
3. Implement the two §8 MUSTs: omit `community_id` on an embedded snapshot,
omit `seed` when it byte-equals `current`, and rewrite `seed`'s cosmetic
fields (`name`, `relays`, each channel's `name`) from `current` on every
serialization.
4. `build_list_event`/`parse_list_event` take the fragment index and emit/read
the `d` tag.
Tests: round-trip the `examples.md` §6.2 payload verbatim; `merge` convergence
for two devices and mixed-age fragments; `frags` disagreement resolves to the
larger value; a repack does not shed unknown fields.
**As built.** The §8 rules live behind private wire structs (`WireList`,
`WireEntry`, `WireSnapshot`, `WireChannel`), so a writer re-encodes on every
serialization while the public types keep their internal hex/`PublicKey`
spellings and `cord05` stays hex. Three deviations from the sketch above:
- `is_complete` takes the set of fragment indices a client holds, not a count:
a count is wrong when the indices are sparse.
- The reader tolerates non-zero base64url trailing bits. The spec's own §6.2
example has five such values, so a strict decoder rejects the worked example;
the writer still emits the canonical spelling.
- The third omission MUST was implemented too: an entry whose `added_at` does
not outrun its tombstone is not written. It is a serialization rule exactly
like the other two, so it belongs here rather than in Phase D.
`parse_list_event` validates the `d` tag but returns just the `CommunityList`;
`fragment_index(event)` reads the index, which keeps `sync.rs` untouched until
Phase C. `MAX_MEMBERSHIPS = 50` is kept for now as a stopgap (see risks): §8 has
no membership limit, and Phase D's fragmentation is what removes the cap.
### Phase B — materialize a community from join material (pure)
`crates/concord/src/store.rs`, `crates/concord/src/cords/cord02/list.rs`
1. `CommunityState::from_join_material(material: &JoinMaterial, added_at_ms:
u64) -> Result<Self>`: identity/owner/salt/root/root_epoch from the material;
`control_pks = { root_epoch → control_pk }`; `relays` parsed; `channels` from
the grants; `control_root` when present; `heads` empty (the first control
fold fills them); `banned` empty; `dissolved` false.
2. Carry the private channel key: add `key: Option<[u8; 32]>` to
`ChannelKeyRef` (or a parallel map) so a grant's `key` has a home. Without
this, a private channel is silently read-only-until-rekey.
Tests: a material with and without `control_root`; a private grant's key
survives; `from_join_material` then `planes()` yields the control `control_pk`
plus the guestbook and public channels, i.e. a subscription filter that
addresses real planes.
### Phase C — fetch the List from relays, then load
`crates/community/src/sync.rs`
1. `load` becomes:
- resolve where to ask: the account's NIP-65 write relays (kind `10002`) plus
the pool's connected relays. If only the app's bootstrap relays are queried,
a List published by another client (e.g. accordion on `relay.damus.io` /
`nos.lol`) will simply not be found.
- `client.fetch_events(Filter::new().kind(33302).author(self_pk))` — one
filter returns every fragment. Fetched events are persisted by the client
(`nostr-sdk/src/relay/inner.rs:1291`), so the database read stays valid.
- merge fragments → `CommunityList`.
- for each entry whose `is_live(&id)`: if a state document exists, keep its
`heads` (the fold's authority) and refresh relays/keys from `current`;
otherwise `from_join_material(..)`.
- `store::save_state` each result so the next `load` is warm.
2. `load_list` keeps reading `client.database()` — after the fetch it is
populated. It must stop using `.limit(1)`.
3. Drop the `states.retain(..)` shape: the List is now the *source* of states,
not just a filter over local ones. A local state whose membership is
tombstoned is still dropped, but a List entry with no local state now
produces one.
Tests (no network, `nostr-memory`): a `33302` fragment written by the account is
discovered with **no** state document present; a tombstoned id is dropped; a
missing fragment leaves the rest usable. A `nostr_sdk::local_relay::LocalRelay`
(in-process relay, public in this pinned revision) can drive the real
fetch/subscribe path end to end.
### Phase D — publish
`crates/community/src/sync.rs`, `crates/concord/src/store.rs`
1. `create` appends to the List and publishes the fragment read-modify-write per
§8, targeting the metadata's relays.
2. `create` publishes the genesis wraps to those relays. Today it only
`client.database().save_event(wrap)`s, so a created community is invisible to
every other account.
3. Leave uses a tombstone; a repack requires the complete List and is a
non-goal until memberships outgrow one fragment.
### Phase E — verify live
`RUST_LOG=info cargo run -p coop`, sign in with the accordion account that
already belongs to communities. Expect `community {id}: subscribing to ..` and
rows in the sidebar. This is the first time the path can be exercised at all.
## Validation per phase
- `cargo test -p concord` (A, B), `cargo test -p community` (B, C, D).
- `cargo clippy --workspace --all-targets`, `cargo fmt --all -- --check`.
- A is provable against the spec's worked example, so it needs no relay.
- C is provable with `nostr-memory` + `LocalRelay`, so it needs no network.
- E is the only step that needs real relays.
## Risks and open decisions
- **Base64url is case-significant and coop's ids are hex everywhere else.**
Confine the codec to `cord02::list`; any normalisation that case-folds will
silently corrupt §8 values. **Resolved in Phase A**: the codec is private to
`list.rs` and never case-folds.
- **`MAX_MEMBERSHIPS = 50` is not in the spec.** §8 has no membership limit; its
only bound is the 65,536-byte *encoded event*. `fits()` still measures the
NIP-44 plaintext, which understates that by roughly a third, so the count cap is
kept as a conservative stopgap until Phase D measures the built event and
fragments on write.
- **Relay selection for the fetch is the difference between finding the account's
List and not.** NIP-65 write relays + pool, or a user-visible relay setting?
- **Private channels stay unreadable until `ChannelKeyRef` carries the grant key**
(Phase B.2). Public discovery works without it.
- **Two writers, one key.** Once coop publishes `33302`, an account used from
both accordion and coop has both clients writing the List. §8's
read-modify-write is what keeps that from losing memberships — it is not
optional.
- **`store::save_state` signs with a per-process random key.** Harmless while it
stays local, but it means the state document can never be published or
compared; if a future phase wants it on the wire, it needs the account signer.
docs/concord-usage.md
+43 -9
View File
@@ -413,18 +413,47 @@ A member's own memberships, synced across their devices:
```rust
use concord::cord02::list;
let material = cord02::list::join_material(&invite, staff.then_some(&control_root));
let mut mine = cord02::list::parse_list_event(&my_keys, &event).await?;
mine = cord02::list::merge(mine, cord02::list::CommunityList {
entries: vec![cord02::list::CommunityListEntry { community_id, seed: material.clone(), current: material, added_at: now_ms, extra: Default::default() }],
..Default::default()
let material = list::join_material(&invite, staff.then_some(&control_root));
let mut mine = list::parse_list_event(&my_keys, &event).await?; // validates the d tag
mine = list::merge(mine, list::CommunityList {
entries: vec![list::CommunityListEntry { community_id, seed: material.clone(), current: material, added_at: now_ms, extra: Default::default() }],
..Default::default() // frags: 1
});
let event = cord02::list::build_list_event(&my_keys, &mine).await?; // kind 13302, NIP-44 to self
let event = list::build_list_event(&my_keys, &mine, 0).await?; // kind 33302, d = fragment 0
```
Kind `33302` is **addressable and fragmented**: one event per fragment, its `d`
tag the fragment index in decimal. `frags` in the payload declares how many the
List has, and `is_complete(held_indices)` answers whether the client has a
fragment at every index below it. `merge` resolves a `frags` disagreement to the
larger value. (`13302`, the single-event List, is retired by the spec — a
replaceable kind cannot fragment.)
The payload's 32-byte values are **unpadded base64url at every depth**, which is
section-scoped to §8: CORD-05 invites stay hex. The writer re-encodes them on
every serialization, so its output is always the canonical 43-character spelling;
the reader also accepts non-zero trailing bits, because the spec's own worked
example contains them and no reader can tell a mis-encoded named field from a
correct one. The codec is `utils::base64url` and the wire structs behind the
List's `Serialize`/`Deserialize` are the only callers, so no other encoding path
is touched.
Three write-time rules are folded into serialization, so an in-memory document
and its wire form differ:
- an embedded snapshot omits `community_id` and inherits the entry's;
- `seed` is omitted when it equals `current`, and its cosmetic fields (`name`,
`relays`, each channel's `name`) are overwritten from `current` first, so a
rename collapses the snapshots instead of forking them;
- an entry whose `added_at` does not outrun its tombstone is omitted — the
tombstone alone carries the state.
`is_live(&id)` answers joined-versus-left: a tombstone is terminal until a
strictly newer join outruns it. `fits()` is the write gate — 50 memberships and
the NIP-44 size cap, both protocol constants.
strictly newer join outruns it. `fits()` is the write gate: 50 memberships and
the NIP-44 plaintext cap. The 50 is a stopgap inherited from the retired
single-event design — §8 has **no membership limit**, its only bound is the
65,536-byte encoded event, and the real fix is to start a new fragment on write
(see `docs/concord-community-discovery-plan.md`, Phase D).
## GPUI integration
@@ -545,7 +574,12 @@ client.subscribe(filter).with_id(sub_id).await?;
re-folds on an inbound wrap. The sidebar observes the registry, logs
`CommunityEvent::Error` through `log::error!`, and its "New community" row opens
a name prompt that calls `CommunityRegistry::create`. `create` still persists
the genesis locally without publishing it to the metadata's relays.
the genesis locally without publishing it to the metadata's relays. Discovery
is local-only: `load` reads the state documents already in
`client.database()` and never fetches the account's CORD-02 Community List
(`33302`) from relays, so a fresh install — or one signing in as an account
that joined elsewhere — finds nothing and never subscribes. See
`docs/concord-community-discovery-plan.md`.
- **Account-key writers take any signer, not `&Keys`.** `genesis`,
`ControlWriter`, the guestbook and chat `seal_rumor`s, the `list` builders, and
the `cord05` invite writers (`build_direct_invite` / `unwrap_direct_invite`,