coop/docs/concord-community-discovery-plan.md

Concord discovery: why no community ever reaches subscribe

Audit + fix plan. Read alongside docs/concord-usage.md and docs/concord-simplification-plan.md.

Symptom

crates/community/src/lib.rs::subscribe is never called, so no wrap is ever subscribed to and the sidebar stays empty. community load: 0 state document(s) found is the only clue.

Root cause

CommunityRegistry::load only ever reads the local database. Nothing in the discovery path touches a relay.

community::init
  └─ SignerChanged → load
       └─ sync::load
            ├─ store::load_states(client)   → client.database().query(..)   // local only
            └─ load_list(client, ..)        → client.database().query(..)   // local only, .limit(1)
       → track([])
       → sync_subscriptions: `for community in self.communities` runs zero times
       → subscribe never called
       → no relay is ever queried
       → the database never fills
       → load stays empty forever

The loop is self-reinforcing: the local database is populated by the subscriptions that the empty load prevents. That is why an account which belongs to several communities in another client still shows nothing — a fresh install has no concord/* state document, and coop has no way to ask for one.

Confirmed by inspection:

Location	What it does
crates/community/src/lib.rs:167-191	load → sync::load, then track(states)
crates/community/src/sync.rs:125-137	load = store::load_states + load_list
crates/concord/src/store.rs:314-341	load_states queries client.database() only
crates/community/src/sync.rs:139-153	load_list queries client.database() only, .limit(1)
crates/community/src/lib.rs:233-276	sync_subscriptions skips everything when communities is empty

subscribe itself is correct. Do not debug it.

What the protocol actually says

Read from the spec (concord-protocol/concord, the submodule referenced by accordion.chat): 02.md §8 and examples.md §6.2.

A member's memberships live in the Community List, on relays:

• Kind 33302, addressable, NIP-44-encrypted to self, signed by the member's real key, one event per fragment with d = the fragment index in decimal ("0", "1", …). 13302 is explicitly retired ("the single-event Community List, superseded by 33302 once it outgrew one event — a replaceable kind cannot fragment", 02.md:314).
• Every 32-byte value at any depth is unpadded base64url, not hex. This is section-scoped: CORD-05 invite fields stay hex (examples.md §6.3).
• Join material is the membership subset — owner, owner_salt, community_root, root_epoch, control_pk, channels, relays, name, plus control_root when held. It is the only durable home of a member's keys.
• The two snapshots solve opposite problems: seed is the earliest epoch held (backfill anchor), current the latest ("so a fresh device reconstructs the Community instantly with no epoch-by-epoch walk"). seed is omitted when equal to current; embedded snapshots omit community_id (inherited).
• A client holds the complete List when it holds a fragment at every index below frags; it unions fragments and merges, so a partial read is safe.

Two consequences for coop:

1. The state document is a coop invention. store::{save_state, load_state, load_states} write kind 30078 with d = concord/<id>, signed by a per-process LOCAL_KEYS, and never leave the machine. No equivalent exists anywhere in the spec. It is a local cache and must never be treated as the discovery source.
2. Discovery is: subscribe to my 33302 → materialize a community from current join material → subscribe to its planes → fold. The fold produces the authoritative state; the List only supplies the keys to start.

Divergences (coop vs spec)

#	Spec	coop today
1	kind 33302, addressable	cord02::list::KIND_COMMUNITY_LIST = 13302 (retired)
2	one event per fragment, d = index, frags declared	no frags, single event, d unused, load_list .limit(1)
3	32-byte values unpadded base64url at any depth	hex: JoinMaterial.owner/control_root (PublicKey/String), CommunityId serde, ChannelGrant.key
4	seed omitted when equal to current; embedded snapshot omits community_id; seed's cosmetic fields rewritten from current	both snapshots always serialized verbatim; community_id always present
5	fetch from relays	local database only — fixed in Phase C
6	materialize CommunityState from join material	no such path; only CommunityState::from_genesis
7	publish the List on create/join (read-modify-write)	build_list_event is referenced only by tests and docs
8	private channel keys ride in join material	ChannelKeyRef has no key field

Divergences 1–4 meant that even if the fetch existed, coop could neither read what accordion wrote nor write something accordion could read. Phases A, B and C are done, so 1–6 are resolved; 7 and 8 remain (8 only in that private planes are still not subscribed).

Plan

Ordered so each phase is independently reviewable and testable. Nothing here touches the frozen HKDF derivations or cord01 envelope semantics.

Phase A — make the List interoperable (pure, no I/O) — DONE

crates/concord/src/cords/cord02/list.rs

1. KIND_COMMUNITY_LIST → 33302; add frags: u64 to CommunityList and is_complete(&self, frags) -> bool.
2. Add a base64url codec for the §8 value set and apply it to every 32-byte field at every depth. Because JoinMaterial currently types owner and control_pk as PublicKey (nostr's hex serde), this needs either wire newtypes or serialize_with/deserialize_with helpers. Keep it local to the List: cord05 stays hex.
3. Implement the two §8 MUSTs: omit community_id on an embedded snapshot, omit seed when it byte-equals current, and rewrite seed's cosmetic fields (name, relays, each channel's name) from current on every serialization.
4. build_list_event/parse_list_event take the fragment index and emit/read the d tag.

Tests: round-trip the examples.md §6.2 payload verbatim; merge convergence for two devices and mixed-age fragments; frags disagreement resolves to the larger value; a repack does not shed unknown fields.

As built. The §8 rules live behind private wire structs (WireList, WireEntry, WireSnapshot, WireChannel), so a writer re-encodes on every serialization while the public types keep their internal hex/PublicKey spellings and cord05 stays hex. Three deviations from the sketch above:

• is_complete takes the set of fragment indices a client holds, not a count: a count is wrong when the indices are sparse.
• The reader tolerates non-zero base64url trailing bits. The spec's own §6.2 example has five such values, so a strict decoder rejects the worked example; the writer still emits the canonical spelling.
• The third omission MUST was implemented too: an entry whose added_at does not outrun its tombstone is not written. It is a serialization rule exactly like the other two, so it belongs here rather than in Phase D.

parse_list_event validates the d tag but returns just the CommunityList; fragment_index(event) reads the index, which keeps sync.rs untouched until Phase C. MAX_MEMBERSHIPS = 50 is kept for now as a stopgap (see risks): §8 has no membership limit, and Phase D's fragmentation is what removes the cap.

Phase B — materialize a community from join material (pure) — DONE

crates/concord/src/store.rs, crates/concord/src/cords/cord02/list.rs

1. CommunityState::from_join_material(material: &JoinMaterial, added_at_ms: u64) -> Result<Self>: identity/owner/salt/root/root_epoch from the material; control_pks = { root_epoch → control_pk }; relays parsed; channels from the grants; control_root when present; heads empty (the first control fold fills them); banned empty; dissolved false.
2. Carry the private channel key: add key: Option<[u8; 32]> to ChannelKeyRef (or a parallel map) so a grant's key has a home. Without this, a private channel is silently read-only-until-rekey.

Tests: a material with and without control_root; a private grant's key survives; from_join_material then planes() yields the control control_pk plus the guestbook and public channels, i.e. a subscription filter that addresses real planes.

As built. from_join_material does not verify community_id against owner/owner_salt: the List is signed by the member's own key and encrypted to self, and the invite path already validates that binding in CommunityInvite::validate. private on a materialized channel is simply key.is_some() — the spec's channels carry only the Private Channel keys a member was granted, so a grant with no key is a public channel. Nothing else changed: from_genesis and apply_fold construct every channel with key: None, and planes() still skips private channels, whose address derives from the granted key rather than the community_root. Carrying the key is what makes subscribing to them possible later; it is not needed to fix discovery.

Two tests. In concord, from_join_material (with and without control_root, a granted key surviving, a public grant staying keyless). In community, planes() plus subscription_filter over a state built field-by-field (control

• guestbook + public channel addressed, private skipped) — JoinMaterial and ChannelGrant cannot be constructed from community because their extra field's type is crate-private, so the materialization and the plane derivation are each proved where they live.

Phase C — the List drives load — DONE

crates/community/src/sync.rs, crates/community/src/lib.rs

1. subscribe_list(client, self_pk) subscribes to Kind::Custom(33302) author(self_pk) under a dedicated concord/list subscription id, using ReqTarget::auto. With gossip enabled, auto breaks the filter down by author, so it queries the account's NIP-65 write relays and adds/connects them itself — bootstrap relays alone would miss a List published elsewhere.
2. CommunityRegistry calls subscribe_list once per signer (signer change and the initial defer). It is deliberately not called from load: re-subscribing on every List event would re-deliver the List and loop. reset does not unsubscribe it either — subscribe_list replaces the subscription itself, and a reset-issued unsubscribe could race the replacement and cancel discovery.
3. The notification listener routes a concord/list event to a new Signal::List, whose consumer re-runs load. Community planes keep using Signal::Event(id).
4. load_list reads every 33302 event by self_pk from the database, keeps the newest event per fragment index, decrypts and merges them. .limit(1) is gone. An incomplete List is read normally — a missing fragment is news not yet heard.
5. load unions two sources: every live List entry (materialized with from_join_material, or refreshed if a state document already exists) and every held local state the List does not mention. A held membership is dropped only when a tombstone outranks its added_at_ms; absence from the List is never a fact. Each list-derived state is save_stated, so the next load is warm.
6. refresh(held, fresh) keeps the fold's authority (heads, banned, dissolved) and the control planes it learned, and takes the List's identity, relays, and channel keys. Channels are merged by id rather than replaced, so a public channel the fold discovered is not shed by a List snapshot that predates it.

As built, deviating from the sketch above. The plan called for client.fetch_events(..); the SDK's own recommendation is to keep the request path on a subscription and read the database. This is safer than it sounds: a relay's event is persisted at nostr-sdk/src/relay/inner.rs:1291 before the notification is emitted, so a subscription plus a database read loses nothing and needs no explicit save. The subscription is set up with ReqTarget::auto rather than a hand-built NIP-65 relay map, because gossip already resolves the author's write relays and connects them on demand.

Tests (no network): a fragment in the database with no state document materializes a community and writes one; a tombstone at u64::MAX drops a held membership; a two-fragment List with only fragment 0 delivered still yields its membership; a held membership the List never mentions is kept alongside the discovered one; refresh keeps heads/banned/dissolved and both control planes while taking the List's keys; and the concord/list id is not read as a community subscription. Fragment events are built with build_list_event from a §8 JSON payload, so the test exercises the real decrypt-and-merge path without a relay.

Phase D — publish

crates/community/src/sync.rs, crates/concord/src/store.rs

1. create appends to the List and publishes the fragment read-modify-write per §8, targeting the metadata's relays.
2. create publishes the genesis wraps to those relays. Today it only client.database().save_event(wrap)s, so a created community is invisible to every other account.
3. Leave uses a tombstone; a repack requires the complete List and is a non-goal until memberships outgrow one fragment.

Phase E — verify live

RUST_LOG=info cargo run -p coop, sign in with the accordion account that already belongs to communities. Expect community {id}: subscribing to .. and rows in the sidebar. This is the first time the path can be exercised at all.

Validation per phase

• cargo test -p concord (A, B), cargo test -p community (B, C, D).
• cargo clippy --workspace --all-targets, cargo fmt --all -- --check.
• A is provable against the spec's worked example, so it needs no relay.
• C is provable with nostr-memory: fragments are built with build_list_event and saved as the subscription would have, then load reads them. No relay, no LocalRelay.
• E is the only step that needs real relays.

Risks and open decisions

• Base64url is case-significant and coop's ids are hex everywhere else. Confine the codec to cord02::list; any normalisation that case-folds will silently corrupt §8 values. Resolved in Phase A: the codec is private to list.rs and never case-folds.
• MAX_MEMBERSHIPS = 50 is not in the spec. §8 has no membership limit; its only bound is the 65,536-byte encoded event. fits() still measures the NIP-44 plaintext, which understates that by roughly a third, so the count cap is kept as a conservative stopgap until Phase D measures the built event and fragments on write.
• Relay selection is the difference between finding the account's List and not. Resolved in Phase C by ReqTarget::auto, whose gossip path resolves the filter's author to their NIP-65 write relays and connects them. A List published only to relays with no NIP-65 entry is still unreachable; that is a user-visible relay setting if it ever bites.
• Private channels stay unsubscribed until planes() derives their address from the granted key (Phase B gave ChannelKeyRef a home for it, but the discovery fix does not need it). Public discovery works regardless.
• Two writers, one key. Once coop publishes 33302, an account used from both accordion and coop has both clients writing the List. §8's read-modify-write is what keeps that from losing memberships — it is not optional.
• store::save_state signs with a per-process random key. Harmless while it stays local, but it means the state document can never be published or compared; if a future phase wants it on the wire, it needs the account signer.