diff --git a/docs/concord-usage.md b/docs/concord-usage.md deleted file mode 100644 index 74c23a08..00000000 --- a/docs/concord-usage.md +++ /dev/null @@ -1,780 +0,0 @@ -# Using the Concord backend - -`crates/concord` is a protocol crate: derivations, envelopes, folds and the local -state document. It has no GPUI dependency and owns no strings a user reads — the -UI layer decides every rendering. This document is the map from a UI action to -the calls it makes. - -A community is addressed by a `community_id` (never on the wire) plus three -secrets: `community_root` (read access — holding it *is* membership), -`control_root` (write access to the Control Plane, held by staff), and per-Channel -keys for private channels. Authority is a roster of owner-rooted signed grants, -folded independently by every client. - -## Modules - -Files follow the CORD documents. The frozen derivations of Appendix A and the id -vocabulary are shared substrate — every document calls them — so they live outside -`cords` in `utils::derive` (re-exported as `derive`) and `types`. - -| Module | Owns | -| --- | --- | -| `cord01` | Private Streams: the seal/wrap envelope and the NIP-44 helpers | -| `cord02` | Communities: identity, epochs, metadata, the Control Plane fold and writer | -| `cord02::guestbook` | Joins, leaves, kicks, snapshots, the member list | -| `cord02::list` | The Community List (a member's own memberships, across devices) | -| `cord03` | Channels: Channel metadata and the Chat Plane | -| `cord04` | Roles: chained editions, parse/hash/fold, the roster, permissions, the banlist | -| `cord04::pins` | Pin Lists, and the key disclosure a keyless reader verifies | -| `cord05` | Invite bundles, links, the Direct Invite, the Invite List | -| `cord06` | Key rotations, refounding, compaction, dissolution | -| `derive` | Every frozen HKDF derivation and coordinate | -| `state` | The community state document and its pure folds — no I/O | - -`CommunityId`, `ChannelId`, `RoleId`, `Epoch` and `Extra` (crate-internal) come from -the private `types` module and are re-exported at the crate root. - -Nothing in `concord` touches a relay or a database. The local rumor cache, the -state documents and the relay paging walk live one crate up, in -`community::cache` and `community::history`, and are what a client actually calls -(read on below). - -CORD-07 (audio/video) is unimplemented. CORD-08's timer has no file of its own: it -lives in the metadata it reads (`cord02`) and the fold it filters (`cord03`). - -Read `CommunityId` as "this community", `ChannelId` as "this channel", `Epoch` as -"which key generation". Nothing else in the API needs internal state. - -## Creating a community - -```rust -use concord::cord02::{self, CommunityMetadata}; -use concord::state::CommunityState; -use community::cache::save_state; - -let metadata = CommunityMetadata { name: "Room".into(), ..Default::default() }; -let minted = cord02::genesis(&owner_keys, &metadata, now_secs).await?; - -// minted.identity — community_id, owner, owner_salt (verify() recomputes it) -// minted.wraps — the two owner-signed genesis editions, already sealed -// minted.channel_id — the #general channel -for wrap in &minted.wraps { - client.send_event(wrap).to(&relays).await?; -} -``` - -The owner then needs the folded state, which is also what every member does on -join: - -```rust -use concord::derive::{control_group_key, control_signer_group_key}; -use concord::cord04::ParsedEdition; - -let read = control_group_key(&minted.community_root, &minted.identity.community_id, Epoch(0))?; -let signer = control_signer_group_key(&minted.control_root, &minted.identity.community_id, Epoch(0))?; -let editions: Vec = minted - .wraps - .iter() - .map(|wrap| cord02::open_edition(wrap, &read, &signer.pk(), true)) - .collect::>()?; - -let mut state = CommunityState::from_genesis(&minted, &editions, added_at_ms)?; -save_state(&client, &state).await?; -``` - -Put the community's relay list into `state.relays` and add those relays to the -client explicitly — coop's client is a gossip client with no background refresh. - -Creating is not finished until the membership is announced. The two writes are -independent and both best-effort: the genesis wraps go to the community's -relays, and the membership goes to the account's own Community List (below), so a -new device — or another client — can find the community without an invite. -`crates/community`'s `sync::create` performs both. - -## Joining - -An invite link resolves to a bundle: - -```rust -use concord::cord05::{self, BundleState, invite_bundle_key}; - -let link = cord05::parse_link(url)?; // link_signer, token, bootstrap_relays, naddr - -// The crate does no I/O: fetch the naddr from the fragment's relays, then: -let invite = match cord05::parse_bundle_event(&event, &link.link_signer, &invite_bundle_key(&link.token))? { - BundleState::Live(invite) => invite, // validate() already ran - BundleState::Revoked => return Ok(None), // a tombstone at the coordinate -}; -``` - -A Direct Invite arrives as a NIP-59 gift wrap addressed to the member: - -```rust -let (inviter, invite) = cord05::unwrap_direct_invite(&wrap, &my_keys).await?; -``` - -Either way the invite carries `community_id`, `owner`, `owner_salt`, -`community_root`, `root_epoch`, `control_pk`, the granted `channels` -(`ChannelGrant { id, key, epoch, name }`) and the relay set. `invite.expired(now_ms)` -is a preview rule: a past expiry still renders, but the join is refused. - -Then publish a join so the member list sees the member before any backfill: - -```rust -use concord::derive::guestbook_group_key; -use concord::cord02::guestbook; - -let guestbook = guestbook_group_key(&invite.community_root, &invite.community_id, invite.root_epoch)?; -let rumor = cord02::guestbook::build_join(my_pk, Some((creator_npub, label)), now_ms); -let (wrap, _) = cord02::guestbook::seal_rumor(&rumor, &guestbook, &my_keys).await?; -client.send_event(&wrap).to(&relays).await?; -``` - -## Reading the Control Plane - -```rust -use concord::cord02::{self, ControlFold}; - -let editions: Vec = wraps - .iter() - .filter_map(|wrap| cord02::open_edition(wrap, &read, &control_pk, true).ok()) - .collect(); - -let control: ControlFold = - cord02::fold_control(&owner, &community_id, &editions, &state.floors(), &state.banned); -state.apply_fold(&control); -``` - -`ControlFold` is everything the community UI needs: - -| Field | Use | -| --- | --- | -| `community` | name, description, icon, banner, `message_expiration` | -| `channels` | the channel list; `deleted: true` means drop it | -| `roles` | `role()`, `roles_of()`, `effective_permissions()`, `is_authorized()`, `is_staff()` | -| `banned` | the banlist | -| `registries` / `is_public()` | each invite creator's live link signers | -| `pins` / `pin_content(id, channel)` | Pin List content per channel | -| `floors` | the committed heads the next fold is judged against | -| `gapped` | a chain hole: refetch the Control Plane before trusting what is missing | - -`None` on `community` or a channel means "this client saw no authorized edition", -never "the value is gone" — keep what the state already holds rather than walking -the community backwards. Feed `state.floors()` and `state.banned` into the next -fold; they are its memory. - -## Sending a message - -```rust -use concord::cord03::{self, build_message}; -use concord::derive::channel_group_key; - -let plane = channel_group_key(&community_root, &channel, epoch)?; // public channel -let rumor = build_message(my_pk, &channel, epoch, text, None, at_ms, timer); -let (wrap, wrap_key) = cord03::seal_rumor(&rumor, &plane, &my_keys, false).await?; -client.send_event(&wrap).to(&relays).await?; -``` - -- `epoch` is the **root** epoch for a public channel: its plane is - `channel_group_key(community_root, channel, root_epoch)`, so a Refounding moves - it along with the root. A private channel passes its own current channel epoch - (`state.channels` carries it) and derives from its own key instead of - `community_root`. -- `timer` is `control.community.message_expiration`; pass `None` when it is off. - The builder attaches the NIP-40 tag and `seal_rumor` mirrors it onto the wrap, - so relays drop the ciphertext too. -- `ephemeral: true` picks kind `21059` for typing indicators. Keep the returned - wrap key if the message may be deleted later — a kind-5 delete needs it. -- That `send_event` is the whole publish path; there is no optimistic echo. Feed - the wrap through the same ingest path the subscription uses so send-then-read - never waits on a relay round trip. - -`build_edit`, `build_reaction` and `build_delete` are the same shape, each a rumor -about an existing `EventId` rather than a mutation. - -## Reading a channel - -```rust -use concord::cord03::{self, fold, plane_keys}; -use community::cache; - -let planes = plane_keys(&held, &channel)?; // &[(Epoch, secret)] -let mut rumors = Vec::new(); - -for wrap in &wraps { - let Some((epoch, group)) = planes.iter().find(|(_, group)| group.pk() == wrap.pubkey) else { - continue; - }; - let Ok((opened, rumor)) = cord03::open(wrap, group, &channel, *epoch) else { - continue; - }; - cache::cache_rumor(&client, &channel, &opened).await?; - rumors.push(rumor); -} - -let messages = fold(&rumors, Timestamp::now(), |actor, citation, author| { - citation_ok(&owner, &community_id, actor, citation, &control.roles.floors) - && control.roles.can_act_on_member(actor, &owner, author, Permissions::MANAGE_MESSAGES) -}); -``` - -`ChatMessage` carries `content`, `at_ms`, `edited_at`, `deleted`, `reactions`, -`reply_to` and `thread_root` already resolved. The fold drops expired rumors; a -`deleted` row is still returned so the timeline keeps its shape. - -Relay history pages through the local cache: - -```rust -use community::history::{self, PageRegistry, Window}; - -// Every key the client still holds for the channel, newest epoch first: the -// state's current key (`ChannelKeyRef::key` at `epoch`) plus every `priors` -// entry a rotation stepped off. A public channel derives one per held root. -let held: Vec = state.held_keys(&channel); - -// The same registry `CommunityRegistry` shares with its notification pump: it is -// what carries each page's EOSE and CLOSED back to the round waiting on it. -let page = history::page( - client, - &pages, - &channel, - &held, - &relays, - Window::opening(cursor), - 20, - 50, -) -.await?; -let cached = cache::query_rumors(&client, &channel, None, 50, Some(&cord03::ROW_KINDS)).await?; -``` - -A key's `retired_at` (a `Timestamp`, set when a rotation supersedes it) is a read -cutoff: a wrap at that epoch with a later `created_at` is refused, so a retired -epoch is history and never a live plane an ejected holder can keep writing into. - -A relay is only ever *asked*: `history::page` installs **one REQ per page** over -`ReqTarget::manual` for the community's own relays (which is what makes the client -verify a wrap, deduplicate it and persist it), waits for every relay to settle, -and then reads the page back out of the local database — no wrap is ever consumed -straight off the wire, and `fetch_events` is not used anywhere. The wrap lands in -the client's shared event store through that ordinary ingest path; `cache_rumor` -then puts the decrypted rumor beside it, and only the rumor is ever served to a -reader. - -EOSE and CLOSED are **not** watched inside the page. The registry's notification -pump is the only consumer of `client.notifications()`; it delivers one -`PageReport` per settling relay into the `PageRegistry` under the page's -subscription id, and the page waits on that channel with `PAGE_TIMEOUT`. The id -is opaque (`concord-history-`) because a NIP-01 id is capped at 64 characters, -and the registry is what maps it back to the waiting round. - -A page ends on EOSE. A CLOSED ends it as unanswered, *except* NIP-42's -`auth-required`: the SDK re-issues that REQ under the same subscription id once -the handshake completes, so the page waits for the resubscribed answer rather -than writing off a relay that only wanted to authenticate. - -`history::page` walks newest-first across every held epoch, caches what it opens, and -reports what it saw: `oldest`/`newest` feed the caller's `ChannelCursor`, -`exhausted` is earned only by a short page *after* history was seen, and an -all-empty answer sets `failed` so a later round re-asks instead of sealing the -channel at "no more history". `unreadable` counts the wraps the page reached that -no held key could open — sealed past the cutoff their key's rotation set, or bound -to another channel — because those are history the reader is missing, not history -that is not there. Page down with `Window::older_than(seen.oldest)`, -open a channel with `Window::opening(cursor)` (wide cold, `newest - 60s` warm), -and read the region between two cursors with `Window::between(..)`. A `Window` -carries `Timestamp`s, so its bounds go straight into a NIP-01 filter with no unit -conversion; only a reader's millisecond `at_ms` narrows to a second at the query. -`query_rumors` is the read path when the group keys are gone; pass `kinds` to -budget rows apart from the events that only decorate them. `cache::wrapper_index` -reads the cached rows back keyed by the wrap they came from, which is what lets -`sync::fold` observe author and message times without re-opening a wrap it already -cached, and `cache::purge_expired(client, &channel, now)` runs at the top of every -round — the timer is cooperative, so the local store is the artifact that has to -forget. - -`sync::fold` counts the same thing over the whole store, per channel, in -`Snapshot.unreadable`. `sync_round` sums a round's pages into -`Progress.unreadable`, and `Community` keeps both per channel: `progress(channel)` -is the last completed round, `unreadable(channel)` is the count, and -`missing_key(channel)`, `channel_removed_at(channel)`, `removed_at()` and -`stranded()` are the rest of the honest read surface a panel needs to tell an -empty room from one it cannot read. `Community::due(channel)` is the other half — -whether a round is worth asking for yet (a round for `Older` and an explicit retry -never ask it), and `Community::tick`, driven by `CommunityRegistry` once per -`community::MIN_ROUND_INTERVAL`, re-folds and re-rounds the active channel of a -community whose last round is older than `STALE_AFTER`. - -`ChatAction::TimerNotice { seconds }` is a policy notice, not a message: render it -as an inline row only when its author passes -`control.roles.is_authorized(&author, &owner, Permissions::MANAGE_METADATA)`. - -## Membership - -```rust -let states = cord02::guestbook::coalesce(&rumors, now_ms, &refounders, |actor, target, citation| { - citation_ok(&owner, &community_id, actor, citation, &control.roles.floors) - && control.roles.can_act_on_member(actor, &owner, target, Permissions::KICK) -}); -let members = cord02::guestbook::complete_memberlist(&states, &observed, &granted, &control.banned, &BTreeMap::new()); -``` - -- `refounders` is the set of npubs whose rotations minted an epoch this client - verified (`CommunityState.refounders`) — a snapshot chunk is honored only from - one of them, and an empty set honors none. A rotation that delivered this - client no key still names its minter, so the set is filled by any base rotation - whose continuity verifies against a root held (`rekey::walk`); a List entry's - `refounder`, read by `list::JoinMaterial::refounder`, is the other source. -- `observed` is npub → ms for every author this client has seen publish anything - usable, which is what makes a member visible before their Join arrives. Only - count it forward. -- `granted` is every npub the roster ranks; they are members with no Guestbook - entry at all. -- `banned_at` is empty today, so a ban is terminal in the fold. Fill it when the - banlist head's timestamp is plumbed through. -- Removal is three separate actions, composed by the caller: strip the grant - (immediate and cheap), then the kick directive, then — for a ban — the rotation - that actually enforces it. - -## Moderation writes - -Every Control Plane write goes through one writer and one edition shape: - -```rust -let writer = ControlWriter { author: my_pk, read: read.clone(), signer: signer.clone() }; -let head = control.floors.get(entity).cloned(); - -let (wrap, new_head) = writer.set_community_metadata( - &my_keys, &community_id, &metadata, head.as_ref(), citation, now_secs).await?; -``` - -`citation` is the `vac` the actor acts under — `None` only for the owner. Build it -from the folded Grant that ranks them (`AuthorityCitation { entity, version, hash }`) -and pass the head from the current fold, so the chain cannot silently fork. - -Wrappers: `set_community_metadata`, `set_channel_metadata`, `set_role`, -`set_grant`, `set_banlist`, `set_registry`, `set_pin_list`, plus raw `publish`. -A ban is a `set_banlist` followed by a base rekey; a kick is a `set_grant` with an -empty `role_ids` followed by `cord02::guestbook::build_kick`. - -## Pins - -```rust -use concord::cord04::pins; - -let entry = cord04::pins::build_entry(&opened_message, &plane, &channel)?; -let head_content = control.pin_content(&community_id, &channel).unwrap_or(""); -let read = cord04::pins::read_list(head_content, |epoch| channel_group_key(&root, &channel, epoch).ok()); -let content = cord04::pins::publishable(&read, channel_is_private, &plane, epoch)?; -let (wrap, _) = writer.set_pin_list( - &my_keys, &community_id, &channel, &content, head, citation, now_secs).await?; -``` - -Reading is verification: `read_list` decodes either content form (public, or -sealed under the channel key at the named epoch), and -`cord04::pins::verify_entry(entry, &channel)` returns a `VerifiedPin` with the proven -author, words and time — no history and no old keys needed. `read.sealed` means -the list is sealed under an epoch this client never held: show it as unavailable, -and never write from it (`publishable` refuses). `cord04::pins::killed_by(&pin, &delete)` -answers whether a folded kind-5 erases an entry. - -## Invites - -```rust -use concord::derive::{invite_bundle_key, TOKEN_LEN}; -use concord::cord05::{self, InviteEntry, InviteTombstone}; - -let token: [u8; TOKEN_LEN] = /* 16 bytes from any CSPRNG */; -let bundle_key = invite_bundle_key(&token); -let link_signer = Keys::generate(); -let bundle = cord05::build_bundle_event(&link_signer, &invite, &bundle_key)?; -let url = cord05::build_invite_url(BASE, &link_signer.public_key(), &token, &relays)?; -``` - -A link is a coordinate plus a fragment: the naddr fetches the bundle, the token -unlocks it, and the fragment names the relays to fetch from. -`cord05::stock_relays()` is what a fragment with no relays of its own means. - -The `link_signer` secret is what lets the creator refresh or retire the link, so -keep it against the token in the member's own Invite List — a local document -encrypted to self, exactly like the Community List: - -```rust -let mut list = cord05::parse_invite_list(&my_keys, &event).await?; -list.entries.push(InviteEntry { - token: HEXLOWER.encode(&token), - signer_sk: link_signer.secret_key().to_secret_hex(), - community_id: invite.community_id, - url, - label: None, - created_at: now_ms, - expires_at: None, - extra: Default::default(), -}); -let event = cord05::build_invite_list(&my_keys, &list).await?; // kind 13303 - -// Retiring is a tombstone, never a deletion: it beats a stale copy terminally. -list.tombstones.push(InviteTombstone { - token: HEXLOWER.encode(&token), - community_id: invite.community_id, - extra: Default::default(), -}); -``` - -`merge_invite_lists` merges two devices' copies, `is_live(&token_hex)` answers -whether a link still stands, and `fits()` is the write gate. - -## Rekeys, refounding and dissolution - -A rotation is authority plus delivery: `rekey_authorized(&control.roles, &owner, &me, permission, &removed)` -gates it, `plan_rotation(scope, epoch)` mints what it delivers, and -`build_rekey_chunks` seals one blob per remaining member: - -```rust -use concord::derive::epoch_key_commitment; -use concord::cord06::{self, RekeyScope, RotationPlan}; - -let scope = RekeyScope::Channel(channel_id); // or RekeyScope::Base -let plan = cord06::plan_rotation(scope, Epoch(epoch + 1))?; -let new_key = plan.new_key(); - -// A base rotation delivers the new control-plane keys beside the root; a channel -// rotation delivers only that channel's fresh key. -let (control_pk, control_root) = match &plan { - RotationPlan::Base(refounding) => { - let pk = refounding.signer(&community_id)?.pk().to_bytes(); - (Some(pk), is_staff.then_some(&refounding.new_control_root)) - } - RotationPlan::Channel { .. } => (None, None), -}; - -let mut blobs = Vec::with_capacity(members.len()); - -for member in &members { - blobs.push( - cord06::build_blob( - &my_keys, - member, - scope, - plan.epoch(), - &new_key, - control_pk.as_ref(), - control_root, - ) - .await?, - ); -} - -let rekey_group = cord06::rekey_group(scope, &community_root, &community_id, plan.epoch())?; -let wraps = cord06::build_rekey_chunks( - &my_keys, - &rekey_group, - scope, - plan.epoch(), - Epoch(epoch), - &epoch_key_commitment(Epoch(epoch), &community_root), - &blobs, - citation, - false, - now_secs, -) -.await?; -``` - -On the receiving side, `cord06::parse_rekey_chunk(&opened)` per wrap, then -`collect_rotations(&chunks)`, then `am_i_removed(&rotation, &me)` — which is -`None` until every chunk is held, because an incomplete set is never a removal. A -member finds their delivery with `find_my_blobs` / `open_blob`, and adopts the key -only if the plaintext binds to the scope and epoch they expect and its `prevcommit` -matches the key they already hold. Two concurrent rotations settle on `fork_winner`. - -`community::rekey::adopt` is that receiver, run against the local database: it -walks a scope forward one epoch at a time off the key actually held (never -waiving a gap, never adopting a fork), keeps each stepped-off key as a prior -with the rotation's publish time as its read cutoff, and reports a removal or a -strand when a complete rotation carries no blob for the member. - -`community::rekey::rotate` is the sender, run against the held state: it resolves -the epoch and key the scope is stepping off, refuses a rotation that would skip or -cut off its own rotator, delivers one blob per recipient (`Rewrite.recipients`, -which the caller names because a private channel's audience is not in the local -state), marks the rotation severed when it excludes somebody, and — for a base -scope — carries the settled Control Plane heads onto the new epoch's groups with -`cord06::compact`. `Community::rotate` checks `Rewrite::authorized` first, the same -`rekey_authorized` under the same permissions `adopt` applies, publishes to the -community's relays, and then adopts what it published through `rekey::adopt` -rather than by construction: a rotation this client wrote and one it received take -one path. - -The blob plaintext is a fixed-width binary record, but a signer's NIP-44 is -text-only, so `build_blob` carries it base64-encoded inside the envelope. -`open_blob` mirrors that, so the record layout and the `locator` are unchanged. - -Dissolution is owner-only and terminal: - -```rust -let rumor = cord06::dissolved_tombstone_rumor(owner_pk, &community_id, now_secs); -let wrap = cord06::seal_dissolved(&rumor, &community_id, &my_keys, now_secs).await?; - -// A receiver seals the community read-only on sight. -if cord06::verify_dissolved(&wrap, &identity) { - state.dissolved = true; -} -``` - -## The Community List - -A member's own memberships, synced across their devices: - -```rust -use concord::cord02::list; - -let entry = concord::state::list_entry(&state, &metadata.name); // state → §8 material -let held = list::parse_list_event(&my_keys, &event).await?; // validates the d tag -let mine = held.joined(entry); // community_id-keyed union -let event = list::build_list_event(&my_keys, &mine, 0, now_secs).await?; // kind 33302, d = 0 -client.send_event(&event).to_nip65().await?; // account's own write relays -``` - -The publish needs no separate database write: `send_event` persists the event -locally *before* it resolves targets, so the fragment is available to the -`concord/list` read path even if every relay is unreachable. - -`join_material(&invite, control_root)` makes the §8 material from a CORD-05 -invite; `concord::state::list_entry(state, name)` makes it from a `CommunityState`, and is -what a write path uses after a create, join or rename. `joined` and `tombstoned` -are the two mutations: both are `community_id`-keyed unions, so neither an append -nor a leave can lose a membership the other writer has. - -The entry is signed by the member's real key and sealed to that same key -(`seal_to_self`), so only the member's devices read it — a stranger's -`parse_list_event` fails rather than returning a partial list. Publishing goes to -the member's **NIP-65 write relays**, the same set the `concord/list` -subscription resolves for its `author` filter. - -Kind `33302` is **addressable and fragmented**: one event per fragment, its `d` -tag the fragment index in decimal. `frags` in the payload declares how many the -List has, and `is_complete(held_indices)` answers whether the client has a -fragment at every index below it. `merge` resolves a `frags` disagreement to the -larger value. (`13302`, the single-event List, is retired by the spec — a -replaceable kind cannot fragment.) - -A join material may also carry two hex extensions this crate does not write but -does read, because they are the only place a device that never held a rotation can -learn who minted an epoch: `refounder` names the npub whose Refounding minted the -entry's `root_epoch`, and `held_roots` is the retained prior epochs -(`[{epoch, key, refounder?, control_pk?, retired_at?}]`). `JoinMaterial::refounder()` -and `JoinMaterial::held_roots()` decode both; unknown fields are round-tripped -regardless, so a document that carries them keeps them. - -The payload's 32-byte values are **unpadded base64url at every depth**, which is -section-scoped to §8: CORD-05 invites stay hex. The writer re-encodes them on -every serialization, so its output is always the canonical 43-character spelling; -the reader also accepts non-zero trailing bits, because the spec's own worked -example contains them and no reader can tell a mis-encoded named field from a -correct one. The codec is `utils::base64url` and the wire structs behind the -List's `Serialize`/`Deserialize` are the only callers, so no other encoding path -is touched. - -Three write-time rules are folded into serialization, so an in-memory document -and its wire form differ: - -- an embedded snapshot omits `community_id` and inherits the entry's; -- `seed` is omitted when it equals `current`, and its cosmetic fields (`name`, - `relays`, each channel's `name`) are overwritten from `current` first, so a - rename collapses the snapshots instead of forking them; -- an entry whose `added_at` does not outrun its tombstone is omitted — the - tombstone alone carries the state. - -`is_live(&id)` answers joined-versus-left: a tombstone is terminal until a -strictly newer join outruns it. `fits()` is the write gate: 50 memberships and -the NIP-44 plaintext cap. The 50 is a stopgap inherited from the retired -single-event design — §8 has **no membership limit**, its only bound is the -65,536-byte encoded event, and the real fix is to start a new fragment on write. -Until that lands, an append onto a List that already spans more than one fragment -is refused rather than performed against a partial read, because placing a new -membership needs a repack. The community stays local (`load` keeps a membership -the List never mentions) and the write is deferred with a warning. - -Discovery is a **subscription, not a fetch**: subscribe with -`Filter::new().kind(Kind::Custom(KIND_COMMUNITY_LIST)).author(my_pk)` and read the -fragments back out of `client.database()`. The client persists a relay's event -before it notifies, so a subscription plus a database read loses nothing and -needs no explicit save. Parse each event with `parse_list_event`, keep the newest -per `fragment_index`, and `merge` them — reading an incomplete List is safe, since -a missing fragment is only news not yet heard. - -## GPUI integration - -`crates/concord` stays GPUI-free; the registry and sync engine live in -`crates/community`. That layer adds a registry global and one -entity per community, and moves every decrypt, verification, fold and I/O off -the foreground thread. - -### Entities - -Same shape as `ChatRegistry`: - -```rust -pub fn init(cx: &mut App) { - CommunityRegistry::set_global(cx.new(CommunityRegistry::new), cx); -} - -impl CommunityRegistry { - pub fn global(cx: &App) -> Entity { - cx.global::().0.clone() - } -} -``` - -Call it after `cord03::init` in `desktop/src/main.rs` and `web/src/lib.rs`, and -subscribe to `NostrRegistry` for `SignerChanged` so the communities reset with -the account. - -- `CommunityRegistry` holds `communities: Vec>`, an index by - `CommunityId`, and `tasks: SmallVec<[Task>; 2]>`. -- `Community` owns one `CommunityState`, the last `ControlFold`, the member list - and the channel list. Views render `Entity`; no protocol state - lives in a view. -- `CommunityState::apply_fold` is one assignment: run it in the task that - produced the fold and send only the result to the foreground. -- Emit an event on every fold so dependents re-read. - -### Foreground and background - -A background task never touches an entity. It sends results through a bounded -`flume` channel that a foreground `cx.spawn` drains with `this.update(...)`. - -```rust -let (signal_tx, signal_rx) = flume::bounded::(256); -let client = client.clone(); - -// Background: open, verify, fold — no entities. -self.ingress = Some(cx.background_spawn(async move { - for wrap in &wraps { - let Some(plane) = planes.iter().find(|plane| plane.group.pk() == wrap.pubkey) else { - continue; - }; - let (opened, rumor) = cord03::open(wrap, &plane.group, &plane.channel, plane.epoch)?; - cache::cache_rumor(&client, &plane.channel, &opened).await?; - signal_tx.send_async(Signal::Chat { channel: plane.channel, rumor }).await?; - } - Ok(()) -})); - -// Foreground: the only place entities change. -self.consumer = Some(cx.spawn(async move |this, cx| { - while let Ok(signal) = signal_rx.recv_async().await { - this.update(cx, |this, cx| this.apply(signal, cx))?; - } - Ok(()) -})); -``` - -- Every cache function takes the `&Client` and reaches the database through - `client.database()`, so clone the `Client` into the background task. -- Keep long-lived tasks in fields — dropping a `Task` cancels it. Assign `None` - to an `Option>` before respawning it; a signer change replaces both - the listener and the consumer. -- `cx.spawn` when the work updates an entity after awaiting, and - `cx.background_spawn` when it only produces a value. A query the foreground awaits can be returned straight out: `fn messages(&self, cx: &App) -> Task, Error>>`. -- Do the first load in `cx.defer_in(window, ...)` so `init` returns before the - first relay request. -- NIP-46 signing is async: call `signer.get_public_key_async()` / - `sign_event_async` inside the background task. Every account-key writer takes - any signer (`Keys` or the app's `UniversalSigner`) and is `async`, so `await` - it there rather than requiring device keys. - -### Subscriptions - -A wrap is addressed to a plane, so the plane's public key is the routing key and -one `Filter` per held plane is enough: - -```rust -let filter = Filter::new() - .kinds([Kind::from(KIND_WRAP), Kind::from(KIND_WRAP_EPHEMERAL)]) - .pubkey(plane.group.pk()) - .since(joined_at); -client.subscribe(filter).with_id(sub_id).await?; -``` - -- `pubkeys([...])` carries every plane of a community on one subscription. Call - `subscribe` again with the new address whenever a join, a channel add or a - rekey fold changes it. -- Route inbound events by `subscription_id` from `RelayMessage::Event`, never by - kind. -- Watch one epoch ahead for the base, and a **window** of channel epochs under - every held root for private channels: `base_rekey_group_key(&root_N, &id, - Epoch(N + 1))`, plus `channel_rekey_group_key(&root, &channel, Epoch(channel_epoch - + ahead))` for `ahead in 1..=8` and each root in `state.roots()`. A Refounding - seals its channel rekeys under the root current when it was minted, so a member - who adopted the base rotation first must still ask under the prior root; the - window is what lets a member who missed a rotation catch up, or learn they were - cut. This is `community::rekey::watches`, installed as a second standing REQ - whose id resolves back to its community through `rekey::WatchRegistry` (the id - cannot carry a 64-hex community id within the NIP-01 length cap). -- `community::rekey::adopt` then reads those wraps **from the database** and walks - each scope forward one epoch at a time: a complete, authorized rotation whose - `prevcommit` extends the key actually held hands over the next key (raced - rotations settle on the lowest); a gap is fetched, never waived; a fork is never - adopted. An addressed-but-unverifiable rotation is neither adopted nor read as a - removal. Only a complete rotation at/after the join, from a rotator who outranks - the member, with no blob for them, is a removal; one that predates the join is a - strand — a stale invite landed the member on a superseded epoch. - -### Tests - -`cx.background_executor().timer(..)` for delays, never `smol::Timer`, or -`run_until_parked()` finds nothing left to run. Push a wrap into the channel and -`run_until_parked()` to drive the foreground consumer. - -## Not wired up yet - -- **`crates/concord` stays protocol-only; the registry lives in - `crates/community`.** `concord` has no subscriptions, no `init`, and no - `Entity`; `community::CommunityRegistry` owns one `Entity` - per state document, subscribes when a community's plane set changes, and - re-folds on an inbound wrap. The sidebar subscribes to the registry, surfaces - `CommunityEvent::Error` as a window notification, and its "New community" row - in the Communities tab dispatches `Command::NewCommunity`, whose name prompt - calls `CommunityRegistry::create`. `create` persists the - genesis locally, publishes the wraps to the community's relays, and records the - membership in the account's Community List — all best-effort, so a relay that is - down warns without losing the community. Discovery - subscribes to the account's CORD-02 Community List (`33302`) under the - `concord/list` subscription id and reads the fragments back out of - `client.database()` — the SDK persists a relay's event before notifying, so the - read is always current. A `concord/list` notification re-runs `load`, which - materializes a community from each live List entry (`from_join_material`) and - keeps any state document the List does not mention, so a fresh install — or one - signing in as an account that joined elsewhere — finds its communities. See - `docs/concord-community-discovery-plan.md` (including its note on the retired - `13302` the current reference client still writes). -- **Account-key writers take any signer, not `&Keys`.** `genesis`, - `ControlWriter`, the guestbook and chat `seal_rumor`s, the `list` builders, and - the `cord05` invite writers (`build_direct_invite` / `unwrap_direct_invite`, - `build_invite_list` / `parse_invite_list`) and `cord06` blob writers - (`build_blob` / `open_blob`) are `async` and generic over the SDK's - `AsyncGetPublicKey` / `AsyncSignEvent` / `AsyncNip44` traits, so a `Keys` and an - app `UniversalSigner` both work. The NIP-59 paths (`build_direct_invite`, - `unwrap_direct_invite`) stay `Sized` because the SDK's gift-wrap helpers are. - Group-key and locally-held-secret writers (`cord01` wrap functions, - `cord05::build_bundle_event`, `community::cache`) still take the raw key material they - genuinely need. -- **`crates/chat/src/lib.rs::handle_notifications` treats every kind 1059 event as - a NIP-59 gift wrap for the current user.** Concord wraps are kind 1059 too, so - that handler must route by subscription id before any concord subscription goes - live, or every stream wrap lands in the DM trash and raises a toast. -- **Rotation-delivered plane keys are persisted, and history spans them.** - `CommunityState.held_roots` keeps every root a rotation stepped off (with the - retired root's Control signer and the publish time that retires it), and each - `ChannelKeyRef.priors` keeps every channel key a rotation stepped off. The read - side derives planes from all of them, and a retired key's `retired_at` is a hard - read cutoff at both `history::page` and `sync::fold`. The state - `Community::removed_at`/`Community::stranded`/`channel_removed_at` carries is - rendered as a notice **and** enforced at write time: `Community::send` refuses - when `channel_secret` is `None` (a removal, a strand, a channel cut, or a key we - never held), because a wrap sealed under a superseded root would reach nobody who - rotated. A rotation this client publishes itself (`Community::rotate`) is adopted - through the same `rekey::adopt` an arriving one goes through, so the held state - and the wire cannot diverge. diff --git a/docs/nip29-backend-plan.md b/docs/nip29-backend-plan.md deleted file mode 100644 index 77107bcc..00000000 --- a/docs/nip29-backend-plan.md +++ /dev/null @@ -1,650 +0,0 @@ -# NIP-29 groups backend - -Coop has two relay-backed backends today: `crates/chat` (NIP-17 gift-wrap DMs, -rendered by `crates/chat_ui`) and `crates/community` (Concord). This document is a -plan, not a description of shipped code; nothing below is wired up yet. It covers -the backend only — reading, writing, joining and leaving relay-based groups as -defined by [NIP-29](https://github.com/nostr-protocol/nips/blob/master/29.md) — -and ends with the hooks a UI will use later. - -## The spec in backend terms - -A group is a `(relay, id)` pair. The relay enforces membership and signs the -group's state; there is no cross-relay identity, no encryption, and no portable -"create". The same `id` on two relays is two groups (a fork), which the plan -models as two entities. - -| Concern | Wire | Backend rule | -| --- | --- | --- | -| Group id | random string, relay-chosen | `h` tag on everything a user sends; `d` tag on relay state events | -| Timeline | any event kind + `h` | v1 renders kinds `1`, `9`, `11`, `1111` and `7`; every other kind is cached for the `previous` window but not rendered, except the membership kinds below | -| Timeline refs | `previous` tag, first 8 hex chars of event ids | built from the last 50 events seen on that relay (any kind — NIP-29 timeline references), excluding our own; send up to 3 | -| Relay state | `39000` metadata, `39001` admins, `39002` members, `39003` roles, `39004` LiveKit participants | relay-signed, addressable by `d`; read, never written by the client | -| Membership | `9021` join, `9022` leave (user-signed); relay answers with `9000` put-user / `9001` remove-user | membership = latest `9000`/`9001` naming our pubkey; no event means not a member, and `Pending`/`Refused` are tracked locally for private groups | -| Own group list | `10009` (NIP-51 simple groups) | `["group", id, relay, name?]` plus `["r", relay]`; replaceable, published to NIP-65 relays | -| Reference | `naddr1…` of kind `39000` with the relay self pubkey and a relay hint, optionally `?invite=` | the self pubkey is the author of the fetched `39000`; no `39000` means no canonical naddr | - -Two behaviours that shape the code: - -- **Late publication is rejected.** Relays drop messages with old timestamps, so - there is no offline send queue in v1: a send either goes now or fails visibly. -- **`previous` must reference events the relay already has.** Relays may enforce - at least 3 references; a group we have little history for may have its sends - rejected until the backlog loads. The relay's `OK false` message is surfaced - as-is. - -### Not in v1 - -Moderation writes (`9000`–`9010`), group creation (`9007`, relay-dependent), -pinned events (`39005`/`9010`), subgroups (`parent`/`child` — parsed but inert), -LiveKit audio/video (`livekit` tag, `39004`, the token endpoint), fork/migration -detection through admins' `10009`, and relay browsing (`{"kinds":[39000]}` to -list a relay's public groups). All are additive; none change the model above. - -## What the local rust-nostr checkout gives us - -Cross-checked against the pinned checkout -(`~/.cargo/git/checkouts/nostr-619b808bb247a9ed/01f1950`): - -- **Kinds exist** for every NIP-29 event in v1: `Kind::GroupPutUser` (9000), - `GroupRemoveUser` (9001), `GroupEditMetadata` (9002), `GroupDeleteEvent` - (9005), `GroupCreateGroup` (9007), `GroupDeleteGroup` (9008), - `GroupCreateInvite` (9009), `GroupJoinRequest` (9021), `GroupLeaveRequest` - (9022), `GroupMetadata` (39000), `GroupAdmins` (39001), `GroupMembers` - (39002), `GroupRoles` (39003), `GroupLivekitParticipants` (39004), and - `Kind::SimpleGroups` (10009). `Kind::ChatMessage` is kind 9. -- **`9010` and `39005` are missing** from this revision (`Kind::Custom(9010)`, - `Kind::Custom(39005)` if they are ever needed — they are not, in v1). -- **There is no NIP-29 module.** `nostr/src/nips/` has no `nip29.rs`, and no - `GroupId`, metadata type, parser or builder exists. Everything in `protocol.rs` - below is ours; it is thin — tags and filters, no crypto. -- **The APIs to lean on** are all present and already used in the app: - - `Tag::custom`, `Tag::event`, `Tag::identifier`, `Tag::public_key`, - `Tag::content`, `Tag::as_slice`. - - `Filter::custom_tag(SingleLetterTag::LOWERCASE_H, id)`, - `.identifier(id)`, `.author`, `.kinds`, `.limit`, `.since`. - - `ReqTarget::single(url, filters)` for a group-scoped REQ, - `client.subscribe(..).with_id(..)`, `client.fetch_events(..)`, - `client.send_event(&event).to([url])`. - - `EventBuilder::new(kind, content).tags(tags).finalize_async(&signer)` — the - app's `UniversalSigner` works here (NIP-46 included), as it does in - `crates/chat/src/room.rs`. - - `client.database().query/save_event/delete`. - - `Nip19Coordinate { coordinate, relays }` with `ToBech32`/`FromBech32` for - `naddr`. -- **NIP-11 is not exposed.** `Relay` in this SDK has no document accessor, so the - relay self pubkey is taken from the author of the group's `39000` event; a - group whose relay publishes no `39000` therefore has no canonical naddr and - cannot be shared or joined in v1. - -## What the app already does (and we copy) - -The NIP-17 backend is the template. `crates/chat` has a global `ChatRegistry` -entity, one `Room` entity per conversation, a background task over -`client.notifications()` that sends signals through a `flume` channel, and a -foreground `cx.spawn` consumer that is the only place entities change. Local -messages are cached as `ApplicationSpecificData` (30078) events with a `d` tag -for the target id and an `r` tag for the room key (`set_rumor`/`get_rumor`, -`crates/chat/src/lib.rs`). `crates/chat_ui` renders `chat::Message` into a -`ListState`, resolving names through `PersonRegistry`. - -`crates/community` shows the same pattern for relay-scoped state: routing -notifications by subscription id (`route_of`), bringing relays into the pool -before use (`connect_relays`), and treating a `CLOSED` message as a refusal. -Replacing a list is done in `crates/workspace/src/panels/contact_list.rs`: -sign the builder, `client.send_event(&event).to_nip65().await` — the client saves -the event locally even when the publish fails. - -NIP-29 reuses all of this and drops what only NIP-17 needs: no encryption, no -per-member fan-out, no gift wraps, no `SendReport` per receiver — one relay, one -event, one `OK`. - -## Where this lands - -A new crate `crates/nip29`, because NIP-29 is a different transport (plain relay -events), a different state authority (relay-signed `39000`s) and a different UI -later. Making `chat::Room` polymorphic over gift-wrapped DMs and relay groups -would put two protocols in one entity. The alternative — modules inside -`crates/chat` — saves a `Cargo.toml` but was set aside (see Decisions). - -| File | Owns | -| --- | --- | -| `crates/nip29/src/protocol.rs` | Pure NIP-29 types and their `impl` blocks: `GroupId`, `GroupKey`, `GroupReference`, `GroupMetadata`, admins/members/roles, `GroupList`, `Membership`, `TimelineWindow`. No I/O. | -| `crates/nip29/src/group.rs` | `Group` entity: state, timeline, membership, `connect`, `ingest`, `send_message`, `send_reaction`, `join`, `leave`, `GroupEvent`. | -| `crates/nip29/src/lib.rs` | `GroupsRegistry` global, `init`, standing subscriptions and the signal pump, the envelope cache, `load`/`publish` of the account's `10009`. | -| `desktop/src/main.rs`, `web/src/lib.rs` | `nip29::init(cx)` after `chat::init(cx)`. | -| UI later | `crates/chat_ui` (reuse `chat::Message` rendering) or a new `crates/nip29_ui`; sidebar section; a join dialog. | - -`protocol` is re-exported at the crate root (`pub use protocol::*`) so callers -write `nip29::GroupId`, `nip29::GroupMetadata`. The crate's dependencies are the -ones `chat` already carries: `state` (client and signer), `chat` (`Message`), -`gpui`, `nostr-sdk`, `anyhow`, `flume`, `futures`, `smallvec`, `log`; -dev-dependencies `nostr-memory` and `nostr-sdk` with the `local-relay` feature -for the tests below. - -## Data flow - -``` -own 10009 (NIP-65 relays) group relay (one per group) - │ load / subscribe │ REQ #h= (timeline + 9000/9001) - ▼ │ REQ #d= (39000..39003) -GroupsRegistry ──ensure relay──► Group entity ◄──ingest── background listener - │ index: GroupKey │ (saves envelope first) - │ │ - └── flume Signal ◄──────────────────────┘ - ▲ - UI later: GroupPanel (Vec), JoinDialog (naddr) -``` - -A group's subscriptions are the standing `grp-` REQ (timeline + state + -membership, the timeline replayed up to 100 events) and, on demand, a -`pg-` REQ that re-asks the same timeline filter with `until` set to the -oldest event hydrated so far. Both are recognised by the same pump and land in -the same envelope cache, so paging needs no separate read path. - -## Phase 0 — `crates/nip29/src/protocol.rs` (pure, no I/O) - -The protocol is expressed as types with `impl` blocks: a parsed thing is a struct -and every operation on it is a method, so call sites read `id.message(...)`, -`metadata.naddr(...)`, `admins.contains(&pk)` instead of handing the pieces to -free builder functions. - -```rust -/// The relay-chosen group id. -pub struct GroupId(String); - -impl GroupId { - pub fn new(id: impl Into) -> Result; // trimmed, non-empty - pub fn as_str(&self) -> &str; - pub fn h_tag(&self) -> Tag; // ["h", id] - - pub fn timeline_filter(&self, limit: usize) -> Filter; // #h = id - pub fn state_filter(&self) -> Filter; // #d = id, kinds 39000..39003 - pub fn membership_filter(&self, me: PublicKey) -> Filter; // #h = id, #p = me, kinds 9000/9001 - - pub fn message(&self, content: String, replies: &[EventId], previous: Option) -> EventBuilder; // kind 9 - pub fn reaction(&self, target: EventId, emoji: String, previous: Option) -> EventBuilder; // kind 7 + e - pub fn join_request(&self, code: Option<&str>, reason: &str) -> EventBuilder; // 9021 - pub fn leave_request(&self, reason: &str) -> EventBuilder; // 9022 -} - -/// Entity identity: the same id on another relay is another group (a fork). -pub struct GroupKey { relay: RelayUrl, id: GroupId } - -impl GroupKey { - pub fn new(relay: RelayUrl, id: GroupId) -> Self; - pub fn relay(&self) -> &RelayUrl; - pub fn id(&self) -> &GroupId; - pub fn cache_tag(&self) -> String; // "|" - pub fn uniq_id(&self) -> u64; // hashed, for panel and element ids -} - -/// What a user pastes to join: naddr1…(+"?invite="). -pub struct GroupReference { id: GroupId, relay: RelayUrl, invite: Option } - -impl GroupReference { - pub fn parse(input: &str) -> Result; - pub fn id(&self) -> &GroupId; - pub fn relay(&self) -> &RelayUrl; - pub fn invite(&self) -> Option<&str>; - pub fn key(&self) -> GroupKey; -} - -impl FromStr for GroupReference { .. } // so a pasted naddr can use `.parse()` too - -/// kind:39000, relay-signed. -pub struct GroupMetadata { - pub name: Option, pub picture: Option, pub banner: Option, - pub about: Option, pub private: bool, pub restricted: bool, pub hidden: bool, - pub closed: bool, pub livekit: bool, pub parent: Option, - pub supported_kinds: Option>, // None = unspecified (all); Some(vec![]) = none -} - -impl GroupMetadata { - pub fn parse(event: &Event) -> Result; - pub fn name(&self) -> Option<&str>; - pub fn supports(&self, kind: Kind) -> bool; - pub fn naddr(&self, relay: &RelayUrl, invite: Option<&str>) -> Result; - // Coordinate { kind: 39000, public_key: , identifier: } -} - -/// kind:39001 — each pubkey with its role labels. -pub struct GroupAdmins(Vec<(PublicKey, Vec)>); - -impl GroupAdmins { - pub fn parse(event: &Event) -> Result; - pub fn entries(&self) -> &[(PublicKey, Vec)]; - pub fn contains(&self, public_key: &PublicKey) -> bool; -} - -/// kind:39002 — may be absent or a subset; absence means "not published". -pub struct GroupMembers(Vec); - -impl GroupMembers { - pub fn parse(event: &Event) -> Result; - pub fn contains(&self, public_key: &PublicKey) -> bool; - pub fn iter(&self) -> impl Iterator; -} - -/// kind:39003 — role names are relay-defined. -pub struct GroupRoles(Vec); -pub struct Role { pub name: String, pub description: Option } - -impl GroupRoles { - pub fn parse(event: &Event) -> Result; - pub fn roles(&self) -> &[Role]; -} - -/// kind:39004 — who is live in the group's AV room; `participant` tags, not `p`. -/// Absence means nothing is known, which is not the same as an empty room. -pub struct GroupParticipants(Vec); - -impl GroupParticipants { - pub fn parse(event: &Event) -> Result; - pub fn contains(&self, public_key: &PublicKey) -> bool; - pub fn iter(&self) -> impl Iterator; -} - -/// kind:10009 — the account's own list. -pub struct GroupList(Vec); -pub struct GroupListEntry { pub id: GroupId, pub relay: RelayUrl, pub name: Option } - -impl GroupList { - pub fn parse(event: &Event) -> Result; - pub fn entries(&self) -> &[GroupListEntry]; - pub fn upsert(&mut self, entry: GroupListEntry); - pub fn remove(&mut self, key: &GroupKey); - pub fn to_builder(&self) -> EventBuilder; // kind 10009, group + r tags -} - -pub enum Membership { - Unknown, - Pending { since: Timestamp }, // our 9021 went out; no answer yet - Member, - Refused { reason: Option }, // the relay's OK false / CLOSED message - Removed, -} - -impl Membership { - /// Folds 9000/9001 only; `Pending` and `Refused` are set by the join path. - pub fn from_events(events: &[Event], me: PublicKey) -> Self; -} - -/// The newest-first window (≤50) that `previous` references are drawn from. -pub struct TimelineWindow(Vec); - -impl TimelineWindow { - pub fn new() -> Self; - pub fn push(&mut self, event: Event); // dedups, caps at 50 - pub fn previous_tag(&self, me: PublicKey) -> Option; // excludes `me`, ≤3 distinct 8-hex -} -``` - -Details that must be right: - -- Every user event carries `["h", id]` (from `GroupId::h_tag`); `previous` is - added only when we have at least one eligible event, and its values are the - first 8 characters of the hex ids. -- `GroupReference::parse` splits `?invite=` before bech32 decoding; a missing - relay hint is an error (there is nowhere to send the join request). -- `GroupList::parse` accepts `group` tags with 1–3 values (id, relay, name). -- `Membership::from_events` takes the newest `9000`/`9001` whose `p` tags name - `me`; on an exact tie it prefers `9001` (removal). `Pending` and `Refused` are - not things the relay folds into an event: the join path sets them from the - `9021` send and from the relay's refusal message. -- `GroupMetadata::supports` distinguishes "tag absent" from "tag present with no - values": the first means all kinds, the second means no text messages. -- The builders return `EventBuilder`s so the caller decides when to sign; nothing - in this file does I/O. - -## Phase 1 — read path (`group.rs` + `lib.rs`) - -`GroupsRegistry` (global entity, mirroring `CommunityRegistry`): - -```rust -pub struct GroupsRegistry { - groups: Vec>, - index: HashMap>, - /// The account's 10009, loaded at start and mutated by join/leave - list: GroupList, - /// subscription id → group, rebuilt whenever a group resubscribes - routes: Arc>>, - signal_tx: flume::Sender, signal_rx: flume::Receiver, - notification_listener: Option>>, // cancelled on signer change - signal_consumer: Option>>, - tasks: SmallVec<[Task>; 2]>, - _subscriptions: SmallVec<[Subscription; 2]>, -} - -impl GroupsRegistry { - pub fn global(cx: &App) -> Entity; - pub fn groups(&self) -> &[Entity]; - pub fn group(&self, key: &GroupKey) -> Option>; - pub fn join(&mut self, reference: GroupReference, cx: &mut Context); - pub fn leave(&mut self, key: &GroupKey, cx: &mut Context); - fn load(&mut self, cx); // own 10009 from the database into `self.list` - fn publish_list(&mut self, cx); // sign `self.list.to_builder()`, to_nip65, best-effort -} -``` - -- **Subscriptions.** One standing REQ per group, two filters in one id: - `key.id().timeline_filter(100)` and `key.id().state_filter()`. The id is - `grp-<8 hex of the group key hash>` (NIP-01 caps ids at 64 chars; the real key - lives in `routes`). The relay is added and connected first - (`add_relay(url).and_connect()`), like `community::sync::connect_relays`. On - open, also issue a one-shot `key.id().membership_filter(me)` fetch so a restart - learns `9000`/`9001` immediately; the state filter is fetched once (limit 1 per - kind) so the sidebar has names before the live REQ settles. A dead relay costs - only its own group. -- **Pump.** The listener matches `ClientNotification::Message` and routes - `RelayMessage::Event`/`EndOfStoredEvents`/`Closed` by subscription id through - `routes`. For `h`-tagged events it saves the envelope first, then sends - `Signal::Event(key, event)`; state events send `Signal::State(key, event)`; - `Closed` sends `Signal::Refused(key, message)`. The consumer calls - `Group::ingest`, which is the only place entity state changes. -- **Envelope cache.** Same shape as `set_rumor`/`get_rumor` in `crates/chat`: - `ApplicationSpecificData` (30078, addressable) with `d` = event id (dedup on - replay), `r` = `"|"` (per-fork history), `k` = kind, content = the - raw event JSON, signed by a crate-local `LOCAL_KEYS`. Every `h`-tagged event - the standing subscription delivers is enveloped — any kind, because the - `previous` window is literally "the last 50 events seen" (NIP-29 timeline - references) — while the `d`-tagged state events (`39000`–`39003`) are not. - Reading is `Filter::new().custom_tag(LOWERCASE_R, key)`, parsed and sorted by - `created_at`, then partitioned: the render set (`1`, `7`, `9`, `11`, `1111`) - becomes messages, `9000`/`9001` fold membership, the rest only feed the - window. This is what makes the timeline fork-safe and restart-safe without - teaching the database about relays. -- **`Group` state.** `GroupKey`, `metadata: Option`, - `metadata_event: Option` (what `GroupMetadata::naddr` is built from), - `members`, `admins`, `roles`, `membership: Membership` (including the local - `Pending`/`Refused` access states), `messages: Vec` for the render set - (reusing `chat::Message`, which already parses any event into display fields), - an id index, a `TimelineWindow` for `previous`, `loading`, `refused`. It emits - `GroupEvent::{Incoming(Event), Updated, Refused(String), Eose}` — the panel - subscribes to these the way `ChatPanel` subscribes to `RoomEvent`. -- **`load()`.** Subscribe to our own `10009` (`ReqTarget::auto`, limit 1), then - read the newest one from the database into `self.list`, and add one group per - entry. A newer `10009` arriving from another device is merged - (`GroupList::upsert`) and the new groups are subscribed; removals by another - client cannot be detected in a replaceable list and are not guessed at. - -## Phase 2 — write path - -All writes sign with the app signer (`finalize_async(&signer)`), send to the -group's relay only, and treat the relay's `OK` as the truth. - -- **Send a message.** Build with - `key.id().message(content, &replies, window.previous_tag(me))`, sign, then - `client.send_event(&event).to([relay])`. Kind 9 is the default message kind; - when `metadata.supports(Kind::ChatMessage)` is false the composer is disabled - rather than sending a kind the relay does not take. On acceptance: save the - envelope, insert into `messages`, emit `GroupEvent::Incoming`. On `OK false`: - return the relay's message (it already carries `invalid:` / `restricted:` / - `duplicate:` prefixes) and let the panel show it. Reactions go through the same - path with `key.id().reaction(..)`. -- **Join.** `GroupReference::parse` → error if no relay hint → add/connect the - relay → fetch `39000` (`limit 1`) to confirm the group and learn the self - pubkey; `GroupMetadata::parse` it; a hidden group may refuse this — proceed - without metadata rather than failing → create the entity, fetch state + - membership, subscribe → send `key.id().join_request(code, ..)` — the `h` tag - plus the `code` tag when an invite code was parsed → mark - `Membership::Pending { since }` and `self.list.upsert(..)`, so the group stays - listed while it waits. The relay either emits `9000` (member), refuses the - `9021` with `OK false` (set `Refused { reason }` from its message — the NIP - says the message explains whether the rejection is final, pending review or - payment-required), or refuses the timeline REQ with `restricted:` / - `auth-required:` in a `CLOSED` (keep the state, do not raise an error toast). - All three are observed on the standing subscription or the send result. -- **Leave.** Send `key.id().leave_request(..)`; on `OK`, forget the group's - envelopes (`client.database().delete(filter)` by `r` tag), drop the entity and - `self.list.remove(key)`, then publish. The relay also emits `9001`, which the - (now gone) subscription would have seen. -- **Publish `10009`.** Sign `self.list.to_builder()` and publish with - `to_nip65()`. The registry's `GroupList` is the one loaded at start, join - `upsert`s an entry and leave `remove`s one, so a rewrite can never drop a group - we loaded. The client saves the event locally even when the publish fails, so - the list survives offline. - -## Phase 3 — later, in rough order - -1. Moderation writes: `9000` put-user (roles), `9001` remove-user, `9002` - edit-metadata, `9005` delete-event, `9009` create-invite; gate the UI on the - caller appearing in `39001`, let the relay be the enforcer. **Done** as pure - writes: builders on `GroupId`, senders on `Group` sharing `dispatch`. - `edit_metadata` takes a `GroupMetadata`, which is now `Default`-able (its - private `id`/`author` origin are `Option`) so the UI can start an edit from - scratch or from `Group::metadata()`. **`9005` is write-only for now**: the - relay stops serving the event but nothing is removed locally, so the deleted - message lingers in the cache and timeline. The UI phase must drop the target - envelope (filter on its `d` = target id and the group's `r` cache tag) and - evict it from `messages`/`index`, otherwise only a reload clears it. -2. Group creation (`9007`) where the relay supports it, and relay browsing. - **Done.** `GroupId::create_group` reuses the `9002` metadata tags, so the - proposed id (`h`) and the initial name/flags go in one event; the relay - republishes `39000`, which the standing subscription picks up. - `GroupsRegistry::create` mirrors `join`: `ensure` → optimistic `10009` entry - + `publish_list` → publish, and on `OK false` it `forget`s the group and emits - `GroupsEvent::Error(reason)` (that variant was previously unused). There is - no NIP-11 support check available, so "where the relay supports it" is - enforced by the relay refusing the `9007`. - `GroupsRegistry::browse(relay, cx)` follows the same pipeline as every - other read: it registers a `brw-` subscription (an opaque id, like - `grp-`; the relay is keyed in state, not encoded in the id, because the - pump needs a prefix to recognise its own ids and a bare - `SubscriptionId::generate()` would be indistinguishable from the other - crates' ids on the shared client). `add_relay` + - `subscribe` (auto-close on EOSE, 10s cap), and returns immediately. The - **global pump** observes the events, and `handle_event`/`handle_eose`/ - `handle_closed` fold each relay-signed `39000` into `self.browse`, keyed by - the subscription id, so no DB scoping is needed. `GroupsRegistry::browsed(relay)` - returns `Browse { relay, loading, groups: Vec }` for the UI, and - `GroupsEvent::Updated` says when to re-read it. No `fetch_events` anywhere: - the DB cannot be scoped to a relay (a stored event carries no origin), so the - subscription's own events are the source. `join` now delegates to a new - `join_key(key, invite)`, which is what a browse result needs (a candidate has - a key, not a `GroupReference`). -3. Pins and subgroups. **Pins done:** `Pin::{Event, Address}` and - `GroupPins` in `protocol.rs`, with `PIN_LIST` (39005) and `UPDATE_PIN_LIST` - (9010) as `u16` consts used through `Kind::Custom`. `39005` joins - `STATE_KINDS`, so the standing subscription fetches it, `absorb_state` folds - it into `Group::pins()`, and `Group::update_pins(pins)` writes `9010` through - the shared `dispatch`. **Subgroups parsed:** `GroupMetadata` gained - `pub children: Vec` (order preserved, parsed from `child` tags) and - `metadata_tags` re-emits them — `9002` requires it, an edit that drops a - child is rejected. `parent`/`children` are per-relay, so assembling the tree - from `Group::metadata()` is UI work; there is no backend tree helper. -4. Range backfill: page the timeline with `until` for "load more". **Done.** - `GroupId::timeline_page(limit, until)` is `timeline_filter` plus `.until(..)`, - and `Group::load_page` sends it as a `pg-` subscription (auto-close on - EOSE, 10s cap) alongside the standing `grp-` one, which keeps running so - paging cannot replace the live tail. The read path is the usual one: `pg-` - joins `is_own_subscription`, the id is registered in `routes` against the same - key, and `handle_event` therefore caches each page event and `reload`s the - group, so it arrives through the envelope cache and the timeline like any - other event. `GroupsRegistry::load_more(key, cx)` is the entry point: it does - nothing while a page is in flight (`Group::loading_more`) or before anything is - cached, and registers the route before `Group::load_page` sets the flag. - `handle_eose` and a page `CLOSED` remove the route and call - `Group::page_finished`, which clears the flag and emits `GroupEvent::Updated` - so a panel's spinner can settle. The cursor is `Group::oldest()`, the oldest - `h`-tagged event ever hydrated, updated by `track` next to `window.push` — - *not* read from `TimelineWindow`, which keeps only the newest 50 events, so a - cursor taken from it would never move and would page the same window forever. - `until` is inclusive, so a page repeats the event it starts from; `hydrate` - dedups by id, so that costs one redundant fetch, never a gap. `forget` also - drops and unsubscribes the `pg-` route (`reset` already sweeps `routes`). Two - gaps left for the UI phase: a relay that accepts the REQ but never EOSEs leaves - `loading_more` true (the auto-close timeout closes the subscription locally and - emits no notification — the same stickiness as `Browse::loading`), and a group - whose next page shares the `until` second cannot advance the cursor, so it can - re-fetch the same page. Neither loops on its own; both need a "load older - messages" button bound to `Group::loading_more()` to be observable at all. -5. LiveKit AV and fork/migration detection, split in three: - 1. **`39004` participants. Done.** `GroupParticipants` joins `protocol.rs` - with the same shape as `GroupMembers` but reads `participant` tags (the NIP - does not use `p` here); one malformed pubkey fails the whole event, as in - `GroupAdmins`. The kind joins `STATE_KINDS`, so the standing subscription - already carries it, and `absorb_state` folds it into `Group::participants()`. - Nothing is written by us — the relay publishes it and updates it as people - come and go, so it rides the live tail like any other state event. - "Absent" and "empty" are deliberately the same value here: unlike - `supported_kinds`, the NIP gives the empty list no special meaning, so the - UI reads "no one is live" either way. - 2. The `/.well-known/nip29/livekit/` token endpoint and the `204` - probe at `/.well-known/nip29/livekit`. **Done.** The HTTP client is the one - GPUI already owns: `cx.http_client()` returning `Arc`, - already wired to `reqwest_client::ReqwestClient` on desktop and to `fetch` - on wasm — the same path `state::nip05` uses, which is why no HTTP - dependency lands in this crate. Its `get()` convenience cannot set headers, - so the request goes through `HttpClient::send` with a `Builder`, an - explicit `RedirectPolicy::NoFollow` (the `Authorization` header is signed - for exactly one URL, so a redirect could only ever fail verification) and a - 10s `timeout` — both are opt-in extensions, and a request without them has - none. The header itself is NIP-98 from the SDK: `nostr/nip98` is now - enabled on the workspace `nostr` entry and nip29 declares `nostr` itself, - so `HttpData::new(url, HttpMethod::GET).to_authorization(&signer)` builds - `Nostr ` — the `u` + `method` tags that the crate's own - `verify_auth_header` demands, though the NIP's prose mentions only `u`. - `GroupKey::livekit_token_url` / `livekit_probe_url` derive the endpoint - purely: the relay URL is rewritten to http(s) (keeping host and port), - reset to the origin, and the path built with `pop_if_empty().push(..)` so - each segment is escaped — a relay with its own path prefix loses it, since - the NIP puts the endpoint at the absolute path. `Group::livekit_session` - and `Group::livekit_supported` are the two calls, both `&App` + - `background_spawn` like `publish`. The session is returned, not stored: - tokens are per-join artifacts, not group state, so the caller owns it. - Only `livekit_supported`'s unauthenticated GET ignores status beyond - `204`; the token endpoint's response shape is **not specified by the NIP** - (see Decisions) and no LiveKit client library is involved — the backend - stops at the JWT and server URL. - 3. Fork/migration detection. **Done.** One standing `nip29-forks` - subscription over `{authors: , kinds: [10009]}` - on `ReqTarget::auto` — the admins' own group lists live on their outbox - relays, so this works with the group's relay down, and the admin pubkeys - come from `39001`, which is hydrated from the envelope cache, so it works - after a restart too. Each event is folded into `forks` (author → newest - `GroupList`, guarded on `created_at` because `10009` is replaceable), then - `evaluate_forks` recomputes every group. `Group::elsewhere()` is `Some` - when an admin of that group lists the same group id at a different relay; - the data cannot distinguish a move from a fork, hence the neutral name. - Evaluation is incremental, on each event, rather than batched on EOSE: - `ReqTarget::auto` fans out to several relays, so the first EOSE would - otherwise evaluate — and clear — against one relay's partial view. - Nothing is auto-joined, since the NIP asks the client to *offer*; the - offer is `GroupsRegistry::join_key(elsewhere, None, cx)`, which is exactly - "fetch from the new relay and update our own `10009`". Two triggers, - because the NIP's "periodically" and its "MUST when the relay is offline" - are different things. Reactively, the group observer calls - `sync_forks(false)` on `GroupEvent::Updated`, subscribing as soon as the - first `39001` lands and re-subscribing only when the admin set actually - changes — the guard matters, since `Updated` also fires for metadata, - pins, membership and a finished page. Periodically, a - `background_executor().timer` loop started in `load` pushes - `Signal::Forks` every 15 minutes and `sync_forks(true)` re-queries - regardless: that is the recovery path for a subscription that died or a - client that was offline, not the detection path. `reset` drops the fork - state and unsubscribes the REQ; a `CLOSED` on it is only logged, because - the tick retries and there is nothing to repair. - -## UI later — the hooks this backend leaves - -| UI action | Call | -| --- | --- | -| Paste an naddr and join (or retry a refused join) | `GroupReference::parse` → `GroupsRegistry::join` | -| Open a group | `GroupsRegistry::group(&key, cx)` → panel subscribes to `GroupEvent` | -| Timeline | `Group::messages()` (already `chat::Message`s) | -| Load older messages | `GroupsRegistry::load_more(&key, cx)`; spinner from `Group::loading_more()` | -| Send / react | `Group::send_message` / `send_reaction` | -| Members, admins, roles, flags | `Group::members/admins/roles/metadata` | -| Who is in the AV room | `Group::participants()` | -| Join AV (get a LiveKit session) | `Group::livekit_session(cx)` → `await` → `LivekitSession { token, server_url }` | -| Is AV available here | `Group::livekit_supported(cx)` after seeing `metadata().livekit` | -| The group may have moved or been forked | `Group::elsewhere()` → offer `GroupsRegistry::join_key(elsewhere, None, cx)` | -| Membership badge (member, pending, refused, removed) | `Group::membership()` | -| Copy an invite link | `GroupMetadata::naddr(relay, invite)` | -| Leave | `GroupsRegistry::leave` | -| Create a group | `GroupMetadata { ..Default::default() }` → `GroupsRegistry::create(key, metadata, cx)`; failure surfaces as `GroupsEvent::Error` | -| Browse a relay's groups | `GroupsRegistry::browse(relay, cx)` → re-read `GroupsRegistry::browsed(relay)` on `GroupsEvent::Updated`; join a result with `GroupsRegistry::join_key(candidate.key, None, cx)` | -| Moderate (put/remove user, edit metadata, delete event, invite) | `Group::put_user/remove_user/edit_metadata/delete_event/create_invite` | -| Pins | `Group::pins()` to read, `Group::update_pins(pins)` to replace the list | -| Subgroup tree | `Group::metadata()` → `parent`/`children`, assembled per relay | - -Where it lands: the sidebar gains a section reading `GroupsRegistry::groups()` -(the same shape as its Communities section) with pending and refused groups -listed alongside members; a new panel renders `Vec` with the existing -`ListState` + `ui::message::MessageRow` and `PersonRegistry` for authors. The -message *data* is already shared (`chat::Message`), so only `ChatPanel`'s -row-rendering methods need lifting into `ui` (or copying) rather than the type. A -join dialog accepts a pasted naddr (with or without `?invite=`). System -notifications can hang off `GroupEvent::Incoming` when the window is inactive. - -Nothing new is needed from GPUI to support this backend. The local checkouts -are `gpui-pre` 0.3.7 (aliased as `gpui`) and `gpui-base` 0.7.0, which `crates/ui` -styles; the GPUI APIs the backend uses are the ones `chat` and `community` -already use — `Entity`/`WeakEntity`, `Context::spawn`, `background_spawn`, -`EventEmitter`, `Subscription` — plus two that `state` already uses: -`App::http_client()` for the AV token fetch and -`App::background_executor().timer(..)` for the fork tick. The panel's components -(`ListState`, `MessageRow`, `Avatar`, `Button`, `Scrollbar`) already exist in the -app. - -## Tests - -- `protocol.rs` unit tests: `GroupReference::parse` (naddr, `?invite=`, garbage, - missing relay hint, an naddr whose kind is not `39000`); `GroupMetadata::parse` - flags and `supports` with `supported_kinds` absent vs empty; admins with roles; - members; roles; `GroupList` round trip (`parse` → `upsert` → `to_builder`); - `TimelineWindow::previous_tag` (excludes self, caps at 3, 8-char prefixes, - any-kind window, `None` when empty); `Membership::from_events` (`9000` vs - `9001`, newest wins, tie prefers removal; `Pending`/`Refused` are exercised - through the join path); `GroupId::message` carries `h` and the reply `e` tags; - `GroupMetadata::naddr` round trip with and without an invite code. -- Cache round trip with `nostr-memory` as a dev-dependency (the pattern - `crates/community` already uses): save two envelopes (a kind 9 and a kind - `9000`), read them back sorted, assert the partition — the kind 9 renders, the - `9000` only folds membership — re-save one and confirm no duplicate, `delete` - clears the group. -- Routing: subscription id ↔ `GroupKey` mapping survives a resubscribe. -- One integration test against the SDK's `LocalRelay` - (`nostr-sdk` dev-dependency with the `local-relay` feature): seed a `39000` - and a kind 9 with an `h` tag, run `Group::connect`, assert metadata and the - timeline land. Timeouts in GPUI-driven tests use - `cx.background_executor().timer(..)`, never `smol::Timer`, per `AGENTS.md`. - -## Decisions - -1. **Crate.** `crates/nip29` (the body above is written for it). -2. **Timeline kinds.** Render `1`, `9`, `11`, `1111` and `7`; every other kind - is still cached and feeds the `previous` window, per the NIP's timeline - references rule. -3. **Default message kind.** Kind 9; the composer is disabled when - `supported_kinds` is present and lacks it. -4. **Pending joins.** Pending (and refused) groups are listed in the sidebar, - with an access state on the row. -5. **`10009` merge.** Union-only; removals made by another client are not - detected and no diffing is done. -6. **Leave and the cache.** Leaving wipes the group's envelopes. -7. **Restricted/private groups.** A visible access state is required: - `Membership::{Unknown, Pending, Member, Refused, Removed}` with the relay's - refusal reason shown and a retry, while the client's NIP-42 authenticator - handles the reads. -8. **References.** `naddr` only (`?invite=` accepted); `id@relay` is not. A group - whose relay publishes no `39000` cannot be referenced or joined in v1. -9. **LiveKit response shape.** The NIP says the token endpoint returns "the - LiveKit JWT and the URL of the LiveKit server" but names no fields, so - `LivekitSession::parse` reads `token` (or `jwt`) and `url` (or `server_url`) - and fails loudly on anything else. This is a guess, not a spec, and it is the - one part of the AV path that a real relay can contradict; it is worth - re-checking against whichever relay is targeted before the UI lands. - Likewise `livekit_supported` treats "not `204`" as "no AV" rather than an - error, because it only feeds a button. -10. **`Monitor`.** Not adopted. It is only reachable through - `ClientBuilder::monitor(Monitor::new(n))`, so nothing can read per-relay - status until `crates/state` opts in, and its `StatusChanged` stream does not - distinguish a permanent outage from a reconnect — so it would need filtering - before it could mean anything. The only thing it buys here is running the - admins' `10009` check the instant a group's relay drops, rather than at the - next 15-minute tick. The NIP frames that as MUST, but the substance is - already covered: the admins come from the cached `39001`, so the check runs - with the relay gone, just later. If that latency ever matters, the hook is - `sync_forks(true, cx)` driven by a `StatusChanged` consumer — one place.