coop/docs/nip29-backend-plan.md

NIP-29 groups backend

Coop has two relay-backed backends today: crates/chat (NIP-17 gift-wrap DMs, rendered by crates/chat_ui) and crates/community (Concord). This document is a plan, not a description of shipped code; nothing below is wired up yet. It covers the backend only — reading, writing, joining and leaving relay-based groups as defined by NIP-29 — and ends with the hooks a UI will use later.

The spec in backend terms

A group is a (relay, id) pair. The relay enforces membership and signs the group's state; there is no cross-relay identity, no encryption, and no portable "create". The same id on two relays is two groups (a fork), which the plan models as two entities.

Concern	Wire	Backend rule
Group id	random string, relay-chosen	h tag on everything a user sends; d tag on relay state events
Timeline	any event kind + h	v1 renders kinds 1, 9, 11, 1111 and 7; every other kind is cached for the previous window but not rendered, except the membership kinds below
Timeline refs	previous tag, first 8 hex chars of event ids	built from the last 50 events seen on that relay (any kind — NIP-29 timeline references), excluding our own; send up to 3
Relay state	39000 metadata, 39001 admins, 39002 members, 39003 roles	relay-signed, addressable by d; read, never written by the client
Membership	9021 join, 9022 leave (user-signed); relay answers with 9000 put-user / 9001 remove-user	membership = latest 9000/9001 naming our pubkey; no event means not a member, and Pending/Refused are tracked locally for private groups
Own group list	10009 (NIP-51 simple groups)	["group", id, relay, name?] plus ["r", relay]; replaceable, published to NIP-65 relays
Reference	naddr1… of kind 39000 with the relay self pubkey and a relay hint, optionally ?invite=<code>	the self pubkey is the author of the fetched 39000; no 39000 means no canonical naddr

Two behaviours that shape the code:

• Late publication is rejected. Relays drop messages with old timestamps, so there is no offline send queue in v1: a send either goes now or fails visibly.
• previous must reference events the relay already has. Relays may enforce at least 3 references; a group we have little history for may have its sends rejected until the backlog loads. The relay's OK false message is surfaced as-is.

Not in v1

Moderation writes (9000–9010), group creation (9007, relay-dependent), pinned events (39005/9010), subgroups (parent/child — parsed but inert), LiveKit audio/video (livekit tag, 39004, the token endpoint), fork/migration detection through admins' 10009, and relay browsing ({"kinds":[39000]} to list a relay's public groups). All are additive; none change the model above.

What the local rust-nostr checkout gives us

Cross-checked against the pinned checkout (~/.cargo/git/checkouts/nostr-619b808bb247a9ed/01f1950):

• Kinds exist for every NIP-29 event in v1: Kind::GroupPutUser (9000), GroupRemoveUser (9001), GroupEditMetadata (9002), GroupDeleteEvent (9005), GroupCreateGroup (9007), GroupDeleteGroup (9008), GroupCreateInvite (9009), GroupJoinRequest (9021), GroupLeaveRequest (9022), GroupMetadata (39000), GroupAdmins (39001), GroupMembers (39002), GroupRoles (39003), GroupLivekitParticipants (39004), and Kind::SimpleGroups (10009). Kind::ChatMessage is kind 9.
• 9010 and 39005 are missing from this revision (Kind::Custom(9010), Kind::Custom(39005) if they are ever needed — they are not, in v1).
• There is no NIP-29 module. nostr/src/nips/ has no nip29.rs, and no GroupId, metadata type, parser or builder exists. Everything in protocol.rs below is ours; it is thin — tags and filters, no crypto.
• The APIs to lean on are all present and already used in the app:
  • Tag::custom, Tag::event, Tag::identifier, Tag::public_key, Tag::content, Tag::as_slice.
  • Filter::custom_tag(SingleLetterTag::LOWERCASE_H, id), .identifier(id), .author, .kinds, .limit, .since.
  • ReqTarget::single(url, filters) for a group-scoped REQ, client.subscribe(..).with_id(..), client.fetch_events(..), client.send_event(&event).to([url]).
  • EventBuilder::new(kind, content).tags(tags).finalize_async(&signer) — the app's UniversalSigner works here (NIP-46 included), as it does in crates/chat/src/room.rs.
  • client.database().query/save_event/delete.
  • Nip19Coordinate { coordinate, relays } with ToBech32/FromBech32 for naddr.
• NIP-11 is not exposed. Relay in this SDK has no document accessor, so the relay self pubkey is taken from the author of the group's 39000 event; a group whose relay publishes no 39000 therefore has no canonical naddr and cannot be shared or joined in v1.

What the app already does (and we copy)

The NIP-17 backend is the template. crates/chat has a global ChatRegistry entity, one Room entity per conversation, a background task over client.notifications() that sends signals through a flume channel, and a foreground cx.spawn consumer that is the only place entities change. Local messages are cached as ApplicationSpecificData (30078) events with a d tag for the target id and an r tag for the room key (set_rumor/get_rumor, crates/chat/src/lib.rs). crates/chat_ui renders chat::Message into a ListState, resolving names through PersonRegistry.

crates/community shows the same pattern for relay-scoped state: routing notifications by subscription id (route_of), bringing relays into the pool before use (connect_relays), and treating a CLOSED message as a refusal. Replacing a list is done in crates/workspace/src/panels/contact_list.rs: sign the builder, client.send_event(&event).to_nip65().await — the client saves the event locally even when the publish fails.

NIP-29 reuses all of this and drops what only NIP-17 needs: no encryption, no per-member fan-out, no gift wraps, no SendReport per receiver — one relay, one event, one OK.

Where this lands

A new crate crates/nip29, because NIP-29 is a different transport (plain relay events), a different state authority (relay-signed 39000s) and a different UI later. Making chat::Room polymorphic over gift-wrapped DMs and relay groups would put two protocols in one entity. The alternative — modules inside crates/chat — saves a Cargo.toml but was set aside (see Decisions).

File	Owns
crates/nip29/src/protocol.rs	Pure NIP-29 types and their impl blocks: GroupId, GroupKey, GroupReference, GroupMetadata, admins/members/roles, GroupList, Membership, TimelineWindow. No I/O.
crates/nip29/src/group.rs	Group entity: state, timeline, membership, connect, ingest, send_message, send_reaction, join, leave, GroupEvent.
crates/nip29/src/lib.rs	GroupsRegistry global, init, standing subscriptions and the signal pump, the envelope cache, load/publish of the account's 10009.
desktop/src/main.rs, web/src/lib.rs	nip29::init(cx) after chat::init(cx).
UI later	crates/chat_ui (reuse chat::Message rendering) or a new crates/nip29_ui; sidebar section; a join dialog.

protocol is re-exported at the crate root (pub use protocol::*) so callers write nip29::GroupId, nip29::GroupMetadata. The crate's dependencies are the ones chat already carries: state (client and signer), chat (Message), gpui, nostr-sdk, anyhow, flume, futures, smallvec, log; dev-dependencies nostr-memory and nostr-sdk with the local-relay feature for the tests below.

Data flow

own 10009 (NIP-65 relays)              group relay (one per group)
      │ load / subscribe                     │  REQ #h=<id>  (timeline + 9000/9001)
      ▼                                      │  REQ #d=<id>  (39000..39003)
GroupsRegistry ──ensure relay──► Group entity ◄──ingest── background listener
      │ index: GroupKey                       │              (saves envelope first)
      │                                       │
      └── flume Signal ◄──────────────────────┘
                 ▲
     UI later: GroupPanel (Vec<Message>), JoinDialog (naddr)

Phase 0 — crates/nip29/src/protocol.rs (pure, no I/O)

The protocol is expressed as types with impl blocks: a parsed thing is a struct and every operation on it is a method, so call sites read id.message(...), metadata.naddr(...), admins.contains(&pk) instead of handing the pieces to free builder functions.

/// The relay-chosen group id.
pub struct GroupId(String);

impl GroupId {
    pub fn new(id: impl Into<String>) -> Result<Self, Error>;   // trimmed, non-empty
    pub fn as_str(&self) -> &str;
    pub fn h_tag(&self) -> Tag;                                 // ["h", id]

    pub fn timeline_filter(&self, limit: usize) -> Filter;      // #h = id
    pub fn state_filter(&self) -> Filter;                       // #d = id, kinds 39000..39003
    pub fn membership_filter(&self, me: PublicKey) -> Filter;   // #h = id, #p = me, kinds 9000/9001

    pub fn message(&self, content: String, replies: &[EventId], previous: Option<Tag>) -> EventBuilder; // kind 9
    pub fn reaction(&self, target: EventId, emoji: String, previous: Option<Tag>) -> EventBuilder;      // kind 7 + e
    pub fn join_request(&self, code: Option<&str>, reason: &str) -> EventBuilder;                       // 9021
    pub fn leave_request(&self, reason: &str) -> EventBuilder;                                          // 9022
}

/// Entity identity: the same id on another relay is another group (a fork).
pub struct GroupKey { relay: RelayUrl, id: GroupId }

impl GroupKey {
    pub fn new(relay: RelayUrl, id: GroupId) -> Self;
    pub fn relay(&self) -> &RelayUrl;
    pub fn id(&self) -> &GroupId;
    pub fn cache_tag(&self) -> String;   // "<relay>|<id>"
    pub fn uniq_id(&self) -> u64;        // hashed, for panel and element ids
}

/// What a user pastes to join: naddr1…(+"?invite=<code>").
pub struct GroupReference { id: GroupId, relay: RelayUrl, invite: Option<String> }

impl GroupReference {
    pub fn parse(input: &str) -> Result<Self, Error>;
    pub fn id(&self) -> &GroupId;
    pub fn relay(&self) -> &RelayUrl;
    pub fn invite(&self) -> Option<&str>;
    pub fn key(&self) -> GroupKey;
}

impl FromStr for GroupReference { .. }   // so a pasted naddr can use `.parse()` too

/// kind:39000, relay-signed.
pub struct GroupMetadata {
    pub name: Option<String>, pub picture: Option<String>, pub banner: Option<String>,
    pub about: Option<String>, pub private: bool, pub restricted: bool, pub hidden: bool,
    pub closed: bool, pub livekit: bool, pub parent: Option<GroupId>,
    pub supported_kinds: Option<Vec<Kind>>,   // None = unspecified (all); Some(vec![]) = none
}

impl GroupMetadata {
    pub fn parse(event: &Event) -> Result<Self, Error>;
    pub fn name(&self) -> Option<&str>;
    pub fn supports(&self, kind: Kind) -> bool;
    pub fn naddr(&self, relay: &RelayUrl, invite: Option<&str>) -> Result<String, Error>;
    // Coordinate { kind: 39000, public_key: <the event author>, identifier: <the d tag> }
}

/// kind:39001 — each pubkey with its role labels.
pub struct GroupAdmins(Vec<(PublicKey, Vec<String>)>);

impl GroupAdmins {
    pub fn parse(event: &Event) -> Result<Self, Error>;
    pub fn entries(&self) -> &[(PublicKey, Vec<String>)];
    pub fn contains(&self, public_key: &PublicKey) -> bool;
}

/// kind:39002 — may be absent or a subset; absence means "not published".
pub struct GroupMembers(Vec<PublicKey>);

impl GroupMembers {
    pub fn parse(event: &Event) -> Result<Self, Error>;
    pub fn contains(&self, public_key: &PublicKey) -> bool;
    pub fn iter(&self) -> impl Iterator<Item = &PublicKey>;
}

/// kind:39003 — role names are relay-defined.
pub struct GroupRoles(Vec<Role>);
pub struct Role { pub name: String, pub description: Option<String> }

impl GroupRoles {
    pub fn parse(event: &Event) -> Result<Self, Error>;
    pub fn roles(&self) -> &[Role];
}

/// kind:10009 — the account's own list.
pub struct GroupList(Vec<GroupListEntry>);
pub struct GroupListEntry { pub id: GroupId, pub relay: RelayUrl, pub name: Option<String> }

impl GroupList {
    pub fn parse(event: &Event) -> Result<Self, Error>;
    pub fn entries(&self) -> &[GroupListEntry];
    pub fn upsert(&mut self, entry: GroupListEntry);
    pub fn remove(&mut self, key: &GroupKey);
    pub fn to_builder(&self) -> EventBuilder;   // kind 10009, group + r tags
}

pub enum Membership {
    Unknown,
    Pending { since: Timestamp },          // our 9021 went out; no answer yet
    Member,
    Refused { reason: Option<String> },    // the relay's OK false / CLOSED message
    Removed,
}

impl Membership {
    /// Folds 9000/9001 only; `Pending` and `Refused` are set by the join path.
    pub fn from_events(events: &[Event], me: PublicKey) -> Self;
}

/// The newest-first window (≤50) that `previous` references are drawn from.
pub struct TimelineWindow(Vec<Event>);

impl TimelineWindow {
    pub fn new() -> Self;
    pub fn push(&mut self, event: Event);                       // dedups, caps at 50
    pub fn previous_tag(&self, me: PublicKey) -> Option<Tag>;   // excludes `me`, ≤3 distinct 8-hex
}

Details that must be right:

• Every user event carries ["h", id] (from GroupId::h_tag); previous is added only when we have at least one eligible event, and its values are the first 8 characters of the hex ids.
• GroupReference::parse splits ?invite= before bech32 decoding; a missing relay hint is an error (there is nowhere to send the join request).
• GroupList::parse accepts group tags with 1–3 values (id, relay, name).
• Membership::from_events takes the newest 9000/9001 whose p tags name me; on an exact tie it prefers 9001 (removal). Pending and Refused are not things the relay folds into an event: the join path sets them from the 9021 send and from the relay's refusal message.
• GroupMetadata::supports distinguishes "tag absent" from "tag present with no values": the first means all kinds, the second means no text messages.
• The builders return EventBuilders so the caller decides when to sign; nothing in this file does I/O.

Phase 1 — read path (group.rs + lib.rs)

GroupsRegistry (global entity, mirroring CommunityRegistry):

pub struct GroupsRegistry {
    groups: Vec<Entity<Group>>,
    index: HashMap<GroupKey, Entity<Group>>,
    /// The account's 10009, loaded at start and mutated by join/leave
    list: GroupList,
    /// subscription id → group, rebuilt whenever a group resubscribes
    routes: Arc<RwLock<HashMap<SubscriptionId, GroupKey>>>,
    signal_tx: flume::Sender<Signal>, signal_rx: flume::Receiver<Signal>,
    notification_listener: Option<Task<Result<()>>>,  // cancelled on signer change
    signal_consumer: Option<Task<Result<()>>>,
    tasks: SmallVec<[Task<Result<()>>; 2]>,
    _subscriptions: SmallVec<[Subscription; 2]>,
}

impl GroupsRegistry {
    pub fn global(cx: &App) -> Entity<Self>;
    pub fn groups(&self) -> &[Entity<Group>];
    pub fn group(&self, key: &GroupKey) -> Option<Entity<Group>>;
    pub fn join(&mut self, reference: GroupReference, cx: &mut Context<Self>);
    pub fn leave(&mut self, key: &GroupKey, cx: &mut Context<Self>);
    fn load(&mut self, cx);           // own 10009 from the database into `self.list`
    fn publish_list(&mut self, cx);   // sign `self.list.to_builder()`, to_nip65, best-effort
}

• Subscriptions. One standing REQ per group, two filters in one id: key.id().timeline_filter(100) and key.id().state_filter(). The id is grp-<8 hex of the group key hash> (NIP-01 caps ids at 64 chars; the real key lives in routes). The relay is added and connected first (add_relay(url).and_connect()), like community::sync::connect_relays. On open, also issue a one-shot key.id().membership_filter(me) fetch so a restart learns 9000/9001 immediately; the state filter is fetched once (limit 1 per kind) so the sidebar has names before the live REQ settles. A dead relay costs only its own group.
• Pump. The listener matches ClientNotification::Message and routes RelayMessage::Event/EndOfStoredEvents/Closed by subscription id through routes. For h-tagged events it saves the envelope first, then sends Signal::Event(key, event); state events send Signal::State(key, event); Closed sends Signal::Refused(key, message). The consumer calls Group::ingest, which is the only place entity state changes.
• Envelope cache. Same shape as set_rumor/get_rumor in crates/chat: ApplicationSpecificData (30078, addressable) with d = event id (dedup on replay), r = "<relay>|<id>" (per-fork history), k = kind, content = the raw event JSON, signed by a crate-local LOCAL_KEYS. Every h-tagged event the standing subscription delivers is enveloped — any kind, because the previous window is literally "the last 50 events seen" (NIP-29 timeline references) — while the d-tagged state events (39000–39003) are not. Reading is Filter::new().custom_tag(LOWERCASE_R, key), parsed and sorted by created_at, then partitioned: the render set (1, 7, 9, 11, 1111) becomes messages, 9000/9001 fold membership, the rest only feed the window. This is what makes the timeline fork-safe and restart-safe without teaching the database about relays.
• Group state. GroupKey, metadata: Option<GroupMetadata>, metadata_event: Option<Event> (what GroupMetadata::naddr is built from), members, admins, roles, membership: Membership (including the local Pending/Refused access states), messages: Vec<Message> for the render set (reusing chat::Message, which already parses any event into display fields), an id index, a TimelineWindow for previous, loading, refused. It emits GroupEvent::{Incoming(Event), Updated, Refused(String), Eose} — the panel subscribes to these the way ChatPanel subscribes to RoomEvent.
• load(). Subscribe to our own 10009 (ReqTarget::auto, limit 1), then read the newest one from the database into self.list, and add one group per entry. A newer 10009 arriving from another device is merged (GroupList::upsert) and the new groups are subscribed; removals by another client cannot be detected in a replaceable list and are not guessed at.

Phase 2 — write path

All writes sign with the app signer (finalize_async(&signer)), send to the group's relay only, and treat the relay's OK as the truth.

• Send a message. Build with key.id().message(content, &replies, window.previous_tag(me)), sign, then client.send_event(&event).to([relay]). Kind 9 is the default message kind; when metadata.supports(Kind::ChatMessage) is false the composer is disabled rather than sending a kind the relay does not take. On acceptance: save the envelope, insert into messages, emit GroupEvent::Incoming. On OK false: return the relay's message (it already carries invalid: / restricted: / duplicate: prefixes) and let the panel show it. Reactions go through the same path with key.id().reaction(..).
• Join. GroupReference::parse → error if no relay hint → add/connect the relay → fetch 39000 (limit 1) to confirm the group and learn the self pubkey; GroupMetadata::parse it; a hidden group may refuse this — proceed without metadata rather than failing → create the entity, fetch state + membership, subscribe → send key.id().join_request(code, ..) — the h tag plus the code tag when an invite code was parsed → mark Membership::Pending { since } and self.list.upsert(..), so the group stays listed while it waits. The relay either emits 9000 (member), refuses the 9021 with OK false (set Refused { reason } from its message — the NIP says the message explains whether the rejection is final, pending review or payment-required), or refuses the timeline REQ with restricted: / auth-required: in a CLOSED (keep the state, do not raise an error toast). All three are observed on the standing subscription or the send result.
• Leave. Send key.id().leave_request(..); on OK, forget the group's envelopes (client.database().delete(filter) by r tag), drop the entity and self.list.remove(key), then publish. The relay also emits 9001, which the (now gone) subscription would have seen.
• Publish 10009. Sign self.list.to_builder() and publish with to_nip65(). The registry's GroupList is the one loaded at start, join upserts an entry and leave removes one, so a rewrite can never drop a group we loaded. The client saves the event locally even when the publish fails, so the list survives offline.

Phase 3 — later, in rough order

1. Moderation writes: 9000 put-user (roles), 9001 remove-user, 9002 edit-metadata, 9005 delete-event, 9009 create-invite; gate the UI on the caller appearing in 39001, let the relay be the enforcer.
2. Group creation (9007) where the relay supports it, and relay browsing (fetch_events(ReqTarget::single(relay, [Filter::new().kind(Kind::GroupMetadata).limit(..)]))).
3. Pins: 9010 write, 39005 read (Kind::Custom); subgroups: parse parent/child and lay groups out as a tree.
4. Range backfill: page the timeline with until for "load more".
5. LiveKit AV (livekit tag, 39004, the /.well-known/nip29/livekit token endpoint with a NIP-98 27235 authorization event) and fork/migration detection through the admins' 10009.

UI later — the hooks this backend leaves

UI action	Call
Paste an naddr and join (or retry a refused join)	GroupReference::parse → GroupsRegistry::join
Open a group	GroupsRegistry::group(&key, cx) → panel subscribes to GroupEvent
Timeline	Group::messages() (already chat::Messages)
Send / react	Group::send_message / send_reaction
Members, admins, roles, flags	Group::members/admins/roles/metadata
Membership badge (member, pending, refused, removed)	Group::membership()
Copy an invite link	GroupMetadata::naddr(relay, invite)
Leave	GroupsRegistry::leave

Where it lands: the sidebar gains a section reading GroupsRegistry::groups() (the same shape as its Communities section) with pending and refused groups listed alongside members; a new panel renders Vec<Message> with the existing ListState + ui::message::MessageRow and PersonRegistry for authors. The message data is already shared (chat::Message), so only ChatPanel's row-rendering methods need lifting into ui (or copying) rather than the type. A join dialog accepts a pasted naddr (with or without ?invite=). System notifications can hang off GroupEvent::Incoming when the window is inactive.

Nothing new is needed from GPUI to support this backend. The local checkouts are gpui-pre 0.3.5 (aliased as gpui) and gpui-base 0.7.0, which crates/ui styles; the only GPUI APIs the backend uses are the ones chat and community already use — Entity/WeakEntity, Context::spawn, background_spawn, EventEmitter, Subscription — and the panel's components (ListState, MessageRow, Avatar, Button, Scrollbar) already exist in the app.

Tests

• protocol.rs unit tests: GroupReference::parse (naddr, ?invite=, garbage, missing relay hint, an naddr whose kind is not 39000); GroupMetadata::parse flags and supports with supported_kinds absent vs empty; admins with roles; members; roles; GroupList round trip (parse → upsert → to_builder); TimelineWindow::previous_tag (excludes self, caps at 3, 8-char prefixes, any-kind window, None when empty); Membership::from_events (9000 vs 9001, newest wins, tie prefers removal; Pending/Refused are exercised through the join path); GroupId::message carries h and the reply e tags; GroupMetadata::naddr round trip with and without an invite code.
• Cache round trip with nostr-memory as a dev-dependency (the pattern crates/community already uses): save two envelopes (a kind 9 and a kind 9000), read them back sorted, assert the partition — the kind 9 renders, the 9000 only folds membership — re-save one and confirm no duplicate, delete clears the group.
• Routing: subscription id ↔ GroupKey mapping survives a resubscribe.
• One integration test against the SDK's LocalRelay (nostr-sdk dev-dependency with the local-relay feature): seed a 39000 and a kind 9 with an h tag, run Group::connect, assert metadata and the timeline land. Timeouts in GPUI-driven tests use cx.background_executor().timer(..), never smol::Timer, per AGENTS.md.

Decisions

1. Crate. crates/nip29 (the body above is written for it).
2. Timeline kinds. Render 1, 9, 11, 1111 and 7; every other kind is still cached and feeds the previous window, per the NIP's timeline references rule.
3. Default message kind. Kind 9; the composer is disabled when supported_kinds is present and lacks it.
4. Pending joins. Pending (and refused) groups are listed in the sidebar, with an access state on the row.
5. 10009 merge. Union-only; removals made by another client are not detected and no diffing is done.
6. Leave and the cache. Leaving wipes the group's envelopes.
7. Restricted/private groups. A visible access state is required: Membership::{Unknown, Pending, Member, Refused, Removed} with the relay's refusal reason shown and a retry, while the client's NIP-42 authenticator handles the reads.
8. References. naddr only (?invite= accepted); id@relay is not. A group whose relay publishes no 39000 cannot be referenced or joined in v1.