561 lines
41 KiB
Markdown
561 lines
41 KiB
Markdown
# Concord support — backend & API plan
|
||
|
||
Concord is an encrypted community/channel protocol over Nostr: shared-key "Private Streams" (CORD-01), communities with a self-certifying owner id and three planes (CORD-02), public/private channels (CORD-03), an owner-rooted signed roster (CORD-04), invites (CORD-05), rekeys/refoundings (CORD-06) and disappearing messages (CORD-08).
|
||
|
||
Scope of this plan: **backend + public Rust API**. No views, no widgets, no copy.
|
||
|
||
## 1. Scope
|
||
|
||
**In scope**
|
||
|
||
- New `crates/concord` crate: derivations, stream codec, control/chat/guestbook planes, authority fold, ephemeral state, invites, rekeys, dissolution, storage, sync engine.
|
||
- Public API for a future UI: `ConcordRegistry` + `Entity<Community>` / `Entity<Channel>` + events, mirroring the shape of `crates/chat`.
|
||
- The minimum surgical edits to existing crates required for coexistence (see §11).
|
||
|
||
**Out of scope**
|
||
|
||
- Any UI work.
|
||
- CORD-07 audio/video. Reserve `23313`, the `concord/voice-*` labels and the `voice` metadata flag so nothing else claims them, and implement nothing.
|
||
- Pins (CORD-04 §7) ship in the last milestone; the design accounts for `vsk 11` early so the fold is not retrofitted.
|
||
- Cross-client interop testing (Vector/Armada/Grimoire). Tracked as follow-up work, not blocking.
|
||
|
||
## 2. Sources of truth
|
||
|
||
| Doc | What we take from it |
|
||
| --- | --- |
|
||
| CORD-01 | Stream event shape, seal forms 20013/20014, encoding rules, binding, deletions |
|
||
| CORD-02 | `community_id`, `community_root`, `control_root`, epochs, 3 planes, metadata, invites, Community List, dissolution |
|
||
| CORD-03 | Channel keying, metadata, message kinds, `channel`/`epoch` binding, threads vs quotes |
|
||
| CORD-04 | Editions, `vac`, the roster, permission bits, banlist, the three removals, pins |
|
||
| CORD-05 | Bundle, link (naddr + fragment), relay dictionary, Invite List, Registry, Direct Invite |
|
||
| CORD-06 | Rekey blobs, chunking, `prevcommit` continuity, Refounding, compaction, races |
|
||
| CORD-08 | `message_expiration`, NIP-40 tagging, ingest/purge enforcement, timer notice 1740 |
|
||
|
||
Appendix A (derivations) and Appendix B (kinds) of CORD-02 are **frozen**: every labeled byte and every kind number is part of the wire format. Treat both as constants with golden-vector tests.
|
||
|
||
Reference implementations for cross-checking behaviour (not for copying code): Vector (`crates/vector-core/src/community/v2/*`), Armada, Grimoire.
|
||
|
||
## 3. Reuse map — nostr-sdk APIs we build on
|
||
|
||
Verified against the pinned revision (`nostr` 0.45.4 / `nostr-sdk` 0.45.2, git `b230cec`).
|
||
|
||
| Concord need | Existing API |
|
||
| --- | --- |
|
||
| NIP-44 under a raw conversation key | `nostr::nips::nip44::v2::{ConversationKey, encrypt_to_bytes_with_nonce, decrypt_to_bytes}` |
|
||
| NIP-44 conversation key for a keypair | `ConversationKey::derive(&SecretKey, &PublicKey)` (self-ECDH for streams) |
|
||
| NIP-44 under a signer | `nip44::{encrypt, decrypt}` (already wrapped by `state::UniversalSigner`) |
|
||
| Event id recomputation | `EventId::compute(pubkey, created_at, kind, tags, content)`, `UnsignedEvent::compute_id` |
|
||
| Event (de)serialization | `Event::{from_json, as_json, verify}`, `UnsignedEvent::from_json` |
|
||
| Building events | `EventBuilder::new(..).tags(..).custom_created_at(..)`, `FinalizeEvent::finalize(&Keys)` |
|
||
| Tags | `Tag::{custom, identifier, public_key, expiration}`, `Tags`, `SingleLetterTag` |
|
||
| Kinds | `Kind::GiftWrap` (1059), `Kind::Custom(21059|20013|20014|3308|…)`, `Kind::is_ephemeral` |
|
||
| Publish | `Client::send_event(&event).to(relays).ack_policy(AckPolicy::none())` |
|
||
| Subscribe | `Client::subscribe(target).with_id(..).close_on(..)`, `SubscribeAutoCloseOptions`, `ReqExitPolicy` |
|
||
| Backfill | `Client::fetch_events(target)`, `Client::stream_events(target)` |
|
||
| Local persistence | `Client::database()` → `query(Filter)`, `save_event(&Event)` |
|
||
| Invite link parsing | `Nip19::from_bech32` → `Nip19::Coordinate(Nip19Coordinate)` |
|
||
| Signer abstraction | `state::UniversalSigner` (`AsyncSignEvent` + `AsyncNip44`) |
|
||
| Relay auth | `nostr_sdk::{Authenticator, SignerAuthenticator}` |
|
||
|
||
**Not needed.** `secp256k1` (use `nostr::SecretKey::from_slice` + `Keys::new`), `base64` (use `data_encoding::BASE64`, already a workspace dep), `bech32` (NIP-19 is in the SDK), any new storage engine (the client's LMDB database is enough), any new HTTP client.
|
||
|
||
**Add one dependency:** `hkdf = "0.12"` (already in `Cargo.lock` transitively). Add it to `[workspace.dependencies]` and to the new crate. `sha2` is already a workspace dep.
|
||
|
||
## 4. Crate layout
|
||
|
||
New crate `crates/concord`, picked up automatically by the `crates/*` workspace member glob.
|
||
|
||
```
|
||
crates/concord/
|
||
Cargo.toml
|
||
src/lib.rs init, ConcordRegistry, ConcordEvent, signal bus, subscriptions, ingest pipeline
|
||
src/derive.rs frozen HKDF / group_key / locators / commitments + golden vectors
|
||
src/stream.rs CORD-01: seal + wrap + open (SealForm, OpenedStream), channel/epoch binding
|
||
src/edition.rs CORD-04 §1: canonical signing bytes, edition hash, parse, fold
|
||
src/control.rs control plane view, genesis, content types, roster fold, authority checks
|
||
src/guestbook.rs CORD-02 §5: join/leave/kick/snapshot, coalesce, complete memberlist
|
||
src/chat.rs CORD-03: channel plane — message/edit/delete/reaction builders + message view
|
||
src/invite.rs CORD-05: bundle, link, registry, Invite List, Direct Invite
|
||
src/rekey.rs CORD-06: blob codec, continuity, refounding, compaction, dissolution
|
||
src/store.rs local persistence + opened-rumor cache + history queries
|
||
```
|
||
|
||
`Community` and `Channel` GPUI entities live in `src/lib.rs` next to the registry — they are the public surface, not a separate concern. Ten modules, each with real content; no single-fn files.
|
||
|
||
Dependencies: `common`, `state`, `person`, `device`, `settings`, `gpui`, `nostr` (for `nip44` features), `nostr-sdk`, `hkdf`, `sha2`, `serde`, `serde_json`, `anyhow`, `flume`, `smallvec`, `itertools`, `futures`, `log`, `instant`. Everything under `cfg(not(target_arch = "wasm32"))` follows the `chat`/`state` split so the crate still builds for web.
|
||
|
||
## 5. Core types
|
||
|
||
```rust
|
||
pub struct CommunityId([u8; 32]); // sha256 commitment, never on the wire
|
||
pub struct ChannelId([u8; 32]);
|
||
pub struct Epoch(pub u64);
|
||
|
||
/// A derived stream: signing keypair + the self-ECDH conversation key that
|
||
/// encrypts the wraps. Memoised in a bounded process-wide cache.
|
||
pub struct GroupKey { keys: Keys, conversation: ConversationKey }
|
||
impl GroupKey {
|
||
pub fn pk(&self) -> PublicKey;
|
||
pub fn keys(&self) -> &Keys;
|
||
pub fn conversation(&self) -> &ConversationKey;
|
||
}
|
||
|
||
pub enum SealForm { Encrypted, Plaintext }
|
||
|
||
pub struct OpenedStream {
|
||
pub rumor_id: EventId,
|
||
pub author: PublicKey, // the seal's verified pubkey
|
||
pub seal_form: SealForm,
|
||
pub seal: Event, // retained: compaction re-wraps plaintext seals verbatim
|
||
pub wrapper_id: EventId,
|
||
pub at_ms: u64, // created_at * 1000 + ms tag
|
||
pub rumor: UnsignedEvent,
|
||
}
|
||
```
|
||
|
||
Ordering everywhere uses `at_ms`, never `created_at`, and ties break on the lower inner rumor id.
|
||
|
||
## 6. Frozen derivations (`derive.rs`)
|
||
|
||
```rust
|
||
fn build_info(label: &str, id: &[u8; 32], epoch: Option<u64>) -> Vec<u8>; // label ‖ 0x00 ‖ id[32] ‖ epoch_be[8]?
|
||
fn hkdf32(ikm: &[u8], info: &[u8]) -> [u8; 32]; // HKDF-SHA256, zero-length salt, L = 32
|
||
fn hkdf_to_secret_key(ikm: &[u8], info: &[u8]) -> SecretKey; // A.3 scalar_normalize, counter from 0
|
||
|
||
fn group_key(label: &str, secret: &[u8], id: &[u8; 32], epoch: Option<u64>) -> GroupKey;
|
||
|
||
pub fn channel_group_key(secret: &[u8; 32], channel: &ChannelId, epoch: Epoch) -> GroupKey;
|
||
pub fn control_group_key(root: &[u8; 32], id: &CommunityId, epoch: Epoch) -> GroupKey; // read key
|
||
pub fn control_signer_group_key(root: &[u8; 32], id: &CommunityId, epoch: Epoch) -> GroupKey; // write key
|
||
pub fn guestbook_group_key(root: &[u8; 32], id: &CommunityId, epoch: Epoch) -> GroupKey;
|
||
pub fn channel_rekey_group_key(root: &[u8; 32], channel: &ChannelId, epoch: Epoch) -> GroupKey;
|
||
pub fn base_rekey_group_key(root: &[u8; 32], id: &CommunityId, epoch: Epoch) -> GroupKey;
|
||
pub fn dissolved_group_key(id: &CommunityId) -> GroupKey; // no epoch field
|
||
|
||
pub fn community_id_of(owner_xonly: &[u8; 32], owner_salt: &[u8; 32]) -> CommunityId; // plain SHA-256
|
||
pub fn epoch_key_commitment(epoch: Epoch, key: &[u8; 32]) -> [u8; 32]; // plain SHA-256
|
||
pub fn grant_locator(id: &CommunityId, member: &[u8; 32]) -> [u8; 32];
|
||
pub fn banlist_locator(id: &CommunityId) -> [u8; 32];
|
||
pub fn pins_locator(id: &CommunityId, channel: &ChannelId) -> [u8; 32];
|
||
pub fn invite_links_locator(id: &CommunityId, creator: &[u8; 32]) -> [u8; 32];
|
||
pub fn recipient_locator(rotator: &[u8; 32], recipient: &[u8; 32], scope: &[u8; 32], epoch: Epoch) -> [u8; 32];
|
||
pub fn invite_bundle_key(token: &[u8; 16]) -> [u8; 32]; // raw hkdf32 output; used as a NIP-44 conversation key
|
||
```
|
||
|
||
Rules that must be enforced by construction, not by convention:
|
||
|
||
- Hex is lowercase everywhere; pubkeys are x-only hex, never bech32; tag numbers are decimal with no leading zeros (`"4"`, never `04`/`+4`).
|
||
- The epoch field is *omitted*, not zeroed, for labels with no epoch (`concord/dissolved`, locators, `concord/community`).
|
||
- `scalar_normalize` retries by appending a counter byte to the same `info`, starting at `0`.
|
||
- Labels and commitments are append-only. A test asserts every label is unique and that the label table matches Appendix A.6 exactly.
|
||
|
||
**Golden vectors.** `derive.rs` carries a `#[cfg(test)]` block pinning every derivation output, seeded from the independent Python vectors published by the Vector implementation (channel/control/control-signer/guestbook at epoch 0 and at `0x0102030405060708`, dissolved, all four locators, invite key, community id, epoch commitment). One vector is missing upstream — `pins_locator` — so we mint it from our own implementation and pin it, flagged in the test as self-referential. Changing any pinned value means the wire format changed.
|
||
|
||
## 7. Stream codec (`stream.rs`)
|
||
|
||
```rust
|
||
pub const KIND_WRAP: u16 = 1059;
|
||
pub const KIND_WRAP_EPHEMERAL: u16 = 21059;
|
||
pub const KIND_SEAL_ENCRYPTED: u16 = 20013;
|
||
pub const KIND_SEAL_PLAINTEXT: u16 = 20014;
|
||
pub const NIP44_MAX_PLAINTEXT: usize = 65_535;
|
||
|
||
pub fn seal_content(rumor: &UnsignedEvent, form: SealForm, group: &GroupKey) -> Result<String, StreamError>;
|
||
pub fn build_seal(rumor: &UnsignedEvent, form: SealForm, group: &GroupKey, author: &Keys) -> Result<Event, StreamError>;
|
||
pub fn wrap_seal(seal: &Event, group: &GroupKey, wrap_kind: u16, at: Timestamp, extra: &[Tag]) -> Result<(Event, Keys), StreamError>;
|
||
|
||
pub fn open_wrap(wrap: &Event, group: &GroupKey) -> Result<OpenedStream, StreamError>;
|
||
pub fn open_wrap_at(wrap: &Event, address: &PublicKey, conversation: &ConversationKey, verify_wrap_sig: bool) -> Result<OpenedStream, StreamError>;
|
||
|
||
pub fn channel_binding_tags(channel: &ChannelId, epoch: Epoch) -> Vec<Tag>;
|
||
pub fn check_channel_binding(rumor: &UnsignedEvent, channel: &ChannelId, epoch: Epoch) -> Result<(), StreamError>;
|
||
pub fn build_rumor(kind: u16, author: PublicKey, content: &str, tags: Vec<Tag>, at_ms: u64) -> UnsignedEvent; // appends ["ms", n]
|
||
pub fn resolve_ms_strict(rumor: &UnsignedEvent) -> Result<u64, StreamError>;
|
||
```
|
||
|
||
Design points that are easy to get wrong:
|
||
|
||
- The wrap is signed by the **stream key** with a random ephemeral `p` tag — NIP-59 reversed. `extra` is how the caller mirrors a NIP-40 expiration onto the wrap.
|
||
- The seal is signed by the **real author** and carries `created_at` equal to the rumor's. It is never published bare.
|
||
- Control plane **must** use the plaintext seal; chat, guestbook and rekey planes **must** use the encrypted one. Each plane asserts its own form at both ends.
|
||
- The control plane is a write-restricted stream: the wrap key derives from `control_root` while the content is encrypted under the `community_root`-derived conversation key. `open_wrap_at` takes the two halves separately for this reason.
|
||
- Open order: kind → address match → wrap signature (only when `verify_wrap_sig`) → NIP-44 open → seal kind → seal signature → rumor parse → `rumor.pubkey == seal.pubkey` → recompute the rumor id and reject a mismatch → strict `ms` resolve.
|
||
- Enforce the 65 535-byte NIP-44 cap at every nesting layer before publishing.
|
||
- Do not verify the wrap signature for ordinary planes: every reader holds the group key, so the signature proves nothing. It matters only for the restricted control plane and for rekeys.
|
||
- The ephemeral wrap keypair is returned to the caller so a client may NIP-09-scrub its own wrap later.
|
||
|
||
## 8. Planes, state and folds
|
||
|
||
### 8.1 Editions and authority (`edition.rs`, `control.rs`)
|
||
|
||
```rust
|
||
pub const EDITION_LABEL: &[u8] = b"vector-community/v1/edition"; // frozen, cross-client
|
||
|
||
pub fn edition_hash(entity: &[u8; 32], version: u64, prev: Option<&[u8; 32]>, content: &[u8]) -> [u8; 32];
|
||
pub struct ParsedEdition { author: PublicKey, vsk: String, entity: [u8; 32], version: u64,
|
||
prev: Option<[u8; 32]>, content: String, self_hash: [u8; 32] };
|
||
pub fn parse_edition(rumor: &UnsignedEvent) -> Result<ParsedEdition, EditionError>;
|
||
|
||
pub struct FoldResult { pub head: Option<usize>, pub gap: bool, pub anchored: bool }
|
||
pub fn fold(editions: &[EditionMeta], floor: u64, floor_hash: Option<&[u8; 32]>) -> FoldResult;
|
||
pub fn bootstrap_head(editions: &[EditionMeta], floor: u64) -> Option<usize>;
|
||
```
|
||
|
||
- Tag grammar: `["vsk", sub]`, `["eid", hex32]`, `["ev", decimal]`, `["ep", hex32]`, `["vac", eid, version, hash]`. Duplicates of any of the five reject the edition; `ev` must pass a decimal check before parsing.
|
||
- Tie-break at equal version is the lower **inner rumor id**, never `created_at`.
|
||
- `gap` is a safety signal: a tracking client (already holds the floor) fails closed for that entity and refetches; a bootstrapping client (floor 0) may accept the highest authority-verified head, which is what makes compaction re-wrapping work.
|
||
- Entity coordinates are `vsk 0` → `community_id`, `1` → `role_id`, `2` → `channel_id`, `3` → `grant_locator`, `4` → `banlist_locator`, `8` → `invite_links_locator`, `11` → `pins_locator`. All derive from `community_id` only, so a refounding re-wraps heads verbatim.
|
||
|
||
```rust
|
||
pub const P_MANAGE_ROLES: u64 = 1 << 0; // …bit table from CORD-04 §3, frozen; retired bits are burned
|
||
pub struct CommunityRoles { roles: BTreeMap<[u8; 32], Role>, grants: BTreeMap<PublicKey, Grant> }
|
||
impl CommunityRoles {
|
||
pub fn permissions_of(&self, member: &PublicKey) -> u64; // union of role bits
|
||
pub fn position_of(&self, member: &PublicKey, owner: &PublicKey) -> u32;
|
||
pub fn is_authorized(&self, actor: &PublicKey, owner: &PublicKey, bit: u64) -> bool;
|
||
pub fn is_authorized_in(&self, actor: &PublicKey, owner: &PublicKey, channel: &ChannelId, bit: u64) -> bool;
|
||
pub fn outranks(&self, actor: &PublicKey, owner: &PublicKey, target_position: u32) -> bool;
|
||
pub fn can_act_on(&self, actor: &PublicKey, owner: &PublicKey, target: &PublicKey, bit: u64) -> bool;
|
||
pub fn is_staff(&self, member: &PublicKey, owner: &PublicKey) -> bool; // the six control bits, CORD-04 §3
|
||
}
|
||
```
|
||
|
||
Authority rules to encode once and test hard:
|
||
|
||
- The owner is position 0, derived from `community_id`, and is never removable.
|
||
- No edition may claim a `position` at or above its own signer's, including the owner: no Role may claim 0.
|
||
- The actor must hold the required bit **and strictly outrank** the target. Equal cannot act on equal.
|
||
- A `vac` citation is a sync floor, not a verdict: block until the cited Grant version is folded, verify its hash, then judge against the *current* roster.
|
||
- A staff-making Grant carries `control_wrap`, a NIP-44 pairwise ciphertext of `epoch_be[8] ‖ control_root[32]`, and is adopted **only if it derives to the `control_pk` the member already holds** for the named epoch.
|
||
- Banlist is one replaced entity; mutations carry a re-heal step (re-fold after publish, re-apply if the addition lost the tiebreak).
|
||
|
||
### 8.2 Communities, channels, metadata
|
||
|
||
`CommunityMetadata` carries `name` (≤ 64 bytes), `description` (≤ 10 000 bytes), `relays` (truncated on read and write to 5), `icon` and `banner` as encrypted-blob pointers (`{url, key, nonce, hash}`), `message_expiration`, and the optional `custom` object. `ChannelMetadata` carries `name`, `private`, optional `voice`, `deleted`, optional `custom`.
|
||
|
||
Every content struct uses `#[serde(flatten)] extra: serde_json::Map<String, Value>` and round-trips unknown fields. A name edit by an older client must not wipe another client's `custom` keys. Round-trip discipline gets its own test.
|
||
|
||
Channel keying follows CORD-03 §1: a public channel derives from `community_root` at the base epoch, a private one from its own random key at its own epoch. Public→private is a rekey at the next channel epoch (monotonic, never reset); private→public derives from the base again and the prior private history stays unreadable to later joiners.
|
||
|
||
### 8.3 Guestbook and member list (`guestbook.rs`)
|
||
|
||
```rust
|
||
pub enum GuestbookEntry {
|
||
Join { member: PublicKey, at_ms: u64, invited_by: Option<(String, String)> },
|
||
Leave { member: PublicKey, at_ms: u64 },
|
||
Kick { actor: PublicKey, target: PublicKey, at_ms: u64, citation: Option<AuthorityCitation> },
|
||
Snapshot { refounder: PublicKey, members: Vec<PublicKey>, snapshot_id: [u8; 32], chunk: (u32, u32), at_ms: u64 },
|
||
}
|
||
|
||
pub fn coalesce(events: &[GuestbookEvent], now_ms: u64, snapshot_authority: Option<&PublicKey>,
|
||
can_kick: impl Fn(&PublicKey, &PublicKey, Option<&AuthorityCitation>) -> bool)
|
||
-> BTreeMap<PublicKey, MemberState>;
|
||
|
||
pub fn complete_memberlist(coalesced: &BTreeMap<PublicKey, MemberState>,
|
||
observed: &BTreeMap<PublicKey, u64>, // author → newest ms published
|
||
banned: &BTreeSet<PublicKey>, banned_at: &BTreeMap<PublicKey, u64>,
|
||
refound: Option<&Refound>) -> BTreeSet<PublicKey>;
|
||
```
|
||
|
||
- Entries dated more than an hour ahead of local time are dropped. An `ms` outside `0..999` drops the entry rather than being interpreted.
|
||
- Coalesce is per npub, one final state each, by millisecond time, ties on the lower inner rumor id.
|
||
- A Kick counts only when its signer holds `KICK` and outranks the target.
|
||
- A Snapshot counts only from the npub whose Refounding minted the epoch. There is deliberately no owner fallback.
|
||
- The member list is `coalesced Joined ∪ observed authors − banlist`, and observation counts **forward only** (an author re-enters on activity newer than their latest Leave/Kick/Ban). A Grant holder with `ms == 0` is present by construction.
|
||
|
||
### 8.4 Chat plane (`chat.rs`)
|
||
|
||
```rust
|
||
pub struct ChatMessage {
|
||
pub id: EventId, // recomputed rumor id
|
||
pub author: PublicKey,
|
||
pub channel: ChannelId,
|
||
pub epoch: Epoch,
|
||
pub kind: Kind, // 9 | 1111 | 3302 | 1740 | 15
|
||
pub content: String,
|
||
pub media: Vec<SharedUri>,
|
||
pub mentions: Vec<Mention>,
|
||
pub reply_to: Option<EventId>, // lowercase `e`/`q`
|
||
pub thread_root: Option<EventId>, // uppercase `E` for 1111
|
||
pub at_ms: u64,
|
||
pub expiration: Option<Timestamp>,
|
||
pub edited_at: Option<u64>, // folded from 3302
|
||
pub deleted: bool, // folded from 5
|
||
pub reactions: BTreeMap<PublicKey, String>,
|
||
}
|
||
```
|
||
|
||
Sends funnel through one function so the rules cannot drift:
|
||
|
||
```rust
|
||
fn publish_chat(store, client, community, channel, epoch, group, rumor, at_ms, ephemeral) -> Task<Result<Output<EventId, EventSendStatus>, Error>>;
|
||
```
|
||
|
||
It builds the seal + wrap, mirrors the NIP-40 tag onto the wrap for durable kinds, publishes via `send_event(..).to(relays)`, retains the ephemeral wrap key for later NIP-09 scrubbing, and locally echoes its own wrap through the same ingest path so send-then-read works without waiting on a relay round-trip.
|
||
|
||
Disappearing messages (CORD-08) live here: `message_expiration` is read from the folded metadata, `["expiration", created_at + t]` is attached to every durable Chat rumor and to the wrap, kinds 5 and 1740 are exempt, ingest refuses an already-expired rumor, a periodic sweep purges stored ones, and the kind 1740 timer notice renders only when its author holds `MANAGE_METADATA`.
|
||
|
||
### 8.5 Invites (`invite.rs`)
|
||
|
||
```rust
|
||
pub struct CommunityInvite { community_id, owner, owner_salt, community_root, root_epoch,
|
||
control_pk: Option<String>, channels: Vec<ChannelGrant>,
|
||
relays: Vec<String>, name: String, icon: Option<ImageRef>,
|
||
expires_at: Option<u64>, creator_npub: Option<String>, label: Option<String>,
|
||
extra: Map<String, Value> }
|
||
impl CommunityInvite { pub fn validate(&self) -> Result<(), InviteError>; // recompute community_id
|
||
pub fn expired(&self, now_ms: u64) -> bool; }
|
||
|
||
pub fn build_bundle(token: &[u8; 16], link_signer: &Keys, invite: &CommunityInvite) -> Result<Event, InviteError>; // 33301, d = ""
|
||
pub fn build_revocation(link_signer: &Keys) -> Result<Event, InviteError>; // vsk 9
|
||
pub fn parse_link(input: &str) -> Result<ParsedInviteLink, InviteError>;
|
||
pub fn encode_fragment(relays: &[RelayUrl], token: &[u8; 16]) -> String; // version byte 4, flags, ≤ 3 relays, base64url
|
||
pub fn decode_fragment(fragment: &str) -> Result<(Vec<RelayUrl>, [u8; 16]), InviteError>;
|
||
pub fn build_direct_invite(receiver: &PublicKey, invite: &CommunityInvite, signer: &UniversalSigner) -> Task<Result<Event, Error>>; // 3313 rumor → 13 seal → k-tagged 1059
|
||
```
|
||
|
||
The link rides `naddr` (`Nip19Coordinate` for kind 33301, link signer, empty `d`) in the path and the token + bootstrap relays in the fragment. A fragment is never sent to a server. The bundle is decrypted with `invite_bundle_key(token)`, and the joiner must recompute `community_id` from `owner` + `owner_salt`.
|
||
|
||
Bounds before allocation: reject a bundle with more than 256 channels, truncate the relay list to 5, refuse an expired one.
|
||
|
||
### 8.6 Rekeys and refoundings (`rekey.rs`)
|
||
|
||
```rust
|
||
pub enum RekeyScope { Channel(ChannelId), Base }
|
||
pub fn encode_blob_plaintext(scope, epoch, new_root, control_pk, control_root) -> Vec<u8>; // 72 | 104 | 136 bytes
|
||
pub fn parse_blob_plaintext(bytes: &[u8], scope, epoch) -> Result<KeyDelivery, RekeyError>;
|
||
pub fn build_rekey_rumor(rotator, scope, new_epoch, prev_epoch, prev_commit, blobs, chunk) -> UnsignedEvent;
|
||
pub fn plan_refounding(fold, removed: &[PublicKey]) -> Result<Refounding, RekeyError>;
|
||
pub fn compact(fold, epoch, new_control_root, ...) -> Vec<Event>; // re-wrap heads verbatim, plaintext seals preserved
|
||
```
|
||
|
||
- The subscription for rekeys is precomputed from the *next* epoch's address, per private channel and once for the base.
|
||
- A receiver accepts a key only after: locating its blob, decrypting with the rotator↔recipient conversation key, checking the bound `scope` and `epoch` inside the plaintext, and matching `prevcommit` against the key it currently holds.
|
||
- Only after holding **all** `n` chunks of one `(rotator, newepoch, prevcommit)` set, with none containing its locator, may a client conclude it was removed.
|
||
- Send cap 80 blobs per event, accept cap 120 (Vector's documented erratum: the CORD-01 double envelope pushes 120 blobs past a 64 KB relay limit). Record the reason in a comment so nobody "fixes" it back.
|
||
- Compacted control heads are re-wrapped with their original signature intact, which is exactly why the control plane uses the plaintext seal.
|
||
- Two concurrent refoundings converge on the lexicographically lowest new base key; the heal is down-only.
|
||
- Authority: a channel rekey needs `MANAGE_CHANNELS`, a refounding needs `BAN`, and in both the rotator must strictly outrank every removed target. Holding a key is never authority.
|
||
|
||
Dissolution (CORD-02 §9) also lives here: a chainless, owner-signed `vsk 10` tombstone at `dissolved_group_key(id)`, plaintext-sealed, and a verifier **must** refuse any tombstone whose `eid` is not the community's own id (including the all-zero placeholder — accepting it lets an owner's genuine tombstone for one community be re-wrapped at another of theirs and kill it permanently). On sight the community is sealed read-only: subscriptions halt, nothing new is honored, existing history stays readable, and a member's delete of their own message is still honored.
|
||
|
||
## 9. Storage (`store.rs`)
|
||
|
||
Three layers, no new storage engine:
|
||
|
||
1. **Raw wraps** (kind 1059) are persisted automatically by the SDK's relay pool when a subscription or fetch matches a filter. Nothing to write.
|
||
2. **Opened rumors** are cached locally as NIP-78 `Kind::ApplicationSpecificData` events signed by a session-local keypair, exactly like `chat::set_rumor`. Tags: `["d", rumor_id]` (replace key), `["c", channel_hex]`, `["p", author]`, `["k", kind]`, `["e", wrap_id]`, `["t", "concord"]`. Contents are the rumor JSON.
|
||
- The `c`/`t` keys deliberately differ from chat's `r` key so the two message namespaces can never collide in one database.
|
||
- The read path dedupes by rumor id and keeps the newest `created_at`, because the local signing key changes per session.
|
||
3. **Community state** — one local document per community, `Kind::ApplicationSpecificData` with `["d", "concord/<community_id>"]`:
|
||
|
||
```rust
|
||
pub struct CommunityState {
|
||
pub id: CommunityId,
|
||
pub owner: PublicKey,
|
||
pub owner_salt: [u8; 32],
|
||
pub community_root: [u8; 32],
|
||
pub root_epoch: Epoch,
|
||
pub control_root: Option<[u8; 32]>, // present iff the holder is staff
|
||
pub control_pks: BTreeMap<Epoch, PublicKey>,
|
||
pub channels: Vec<ChannelKeyRef>, // id, key, epoch, name, private
|
||
pub epoch_keys: Vec<([u8; 32], Epoch, [u8; 32])>, // (scope, epoch, key) — the history backfill index
|
||
pub relays: Vec<RelayUrl>,
|
||
pub heads: BTreeMap<[u8; 32], (u64, [u8; 32], EventId)>, // entity → (version, self_hash, inner id)
|
||
pub guestbook: Vec<GuestbookEvent>,
|
||
pub observed: BTreeMap<PublicKey, u64>,
|
||
pub banned: BTreeSet<PublicKey>,
|
||
pub dissolved: bool,
|
||
pub added_at_ms: u64,
|
||
}
|
||
```
|
||
|
||
Writes are debounced (a fold head changes on every edition); reads load once at init.
|
||
|
||
**Decision, stated for the record:** this document stores raw community keys unencrypted in a plaintext local database. That matches the existing posture — `chat` already caches decrypted message rumors in the same LMDB. If that posture ever changes, the state document is the one to wrap with NIP-44-to-self, since it is the only local artifact holding keys.
|
||
|
||
History queries:
|
||
|
||
```rust
|
||
pub async fn query_messages(&self, channel: &ChannelId, until: Option<Timestamp>, limit: usize) -> Result<Vec<Event>, Error>;
|
||
pub async fn backfill(&self, plane_authors: &[PublicKey], relays: &[RelayUrl], until: Option<Timestamp>, limit: usize) -> Result<Vec<Event>, Error>;
|
||
```
|
||
|
||
`query_messages` reads the local cache (`Filter::new().kind(ApplicationSpecificData).custom_tag(LOWERCASE_C, channel_hex)`); `backfill` pages relays newest-first with `until`, deduplicating by wrap id and stepping past same-second walls.
|
||
|
||
## 10. Sync engine and GPUI conventions
|
||
|
||
`ConcordRegistry` mirrors `ChatRegistry`'s shape exactly: a foreground GPUI entity holding `Entity<Community>` handles, a `flume` signal bus, one background notification listener, one foreground consumer, and task slots that are cleared when the signer changes.
|
||
|
||
**Subscription.** Community relays come from the folded metadata. `init`/`join` add them to the client (`client.add_relay(url).and_connect()`), then:
|
||
|
||
```rust
|
||
let filter = Filter::new()
|
||
.kinds([Kind::GiftWrap, Kind::Custom(KIND_WRAP_EPHEMERAL)])
|
||
.authors(plane_authors) // guestbook, control signer, all held channel planes,
|
||
// rekey addresses for epoch + 1, dissolved address
|
||
.since(Timestamp::from_secs(now - FRESH_WINDOW))
|
||
.limit(0); // live tail only; history comes from backfill
|
||
client.subscribe(target).with_id(SubscriptionId::new(CONCORD_SUB)).await?;
|
||
```
|
||
|
||
Targeted subscribe against the community relays, with a pool-wide subscribe as the fallback path. Rebuild idempotently whenever a plane's address changes (join, channel added, rekey folded).
|
||
|
||
**Routing.** `dispatch` matches on the `subscription_id` carried in `RelayMessage::Event`, dedupes by wrap id (both subscriptions and several relays deliver the same wrap), then recognises the plane by **wrap author** against the derived addresses it holds — never by trial decryption. Recognition order: held channel planes → guestbook → control signer → rekey addresses → dissolved.
|
||
|
||
**Ingest pipeline.** Unwrap, verify and fold all happen inside `cx.background_spawn`, never on the foreground thread: secp256k1 verification per edition and per seal is far too expensive for the UI thread.
|
||
|
||
```rust
|
||
enum Signal {
|
||
Chat { community: CommunityId, channel: ChannelId, message: Box<ChatMessage> },
|
||
Control { community: CommunityId, heads: Vec<EntityHead>, roster: Box<FoldedRoster> },
|
||
Guestbook { community: CommunityId, members: BTreeSet<PublicKey> },
|
||
Rekey { community: CommunityId, scope: RekeyScope, epoch: Epoch },
|
||
Dissolved(CommunityId),
|
||
Eose(SubscriptionId),
|
||
Error(ConcordError),
|
||
}
|
||
```
|
||
|
||
The consumer is `cx.spawn(async move |this, cx| { while let Ok(signal) = rx.recv_async().await { this.update(cx, |this, cx| this.apply(signal, cx))?; } })`, which updates entities and calls `cx.notify()`.
|
||
|
||
Rules taken from the project guidelines:
|
||
|
||
- Crypto, folding, database queries and network I/O only in `cx.background_spawn`.
|
||
- Foreground tasks are `cx.spawn` with `this.update(cx, ..)`; any entity update happens there, and the inner `cx` is always used.
|
||
- Tasks are stored in fields (`tasks`, `listener`, `consumer`) so they are cancelled on signer change and dropped with the registry. `detach()` only for genuinely fire-and-forget work such as the local state save.
|
||
- Long-running paging is bounded by explicit page and step caps, not by unbounded loops.
|
||
- Every fallible path returns `Result` and surfaces through `ConcordEvent::Error`; nothing is silently swallowed.
|
||
|
||
**Registry API.**
|
||
|
||
```rust
|
||
pub fn init(window: &mut Window, cx: &mut App);
|
||
pub struct ConcordRegistry { /* … */ }
|
||
impl ConcordRegistry {
|
||
pub fn global(cx: &App) -> Entity<Self>;
|
||
pub fn loading(&self) -> bool;
|
||
pub fn communities(&self) -> Vec<Entity<Community>>;
|
||
pub fn community(&self, id: &CommunityId, cx: &App) -> Option<WeakEntity<Community>>;
|
||
pub fn find(&self, query: &str, cx: &App) -> Vec<Entity<Community>>;
|
||
|
||
pub fn create(&mut self, params: CommunityParams, cx: &mut Context<Self>) -> Task<Result<CommunityId, Error>>;
|
||
pub fn join(&mut self, link: &str, cx: &mut Context<Self>) -> Task<Result<CommunityId, Error>>;
|
||
pub fn accept_direct_invite(&mut self, rumor: &UnsignedEvent, cx: &mut Context<Self>) -> Task<Result<CommunityId, Error>>;
|
||
pub fn leave(&mut self, id: &CommunityId, cx: &mut Context<Self>);
|
||
pub fn discard_invite(&mut self, id: &CommunityId, cx: &mut Context<Self>);
|
||
pub fn refresh(&mut self, id: &CommunityId, cx: &mut Context<Self>);
|
||
pub fn shutdown(&mut self, cx: &mut Context<Self>); // halt subscriptions, keep our own state
|
||
}
|
||
```
|
||
|
||
**Community API** (`Entity<Community>`, `EventEmitter<CommunityEvent>`):
|
||
|
||
```rust
|
||
pub fn id(&self) -> CommunityId;
|
||
pub fn owner(&self) -> PublicKey;
|
||
pub fn name(&self) -> SharedString; pub fn description(&self) -> Option<SharedString>;
|
||
pub fn icon(&self) -> Option<ImageRef>;
|
||
pub fn relays(&self) -> Vec<RelayUrl>;
|
||
pub fn epoch(&self) -> Epoch;
|
||
pub fn dissolved(&self) -> bool;
|
||
pub fn channels(&self) -> Vec<Entity<Channel>>;
|
||
pub fn channel(&self, id: &ChannelId, cx: &App) -> Option<WeakEntity<Channel>>;
|
||
pub fn members(&self) -> BTreeSet<PublicKey>;
|
||
pub fn banned(&self) -> BTreeSet<PublicKey>;
|
||
pub fn roles(&self) -> &CommunityRoles;
|
||
pub fn permissions(&self, member: &PublicKey) -> u64;
|
||
pub fn is_staff(&self, member: &PublicKey) -> bool;
|
||
pub fn message_expiration(&self) -> Option<u64>;
|
||
|
||
// authority actions — each returns a publish task and nothing optimistic
|
||
pub fn set_metadata(&mut self, meta: CommunityMetadata, cx: &mut Context<Self>) -> Task<Result<(), Error>>;
|
||
pub fn create_channel(&mut self, name: &str, private: bool, cx: &mut Context<Self>) -> Task<Result<ChannelId, Error>>;
|
||
pub fn edit_channel(&mut self, id: &ChannelId, meta: ChannelMetadata, cx: &mut Context<Self>) -> Task<Result<(), Error>>;
|
||
pub fn create_role(&mut self, role: Role, cx: &mut Context<Self>) -> Task<Result<(), Error>>;
|
||
pub fn assign_roles(&mut self, member: &PublicKey, roles: &[[u8; 32]], cx: &mut Context<Self>) -> Task<Result<(), Error>>;
|
||
pub fn ban(&mut self, members: &[PublicKey], cx: &mut Context<Self>) -> Task<Result<(), Error>>;
|
||
pub fn unban(&mut self, members: &[PublicKey], cx: &mut Context<Self>) -> Task<Result<(), Error>>;
|
||
pub fn kick(&mut self, member: &PublicKey, cx: &mut Context<Self>) -> Task<Result<(), Error>>;
|
||
pub fn rekey_channel(&mut self, id: &ChannelId, removed: &[PublicKey], cx: &mut Context<Self>) -> Task<Result<(), Error>>;
|
||
pub fn refound(&mut self, removed: &[PublicKey], cx: &mut Context<Self>) -> Task<Result<(), Error>>;
|
||
pub fn dissolve(&mut self, cx: &mut Context<Self>) -> Task<Result<(), Error>>;
|
||
pub fn create_invite(&mut self, params: InviteParams, cx: &mut Context<Self>) -> Task<Result<String, Error>>;
|
||
pub fn revoke_invite(&mut self, token: &[u8; 16], cx: &mut Context<Self>) -> Task<Result<(), Error>>;
|
||
pub fn direct_invite(&mut self, receiver: &PublicKey, cx: &mut Context<Self>) -> Task<Result<(), Error>>;
|
||
pub fn save_community_list(&mut self, cx: &mut Context<Self>) -> Task<Result<(), Error>>; // kind 13302, multi-device sync
|
||
```
|
||
|
||
**Channel API** (`Entity<Channel>`): `id`, `name`, `private`, `epoch`, `deleted`, plus
|
||
|
||
```rust
|
||
pub fn messages(&self, until: Option<Timestamp>, limit: usize, cx: &App) -> Task<Result<Vec<ChatMessage>, Error>>;
|
||
pub fn send(&self, content: &str, reply_to: Option<EventId>, cx: &App) -> Task<Result<Output<EventId, EventSendStatus>, Error>>;
|
||
pub fn send_file(&self, file: FileAttachment, reply_to: Option<EventId>, cx: &App) -> Task<Result<Output<EventId, EventSendStatus>, Error>>;
|
||
pub fn edit(&self, id: EventId, content: &str, cx: &App) -> Task<Result<Output<EventId, EventSendStatus>, Error>>;
|
||
pub fn delete(&self, id: EventId, cx: &App) -> Task<Result<Output<EventId, EventSendStatus>, Error>>;
|
||
pub fn react(&self, id: EventId, emoji: &str, cx: &App) -> Task<Result<Output<EventId, EventSendStatus>, Error>>;
|
||
pub fn typing(&self, cx: &App) -> Task<Result<(), Error>>; // kind 23311, ephemeral
|
||
pub fn pin(&self, id: EventId, cx: &App) -> Task<Result<(), Error>>; // vsk 11, PIN_MESSAGES
|
||
```
|
||
|
||
`CommunityEvent` and `ChannelEvent` mirror `ChatEvent`: one variant per thing the UI has to react to (`Updated`, `Members`, `Added`, `Removed`, `Dissolved`, `Error`, plus channel-level `Incoming`, `Reload`).
|
||
|
||
## 11. Integration with existing crates
|
||
|
||
1. **`crates/chat/src/lib.rs` — required fix.** `handle_notifications` currently treats *every* kind 1059 event as a NIP-17 wrap for the current user and pushes an unwrap failure into the trash. Concord wraps are kind 1059 with an ephemeral `p` tag, so they would flood the trash and leak error toasts. Route by `subscription_id` from `RelayMessage::Event` against `sub_id1`/`sub_id2`, and drop the `if rumor.tags.is_empty()` heuristic once the real recipient check is in place.
|
||
2. **`desktop/src/main.rs` and `web/src/lib.rs`** — add `concord::init(window, cx)` after `chat::init(window, cx)`.
|
||
3. **`Cargo.toml`** — add `hkdf = "0.12"` to `[workspace.dependencies]`; add the crate to `desktop` and `web` dependencies. No other workspace changes.
|
||
4. **No changes** to `state`, `person`, `device`, `settings`, `common`, or `ui`.
|
||
|
||
## 12. Security invariants to test, not to assume
|
||
|
||
Each of these has burned a real implementation, or is a documented cross-client trap:
|
||
|
||
- Recompute every rumor id and reject a claimed mismatch; never trust an embedded `id`.
|
||
- Require `rumor.pubkey == seal.pubkey`.
|
||
- Require the plaintext seal form on Control and the encrypted form on Chat/Guestbook/Rekey — a strict reader must drop a mis-sealed edition rather than fold a chain a later compaction would fork.
|
||
- Check `channel` **and** `epoch` against the plane whose key opened the wrap; reject duplicates of either tag.
|
||
- Reject duplicate `vsk`/`eid`/`ev`/`ep`/`vac` tags; require decimal-with-no-leading-zeros on every numeric tag.
|
||
- Refuse a tombstone whose `eid` is not this community's id.
|
||
- Adopt a `control_root` from a Grant only if it derives to the `control_pk` held for that epoch; adopt a rekey blob only if its bound plaintext matches the scope and epoch and its `prevcommit` matches the key currently held.
|
||
- Never conclude removal from a partial rekey chunk set.
|
||
- Drop guestbook entries more than an hour in the future; treat an out-of-range `ms` as malformed, not as an interpretation opportunity.
|
||
- Never honour a Snapshot from anyone but the refounder of that epoch.
|
||
- Refuse to write a Pin List from a list the writer could not read.
|
||
- Enforce the NIP-44 65 535-byte cap at every layer before publishing, and the 5-relay / 256-channel / 50-membership / 100-roles / 64-roles-per-member / 500-banlist / 25-pin caps at their ingest and write points.
|
||
- Lowercase hex only; x-only pubkeys only; no version tag anywhere.
|
||
|
||
## 13. Milestones
|
||
|
||
| # | Deliverable | Done when |
|
||
| --- | --- | --- |
|
||
| M0 | Crate skeleton, `derive.rs`, golden vectors, workspace wiring | `cargo test -p concord` pins every derivation from an independent vector set; all labels match Appendix A.6 |
|
||
| M1 | `stream.rs` + `store.rs` | seal/wrap/open round-trips for both seal forms; malformed inputs rejected in the documented order; local cache reads back after a restart |
|
||
| M2 | `edition.rs` + `control.rs` genesis | a community is created and published; its two genesis wraps open at a second client sharing the keys; edition hash matches the cross-client vector |
|
||
| M3 | Control fold + roster + metadata/channels | fold tests for chains, gaps, downgrade refusal, fork tiebreak, compaction dangle; metadata and channel edits visible to a second client |
|
||
| M4 | Chat plane | send/receive/edit/delete/reaction across two identities; history pages backwards across an epoch boundary; binding checks reject a foreign channel/epoch |
|
||
| M5 | Guestbook + member list + moderation | join/leave/kick/ban converge to the same memberlist on both clients; every authority rule from §8.1 has a negative test |
|
||
| M6 | Invites + Community List | link mint → fetch → join round-trips; revoked link refuses; direct invite lands in the recipient's giftwrap inbox via the `k` tag; a second device reconstructs membership from 13302 |
|
||
| M7 | Rekeys + refounding + dissolution | a removed member stops reading after a rekey; continuity and race rules tested; a tombstone seals the community and a foreign-id tombstone is refused |
|
||
| M8 | Pins + disappearing messages + hardening | pins verify from a keyless reader's view; expiry is refused at ingest and purged by the sweep; the audit of §12 is complete with a test per bullet |
|
||
|
||
Ordering is deliberately dependency-first: each milestone is usable on its own, and nothing in M2+ depends on a later milestone.
|
||
|
||
## 14. Open questions and risks
|
||
|
||
1. **Community List kind.** CORD-02 §8 specifies `13302`, replaceable. Vector has retired it in favour of fragmented `33302`, because a replaceable kind holds one event per pubkey and so cannot shard past the NIP-44 size cap. We implement `13302` per spec, enforce the 50-membership cap and pre-publish size check, and treat `33302` as an interop follow-up. Confirm with Armada before writing the multi-device code.
|
||
2. **NIP-42 for stream-authored REQs.** Relays that gate kind 1059 by author (for example `ditto-relay`'s `AUTH_KINDS`) need an AUTH event signed by that plane's derived key. `nostr-sdk`'s `Authenticator` is per-client and returns one identity, while a Concord client holds many plane keys — so this cannot be solved by swapping the authenticator. Options: contribute a per-REQ auth hook upstream, or accept that such relays are unsupported and prefer relays without the gate. Decide before M7; the default is "documented limitation" plus a relay-capability check.
|
||
3. **`invite_bundle_key`.** Appendix A.6 says the label "yields the public-invite decrypt key" without stating whether that is the raw HKDF output used as a NIP-44 conversation key or the `conv_key` of a normalized keypair. The reference implementation uses the raw output. Pin a vector and verify against Armada early — this one decides whether links open at all.
|
||
4. **Missing golden vector for `pins_locator`.** Upstream publishes none. Ours will be self-referential; flag it in the test.
|
||
5. **Relay set.** Up to 5 recommended, and both reads and writes fan out across them. Coop's client is a gossip client with `no_background_refresh`, so community relays must be added explicitly and re-added on metadata change.
|
||
6. **Local plaintext state.** §9 records the decision. Revisit only if the local database stops being treated as trusted.
|
||
7. **Was a `community_id` ever hashed into a tag?** No — it must never appear on the wire. A lint-style test should assert it appears in no published event other than inside an invite bundle and a direct invite.
|
||
|
||
## 15. Test strategy
|
||
|
||
- **Unit, pure:** derivations against golden vectors, edition hash, fold, coalesce, memberlist, blob codec, caps and rejection paths. These need no GPUI context and should be exhaustive — they are where cross-client divergence is caught.
|
||
- **Integration, GPUI:** `TestAppContext` with two registries sharing an in-memory database, driving wraps through the ingest path; timeouts and delays use `cx.background_executor().timer(..)` per the project guidelines, never `smol::Timer`.
|
||
- **Round-trip:** every builder paired with its parser, asserting the parse produces the identical structure, including unknown-field round-tripping on all content types.
|
||
- **Negative:** every bullet in §12 gets a test that constructs the hostile input and asserts the drop.
|