add basic concord backend

PLAN.md

@@ -46,7 +46,7 @@ Verified against the pinned revision (`nostr` 0.45.4 / `nostr-sdk` 0.45.2, git `
 | NIP-44 under a signer | `nip44::{encrypt, decrypt}` (already wrapped by `state::UniversalSigner`) |
 | Event id recomputation | `EventId::compute(pubkey, created_at, kind, tags, content)`, `UnsignedEvent::compute_id` |
 | Event (de)serialization | `Event::{from_json, as_json, verify}`, `UnsignedEvent::from_json` |
-| Building events | `EventBuilder::new(..).tags(..).custom_created_at(..)`, `FinalizeEvent::finalize(&Keys)` |
+| Building events | `EventBuilder::new(..).tags(..).custom_created_at(..)`, `FinalizeEvent::finalize(&Keys)`; `UnsignedEvent::new(..)` for rumors, whose tags are the author's contract and must not be normalized |
 | Tags | `Tag::{custom, identifier, public_key, expiration}`, `Tags`, `SingleLetterTag` |
 | Kinds | `Kind::GiftWrap` (1059), `Kind::Custom(21059|20013|20014|3308|…)`, `Kind::is_ephemeral` |
 | Publish | `Client::send_event(&event).to(relays).ack_policy(AckPolicy::none())` |
@@ -59,16 +59,16 @@ Verified against the pinned revision (`nostr` 0.45.4 / `nostr-sdk` 0.45.2, git `
 
 **Not needed.** `secp256k1` (use `nostr::SecretKey::from_slice` + `Keys::new`), `base64` (use `data_encoding::BASE64`, already a workspace dep), `bech32` (NIP-19 is in the SDK), any new storage engine (the client's LMDB database is enough), any new HTTP client.
 
-**Add one dependency now:** `hkdf = "0.12"` (already in `Cargo.lock` transitively). Add it to `[workspace.dependencies]` and to the new crate. `sha2` is already a workspace dep. **Two more at M8:** the Pin List's per-message key disclosure needs `chacha20 = "0.9"` (already in the tree because we enable nostr's `nip44`, which is where `chacha20` comes from) and `hmac = "0.12"` (already in the tree via `hkdf`) — see §14.8. Zero new crates, all three are direct-dependency lines only.
+**Dependencies added so far:** `hkdf = "0.12"` at M0 (already in `Cargo.lock` transitively) and `rand = "0.10"` at M1 for the NIP-44 nonce, pinned to the exact instance `nostr` already builds (`default-features = false`, features `std` + `sys_rng`) so `nostr`'s `os-rng` and ours unify on one `rand`/`getrandom`. `sha2` and `data-encoding` were already workspace deps. **Two more at M8:** the Pin List's per-message key disclosure needs `chacha20 = "0.9"` (already in the tree because we enable nostr's `nip44`, which is where `chacha20` comes from) and `hmac = "0.12"` (already in the tree via `hkdf`) — see §14.8. Zero new crates so far, and all of these are direct-dependency lines only.
 
 **Why not depend on Vector's crates.** `vector-core` (published, MIT) holds the only other Rust Concord implementation, in `src/community/v2/*`. It is not reusable as a dependency, and the "reuse their crypto" argument does not hold:
 
 - **No shared types.** It exact-pins the nostr family (`nostr = "=0.45.1"`, `nostr-sdk = "=0.45.1"`, `nostr-connect = "=0.45.1"`, `nostr-blossom = "=0.45.0"`) with the note that a caret range would let a consumer resolve a mixed set, while we track git master (`b230cec`, 0.45.4 / 0.45.2). A registry 0.45.1 and a git 0.45.4 cannot unify, so a build linking both carries two `nostr` crates whose `Event`/`Keys`/`PublicKey`/`Client` are unrelated types.
 - **Not wasm-buildable.** `rusqlite` (bundled C SQLite), `libc`, `rustls`, `reqwest`, `image`, `bip39`, and a `tokio` `net` + `rt-multi-thread` requirement; `VectorCore::init` installs a process-global rustls provider and raises the fd limit. Coop's `web` target is wasm32.
 - **It is an application core, not a Concord library.** 80k+ lines over 111 files, built on process-global singletons (`state::STATE`, `MY_SECRET_KEY`, one app-data dir, one live account, `traits::set_event_emitter`) and its own SQLite schema, relay pool and blocking `listen()` loop. Adopting it means handing it the nsec and letting it own the client, the database and the event loop — replacing `state`, `chat` and `person` rather than reusing a component. Its `login` stores raw secret-key bytes in that global vault, so an account whose key lives in a signer cannot drive it.
-- **There is no cryptography to share.** Both implementations call the same audited crates — `hkdf`, `sha2`, nostr's secp256k1 keypair, and nostr's NIP-44 v2. Vector's comment on that same dependency is "audited RustCrypto crate rather than a hand-rolled construction". Our `derive.rs` has no curve arithmetic, AEAD or randomness of its own: it holds the frozen `info` layout and label table, which are the wire format, not a primitive.
+- **There is no cryptography to share.** Both implementations call the same audited crates — `hkdf`, `sha2`, nostr's secp256k1 keypair, and nostr's NIP-44 v2. Vector's comment on that same dependency is "audited RustCrypto crate rather than a hand-rolled construction". Confirmed in M1 by reading `community/cipher.rs`: it is a ~20-line wrapper that draws an OS nonce, calls `nostr::nip44::v2::encrypt_to_bytes_with_nonce`, and base64s the result — which is precisely what `stream.rs` does. Their `stream.rs` likewise calls `nostr_sdk::prelude::nip44::v2::{decrypt_to_bytes, ConversationKey}` directly. Our `derive.rs` holds the frozen `info` layout and label table, and `stream.rs` the seal/wrap ordering; both are wire format, not primitives.
 
-So Vector's crates earn their place as an **oracle, not a dependency**: the golden vectors in `derive.rs` are their published data, produced by an independent implementation.
+So Vector's crates earn their place as an **oracle, not a dependency**: the golden vectors in `derive.rs` are their published data, and their `community/v2/stream.rs` was diffed against our §7 before the codec was written. It agrees on every wire detail, and contributed the `ms` first-wins rule, the Control Plane's no-`ms` rumor shape and the `rewrap_seal` contract.
 
 ## 4. Crate layout
 
@@ -91,7 +91,9 @@ crates/concord/
 
 `Community` and `Channel` GPUI entities live in `src/lib.rs` next to the registry — they are the public surface, not a separate concern. Ten modules, each with real content; no single-fn files.
 
-Dependencies: `common`, `state`, `person`, `device`, `settings`, `gpui`, `nostr` (for `nip44` features), `nostr-sdk`, `hkdf`, `sha2`, `serde`, `serde_json`, `anyhow`, `flume`, `smallvec`, `itertools`, `futures`, `log`, `instant`. Everything under `cfg(not(target_arch = "wasm32"))` follows the `chat`/`state` split so the crate still builds for web.
+Dependencies: `common`, `state`, `person`, `device`, `settings`, `gpui`, `nostr` (for `nip44` features), `nostr-sdk`, `hkdf`, `sha2`, `data-encoding`, `rand`, `serde`, `serde_json`, `anyhow`, `flume`, `smallvec`, `itertools`, `futures`, `log`, `instant`. Everything under `cfg(not(target_arch = "wasm32"))` follows the `chat`/`state` split so the crate still builds for web.
+
+Declare only what a milestone actually uses. As of M1 the crate depends on `nostr`, `nostr-sdk`, `hkdf`, `sha2`, `data-encoding`, `rand`, `anyhow` (plus `nostr-memory`, `serde_json`, `smol` for tests). `rand` is pinned to the `0.10.2` instance `nostr` already builds and shares its `getrandom`, which the `web` crate already enables `wasm_js` on — so no new package and no new wasm obligation.
 
 ## 5. Core types
 
@@ -185,7 +187,7 @@ Rules that must be enforced by construction, not by convention:
 
 **Golden vectors.** `derive.rs` pins all 18 published vectors (the seed and `pk` for channel, control, control-signer and guestbook; both keyed labels at epoch `0` and at `0x0102030405060708`; both rekey labels at epoch 1; dissolved; all four locators; the invite key; the community id; the epoch-key commitment), cross-checked against an independent Python implementation (RFC 5869 HKDF plus pure-integer secp256k1) before being frozen. One vector is missing upstream — `pins_locator` — so we mint it from our own implementation and pin it, flagged in the test as self-referential. Changing any pinned value means the wire format changed.
 
-## 7. Stream codec (`stream.rs`)
+## 7. Stream codec (`stream.rs`) — implemented in M1
 
 ```rust
 pub const KIND_WRAP: u16 = 1059;
@@ -194,29 +196,56 @@ pub const KIND_SEAL_ENCRYPTED: u16 = 20013;
 pub const KIND_SEAL_PLAINTEXT: u16 = 20014;
 pub const NIP44_MAX_PLAINTEXT: usize = 65_535;
 
+pub enum SealForm { Encrypted, Plaintext }
+
+pub struct OpenedStream {
+    pub rumor_id: EventId,
+    pub author: PublicKey,
+    pub seal_form: SealForm,
+    pub seal: Event,
+    pub wrapper_id: EventId,
+    pub at_ms: u64,
+    pub rumor: UnsignedEvent,
+}
+
+pub fn split_ms(at_ms: u64) -> (u64, u16);
+pub fn resolve_ms_strict(rumor: &UnsignedEvent) -> Result<u64, StreamError>;
+
 pub fn seal_content(rumor: &UnsignedEvent, form: SealForm, group: &GroupKey) -> Result<String, StreamError>;
 pub fn build_seal(rumor: &UnsignedEvent, form: SealForm, group: &GroupKey, author: &Keys) -> Result<Event, StreamError>;
 pub fn wrap_seal(seal: &Event, group: &GroupKey, wrap_kind: u16, at: Timestamp, extra: &[Tag]) -> Result<(Event, Keys), StreamError>;
+pub fn rewrap_seal(seal: &Event, new_group: &GroupKey, at: Timestamp) -> Result<(Event, Keys), StreamError>;
 
 pub fn open_wrap(wrap: &Event, group: &GroupKey) -> Result<OpenedStream, StreamError>;
-pub fn open_wrap_at(wrap: &Event, address: &PublicKey, conversation: &ConversationKey, verify_wrap_sig: bool) -> Result<OpenedStream, StreamError>;
+pub fn open_wrap_at(wrap: &Event, address: &PublicKey, conversation: &ConversationKey, verify_wrap_signature: bool) -> Result<OpenedStream, StreamError>;
+
+pub fn build_rumor_ms(kind: u16, author: PublicKey, content: &str, tags: Vec<Tag>, at_ms: u64) -> UnsignedEvent;
+pub fn build_rumor_secs(kind: u16, author: PublicKey, content: &str, tags: Vec<Tag>, at_secs: u64) -> UnsignedEvent;
 
 pub fn channel_binding_tags(channel: &ChannelId, epoch: Epoch) -> Vec<Tag>;
 pub fn check_channel_binding(rumor: &UnsignedEvent, channel: &ChannelId, epoch: Epoch) -> Result<(), StreamError>;
-pub fn build_rumor(kind: u16, author: PublicKey, content: &str, tags: Vec<Tag>, at_ms: u64) -> UnsignedEvent;  // appends ["ms", n]
-pub fn resolve_ms_strict(rumor: &UnsignedEvent) -> Result<u64, StreamError>;
 ```
 
+`StreamError` is a typed enum, not `anyhow`: the caller has to tell a drop from a fatal, and M1's acceptance criterion is that rejections happen in the documented order.
+
+Refinements against the draft this plan opened with, decided after diffing Vector's `crates/vector-core/src/community/v2/stream.rs` (Concord has no crate of its own there, and `envelope.rs` does not exist):
+
+- `build_rumor` became `build_rumor_ms` plus `build_rumor_secs`. The Control Plane edition carries **no** `ms` tag, because editions fold by version, not by time.
+- `rewrap_seal` was added to the codec. Without it the plaintext-seal carry-forward has no expression, and M7's compaction is its only caller.
+- NIP-44 is reached through `nostr`'s own `nip44::v2::{encrypt_to_bytes_with_nonce, decrypt_to_bytes, ConversationKey}`, with a fresh OS nonce per message and `data_encoding::BASE64` for carriage. This is exactly what Vector does; there is no cryptography of theirs to reuse.
+
 Design points that are easy to get wrong:
 
 - The wrap is signed by the **stream key** with a random ephemeral `p` tag — NIP-59 reversed. `extra` is how the caller mirrors a NIP-40 expiration onto the wrap.
 - The seal is signed by the **real author** and carries `created_at` equal to the rumor's. It is never published bare.
 - Control plane **must** use the plaintext seal; chat, guestbook and rekey planes **must** use the encrypted one. Each plane asserts its own form at both ends.
 - The control plane is a write-restricted stream: the wrap key derives from `control_root` while the content is encrypted under the `community_root`-derived conversation key. `open_wrap_at` takes the two halves separately for this reason.
-- Open order: kind → address match → wrap signature (only when `verify_wrap_sig`) → NIP-44 open → seal kind → seal signature → rumor parse → `rumor.pubkey == seal.pubkey` → recompute the rumor id and reject a mismatch → strict `ms` resolve.
+- Open order: kind → address match → wrap signature (only when `verify_wrap_signature`) → NIP-44 open → seal kind → seal signature → rumor parse → `rumor.pubkey == seal.pubkey` → recompute the rumor id and reject a mismatch → strict `ms` resolve.
 - Enforce the 65 535-byte NIP-44 cap at every nesting layer before publishing.
 - Do not verify the wrap signature for ordinary planes: every reader holds the group key, so the signature proves nothing. It matters only for the restricted control plane and for rekeys.
 - The ephemeral wrap keypair is returned to the caller so a client may NIP-09-scrub its own wrap later.
+- **A duplicate `ms` tag takes the first value; it is not rejected.** Rejecting made Vector and Armada disagree on whether the event *exists*, and because `ms` orders messages that divergence reached membership. `ms` is the publisher's own value, so conceding a second tag grants an attacker no reach a single one did not. A present-but-valueless `ms`, or one that is not a lone canonical decimal in `0..=999`, is `BadMs` and the event is dropped, never clamped — `u64::from_str` alone would accept a leading `+`, a second encoding a strict peer rejects, so the digit check comes first.
+- A binding tag that names the same key twice is rejected outright, since first-match would then be the reader's choice rather than the author's; a valueless tag counts as absent, so a true absence reports `MissingTag`.
 
 ## 8. Planes, state and folds
 
@@ -373,14 +402,26 @@ pub fn compact(fold, epoch, new_control_root, ...) -> Vec<Event>;   // re-wrap h
 
 Dissolution (CORD-02 §9) also lives here: a chainless, owner-signed `vsk 10` tombstone at `dissolved_group_key(id)`, plaintext-sealed, and a verifier **must** refuse any tombstone whose `eid` is not the community's own id (including the all-zero placeholder — accepting it lets an owner's genuine tombstone for one community be re-wrapped at another of theirs and kill it permanently). On sight the community is sealed read-only: subscriptions halt, nothing new is honored, existing history stays readable, and a member's delete of their own message is still honored.
 
-## 9. Storage (`store.rs`)
+## 9. Storage (`store.rs`) — local layer implemented in M1
 
 Three layers, no new storage engine:
 
 1. **Raw wraps** (kind 1059) are persisted automatically by the SDK's relay pool when a subscription or fetch matches a filter. Nothing to write.
 2. **Opened rumors** are cached locally as NIP-78 `Kind::ApplicationSpecificData` events signed by a session-local keypair, exactly like `chat::set_rumor`. Tags: `["d", rumor_id]` (replace key), `["c", channel_hex]`, `["p", author]`, `["k", kind]`, `["e", wrap_id]`, `["t", "concord"]`. Contents are the rumor JSON.
    - The `c`/`t` keys deliberately differ from chat's `r` key so the two message namespaces can never collide in one database.
-   - The read path dedupes by rumor id and keeps the newest `created_at`, because the local signing key changes per session.
+   - `created_at` is the **message's own second** (from `at_ms`), not the wall clock. Otherwise `until` and the ordering would page on cache time rather than message time.
+   - The read path dedupes by rumor id and keeps the newest `created_at`, because the local signing key changes per session and each session leaves its own copy. The query therefore carries no filter `limit` — every copy has to be in hand before they can be collapsed — and the cap is applied to the deduplicated result instead.
+   - The layer takes `&dyn NostrDatabase`, not `&Client`: it is local-only, which keeps it testable without a relay or a GPUI context.
+
+```rust
+pub async fn cache_rumor(database: &dyn NostrDatabase, channel: &ChannelId, opened: &OpenedStream) -> Result<()>;
+pub async fn query_rumors(database: &dyn NostrDatabase, channel: &ChannelId, until: Option<Timestamp>, limit: usize) -> Result<Vec<UnsignedEvent>>;
+```
+
+`query_rumors` returns `UnsignedEvent`, not `Event`: the cached payload *is* a rumor, which is also what `OpenedStream` carries, so the caller never has to re-parse.
+
+**Deferred to M4:** the relay-paging `backfill`. It is network history paging whose "step past the same-second wall" policy belongs with the sync engine, and M1 has no subscription to test it against.
+
 3. **Community state** — one local document per community, `Kind::ApplicationSpecificData` with `["d", "concord/<community_id>"]`:
 
 ```rust
@@ -404,19 +445,10 @@ pub struct CommunityState {
 }
 ```
 
-Writes are debounced (a fold head changes on every edition); reads load once at init.
+Writes are debounced (a fold head changes on every edition); reads load once at init. This layer lands in M2, once `Community` exists to hold it.
 
 **Decision, stated for the record:** this document stores raw community keys unencrypted in a plaintext local database. That matches the existing posture — `chat` already caches decrypted message rumors in the same LMDB. If that posture ever changes, the state document is the one to wrap with NIP-44-to-self, since it is the only local artifact holding keys.
 
-History queries:
-
-```rust
-pub async fn query_messages(&self, channel: &ChannelId, until: Option<Timestamp>, limit: usize) -> Result<Vec<Event>, Error>;
-pub async fn backfill(&self, plane_authors: &[PublicKey], relays: &[RelayUrl], until: Option<Timestamp>, limit: usize) -> Result<Vec<Event>, Error>;
-```
-
-`query_messages` reads the local cache (`Filter::new().kind(ApplicationSpecificData).custom_tag(LOWERCASE_C, channel_hex)`); `backfill` pages relays newest-first with `until`, deduplicating by wrap id and stepping past same-second walls.
-
 ## 10. Sync engine and GPUI conventions
 
 `ConcordRegistry` mirrors `ChatRegistry`'s shape exactly: a foreground GPUI entity holding `Entity<Community>` handles, a `flume` signal bus, one background notification listener, one foreground consumer, and task slots that are cleared when the signer changes.
@@ -537,7 +569,7 @@ pub fn pin(&self, id: EventId, cx: &App) -> Task<Result<(), Error>>; // vsk 11,
 
 ## 11. Integration with existing crates
 
-1. **`crates/chat/src/lib.rs` — required fix.** `handle_notifications` currently treats *every* kind 1059 event as a NIP-17 wrap for the current user and pushes an unwrap failure into the trash. Concord wraps are kind 1059 with an ephemeral `p` tag, so they would flood the trash and leak error toasts. Route by `subscription_id` from `RelayMessage::Event` against `sub_id1`/`sub_id2`, and drop the `if rumor.tags.is_empty()` heuristic once the real recipient check is in place.
+1. **`crates/chat/src/lib.rs` — required fix, applied in M2.** `handle_notifications` currently treats *every* kind 1059 event as a NIP-59 gift wrap for the current user and pushes an unwrap failure into the trash. Concord wraps are kind 1059 with an ephemeral `p` tag, so they would flood the trash and leak error toasts. Route by `subscription_id` from `RelayMessage::Event` against `sub_id1`/`sub_id2`, and drop the `if rumor.tags.is_empty()` recipient heuristic. M1 did not need it because nothing subscribes yet.
 2. **`desktop/src/main.rs` and `web/src/lib.rs`** — add `concord::init(window, cx)` after `chat::init(window, cx)`.
 3. **`Cargo.toml`** — add `hkdf = "0.12"` to `[workspace.dependencies]`; add the crate to `desktop` and `web` dependencies. No other workspace changes.
 4. **No changes** to `state`, `person`, `device`, `settings`, `common`, or `ui`.
@@ -550,7 +582,7 @@ Each of these has burned a real implementation, or is a documented cross-client
 - Require `rumor.pubkey == seal.pubkey`.
 - Require the plaintext seal form on Control and the encrypted form on Chat/Guestbook/Rekey — a strict reader must drop a mis-sealed edition rather than fold a chain a later compaction would fork.
 - Check `channel` **and** `epoch` against the plane whose key opened the wrap; reject duplicates of either tag.
-- Reject duplicate `vsk`/`eid`/`ev`/`ep`/`vac` tags; require decimal-with-no-leading-zeros on every numeric tag.
+- Reject duplicate `vsk`/`eid`/`ev`/`ep`/`vac` tags; require decimal-with-no-leading-zeros on every numeric tag. The one exception is `ms`, which takes its first value rather than erroring — see §7 for why rejecting it reached membership.
 - Refuse a tombstone whose `eid` is not this community's id.
 - Adopt a `control_root` from a Grant only if it derives to the `control_pk` held for that epoch; adopt a rekey blob only if its bound plaintext matches the scope and epoch and its `prevcommit` matches the key currently held.
 - Never conclude removal from a partial rekey chunk set.
@@ -564,11 +596,11 @@ Each of these has burned a real implementation, or is a documented cross-client
 
 | # | Deliverable | Done when |
 | --- | --- | --- |
-| M0 | Crate skeleton, `derive.rs`, golden vectors, workspace wiring | `cargo test -p concord` pins every derivation from an independent vector set; all labels match Appendix A.6 |
-| M1 | `stream.rs` + `store.rs` | seal/wrap/open round-trips for both seal forms; malformed inputs rejected in the documented order; local cache reads back after a restart |
+| M0 | Crate skeleton, `derive.rs`, golden vectors, workspace wiring | ✅ `cargo test -p concord` pins every derivation; all labels match Appendix A.6 |
+| M1 | `stream.rs` + `store.rs` | ✅ seal/wrap/open round-trips for both seal forms; hostile wraps rejected in the documented order; the local cache reads back with the group key gone |
 | M2 | `edition.rs` + `control.rs` genesis | a community is created and published; its two genesis wraps open at a second client sharing the keys; edition hash matches the cross-client vector |
 | M3 | Control fold + roster + metadata/channels | fold tests for chains, gaps, downgrade refusal, fork tiebreak, compaction dangle; metadata and channel edits visible to a second client |
-| M4 | Chat plane | send/receive/edit/delete/reaction across two identities; history pages backwards across an epoch boundary; binding checks reject a foreign channel/epoch |
+| M4 | Chat plane | send/receive/edit/delete/reaction across two identities; history pages backwards across an epoch boundary (relay `backfill` lands here); binding checks reject a foreign channel/epoch |
 | M5 | Guestbook + member list + moderation | join/leave/kick/ban converge to the same memberlist on both clients; every authority rule from §8.1 has a negative test |
 | M6 | Invites + Community List | link mint → fetch → join round-trips; revoked link refuses; direct invite lands in the recipient's giftwrap inbox via the `k` tag; a second device reconstructs membership from 13302 |
 | M7 | Rekeys + refounding + dissolution | a removed member stops reading after a rekey; continuity and race rules tested; a tombstone seals the community and a foreign-id tombstone is refused |
@@ -576,6 +608,8 @@ Each of these has burned a real implementation, or is a documented cross-client
 
 Ordering is deliberately dependency-first: each milestone is usable on its own, and nothing in M2+ depends on a later milestone.
 
+M1 closed with `cargo test -p concord` (5 tests), `cargo clippy -p concord --all-targets`, and `cargo fmt -p concord --check` all clean. `rand` was added to the workspace pinned to the same `0.10.2` instance `nostr` already builds, so `Cargo.lock` gained no package.
+
 ## 14. Open questions and risks
 
 1. **Community List kind.** CORD-02 §8 specifies `13302`, replaceable. Vector has retired it in favour of fragmented `33302`, because a replaceable kind holds one event per pubkey and so cannot shard past the NIP-44 size cap. We implement `13302` per spec, enforce the 50-membership cap and pre-publish size check, and treat `33302` as an interop follow-up. Confirm with Armada before writing the multi-device code.