reya/coop
1
0
Fork 0
Code Issues Pull Requests 1 Releases 3 Activity

updat

Browse Source
This commit is contained in:
reya
2026-09-22 16:38:08 +07:00
parent 6fac58d494
commit 1c5ef58049
11 changed files with 808 additions and 181 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/community-history-sync-plan.md
+162 -24
View File
@@ -35,9 +35,20 @@ subscriptions too, which is what a page REQ is (`nostr-sdk/src/relay/inner.rs`
`RelayNotification::Authenticated` arm). Only `AuthenticationFailed` ends a relay's
wait, and it is reported rather than rendered as an empty channel.
**Revision 3.** One more correction, from a review of what revision 2 landed:
4. **Time is a type, not a `u64` with the unit in its name.** The read path held
three different units in the same `u64`: `HeldKey`/`HeldRoot.retired_at` held
seconds (but were compared with a `Timestamp` through `.as_secs()`),
`ChannelCursor` held milliseconds that were only ever seconds multiplied by a
thousand, and `CommunityState.channel_cuts` held epochs. Every filter boundary
divided by a thousand on the way out. §10 types the time fields (`Timestamp`,
`Epoch`) and leaves a millisecond only where a millisecond is real.
Read path today (phase 1 landed the walk, phase 2a moved each layer, phase 2b
swapped the transport and put the pump in charge of settling pages, phase 3 added
the rekey watch and held epochs, phase 4 made an empty or unreadable room say so):
the rekey watch and held epochs, phase 4 made an empty or unreadable room say so,
phase 5 typed the times they all compare):
```
CommunityPanel::load community_ui/src/lib.rs:192
@@ -277,10 +288,10 @@ impl Window {
/// Nothing held: ask wide, the round pages down from there.
/// Something held: ask only for what is new, plus an overlap for the seam.
pub fn opening(cursor: ChannelCursor) -> Self {
match cursor.newest_ms {
Some(newest_ms) => Window {
since_ms: Some(newest_ms.saturating_sub(CURSOR_OVERLAP_MS)),
until_ms: None,
match cursor.newest {
Some(newest) => Window {
since: Some(newest - CURSOR_OVERLAP),
until: None,
},
None => Window::default(),
}
@@ -296,13 +307,13 @@ pub fn live_window(state: &CommunityState) -> Window {
let floor = state
.channels
.iter()
.filter_map(|channel| state.cursors.get(&channel.id)?.newest_ms)
.filter_map(|channel| state.cursors.get(&channel.id)?.newest)
.min();
match floor {
Some(floor) => Window {
since_ms: Some(floor.saturating_sub(CURSOR_OVERLAP_MS)),
until_ms: None,
since: Some(floor - CURSOR_OVERLAP),
until: None,
},
None => Window::default(),
}
@@ -352,7 +363,7 @@ relay.
Rules carried over from phase 1, none of which are negotiable:
- **A page boundary is exclusive.** `until = oldest_seen_ms - 1`, so consecutive
- **A page boundary is exclusive.** `until = oldest_seen - 1`, so consecutive
pages never share an event and the walk terminates.
- **`exhausted` is earned.** Only `raw > 0` plus a bottomed-out page on every
relay that answered may set it. An all-empty page sets `failed`, so the next
@@ -366,8 +377,8 @@ Rules carried over from phase 1, none of which are negotiable:
as refused, and it is surfaced.
- **One cursor, one filter per page.** The database is not per relay, so per-relay
cursors do not exist; a relay is a source that fills the database.
- **`newest_ms` advances only on a complete round** (`!newest.failed && bridge.exhausted`),
`oldest_ms` only walks down on a complete page, and `exhausted` is cleared by a
- **`newest` advances only on a complete round** (`!newest.failed && bridge.exhausted`),
`oldest` only walks down on a complete page, and `exhausted` is cleared by a
rekey.
The cold/warm distinction is now the *window rule* rather than a special case:
@@ -375,9 +386,9 @@ The cold/warm distinction is now the *window rule* rather than a special case:
| State of the channel | Window |
| -------------------- | ------ |
| no cursor, nothing cached (cold open) | `Window::opening` → `since = None`: subscribe for everything the relays have, then page down |
| cursor with `newest_ms` (warm open) | `Window::opening` → `since = newest - CURSOR_OVERLAP`: new data only |
| cursor with `oldest_ms` | `Window::older_than(oldest_ms)`: older data, on demand |
| a hole between `saved.newest_ms` and the newest seen | `Window::between(..)`: the bridge |
| cursor with `newest` (warm open) | `Window::opening` → `since = newest - CURSOR_OVERLAP`: new data only |
| cursor with `oldest` | `Window::older_than(oldest)`: older data, on demand |
| a hole between `saved.newest` and the newest seen | `Window::between(..)`: the bridge |
So an open does two things, and they follow the same rule: it makes sure the live
REQ is installed — wide when nothing is held, which is where "subscribe to get all
@@ -390,9 +401,9 @@ And the passes fall out of that rule:
- **Newest pass** (`CatchUp`): one page at `Window::opening(saved)`. Cold, this is
"get all data"; warm, this is "get only new data".
- **Bridge**: while the previous page was *full* and still above
`saved.newest_ms`, keep walking down with `until = oldest - 1`, bounded by
`CATCH_UP_PAGES`. A short page ends it, which is what lets `newest_ms` advance.
- **Older pass**: resume at `saved.oldest_ms.or(oldest_seen)` and walk down,
`saved.newest`, keep walking down with `until = oldest - 1`, bounded by
`CATCH_UP_PAGES`. A short page ends it, which is what lets `newest` advance.
- **Older pass**: resume at `saved.oldest.or(oldest_seen)` and walk down,
bounded by `CATCH_UP_PAGES` on a catch-up and `LOAD_OLDER_PAGES` on a scroll-up.
Sizes stay the reference numbers: `PAGE_WRAPS = 50`, `CATCH_UP_PAGES = 20`,
@@ -452,7 +463,7 @@ the next ones:
- `ChannelKeyRef` gains
`#[serde(default, skip_serializing_if = "Vec::is_empty")] priors: Vec<HeldKey>`
where `HeldKey { epoch: Epoch, key: [u8; 32], retired_at: Option<u64> }`, and
where `HeldKey { epoch: Epoch, key: [u8; 32], retired_at: Option<Timestamp> }`, and
`CommunityState` gains the same shape for roots (`HeldRoot { epoch, key,
`control_pk, retired_at }`). `channel_secret` returns every held epoch,
`history::page` already takes `&[(Epoch, [u8; 32])]`, and `sync::refresh` stops
@@ -521,9 +532,49 @@ Three deviations from the sketch above, all of them smaller than planned:
adoptable, so a member who missed several rotations catches up in one database
read instead of one pass per poll. The lookahead window is what feeds it.
Still deferred, and named here so it is not mistaken for landed: a rotation this
client published itself is not adopted locally (no rekey writer exists yet, so the
watch only ever adopts another member's).
#### The writer (phase 6) — **landed**
Phase 3 gave the client a receiver; phase 6 gives it a hand. `rekey::rotate`
builds the rotation from the key the client actually holds and publishes it, and
`Community::rotate` is the entry a moderation surface will call:
- **One plan, drawn in the protocol crate.** `cord06::plan_rotation(scope, epoch)`
mints what the rotation delivers — a `Refounding` (new root and control pair)
for a base scope, a fresh key for a channel — beside `plan_refounding`, which is
now the base arm of it. No caller ever holds a rotation secret before the
rotation exists.
- **Authority first, and the same authority the receiver applies.** `Rewrite::authorized`
runs `cord06::rekey_authorized` under `permissions(scope)` — the same list
`adopt` now walks with, shared instead of written twice — so the client cannot
publish a rotation it would not itself adopt. `Community::rotate` adds the three
states the role fold cannot see: a removal, a strand and a ban.
- **One blob per recipient, and the rotator is one of them.** A rotation that
delivered no blob to its own rotator would strand them, so the writer refuses
one. Staff get the new Control Plane root beside the key, everyone else gets the
key alone.
- **A refounding carries its heads.** `carry_heads` reads the control editions
back out of the store, picks the ones the settled floors name, and `compact`s
them onto the new epoch's groups. Without it the new Control Plane starts empty
and the next reader that does not hold the old root folds no roles, metadata or
banlist at all. (`compact` re-signs nothing, so this only works on the plaintext
seals the Control Plane already uses.)
- **Then it adopts itself, through the ordinary receiver.** The chunks are saved
into the local database as they are sent, so `Community::rotate` calls the same
`rekey::adopt` the watch calls instead of adopting by construction: one path for
every rotation, whoever wrote it. Publish failures are logged per relay and do
not change the held state, so a rotation that only reached some relays still
leaves this client consistent — which is the same asymmetry the reference has.
- **The receiver now keeps the signing root it is handed.** `BaseAdoption`
carries the delivered `control_root` through to `state.control_root`; before
this, a refounding's root was dropped at the receiver and a staff member went on
signing under the epoch they had left.
What is still not here: the trigger. Kicking a member, choosing a channel's
remaining audience, and rendering any of it is the moderation and
community-management surface §8 keeps out of scope, so `Community::rotate` is an
API with no caller in the app yet — and `Rewrite.recipients` is the caller's to
name, because a private channel's audience is in each member's own invite and not
in the local state.
### 7. Honest states (phase 4) — **landed**
@@ -659,6 +710,50 @@ NIP-44-open them all. The plan's per-channel work queue in the pump turned out t
be unnecessary for that: the first fold after a live wrap already opens exactly
the uncached wraps.
### 10. Time is a type: `Timestamp`, not padded milliseconds (phase 5) — **landed**
The read path compared times through a bare `u64` in three units, and each
boundary paid for the confusion:
| Value | Held | What it cost |
| ----- | ---- | ------------ |
| `HeldKey.retired_at`, `HeldRoot.retired_at`, `Plane.retired_at` | seconds | compared with `wrap.created_at.as_secs()`, written as `at_ms / 1000` |
| `ChannelCursor.newest_ms`/`oldest_ms`, `WrapPage`, `Window`, the `Walk` bounds | milliseconds, always `secs * 1000` | `* 1000` on every accepted page, `/ 1000` on every filter |
| `CommunityState.channel_cuts` | epochs | `Epoch` unpacked to `u64` and rewrapped at the boundary |
| `added_at_ms`, a rumor's `at_ms` | milliseconds, genuinely | — |
What landed:
- `HeldKey`/`HeldRoot.retired_at`, `ChannelCursor.{newest,oldest}`,
`WrapPage.{newest,oldest}`, `Plane.retired_at`, `Window.{until,since}` and the
`Walk` bounds are `Timestamp`s. A window bound and a wrap's `created_at` are now
the same type, so `read_under` and `Plane::accepts` compare them directly,
`wrap_filter`/`live_filter` hand them to `until`/`since` unchanged, and the
page's exclusive boundary is written `oldest - 1` instead of `oldest_ms - 1`
truncated back to seconds.
- `CURSOR_OVERLAP` is a `Duration` (`60s`), not `60_000`, so
`newest - CURSOR_OVERLAP` reads as what it is.
- `channel_cuts` is `BTreeMap<ChannelId, Epoch>`, which is what
`rekey::Adoptions::cuts` already carried.
- **The two millisecond fields stay milliseconds.** A rumor's time is genuinely
finer than a wrap's second-granular `created_at`: CORD-01 carries the sub-second
offset in an `ms` tag and `resolve_ms_strict` puts it back, so a reader's
`at_ms` and `timeline`'s `before_ms` keep it. The cord02 list document's
`added_at`/`removed_at` keep it too, and not merely for fidelity: `added_at` is
an ordering key judged against a tombstone's `removed_at` under a strict `>`,
so narrowing both to seconds could tie a rejoin with the leave that preceded it.
- **A stored cursor is retired, not reinterpreted.** A document written when the
boundaries were milliseconds would deserialize into a `Timestamp` far in the
future, and its `exhausted` would then wedge the older pass for good. The map's
serde key changed (`cursors` → `channel_cursors`), so such a record is dropped
and the next round rebuilds the cursor from the wraps it reads;
`a_cursor_stored_in_the_old_unit_is_dropped_rather_than_reinterpreted` pins
both halves of that. `retired_at` needed no such treatment, because it was
already written in seconds.
No behaviour changes with it: the walk paged the same regions before and after,
because every millisecond it held was a second multiplied by a thousand.
## Order of work
### Phase 2a — move the code (no behaviour change) — **landed**
@@ -777,13 +872,42 @@ Five recorded deviations:
Phase 4 leaves the client honest about what it can see and what it can write.
What it does not do is make it see more: the rekey writer, and with it adopting a
rotation this client published itself, remains open (see §6).
rotation this client published itself, was still open there (§6, closed in
phase 6).
Each phase leaves the client consistent on its own. Phase 2a was invisible; phase
2b is what makes "open a community" a subscription and a database read; phase 3
is what keeps a rekey from stranding history; phase 4 is what makes an empty or
unreadable room tell the truth.
### Phase 5 — typed time — **landed**
§10. `HeldKey`/`HeldRoot.retired_at`, `ChannelCursor`, `WrapPage`, `Plane`,
`Window` and the `Walk` bounds are `Timestamp`s; `channel_cuts` is `Epoch`;
`CURSOR_OVERLAP` is a `Duration`; and the cursor map's serde key was retired so no
stored millisecond cursor is read as seconds. No behaviour change, no relay
contact, no new dependency — the units the read path already compared by hand are
now the types it compares.
Gate, all green: `cargo test -p concord -p community` (51 + 32),
`cargo clippy -p concord -p community -p community_ui --all-targets`,
`cargo +nightly fmt -p concord -p community -p community_ui --check`,
`cargo check -p workspace --all-targets`.
### Phase 6 — the rekey writer — **landed**
§6's writer: `cord06::plan_rotation`, `rekey::{Rewrite, rotate}`, `carry_heads`,
and `Community::rotate`, which publishes a rotation and then adopts it through the
receiver phase 3 built. The receiver also keeps the signing root a refounding
hands it now, instead of dropping it.
Gate, all green: `cargo test -p concord -p community` (51 + 32),
`cargo clippy -p concord -p community -p community_ui --all-targets`,
`cargo +nightly fmt -p concord -p community -p community_ui --check`,
`cargo check -p workspace --all-targets`. No test was added for the writer itself:
the publish half needs a relay, and the same reason the reference's own rekey
paths are exercised by hand applies here — see `Tests` below.
## Phase 1 status (landed)
What phase 1 delivered, and what phase 2 replaces:
@@ -832,6 +956,10 @@ outstanding are the ones that need a GPUI harness or two live accounts.
`Window::opening(saved)` starts at `newest - CURSOR_OVERLAP`; `Window::older_than`
never includes the boundary event; `sync::live_window` is wide cold and resumes
at the oldest held cursor warm.
- The cursor's unit — **landed in phase 5**: a merge only moves forward and never
earns `exhausted`; a document whose cursors were stored in the old millisecond
encoding reads as no cursors at all, while a typed one round-trips
(`concord/src/state.rs`).
- The subscription plan — **landed in 2b**: a private channel's plane appears when
the key is held and is absent when it is not; `plane_filter` asks for both wrap
kinds and addresses every readable plane. The page-REQ half (a warm open's REQ
@@ -845,8 +973,15 @@ outstanding are the ones that need a GPUI harness or two live accounts.
rotator who outranks us, published after we joined, reads as a removal; a
channel rotation replaces the key and keeps the prior. The pump's half
(`community/src/lib.rs`) is that a rekey watch's event wakes its community.
- The page registry — **landed in 2b**: a page that has already unregistered
receives nothing.
- The rekey writer — **landed in phase 6, structurally only.** No test covers it:
the half that could be tested without a relay (that `plan_rotation` mints two
unrelated keys, that `Rewrite::authorized` agrees with the receiver) would not
catch the failures that matter (a blob addressed to the wrong epoch, a chunk set
the receiver cannot collect, a refounding that carries no head), and the half
that would — publish, re-read, adopt — needs a real relay and a second account.
What stands in for it is that the writer cannot take a path the receiver does
not: same `permissions`, same `build_rekey_chunks`, and adoption goes through
`rekey::adopt` rather than beside it.
- The read path: the side-event budget folds an edit/delete/reaction older than
the row window onto its message.
- What cannot be read — **landed in phase 4**: a wrap sealed after the rotation
@@ -885,6 +1020,9 @@ outstanding are the ones that need a GPUI harness or two live accounts.
Unread badges, notifications, message threads, pins, typing indicators and
presence (21059 wraps are wired for routing here, not for those features), file
and media rendering, moderation actions, and the community-management surfaces.
The rekey writer is the one exception, and only its mechanism: `Community::rotate`
exists to be called, but nothing in the app calls it. Kicking a member, choosing
the audience a rotation keeps, and showing any of it stay out.
`crates/chat`'s DM path shares none of this code and is not touched; a later
change can lift the page walk/cursor into a shared module if DMs grow the same
paging.
docs/concord-usage.md
+48 -26
View File
@@ -245,7 +245,7 @@ let page = history::page(
let cached = cache::query_rumors(&client, &channel, None, 50, Some(&cord03::ROW_KINDS)).await?;
```
A key's `retired_at` (epoch seconds, set when a rotation supersedes it) is a read
A key's `retired_at` (a `Timestamp`, set when a rotation supersedes it) is a read
cutoff: a wrap at that epoch with a later `created_at` is refused, so a retired
epoch is history and never a live plane an ejected holder can keep writing into.
@@ -271,21 +271,24 @@ the handshake completes, so the page waits for the resubscribed answer rather
than writing off a relay that only wanted to authenticate.
`history::page` walks newest-first across every held epoch, caches what it opens, and
reports what it saw: `oldest_ms`/`newest_ms` feed the caller's `ChannelCursor`,
reports what it saw: `oldest`/`newest` feed the caller's `ChannelCursor`,
`exhausted` is earned only by a short page *after* history was seen, and an
all-empty answer sets `failed` so a later round re-asks instead of sealing the
channel at "no more history". `unreadable` counts the wraps the page reached that
no held key could open — sealed past the cutoff their key's rotation set, or bound
to another channel — because those are history the reader is missing, not history
that is not there. Page down with `Window::older_than(seen.oldest_ms)`,
open a channel with `Window::opening(cursor)` (wide cold, `newest_ms - 60s` warm),
and read the region between two cursors with `Window::between(..)`. `query_rumors`
is the read path when the group keys are gone; pass `kinds` to budget rows apart
from the events that only decorate them. `cache::wrapper_index` reads the cached
rows back keyed by the wrap they came from, which is what lets `sync::fold` observe
author and message times without re-opening a wrap it already cached, and
`cache::purge_expired(client, &channel, now)` runs at the top of every round — the
timer is cooperative, so the local store is the artifact that has to forget.
that is not there. Page down with `Window::older_than(seen.oldest)`,
open a channel with `Window::opening(cursor)` (wide cold, `newest - 60s` warm),
and read the region between two cursors with `Window::between(..)`. A `Window`
carries `Timestamp`s, so its bounds go straight into a NIP-01 filter with no unit
conversion; only a reader's millisecond `at_ms` narrows to a second at the query.
`query_rumors` is the read path when the group keys are gone; pass `kinds` to
budget rows apart from the events that only decorate them. `cache::wrapper_index`
reads the cached rows back keyed by the wrap they came from, which is what lets
`sync::fold` observe author and message times without re-opening a wrap it already
cached, and `cache::purge_expired(client, &channel, now)` runs at the top of every
round — the timer is cooperative, so the local store is the artifact that has to
forget.
`sync::fold` counts the same thing over the whole store, per channel, in
`Snapshot.unreadable`. `sync_round` sums a round's pages into
@@ -415,25 +418,25 @@ whether a link still stands, and `fits()` is the write gate.
## Rekeys, refounding and dissolution
A rotation is authority plus delivery: `rekey_authorized(&control.roles, &owner, &me, permission, &removed)`
gates it, `plan_refounding(epoch)` mints the new pair, and `build_rekey_chunks`
seals one blob per remaining member:
gates it, `plan_rotation(scope, epoch)` mints what it delivers, and
`build_rekey_chunks` seals one blob per remaining member:
```rust
use concord::derive::epoch_key_commitment;
use concord::cord06::{self, RekeyScope};
use concord::cord06::{self, RekeyScope, RotationPlan};
let scope = RekeyScope::Channel(channel_id); // or RekeyScope::Base
let plan = cord06::plan_refounding(Epoch(epoch + 1))?;
let plan = cord06::plan_rotation(scope, Epoch(epoch + 1))?;
let new_key = plan.new_key();
// A base rotation delivers the new control-plane keys beside the root; a channel
// rotation delivers only that channel's fresh key.
let new_key = plan.new_root;
let (control_pk, control_root) = match scope {
RekeyScope::Base => {
let pk = plan.signer(&community_id)?.pk().to_bytes();
(Some(pk), is_staff.then_some(&plan.new_control_root))
let (control_pk, control_root) = match &plan {
RotationPlan::Base(refounding) => {
let pk = refounding.signer(&community_id)?.pk().to_bytes();
(Some(pk), is_staff.then_some(&refounding.new_control_root))
}
RekeyScope::Channel(_) => (None, None),
RotationPlan::Channel { .. } => (None, None),
};
let mut blobs = Vec::with_capacity(members.len());
@@ -441,18 +444,24 @@ let mut blobs = Vec::with_capacity(members.len());
for member in &members {
blobs.push(
cord06::build_blob(
&my_keys, member, scope, plan.epoch, &new_key, control_pk.as_ref(), control_root,
&my_keys,
member,
scope,
plan.epoch(),
&new_key,
control_pk.as_ref(),
control_root,
)
.await?,
);
}
let rekey_group = cord06::rekey_group(scope, &community_root, &community_id, plan.epoch)?;
let rekey_group = cord06::rekey_group(scope, &community_root, &community_id, plan.epoch())?;
let wraps = cord06::build_rekey_chunks(
&my_keys,
&rekey_group,
scope,
plan.epoch,
plan.epoch(),
Epoch(epoch),
&epoch_key_commitment(Epoch(epoch), &community_root),
&blobs,
@@ -476,6 +485,18 @@ waiving a gap, never adopting a fork), keeps each stepped-off key as a prior
with the rotation's publish time as its read cutoff, and reports a removal or a
strand when a complete rotation carries no blob for the member.
`community::rekey::rotate` is the sender, run against the held state: it resolves
the epoch and key the scope is stepping off, refuses a rotation that would skip or
cut off its own rotator, delivers one blob per recipient (`Rewrite.recipients`,
which the caller names because a private channel's audience is not in the local
state), marks the rotation severed when it excludes somebody, and — for a base
scope — carries the settled Control Plane heads onto the new epoch's groups with
`cord06::compact`. `Community::rotate` checks `Rewrite::authorized` first, the same
`rekey_authorized` under the same permissions `adopt` applies, publishes to the
community's relays, and then adopts what it published through `rekey::adopt`
rather than by construction: a rotation this client wrote and one it received take
one path.
The blob plaintext is a fixed-width binary record, but a signer's NIP-44 is
text-only, so `build_blob` carries it base64-encoded inside the envelope.
`open_blob` mirrors that, so the record layout and the `locator` are unchanged.
@@ -737,5 +758,6 @@ client.subscribe(filter).with_id(sub_id).await?;
rendered as a notice **and** enforced at write time: `Community::send` refuses
when `channel_secret` is `None` (a removal, a strand, a channel cut, or a key we
never held), because a wrap sealed under a superseded root would reach nobody who
rotated. The residual gap: this client does not adopt a rotation it published
itself, because there is no rekey writer yet.
rotated. A rotation this client publishes itself (`Community::rotate`) is adopted
through the same `rekey::adopt` an arriving one goes through, so the held state
and the wire cannot diverge.