# Plan: bounded slow-consumer handling for the TCP publisher

Branch: `slow_consumer`. Status: implemented (2026-08-25); all tests pass.

## Revision: simplification (2026-08-25, supersedes parts of the text below)

* **Close-if-slow is dropped entirely.** No `PUBLISHER_CLOSE_SLOW` env var, no
  `close_slow` connection-string option, no close logic in `PubClientTcp` — slow
  consumers are always skipped ahead. `slow_client_detected(last_sent, oldest_available)`
  loses its `closing` parameter. This also removed the initial-request special case in
  `dispatch_initial_request` (a stale requested key is just another gap for
  `schedule_write` to skip) and the client `close()` machinery.
* **max_depth range is 3–60 inclusive** (was 3–59).
* **One validation site.** `PublisherOptions::parse_max_depth(text)` (throws) is the
  single parse/validate routine; `from_environment()` catches and warns with the default,
  `with_arguments()` lets it propagate. No duplicated strtol/range logic.

Implementation note — one deviation from the plan as written: on a client's *initial*
request (`dispatch_initial_request`), a gap after the requested key always skips ahead
(firing `slow_client_detected` with `closing=false`), even in close mode. Closing there
would put a reconnecting slow client into a permanent connect/close loop, since
`has_gap_after(old_key)` never becomes false again. Close mode therefore means: the
stalled connection is dropped, and the subscriber's reconnect resumes from the oldest
retained message.

## Problem

`PubClientTcp::last_message_` holds a `boost::intrusive_ptr<MessageBlock>`, and every
`MessageBlock` owns its `next_` link. `BoundedList` bounds only what the *list* references;
a stalled client pins the entire suffix chain from its last-sent block up to the head, so
memory grows without bound while it is stalled. Everything runs on one `io_context` thread
per transport, so this is a lifetime problem, not a race.

## Design goals

* Clients store a **key**, not a node. They hold at most one message (the in-flight one).
* The history container is a **deque of shared, immutable blocks with no linkage**
  (no next/prev/parent). Holding a block pins exactly that block and nothing else.
* The container hands out **only `shared_ptr`s** — no iterators, references, or element
  pointers — so misuse is prevented by the type system, not by documentation.
* No `boost::optional` (easy to accidentally move from); nullable `shared_ptr` instead.
* Slow consumers either **skip ahead** (default) or are **disconnected**, chosen by env var.
* History depth configurable by env var.

## 1. New types (`pub_sub/private/pub_intl.hh`)

Replace `BoundedList` and the intrusive `MessageBlock` with:

```cpp
/// Immutable, heap-allocated, shared. Owns a header + payload ref. No next/prev/parent.
class MessageBlock {
public:
    using BuffersType = std::array<boost::asio::const_buffer, 2>;
    MessageBlock(KeyType key, Message msg);         // builds buffers_ from this; object never moves
    MessageBlock(const MessageBlock&) = delete;     // never copied/moved — only shared
    MessageBlock& operator=(const MessageBlock&) = delete;

    KeyType            key()     const noexcept;
    std::size_t        size()    const noexcept;
    const void*        data()    const noexcept;
    const Message&     msg()     const noexcept;
    const BuffersType& buffers() const noexcept;    // stable for the block's lifetime
private:
    BlockHeader header_;
    Message     msg_;       // Message::data_ is a shared_ptr; copying is a refcount bump
    BuffersType buffers_;
};
using MessageBlockPtr = std::shared_ptr<const MessageBlock>;   // const: shared blocks are never mutated

class MessageHistory {
public:
    explicit MessageHistory(std::size_t capacity);            // no default ctor: owner must choose depth
    MessageBlockPtr push(KeyType key, Message msg);           // make_shared; evicts front when full; returns new block
    MessageBlockPtr first_after(KeyType key) const noexcept;  // nullptr if none  (TCP resume)
    MessageBlockPtr find(KeyType key)        const noexcept;  // nullptr if none  (UDP retransmit)
    MessageBlockPtr newest()                 const noexcept;  // nullptr if empty
    bool has_gap_after(KeyType last_key)     const noexcept;  // anything_evicted_ && last_key < evicted_up_to_
    bool empty() const noexcept;
    std::size_t size() const noexcept;
    std::size_t capacity() const noexcept;
private:
    std::deque<MessageBlockPtr> blocks_;
    std::size_t capacity_;
    KeyType     evicted_up_to_{0};   // key of the most recently evicted block
    bool        anything_evicted_{false};
};
```

Delete: `BoundedList`, `MessageBlock::next()/next(ptr)`, `make_message_block`,
`intrusive_ptr_add_ref/release(MessageBlock*)`, `ref_cnt_`. `boost/intrusive_ptr` remains
only for `PubClientTcp`.

`std::shared_ptr` + `make_shared` is one allocation per publish (same as today). The I/O
thread is single-threaded so atomic refcounting cost is negligible; not worth a hand-rolled
intrusive count.

## 2. `PublisherOptions` + environment parsing (`pub_intl.hh`, `pub.cc`)

```cpp
struct PublisherOptions {
    std::size_t max_depth{8};
    bool        close_slow_clients{false};
    static PublisherOptions from_environment();
};
```

* `PUBLISHER_MAX_DEPTH`: parse with `strtol`; accept only if the whole string is consumed and
  `2 < v < 60` (i.e. 3–59, exclusive as specified). Otherwise keep 8 and print one
  `std::cerr` warning.
* `PUBLISHER_CLOSE_SLOW`: enabled iff the value is exactly the literal `"1"`.
* Read once in `create_publisher_intl()` (`pub.cc`), stored on `PublisherIntl`.
* `PublisherIntlTcp` / `PublisherIntlUdp` constructors gain a `PublisherOptions` parameter,
  defaulted to `from_environment()` so existing test call sites
  (`PublisherIntlTcp("127.0.0.1", 0, debug)`) still compile; `pub.cc` passes it explicitly.
  The public plugin ABI (`PublisherPluginApi::publish`) is unchanged.
* `max_depth` sizes `MessageHistory` in both TCP and UDP (UDP gets a deeper retransmit
  window). `close_slow_clients` applies to TCP only.

## 3. `PubClientTcp` send loop (`pub_sub/private/pub_tcp.hh`)

Replace `MessageBlockPtr last_message_` with:

```cpp
KeyType         last_key_{0};
bool            has_position_{false};   // false until first send / non-NEXT request
MessageBlockPtr in_flight_;             // the only block a client ever holds
```

* `dispatch_initial_request()`: request == `NEXT_PUB_MSG()` → `has_position_ = false; sleep()`.
  Otherwise `last_key_ = request; has_position_ = true; schedule_write_or_sleep()`.
* `schedule_write()`:
  1. `if (!has_position_) { auto b = messages_.newest(); if (!b) return false; write(b); return true; }`
  2. `if (messages_.has_gap_after(last_key_))`: call
     `debug_.slow_client_detected(last_key_, oldest_available, close)`; if
     `close_slow_clients_` → cancel/close the socket and `return true` (removes it from the
     sleep list; the pending-op error path drops the last references). Otherwise fall through:
     the gap is skipped.
  3. `auto b = messages_.first_after(last_key_); if (!b) return false; write(b); return true;`
* `write(MessageBlockPtr b)`: `in_flight_ = std::move(b); last_key_ = in_flight_->key();`
  `async_write(s_, in_flight_->buffers(), ...)`. Completion handler: `in_flight_.reset()`
  first (release payload ASAP), then `schedule_write_or_sleep()`.
* `MessageHistory&` stays a reference member; the history is owned by `PublisherIntlTcp`,
  which tears down clients via `context_.stop()` + join (unchanged lifetime).
* Remove the commented-out `std::cout` debug blocks in touched functions.

## 4. UDP (`pub_sub/private/pub_udp.hh`)

* `send_message(key, msg)`: `auto b = messages_.push(key, msg);` then fragment from
  `b->data()` — one block created and stored, no duplicate buffer construction.
* `send_partial_message` takes `const MessageBlock&`.
* `register_retry()`: `auto b = messages_.find(req.msg_key); if (!b) return;`
  `context_.post([this, b, start, count]{ send_partial_message(*b, start, count); });`
  — the posted lambda owns one payload and nothing else.
* Delete `find_old_message`.

## 5. Debug hook + SOVERSION

* `common.hh`: `PubDebugNotices::slow_client_detected(KeyType last_sent, KeyType oldest_available, bool closing) {}`.
  Adds a vtable slot → bump `PUB_SUB_SOVERSION_CURRENT` and `AGE` in `pub_sub/CMakeLists.txt`
  (libtool rules documented there).

## 6. Tests (`pub_sub/tests/`)

* `test_message_history.cc` (new): eviction order; capacity honoured; `first_after` /
  `find` / `newest` at boundaries; `has_gap_after` before/after eviction; **lifetime test**:
  push N blocks with distinct payloads, retain one `MessageBlockPtr`, push until all evict,
  assert the retained payload's `use_count()` == test ref + block ref and the other N-1
  payloads' `weak_ptr`s are expired; reset the retained block → only the test's ref remains.
  `static_assert(!std::is_copy_constructible<MessageBlock>::value)`.
* `test_publisher_options.cc` (new): `setenv`/`unsetenv` matrix — depth 3 and 59 accepted;
  2, 60, `""`, `"abc"`, `"5x"`, `"-1"` rejected; close-slow `"1"` on, `"0"`/`"true"`/`"01"`/`" 1"` off.
* `test_full_subscription_internal.cc`: depth-3 publisher, subscriber handler blocked on a
  latch, publish ~10 messages, release:
  * skip mode → subscriber observes a key gap then resumes with latest keys;
  * close mode → `connection_terminated` fires on the subscriber, existing reconnect path
    resumes delivery.
  Existing "send through tcp" / "reset connection" cases must pass unchanged.
* Fix `add_test(NAME test_pub_sub_asan COMMAND test_pub_sub ...)` → `test_pub_sub_asan` so
  ASan actually runs under `ctest`.
* `test_sender.cc` / `test_slow_receiver.cc`: mention the two env vars for manual runs.

## 7. Docs / packaging

* README: document `PUBLISHER_MAX_DEPTH` (3–59) and `PUBLISHER_CLOSE_SLOW=1`, and the new
  slow-consumer semantics (skip by default).
* `version.txt` + `debian/changelog` → 0.4.0 (behaviour change). CMake enforces they match.
* CLAUDE.md: update the Threading/ownership paragraph.

## Decisions baked in

* Depth range exclusive: 3–59. `PUBLISHER_CLOSE_SLOW` accepts only the literal `1`.
* Invalid env values fall back to defaults with a one-line `std::cerr` warning.
* `MessageBlock` is non-copyable/non-movable and only ever lives behind `shared_ptr<const>`.
* `MessageHistory` has no default constructor so the env-derived depth cannot be bypassed by `{}`.
* Env vars are read once per `Publisher` creation, not per destination.

## Order of work

1 (+ `test_message_history.cc`) → 2 → 3 → 4 → 5 → remaining 6 → 7.

## Addendum: publisher options in the connection string

Status: implemented (2026-08-25); all tests pass. Ships in the same (unreleased) 0.4.0.

Implementation note: the new udp plugin test was the first to ever destroy a
`PublisherIntlUdp`, exposing a pre-existing crash — its defaulted destructor ran
`std::thread`'s destructor on a joinable thread (`std::terminate`). Fixed by resetting
the work guard, stopping the context, and joining in `~PublisherIntlUdp`.

Decision (2026-08-25): out-of-tree plugins exist and are in use, and this is wanted
quickly, so this takes the **middle path**: the plugin ABI (`publish(std::string, ...)`),
`prefix()` matching in `add_destination`, and `parse_pipeline` are all unchanged — a query
at the end of a stage cannot affect prefix matching, so existing plugins and connection
strings behave exactly as before. The parse helper is public and structured
(scheme + path + arguments); the built-in transports use it internally and custom plugins
can opt in, but nothing is imposed on them. The full `vector<StageAddress>` pipeline /
struct-typed plugin API is deferred to a deliberate plugin-API v2.

### Syntax and semantics

Query parameters on transport stages: `tcp://host:port?max_depth=16&close_slow=1`.

* Option keys mirror the env vars, lowercased without the prefix: `max_depth`, `close_slow`.
* **Precedence: connection string > environment > built-in default.** Env vars remain the
  per-`Publisher` default (read once, as today); the connection string overrides per
  destination.
* **Invalid values, unknown keys, and duplicate keys throw** (`std::runtime_error` out of
  `add_destination`, like other malformed connection strings). This deliberately differs
  from env parsing (warn + default): an env var may be set outside the app's control, but
  a connection string is explicit configuration — the connection cannot be made.
* `close_slow` accepts `"1"` (on) and `"0"` (off) — `"0"` lets a destination opt out of an
  env-enabled close. Anything else throws. `max_depth` must be a whole-string integer in
  3–59, else throws.
* **`close_slow` is accepted but ignored on `udp://` / `multi://`** (validated, not
  applied — the option is TCP-only).
* `zstd://` keeps its existing `zstd://level` form, unchanged.
* Subscriber side: out of scope. No subscriber options exist; a `?` in a subscriber string
  keeps failing address parsing as it does today.

### 1. Public helper: parse a stage into path + arguments

New public header `pub_sub/include/cds-pubsub/stage_address.hh` (+ `stage_address.cc`),
usable by custom plugins loaded via `load_plugin()`:

```cpp
namespace pub_sub
{
    struct StageAddress
    {
        std::string scheme;    // "tcp" for "tcp://..."
        std::string path;      // everything after "://", query stripped
        std::map< std::string, std::string > arguments;
    };
    // Parse "scheme://path?k1=v1&k2=v2".  Throws std::runtime_error if
    // "://" is absent or the scheme is empty.  The query is everything
    // after the FIRST '?'; no '?' or a bare trailing '?' -> empty
    // arguments.  Each component is key=value split at the first '='
    // (values may contain '=' and '?', and may be empty).  Throws on an
    // empty component, a missing '=', an empty key, or a duplicate key.
    // No URL-decoding.
    StageAddress parse_stage_address( const std::string& stage );
}
```

Values are validated by the consumer, not the parser (an empty value is a parse-level
pass, a validation-level throw). Adding one exported function + header is covered by the
SOVERSION current/age bump already staged for 0.4.0; add the header to `PUBLIC_HEADERS`.

### 2. `PublisherOptions::with_arguments` (`pub_intl.hh`)

```cpp
// Returns a copy with overrides applied.  Throws std::runtime_error on an
// unknown key or invalid value ("max_depth": whole-string integer in 3-59;
// "close_slow": exactly "1" or "0").
PublisherOptions with_arguments(
    const std::map< std::string, std::string >& arguments ) const;
```

Both keys are accepted for every transport; the UDP publisher simply never reads
`close_slow_clients`, which implements "ignored on UDP" without a second validation
table.

### 3. Plugin changes

Move the four built-in publisher plugin classes out of `pub.cc` into a new private
header `pub_sub/private/pub_plugins_intl.hh` (tests include `private/`, so they can then
construct plugins directly and assert that options landed on the instances). `pub.cc`
keeps `create_publisher_intl()` and the facade methods.

In `TcpPublisherPluginApi` / `UdpPublisherPluginApi` / `UdpMultiPublisherPluginApi`::`publish`:

```cpp
auto stage = parse_stage_address( address );
// scheme check replaces the raw prefix check: stage.scheme + "://" == prefix()
auto options = options_.with_arguments( stage.arguments );
// existing address parsing, now on stage.path (no substr fiddling)
```

`PublisherIntl::add_destination`'s prefix matching still sees the raw stage string; the
prefix is at position 0 and the query at the end, so plugin selection is unchanged.
`PubPluginZLibFilter` keeps its raw-string parsing (`zstd://3`), untouched.

### 4. Tests

* `test_stage_address.cc` (new): scheme/path split (`tcp://h:1`, `zstd://3`, empty
  path `x://`); no query; one and several arguments; value containing `=` and `?`;
  empty value; bare trailing `?`; throws on missing `://`, empty scheme, empty
  component (`a=1&&b=2`), missing `=`, empty key (`=1`), duplicate key.
* `test_publisher_options.cc` additions: `with_arguments` matrix — `max_depth` 3/59
  accepted, 2/60/`""`/`"abc"`/`"5x"`/`"-1"` throw; `close_slow` `"1"`→on, `"0"`→off,
  `"true"`/`"01"`/`" 1"` throw; unknown key throws; empty map keeps the base options;
  precedence test — env depth 5 + argument depth 10 → 10.
* Plugin-level (via `pub_plugins_intl.hh`): build an instance via
  `TcpPublisherPluginApi::publish( "tcp://127.0.0.1:0?max_depth=3&close_slow=1", ... )`,
  `dynamic_cast` to `PublisherIntlTcp*`, assert `messages_.capacity() == 3` and
  `close_slow_clients_`. Env-precedence variant: plugin built from an env with depth 5,
  string says 3 → 3; no arguments → 5. UDP variant: `?max_depth=10&close_slow=1`
  constructs with capacity 10 (close validated but ignored). Invalid values throw from
  `publish()` without creating sockets.
* Facade-level: `Publisher::add_destination` throws for `?max_depth=99`, `?bogus=1`,
  and `?close_slow=yes`; existing option-free strings still work.

### 5. Docs / packaging

* README: connection-string option syntax, precedence order, throw-on-invalid, and the
  UDP-ignores-`close_slow` note.
* CLAUDE.md: extend the plugin/pipeline paragraph with the `?key=value` syntax and the
  `parse_stage_address` helper.
* Amend the (unreleased) 0.4.0 `debian/changelog` entry; no further version bump.

### Decisions baked in

* Options ride on the transport stage, not a dedicated `opts://` stage — pipelines build
  back-to-front, so only the transport plugin can size its own history at construction.
* Middle path: plugin ABI, prefix matching, and `parse_pipeline` untouched because
  out-of-tree plugins are in use. Helper is public API so third-party plugins can adopt
  the same convention; for them it stays opt-in (plugins receive the raw stage string).
* `StageAddress` carries the scheme so a stage parses into everything a plugin needs;
  the `vector<StageAddress>` pipeline with struct-typed plugin signatures is deferred to
  a future plugin-API v2 (it would be a source/ABI break for existing plugins).
* Parser is mechanical (scheme + path + string map); all option semantics live in
  `PublisherOptions::with_arguments`.
* `close_slow` is validated everywhere but only read by the TCP transport — "ignored on
  UDP" without a per-transport validation table.

## Alternatives considered

* `expire()` on list nodes with payload nulling (original idea): works but leaves an
  unbounded chain of empty nodes and adds a second lifetime mechanism.
* Value-type snapshot returned via `boost::optional`: rejected — easy to move-from by accident.
