Problem
Today the CXO publisher rides one transport per peer-pair (dmsg only). When dmsg congests (e.g. a backed-up subscriber slows down delivery, a dmsg-server goes flappy, the local visor's session count maxes out), the publisher has no fallback. Messages either stall in the send queue, or — post-#2538 — the conn dies after `sendMsgQueueTimeout` and the peer's subscriber sees a gap.
For app-level traffic, the analogous gap is being addressed by the multi-transport Plan B framework in #2616 (UDP-over-skywire-routes). Group chat's CXO traffic should get the same treatment so chat resilience doesn't lag the app surface.
Goal
Allow the CXO publisher to fan a message out over BOTH dmsg AND a skynet route to the same peer, with deterministic dedup on the subscriber side. Either path succeeds → subscriber sees the message exactly once.
Proposed shape
Transport selection
Each subscriber connection currently uses one underlying transport (dmsg.Stream). Add a second logical "channel" per subscriber: a skynet RouteGroup or DatagramRouteGroup to the same peer.
The publisher's send path:
- Encode message once.
- WriteAll to (dmsg.Stream, skynet.RouteGroup) in parallel.
- Track per-send delivery state: `{dmsgDelivered: bool, skynetDelivered: bool, errors: []}`.
- Success = either delivered. Failure = both errored.
Subscriber dedup
Messages carry a content hash + sender-seq tuple. Subscriber maintains a small (256-entry) bloom filter of recently-seen (sender, seq) tuples; second arrival via the alt channel is silently dropped.
Path failure isolation
A dmsg-only failure no longer ends the per-peer Conn; the skynet path covers it. Same for skynet-only. Both-fail-N-times-in-a-row triggers the existing reconnect machinery (#2606 cascade).
Negotiation
On Subscribe, peer negotiates supported channels ("I have dmsg only" vs "I have both"). Backward-compatible: pre-PR peers get dmsg-only as today; only when BOTH peers advertise skynet does the publisher attempt the second channel.
Out of scope (v1)
- Adaptive path selection (always-fan-out is fine for chat-level traffic volumes; future tuning can throttle)
- Cross-path latency reporting / per-path stats
- Three+ transport channels (start with two)
Implementation pieces (rough)
- `pkg/cxo/node/conn.go` — add a second underlying writer slot per Conn
- `pkg/cxo/node/handshake.go` — negotiate alt-channel during the same handshake that sets up dmsg
- `pkg/cxo/node/conn.go` — sendMsg fans out to both channels, tracks per-channel delivery
- `pkg/cxo/node/conn.go` — receiveMsg dedupes via (sender, seq) bloom
- Tests: round-trip with dmsg-only, with skynet-only, with both; replay/dedup behavior
Owner
Open. Compatible with #2616 framework as a complement (#2616 is for app traffic, this is for the CXO control+chat traffic).
Related
Problem
Today the CXO publisher rides one transport per peer-pair (dmsg only). When dmsg congests (e.g. a backed-up subscriber slows down delivery, a dmsg-server goes flappy, the local visor's session count maxes out), the publisher has no fallback. Messages either stall in the send queue, or — post-#2538 — the conn dies after `sendMsgQueueTimeout` and the peer's subscriber sees a gap.
For app-level traffic, the analogous gap is being addressed by the multi-transport Plan B framework in #2616 (UDP-over-skywire-routes). Group chat's CXO traffic should get the same treatment so chat resilience doesn't lag the app surface.
Goal
Allow the CXO publisher to fan a message out over BOTH dmsg AND a skynet route to the same peer, with deterministic dedup on the subscriber side. Either path succeeds → subscriber sees the message exactly once.
Proposed shape
Transport selection
Each subscriber connection currently uses one underlying transport (dmsg.Stream). Add a second logical "channel" per subscriber: a skynet RouteGroup or DatagramRouteGroup to the same peer.
The publisher's send path:
Subscriber dedup
Messages carry a content hash + sender-seq tuple. Subscriber maintains a small (256-entry) bloom filter of recently-seen (sender, seq) tuples; second arrival via the alt channel is silently dropped.
Path failure isolation
A dmsg-only failure no longer ends the per-peer Conn; the skynet path covers it. Same for skynet-only. Both-fail-N-times-in-a-row triggers the existing reconnect machinery (#2606 cascade).
Negotiation
On Subscribe, peer negotiates supported channels ("I have dmsg only" vs "I have both"). Backward-compatible: pre-PR peers get dmsg-only as today; only when BOTH peers advertise skynet does the publisher attempt the second channel.
Out of scope (v1)
Implementation pieces (rough)
Owner
Open. Compatible with #2616 framework as a complement (#2616 is for app traffic, this is for the CXO control+chat traffic).
Related