fix(topology): reattach fanout on cross-type producer key swap#25725
Open
pront wants to merge 2 commits into
Open
fix(topology): reattach fanout on cross-type producer key swap#25725pront wants to merge 2 commits into
pront wants to merge 2 commits into
Conversation
get_changed_outputs only iterated to_change sets, missing keys that disappear as one kind of source and reappear as another (e.g. an enrichment-table-derived source removed while a regular source with the same key is added). Downstream components whose input stays the same would silently stop receiving events after such a reload. Extend the set to also include cross-type respawns: (sources.to_remove | enrichment_tables.to_remove) intersected with (sources.to_add | enrichment_tables.to_add). Enrichment-table sink keys can appear in enrichment_tables.* but never match output_ids.component, so their inclusion in the union is harmless. Note: this addresses the codex-flagged reattach gap by inspection. A minimal isolated repro of the end-to-end symptom (unchanged sink loses events post-reload) was attempted but hit other reload-path issues that need separate investigation.
6b7e070 to
037d160
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a topology reload bug where downstream components whose input key stayed the same across reload would silently stop receiving events if the upstream producer changed kind (e.g. an enrichment-table-derived source at key
Xreplaced by a regular source at keyX, or a transform atXreplaced by a source atX).get_changed_outputspreviously only iteratedto_changesets, missing cross-type respawns. It now also considers keys that appear in(sources.to_remove ∪ transforms.to_remove ∪ enrichment_tables.to_remove)intersected with(sources.to_add ∪ transforms.to_add ∪ enrichment_tables.to_add).Vector configuration
Reproducer config A (pre-reload):
Reloaded config B:
How did you test this PR?
Manual end-to-end reproduction with
./target/debug/vector --config live.yaml, swapping A -> B via SIGHUP:observerreceives 0 events after reload (bug confirmed).observercontinues receivingdemo_logsevents after reload.Also ran
cargo checkwith the topology feature set. A unit-test-only regression could not be reliably reproduced in the current test harness (the reattach path fires but events don't flow viabasic_source/basic_sinkafter reload — a separate test-infra issue), so the branch relies on the manual reproducer above. Happy to add a test if a reviewer knows a working scaffold.Change Type
Is this a breaking change?
Does this PR include user facing changes?
no-changeloglabel to this PR.References