See https://wikitech.wikimedia.org/wiki/User:Stashbot
Avatar by Freepik from www.flaticon.com CC 3.0 BY.
User Details
User Details
- User Since
- Feb 10 2016, 12:29 AM (544 w, 3 d)
- Roles
- Bot
- Availability
- Available
- IRC Nick
- stashbot
- LDAP User
- Stashbot
- MediaWiki User
- Unknown
Today
Today
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-18T00:11:23Z] <bking@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs2018.codfw.wmnet, repooling source-only afterwards
Yesterday
Yesterday
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T23:53:18Z] <bking@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs2017.codfw.wmnet -> wdqs2026.codfw.wmnet, repooling source-only afterwards
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T23:09:34Z] <bking@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs2018.codfw.wmnet, repooling source-only afterwards
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T23:08:03Z] <bking@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs2017.codfw.wmnet -> wdqs2026.codfw.wmnet, repooling source-only afterwards
Stashbot added a comment to T425477: Stream live progress during web updates.
Mentioned in SAL (#wikimedia-cloud) [2026-07-17T21:06:34Z] <wmbot~jeanfred@tools-bastion-15> Deploy ea1afda (Add integration tests for SSE streaming error scenarios) for T425477
Stashbot added a comment to T425477: Stream live progress during web updates.
Mentioned in SAL (#wikimedia-cloud) [2026-07-17T21:06:32Z] <wmbot~jeanfred@tools-bastion-15> Deploy b2fd525 (Add retry button for transient errors and connection loss in SSE template) for T425477
Stashbot added a comment to T425477: Stream live progress during web updates.
Mentioned in SAL (#wikimedia-cloud) [2026-07-17T21:06:29Z] <wmbot~jeanfred@tools-bastion-15> Deploy 8926c8f (Render differentiated error UIs in SSE template) for T425477
Stashbot added a comment to T425477: Stream live progress during web updates.
Mentioned in SAL (#wikimedia-cloud) [2026-07-17T21:06:27Z] <wmbot~jeanfred@tools-bastion-15> Deploy 87dcff4 (Escape dynamic content in innerHTML in SSE template) for T425477
Stashbot added a comment to T425477: Stream live progress during web updates.
Mentioned in SAL (#wikimedia-cloud) [2026-07-17T21:06:25Z] <wmbot~jeanfred@tools-bastion-15> Deploy 9deb932 (Extract handlers into named functions in SSE template) for T425477
Stashbot added a comment to T425477: Stream live progress during web updates.
Mentioned in SAL (#wikimedia-cloud) [2026-07-17T21:06:23Z] <wmbot~jeanfred@tools-bastion-15> Deploy 5d8de99 (Enrich SSE error events with error_type, error_category, and query) for T425477
Stashbot added a comment to T425477: Stream live progress during web updates.
Mentioned in SAL (#wikimedia-cloud) [2026-07-17T21:06:20Z] <wmbot~jeanfred@tools-bastion-15> Deploy fc20c17 (Add ErrorCategory enum and declare it on exception classes) for T425477
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T21:05:03Z] <bking@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1022.eqiad.wmnet -> wdqs1026.eqiad.wmnet, repooling source-only afterwards
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T20:57:16Z] <bking@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs2026.codfw.wmnet -> wdqs2017.codfw.wmnet, repooling source-only afterwards
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T20:11:56Z] <bking@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs2026.codfw.wmnet -> wdqs2017.codfw.wmnet, repooling source-only afterwards
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T20:08:06Z] <bking@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1022.eqiad.wmnet -> wdqs1026.eqiad.wmnet, repooling source-only afterwards
Stashbot added a comment to T432426: Network unavailable on a few VMs.
Mentioned in SAL (#wikimedia-cloud) [2026-07-17T19:56:00Z] <andrewbogott> hard rebooting wsexport-calibre01, T432426
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T19:55:37Z] <bking@deploy2003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 12s)
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T19:55:29Z] <bking@deploy2003> Started deploy [wdqs/wdqs@e8fb00c]: T430880
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T19:55:21Z] <bking@deploy2003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 09s)
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T19:55:15Z] <bking@deploy2003> Started deploy [wdqs/wdqs@e8fb00c]: T430880
Stashbot added a comment to T432426: Network unavailable on a few VMs.
Mentioned in SAL (#wikimedia-cloud) [2026-07-17T19:54:56Z] <andrewbogott> hard rebooting demo-patrollers and reader-embedding, reporting as down. Cleanup for T432426
Stashbot added a comment to T432426: Network unavailable on a few VMs.
Mentioned in SAL (#wikimedia-cloud) [2026-07-17T19:53:18Z] <andrewbogott> hard rebooted logging-logstash-03 and logging-logstash-04 for T432426
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T19:50:43Z] <bking@deploy2003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 08s)
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T19:50:39Z] <bking@deploy2003> Started deploy [wdqs/wdqs@e8fb00c]: T430880
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T19:50:18Z] <bking@deploy2003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 10m 03s)
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T19:40:19Z] <bking@deploy2003> Started deploy [wdqs/wdqs@e8fb00c]: T430880
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T19:40:02Z] <bking@deploy2003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 15s)
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T19:39:53Z] <bking@deploy2003> Started deploy [wdqs/wdqs@e8fb00c]: T430880
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T19:37:48Z] <bking@deploy2003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 16s)
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T19:37:36Z] <bking@deploy2003> Started deploy [wdqs/wdqs@e8fb00c]: T430880
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T19:33:46Z] <bking@deploy2003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 12s)
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T19:33:41Z] <bking@deploy2003> Started deploy [wdqs/wdqs@e8fb00c]: T430880
Stashbot added a comment to T432418: Score extension broken - everything "Impossible de compiler".
Mentioned in SAL (#wikimedia-operations) [2026-07-17T16:42:04Z] <reedy@deploy2003> Finished scap sync-world: Backport for [[gerrit:1311872|generatePngAndMidi.sh: Remove SCORE_SAFE (T432418)]] (duration: 10m 29s)
Stashbot added a comment to T432418: Score extension broken - everything "Impossible de compiler".
Mentioned in SAL (#wikimedia-operations) [2026-07-17T16:33:29Z] <reedy@deploy2003> reedy, hartman: Backport for [[gerrit:1311872|generatePngAndMidi.sh: Remove SCORE_SAFE (T432418)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
Stashbot added a comment to T432418: Score extension broken - everything "Impossible de compiler".
Mentioned in SAL (#wikimedia-operations) [2026-07-17T16:31:35Z] <reedy@deploy2003> Started scap sync-world: Backport for [[gerrit:1311872|generatePngAndMidi.sh: Remove SCORE_SAFE (T432418)]]
Stashbot added a comment to T432254: scap on deployment-deploy04 failing to reach logging-logstash-04.logging.eqiad1.wikimedia.cloud.
Mentioned in SAL (#wikimedia-cloud) [2026-07-17T15:12:53Z] <bd808> Hard reboot of logging-logstash-04.logging.eqiad1.wikimedia.cloud; non-responsive to network (T432254)
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T02:46:15Z] <ryankemper@cumin2003> END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T430880, Restore categories on wdqs1025 after Bookworm reimage) xfer categories from wdqs2020.codfw.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T02:46:12Z] <ryankemper@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, Restore categories on wdqs1025 after Bookworm reimage) xfer categories from wdqs2020.codfw.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T02:44:46Z] <ryankemper@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, Restore categories on wdqs1025 after Bookworm reimage) xfer categories from wdqs2020.codfw.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T02:37:43Z] <ryankemper@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, Restore categories on wdqs1025 after Bookworm reimage) xfer categories from wdqs2020.codfw.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T01:12:58Z] <ryankemper@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, Restore wdqs1025 after Bookworm reimage) xfer wikidata_main from wdqs2020.codfw.wmnet -> wdqs1025.eqiad.wmnet, repooling source-only afterwards
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T01:07:57Z] <ryankemper@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, Restore wdqs1027 after Bookworm reimage) xfer scholarly_articles from wdqs2026.codfw.wmnet -> wdqs1027.eqiad.wmnet, repooling both afterwards
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T00:23:46Z] <ryankemper@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, Restore wdqs1025 after Bookworm reimage) xfer wikidata_main from wdqs2020.codfw.wmnet -> wdqs1025.eqiad.wmnet, repooling source-only afterwards
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-17T00:23:13Z] <ryankemper@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, Restore wdqs1027 after Bookworm reimage) xfer scholarly_articles from wdqs2026.codfw.wmnet -> wdqs1027.eqiad.wmnet, repooling both afterwards
Thu, Jul 16
Thu, Jul 16
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T23:50:15Z] <ryankemper@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, xfer to freshly reimaged/scap-deployed wdqs2025 after Bookworm reimage) xfer wikidata_main from wdqs2020.codfw.wmnet -> wdqs2025.codfw.wmnet, repooling source-only afterwards
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T23:12:17Z] <ryankemper> T430880 depooled dnsdisc of wdqs-internal-scholarly-eqiad bc we only have 1 host there
Stashbot added a comment to T431146: Error “The refresh token is invalid.” after migrating Wikidata Lexeme Forms to OAuth 2.
Mentioned in SAL (#wikimedia-cloud) [2026-07-16T23:08:51Z] <wmbot~lucaswerkmeister@tools-bastion-15> deployed 418e934c5a (update Manbhumi item ID after merge) [actually deployed 30878c6831, temporary T431146 branch rebased on top of that)
Stashbot added a comment to T398401: Rename of a repository is not replicated.
Mentioned in SAL (#wikimedia-releng) [2026-07-16T23:06:56Z] <mutante> gerrit1003/2002/2003: rm /srv/gerrit/.ssh/config and revert gerrit:1311577 which fixed T398401 but caused T432413 - replication is working again
Stashbot added a comment to T432413: Creating a Patchdemo wiki for VisualEditor fails on missing ref from gerrit-replica.
Mentioned in SAL (#wikimedia-releng) [2026-07-16T23:06:56Z] <mutante> gerrit1003/2002/2003: rm /srv/gerrit/.ssh/config and revert gerrit:1311577 which fixed T398401 but caused T432413 - replication is working again
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T23:01:07Z] <ryankemper@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, xfer to freshly reimaged/scap-deployed wdqs2025 after Bookworm reimage) xfer wikidata_main from wdqs2020.codfw.wmnet -> wdqs2025.codfw.wmnet, repooling source-only afterwards
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T22:32:08Z] <bking@deploy2003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 27s)
Stashbot added a comment to T430880: Migrate WDQS hosts to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T22:32:02Z] <bking@deploy2003> Started deploy [wdqs/wdqs@e8fb00c]: T430880
Stashbot added a comment to T430986: MediaWiki\Upload\Exception\UploadChunkFileException: Error storing file in '{chunkPath}': backend-fail-internal; local-swift-codfw.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T22:26:23Z] <ladsgroup@deploy2003> Finished scap sync-world: Backport for [[gerrit:1311420|Upload: Do not throw for failure to save a chunk file (T430986)]], [[gerrit:1311529|AddImage: Request only standard thumbnail sizes (T428797)]] (duration: 08m 51s)
Stashbot added a comment to T428797: [wmf.5-regression] Add image dialog doesn't display image.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T22:26:23Z] <ladsgroup@deploy2003> Finished scap sync-world: Backport for [[gerrit:1311420|Upload: Do not throw for failure to save a chunk file (T430986)]], [[gerrit:1311529|AddImage: Request only standard thumbnail sizes (T428797)]] (duration: 08m 51s)
Stashbot added a comment to T430986: MediaWiki\Upload\Exception\UploadChunkFileException: Error storing file in '{chunkPath}': backend-fail-internal; local-swift-codfw.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T22:19:19Z] <ladsgroup@deploy2003> ladsgroup, urbanecm: Backport for [[gerrit:1311420|Upload: Do not throw for failure to save a chunk file (T430986)]], [[gerrit:1311529|AddImage: Request only standard thumbnail sizes (T428797)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
Stashbot added a comment to T428797: [wmf.5-regression] Add image dialog doesn't display image.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T22:19:19Z] <ladsgroup@deploy2003> ladsgroup, urbanecm: Backport for [[gerrit:1311420|Upload: Do not throw for failure to save a chunk file (T430986)]], [[gerrit:1311529|AddImage: Request only standard thumbnail sizes (T428797)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
Stashbot added a comment to T430986: MediaWiki\Upload\Exception\UploadChunkFileException: Error storing file in '{chunkPath}': backend-fail-internal; local-swift-codfw.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T22:17:32Z] <ladsgroup@deploy2003> Started scap sync-world: Backport for [[gerrit:1311420|Upload: Do not throw for failure to save a chunk file (T430986)]], [[gerrit:1311529|AddImage: Request only standard thumbnail sizes (T428797)]]
Stashbot added a comment to T428797: [wmf.5-regression] Add image dialog doesn't display image.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T22:17:32Z] <ladsgroup@deploy2003> Started scap sync-world: Backport for [[gerrit:1311420|Upload: Do not throw for failure to save a chunk file (T430986)]], [[gerrit:1311529|AddImage: Request only standard thumbnail sizes (T428797)]]
Stashbot added a comment to T432042: Add logging for various re-authentication methods (AuthPopup vs DataStash flow).
Mentioned in SAL (#wikimedia-operations) [2026-07-16T21:45:00Z] <sbassett@deploy2003> Finished scap sync-world: Backport for [[gerrit:1311482|Use non-sampled authentication log channel instead of authevents (T432042)]] (duration: 06m 31s)
Stashbot added a comment to T432042: Add logging for various re-authentication methods (AuthPopup vs DataStash flow).
Mentioned in SAL (#wikimedia-operations) [2026-07-16T21:40:13Z] <sbassett@deploy2003> sbassett: Backport for [[gerrit:1311482|Use non-sampled authentication log channel instead of authevents (T432042)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
Stashbot added a comment to T432042: Add logging for various re-authentication methods (AuthPopup vs DataStash flow).
Mentioned in SAL (#wikimedia-operations) [2026-07-16T21:38:29Z] <sbassett@deploy2003> Started scap sync-world: Backport for [[gerrit:1311482|Use non-sampled authentication log channel instead of authevents (T432042)]]
Stashbot added a comment to T432042: Add logging for various re-authentication methods (AuthPopup vs DataStash flow).
Mentioned in SAL (#wikimedia-operations) [2026-07-16T21:30:28Z] <sbassett@deploy2003> Finished scap sync-world: Backport for [[gerrit:1311481|Cleanup: remove reauth indicator from log message (T432042)]] (duration: 08m 19s)
Stashbot added a comment to T432042: Add logging for various re-authentication methods (AuthPopup vs DataStash flow).
Mentioned in SAL (#wikimedia-operations) [2026-07-16T21:23:51Z] <sbassett@deploy2003> sbassett: Backport for [[gerrit:1311481|Cleanup: remove reauth indicator from log message (T432042)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
Stashbot added a comment to T432042: Add logging for various re-authentication methods (AuthPopup vs DataStash flow).
Mentioned in SAL (#wikimedia-operations) [2026-07-16T21:22:08Z] <sbassett@deploy2003> Started scap sync-world: Backport for [[gerrit:1311481|Cleanup: remove reauth indicator from log message (T432042)]]
Stashbot added a comment to T432316: Print/export section missing in 1.47.0-wmf.11.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T20:41:10Z] <aude@deploy2003> Finished scap sync-world: Backport for [[gerrit:1311472|Preserve menus after toolbox (e.g. print/export) in page tools (T432316)]] (duration: 07m 34s)
Stashbot added a comment to T432316: Print/export section missing in 1.47.0-wmf.11.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T20:35:16Z] <aude@deploy2003> aude: Backport for [[gerrit:1311472|Preserve menus after toolbox (e.g. print/export) in page tools (T432316)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
Stashbot added a comment to T432316: Print/export section missing in 1.47.0-wmf.11.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T20:33:36Z] <aude@deploy2003> Started scap sync-world: Backport for [[gerrit:1311472|Preserve menus after toolbox (e.g. print/export) in page tools (T432316)]]
Stashbot added a comment to T430589: CVN: Replace Bullseye nfs server with a new Trixie server.
Mentioned in SAL (#countervandalism) [2026-07-16T20:04:12Z] <Krinkle> Delete cvn-nfs1 per T430589
Stashbot added a comment to T432396: Switchover s2 master (db2207 -> db2204).
Mentioned in SAL (#wikimedia-operations) [2026-07-16T20:02:58Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Depool db2207 T432396', diff saved to https://phabricator.wikimedia.org/P94893 and previous config saved to /var/cache/conftool/dbconfig/20260716-200257-marostegui.json
Stashbot added a comment to T432396: Switchover s2 master (db2207 -> db2204).
Mentioned in SAL (#wikimedia-operations) [2026-07-16T20:01:57Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Promote db2204 to s2 primary T432396', diff saved to https://phabricator.wikimedia.org/P94892 and previous config saved to /var/cache/conftool/dbconfig/20260716-200157-marostegui.json
Stashbot added a comment to T432396: Switchover s2 master (db2207 -> db2204).
Mentioned in SAL (#wikimedia-operations) [2026-07-16T20:00:50Z] <marostegui> Starting emergency s2 codfw failover from db2207 to db2204 - T432396
Stashbot added a comment to T432396: Switchover s2 master (db2207 -> db2204).
Mentioned in SAL (#wikimedia-operations) [2026-07-16T19:56:29Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Set db2204 with weight 0 T432396', diff saved to https://phabricator.wikimedia.org/P94891 and previous config saved to /var/cache/conftool/dbconfig/20260716-195628-marostegui.json
Stashbot added a comment to T432396: Switchover s2 master (db2207 -> db2204).
Mentioned in SAL (#wikimedia-operations) [2026-07-16T19:55:54Z] <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s2 T432396
Stashbot added a comment to T430879: Migrate WCQS to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T19:23:26Z] <bking@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430879, restore data on newly-reimaged host) xfer commons from wcqs1002.eqiad.wmnet -> wcqs1001.eqiad.wmnet, repooling source-only afterwards
Stashbot added a comment to T428971: Allow configuration of canary and production checks based on deployment target.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T18:55:42Z] <dancy@deploy2003> Finished scap sync-world: testing T428971 (duration: 02m 41s)
Stashbot added a comment to T428971: Allow configuration of canary and production checks based on deployment target.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T18:53:01Z] <dancy@deploy2003> Started scap sync-world: testing T428971
Stashbot added a comment to T432360: Buildkit v0.31.2 released.
Mentioned in SAL (#wikimedia-releng) [2026-07-16T18:51:29Z] <dancy> Updated buildkit to v0.31.2 gitlab-cloud-runners (staging and production) (T432360)
Stashbot added a comment to T430830: 1.47.0-wmf.11 deployment blockers.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T18:18:46Z] <jhuneidi@deploy2003> rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.11 refs T430830
Stashbot added a comment to T430879: Migrate WCQS to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T18:08:00Z] <bking@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430879, restore data on newly-reimaged host) xfer commons from wcqs1002.eqiad.wmnet -> wcqs1001.eqiad.wmnet, repooling source-only afterwards
Stashbot added a comment to T430879: Migrate WCQS to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T18:06:33Z] <bking@deploy2003> Finished deploy [wdqs/wdqs@e8fb00c] (wcqs): T430879 (duration: 00m 23s)
Stashbot added a comment to T430879: Migrate WCQS to Bookworm or later.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T18:06:23Z] <bking@deploy2003> Started deploy [wdqs/wdqs@e8fb00c] (wcqs): T430879
Stashbot added a comment to T430589: CVN: Replace Bullseye nfs server with a new Trixie server.
Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-16T17:30:39Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) (T430589)
Stashbot added a comment to T430589: CVN: Replace Bullseye nfs server with a new Trixie server.
Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-16T17:30:38Z] <andrew@cloudcumin1001> START - Cookbook wmcs.nfs.migrate_service (T430589)
Stashbot added a comment to T430589: CVN: Replace Bullseye nfs server with a new Trixie server.
Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-16T17:30:25Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) (T430589)
Stashbot added a comment to T430589: CVN: Replace Bullseye nfs server with a new Trixie server.
Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-16T17:26:44Z] <andrew@cloudcumin1001> START - Cookbook wmcs.nfs.migrate_service (T430589)
Stashbot added a comment to T430589: CVN: Replace Bullseye nfs server with a new Trixie server.
Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-16T17:26:16Z] <andrew@cloudcumin1001> END (ERROR) - Cookbook wmcs.nfs.migrate_service (exit_code=97) (T430589)
Stashbot added a comment to T430589: CVN: Replace Bullseye nfs server with a new Trixie server.
Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-16T17:26:11Z] <andrew@cloudcumin1001> START - Cookbook wmcs.nfs.migrate_service (T430589)
Stashbot added a comment to T430589: CVN: Replace Bullseye nfs server with a new Trixie server.
Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-16T17:24:54Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) (T430589)
Stashbot added a comment to T430589: CVN: Replace Bullseye nfs server with a new Trixie server.
Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-16T17:24:44Z] <andrew@cloudcumin1001> START - Cookbook wmcs.nfs.migrate_service (T430589)
Stashbot added a comment to T432374: Project deployment-prep instance deployment-cache-text08 is down.
Mentioned in SAL (#wikimedia-releng) [2026-07-16T17:07:04Z] <bd808> Hard reboot of deployment-cache-text08.deployment-prep.eqiad1.wikimedia.cloud via Horizon; console shows OOM (T432374)
Stashbot added a comment to T432326: DiskSpace (contint1003).
Mentioned in SAL (#wikimedia-releng) [2026-07-16T16:34:17Z] <dancy> Restarted docker on contint2003 (T432326)
Stashbot added a comment to T432326: DiskSpace (contint1003).
Mentioned in SAL (#wikimedia-releng) [2026-07-16T16:33:03Z] <dancy> Restarted docker on contint1003 (T432326)
Stashbot added a comment to T349582: Set $wgMathInternalRestbaseURL explictly in production.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T14:57:11Z] <krinkle@deploy2003> Started scap sync-world: Backport for [[gerrit:1310596|Set $wgMathInternalRestbaseURL explicitly (T349582)]]
Stashbot added a comment to T432108: stunnel4 keeps not exiting cleanly on Bookworm(?).
Mentioned in SAL (#wikimedia-operations) [2026-07-16T14:27:55Z] <kamila@deploy2003> Finished scap sync-world: Test deployment to check rsync is working - T432108 (duration: 02m 57s)
Stashbot added a comment to T432108: stunnel4 keeps not exiting cleanly on Bookworm(?).
Mentioned in SAL (#wikimedia-operations) [2026-07-16T14:24:58Z] <kamila@deploy2003> Started scap sync-world: Test deployment to check rsync is working - T432108
Stashbot added a comment to T432108: stunnel4 keeps not exiting cleanly on Bookworm(?).
Mentioned in SAL (#wikimedia-operations) [2026-07-16T14:21:44Z] <kamila@deploy2003> sync-world aborted: Test deployment to check rsync is working - T432108 (duration: 00m 36s)
Stashbot added a comment to T430922: codfw: rack B6 maintenance.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T14:21:23Z] <topranks> reboot lsw1-b6-codfw to upgrade JunOS T430922
Stashbot added a comment to T432108: stunnel4 keeps not exiting cleanly on Bookworm(?).
Mentioned in SAL (#wikimedia-operations) [2026-07-16T14:21:10Z] <kamila@deploy2003> Started scap sync-world: Test deployment to check rsync is working - T432108
Stashbot added a comment to T430922: codfw: rack B6 maintenance.
Mentioned in SAL (#wikimedia-operations) [2026-07-16T14:02:08Z] <topranks> beginning depools for lsw1-b6-codfw maintenance T430922
Stashbot added a comment to T431248: [toolsbeta,infra,logging] We are reaching 96% of quota by objects.
Mentioned in SAL (#wikimedia-cloud) [2026-07-16T13:55:57Z] <Raymond_Ndibe> increased the object count quota to 500k T431248