See also T354042: WikimediaDebug features (e.g. logging and profiling) for MediaWiki JobQueue jobs which is related but subtly different. That task is about profiling one job, which may be easier.

This should now be fixed for this use case, general discussion continues in parent. Feel free to reopen if issues persist.

Change #1306643 merged by Clément Goubert:

[operations/puppet@production] trafficserver::backend: Remove X-W-D for api.w.o

https://gerrit.wikimedia.org/r/1306643

Change #1306643 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] trafficserver::backend: Remove X-W-D for api.w.o

https://gerrit.wikimedia.org/r/1306643

The reason that happens is the X-Wikimedia-Debug header (or the extension which just sets that header) bypasses ATS at the edge where the routing to the rest gateway for api.wikimedia.org API endpoints is done. It forces you to the mw-debug MediaWiki-On-Kubernetes deployment, which redirects everything under api.wikimedia.org to the API Doc page.

I discovered this while backporting a change to a EditCheck that had to access an API hosted on api.wikimedia.org, which seems to be fundamentally untestable on the testservers as a result of this behavior.

Thanks, @Krinkle - That helps clarify. For noc.wikimedia.org, presence of the x-wikimedia-debug header affects caching behavior and response header cleanup, but has no role in backend selection at ATS. Instead, requests always route to mw-misc, which also happens to be updated in the testservers stage, so the "feedback loop" for testing merged code is sort of equivalent.

@Scott_French If I understand correctly, this means that:

So, the way this currently works is that mw-misc (home of noc.wikimedia.org) is updated during the testservers deployment stage, just like mw-debug. Meaning, hitting noc during that stage should observe the new changes immediately.

From the task description:

The last fn-EditAction flame graph was generated 27 March 2025:
https://performance.wikimedia.org/arclamp/svgs/daily/2025-03-27.excimer-k8s-php8.all.fn-EditAction.svgz

Mentioned in SAL (#wikimedia-operations) [2026-01-27T09:13:12Z] <cwhite@deploy2002> Finished deploy [performance/arc-lamp@03c538c]: T391517 (duration: 00m 08s)

Mentioned in SAL (#wikimedia-operations) [2026-01-27T09:13:04Z] <cwhite@deploy2002> Started deploy [performance/arc-lamp@03c538c]: T391517

Change #1231219 merged by jenkins-bot:

[performance/arc-lamp@master] Fix matching of MediaWiki EditAction stack frames

https://gerrit.wikimedia.org/r/1231219

Change #1231219 had a related patch set uploaded (by Ori; author: Ori):

[performance/arc-lamp@master] Fix matching of MediaWiki EditAction stack frames

https://gerrit.wikimedia.org/r/1231219

This is still broken and makes various performance analysis and improvements hard or impossible.

@A_smart_kitten thanks for tagging us.

CTT: tagging you as per decline note. This seems like a more general request than one that's specific to the now-decommissioned Mobile-Content-Service?

I cleaned up the residual instances manually on arclamp1001 and arclamp2001.

As part of the decommission of Mobile-Content-Service, this task is no longer valid. See T328036 for reference.
If you believe this is reproducible as part of Page Content Service nowadays used, please reach out to Content-Transform-Team. Thanks.

With the rollout of PHP 8.3... Especially in these very early stages, I'd love to be able to stick my activity onto the PHP 8.3 hosts, and frankly leave it there for the near term future.