Experiment Platform Sprint for June 25 - July 16 2026.
Details
Yesterday
Thu, Jul 16
Because of the issue we have been dealing with via T431015: GrowthBook staging sluggish and throws authentication errors, the Checklist autocompletion feature will be fully disabled for now. We have found that the number of requests done by that feature (about 20 req/minute) was creating a high number of concurrent jobs that were competing for same resources (to access and modify some structures) and that has been causing a excessive high CPU load in the GrowthBook's database.
Change #1311414 merged by jenkins-bot:
[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.4.9 release to production
Change #1311413 merged by jenkins-bot:
[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.4.9 release to staging
Change #1311414 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):
[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.4.9 release to production
Change #1311413 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):
[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.4.8 release to staging
Disable Checklist completion
After reviewing both proposals based on the great comparison table here, some numbers there and some other considerations, a decision was made a we will implement the approach based on UI:
- The numbers showed that the implementation effort was lower for the MR-based on
- The numbers also showed that there would be a constant additional effort in the MR-based one because every exception would mean a deployment. And that could be a blocker in some cases. An unrelated deployment that is being delayed for any reason could delay an exception that needs to be applied
- We have also considered that the additional effort in implementing the UI can be an investment for the future: No additional constant effort will be needed (no deployments) to register and activate a traffic limit exception
- Also the new UI can fit with the new lading page we are considering for test-kitchen.wikimedia.org
Wed, Jul 15
From our side, Experiment Platform team, we have already disabled the feature that caused this issue in both staging and production environments.
I have stopped the A/A test.
We ran this A/A test to check whether there's something special about certain wikis (Wikidata, Meta-Wiki, Wikimedia Commons, and MediaWiki) and Test Kitchen SDKs that might have contributed to the SRM issue we saw in PSI's Email Confirmation Banner A/B test. After running this A/A test for about 3 weeks we do not see any SRM on those wikis, which has improved our confidence in the platform.
Change #1298112 merged by jenkins-bot:
[mediawiki/extensions/TestKitchen@master] TestKitchen: Capture experiment version in analytics events