User Details
- User Since
- Feb 6 2023, 4:18 PM (179 w, 4 d)
- Availability
- Available
- IRC Nick
- sfaci
- LDAP User
- Santiago Faci
- MediaWiki User
- SFaci-WMF [ Global Accounts ]
Thu, Jul 16
Because of the issue we have been dealing with via T431015: GrowthBook staging sluggish and throws authentication errors, the Checklist autocompletion feature will be fully disabled for now. We have found that the number of requests done by that feature (about 20 req/minute) was creating a high number of concurrent jobs that were competing for same resources (to access and modify some structures) and that has been causing a excessive high CPU load in the GrowthBook's database.
After reviewing both proposals based on the great comparison table here, some numbers there and some other considerations, a decision was made a we will implement the approach based on UI:
- The numbers showed that the implementation effort was lower for the MR-based on
- The numbers also showed that there would be a constant additional effort in the MR-based one because every exception would mean a deployment. And that could be a blocker in some cases. An unrelated deployment that is being delayed for any reason could delay an exception that needs to be applied
- We have also considered that the additional effort in implementing the UI can be an investment for the future: No additional constant effort will be needed (no deployments) to register and activate a traffic limit exception
- Also the new UI can fit with the new lading page we are considering for test-kitchen.wikimedia.org
Wed, Jul 15
From our side, Experiment Platform team, we have already disabled the feature that caused this issue in both staging and production environments.
@Sfaci Is that ok if I re-assign the task to you now, to figure out how to avoid creating these events in the first place?
Tue, Jul 14
Fri, Jul 10
As part of this work, and considering that test-kitchen on staging can be also posting comments in Phabricator, the Phabricator Task Custom Field has been set as optional to let who is testing there leave that field empty and avoid flooding phabricator tickets with comments while testing something unrelated to this alerting system
Thu, Jul 9
We have realized that the experiment was going to start running on a Friday which is not recommended (no deployments on Friday), so the start date have been changed and the experiment will start next Monday: 2026-07-13
Your experiment The perfect experiment failed validation and its status has been reverted to draft in GrowthBook.
Your experiment The perfect experiment failed validation and its status has been reverted to draft in GrowthBook.
Wed, Jul 8
- The experiment traffic has been manually updated in the backend side (50%( and is active already in TKUI.
- Regarding Once experiment is active in TK UI, switch experiment config providers from GB to TK UI by tagging the experiment (in GB) with external imported tag, because the experiment in GB is scheduled to start before the TKUI one have actually started, and is already tagged as external imported, we can assume that everything will be fine regarding TKUI prevailing over GB. Now TKUI is the source and, once the GB experiment start, TKUI will keep being the source for two reasons: the machine-readable name is the same (TKUI prevails in this case by default) and also because the experiment in GB is tagged already.
https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/127 was closed because https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/128, which has been merged already, was updated to include also phase_index as a new field for base schemas
Tue, Jul 7
Side note: if GB did have a mechanism to support restricting a tag so that only certain users could add it, I'd suggest we use the tags in GB as the source of truth, and add a check for those tags to the conversion layer logic in TK, and do the increased limit there. Without that ability to limit a tag to only admins or users with a specific permission for that tag, I don't think the approach works.