Summary
Update the current Revise Tone Test Kitchen experiment configuration and restart the experiment with revised task availability and sorting rules. The goal is to isolate the impact of the Revise Tone task by adjusting which Suggested Edits are available to treatment and control groups.
Context:
We are seeing newcomers complete Revise Tone tasks successfully, and the revert rate is quite low (T408642: Product Analytics: Revise Tone - Leading Indicator plan of action).
However, early data suggests that newcomers are still most successful with the Add a Link task, which is the easiest Suggested Edit available. When Revise Tone is sorted first for the treatment group, newcomers are less likely to encounter Add a Link tasks. As a result, very new account holders in the treatment group may be less likely to constructively activate compared with the control group, which is much more likely to surface Add a Link tasks.
This change is intended to better isolate the impact of the Revise Tone task on newcomer editing behavior by removing the sorting bias and adjusting which tasks are available to each experiment group.
User Story
As a Growth Product Manager, I want to run an experiment where Revise Tone is available to the treatment group without being artificially prioritized in sorting, so that we can measure the effect of the task itself on constructive editing behavior.
As a new editor looking for Suggested Edits, I want to see a set of tasks that I can easily complete so that I can contribute to Wikipedia and improve articles.
Changes requested
Remove sorting override
- Remove the change that currently surfaces Revise Tone first for all treatment group participants.
- Suggested Edits should return to the standard sorting behavior (for both control and treatment groups)
Update experiment groups
Treatment group
- Available tasks:
- Revise Tone
- Add a Link
- Not available:
- Copyedit
Control group
- Maintain the current status quo task set:
- Add a Link
- Copyedit
- Not available:
- Revise Tone
Restart the Test Kitchen experiment
Release date: March 19, 2026
End date: June 30, 2026
Acceptance criteria
- The sorting override that prioritizes Revise Tone is removed.
- Treatment participants can access Revise Tone and Add a Link, but Copyedit is not available.
- Control participants retain access to Add a Link and Copyedit, but Revise Tone is not available.
- The Test Kitchen experiment is restarted with the new configuration.
- The copyedit task is instrumented so that we can measure task completion
Experiment start and end dates are set to March 19, 2026 – June 30, 2026.
