Improve e2e test flakiness
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Jakob_WMDE
	Oct 13 2023, 11:12 AM

Description

As we add more REST endpoints, the authorization e2e tests seem to become increasingly flaky.

Example (failures on two consecutive rechecks on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/965682 PS2):

12:44:54   1) Auth
12:44:54        Authorization
12:44:54          Protected entity page
12:44:54            Permission denied - PATCH /entities/items/{item_id}/descriptions:
12:44:54 
12:44:54       AssertionError: 
12:44:54 7Response body: { en: 'random-test-description-LR5rdLQSeH' }8
12:44:54 Invalid status: expected 200 to equal 403      
12:44:54       actual expected
12:44:54       
12:44:54       200    403

This is likely due to a race condition between the API call protecting the entity under test and the following request that's expected to result in the permission error.

Possible solution: Check the entity protection status prior to making the second request and wait a few milliseconds if it's not protected yet.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	REST: Enable retries for flaky blocked user e2e test	mediawiki/extensions/Wikibase	master	+4 -1
	REST: Add timeout and retry to flaky e2e test	mediawiki/extensions/Wikibase	master	+5 -4

Customize query in gerrit

Related Objects

Mentioned In: T349075: mediawiki-quibble-apitests fail for extension math since 17.10.23

Event Timeline

Jakob_WMDE created this task.Oct 13 2023, 11:12 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 13 2023, 11:12 AM

Jakob_WMDE updated the task description. (Show Details)Oct 13 2023, 11:13 AM

Michael subscribed.Oct 13 2023, 4:49 PM

WMDE-leszek subscribed.Oct 16 2023, 1:29 PM

Silvan_WMDE moved this task from Tech backlog to Sprint 6 on the Wikibase Reuse Team board.Oct 17 2023, 9:50 AM

Silvan_WMDE edited projects, added Wikibase Reuse Team (Sprint 6); removed Wikibase Reuse Team.

Lucas_Werkmeister_WMDE mentioned this in T349075: mediawiki-quibble-apitests fail for extension math since 17.10.23.Oct 17 2023, 2:38 PM

According to https://integration.wikimedia.org/zuul/ this affects several changes for wikibase thus raising the priority.

Change 966583 had a related patch set uploaded (by Ollie Shotton; author: Ollie Shotton):

[mediawiki/extensions/Wikibase@master] REST: Add timeout if entity isn't protected yet in e2e Auth test

https://gerrit.wikimedia.org/r/966583

gerritbot added a project: Patch-For-Review.Oct 17 2023, 5:15 PM

Ollie.Shotton_WMDE moved this task from To Do to Doing on the Wikibase Reuse Team (Sprint 6) board.Oct 17 2023, 5:16 PM

Ollie.Shotton_WMDE claimed this task.Oct 18 2023, 3:06 PM

Ollie.Shotton_WMDE moved this task from Doing to Peer Review on the Wikibase Reuse Team (Sprint 6) board.

Change 966583 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] REST: Add timeout and retry to flaky e2e test

https://gerrit.wikimedia.org/r/966583

Maintenance_bot removed a project: Patch-For-Review.Oct 19 2023, 12:11 PM

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.2; 2023-10-24).Oct 19 2023, 1:00 PM

Silvan_WMDE moved this task from Peer Review to Done on the Wikibase Reuse Team (Sprint 6) board.Oct 20 2023, 9:01 AM

I still see this problem. See for example https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Math/+/969484

@Physikerwelt if I see correctly in https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81-docker/13968/console, the failed end to end test was not Wikibase REST API's but mediawiki one

21:32:37 [0-1] Error in "Wikitext Editor.Closing editor (browser button)"

OK, sorry @WMDE-leszek, there seem to be several problems, lately.
What about this one?
From https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Math/+/969510 (this evening) ->
https://integration.wikimedia.org/ci/job/mediawiki-quibble-apitests-vendor-php74-docker/45726/console

00:07:57.622   1 failing
00:07:57.622 
00:07:57.622   1) Auth
00:07:57.626        Authorization
00:07:57.626          Blocked user
00:07:57.626            cannot edit if blocked:
00:07:57.627 
00:07:57.627       AssertionError: 
00:07:57.627 7Response body: {
00:07:57.627   message: 'Error: exception of type Wikibase\\Repo\\RestApi\\Infrastructure\\DataAccess\\Exceptions\\EntityUpdateFailed: <Error, collected 2 message(s) on the way, array value set>\n' +
00:07:57.627     '+----------+---------------------------+--------------------------------------+\n' +
00:07:57.627     '| error    | blockedtext               | [[User:Mindyx5bvjoTilT|‪Mindyx5bvjoT |\n' +
00:07:57.627     '|          |                           | ilT‬]] | testing | 127.0.0.1 | ‪Mind |\n' +
00:07:57.627     '|          |                           | yx5bvjoTilT‬ | 24 | infinite | ‪Mind |\n' +
00:07:57.627     '|          |                           | yx5bvjoTilT‬ | 15:59, 29 October 202 |\n' +
00:07:57.627     '|          |                           | 3                                    |\n' +
00:07:57.628     '| error    | permissionserrors         |                                      |\n' +
00:07:57.628     '+----------+---------------------------+--------------------------------------+\n',

Change 969750 had a related patch set uploaded (by Jakob; author: Jakob):

[mediawiki/extensions/Wikibase@master] REST: Enable retries for flaky blocked user e2e test

https://gerrit.wikimedia.org/r/969750

gerritbot added a project: Patch-For-Review.Oct 30 2023, 1:21 PM

Change 969750 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] REST: Enable retries for flaky blocked user e2e test

https://gerrit.wikimedia.org/r/969750

Maintenance_bot removed a project: Patch-For-Review.Oct 31 2023, 9:30 AM

ReleaseTaggerBot edited projects, added MW-1.42-notes (1.42.0-wmf.4; 2023-11-07); removed MW-1.42-notes (1.42.0-wmf.2; 2023-10-24).Oct 31 2023, 2:02 PM

Ifrahkhanyaree_WMDE closed this task as Resolved.Oct 31 2023, 2:56 PM

Ifrahkhanyaree_WMDE moved this task from Backlog to Done on the Wikibase REST API (WPP) board.Nov 27 2023, 8:41 AM