[Epic] Replace current manual regression test with instrumented Espresso test.
Closed, ResolvedPublic2 Estimated Story Points
Actions

Assigned To

Authored By

	Dbrant
	Oct 30 2024, 8:04 PM

Description

Our current regression test, which we perform once per every release cycle, is performed manually by our QA staff.

This kind of test could potentially be automated by making an instrumented test that performs all the same steps programmatically, and reports if there are failures.

Let's work towards making an Espresso-driven test that aims to be 1:1 with our current regression/smoke test.

Related Objects
Search...

Status	Assigned	Task
Resolved	Dbrant	T358021 [XL] Refine and modularize Espresso tests
Resolved	WRai-WMF	T378649 [Epic] Replace current manual regression test with instrumented Espresso test.
Resolved	WRai-WMF	T378659 [XL] Regression test in Espresso: Editing
Resolved	WRai-WMF	T378658 [M] Regression test in Espresso: "More" menu
Resolved	WRai-WMF	T378656 [M] Regression test in Espresso: Settings
Resolved	WRai-WMF	T378655 [S] Regression test in Espresso: Suggested Edits
Resolved	WRai-WMF	T378654 Regression test in Espresso: Reading lists
Resolved	WRai-WMF	T378653 [S] Regression test in Espresso: Search
Resolved	WRai-WMF	T378652 [M] Regression test in Espresso: Article interactions
Resolved	WRai-WMF	T378651 [S] Regression test in Espresso: Explore Feed

Event Timeline

Dbrant created this task.Oct 30 2024, 8:04 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 30 2024, 8:04 PM

Dbrant triaged this task as Low priority.Oct 30 2024, 8:04 PM

Dbrant added a parent task: T358021: [XL] Refine and modularize Espresso tests.

Dbrant renamed this task from Replace current manual regression test with instrumented Espresso test. to [Epic] Replace current manual regression test with instrumented Espresso test..Oct 30 2024, 8:22 PM

Restricted Application added a project: Epic. · View Herald TranscriptOct 30 2024, 8:22 PM

Dbrant added subtasks: T378659: [XL] Regression test in Espresso: Editing, T378658: [M] Regression test in Espresso: "More" menu, T378656: [M] Regression test in Espresso: Settings, T378655: [S] Regression test in Espresso: Suggested Edits, T378654: Regression test in Espresso: Reading lists, T378653: [S] Regression test in Espresso: Search, T378652: [M] Regression test in Espresso: Article interactions, T378651: [S] Regression test in Espresso: Explore Feed.Oct 30 2024, 8:38 PM

@Dbrant is there a link to the manual regression test cases that one can use as reference for the automation?

Seddon closed subtask T378652: [M] Regression test in Espresso: Article interactions as Resolved.Feb 18 2025, 5:28 PM

Seddon closed subtask T378655: [S] Regression test in Espresso: Suggested Edits as Resolved.

Seddon closed subtask T378656: [M] Regression test in Espresso: Settings as Resolved.

Seddon closed subtask T378651: [S] Regression test in Espresso: Explore Feed as Resolved.

Seddon closed subtask T378653: [S] Regression test in Espresso: Search as Resolved.

Seddon moved this task from Tech Debt Backlog to Android Release - FY2024-25 on the Wikipedia-Android-App-Backlog board.Feb 19 2025, 3:38 PM

Seddon edited projects, added Wikipedia-Android-App-Backlog (Android Release - FY2024-25); removed Wikipedia-Android-App-Backlog.

Dbrant closed subtask T378658: [M] Regression test in Espresso: "More" menu as Resolved.Apr 14 2025, 8:11 PM

Dbrant closed subtask T378654: Regression test in Espresso: Reading lists as Resolved.

WRai-WMF closed subtask T378659: [XL] Regression test in Espresso: Editing as Resolved.Aug 6 2025, 7:33 PM

Dbrant moved this task from Android Release - FY2024-25 to Android Release - FY2025-26 on the Wikipedia-Android-App-Backlog board.Nov 24 2025, 6:25 PM

Dbrant edited projects, added Wikipedia-Android-App-Backlog (Android Release - FY2025-26); removed Wikipedia-Android-App-Backlog (Android Release - FY2024-25).

Dbrant moved this task from Incoming tasks from backlog to Ready for future sprints (P1) on the Wikipedia-Android-App-Backlog (Android Release - FY2025-26) board.Nov 25 2025, 5:53 PM

Dbrant set the point value for this task to 5.

WRai-WMF claimed this task.Nov 25 2025, 7:44 PM

WRai-WMF moved this task from Ready for future sprints (P1) to Doing on the Wikipedia-Android-App-Backlog (Android Release - FY2025-26) board.

Williamrai opened https://github.com/wikimedia/apps-android-wikipedia/pull/6150

Dbrant changed the point value for this task from 5 to 8.Dec 8 2025, 6:29 PM

Dbrant changed the point value for this task from 8 to 2.Dec 9 2025, 5:37 PM

Espresso tests can now run on GitHub Actions. At the moment, they need to be triggered manually by going to Actions in GitHub and selecting either the Android Smoke Test workflow or the Android Saved Article Offline Test workflow.
The Espresso Smoke Test only includes tests that don’t require login, since Captcha prompts appear during login and there’s no way for Espresso to complete them.
Tests that do require login must currently be run locally through Android Studio, using the LoggedInUserTestSuite file.

Nice to have: A Python script that runs the LoggedInUserTestSuite with a single terminal command. There are a few complexities around this, so it might take some time to sort out. For now, I’m keeping it as a nice-to-have since the tests can already be run through Android Studio.

WRai-WMF moved this task from Doing to Code Review on the Wikipedia-Android-App-Backlog (Android Release - FY2025-26) board.Dec 12 2025, 3:47 PM

dbrant closed https://github.com/wikimedia/apps-android-wikipedia/pull/6150

WRai-WMF moved this task from Code Review to Merged and Waiting on the Wikipedia-Android-App-Backlog (Android Release - FY2025-26) board.Jan 9 2026, 3:47 PM

cooltey moved this task from Merged and Waiting to QA signoff on the Wikipedia-Android-App-Backlog (Android Release - FY2025-26) board.Jan 12 2026, 8:19 PM