run e2e tests in isolated child processes by jbedard · Pull Request #23245 · angular/angular-cli

jbedard · 2022-05-28T08:11:25Z

~~Note I'd like to merge #23130 first.~~ DONE

This is to stop mixing the angular-cli test-runner environment with the cli environments being tested. Essentially launching each test in a subprocess to a) not mix with the test-runner PATH, npm modules etc, b) prevent environment (vars, cwd, ...) side effects from one test effecting the next tests or the test-runner.

Prefactor commits (can be merged on their own ahead of time instead of squashing):

Wait for killAllProcesses to actually complete before proceeding. Killing a procedure is async but was not being await-ed. This creates race conditions but once tests are run in a subprocess those are far more likely (the subprocess dies before killing subprocesses completes). EDIT: was done in f70557a
Refactor the main e2e_runner.ts into separate methods, async/await instead of .then.
Distinguish test setup vs tests better (in preparation for tests being launched in subprocesses, also distinguishes setup vs test better in the stdout).

Main change:

A specific version of npm + yarn are installed alongside the angular-cli npm package being tested. This will be required in bazel where nothing outside the sandbox (global node/npm/yarn) is accessible. This also sets specific version numbers which could be dynamic in the future.
Tests run in a sandboxed subprocess environment using those specific npm + yarn + cli packages.
Tests no longer have access to the node_modules or environment of the test-runner, only the modules installed within that sandboxed environment. So things like puppeteer now need to be installed within the sandboxed test environment (previously it was using the one in the git repo node_modules). The protractor used in the tests is now the correct one in the test project node_modules (not the one the cli git repo node_modules), so webdriver-manager must now run on that test project and not only the git repo.

Overview:

launch "setup" in the main process to setup the test environment (no real changes from before)
launch each "initializer" in subprocesses (some of the previous "setup" scripts are now "initializers" which run in the subprocess, but no big changes)
launch each test in the subprocesses

gregmagolan · 2022-05-30T16:52:47Z

+    for (const [initializerIndex, initializer] of allInitializers.entries()) {
+      printSetupHeader(initializer, initializerIndex, allInitializers.length, 'initializer');
+
+      await runInitializer((lastTestRun = initializer));


nit: comment that runInitializer runs ng init and creates test-project

clydin · 2022-06-07T16:35:37Z

This appears to be causing failures for the full Windows CI job and the snapshots job:
https://app.circleci.com/pipelines/github/angular/angular-cli/23169/workflows/2aa69b28-122e-468f-9220-f7a36f6ae56c

jbedard · 2022-06-07T17:05:38Z

Error: Running "cmd.exe /c git clean -dxf" returned error. 1...
warning: failed to remove .angular/cache/14.1.0-next.0/ng-packagr/content-v2/sha512/03/ce/bb4940cc77a94bd0f18e4752d30a7c2895fb086c8b48ae6258a7f2628e0c348bd1e42161f20f7efcbd9b6ff60dda64a1c9710bc3cb530e2831201a1e63d2: Filename too long
warning: failed to remove .angular/cache/14.1.0-next.0/ng-packagr/content-v2/sha512/18/eb/2ce695e9c32252f3fde4a56bcabc8bc319ed487f0afbc5322d41c5c8c9bee6079f160ca19345a84baa8a46e9339d1c1fae76e00465bebf52bf13434791db: Filename too long
warning: failed to remove .angular/cache/14.1.0-next.0/ng-packagr/content-v2/sha512/41/50/af2048670b4a59c036628768bde838d380c8e46d771b0f0893b8ba0af3d05b0de0ec1606973bb4bd8672cd6751d1db6e39d3671240d45c409b844fddae41: Filename too long
...

Any ideas here? Should that folder be gitignored and not be cleaned anyway?

jbedard · 2022-06-07T17:07:15Z

I'll look into the snapshot ones more this morning.

What is run which wasn't run on the PR CI?

clydin · 2022-06-07T17:27:15Z

#23311 should fix the snapshot job. Windows will require more investigation though.

These are the jobs that are only run on main (Windows has a subset run on PRs):

e2e-cli-node-16

angular-cli/.circleci/config.yml

Line 373 in 9c26e28

<<: *only_release_branches

e2e-cli-ng-snapshots

angular-cli/.circleci/config.yml

Lines 365 to 369 in 9c26e28

    
           filters: 
        
             branches: 
        
               only: 
        
                 - renovate/angular 
        
                 - main

e2e-cli-win

angular-cli/.circleci/config.yml

Lines 328 to 332 in 9c26e28

    
                       if (Test-Path env:CIRCLE_PULL_REQUEST) { 
        
                         node tests\legacy-cli\run_e2e.js "--glob={tests/basic/**,tests/i18n/extract-ivy*.ts,tests/build/profile.ts,tests/test/test-sourcemap.ts,tests/misc/check-postinstalls.ts}" --nb-shards=$env:CIRCLE_NODE_TOTAL --shard=$env:CIRCLE_NODE_INDEX 
        
                       } else { 
        
                         node tests\legacy-cli\run_e2e.js --nb-shards=$env:CIRCLE_NODE_TOTAL --shard=$env:CIRCLE_NODE_INDEX 
        
                       }

alan-agius4 · 2022-06-07T17:38:23Z

I briefly took a look from my mobile, what is strange is that the files not being removed are ng-packagr cache files, which strangely isn’t even run in the failings tests. Have these been left overs from previous tests?

I also noticed that some other tests are failing with task kill related errors

Error: Command failed: taskkill /pid 1892 /T /F
ERROR: The process "1892" not found.

angular-automatic-lock-bot · 2022-07-08T08:04:14Z

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}

jbedard force-pushed the e2e-local-npm branch 7 times, most recently from 790a1a9 to bd671ea Compare May 30, 2022 09:51

gregmagolan reviewed May 30, 2022

View reviewed changes

Comment thread tests/legacy-cli/e2e_runner.ts Outdated

gregmagolan reviewed May 30, 2022

View reviewed changes

Comment thread tests/legacy-cli/e2e_runner.ts Outdated

gregmagolan reviewed May 30, 2022

View reviewed changes

Comment thread tests/legacy-cli/e2e/utils/project.ts Outdated

gregmagolan reviewed May 30, 2022

View reviewed changes

Comment thread tests/legacy-cli/e2e/utils/test_process.ts Outdated

jbedard force-pushed the e2e-local-npm branch 4 times, most recently from b527198 to 4558799 Compare May 31, 2022 01:37

jbedard marked this pull request as ready for review May 31, 2022 01:37

jbedard requested review from alan-agius4 and clydin May 31, 2022 01:38

jbedard force-pushed the e2e-local-npm branch 6 times, most recently from 89cb1d1 to e0f1c5c Compare May 31, 2022 18:32

jbedard mentioned this pull request May 31, 2022

run e2e legacy tests under bazel #23074

Merged

jbedard force-pushed the e2e-local-npm branch 4 times, most recently from 8742c1d to 60f0ec9 Compare June 1, 2022 19:59

jbedard force-pushed the e2e-local-npm branch 3 times, most recently from 1424022 to 317f948 Compare June 2, 2022 21:05

alan-agius4 approved these changes Jun 3, 2022

View reviewed changes

alan-agius4 reviewed Jun 3, 2022

View reviewed changes

Comment thread tests/legacy-cli/e2e/utils/project.ts Outdated

alan-agius4 reviewed Jun 3, 2022

View reviewed changes

Comment thread tests/legacy-cli/e2e/utils/project.ts Outdated

alan-agius4 added the target: minor This PR is targeted for the next minor release label Jun 3, 2022

jbedard mentioned this pull request Jun 3, 2022

test: fix ts compiler errors #23130

Merged

jbedard force-pushed the e2e-local-npm branch from 317f948 to 020d97d Compare June 3, 2022 23:14

jbedard requested a review from alan-agius4 June 3, 2022 23:45

jbedard force-pushed the e2e-local-npm branch 3 times, most recently from db25a5b to d0548d6 Compare June 6, 2022 20:12

jbedard added 2 commits June 6, 2022 13:23

test: extract test runner method

26f4d66

test: separate test vs test setup execution

a96b2c7

jbedard force-pushed the e2e-local-npm branch from d0548d6 to 4326d07 Compare June 6, 2022 20:34

test: run tests in isolated subprocess

3777a57

jbedard force-pushed the e2e-local-npm branch from 4326d07 to 3777a57 Compare June 6, 2022 20:36

jbedard mentioned this pull request Jun 7, 2022

test: reduce use of npmrc for test npm configuration #23299

Merged

clydin approved these changes Jun 7, 2022

View reviewed changes

alan-agius4 approved these changes Jun 7, 2022

View reviewed changes

alan-agius4 added the action: merge The PR is ready for merge by the caretaker label Jun 7, 2022

clydin merged commit 9c26e28 into angular:main Jun 7, 2022

angular-automatic-lock-bot Bot locked and limited conversation to collaborators Jul 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

run e2e tests in isolated child processes#23245

run e2e tests in isolated child processes#23245
clydin merged 3 commits into
angular:mainfrom
jbedard:e2e-local-npm

jbedard commented May 28, 2022 •

edited

Loading

Uh oh!

gregmagolan May 30, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clydin commented Jun 7, 2022

Uh oh!

jbedard commented Jun 7, 2022

Uh oh!

jbedard commented Jun 7, 2022

Uh oh!

clydin commented Jun 7, 2022

Uh oh!

alan-agius4 commented Jun 7, 2022 •

edited

Loading

Uh oh!

angular-automatic-lock-bot Bot commented Jul 8, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

jbedard commented May 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gregmagolan May 30, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clydin commented Jun 7, 2022

Uh oh!

jbedard commented Jun 7, 2022

Uh oh!

jbedard commented Jun 7, 2022

Uh oh!

clydin commented Jun 7, 2022

Uh oh!

alan-agius4 commented Jun 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

angular-automatic-lock-bot Bot commented Jul 8, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jbedard commented May 28, 2022 •

edited

Loading

alan-agius4 commented Jun 7, 2022 •

edited

Loading