run e2e legacy tests under bazel#23074
Conversation
jbedard
force-pushed
the
legacy-test
branch
30 times, most recently
from
1cfd471 to
52f1c3b
Compare
jbedard
force-pushed
the
legacy-test
branch
6 times, most recently
from
84a6830 to
b015e45
Compare
|
Couple of questions
Both of these are paramount for day to day tasks. |
| # Same as https://circleci.com/orbs/registry/orb/circleci/windows, but named. | ||
| working_directory: ~/ng | ||
| resource_class: windows.medium | ||
| resource_class: windows.2xlarge |
There was a problem hiding this comment.
This will increase the credit usage drastically 500 credits/min compared to the current 40 credits/min
https://circleci.com/product/features/resource-classes/#dev-windows
There was a problem hiding this comment.
I'll try decreasing it again. This was partially to understand why things were flaky or failing which I think was from running 8+ tests concurrently on a single machine. Now it's running 6 in bazel spread over 4 CI shards (the parallelism param).
| docker: | ||
| - image: cimg/node:<< parameters.nodeversion >> | ||
| working_directory: ~/ng | ||
| resource_class: 2xlarge |
There was a problem hiding this comment.
Do we really need to use 2xlarge? None the less likely the default should be large at most.
There was a problem hiding this comment.
I'll try reducing it. All the resource changes was from me playing around with resources/bazel sharding/CI parallelism
| default: 'npm' | ||
| steps: | ||
| - custom_attach_workspace | ||
| - browser-tools/install-chrome |
There was a problem hiding this comment.
Why is chrome needed?
There was a problem hiding this comment.
Without it I'm getting https://app.circleci.com/pipelines/github/angular/angular-cli/26617/workflows/8a85cca7-f1fb-4d5a-8698-83e522f485d4/jobs/352420/steps?invite=true#step-106-1703
... which might just required propper config of chrome within bazel. I can look into it more. It would be great to drop this step which requires 20-30s of CI time.
There was a problem hiding this comment.
This is also used in the unit tests (bazel) and non-bazel e2e tests, so maybe it should stay?
There was a problem hiding this comment.
As I understand this it should not be necessary because we provide the browsers via bazel
|
|
||
| - Run: `node tests/legacy-cli/run_e2e.js` | ||
| - Run a subset of the tests: `node tests/legacy-cli/run_e2e.js tests/legacy-cli/e2e/tests/i18n/ivy-localize-*` | ||
| - Run on a custom set of npm packages (tar files): `node tests/legacy-cli/run_e2e.js --package _angular_cli.tgz _angular_create.tgz dist/*.tgz ...` |
There was a problem hiding this comment.
Would be helpful to write the equivalent of node tests/legacy-cli/run_e2e.js tests/legacy-cli/e2e/tests/i18n/ivy-localize-*.
| var_7_subsets: &all_e2e_subsets ['npm', 'esbuild', 'yarn'] | ||
| var_7_build_types: &all_e2e_build_types ['e2e', 'snapshot'] |
There was a problem hiding this comment.
| var_7_subsets: &all_e2e_subsets ['npm', 'esbuild', 'yarn'] | |
| var_7_build_types: &all_e2e_build_types ['e2e', 'snapshot'] | |
| var_7: &all_e2e_subsets ['npm', 'esbuild', 'yarn'] | |
| var_8: &all_e2e_build_types ['e2e', 'snapshot'] |
Since the key in this pair isn't actually used by yaml, we typical just count rather than providing any "context" in the naming scheme.
| default: 'npm' | ||
| steps: | ||
| - custom_attach_workspace | ||
| - browser-tools/install-chrome |
There was a problem hiding this comment.
As I understand this it should not be necessary because we provide the browsers via bazel
|
|
||
| # Files required by e2e tests | ||
| copy_to_bin( | ||
| name = "dot-files", |
There was a problem hiding this comment.
This is not a dot file right? Were others included at one point?
There was a problem hiding this comment.
I assume it was at one point and has evolved over time. I'll change it 👍
josephperrott
left a comment
josephperrott
left a comment
There was a problem hiding this comment.
LGTM though still other comments that appear to be unaddressed.
|
I think those were all addressed offline and not in the comments here, just no one had pressed resolve. |
alan-agius4
left a comment
alan-agius4
left a comment
There was a problem hiding this comment.
One comment otherwise LGTM. Can you also squash please?
Using execPath ensures the child processes are the same node binary without depending on PATH.
fixup: add ChromeHeadlessNoSandbox browser to to karma builder
|
This issue has been automatically locked due to inactivity. Read more about our automatic conversation locking policy. This action has been performed automatically by a bot. |
3 participants
This runs the legacy-cli/e2e tests via bazel using the bazel built npm packages from
packages/**.After all the pre-factors it's mainly just build config and a few code changes to use the bazel environment vars if present. This can be simplified if the non-bazel versions are dropped in the future. Currently all the CI jobs are prefixed with
bazel-.Build targets / API:
There are 9 bazel targets invoking the e2e tests with different configurations:
//tests/legacy-cli:e2e.{npm,yarn,esbuild,saucelabs}: configured like CI today//tests/legacy-cli:e2e.snapshot.{npm,yarn,esbuild,saucelabs}: ^ using snapshots//tests/legacy-cli:e2e: nothing pre-configured, does not run by default or on CI, is designed to be run and configured manually with bazel cli args.With any of these you can:
bazel testto run as a bazel testbazel runto do the equivalent of--debugand remain running on failure--test_filters="..."which essentially does the e2e_runner--glob="..."--test_arg="..."to specify additional e2e_runner args,--globwould work here tooSharding
Today e2e tests have
--shardand--nb-shardscli params which circleci sets based on theparallelismconfig.Now those cli args are defaulted based on the bazel environment variables set by
nodejs_test(shard_count)and/or theE2E_SHARD_TOTAL+E2E_SHARDenvironment variables I added. This way both bazel and circleci sharding can be combined into one (bazel shards via processes on one circleci environment, circleci sharding in isolated environments).Prefactors: