Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upci: do not run benchmark measurements in circleci #34753
Closed
Conversation
devversion
force-pushed the
devversion:build/split-benchmark-and-e2e-tests
branch
3 times, most recently
from
a76e07c
to
45060b3
devversion
requested review from
IgorMinar and
angular/dev-infra-framework
as
code owners
devversion
force-pushed the
devversion:build/split-benchmark-and-e2e-tests
branch
from
45060b3
to
0cbc083
|
thanks @devversion! I found a few things that I think we could improve, please take a look at the individual comments. |
devversion
force-pushed the
devversion:build/split-benchmark-and-e2e-tests
branch
2 times, most recently
from
bb11dee
to
d838cac
|
@IgorMinar Thanks for reviewing! I addressed your feedback. |
devversion
force-pushed the
devversion:build/split-benchmark-and-e2e-tests
branch
from
d838cac
to
3a6db85
|
thank you! |
IgorMinar
added
PR action: merge
PR action: cleanup
and removed
PR action: review
labels
|
this PR fails in g3 in a catastrophic way with build errors due to internal latency lab tests. we'll need to investigate this and fix it next week. |
|
I don't have enough insight, but I'd guess it's because we either changed the file names of the spec files, or since we updated the test library target names (depending on how it is set up in g3) |
|
@devversion The change in file names caused the files to be included in a |
|
I started a new g3 presubmit. @devversion could you plz rebase your PR on top of the latest master, so we rerun CI with the most up-to-date version? Thank you. |
Currently we run all benchmark perf tests in CircleCI. Since we do not collect any results, we unnecessarily waste CI/RBE resources. Instead, we should just not run benchmark perf tests in CI, but still run the functionality e2e tests which ensure that benchmarks are not broken. We can do this by splitting the perf and e2e tests into separate files/targets.
Currently, based on the file names it's not quite clear whether a given `.spec.ts` file runs benchmark perf or benchmark e2e functionality tests. To disambiguate these, we use new file suffixs. i.e. `e2e-spec.ts` and `perf-spec.ts`.
Previously, when the benchmark tests ran outside of Bazel, developers had the posibility to control how the tests run through command line options. e.g. `--dryrun`. This no longer works reliable in Bazel where command line arguments are not passed to the text executable. To make the global options still usable (as they could be still useful in some cases), we just pass them through the Bazel `--test_env`. This reduces the code we need to read the command line, but still preserves the flexibility in a Bazel idiomatic way.
|
@AndrewKushnir Rebased. |
devversion
force-pushed the
devversion:build/split-benchmark-and-e2e-tests
branch
from
3a6db85
to
6281d78
AndrewKushnir
added a commit
that referenced
this pull request
Jan 29, 2020
AndrewKushnir
added a commit
that referenced
this pull request
Jan 29, 2020
…34753) Previously, when the benchmark tests ran outside of Bazel, developers had the posibility to control how the tests run through command line options. e.g. `--dryrun`. This no longer works reliable in Bazel where command line arguments are not passed to the text executable. To make the global options still usable (as they could be still useful in some cases), we just pass them through the Bazel `--test_env`. This reduces the code we need to read the command line, but still preserves the flexibility in a Bazel idiomatic way. PR Close #34753
AndrewKushnir
added a commit
that referenced
this pull request
Jan 29, 2020
Currently we run all benchmark perf tests in CircleCI. Since we do not collect any results, we unnecessarily waste CI/RBE resources. Instead, we should just not run benchmark perf tests in CI, but still run the functionality e2e tests which ensure that benchmarks are not broken. We can do this by splitting the perf and e2e tests into separate files/targets. PR Close #34753
AndrewKushnir
added a commit
that referenced
this pull request
Jan 29, 2020
AndrewKushnir
added a commit
that referenced
this pull request
Jan 29, 2020
…34753) Previously, when the benchmark tests ran outside of Bazel, developers had the posibility to control how the tests run through command line options. e.g. `--dryrun`. This no longer works reliable in Bazel where command line arguments are not passed to the text executable. To make the global options still usable (as they could be still useful in some cases), we just pass them through the Bazel `--test_env`. This reduces the code we need to read the command line, but still preserves the flexibility in a Bazel idiomatic way. PR Close #34753
sonukapoor
added a commit
to sonukapoor/angular
that referenced
this pull request
Feb 13, 2020
Currently we run all benchmark perf tests in CircleCI. Since we do not collect any results, we unnecessarily waste CI/RBE resources. Instead, we should just not run benchmark perf tests in CI, but still run the functionality e2e tests which ensure that benchmarks are not broken. We can do this by splitting the perf and e2e tests into separate files/targets. PR Close angular#34753
sonukapoor
added a commit
to sonukapoor/angular
that referenced
this pull request
Feb 13, 2020
Currently, based on the file names it's not quite clear whether a given `.spec.ts` file runs benchmark perf or benchmark e2e functionality tests. To disambiguate these, we use new file suffixs. i.e. `e2e-spec.ts` and `perf-spec.ts`. PR Close angular#34753
sonukapoor
added a commit
to sonukapoor/angular
that referenced
this pull request
Feb 13, 2020
…ngular#34753) Previously, when the benchmark tests ran outside of Bazel, developers had the posibility to control how the tests run through command line options. e.g. `--dryrun`. This no longer works reliable in Bazel where command line arguments are not passed to the text executable. To make the global options still usable (as they could be still useful in some cases), we just pass them through the Bazel `--test_env`. This reduces the code we need to read the command line, but still preserves the flexibility in a Bazel idiomatic way. PR Close angular#34753
angular-automatic-lock-bot
bot
commented
Feb 29, 2020
|
This issue has been automatically locked due to inactivity. Read more about our automatic conversation locking policy. This action has been performed automatically by a bot. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
devversion commentedJan 13, 2020
Currently we run all benchmark perf tests in CircleCI. Since we do not
collect any results, we unnecessarily waste CI/RBE resources. Instead,
we should just not run benchmark perf tests in CI, but still run the
functionality e2e tests which ensure that benchmarks are not broken.
We can do this by splitting the perf and e2e tests into separate
files/targets.