🐞 bug report

Affected Package

Not sure - probably something to do with testing or the CLI.

Is this a regression?

Not as far as I know.

Description

Currently, when running tests using ng test, the compile-time error-checking applied to components appears to be significantly laxer than what happens in other CLI workflows such as ng serve and ng build.

I was able to find several ways to write an Angular template incorrectly, which result in (expected) compile errors when you run ng serve or ng build, but when you run ng test, tests which compile these components and make assertions about the resulting DOM succeed without any problems at all.

I found it very surprising that tests of this type would succeed when the components completely fail to compile in other contexts. At first I thought I must be doing something very obvious wrong, but I was able to reproduce the problem with a very simple component using the out-of-the-box configuration generated by ng new --strict at the latest version.

I searched online, and I was surprised to be unable to find any references to this problem. I did find a couple of similar-looking issues where errors in a template did not result in the test failures one would expect, but these are for non-existent HTML elements:

angular/angular#36171
angular/angular#36430

So either this is a different but similar-looking problem, or these previously-reported problems are broader than these existing issues suggest. I tried setting enableIvy = false in my repro (see below), and it made no difference, so this problem does not appear to be Ivy-specific; since the issues above do appear to be Ivy-specific, I suspect this is a different problem.

Either way, such significant discrepancies between test and non-test behaviours raise concerns about the usefulness of Angular tests for ensuring the code quality of components.

🔬 Minimal Reproduction

I created the following GitHub repository to reproduce this issue:

https://github.com/amdw/templates-test-repro

There are two branches, master and nonstrict. In both cases, ng serve fails with a compile error, but ng test succeeds, despite the fact that the tests compile the broken MyComponent and make assertions about the contents of its DOM.

I would expect both commands to fail with a compile error - or at the very least, this behavioural discrepancy should be prominently mentioned in testing-related documentation such as this, with a description of how tests can be configured to have the same compilation behaviour as these other workflows.

The master branch just uses the default compiler settings generated by ng new --strict. MyComponent has an incorrect HTML template: myinput can be undefined according to the component type signature, but one of its fields is being unconditionally dereferenced. Running ng serve or ng build results in a compile error, as expected:

ERROR in src/app/mycomponent/mycomponent.component.html:7:27 - error TS2532: Object is possibly 'undefined'.

However ng test results in 100% success, with no mention in the logs that there is any problem at all.

Initially I thought this might be a problem with the new-ish strictTemplates settings, since that is the context in which I discovered the problem (I was trying to convert an existing application to use strictTemplates; I got to a stage where my tests were 100% green, and thought I must be nearly done, but when I ran ng serve I got a flood of errors caused by type-checking failures in my HTML templates).

However, I did a little more experimentation and found that is not the case. The nonstrict branch switches strictTemplates and strictInjectionParameters to false, and introduces another template error, referencing a component attribute which does not even exist. Once again, ng build and ng serve result in the expected failures:

src/app/mycomponent/mycomponent.component.html:6:39 - error TS2339: Property 'nonexistent' does not exist on type 'MycomponentComponent'.

However, once again, ng test results in 100% success.

🔥 Exception or Error

N/A - the problem is that tests pass when I believe they shouldn't.

🌍 Your Environment

Angular Version:

Angular CLI: 10.0.1
Node: 12.16.3
OS: linux x64

Angular: 10.0.2
... animations, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, router
Ivy Workspace: Yes

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.1000.1
@angular-devkit/build-angular     0.1000.1
@angular-devkit/build-optimizer   0.1000.1
@angular-devkit/build-webpack     0.1000.1
@angular-devkit/core              10.0.1
@angular-devkit/schematics        10.0.1
@angular/cli                      10.0.1
@ngtools/webpack                  10.0.1
@schematics/angular               10.0.1
@schematics/update                0.1000.1
rxjs                              6.5.5
typescript                        3.9.6
webpack                           4.43.0

Anything else relevant?

I'm not aware of anything. I've only tried this on Fedora Workstation 32, but I find it hard to believe this would be OS or browser specific.