Introducing type-checking #2749

rogiervd · 2024-11-06T10:12:34Z

rogiervd
Nov 6, 2024

I would like to propose introducing type checking as an additional check in CI and possibly a pre-commit hook. I would propose leaving type annotations optional, but reviewers can encourage this where it makes sense.

Type checking provides an extra check for correctness. Type checking also encourages type annotations, which (1) are documentation for users and (2) can be used by IDEs to give hints.

Four type checkers are in general use for Python.
Three use a similar paradigm:

mypy is the OG type checker. It requires type annotations where it gets confused.
pyright (Microsoft) presents itself as the fast and updated mypy. It is used by default by VS Code.
pyre (Meta) is faster, and also watches the working directory in the background.

One type checker is different:

pytype (Google) is more lenient but unlike the other checkers does type inference, which makes it much slower.

Personally, I like pytype, but on my indicative test it takes four minutes to run, which may become a pain.
I don't think you'd want to put pytype in a pre-commit check, whereas pyright takes 30 seconds and pyre 10 seconds.
It may be possible to run multiple type checkers to check for the union of the problems they detect.

I imagine the process of adding type checks would involve quite a few PRs over time:

Put in a PR for adding type checking to the CI. This would fail with hundreds of errors and not be merged initially.
Put in multiple PRs to fix various type errors.
Keep rebasing the PR in (1) until it passes.

I am volunteering to drive this, though I would need some help where I don't understand the code well enough.

A few types of changes that would be required to make type checking pass, judging from running a few type checkers over the current tree:

Fixing type annotations in function signatures. In a few places current type annotations are wrong or have incorrect syntax.
Add some type annotations to make the type checker happy.
Fix bugs in a few places.
Add annotations like # type: ignore to import statements for packages that are not in requirements.txt, like matplotlib.
Add annotations like # type: ignore to code that edits attributes of objects at run time. E.g. the Brain class, which is particularly dynamic.
Add assertions like assert argument_name is not None in places where this assumption is currently implicit.
(Possibly, depending on which type checker) Change __init__.py files if necessary, maybe adding __all__ = [...].
Fix object-oriented patterns where they are not strictly kept, for example overrides which change the method signature or base classes that don't strictly declare all the methods that are used elsewhere in the code.
It may be worth updating packages like Numpy to versions that have type annotations. Numpy>=1.20 has numpy.typing, so then ArrayLike can be used.

In the long term, I would envision these changes:

Coders and reviewers will feel the usefulness of type annotations and start using and requesting them more. This will help humans, IDEs, and the type checker understand the code better.
Existing type annotations move from docstrings to the function signature.
Code that edits attributes of objects at run time will require so many # type: ignore that it will have a harder time getting through code review.

TParcollet · 2024-11-06T10:13:46Z

TParcollet
Nov 6, 2024
Maintainer

@pplantinga @mravanelli @asumagic @Adel-Moumen inputs may be welcome here.

0 replies

TParcollet · 2024-11-06T10:50:18Z

TParcollet
Nov 6, 2024
Maintainer

Thanks @rogiervd for this, I think we will all agree that this is important. A few comments:

I think that speed should be prioritized over number of checks up to a certain threshold.
"Add annotations like # type: ignore to import statements for packages that are not in requirements.txt, like matplotlib." - This is actually more of an error than something else, if matplotlib is in the speechbrain library (not in a recipe) it should be in the requirement. If things like this happen, I propose listing them and discussing them in a proper issue. I don't even see what we would have matplotlib in the SB lib.
"Fix object-oriented patterns where they are not strictly kept, for example overrides which change the method signature or base classes that don't strictly declare all the methods that are used elsewhere in the code." -- this should definitely be a sperate PR so we can discuss it.
"It may be worth updating packages like Numpy to versions that have type annotations. Numpy>=1.20 has numpy.typing, so then ArrayLike can be used." -- We don't have a strict dependency on numpy according to our requirement, so that should be fine.
"Code that edits attributes of objects at run time will require so many # type: ignore that it will have a harder time getting through code review." -- Yep, that may be a serious lisibility issue.

2 replies

asumagic Nov 7, 2024
Maintainer

We do have a fair amount of places where matplotlib is used (mostly in some helper/tool functions, e.g. dependency matrices for streaming debug), but it is kept optional in most places.

TParcollet Nov 7, 2024
Maintainer

Interesting, why is it optional then. Maybe we should change that.

pplantinga · 2024-11-06T16:32:17Z

pplantinga
Nov 6, 2024
Maintainer

I have historically been a little skeptical of Python type hints, but I'll try to keep an open mind about it for the purposes of this discussion. Just to explain, I find that often they obscure rather than clarify function definitions, without providing any guarantees.

def function(a, b=None, c=None):
  """
  Arguments
  ---------
  a : torch.Tensor
  b : dict, default None
  c : list of lists of ints, default None

  Returns
  -------
  int or None
  """

is just much clearer to me than:

def function(a : torch.Tensor, b : Optional[Mapping[str, Any]] = None, c : Optional[Iterable[Iterable[int]]] = None) -> Optional[int]:
...

My second impression is that the type system can sometimes be too restrictive, marking perfectly fine code as violating type safety, making it an annoying burden for developers. The risk is that we overuse # type: ignore which just clutters the code.

However, I'm certainly willing to admit that as the toolkit grows and is depended on by more users, perhaps the verbosity and extra developer burden is worth the trouble in order to get people to think more carefully about the structure of the arugments. In addition, perhaps this could encourage us to use more data classes that encapsulate the arguments we send between functions. This is sort of a parallel issue but seems related to me: it seems we ought to have a data class for run_opts and perhaps a few other places.

As for specific tools, I'm not familiar with these but a brief investigation made it seem like pyright is more stable -- slow development at this point, whereas pyre is being actively developed. Leaning towards pyre but honestly, probably whoever puts together the initial PR should try both and see which one is more easy to use.

0 replies

asumagic · 2024-11-07T08:39:22Z

asumagic
Nov 7, 2024
Maintainer

I would be in favor of this. In addition to other comments, I would tend towards one that integrates well with IDEs and/or that is fast to execute. (It's also big enough that it may be required to find a way to approach the problem incrementally)

Something I long wished for would be a way to add tensor shape information and checking to static type information, but last I checked tooling around this looked fairly immature.

One thing where I think we will always fall short is linting in training scripts, since most typing information is gone when you go through hyperpyyaml-loaded namespaces. It would be neat if the chosen linter would allow executing arbitrary code to infer those types from a sample yaml, but that seems tricky.

It may be possible to run multiple type checkers to check for the union of the problems they detect.

This might increase the amount of workarounds in the codebase for each linter. I would stick to one.

0 replies

rogiervd · 2024-11-07T14:08:14Z

rogiervd
Nov 7, 2024
Author

Thanks all for the comments. Picking out a subset of points:

I find that often they obscure rather than clarify function definitions

I think it would be

def function(
    a: torch.Tensor,
    b: Optional[dict] = None,
    c: Optional[List[List[int]]] = None,
) -> Optional[int]:

but still, it's a matter of taste.
By the way, from Python 3.9 (not currently relevant) this can be written as

def function2(
    a: torch.Tensor,
    b: dict | None = None,
    c: list[list[int]] | None = None,
) -> int | None:

My second impression is that the type system can sometimes be too restrictive, marking perfectly fine code as violating type safety

This is specifically why I mentioned the Brain class, which uses setattr(self, arg, hparams[arg]) to dynamically set attributes. My opinion is that there's all sorts of problems with this implementation (correctness, safety, understandability) and the addition of type: ignore in many places would helpfully highlight this. You may disagree.

Something I long wished for would be a way to add tensor shape information and checking to static type information, but last I checked tooling around this looked fairly immature.

Good to know I'm not the only one who feels this. I believe jaxtyping does something like this (also for PyTorch) but at runtime.

One thing where I think we will always fall short is linting in training scripts, since most typing information is gone when you go through hyperpyyaml-loaded namespaces. It would be neat if the chosen linter would allow executing arbitrary code to infer those types from a sample yaml, but that seems tricky.

I think this is out of scope for this proposal, essentially because no one has written a type checker for hyperpyyaml yet. I think the closest thing would be to use pydantic (again, at run time), and a type-annotated dataclass that the hyperyaml can be loaded into. (I would prefer hparams.num_workers over hparams["num_workers"] so that the IDE can help. But: out of scope.)

1 reply

pplantinga Nov 8, 2024
Maintainer

By the way, from Python 3.9 (not currently relevant) this can be written as
def function2(
    a: torch.Tensor,
    b: dict | None = None,
    c: list[list[int]] | None = None,
) -> int | None:

Ooh, I do like this better, once 3.8 is deprecated maybe we can target this.

My second impression is that the type system can sometimes be too restrictive, marking perfectly fine code as violating type safety

This is specifically why I mentioned the Brain class, which uses setattr(self, arg, hparams[arg]) to dynamically set attributes. My opinion is that there's all sorts of problems with this implementation (correctness, safety, understandability) and the addition of type: ignore in many places would helpfully highlight this. You may disagree.

Okay, this is a pretty core part of Brain, if the type checker is going to choke on this we either need a pretty big redesign or, like you said, a lot of type: ignores which may reduce the value of the type checker. Here's a follow-up question I guess, does the PyTorch project itself have any type checking? Because they do this sort of thing a lot as well. Just wondering if we might benefit from studying their choices.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Introducing type-checking #2749

Uh oh!

{{title}}

Uh oh!

Replies: 5 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

Introducing type-checking #2749

Uh oh!

rogiervd Nov 6, 2024

Replies: 5 comments · 3 replies

Uh oh!

TParcollet Nov 6, 2024 Maintainer

Uh oh!

TParcollet Nov 6, 2024 Maintainer

Uh oh!

asumagic Nov 7, 2024 Maintainer

Uh oh!

TParcollet Nov 7, 2024 Maintainer

Uh oh!

pplantinga Nov 6, 2024 Maintainer

Uh oh!

asumagic Nov 7, 2024 Maintainer

Uh oh!

rogiervd Nov 7, 2024 Author

Uh oh!

pplantinga Nov 8, 2024 Maintainer

rogiervd
Nov 6, 2024

Replies: 5 comments 3 replies

TParcollet
Nov 6, 2024
Maintainer

TParcollet
Nov 6, 2024
Maintainer

asumagic Nov 7, 2024
Maintainer

TParcollet Nov 7, 2024
Maintainer

pplantinga
Nov 6, 2024
Maintainer

asumagic
Nov 7, 2024
Maintainer

rogiervd
Nov 7, 2024
Author

pplantinga Nov 8, 2024
Maintainer