Generally we try to follow the actual implementation, until there is a good reason not to. Typing private modules is "better", but it was often neglected, because practically speaking there isn't too much difference, and is slightly more work.

I think what I mean is that it's not obvious what "follow the implementation" means here, because different parts of the implementation are conflicting with each other.

If we have:

_weakref.pyi

class ReferenceType: ...

weakref.pyi

from _weakref import ReferenceType as ReferenceType

Then we have followed the import structure accurately, but ReferenceType has a different qualname in typeshed and at runtime.

If we have instead:

_weakref.pyi

from weakref import ReferenceType as ReferenceType

weakref.pyi

class ReferenceType: ...

Now ReferenceType has a matching qualname in typeshed and at runtime, but we've had to reverse the import structure in order to do that. I'd make the case that the second version follows the implementation a little better, since the qualname is visible to mypy/stubcheck, while (I don't think) the import structure really is. I might be wrong about that?

At any rate, I wasn't suggesting that we shouldn't make these available in the private module namespace.

This is a great question. I don't have a strong opinion, but I lean towards:

Typeshed should follow the declared implementation over the layout. e.g. ReferenceType considers itself to live in the weakref module, that's where the majority of users will get it from, so that's the correct place for it.

I have this preference for two reasons:

• It results in better user facing diagnostics
• It better reflects the truth in alternative Python implementations or cases where C accelerator modules are missing

This is also why in the "various inheritance" patch from #3968 I have a few cases of special casing underscore prefixed modules and classes: python/mypy@master...hauntsaninja:mypy:stubtestbaseclass#diff-55639a1f38efeacbb10988649399037c13ad11e9c82890b453d98ea2bed9caf8R267

I made several MRs along these lines, but then I discovered that the _pydecimal module changes its __name__ attribute, which I hadn't though of as an option: https://github.com/python/cpython/blob/d91e43ed7839b601cbeadd137d89e69e2cc3efda/Lib/_pydecimal.py#L151

Which got me thinking: what if mypy could detect cases where a module level __name__ = "foo" is set in the stubs, and set the name appropriately? Then we could follow both parts of the implementation, layout and naming.

I'm willing to take a run at that MR, if people think it's a reasonable idea.

As it turned out, classes which set their own __module__ attribute were a lot easier to handle than modules which set their own __name__ attribute. I have a draft implementation for that here: python/mypy#16698

With that, typeshed could set __module__ on classes which claim to be from a different module, and stubcheck could perform more accurate comparisons with runtime.

There are a few pull requests that add missing _foo modules (#11187 #11155 #11159 #11174). What problem does that solve? In other words, when does a typeshed user want stubs for a _weakref or _io or _ssl module?

If I understood correctly, we decided to do this (which IMO is a good idea for several reasons):

e.g. ReferenceType considers itself to live in the weakref module, that's where the majority of users will get it from, so that's the correct place for it.

And if you have everything you'll ever need in weakref, then why do you need _weakref?

Note that I'm talking about adding _foo modules for completeness. I'm not saying that we should delete existing _foo stubs. For example, _tkinter stubs contain various things that aren't exposed in tkinter, but can be useful when working with tkinter.

I created those while working on naming and inheritance - those all contain something which is a parent class of something else, so adding them brings the inheritance structure in typeshed closer to what it looks like at runtime. In other cases, it's a naming issue: the mod.Foo type calls itself _mod.Foo at runtime, and adding the _mod module lets typeshed align names that way.

It's a fair question whether that's justified in all cases or not. But that's not actually the subject of this ticket. Note that typeshed already has _weakref, and aligning the naming there would mean moving weakref.ReferenceType out of _weakref (where it's currently defined by both typeshed and runtime) and into weakref (where it will match the name given by runtime).

For another example and one more on subject: I've been investigating the issue of decoupling collections.abc from it's typing aliases (#6257). Runtime defines these classes in _collections_abc.py rather than collections/abc.py so that they can be imported without incurring the performance hit of also importing collections/__init__.py, but then sets __name__ = "collections.abc" in that file so that the various clases call themselves collections.abc.Foo instead of _collections_abc.Foo.

Typeshed has the same performance concerns; If we define them in collections.abc, that's an increase in the number of files involved in the circular dependency cycle that contains typing and builtins. I haven't tested with collections.abc, but making sure that that mypy can resolve that dependency cycle fast enough was already something I had to pay attention to while doing some tests that move these classes from typing.pyi into _collections_abc.pyi. So I think defining them in _collections_abc.pyi but adjusting their naming with either the module-level __name__ = directive or a class-level __module__ = directive is the best option.