The stubs need to have the directory structure stubs/antlr4-python3-runtime/antrl4. The outer directory is the name of the PyPI runtime distribution; the inner directory is the name of the package that can be imported at runtime after installing the runtime distribution

It looks like I can easily clean up the Test / Test typeshed with pyright failures by referencing and tracing through the source. for the Third-party stubtests, what's the best way to resolve issues where the source devs initialize variables to None?

for the Third-party stubtests, what's the best way to resolve issues where the source devs initialize variables to None?

I haven't looked at the antlr4 source code at all, but if the runtime has a function like this:

def foo(name=None): ...

And you've written a stub like this:

def foo(name: str = ...) -> None: ...

Then, to make stubtest happy, you generally need to change it to something like this:

def foo(name: str | None = None) -> None ...

Since the function obviously won't raise an error if None is passed (since None is the default)

Most of the pyright errors seem to be referencing missing generic arguments. The best option is to add the proper generic arguments. But if you don't have the bandwidth for that, it's fine to use Incomplete (from the _typeshed module) for now. This is a marker that proper types need to be added at some point: set -> set[Incomplete], dict -> dict[Incomplete, Incomplete] etc.

I haven't looked at the stubtest problems yet, but this indicates a mismatch between the implementation and the stubs.

@srittau I have half the missing generic arguments fixed in a dev branch. I'm looking at the source to make sure TokenName: list is actually TokenName: list[str]. Thanks for the "Incomplete" hint. If I find one excessively difficult, I'll make sure to use that.

perl -pi -e 's/(def.*)\): \.\.\./$1\) -> Incomplete: \.\.\./g' *.pyi to the rescue

Any advice on these errors?

error: antlr4.xpath.XPathLexer.StringIO.truncate is inconsistent, stub argument "__size" differs from runtime argument "pos" 
Stub: in file /home/runner/work/typeshed/typeshed/stubs/antlr4-python3-runtime/antlr4/xpath/XPathLexer.pyi:73
def (self: io.IOBase, Union[builtins.int, None] =) -> builtins.int 
Runtime: def (self, pos=None, /)

I don’t see any calls to StringIO.truncate in XPathLexer.pyi, and XPathLexer.pyi is not even 73 lines long‽

Any advice on these errors?

error: antlr4.xpath.XPathLexer.StringIO.truncate is inconsistent, stub argument "__size" differs from runtime argument "pos" 
Stub: in file /home/runner/work/typeshed/typeshed/stubs/antlr4-python3-runtime/antlr4/xpath/XPathLexer.pyi:73
def (self: io.IOBase, Union[builtins.int, None] =) -> builtins.int 
Runtime: def (self, pos=None, /)

I don’t see any calls to StringIO.truncate in XPathLexer.pyi, and XPathLexer.pyi is not even 73 lines long‽

Looks like it's because you're re-exporting StringIO from XPathLexer.pyi, when you should just be importing it without re-exporting it. Replace from io import StringIO as StringIO at the top of the file with from io import StringIO.

Thanks! This is the way stubgen created the file. Should I remove the “as Foo” from all the imports?

Last errors:

error: antlr4.tree.Tree.ParserRuleContext is not present at runtime
Stub: in file /home/runner/work/typeshed/typeshed/stubs/antlr4-python3-runtime/antlr4/tree/Tree.pyi:23
Any
Runtime:
MISSING

Thanks! This is the way stubgen created the file.

Yeah, stubgen has bad default settings. In our convenience script for generating new stubs for typeshed, we always run stubgen with the --export-less option: https://github.com/python/typeshed/blob/main/scripts/create_baseline_stubs.py

Should I remove the “as Foo” from all the imports?

Probably

This pull request has a green check. Let me know if there are any other must-dos before merging it.

Could you do a final cleanup pass with the following replacements?

• -> Incomplete: -> :
• : Incomplete) -> )
• : Incomplete, -> ,

I just pushed a bunch of cleanups to your PR branch. I'm not sure exactly what I did that could have triggered new stubtest errors, but stubtest is now, for some reason, complaining that some things in these stubs don't exist at runtime. At least some of the errors appear to be correct, however:

>>> import antlr4.Recognizer
>>> antlr4.Recognizer.RecognizeException
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'antlr4.Recognizer' has no attribute 'RecognizeException'

(At runtime, RecognizeException is set here, but then deleted again here)

Ah, I see stubtest didn't run on 9f184a1, for some reason, which explains why we didn't see those complaints on the commit immediately prior to the cleanups I just pushed.

@AlexWaygood Do you have any advice on resolving the “not present at runtime” issues given the state of the antlr4 source material?

@AlexWaygood Do you have any advice on resolving the “not present at runtime” issues given the state of the antlr4 source material?

Since these things do indeed seem to be not present at runtime when you actually use the package, I would delete them from the stubs

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

All green and good to go from my end.

@AlexWaygood what’s the usual wait time from “all good” to “merge”? Anything I need to do?

@AlexWaygood what’s the usual wait time from “all good” to “merge”? Anything I need to do?

Sorry for the wait. I'm afraid I'm more busy than usual at the moment, so probably won't be able to take a proper look at this soon. Hopefully somebody else on the team will be able to take a look soon, but I'm afraid we're all volunteers, so I can't make any promises about timescale here :(

@AlexWaygood No problem. I noticed there were other mergers and wanted to make sure there wasn’t anything I needed to do.