If we're doing that, we could probably just output it as rust code and include!() it. Actually, we could probably do that anyway and avoid having to depend on serde-json

I think we don't need to care about dependencies for derive module. It will be only used during the build step. We will not use any serde-json code from the end product.

Whats happening to mac test build?

Oh, wait - also my concern about doing this is that it'll make a cpython environment a requirement for building, which I feel is like undesired. Also build times, traverse_all() takes forever.

I can agree about python dependency by how you see the dependency is. Maybe checking for python in local environment or download the pre-built database from internet would work.

I don't agree about build time. It will be run only once unless it needs to be updated, and it requires only one second to run.

time python scripts/generate_docs.py > /dev/null
python scripts/generate_docs.py > /dev/null  0.81s user 0.18s system 97% cpu 1.014 total

Huh, on mine it hangs for ten seconds and then core dumps with an illegal instruction. Are you on an M1? :P

Oh, maybe one of the reasons for that is that it checks every module, right? It needs to be in a totally clean environment to just do the stdlib ones.

Yes, I am using a new clean venv for the build.
(and yes, it is m1. But 10x times is not an expectation)

The build error must be one of the reason we should run scripts on CI

I still really don't think invoking cpython in the build process (and specifically depending on the python environment the build process runs in) is a good idea; even in CI that failure isn't from a stdlib package! it's from python-apt which is probably preinstalled on the runner vm. And also this makes the result of the build vary based on which specific version of python is installed (which I feel like is a bad precedent), and IMO downloading the pre-built database from the web is the same thing as just checking it into the repo in the first place.

I could make a quick patch for that unicode escape issue, but I don't think the stdlib docstrings would ever have a non-ascii character in them.

oh, that explains the reason it only fails on linux!

btw, please check the new build.rs. Unlike ast, doc is totally optional. if it cannot run python, it will build rustpython without document.

I added whitelist to skip third party libraries. it will be faster. The doc generation is totally optional. Maybe we'd better to have doc feature to skip doc embedding to reduce binary size in future.

I... still don't like that the build will be different depending on the python version installed. I mean, it doesn't really matter that the docs are always up to date, right? If we need to update the docs db, we can, I still don't think I see the benefit of generating them as part of the build process.

oh, that reminded me why i made the doc to json file.
For now, this way is increasing doc coverage because we are not managing doc data.
My first plan for json doc db was collecting data from every OS and merge them to single file.

@coolreader18 no instant generation anymore. will you please also review other PRs?