Skip to content

bpo-36500: Enable Windows users to regen grammar, opcodes, tokens and symbols from Visual Studio #12654

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zooba merged 22 commits into from
Dec 17, 2019
Merged

bpo-36500: Enable Windows users to regen grammar, opcodes, tokens and symbols from Visual Studio #12654

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
f291fb1
Add regen-grammar equivalent as optional build project, dependant on …
tonybaloney Apr 2, 2019
0f4b23a
Downgrade solution file
tonybaloney Apr 2, 2019
759e241
Remove win10 toolset configurations manually
tonybaloney Apr 2, 2019
192f358
Update pcbuild.sln
tonybaloney Apr 2, 2019
5eb7795
add regen-keywords and regen-symbols build stages
tonybaloney Apr 2, 2019
4017063
Merge remote-tracking branch 'upstream/pcbuildregen' into pcbuildregen
tonybaloney Apr 2, 2019
0be5632
Copy both C and H for grammar, regen opcodes and order process
tonybaloney Apr 2, 2019
a2c1e0c
All regen stages with comments (except regen-ast)
tonybaloney Apr 2, 2019
b52329f
Added regen-ast in 2 stages
tonybaloney Apr 2, 2019
95eaa9f
Add the other artifacts to the linked files
tonybaloney Apr 2, 2019
1396fd9
📜🤖 Added by blurb_it.
blurb-it[bot] Apr 2, 2019
80d2d9a
Update with feedback from Steve
tonybaloney May 9, 2019
45c7999
Error on regenerated files in Release configuration
tonybaloney May 9, 2019
2162db2
Add regen to the build.bat
tonybaloney May 9, 2019
89246a4
Make regen a seperate project
tonybaloney May 9, 2019
546cea5
Merge branch 'master' into pcbuildregen
tonybaloney Aug 21, 2019
eb87863
Swap rel paths with $(PySourcePath)
tonybaloney Aug 21, 2019
9fca9db
Bind project path correctly
tonybaloney Aug 21, 2019
7863992
merge upstream
tonybaloney Aug 21, 2019
370fcba
Change the regen to be a stage after the primary build so that the us…
tonybaloney Aug 22, 2019
8f93907
Update Misc/NEWS.d/next/Build/2019-04-02-01-59-26.bpo-36500.fyG6_U.rst
tonybaloney Aug 22, 2019
0a6ab75
Update 2019-04-02-01-59-26.bpo-36500.fyG6_U.rst
tonybaloney Aug 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Build/2019-04-02-01-59-26.bpo-36500.fyG6_U.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Added an optional "regen" project to the Visual Studio solution that will
regenerate all grammar, tokens, and opcodes.
17 changes: 16 additions & 1 deletion PCbuild/build.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ echo. automatically by the pythoncore project)
echo. --pgo Build with Profile-Guided Optimization. This flag
echo. overrides -c and -d
echo. --test-marker Enable the test marker within the build.
echo. --regen Regenerate all opcodes, grammar and tokens
echo.
echo.Available flags to avoid building certain modules.
echo.These flags have no effect if '-e' is not given:
Expand Down Expand Up @@ -76,7 +77,8 @@ if "%~1"=="-k" (set kill=true) & shift & goto CheckOpts
if "%~1"=="--pgo" (set do_pgo=true) & shift & goto CheckOpts
if "%~1"=="--pgo-job" (set do_pgo=true) & (set pgo_job=%~2) & shift & shift & goto CheckOpts
if "%~1"=="--test-marker" (set UseTestMarker=true) & shift & goto CheckOpts
if "%~1"=="-V" shift & goto :Version
if "%~1"=="-V" shift & goto Version
if "%~1"=="--regen" (set Regen=true) & shift & goto CheckOpts
rem These use the actual property names used by MSBuild. We could just let
rem them in through the environment, but we specify them on the command line
rem anyway for visibility so set defaults after this
Expand Down Expand Up @@ -154,9 +156,22 @@ echo on
/p:UseTestMarker=%UseTestMarker% %GITProperty%^
%1 %2 %3 %4 %5 %6 %7 %8 %9

if "%Regen%"=="true" call :Regen
@echo off
exit /b %ERRORLEVEL%

:Regen
echo on
call "%dir%find_msbuild.bat" %MSBUILD%
if not ERRORLEVEL 1 %MSBUILD% "%dir%regen.vcxproj" /t:%target% %parallel% %verbose%^
/p:IncludeExternals=%IncludeExternals%^
/p:Configuration=%conf% /p:Platform=%platf%^
/p:UseTestMarker=%UseTestMarker% %GITProperty%^
%1 %2 %3 %4 %5 %6 %7 %8 %9

@echo off
goto :eof

:Version
rem Display the current build version information
call "%dir%find_msbuild.bat" %MSBUILD%
Expand Down
225 changes: 225 additions & 0 deletions PCbuild/regen.vcxproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,225 @@
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup Label="ProjectConfigurations">
<ProjectConfiguration Include="Debug|ARM">
<Configuration>Debug</Configuration>
<Platform>ARM</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Debug|Win32">
<Configuration>Debug</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Debug|x64">
<Configuration>Debug</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="PGInstrument|ARM">
<Configuration>PGInstrument</Configuration>
<Platform>ARM</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="PGInstrument|Win32">
<Configuration>PGInstrument</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="PGInstrument|x64">
<Configuration>PGInstrument</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="PGUpdate|ARM">
<Configuration>PGUpdate</Configuration>
<Platform>ARM</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="PGUpdate|Win32">
<Configuration>PGUpdate</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="PGUpdate|x64">
<Configuration>PGUpdate</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|ARM">
<Configuration>Release</Configuration>
<Platform>ARM</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|Win32">
<Configuration>Release</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|x64">
<Configuration>Release</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
</ItemGroup>
<PropertyGroup Label="Globals">
<ProjectGuid>{21CF2108-2CC9-4005-A6ED-B7965ADE3854}</ProjectGuid>
<Keyword>Win32Proj</Keyword>
<RootNamespace>regen</RootNamespace>
<SupportPGO>false</SupportPGO>
<WindowsTargetPlatformVersion>10.0</WindowsTargetPlatformVersion>
</PropertyGroup>
<Import Project="python.props" />
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
<PropertyGroup Label="Configuration">
<ConfigurationType>Utility</ConfigurationType>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Label="Configuration" Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<PropertyGroup Label="Configuration" Condition="'$(Configuration)|$(Platform)'=='Debug|ARM'">
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<PropertyGroup Label="Configuration" Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<PropertyGroup Label="Configuration" Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<PropertyGroup Label="Configuration" Condition="'$(Configuration)|$(Platform)'=='Release|ARM'">
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<PropertyGroup Label="Configuration" Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<PropertyGroup Label="Configuration" Condition="'$(Configuration)|$(Platform)'=='PGUpdate|x64'">
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<PropertyGroup Label="Configuration" Condition="'$(Configuration)|$(Platform)'=='PGUpdate|ARM'">
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<PropertyGroup Label="Configuration" Condition="'$(Configuration)|$(Platform)'=='PGUpdate|Win32'">
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<PropertyGroup Label="Configuration" Condition="'$(Configuration)|$(Platform)'=='PGInstrument|ARM'">
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<PropertyGroup Label="Configuration" Condition="'$(Configuration)|$(Platform)'=='PGInstrument|x64'">
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<PropertyGroup Label="Configuration" Condition="'$(Configuration)|$(Platform)'=='PGInstrument|Win32'">
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
<ImportGroup Label="ExtensionSettings">
</ImportGroup>
<ImportGroup Label="PropertySheets">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
<Import Project="pyproject.props" />
</ImportGroup>
<PropertyGroup Label="UserMacros" />
<ItemDefinitionGroup>
<ClCompile>
<PreprocessorDefinitions>_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
</Link>
</ItemDefinitionGroup>
<ItemGroup>
<ClCompile Include="..\Programs\_freeze_importlib.c" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="pythoncore.vcxproj">
<Project>{cf7ac3d1-e2df-41d2-bea6-1e2556cdea26}</Project>
<Private>true</Private>
<ReferenceOutputAssembly>true</ReferenceOutputAssembly>
<CopyLocalSatelliteAssemblies>false</CopyLocalSatelliteAssemblies>
<LinkLibraryDependencies>true</LinkLibraryDependencies>
<UseLibraryDependencyInputs>false</UseLibraryDependencyInputs>
</ProjectReference>
</ItemGroup>
<ItemGroup>
<None Include="..\Grammar\Grammar">
</None>
<None Include="..\Grammar\Tokens">
</None>
<None Include="..\Include\graminit.h">
</None>
<None Include="..\Python\graminit.c">
</None>
<None Include="..\Include\token.h">
</None>
<None Include="..\Include\opcode.h">
</None>
<None Include="..\Include\Python-ast.h">
</None>
<None Include="..\Python\Python-ast.c">
</None>
<None Include="..\Lib\keyword.py">
</None>
<None Include="..\Lib\symbol.py">
</None>
<None Include="..\Doc\library\token-list.inc">
</None>
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets">
</ImportGroup>
<Target Name="_RegenGrammar" BeforeTargets="Build">
<!-- Regenerate Include/graminit.h and Python/graminit.c from Grammar/Grammar using pgen-->
<Exec Command="&quot;$(PythonExe)&quot; -m Parser.pgen &quot;$(PySourcePath)Grammar\Grammar&quot; &quot;$(PySourcePath)Grammar\Tokens&quot; &quot;$(IntDir)graminit.h&quot; &quot;$(IntDir)graminit.c&quot;" />
<Copy SourceFiles="$(IntDir)graminit.h" DestinationFiles="$(PySourcePath)Include\graminit.h">
<Output TaskParameter="CopiedFiles" ItemName="_UpdatedH" />
</Copy>
<Copy SourceFiles="$(IntDir)graminit.c" DestinationFiles="$(PySourcePath)Python\graminit.c">
<Output TaskParameter="CopiedFiles" ItemName="_UpdatedC" />
</Copy>
<Warning Text="Grammar updated. You will need to rebuild pythoncore to see the changes." Condition="'@(_UpdatedH)' != '' and '@(_UpdatedC)' != ''" />
</Target>
<Target Name="_RegenAST_H" AfterTargets="_RegenGrammar">
<!-- Regenerate Include/Python-ast.h using Parser/asdl_c.py -h -->
<Exec Command="&quot;$(PythonExe)&quot; &quot;$(PySourcePath)Parser\asdl_c.py&quot; -h &quot;$(IntDir)Python-ast.h&quot; &quot;$(PySourcePath)Parser\Python.asdl&quot;" />
<Copy SourceFiles="$(IntDir)Python-ast.h" DestinationFiles="$(PySourcePath)Include\Python-ast.h">
<Output TaskParameter="CopiedFiles" ItemName="_UpdatedH" />
Copy link
Member

@zooba zooba Apr 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't sufficient for detecting files that were regenerated without changing. Look in the importlib one for the contents check.

(If you want an excuse to learn about target batching in MSBuild - which is ridiculously powerful - you might consider having one target for doing the optional-copy and fail if updated step, then pass it all the target files in an ItemGroup with the generated file as metadata.)

Copy link
Contributor Author

@tonybaloney tonybaloney Apr 2, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zooba It copies them regardless in the same way that the Makefile does, which is why this project is an optional configuration and not part of the Debug or Release build dependencies.
Would you like me to add this check and make regen.vcxproj build automatically?

Copy link
Member

@zooba zooba Apr 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that'll be most reliable.

If you add all the source files as items in the project then it should skip rebuilding if none of them have changed.

Copy link
Contributor Author

@tonybaloney tonybaloney May 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tried the target batching, it doesn't seem to be able to determine any better than this revision whether the contents have changed. The docs say that it uses timestamps on the files to determine (which won't work in this case)

Copy link
Member

@zooba zooba May 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, you'll need to do content comparisons. See what the current freezeimportlib project does.

</Copy>
<Warning Text="Python-ast.h updated. You will need to rebuild pythoncore to see the changes." Condition="'@(_UpdatedH)' != '' and '@(_UpdatedC)' != ''" />
</Target>
<Target Name="_RegenAST_C" AfterTargets="_RegenGrammar">
<!-- Regenerate Python/Python-ast.c using Parser/asdl_c.py -c -->
<Exec Command="&quot;$(PythonExe)&quot; &quot;$(PySourcePath)Parser\asdl_c.py&quot; -c &quot;$(IntDir)Python-ast.c&quot; &quot;$(PySourcePath)Parser\Python.asdl&quot;" />
<Copy SourceFiles="$(IntDir)Python-ast.c" DestinationFiles="$(PySourcePath)Python\Python-ast.c">
<Output TaskParameter="CopiedFiles" ItemName="_UpdatedH" />
</Copy>
<Warning Text="Python-ast.c updated. You will need to rebuild pythoncore to see the changes." Condition="'@(_UpdatedH)' != '' and '@(_UpdatedC)' != ''" />
</Target>
<Target Name="_RegenOpcodes" AfterTargets="_RegenAST_C">
<!-- Regenerate Include/opcode.h from Lib/opcode.py using Tools/scripts/generate_opcode_h.py-->
<Exec Command="&quot;$(PythonExe)&quot; $(PySourcePath)Tools\scripts\generate_opcode_h.py &quot;$(PySourcePath)Lib\opcode.py&quot; &quot;$(IntDir)opcode.h&quot;" />
<Copy SourceFiles="$(IntDir)opcode.h" DestinationFiles="$(PySourcePath)Include\opcode.h">
<Output TaskParameter="CopiedFiles" ItemName="_Updated" />
</Copy>
<Warning Text="Opcodes updated. You will need to rebuild pythoncore to see the changes." Condition="'@(_Updated)' != ''" />
</Target>
<Target Name="_RegenTokens" AfterTargets="_RegenOpcodes">
<!-- Regenerate Doc/library/token-list.inc from Grammar/Tokens using Tools/scripts/generate_token.py-->
<Exec Command="&quot;$(PythonExe)&quot; $(PySourcePath)Tools\scripts\generate_token.py rst &quot;$(PySourcePath)Grammar\Tokens&quot; &quot;$(PySourcePath)Doc\library\token-list.inc&quot;" />
<!-- Regenerate Include/token.h from Grammar/Tokens using Tools/scripts/generate_token.py-->
<Exec Command="&quot;$(PythonExe)&quot; $(PySourcePath)Tools\scripts\generate_token.py h &quot;$(PySourcePath)Grammar\Tokens&quot; &quot;$(PySourcePath)Include\token.h&quot;" />
<!-- Regenerate Parser/token.c from Grammar/Tokens using Tools/scripts/generate_token.py-->
<Exec Command="&quot;$(PythonExe)&quot; $(PySourcePath)Tools\scripts\generate_token.py c &quot;$(PySourcePath)Grammar\Tokens&quot; &quot;$(PySourcePath)Parser\token.c&quot;" />
<!-- Regenerate Lib/token.py from Grammar/Tokens using Tools/scripts/generate_token.py -->
<Exec Command="&quot;$(PythonExe)&quot; $(PySourcePath)Tools\scripts\generate_token.py py &quot;$(PySourcePath)Grammar\Tokens&quot; &quot;$(PySourcePath)Lib\token.py&quot;" />
</Target>
<Target Name="_RegenKeywords" AfterTargets="_RegenTokens">
<!-- Regenerate Lib/keyword.py from Grammar/Grammar and Grammar/Tokens using Parser/pgen-->
<Exec Command="&quot;$(PythonExe)&quot; -m Parser.pgen.keywordgen &quot;$(PySourcePath)Grammar\Grammar&quot; &quot;$(PySourcePath)Grammar\Tokens&quot; &quot;$(IntDir)keyword.py&quot;" />
<Copy SourceFiles="$(IntDir)keyword.py" DestinationFiles="$(PySourcePath)Lib\keyword.py">
<Output TaskParameter="CopiedFiles" ItemName="_Updated" />
</Copy>
<Warning Text="Keywords updated. You will need to rebuild pythoncore to see the changes." Condition="'@(_Updated)' != ''" />
</Target>
<Target Name="_RegenSymbols" AfterTargets="_RegenKeywords">
<!-- Regenerate Lib/symbol.py from Include/graminit.h using Tools/scripts/generate_symbol_py.py-->
<Exec Command="&quot;$(PythonExe)&quot; $(PySourcePath)Tools\scripts\generate_symbol_py.py &quot;$(PySourcePath)Include\graminit.h&quot; &quot;$(PySourcePath)Lib\symbol.py&quot;" />
</Target>
<Target Name="_CleanFiles" BeforeTargets="CoreClean">
<ItemGroup>
<Clean Include="$(IntDir)keyword.py" />
<Clean Include="$(IntDir)opcode.h" />
<Clean Include="$(IntDir)Python-ast.c" />
<Clean Include="$(IntDir)Python-ast.h" />
<Clean Include="$(IntDir)graminit.h.new" />
<Clean Include="$(IntDir)graminit.c.new" />
</ItemGroup>
</Target>
</Project>