bpo-43897: Implement AST validation for Pattern Matching #24771

isidentical · 2021-03-06T10:52:09Z

https://bugs.python.org/issue43897

github-actions · 2021-04-11T00:12:01Z

This PR is stale because it has been open for 30 days with no activity.

brandtbucher

Hi @isidentical!

I think I missed this in the beta-freeze scramble. Any plans to update it for the new AST?

isidentical · 2021-05-16T17:51:39Z

I think I missed this in the beta-freeze scramble. Any plans to update it for the new AST?

I might take a look at it in the upcoming weeks though I don't think I would have enough capacity to catch up with beta 2 (9 days from now). If you have time, I could give access to my fork if not I'll try to make it ready for beta 3.

brandtbucher · 2021-05-16T18:27:12Z

Beta 3 is fine!

isidentical · 2021-06-12T17:45:48Z

Hey @brandtbucher! I've updated the PR with the new AST form, sorry it took a bit long. I didn't have much free time after the AST layout change, and needed to catch up with the new nodes first. Just ported all the missing pieces, there are still some bugs that I can see (for example some of the old parts of the validator code just quit when error happens, but doesn't decrement the recursion limit. Since I didn't touch those and the patch was already big I just skipped them now, might fix later) but nothing too big. Please let me know any missing stuff that I should handle.

bedevere-bot · 2021-06-12T17:57:38Z

🤖 New build scheduled with the buildbot fleet by @isidentical for commit d758c6a1e9f6c01800822c5bdf5e1778161fab3f 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

brandtbucher

Thanks, this looks really good! I just have a few comments:

brandtbucher · 2021-06-30T20:50:24Z

Lib/test/test_ast.py

+                    cases = [
+                        ast.match_case(
+                            pattern=pattern,
+                            body = [ast.Expr(ast.Constant(...))]


Took me a sec to see what was happening here. Maybe clearer as ast.Pass()?

Suggested change

body = [ast.Expr(ast.Constant(...))]

body = [ast.Pass()]

brandtbucher · 2021-06-30T20:53:03Z

Lib/test/test_ast.py

+        ast.MatchMapping(
+            [constant_true, ast.Starred(ast.Name('lol', ast.Load()), ast.Load())],
+            [pattern_x, pattern_1],
+            rest='legit'
+        ),


This looks the exact same as the pattern above, right?

Suggested change

ast.MatchMapping(

[constant_true, ast.Starred(ast.Name('lol', ast.Load()), ast.Load())],

[pattern_x, pattern_1],

rest='legit'

),

brandtbucher · 2021-06-30T20:55:32Z

Python/ast.c

+        case Constant_kind: {
+            /* Ellipsis and immutable sequences are not allowed.
+               For True, False and None, MatchSingleton() should
+               be used */
+            if (!validate_expr(state, exp, Load)) {
+                return 0;
+            }
+            PyObject *literal = exp->v.Constant.value;
+            if (PyLong_CheckExact(literal) || PyFloat_CheckExact(literal) ||
+                PyBytes_CheckExact(literal) || PyComplex_CheckExact(literal) ||
+                PyUnicode_CheckExact(literal)) {
+                return 1;
+            }
+            PyErr_SetString(PyExc_ValueError,
+                            "unexpected constant inside of a literal pattern");
+            return 0;
+        }


I don't think these braces are needed, right?

Suggested change

case Constant_kind: {

/* Ellipsis and immutable sequences are not allowed.

For True, False and None, MatchSingleton() should

be used */

if (!validate_expr(state, exp, Load)) {

return 0;

}

PyObject *literal = exp->v.Constant.value;

if (PyLong_CheckExact(literal) || PyFloat_CheckExact(literal) ||

PyBytes_CheckExact(literal) || PyComplex_CheckExact(literal) ||

PyUnicode_CheckExact(literal)) {

return 1;

}

PyErr_SetString(PyExc_ValueError,

"unexpected constant inside of a literal pattern");

return 0;

}

case Constant_kind:

/* Ellipsis and immutable sequences are not allowed.

For True, False and None, MatchSingleton() should

be used */

if (!validate_expr(state, exp, Load)) {

return 0;

}

PyObject *literal = exp->v.Constant.value;

if (PyLong_CheckExact(literal) || PyFloat_CheckExact(literal) ||

PyBytes_CheckExact(literal) || PyComplex_CheckExact(literal) ||

PyUnicode_CheckExact(literal)) {

return 1;

}

PyErr_SetString(PyExc_ValueError,

"unexpected constant inside of a literal pattern");

return 0;

brandtbucher · 2021-06-30T21:01:33Z

Python/ast.c

            break;
-        case MatchMapping_kind:
-            // TODO: check "rest" target name is valid
+        case MatchMapping_kind: {
            if (asdl_seq_LEN(p->v.MatchMapping.keys) != asdl_seq_LEN(p->v.MatchMapping.patterns)) {
                PyErr_SetString(PyExc_ValueError,
                                "MatchMapping doesn't have the same number of keys as patterns");
                return 0;


Suggested change

return 0;

ret = 0;

break;

brandtbucher · 2021-06-30T21:05:34Z

Python/ast.c

            break;
-        case MatchMapping_kind:
-            // TODO: check "rest" target name is valid
+        case MatchMapping_kind: {


These braces aren't needed either, right?

Suggested change

case MatchMapping_kind: {

case MatchMapping_kind:

brandtbucher · 2021-06-30T21:05:51Z

Python/ast.c

            break;
+        }


See above.

Suggested change

}

brandtbucher · 2021-06-30T21:10:31Z

Python/ast.c

+                } else if (cls->kind == Attribute_kind) {
+                    cls = cls->v.Attribute.value;
+                    continue;
+                } else {


Suggested change

} else if (cls->kind == Attribute_kind) {

cls = cls->v.Attribute.value;

continue;

} else {

}

else if (cls->kind == Attribute_kind) {

cls = cls->v.Attribute.value;

continue;

}

else {

brandtbucher · 2021-06-30T21:11:20Z

Python/ast.c

            break;
+        }
        case MatchClass_kind:
            if (asdl_seq_LEN(p->v.MatchClass.kwd_attrs) != asdl_seq_LEN(p->v.MatchClass.kwd_patterns)) {
                PyErr_SetString(PyExc_ValueError,
                                "MatchClass doesn't have the same number of keyword attributes as patterns");
                return 0;


Suggested change

return 0;

ret = 0;

break;

brandtbucher · 2021-06-30T21:11:28Z

Python/ast.c

        case MatchClass_kind:
            if (asdl_seq_LEN(p->v.MatchClass.kwd_attrs) != asdl_seq_LEN(p->v.MatchClass.kwd_patterns)) {
                PyErr_SetString(PyExc_ValueError,
                                "MatchClass doesn't have the same number of keyword attributes as patterns");
                return 0;
            }
-            // TODO: Restrict cls lookup to being a name or attribute
            if (!validate_expr(state, p->v.MatchClass.cls, Load)) {
                return 0;


Suggested change

return 0;

ret = 0;

break;

bedevere-bot · 2021-06-30T21:13:29Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

brandtbucher · 2021-06-30T21:20:57Z

Just ported all the missing pieces, there are still some bugs that I can see (for example some of the old parts of the validator code just quit when error happens, but doesn't decrement the recursion limit. Since I didn't touch those and the patch was already big I just skipped them now, might fix later) but nothing too big.

Sorry, didn't see this. Feel free to ignore those suggestions, then!

I can find time to go back over those and fix the "_" name validation issue sometime if you're too busy. Just let me know.

isidentical · 2021-07-08T19:48:38Z

I can find time to go back over those and fix the "_" name validation issue sometime if you're too busy. Just let me know.

What do you mean by the _ validation (example case would be welcome)? By the way feel free to apply style suggestions by yourself since it is quite hard to deal with that sort of diffs in GitHub interface :(

brandtbucher · 2021-07-28T17:13:10Z

What do you mean by the _ validation (example case would be welcome)?

Patterns like MatchMapping(..., rest="_"), MatchStar("_"), and MatchAs(..., name="_") should be rejected, but aren't.

Also, MatchStar is allowed as a top-level pattern, but shouldn't be. I'll just merge this and fix those, the remaining recursion depth bugs, and remove some outdated comments in a new PR today (it will be easier to backport to 3.10 that way).

brandtbucher · 2021-07-28T17:17:36Z

Thanks again! Sorry I kept dropping the ball on this, haha.

the-knights-who-say-ni added the CLA signed label Mar 6, 2021

bedevere-bot added the awaiting core review label Mar 6, 2021

isidentical added the skip news label Mar 6, 2021

isidentical requested a review from brandtbucher March 6, 2021 11:04

github-actions bot added the stale Stale PR or inactive for long period of time. label Apr 11, 2021

isidentical changed the title ~~bpo-42128: Implement AST validation for Pattern Matching~~ bpo-43897: Implement AST validation for Pattern Matching Apr 20, 2021

isidentical added the DO-NOT-MERGE label Apr 20, 2021

brandtbucher reviewed May 16, 2021

View reviewed changes

isidentical force-pushed the pattern-matching-validator branch from 0356aea to d758c6a Compare June 12, 2021 17:43

isidentical removed the DO-NOT-MERGE label Jun 12, 2021

isidentical requested a review from brandtbucher June 12, 2021 17:46

isidentical added 🔨 test-with-buildbots Test PR w/ buildbots; report in status section and removed stale Stale PR or inactive for long period of time. labels Jun 12, 2021

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 12, 2021

isidentical requested a review from pablogsal June 28, 2021 07:33

brandtbucher requested changes Jun 30, 2021

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting core review labels Jun 30, 2021

bpo-43897: ast validation for pattern matching nodes

8dcb7d9

isidentical force-pushed the pattern-matching-validator branch from d758c6a to 8dcb7d9 Compare July 8, 2021 20:57

pablogsal mentioned this pull request Jul 10, 2021

[3.10] bpo-43897: ast validation for pattern matching nodes #27074

Merged

brandtbucher merged commit 31bec6f into python:main Jul 28, 2021

bedevere-bot removed the awaiting changes label Jul 28, 2021

Uh oh!

bpo-43897: Implement AST validation for Pattern Matching #24771

bpo-43897: Implement AST validation for Pattern Matching #24771

Uh oh!

Conversation

isidentical commented Mar 6, 2021 • edited by brandtbucher Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 11, 2021

Uh oh!

brandtbucher left a comment

Choose a reason for hiding this comment

Uh oh!

isidentical commented May 16, 2021

Uh oh!

brandtbucher commented May 16, 2021

Uh oh!

isidentical commented Jun 12, 2021

Uh oh!

bedevere-bot commented Jun 12, 2021

Uh oh!

brandtbucher left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Jun 30, 2021

Uh oh!

brandtbucher commented Jun 30, 2021

Uh oh!

isidentical commented Jul 8, 2021

Uh oh!

brandtbucher commented Jul 28, 2021

Uh oh!

brandtbucher commented Jul 28, 2021

Uh oh!

Uh oh!

isidentical commented Mar 6, 2021 •

edited by brandtbucher

Loading