Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exclude with nested "__all__" does not work as expected #1579

Closed
xspirus opened this issue May 29, 2020 · 3 comments · Fixed by #1588
Closed

Exclude with nested "__all__" does not work as expected #1579

xspirus opened this issue May 29, 2020 · 3 comments · Fixed by #1588
Labels
bug

Comments

@xspirus
Copy link
Contributor

@xspirus xspirus commented May 29, 2020

Bug

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

             pydantic version: 1.5.1
            pydantic compiled: False
                 install path: /home/spirus/projects/pydantic/pydantic
               python version: 3.8.3 (default, May 17 2020, 18:15:42)  [GCC 10.1.0]
                     platform: Linux-5.6.14-arch1-1-x86_64-with-glibc2.2.5
     optional deps. installed: ['typing-extensions', 'email-validator', 'devtools']

As I was experimenting with FastAPI and pydantic, I thought that it would be nice to use exclude in order to avoid some Schema duplication (or complex inheritances). Ultimately, I came across a weird bug when trying to use exclude on nested sequences. Below is a minimal example for reproducing the bug.

from typing import List

from pydantic import BaseModel


class A(BaseModel):
    a: int = 1
    b: int = 2
    c: int = 3


class B(BaseModel):
    a: List[A]


class C(BaseModel):
    b: List[B]


def test():
    c = C(b=[B(a=[A()])])
    print(c.dict(exclude={"b": {0: {"a": {"__all__": {"c"}}}}}))
    print(c.dict(exclude={"b": {"__all__": {"a": {"__all__": {"c"}}}}}))
    assert c.dict(exclude={"b": {0: {"a": {"__all__": {"c"}}}}}) == {"b": [{"a": [{"a": 1, "b": 2}]}]}
    assert c.dict(exclude={"b": {"__all__": {"a": {"__all__": {"c"}}}}}) == {"b": [{"a": [{"a": 1, "b": 2}]}]}

Running the above function with pytest the output is:

Test session starts (platform: linux, Python 3.8.3, pytest 5.3.5, pytest-sugar 0.9.2)
rootdir: /home/spirus/projects/pydantic, inifile: setup.cfg
plugins: cov-2.8.1, mock-3.1.0, sugar-0.9.2
collecting ... 

―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― test ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

    def test():
        c = C(b=[B(a=[A()])])
        print(c.dict(exclude={"b": {0: {"a": {"__all__": {"c"}}}}}))
        print(c.dict(exclude={"b": {"__all__": {"a": {"__all__": {"c"}}}}}))
        assert c.dict(exclude={"b": {0: {"a": {"__all__": {"c"}}}}}) == {"b": [{"a": [{"a": 1, "b": 2}]}]}
>       assert c.dict(exclude={"b": {"__all__": {"a": {"__all__": {"c"}}}}}) == {"b": [{"a": [{"a": 1, "b": 2}]}]}
E       AssertionError: assert {'b': [{}]} == {'b': [{'a': ...1, 'b': 2}]}]}
E         Differing items:
E         {'b': [{}]} != {'b': [{'a': [{'a': 1, 'b': 2}]}]}
E         Use -v to get the full diff

example.py:25: AssertionError
------------------------------------------------------------ Captured stdout call ------------------------------------------------------------
{'b': [{'a': [{'a': 1, 'b': 2}]}]}
{'b': [{}]}

 example.py ⨯                                                                                                                  100% ██████████

Results (0.05s):
       1 failed
         - example.py:20 test

I inspected the codebase a little bit, and found out that the bug is produced because of how _normalize_indexes, as it assumes that when the items is a dict containing the __all__ key, then the value is considered by default to be a set. So calling the update, information may be missed.

I am not certain if my usage violates how the __all__ key should be used, but if this should have been correct, I can create a PR with a solution.
@xspirus xspirus added the bug label May 29, 2020
@samuelcolvin
Copy link
Owner

@samuelcolvin samuelcolvin commented May 31, 2020 •
edited

your naming is very confusing 😕, do you think you could clarify with cases than don't reuse names like "b"?

But I think you're right.

@xspirus
Copy link
Contributor Author

@xspirus xspirus commented May 31, 2020

@samuelcolvin you are right. Here is a more non-confusing example I hope :D

from typing import List

from pydantic import BaseModel


class Alpha(BaseModel):
    id: int = 1
    name: str = "Alpha"


class Beta(BaseModel):
    alphas: List[Alpha]


class Model(BaseModel):
    betas: List[Beta]


def test():
    model = Model(
        betas=[Beta(alphas=[Alpha(), Alpha(id=2)]), Beta(alphas=[Alpha(id=3)])]
    )
    assert model.dict() == {
        "betas": [
            {
                "alphas": [
                    {"id": 1, "name": "Alpha"},
                    {"id": 2, "name": "Alpha"},
                ]
            },
            {"alphas": [{"id": 3, "name": "Alpha"}]},
        ]
    }
    assert model.dict(
        exclude={
            "betas": {
                0: {"alphas": {"__all__": {"name"}}},
                1: {"alphas": {"__all__": {"name"}}},
            }
        }
    ) == {
        "betas": [{"alphas": [{"id": 1}, {"id": 2}]}, {"alphas": [{"id": 3}]}]
    }
    assert model.dict(
        exclude={"betas": {"__all__": {"alphas": {"__all__": {"name"}}}}}
    ) == {
        "betas": [{"alphas": [{"id": 1}, {"id": 2}]}, {"alphas": [{"id": 3}]}]
    }

Output using pytest is:

Test session starts (platform: linux, Python 3.8.3, pytest 5.4.2, pytest-sugar 0.9.3)
rootdir: /home/spirus/projects/local/pydantic, inifile: setup.cfg
plugins: cov-2.8.1, sugar-0.9.3, mock-3.1.0
collecting ...

―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― test ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

    def test():
        model = Model(
            betas=[Beta(alphas=[Alpha(), Alpha(id=2)]), Beta(alphas=[Alpha(id=3)])]
        )
        assert model.dict() == {
            "betas": [
                {
                    "alphas": [
                        {"id": 1, "name": "Alpha"},
                        {"id": 2, "name": "Alpha"},
                    ]
                },
                {"alphas": [{"id": 3, "name": "Alpha"}]},
            ]
        }
        assert model.dict(
            exclude={
                "betas": {
                    0: {"alphas": {"__all__": {"name"}}},
                    1: {"alphas": {"__all__": {"name"}}},
                }
            }
        ) == {
            "betas": [{"alphas": [{"id": 1}, {"id": 2}]}, {"alphas": [{"id": 3}]}]
        }
>       assert model.dict(
            exclude={"betas": {"__all__": {"alphas": {"__all__": {"name"}}}}}
        ) == {
            "betas": [{"alphas": [{"id": 1}, {"id": 2}]}, {"alphas": [{"id": 3}]}]
        }
E       AssertionError: assert {'betas': [{}, {}]} == {'betas': [{'...[{'id': 3}]}]}
E         Differing items:
E         {'betas': [{}, {}]} != {'betas': [{'alphas': [{'id': 1}, {'id': 2}]}, {'alphas': [{'id': 3}]}]}
E         Use -v to get the full diff

example.py:44: AssertionError

 example.py ⨯                                                                                                                                                                  100% ██████████
================================================================================== short test summary info ===================================================================================
FAILED example.py::test - AssertionError: assert {'betas': [{}, {}]} == {'betas': [{'...[{'id': 3}]}]}

Results (0.12s):
       1 failed
         - example.py:19 test

As I mentioned, the problem seems to occur when using __all__ as the index. I think I have found the solution and can submit a PR, once I've written tests to cover these as well.

@pacific0009
Copy link

@pacific0009 pacific0009 commented Jun 1, 2020 •
edited 

from typing import List

from pydantic import BaseModel


class Option(BaseModel):
    text: str
   data: str


class Question(BaseModel):
    options: List[Option]


class Item(BaseModel):
    questions: List[Question]
item = Item(questions =[Question(options=[Option(text='x', data='y')])]


item.dict(exclude={'questions': {'0': {'options': {'__all__': {'data'}}}}})

above is working as expected but

item.dict(exclude={'questions': {'__all__': {'options': {'__all__': {'data'}}}}})

excludes entire option key for each entry in questions instead of excluding only correctAnswer key from each option

@xspirus xspirus mentioned this issue Jun 2, 2020
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

Fix nested all include exclude
3 participants
@samuelcolvin @pacific0009 @xspirus