dbt==0.20.1 commands break on python 3.9.7 #3843

tkoncz · 2021-09-01T15:18:21Z

Many dbt commands (dbt run, dbt test, ...) are breaking on Python 3.9.7 released yesterday: https://docs.python.org/release/3.9.7/whatsnew/changelog.html.

Here is a traceback:

2021-09-01 14:55:10.133377 (MainThread): Running with dbt=0.20.1
2021-09-01 14:55:11.273707 (MainThread): Encountered an error:
2021-09-01 14:55:11.273998 (MainThread): Profile() takes no arguments
2021-09-01 14:55:11.275664 (MainThread): Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/dbt/main.py", line 125, in main
    results, succeeded = handle_and_check(args)
  File "/usr/local/lib/python3.9/site-packages/dbt/main.py", line 203, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/lib/python3.9/site-packages/dbt/main.py", line 242, in run_from_args
    task = parsed.cls.from_args(args=parsed)
  File "/usr/local/lib/python3.9/site-packages/dbt/task/base.py", line 157, in from_args
    return super().from_args(args)
  File "/usr/local/lib/python3.9/site-packages/dbt/task/base.py", line 75, in from_args
    config = cls.ConfigType.from_args(args)
  File "/usr/local/lib/python3.9/site-packages/dbt/config/runtime.py", line 233, in from_args
    project, profile = cls.collect_parts(args)
  File "/usr/local/lib/python3.9/site-packages/dbt/config/runtime.py", line 211, in collect_parts
    profile = cls._get_rendered_profile(
  File "/usr/local/lib/python3.9/site-packages/dbt/config/runtime.py", line 190, in _get_rendered_profile
    return Profile.render_from_args(
  File "/usr/local/lib/python3.9/site-packages/dbt/config/profile.py", line 423, in render_from_args
    return cls.from_raw_profiles(
  File "/usr/local/lib/python3.9/site-packages/dbt/config/profile.py", line 388, in from_raw_profiles
    return cls.from_raw_profile_info(
  File "/usr/local/lib/python3.9/site-packages/dbt/config/profile.py", line 338, in from_raw_profile_info
    return cls.from_credentials(
  File "/usr/local/lib/python3.9/site-packages/dbt/config/profile.py", line 242, in from_credentials
    profile = cls(
TypeError: Profile() takes no arguments

For more details please see the following Slack thread: https://getdbt.slack.com/archives/C99SNSRTK/p1630506829031300

The text was updated successfully, but these errors were encountered:

shr1k · 2021-09-01T15:52:18Z

Can confirm, I faced the same issue. I wonder if it's this change in the Python 3.9.7 release causing the breakage?

bpo-44806: Non-protocol subclasses of typing.Protocol ignore now the init method inherited from protocol base classes. And the PR in question.

The Profile class in dbt ultimately inherits from a Protocol class, and I guess something's going wrong with the dataclass-generated __init__ there?

IANAPG (I Am Not A Python Guru) so I'm mostly just guessing though! 🤷‍♂️

tnightengale · 2021-09-01T20:00:09Z

Surprised there are not more people here... 👀

EDIT: Confirmed that pinning to Python 3.9.6 solved the issue. All credit to the linked Slack thread above!

julianfortune · 2021-09-01T20:52:38Z

I'm having the same issue in a different code base.

I feel confident the change linked in the issue body are the problem, but I don't understand the typing codebase enough to be certain.

I created this minimal example:

from dataclasses import dataclass
from typing import Protocol

class P(Protocol):
    pass

@dataclass
class B(P):
    value: str

print(B("test"))

In 3.9.7:

Traceback (most recent call last):
  File "test.py", line 11, in <module>
    print(B("test"))
TypeError: B() takes no arguments

In 3.9.6:

B(value='test')

julianfortune · 2021-09-01T21:09:40Z

Python Issue

jtcohen6 · 2021-09-02T07:05:59Z

Thanks for jumping into this one, everyone! And thanks for opening the python issue, @julianfortune.

I'm not yet sure whether this represents:

an unintended/unsupported mechanism of protocol inheritance/instantiation, in which case, we should refactor the places where we use Protocol (there's a lot of them)
a regression in python 3.9.7 that will be fixed in a future patch release, at which point we should update python_requires to mark 3.9.7 as unsupported

uriyyo · 2021-09-02T10:19:14Z

As a temporary workaround you can patch dataclasses._set_new_attribute method:

if sys.version_info >= (3, 9, 7):
    import dataclasses

    original_set_new_attr = dataclasses._set_new_attribute


    def patched_set_new_attr(cls, name, value):
        if (
            name == '__init__'
            and any(getattr(c, '_is_protocol', False) for c in cls.__bases__)
            and cls.__init__ is object.__init__
        ):
            del cls.__init__

        return original_set_new_attr(cls, name, value)

    dataclasses._set_new_attribute = patched_set_new_attr

julianfortune · 2021-09-02T13:59:29Z

@uriyyo ’s cpython patch

julianfortune · 2021-09-02T14:01:22Z

@uriyyo Thank you for the workaround, the fix, and a new test case to prevent the regression in the future! 🎉

Startouf · 2021-09-29T09:47:41Z

@julianfortune it's not clear, is it possible to fix the issue now by upgrading dbt or python 3.9 ? Or are we waiting for either dbt or python one to make a new public release ?

If I want to patch manually, where can I find dataclasses._set_new_attribute ? I'm not used to patching classes in python, where can I drop the patch ?

shr1k · 2021-09-29T09:54:37Z

@Startouf -- the latest patch release of dbt implements a workaround (#3855), so you can upgrade dbt to 0.20.2 and not worry about the Python bug.

tkoncz added bug triage labels Sep 1, 2021

julianfortune mentioned this issue Sep 1, 2021

dataclasses that inherit from Protocol subclasses have wrong __init__ python/typing#883

Closed

jtcohen6 removed the triage label Sep 2, 2021

leahwicz mentioned this issue Sep 2, 2021

Excluding Python 3.9.7 #3853

Closed

4 tasks

kwigley mentioned this issue Sep 7, 2021

explicitly define __init__ for Profile class #3855

Merged

4 tasks

kwigley closed this in #3855 Sep 7, 2021

kwigley mentioned this issue Sep 7, 2021

explicitly define __init__ for Profile class #3863

Merged

4 tasks

SamuelGabriel mentioned this issue Nov 5, 2021

Dependencies deepmind/neural_testbed#3

Open

dbt-labs / dbt-core Public

dbt==0.20.1 commands break on python 3.9.7 #3843

dbt==0.20.1 commands break on python 3.9.7 #3843

tkoncz commented Sep 1, 2021

shr1k commented Sep 1, 2021 •

edited

tnightengale commented Sep 1, 2021 •

edited

julianfortune commented Sep 1, 2021

julianfortune commented Sep 1, 2021

jtcohen6 commented Sep 2, 2021

uriyyo commented Sep 2, 2021 •

edited

julianfortune commented Sep 2, 2021

julianfortune commented Sep 2, 2021

Startouf commented Sep 29, 2021 •

edited

shr1k commented Sep 29, 2021

dbt-labs / dbt-core Public

dbt==0.20.1 commands break on python 3.9.7 #3843

dbt==0.20.1 commands break on python 3.9.7 #3843

Comments

tkoncz commented Sep 1, 2021

shr1k commented Sep 1, 2021 • edited

tnightengale commented Sep 1, 2021 • edited

julianfortune commented Sep 1, 2021

julianfortune commented Sep 1, 2021

jtcohen6 commented Sep 2, 2021

uriyyo commented Sep 2, 2021 • edited

julianfortune commented Sep 2, 2021

julianfortune commented Sep 2, 2021

Startouf commented Sep 29, 2021 • edited

shr1k commented Sep 29, 2021

shr1k commented Sep 1, 2021 •

edited

tnightengale commented Sep 1, 2021 •

edited

uriyyo commented Sep 2, 2021 •

edited

Startouf commented Sep 29, 2021 •

edited