Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Disallow direct item access of NotRequired TypedDict properties: … #12095

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Disallow direct item access of NotRequired TypedDict properties: … #12095

wants to merge 4 commits into from
+298 −100

Conversation

@bsmedberg-xometry
Copy link

@bsmedberg-xometry bsmedberg-xometry commented Jan 28, 2022

Description

Disallow direct item access of NotRequired TypedDict properties: these should

always be accessed through .get() because the keys may not be present.

Fixes #12094

Test Plan

Includes unit test coverage for both item access (getitem as an rvalue) and item setting (setitem as an lvalue).

Two existing tests required updates because they partially exercized the now-forbidden codepath using reveal_type.
@bsmedberg-xometry
Disallow direct item access of NotRequired TypedDict properties: thes…
cc9615a 
…e should

always be accessed through .get() because the keys may not be present.

Fixes python#12094
@JelleZijlstra
Copy link
Member

@JelleZijlstra JelleZijlstra commented Jan 28, 2022

cc @davidfstr

@bsmedberg-xometry
Copy link
Author

@bsmedberg-xometry bsmedberg-xometry commented Jan 28, 2022

I hadn't run the mypyc tests before submitting because they weren't part of the default runtests.py setup. I can reproduce the error locally but it seems that mypyc has an unexpected behavior. In mypyc it seems that a typeddict doesn't have a .get() method? At least when I tried to replace d3["c"] with d3.get("c") I get an error back: error: "TD3" has no attribute "get"

@github-actions

This comment has been hidden.

Sign in to view
@bsmedberg-xometry
Copy link
Author

@bsmedberg-xometry bsmedberg-xometry commented Jan 28, 2022

I will come back to this next week to fix .get(), but I think I also need advice on whether this will adversely affect the python ecosystem.

@bsmedberg-xometry
Add stronger type-checking to TypedDict.get such that:
37f9e7b 
1. invalid keys type-check as the default value type (None or the provided
   default)
2. required keys type-check just to the declared type, never the fallback
3. optional keys type-check to the declared type or the default value type

Fixes tests to match the better typecheck results.
@github-actions
Copy link

@github-actions github-actions bot commented Jan 30, 2022

Diff from mypy_primer, showing the effect of this PR on open source code:

graphql-core (https://github.com/graphql-python/graphql-core)
- src/graphql/utilities/extend_schema.py:644: error: Unused "type: ignore" comment
+ src/graphql/utilities/extend_schema.py:204: error: TypedDict "GraphQLDirectiveKwargs" key "args" is not required.
+ src/graphql/utilities/extend_schema.py:239: error: TypedDict "GraphQLInputObjectTypeKwargs" key "name" is not required.
+ src/graphql/utilities/extend_schema.py:252: error: TypedDict "GraphQLInputObjectTypeKwargs" key "fields" is not required.
+ src/graphql/utilities/extend_schema.py:256: error: TypedDict "GraphQLInputObjectTypeKwargs" key "extension_ast_nodes" is not required.
+ src/graphql/utilities/extend_schema.py:262: error: TypedDict "GraphQLEnumTypeKwargs" key "name" is not required.
+ src/graphql/utilities/extend_schema.py:267: error: TypedDict "GraphQLEnumTypeKwargs" key "values" is not required.
+ src/graphql/utilities/extend_schema.py:268: error: TypedDict "GraphQLEnumTypeKwargs" key "extension_ast_nodes" is not required.
+ src/graphql/utilities/extend_schema.py:274: error: TypedDict "GraphQLScalarTypeKwargs" key "name" is not required.
+ src/graphql/utilities/extend_schema.py:276: error: TypedDict "GraphQLScalarTypeKwargs" key "specified_by_url" is not required.
+ src/graphql/utilities/extend_schema.py:284: error: TypedDict "GraphQLScalarTypeKwargs" key "extension_ast_nodes" is not required.
+ src/graphql/utilities/extend_schema.py:291: error: TypedDict "GraphQLObjectTypeKwargs" key "name" is not required.
+ src/graphql/utilities/extend_schema.py:298: error: TypedDict "GraphQLObjectTypeKwargs" key "interfaces" is not required.
+ src/graphql/utilities/extend_schema.py:304: error: TypedDict "GraphQLObjectTypeKwargs" key "fields" is not required.
+ src/graphql/utilities/extend_schema.py:308: error: TypedDict "GraphQLObjectTypeKwargs" key "extension_ast_nodes" is not required.
+ src/graphql/utilities/extend_schema.py:315: error: TypedDict "GraphQLInterfaceTypeKwargs" key "name" is not required.
+ src/graphql/utilities/extend_schema.py:322: error: TypedDict "GraphQLInterfaceTypeKwargs" key "interfaces" is not required.
+ src/graphql/utilities/extend_schema.py:328: error: TypedDict "GraphQLInterfaceTypeKwargs" key "fields" is not required.
+ src/graphql/utilities/extend_schema.py:332: error: TypedDict "GraphQLInterfaceTypeKwargs" key "extension_ast_nodes" is not required.
+ src/graphql/utilities/extend_schema.py:338: error: TypedDict "GraphQLUnionTypeKwargs" key "name" is not required.
+ src/graphql/utilities/extend_schema.py:345: error: TypedDict "GraphQLUnionTypeKwargs" key "types" is not required.
+ src/graphql/utilities/extend_schema.py:348: error: TypedDict "GraphQLUnionTypeKwargs" key "extension_ast_nodes" is not required.
+ src/graphql/utilities/extend_schema.py:635: error: TypedDict "GraphQLSchemaKwargs" key "types" is not required.
+ src/graphql/utilities/extend_schema.py:661: error: TypedDict "GraphQLSchemaKwargs" key "directives" is not required.
+ src/graphql/utilities/extend_schema.py:668: error: TypedDict "GraphQLSchemaKwargs" key "ast_node" is not required.
+ src/graphql/utilities/extend_schema.py:669: error: TypedDict "GraphQLSchemaKwargs" key "extension_ast_nodes" is not required.
+ src/graphql/utilities/build_ast_schema.py:61: error: TypedDict "GraphQLSchemaKwargs" key "ast_node" is not required.
+ src/graphql/utilities/build_ast_schema.py:62: error: TypedDict "GraphQLSchemaKwargs" key "types" is not required.
+ src/graphql/utilities/build_ast_schema.py:75: error: TypedDict "GraphQLSchemaKwargs" key "directives" is not required.
+ tests/type/test_extensions.py:34: error: TypedDict "GraphQLScalarTypeKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:41: error: TypedDict "GraphQLScalarTypeKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:66: error: TypedDict "GraphQLObjectTypeKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:97: error: TypedDict "GraphQLObjectTypeKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:130: error: TypedDict "GraphQLInterfaceTypeKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:161: error: TypedDict "GraphQLInterfaceTypeKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:177: error: TypedDict "GraphQLUnionTypeKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:186: error: TypedDict "GraphQLUnionTypeKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:202: error: TypedDict "GraphQLEnumTypeKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:203: error: TypedDict "GraphQLEnumValueKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:219: error: TypedDict "GraphQLEnumTypeKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:220: error: TypedDict "GraphQLEnumValueKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:243: error: TypedDict "GraphQLInputObjectTypeKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:265: error: TypedDict "GraphQLInputObjectTypeKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:288: error: TypedDict "GraphQLDirectiveKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:289: error: TypedDict "GraphQLArgumentKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:306: error: TypedDict "GraphQLDirectiveKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:307: error: TypedDict "GraphQLArgumentKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:320: error: TypedDict "GraphQLSchemaKwargs" key "extensions" is not required.
+ tests/type/test_extensions.py:329: error: TypedDict "GraphQLSchemaKwargs" key "extensions" is not required.
+ tests/type/test_definition.py:88: error: TypedDict "GraphQLScalarTypeKwargs" key "serialize" is not required.
+ tests/type/test_definition.py:94: error: TypedDict "GraphQLScalarTypeKwargs" key "description" is not required.
+ tests/type/test_definition.py:100: error: TypedDict "GraphQLScalarTypeKwargs" key "specified_by_url" is not required.
+ tests/type/test_definition.py:116: error: TypedDict "GraphQLScalarTypeKwargs" key "parse_value" is not required.
+ tests/type/test_definition.py:117: error: TypedDict "GraphQLScalarTypeKwargs" key "parse_literal" is not required.
+ tests/type/test_definition.py:130: error: TypedDict "GraphQLScalarTypeKwargs" key "serialize" is not required.
+ tests/type/test_definition.py:131: error: TypedDict "GraphQLScalarTypeKwargs" key "parse_value" is not required.
+ tests/type/test_definition.py:132: error: TypedDict "GraphQLScalarTypeKwargs" key "parse_literal" is not required.
+ tests/type/test_definition.py:286: error: TypedDict "GraphQLFieldKwargs" key "args" is not required.
+ tests/type/test_definition.py:300: error: TypedDict "GraphQLFieldKwargs" key "description" is not required.
+ tests/type/test_definition.py:306: error: TypedDict "GraphQLFieldKwargs" key "deprecation_reason" is not required.
+ tests/type/test_definition.py:1111: error: TypedDict "GraphQLEnumTypeKwargs" key "description" is not required.
+ tests/type/test_definition.py:1361: error: TypedDict "GraphQLEnumValueKwargs" key "value" is not required.
+ tests/type/test_definition.py:1367: error: TypedDict "GraphQLEnumValueKwargs" key "value" is not required.
+ tests/type/test_definition.py:1373: error: TypedDict "GraphQLEnumValueKwargs" key "description" is not required.
+ tests/type/test_definition.py:1379: error: TypedDict "GraphQLEnumValueKwargs" key "deprecation_reason" is not required.
+ tests/type/test_definition.py:1425: error: TypedDict "GraphQLInputObjectTypeKwargs" key "description" is not required.
+ tests/type/test_definition.py:1431: error: TypedDict "GraphQLInputObjectTypeKwargs" key "out_type" is not required.
+ tests/type/test_definition.py:1437: error: TypedDict "GraphQLInputObjectTypeKwargs" key "out_type" is not required.
+ tests/type/test_definition.py:1659: error: TypedDict "GraphQLArgumentKwargs" key "description" is not required.
+ tests/type/test_definition.py:1666: error: TypedDict "GraphQLArgumentKwargs" key "out_name" is not required.
+ tests/type/test_definition.py:1672: error: TypedDict "GraphQLArgumentKwargs" key "out_name" is not required.
+ tests/type/test_definition.py:1678: error: TypedDict "GraphQLArgumentKwargs" key "ast_node" is not required.
+ tests/type/test_definition.py:1724: error: TypedDict "GraphQLInputFieldKwargs" key "description" is not required.
+ tests/type/test_definition.py:1731: error: TypedDict "GraphQLInputFieldKwargs" key "out_name" is not required.
+ tests/type/test_definition.py:1737: error: TypedDict "GraphQLInputFieldKwargs" key "out_name" is not required.
+ tests/type/test_definition.py:1743: error: TypedDict "GraphQLArgumentKwargs" key "ast_node" is not required.
+ tests/type/test_definition.py:1794: error: TypedDict "GraphQLInputObjectTypeKwargs" key "fields" is not required.
+ tests/utilities/test_build_client_schema.py:628: error: TypedDict "GraphQLSchemaKwargs" key "assume_valid" is not required.

paasta (https://github.com/yelp/paasta)
+ paasta_tools/instance/hpa_metrics_parser.py:35: error: TypedDict "HPAMetricsDict" key "target_value" is not required.
+ paasta_tools/instance/hpa_metrics_parser.py:60: error: TypedDict "HPAMetricsDict" key "current_value" is not required.
+ paasta_tools/utils.py:2050: error: TypedDict "SystemPaastaConfigDict" key "zookeeper" is not required.
+ paasta_tools/utils.py:2068: error: TypedDict "SystemPaastaConfigDict" key "docker_registry" is not required.
+ paasta_tools/utils.py:2081: error: TypedDict "SystemPaastaConfigDict" key "hacheck_sidecar_volumes" is not required.
+ paasta_tools/utils.py:2095: error: TypedDict "SystemPaastaConfigDict" key "volumes" is not required.
+ paasta_tools/utils.py:2107: error: TypedDict "SystemPaastaConfigDict" key "cluster" is not required.
+ paasta_tools/utils.py:2114: error: TypedDict "SystemPaastaConfigDict" key "dashboard_links" is not required.
+ paasta_tools/utils.py:2130: error: TypedDict "SystemPaastaConfigDict" key "api_endpoints" is not required.
+ paasta_tools/utils.py:2203: error: TypedDict "SystemPaastaConfigDict" key "log_writer" is not required.
+ paasta_tools/utils.py:2216: error: TypedDict "SystemPaastaConfigDict" key "log_reader" is not required.
+ paasta_tools/long_running_service_tools.py:217: error: TypedDict "LongRunningServiceConfigDict" key "nerve_ns" is not required.
+ paasta_tools/drain_lib.py:232: error: TypedDict "SpoolInfo" key "state" is not required.
+ paasta_tools/drain_lib.py:236: error: TypedDict "SpoolInfo" key "state" is not required.
+ paasta_tools/drain_lib.py:308: error: TypedDict "UrlSpec" key "url_format" is not required.
+ paasta_tools/drain_lib.py:325: error: TypedDict "UrlSpec" key "success_codes" is not required.
+ paasta_tools/drain_lib.py:328: error: TypedDict "UrlSpec" key "success_codes" is not required.
+ paasta_tools/kubernetes_tools.py:643: error: TypedDict "AutoscalingParamsDict" key "decision_policy" is not required.
+ paasta_tools/kubernetes_tools.py:654: error: TypedDict "AutoscalingParamsDict" key "metrics_provider" is not required.
+ paasta_tools/kubernetes_tools.py:656: error: TypedDict "AutoscalingParamsDict" key "setpoint" is not required.
+ paasta_tools/kubernetes_tools.py:838: error: TypedDict "SecretVolume" key "secret_name" is not required.
+ paasta_tools/kubernetes_tools.py:969: error: TypedDict "AutoscalingParamsDict" key "metrics_provider" is not required.
+ paasta_tools/kubernetes_tools.py:981: error: TypedDict "AutoscalingParamsDict" key "metrics_provider" is not required.
+ paasta_tools/kubernetes_tools.py:1101: error: TypedDict "KubeContainerResourceRequest" key "cpu" is not required.
+ paasta_tools/kubernetes_tools.py:1102: error: TypedDict "KubeContainerResourceRequest" key "memory" is not required.
+ paasta_tools/kubernetes_tools.py:1103: error: TypedDict "KubeContainerResourceRequest" key "ephemeral-storage" is not required.
+ paasta_tools/kubernetes_tools.py:1264: error: TypedDict "SecretVolume" key "secret_name" is not required.
+ paasta_tools/kubernetes_tools.py:1268: error: TypedDict "SecretVolumeItem" key "key" is not required.
+ paasta_tools/kubernetes_tools.py:1270: error: TypedDict "SecretVolumeItem" key "path" is not required.
+ paasta_tools/kubernetes_tools.py:1272: error: TypedDict "SecretVolume" key "items" is not required.
+ paasta_tools/kubernetes_tools.py:1353: error: TypedDict "SecretVolume" key "container_path" is not required.
+ paasta_tools/kubernetes_tools.py:1635: error: TypedDict "AutoscalingParamsDict" key "metrics_provider" is not required.
+ paasta_tools/setup_prometheus_adapter_config.py:193: error: TypedDict "AutoscalingParamsDict" key "metrics_provider" is not required.
+ paasta_tools/setup_prometheus_adapter_config.py:209: error: TypedDict "AutoscalingParamsDict" key "metrics_provider" is not required.
+ paasta_tools/setup_prometheus_adapter_config.py:223: error: TypedDict "AutoscalingParamsDict" key "setpoint" is not required.
+ paasta_tools/setup_prometheus_adapter_config.py:315: error: TypedDict "AutoscalingParamsDict" key "setpoint" is not required.
+ paasta_tools/setup_prometheus_adapter_config.py:399: error: TypedDict "AutoscalingParamsDict" key "metrics_provider" is not required.
+ paasta_tools/setup_prometheus_adapter_config.py:562: error: TypedDict "AutoscalingParamsDict" key "metrics_provider" is not required.
+ paasta_tools/setup_prometheus_adapter_config.py:573: error: TypedDict "AutoscalingParamsDict" key "prometheus_adapter_config" is not required.
+ paasta_tools/marathon_tools.py:703: error: TypedDict "FormattedMarathonAppDict" key "container" is not required.
+ paasta_tools/marathon_tools.py:704: error: TypedDict "FormattedMarathonAppDict" key "container" is not required.
+ paasta_tools/marathon_tools.py:759: error: TypedDict "FormattedMarathonAppDict" key "env" is not required.
+ paasta_tools/envoy_tools.py:78: error: TypedDict "EnvoyBackend" key "address" is not required.
+ paasta_tools/envoy_tools.py:79: error: TypedDict "EnvoyBackend" key "port_value" is not required.
+ paasta_tools/envoy_tools.py:80: error: TypedDict "EnvoyBackend" key "eds_health_status" is not required.
+ paasta_tools/envoy_tools.py:290: error: TypedDict "EnvoyBackend" key "address" is not required.
+ paasta_tools/envoy_tools.py:325: error: TypedDict "EnvoyBackend" key "address" is not required.
+ paasta_tools/envoy_tools.py:326: error: TypedDict "EnvoyBackend" key "port_value" is not required.
+ paasta_tools/envoy_tools.py:359: error: TypedDict "EnvoyBackend" key "eds_health_status" is not required.
+ paasta_tools/envoy_tools.py:364: error: TypedDict "EnvoyBackend" key "address" is not required.
+ paasta_tools/envoy_tools.py:364: error: TypedDict "EnvoyBackend" key "port_value" is not required.
+ paasta_tools/envoy_tools.py:402: error: TypedDict "EnvoyBackend" key "eds_health_status" is not required.
+ paasta_tools/deployd/common.py:106: error: TypedDict "FormattedMarathonAppDict" key "id" is not required.
+ paasta_tools/smartstack_tools.py:273: error: TypedDict "HaproxyBackend" key "pxname" is not required.
+ paasta_tools/smartstack_tools.py:305: error: TypedDict "HaproxyBackend" key "pxname" is not required.
+ paasta_tools/smartstack_tools.py:316: error: TypedDict "HaproxyBackend" key "status" is not required.
+ paasta_tools/smartstack_tools.py:368: error: TypedDict "HaproxyBackend" key "status" is not required.
+ paasta_tools/smartstack_tools.py:393: error: TypedDict "HaproxyBackend" key "svname" is not required.
+ paasta_tools/smartstack_tools.py:403: error: TypedDict "HaproxyBackend" key "pxname" is not required.
+ paasta_tools/smartstack_tools.py:427: error: TypedDict "HaproxyBackend" key "svname" is not required.
+ paasta_tools/smartstack_tools.py:463: error: TypedDict "HaproxyBackend" key "svname" is not required.
+ paasta_tools/smartstack_tools.py:783: error: TypedDict "HaproxyBackend" key "svname" is not required.
+ paasta_tools/smartstack_tools.py:795: error: TypedDict "HaproxyBackend" key "status" is not required.
+ paasta_tools/smartstack_tools.py:796: error: TypedDict "HaproxyBackend" key "check_status" is not required.
+ paasta_tools/smartstack_tools.py:797: error: TypedDict "HaproxyBackend" key "check_code" is not required.
+ paasta_tools/smartstack_tools.py:798: error: TypedDict "HaproxyBackend" key "lastchg" is not required.
+ paasta_tools/smartstack_tools.py:802: error: TypedDict "HaproxyBackend" key "check_duration" is not required.
+ paasta_tools/instance/kubernetes.py:154: error: TypedDict "HPAMetricsDict" key "name" is not required.
+ paasta_tools/instance/kubernetes.py:160: error: TypedDict "HPAMetricsDict" key "name" is not required.
+ paasta_tools/instance/kubernetes.py:368: error: TypedDict "EnvoyBackend" key "eds_health_status" is not required.
+ paasta_tools/instance/kubernetes.py:371: error: TypedDict "EnvoyBackend" key "address" is not required.
+ paasta_tools/instance/kubernetes.py:371: error: TypedDict "EnvoyBackend" key "port_value" is not required.
+ paasta_tools/instance/kubernetes.py:405: error: TypedDict "HaproxyBackend" key "status" is not required.
+ paasta_tools/setup_marathon_job.py:657: error: TypedDict "FormattedMarathonAppDict" key "id" is not required.
+ paasta_tools/setup_marathon_job.py:893: error: TypedDict "FormattedMarathonAppDict" key "id" is not required.
+ paasta_tools/check_kubernetes_services_replication.py:105: error: TypedDict "KubernetesDeploymentConfigDict" key "monitoring" is not required.
+ paasta_tools/check_kubernetes_services_replication.py:107: error: TypedDict "KubernetesDeploymentConfigDict" key "monitoring" is not required.
+ paasta_tools/autoscaling/autoscaling_service_lib.py:680: error: TypedDict "AutoscalingParamsDict" key "setpoint" is not required.
+ paasta_tools/autoscaling/autoscaling_service_lib.py:830: error: TypedDict "AutoscalingParamsDict" key "decision_policy" is not required.
+ paasta_tools/api/views/instance.py:244: error: TypedDict "FormattedMarathonAppDict" key "id" is not required.
+ paasta_tools/api/views/instance.py:362: error: TypedDict "HaproxyBackend" key "status" is not required.
+ paasta_tools/api/views/instance.py:394: error: TypedDict "EnvoyBackend" key "eds_health_status" is not required.
+ paasta_tools/api/views/instance.py:397: error: TypedDict "EnvoyBackend" key "address" is not required.
+ paasta_tools/api/views/instance.py:397: error: TypedDict "EnvoyBackend" key "port_value" is not required.

pydantic (https://github.com/samuelcolvin/pydantic)
+ pydantic/networks.py:229: error: TypedDict "Parts" key "scheme" is not required.  [typeddict-item-access]
+ pydantic/networks.py:230: error: TypedDict "Parts" key "user" is not required.  [typeddict-item-access]
+ pydantic/networks.py:231: error: TypedDict "Parts" key "password" is not required.  [typeddict-item-access]
+ pydantic/networks.py:235: error: TypedDict "Parts" key "port" is not required.  [typeddict-item-access]
+ pydantic/networks.py:236: error: TypedDict "Parts" key "path" is not required.  [typeddict-item-access]
+ pydantic/networks.py:237: error: TypedDict "Parts" key "query" is not required.  [typeddict-item-access]
+ pydantic/networks.py:238: error: TypedDict "Parts" key "fragment" is not required.  [typeddict-item-access]
+ pydantic/networks.py:247: error: TypedDict "Parts" key "scheme" is not required.  [typeddict-item-access]
+ pydantic/networks.py:254: error: TypedDict "Parts" key "port" is not required.  [typeddict-item-access]
+ pydantic/networks.py:258: error: TypedDict "Parts" key "user" is not required.  [typeddict-item-access]
+ pydantic/networks.py:334: error: TypedDict "Parts" key "scheme" is not required.  [typeddict-item-access]
+ pydantic/networks.py:368: error: TypedDict "Parts" key "ipv4" is not required.  [typeddict-item-access]
+ pydantic/networks.py:368: error: TypedDict "Parts" key "ipv6" is not required.  [typeddict-item-access]

steam.py (https://github.com/Gobot1234/steam.py)
+ steam/trade.py:259: error: TypedDict "ItemDict" key "name_color" is not required.  [typeddict-item-access]
+ steam/trade.py:264: error: TypedDict "ItemDict" key "icon_url_large" is not required.  [typeddict-item-access]

urllib3 (https://github.com/urllib3/urllib3)
+ test/test_ssltransport.py:99: error: TypedDict "_TYPE_PEER_CERT_RET_DICT" key "serialNumber" is not required.  [typeddict-item-access]

cwltool (https://github.com/common-workflow-language/cwltool)
+ cwltool/provenance.py: note: In member "_ro_aggregates" of class "ResearchObject":
+ cwltool/provenance.py:541:44: error: TypedDict "Aggregate" key "bundledAs" is not required.  [typeddict-item-access]

zulip (https://github.com/zulip/zulip)
+ zerver/lib/streams.py:179: error: TypedDict "StreamDict" key "name" is not required.  [typeddict-item-access]
+ zerver/lib/streams.py:642: error: TypedDict "StreamDict" key "name" is not required.  [typeddict-item-access]
+ zerver/lib/streams.py:667: error: TypedDict "StreamDict" key "name" is not required.  [typeddict-item-access]
+ zerver/lib/streams.py:674: error: TypedDict "StreamDict" key "is_web_public" is not required.  [typeddict-item-access]
+ zerver/lib/streams.py:695: error: TypedDict "StreamDict" key "name" is not required.  [typeddict-item-access]
+ zproject/backends.py:1275: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zproject/backends.py:1279: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/views/auth.py:642: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/views/auth.py:650: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1087: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1088: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1089: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1090: error: TypedDict "ExternalAuthDataDict" key "redirect_to" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1111: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1112: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1113: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1114: error: TypedDict "ExternalAuthDataDict" key "redirect_to" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1331: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1333: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1334: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1358: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1359: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1360: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1596: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1597: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1598: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1620: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1621: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:1622: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2597: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2598: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2599: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2632: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2633: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2634: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2653: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2654: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2655: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2700: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2701: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2702: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2723: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2724: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2725: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2803: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2804: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2805: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2852: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2853: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:2854: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:3585: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:3586: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:3587: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:3622: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:3623: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:3624: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:3648: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:3649: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:3650: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:3651: error: TypedDict "ExternalAuthDataDict" key "redirect_to" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:3673: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:3674: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:3675: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:3676: error: TypedDict "ExternalAuthDataDict" key "redirect_to" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:3702: error: TypedDict "ExternalAuthDataDict" key "email" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:3703: error: TypedDict "ExternalAuthDataDict" key "full_name" is not required.  [typeddict-item-access]
+ zerver/tests/test_auth_backends.py:3704: error: TypedDict "ExternalAuthDataDict" key "subdomain" is not required.  [typeddict-item-access]
+ z```

@bsmedberg-xometry
Copy link
Author

@bsmedberg-xometry bsmedberg-xometry commented Jan 31, 2022

@JukkaL I see that you reviewed and approved the previous work on this at #10370 and before that related #9906 - please let me know if you're the right person to review or provide guidance, now that I have the automated tests passing 100%

@thomkeh
Copy link

@thomkeh thomkeh commented Feb 1, 2022

One of the errors in https://github.com/Gobot1234/steam.py is on this line:

self.colour = int(data["name_color"], 16) if "name_color" in data else None

but surely that is valid?

@bsmedberg-xometry
Copy link
Author

@bsmedberg-xometry bsmedberg-xometry commented Feb 1, 2022

That's a good point! From an abstract type-checking perspective, the way I'd want to solve that is with a narrowing pass:
the if "name_color" in data clause would narrow the type of data from TypedDict({name_color?: int}) to TypedDict{{name_color: int})

However, I've not even looked at the way narrowing and inference works in mypy so I'd have to do some research on whether that is a huge or small engineering task.

@davidfstr
Copy link
Contributor

@davidfstr davidfstr commented Feb 1, 2022

One of the errors in https://github.com/Gobot1234/steam.py is on this line:

self.colour = int(data["name_color"], 16) if "name_color" in data else None

but surely that is valid?

If I recall correctly, type narrowing happens for if-statements but not for x-if-y-else-z ternary expressions (but probably should). Assuming that's true, the error could be avoided for now by rewriting as:

if "name_color" in data:
    self.colour = int(data["name_color"], 16)
else:
    self.colour = None

Type narrowing in ternary expressions I expect to be sufficiently difficult to implement that you would want a separate PR for that, if you were to pursue adding that support as well.

@bsmedberg-xometry
Narrow not-required fields into required fields when code does `'lite…
04b260a 
…ral' in typed_dict`.
@bsmedberg-xometry
Add documentation for the stricter item access on TypedDicts.
2c38fd3 
Also document the new and more-ergonomic way to mix required and not-required
items in a TypedDict using the NotRequired annotation.
@bsmedberg-xometry
Copy link
Author

@bsmedberg-xometry bsmedberg-xometry commented Feb 1, 2022

@davidfstr roger, understood.

The new commits I've just added implement the narrowing logic (with a test). I've also updated the documentation to match the new proposed behavior and document the more ergonomic NotRequired method of constructing TypedDict with mixed required/not-required elements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Linked issues

Successfully merging this pull request may close these issues.

Disallow x["foo"] for NotRequired TypedDict access
4 participants
@bsmedberg-xometry @JelleZijlstra @thomkeh @davidfstr