This is likely the most complex use case in this module. The Csv class exhibits default behavior that can be modified based on parameters passed into its __init__ method. By default, cast is set to str, and post_process is set to list. Therefore, if these parameters are explicitly defined, the types provided should be used. However, if they are not specified, the return type of the __call__ method should default to list[str].

If there is a better way to express this, I am open to suggestions.

I haven't looked deeply, but it sounds like you simply need a generic __init__ method.

I think this should be something along these lines. The arguments of cast/post_process are very much defined by the implementation and since they are optional parameters you will need to provide overloads, because otherwise _T/_TCsv will not get bound.

It's also worth noting that you will need to import Iterable to make this suggestion work.

Thank you for this suggestion! I did not realize you could suggest the types of _T and _Csv through the self argument. Really interesting stuff.

After I implemented your suggestions, I am getting some regressions on my code while using this type file. Would you know how to fix it? Also, I am wondering if the Iterable is having some unwanted side effects. Also, the errors I am seeing are coming from mypy, I haven't checked the other type checkers.

from decouple import Choices, Csv
from typing_extensions import reveal_type

# No error and correct type
reveal_type(Csv()(""))  # note: Revealed type is "builtins.list[builtins.str]"

# No error and correct type
reveal_type(Csv(cast=int)(""))  # note: Revealed type is "builtins.list[builtins.int]"

# error: Argument "post_process" to "Csv" has incompatible type "type[set[Any]]"; expected "Callable[[Iterable[str]], list[str]]"  [arg-type]
reveal_type(Csv(post_process=set)(""))  # note: Revealed type is "builtins.list[builtins.str]"

# error: Argument 1 to "Csv" has incompatible type "type[int]"; expected "Callable[[str], str]"  [arg-type]
# error: Argument 4 to "Csv" has incompatible type "type[tuple[Any, ...]]"; expected "Callable[[Iterable[str]], list[str]]"  [arg-type]#
reveal_type(Csv(int, ",", " ", tuple)(""))

# error: Argument "cast" to "Csv" has incompatible type "type[int]"; expected "Callable[[str], str]"  [arg-type]
# error: Argument "post_process" to "Csv" has incompatible type "type[tuple[Any, ...]]"; expected "Callable[[Iterable[str]], list[str]]"  [arg-type]
reveal_type(Csv(cast=int, post_process=tuple)(""))

# error: Argument "cast" to "Csv" has incompatible type "type[int]"; expected "Callable[[str], str]"  [arg-type]
# error: Argument "post_process" to "Csv" has incompatible type "type[tuple[Any, ...]]"; expected "Callable[[Iterable[str]], list[str]]"  [arg-type]
reveal_type(Csv(cast=int, delimiter=",", post_process=tuple)(""))

Yeah, I am not surprised there are some cases where mypy can't infer this correctly, since there's quite some complex matching logic involved and the one type var technically depends on the other, however there is no real way to express this relationship between the two, it would require higher kinded type vars.

The main complication stems from using callables like set, tuple as arguments for post_process, since they are generics in themselves, if you use callables with fixed arguments, then it will work no problem.

I've played around a bit on mypy-play, it needed one more overload to cover all the cases correctly, I've also tried a callback protocol, but that just resulted in very strange behavior of inferring Csv(post_process=set) as Csv[Any, list[Any]] which is very incorrect.

This is the closest I could get, it's not perfect, since it results in unbound type vars for your examples, so if you try to use the result of __call__ mypy will tell you to specify the type, since it can't fully infer it itself. I've also had to give up on some amount of type safety for post_process and define the input as Iterable[Any] rather than Iterable[_T].
https://mypy-play.net/?mypy=latest&python=3.11&gist=8aace8121128e574dae382a18dd3fc02

The main complication stems from using callables like set, tuple as arguments for post_process, since they are generics in themselves, if you use callables with fixed arguments, then it will work no problem.

I was trying to work through that as well, and I have come to same conclusion. However, at least there is a solution.

This is the closest I could get, it's not perfect, since it results in unbound type vars for your examples, so if you try to use the result of call mypy will tell you to specify the type, since it can't fully infer it itself. I've also had to give up on some amount of type safety for post_process and define the input as Iterable[Any] rather than Iterable[_T]. https://mypy-play.net/?mypy=latest&python=3.11&gist=8aace8121128e574dae382a18dd3fc02

I think I have incorporated all of your updates into the PR. Thanks for the suggestions.

I think this upper bound is too strict, I would change this to Collection[Any] at the very least, but technically this can be anything and the docs also state as much (you could e.g. imagine someone calculating an integer hash from the individual values by providing post_process=lambda x: hash(tuple(x))) so I'm not sure if it's worth providing an upper bound.

Yep, that's a great point! Updated!

Same deal here, this takes some overloads to bind correctly and the parameters of cast are given, and choices does need to be a sequence of 2 value tuples, the second value of which is always a string.

Technically you could make this generic in the input value type as well, but since it is meant for dealing with environment variables, it probably makes sense to stick with str as all the examples have as well.

This looks great! Thanks again for the suggestion.

I have one issue, it seems like when cast is used by itself mypy seems to get confused. Would you know of a way to fix that use case?

from decouple import Choices, Csv
from typing_extensions import reveal_type

# error: Argument "cast" to "Choices" has incompatible type "type[int]"; expected "Callable[[str], str]"  [arg-type]
reveal_type(Choices(cast=int)(""))

reveal_type(Choices(flat=[1, 2])("")) # note: Revealed type is "builtins.int"
reveal_type(Choices(choices=[(1, "one"), (2, "two")])("")) #note: Revealed type is "builtins.int"
reveal_type(Choices(cast=int, flat=[1, 2])("")) #note: Revealed type is "builtins.int"
reveal_type(Choices(cast=int, choices=[(1, "one"), (2, "two")])("")) #note: Revealed type is "builtins.int"
reveal_type(Choices(flat=[1, 2], choices=[(1, "one"), (2, "two")])("")) #note: Revealed type is "builtins.int"
reveal_type(Choices(cast=int, flat=[1, 2], choices=[(1, "one"), (2, "two")])("")) #note: Revealed type is "builtins.int"

This needs one additional overload to cover that case:

    @overload
    def __init__(
        self: Choices[_T],
        flat: Sequence[_T] | None = None,
        *,
        cast: Callable[[str], _T],
        choices: Sequence[tuple[_T, str]] | None = None,
    ) -> None: ...

generic overload matching gets complicated really quickly if you want to cover every possible case on arguments other than the first one or two...

I wish overloads would let you violate the requirement of positional optional parameters having to come after all the required ones, it would get rid of a lot of redundant overloads, that have to currently be used in order to express both the positional and keyword version of the overload.

But that would of course require some changes to the language, since the parser would have to ignore these violations specifically only on overloads, the alternative would be to have some typing specific sentinel value that means the parameter is required in order to match that overload, but the parser can still treat it like an optional one. But I am getting off course here.

generic overload matching gets complicated really quickly if you want to cover every possible case on arguments other than the first one or two...

Yeah, I know what you mean.

I wish overloads would let you violate the requirement of positional optional parameters having to come after all the required ones, it would get rid of a lot of redundant overloads, that have to currently be used in order to express both the positional and keyword version of the overload.

But that would of course require some changes to the language, since the parser would have to ignore these violations specifically only on overloads, the alternative would be to have some typing specific sentinel value that means the parameter is required in order to match that overload, but the parser can still treat it like an optional one. But I am getting off course here.

TBH, this is how I thought it would work but the more I dug into making this type module, the more I realized that wasn't the case. But, now I know and thanks for the background!

Shouldn't the Any be _T as the post-processor gets passed whatever came out of cast?

Yes, that was my first suggestion, and we tried that, however mypy will then reject generic classes as a post_process since it can't seem to solve for both type vars simultaneously i.e. it will reject post_process=set but it would accept post_process=set[str].

I agree that this should work, but it's also a very challenging scenario for an overload solver, so I am not surprised mypy can't do it yet.

Should I add a comment that "it should be _T but mypy wasn't able to support it." So in the future, it's more obvious about the decision?

The implementation also supports passing None. In that case, post_process is called with no arguments. Typing that accurately would probably require making a callback protocol for post_process.

Good catch, I glossed over that part in my review. I am however a little concerned how well a callback protocol will behave, considering mypy would match Csv(post_process=set) to Csv[Any, builtins.list[Any]] when I tried to replace it with a callback protocol. Changing the order of overloads, so the first one is fully generic, would probably solve that, but you would still end up with unsolved type vars. Although I am not sure I have tried a callback protocol with Any for the first argument, instead of _T, so maybe that will behave a bit better.

Shouldn't this be Sequence[tuple[_T, object]]?

Also in the overloads below I think the second tuple member in choices should be object, as the implementation ignores it.

Indeed, I had already suggested the same change for choices but I guess it fell victim to manual copy pasta.

While I do agree that the second member in the tuple is ignored and could be anything, I don't think it's a good change.
It expects that when cast is used on the second element in the tuple, for it to return the first element, so while the implementation is bogus and backwards and doesn't verify that, the stubs should probably reflect the intent and the intent was for the second element to be the string representation of the choice.