Used TypeVarTuple and ParamSpec in several places

[agronholm]

Fixes #560.

[gschaffner]

src/anyio/from_thread.py:424: error: Argument 1 to "submit" of "Executor" has incompatible type "Callable[[Callable[[VarArg(*PosArgsT)], Awaitable[T_Retval]], VarArg(*PosArgsT), DefaultNamedArg(str, 'backend'), DefaultNamedArg(dict[str, Any] | None, 'backend_options')], T_Retval]"; expected "Callable[[Callable[[], Coroutine[Any, Any, None]], str, dict[str, Any] | None], None]" [arg-type] appears to be due to python/mypy#16662.

[agronholm]

Sounds like we should just slap a type: ignore[arg-type] on it.

[gschaffner]

closes #560

[gschaffner]

great!

i compared this to my old implementation¹: the main differences seem to be be that while it looks like we both viewed this as a good opportunity to add fuller type-checking to various AnyIO internals (rather than just modifying the public API), you added type-checking in some additional places that i didn't, and i added type-checking in some additional places that you didn't.

i don't believe github's interface allows me to comment suggestions on unmodified files directly on this PR, so instead here they are in the form of commits (which i have rebased atop this branch) that you can consider cherry-picking: 2f876cb...886ebdc

Footnotes

1. https://github.com/agronholm/anyio/compare/master...54ab4c5272bfb29faa385f8d04f6e0fe88472030 & https://github.com/agronholm/anyio/compare/2f876cbb55cc8ffdf408a4b5a1f63d039db8ac0d..54ab4c5272bfb29faa385f8d04f6e0fe88472030 ↩

[agronholm]

At least the type annotations you added to start_task() are wrong: it's declaring that start_task() returns the return value of the target function, when in fact it returns the value passed to task_status.started().

[agronholm]

I've cherry-picked all the changes I thought were valid.

[gschaffner]

At least the type annotations you added to start_task() are wrong: it's declaring that start_task() returns the return value of the target function, when in fact it returns the value passed to task_status.started().

i'm confused—doesn't start_task return a tuple (target_retval_future, task_status_value)?

sample code (click to expand)

from typing import assert_type

import anyio
import anyio.from_thread
from anyio.abc import TaskStatus


async def foo(*, task_status: TaskStatus[int]) -> str:
    task_status.started(1)
    return "asdf"


with anyio.from_thread.start_blocking_portal() as portal:
    fut_retval, task_status_val = portal.start_task(foo)
    retval = fut_retval.result()
    print(f"{retval = }")

    # This is True at runtime.
    print(f"{isinstance(retval, str) = }")

    # But this assertion fails at type-checking time currently.
    assert_type(retval, str)  # error: Expression is of type "Any", not "str"  [assert-type]
    # However, the ``start_task`` annotation change in 05b70e2 makes it pass.

the changes i made to start_task were

     def start_task(
         self,
-        func: Callable[..., Awaitable[Any]],
+        func: Callable[..., Awaitable[T_Retval]],
         *args: object,
         name: object = None,
-    ) -> tuple[Future[Any], Any]:
+    ) -> tuple[Future[T_Retval], Any]:
         """
         Start a task in the portal's task group and wait until it signals for readiness.

@@ -356,7 +357,7 @@ class BlockingPortal:

         """

-        def task_done(future: Future) -> None:
+        def task_done(future: Future[T_Retval]) -> None:
             if not task_status_future.done():
                 if future.cancelled():
                     task_status_future.cancel()
@@ -371,7 +372,7 @@ class BlockingPortal:
         self._check_running()
         task_status_future: Future = Future()
         task_status = _BlockingPortalTaskStatus(task_status_future)
-        f: Future = Future()
+        f: Future[T_Retval] = Future()
         f.add_done_callback(task_done)

(05b70e2#diff-35c3d377d2935dbe1e1e44908929406cc7647d9269d69ba04ef9d353f83b1900R337-R376)

which does not declare that start_task returns the return value of the target function.

[gschaffner]

I have a couple comments about the PR. I've opened them as threads so it's more organized.

Sorry for not using threads in the first place yesterday. I got excited trying to submit a review ASAP since you were waiting for me and mentioned wanting to move fast, but (in retrospect) I think I should have slowed down and organized my communication better.

[gschaffner]

re-submitting a couple of the other earlier suggested changes, as you may also missed them earlier.

[gschaffner]

I would also suggest picking f7d3f5f and 43e3e26. Like T_co, T_Retval should only be used in covariant positions and marking it as such will cause Mypy to error if it's ever accidentally used in a position with the wrong variance.

[agronholm]

I've picked f7d3f5f, but unsure about 43e3e26.

[gschaffner]

(in case anyone is looking at this in the future, there's some more discussion in a Gitter thread: https://matrix.to/#/!JfFIjeKHlqEVmAsxYP:gitter.im/$jPGgnv0ghzCsa33gJ2SFE1YRmFqVFoqwQIeBMMFThqw?via=gitter.im)

[gschaffner]

hooray!

note, #652 (comment) is still needed