Skip to content

Inherited multiprocessing.Process cannot be utilized with multiprocessing.Pool  #103645

@vahvero

Description

@vahvero

Bug report

I am attempting to add an object variable by inheriting multiprocessing.Process. However, attempting to actually use this Process with Pool causes RecursionError.

import multiprocessing as mp
from typing import Type


class ResourceProcess(mp.Process):
    def __init__(
        self,
        *args,
        **kwargs,
    ):
        super().__init__(
            *args,
            **kwargs,
        )
        resource_id = 0
        self._resource_id = resource_id

    @property
    def resource_id(self):
        return self._resource_id


def create_resource_context() -> Type[mp.context.DefaultContext]:
    context = mp.get_context()
    context.Process = ResourceProcess
    return context


def test_resource_pool(ctx):
    with ctx.Pool(2) as pool:
        payload = (tuple(range(3)) for _ in range(5))
        response = pool.starmap(
            _dummy_func,
            payload,
        )

    assert response == (
        expected := list(
            (
                (
                    (0, 1, 2),
                    {},
                ),
            )
            * 5
        )
    ), f"{response=}!={expected}"


def _dummy_func(*args, **kwargs):
    return args, kwargs


if __name__ == "__main__":

    test_resource_pool(mp.get_context())
    print("Standard pool success")
    ctx = create_resource_context()
    test_resource_pool(ctx)
    print("Inherited pool success")

The source of this error is context.py:224 which starts to call itself. If no inheritance is done, pool works as expected.

Your environment

  • CPython versions tested on: 3.11.0
  • Operating system and architecture: Ubuntu 22.04.6, x86_64

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytopic-multiprocessingtype-bugAn unexpected behavior, bug, or error

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions