Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolve timeout() failures on Windows/py3.13 #3415

Merged
merged 5 commits into from
Nov 14, 2024

Conversation

jsiirola
Copy link
Member

Fixes # .

Summary/Motivation:

The timeout() decorator was failing intermittently on Windows under Python 3.13. We would either see errors like:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
    from multiprocessing.spawn import spawn_main; spawn_main(parent_pid=3444, pipe_handle=1048)
                                                  ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Miniconda\envs\test\Lib\multiprocessing\spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Miniconda\envs\test\Lib\multiprocessing\spawn.py", line 130, in _main
    preparation_data = reduction.pickle.load(from_parent)
EOFError: Ran out of input

or

self = <multiprocessing.queues.Queue object at 0x0000021A13039590>, block = True, timeout = 9.999996200000169

    def get(self, block=True, timeout=None):
        if self._closed:
            raise ValueError(f"Queue {self!r} is closed")
        if block and timeout is None:
            with self._rlock:
                res = self._recv_bytes()
            self._sem.release()
        else:
            if block:
                deadline = time.monotonic() + timeout
            if not self._rlock.acquire(block, timeout):
                raise Empty
            try:
                if block:
                    timeout = deadline - time.monotonic()
                    if not self._poll(timeout):
                        raise Empty
                elif not self._poll():
                    raise Empty
                res = self._recv_bytes()
>               self._sem.release()
E               ValueError: semaphore or lock released too many times

I was never quite able to determine why the test was suddenly failing (I suspect that there may actually be a bug in Python 3.13's implementation of Queue). This PR "resolves" the problem by switching from using a multiprocessing.Queue to communicate results back from the child process to the simpler multiprocessing.Pipe. Whereas previously we were seeing the error about 1/3 of the time (out of 25 runs, 7 failed, and 1 hung indefinitely), with this new implementation 25 attempts all passed.

Changes proposed in this PR:

  • switch from multiprocessing.Queue to multiprocessing.Pipe for communicating results back from the subprocess
  • simplify some I/O management
  • update some documentation

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

  1. I agree my contributions are submitted under the BSD license.
  2. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

@blnicho blnicho merged commit fe3f83f into Pyomo:main Nov 14, 2024
28 of 32 checks passed
@jsiirola jsiirola deleted the timeout-error-py3.13win branch November 15, 2024 03:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants