Running Workflows from Python thread (RuntimeError: There is no current event loop in thread)

Ray Core Ray Workflows
1

I’m trying to start a Workflow from a Python thread (using import threading), but it appears Ray is not set up for this. Ray calls asyncio which is unable to find the thread it created.

Ray is trying to create new threads from a thread made by PyClowder. Since it’s creating them from a ‘non-main thread,’ “automatic assignment of a new event loop does not occur. This explains my problem exactly: python 3.x - Python3 asyncio There is no current event loop in thread when spawn a new thread - Stack Overflow

My question: is it possible to edit how Ray creates async threads such that it manually sets the event loop with asyncio.set_event_loop(asyncio.new_event_loop()) ?

Thanks for any info -Kastan

Stack trace:

2022-01-31 10:42:23,320 [Thread-5       ] DEBUG   : pyclowder.connectors - ['kastanvday@gmail.com']
2022-01-31 10:42:23,321 [Thread-5       ] INFO    : pyclowder.connectors - [61a5853c87b65e03e7925706] : StatusMessage.start: Started processing.
2022-01-31 10:42:23,323 [Thread-5       ] DEBUG   : pyclowder.extractors - default check message : {'notifies': ['kastanvday@gmail.com'], 'source': {'id': {'resourceType': "'file", 'id': '61a5853c87b65e03e7925706'}, 'mimeType': 'text/plain', 'extra': {'filename': '10chars.txt'}}, 'jobid': '61f82d8e193c9580af4aa3bf', 'msgid': '61f82d8e193c9580af4aa3c0', 'flags': '', 'intermediateId': '61a5853c87b65e03e7925706', 'host': 'http://host.docker.internal:9000', 'datasetId': '61a41c945e5855829e5923d8', 'id': '61a5853c87b65e03e7925706', 'fileSize': '11', 'filename': '10chars.txt', 'target': '{}', 'secretKey': '70c420d3-918a-45d9-a4e3-987f726c6694', 'activity': 'submitted', 'routing_key': 'extractors.ncsa.rayWorkflow', 'parameters': {}, 'action': 'manual-submission', 'retry_count': 1}
2022-01-31 10:42:23,370 [Thread-5       ] INFO    : pyclowder.connectors - [61a5853c87b65e03e7925706] : StatusMessage.processing: Downloading file.
2022-01-31 10:42:23,461 [Thread-5       ] INFO    : pyclowder.connectors - [61a5853c87b65e03e7925706] : StatusMessage.processing: Loading contents of file...
10:42:23.462680
2022-01-31 10:42:23,479	INFO execution.py:49 -- Workflow job created. [id="rayWorkflow-10:42:23.462680", storage_url="file:///tmp/data"]. Type: FUNCTION.
2022-01-31 10:42:23,481 [Thread-5       ] ERROR   : pyclowder.connectors - [61a5853c87b65e03e7925706] There is no current event loop in thread 'Thread-5'.
Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.7/site-packages/pyclowder/connectors.py", line 426, in _process_message
    self.process_message(self, source_host, secret_key, resource, body)
  File "rayWorkflow.py", line 68, in process_message
    output = self.workflow.run(f"rayWorkflow-{current_time}")
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/workflow/common.py", line 363, in run
    return ray.get(self.run_async(workflow_id, metadata))
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/workflow/common.py", line 405, in run_async
    return run(self, workflow_id, metadata)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/workflow/execution.py", line 56, in run
    ws.save_workflow_user_metadata(metadata)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/workflow/workflow_storage.py", line 416, in save_workflow_user_metadata
    self._put(self._key_workflow_user_metadata(), metadata, True))
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/workflow/workflow_storage.py", line 55, in asyncio_run
    loop = asyncio.get_event_loop()
  File "/home/ray/anaconda3/lib/python3.7/asyncio/events.py", line 644, in get_event_loop
    % threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'Thread-5'.
2022-01-31 10:42:23,483 [Thread-5       ] INFO    : pyclowder.connectors - [61a5853c87b65e03e7925706] : StatusMessage.retry: (#2) There is no current event loop in thread 'Thread-5'.
1 Like
2

@kastan have you managed to solve this? I’m getting similar error, not sure what to do about it

2 Likes
3

I am also having the same issue, This issue seems to be a new one

4

This official example throws RuntimeError

5

The official example for ray.data.llm.vLLMEngineProcessorConfig may throw a RuntimeError if there are version mismatches between Ray and vLLM, missing dependencies, or misconfigured environment variables. Common issues include using an unsupported vLLM version, missing CUDA drivers, or not setting required environment variables like VLLM_USE_V1. Upgrading to the latest Ray and vLLM versions and ensuring all dependencies are installed usually resolves these errors. For example, see the official docs and related issues.

Would you like a step-by-step troubleshooting guide or help with a specific error message?

Sources:

Hint: Mention @RunLLM in the post for followups.

6

Is there any way to reproduce this, its a nightmare to debug this:

RuntimeError: There is no current event loop in thread 'MainThread'.

There is no other log or anything, I’ve tried to use vLLM alone and it deploy correctly with the model sentence-transformers/all-MiniLM-L6-v2 like the example for the embeddings but it doesn’t work either when running the example :frowning:

7

Found it, it is due to this RuntimeError when using the latest uvloop · Issue #57992 · ray-project/ray · GitHub