Integrating OpenCV with FastAPI and Redis RQ Causes Task/Job Failures

[HuseynG]

System Information

// example for python user
OpenCV python version: 4.9.0.80
OpenCV python version headless: 4.9.0.80
Operating System / Platform: MacOS 14.4.1
Python version: 3.9.18

Detailed description

No error is thrown.

Steps to reproduce

I'm working on integrating a set of microservices into our application, focusing on a FastAPI server that interacts with Redis through RQ for task queueing. The setup involves three main files:

1. server.py: Defines FastAPI endpoints.
2. redis_processes.py: Manages Redis connections and queues tasks.
3. worker_tasks.py: Contains the tasks to be enqueued and executed by workers.

The flow is as follows: The FastAPI endpoint receives some inputs and queues jobs in Redis. These jobs are defined in 'worker_tasks.py' and are supposed to be processed by background workers.

However, I'm encountering a peculiar issue: whenever I import 'cv2' (OpenCV) into 'worker_tasks.py', and I receive the following error message: "Work-horse terminated unexpectedly; waitpid returned 6 (signal 6).", but not on terminal though. This problem seems to be specifically related to the inclusion of the OpenCV library in 'worker_tasks.py'.

Here's a simplified version of the code for each component:

'server.py' snippet:

from fastapi import FastAPI, UploadFile, File, Query, HTTPException, Security, Depends
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from typing import List, Optional
from models import FeedbackTask
import redis_processes

app = FastAPI()
security = HTTPBearer()

# Endpoint and authentication details omitted for brevity
@app.post("/processfeedback/")
async def process_feedback(...): # parameters and processing
    return redis_processes.process_feedback(feedback_task)

'redis_processes.py' snippet:

import redis
from rq import Queue
from redis_settings import workers_settings

import threading

import time
from datetime import datetime

from models import FeedbackTask
import worker_tasks as worker_tasks



# Set up Redis connection
redis_conn = redis.Redis(host='localhost', port=6379, db=0)

ai_task_processor_q         = Queue(name= workers_settings[0]['q'],connection=redis_conn)
initial_db_writer_q         = Queue(name= workers_settings[1]['q'],connection=redis_conn)
ai_task_result_writer_q     = Queue(name= workers_settings[2]['q'],connection=redis_conn)


def process_feedback(feedback_process_task: FeedbackTask):
    # tasks.process_feedback(feedback_task)
    now = datetime.now()
    timestamp = int(time.mktime(now.timetuple()))
    feedback_process_task.collection_id = f"{feedback_process_task.user_id}_{timestamp}"

    job_db_w = initial_db_writer_q.enqueue(worker_tasks.add_raw_collection_to_db, feedback_process_task, result_ttl=-1)
    job_ai   = ai_task_processor_q.enqueue(worker_tasks.process_feedback, feedback_process_task, result_ttl=-1)

    # return {"job_id": job_ai.get_id()}
    return {"AI Feedback Process Job Status": job_ai.get_status(),
            "Raw Document Collection Write Status": job_db_w.get_status()}

'worker_tasks.py' snippet:

import cv2  # This import causes the issue
from models import FeedbackTask

def process_feedback(task: FeedbackTask) -> FeedbackTask:
    # Task processing logic here

The tasks fail only after adding 'import cv2' in 'worker_tasks.py'. I've verified that the tasks work as expected when this import is commented out. My initial guess was it might be a problem with worker environments or a conflict between libraries, but I haven't found conclusive evidence to pinpoint the issue.

Has anyone faced a similar issue or does anyone have insights on why importing OpenCV in a worker task could lead to job failures in this setup? Any suggestions on how to debug or resolve this issue would be greatly appreciated.

I am using python 3.9.18, and here is the link for the requirement.txt link

If you would like to replicate the issue, please following the following link.

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
• I updated to the latest OpenCV version and the issue is still there
• There is reproducer code and related data files (videos, images, onnx, etc)

[LaurentBerger]

'The tasks fail only after adding 'import cv2' in 'worker_tasks.py''
Have you got an error message?

[HuseynG]

No, I've encountered a situation where my job fails when executed via RQ, but it doesn't throw any explicit error. This issue seems to be isolated to the cv2 library, as none of the other libraries in my project lead to this behaviour. The error logged by RQ indicates: "Work-horse terminated unexpectedly; waitpid returned 6 (signal 6)." I guess there might be a specific aspect of OpenCV that triggers an abrupt termination of the execution.

[crackwitz]

Context from Stack Overflow

[HuseynG]

Context from Stack Overflow

This is actually my post. And I was advised to raise an issue.

[crackwitz]

I linked this context into this issue because it wasn't linked yet. I was the one who recommended that you file an issue with the other projects because they probably do something strange with their tasking, like copying processes, or otherwise spawning them in unexpected ways that cause OpenCV initialization to fail (OpenCL and IDK what else).

More context: issue on rq/rq and issue on redis/redis-py

[LaurentBerger]

I know nothing about fastapi
in python can you try
import cv2

second in your example I have got an error:
from models import FeedbackTask
ModuleNotFoundError: No module named 'models'

what is models?

[HuseynG]

Import cv2 is not throwing an error, but rather causes issue with RQ/Redis. No other library has this issue i my project except for CV2.

In regards to the models, it is just a 'models.py' file containing pydantic models here is the content:

from typing import List, Optional
from pydantic import BaseModel
from fastapi import UploadFile

class FeedbackTask(BaseModel):
    user_id: str
    #...

[LaurentBerger]

New error
File "C:\Users\laurent\Desktop\fastapi\server.py", line 5, in
import redis_processes
from redis_settings import workers_settings
ModuleNotFoundError: No module named 'redis_settings'

[HuseynG]

New error File "C:\Users\laurent\Desktop\fastapi\server.py", line 5, in import redis_processes from redis_settings import workers_settings ModuleNotFoundError: No module named 'redis_settings'

I created a simpler version of the project, please follow the instructions, link.

[LaurentBerger]

Ok I have got something :
uvicorn server:app --reload --port 8000
INFO: Will watch for changes in these directories: ['C:\Users\laurent\Desktop\fastapi']
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: Started reloader process [22284] using StatReload
INFO: Started server process [24560]
INFO: Waiting for application startup.
INFO: Application startup complete.

and in mozilla

{"detail":"Not Found"}

[HuseynG]

Ok I have got something : uvicorn server:app --reload --port 8000 INFO: Will watch for changes in these directories: ['C:\Users\laurent\Desktop\fastapi'] INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit) INFO: Started reloader process [22284] using StatReload INFO: Started server process [24560] INFO: Waiting for application startup. INFO: Application startup complete.

and in mozilla

{"detail":"Not Found"}

Can you please elaborate? There is not frontend for the project. I recommend using postman, like this way:

[LaurentBerger]

I have got an error message 127.0.0.1:52852 - "GET /processfeedback/?user_id=1234 HTTP/1.1" 405 Method Not Allowed

[HuseynG]

I have got an error message 127.0.0.1:52852 - "GET /processfeedback/?user_id=1234 HTTP/1.1" 405 Method Not Allowed

because it is post method, not get method.

[LaurentBerger]

In console I have got
INFO: 127.0.0.1:53262 - "POST /processfeedback/?user_id=1235 HTTP/1.1" 200 OK

and in mozilla debugger

{"AI Feedback Process Job Status":"queued","Raw Document Collection Write Status":"queued"}

and in rq info
Task_A_Processor_Q |████ 4, 0 executing, 0 finished, 0 failed
1 queues, 4 jobs total

0 workers, 1 queues

Updated: 2024-04-10 16:21:52.397474

[asmorkalov]

@HuseynG OpenCV is native library with significant set of native dependencies. I suspect, that OpenCV and your software stack have some intersaction in list of dynamic dependencies, but their versions are different. The same thing happens time to time in pair opencv-python - pyQt. You can run lsof or htop -l or use gdb to list of libraries loaded by python interpreter. Just compare result with and without OpenCV.