upstream request timeout

[jayvdb]

We run osv-scanner as part of our CI, and have been seeing lots of the following error for the last few days

scan failed server response error: {"code":504,"message":"upstream request timeout"}

Probably this is related to google/osv.dev#2039

We are scanning a Cargo.lock which is size 202485 and 758 entries in it, so it is a bit large.

AFAICS there is no way to configure the timeout. I am wondering whether that might be a useful addition, as we wou'd prefer it take longer rather than cause a failure in a large complicated CI workflow.

[another-rex]

We're currently looking into the timeout situation, there seems to be a recent uptick in the number of timeouts we are getting.

Can you share the time it took to timeout for your CI?

[jayvdb]

This command is part of a much larger step in our CI (standard GitHub Actions), so we cant give a duration for the timeout.
I'll run it locally a few times to see if I can reproduce locally.

[jayvdb]

Scanned /home/jayvdb/.../Cargo.lock file and found 758 packages
scan failed server response error: {"message":"upstream request timeout","code":504}


real	1m0.958s
user	0m0.055s
sys	0m0.018s

[jayvdb]

out of 100 attempts, 12 failed with that error.

[oliverchang]

Thansk @jayvdb ! Is there any way you'd be able to share the Cargo.lock with us? If it's something that's OK to share with us privately, you can email ochang AT google.com.

[jayvdb]

Sent via email

[another-rex]

The timeout issue should be mitigated now! Please test it out!

AFAICS there is no way to configure the timeout. I am wondering whether that might be a useful addition, as we wou'd prefer it take longer rather than cause a failure in a large complicated CI workflow.

Timeout is on the server side, and ideally no queries should come close to the 60 seconds limit, no matter how many packages are being queried (it's split into chunks of 1000). We're looking at more options to prevent this from happening in the future.

[jayvdb]

We had a enough CI jobs run without this occurring for me to be happy that it is resolved.

As this is essentially a repeat of google/osv.dev#1363 , I do believe something additional is needed.

A few ideas come to mind:

• have a --verbose mode which emits a note about each chunk of 1000 being processed, and how long the chunk took to complete.
• when that timeout error occurs in osv-scanner, report how long the application was running
• add a timeout duration arg, so the end user can figure out how long is needed to get a success.
• print out a warning any time a chunk of 1000 takes longer than a hard-coded "reasonable" time period to complete (i.e. for 88 of my 100 uses, I would see warnings)
• configurable retries

[another-rex]

I opened #860 to update the retry logic, and we are also doing a few things in the backend to reduce the number of timeouts that will happen as well.

There are some plans for adding verbosity levels to tell the user more information about what is happening (first part is completed in #727 ), though nothing concrete there yet.