Adjust crawler pvc on exit code 3 (out of storage)

[ikreymer]

crawler 1.5.0 now has an exit code 3 for when crawler is actually out of disk space. The operator should handle this by immediately adjusting the PVC size.

Ideally, crawler will be improved to avoid this, but since this can still happen, operator should be able to respond and fix the issue.

Testing:

• A bit tricky to test, but can be tested by setting small crawler_storage value and crawling a site with lots of large static files.

[ikreymer]

@tw4l updated testing instructions. Easiest to test on dev, lets say if storage is initially 11GB, can kubectl exec into a crawler, and then do fallocate -l 11GB ./blank or however space is left to use up all the space. The crawler should then exit with a 3, or if it exits with another code, will restart and then exit with 3.

kubectl get cjs -n crawlers -o yaml for the job, should include newStorage and kubectl get pvc -n crawlers should show the pvc with more storage, eg. 13GB in this case. Operator should have a log of something like: Attempting to adjust storage to 13Gi ...

[tw4l]

Looks good, and operator and cjs logs in local dev environment looked right even if it wasn't actually resized (because Docker Desktop doesn't support that). To test on dev before merging.