Optimize plugin fails with "No such file or directory" in .cache directory

[chipsenkbeil]

Context

I recently tried adding the optimize plugin to work with a variety of PNGs I use on https://distant.dev/. It appears that I am hitting the issue described in #5098, so taking a stab at reporting it.

I'm using commit fa5bd3f45024328d973977db80cccd5b1df89228 of Insiders.

Bug description

Running mkdocs build or mkdocs serve with the optimize plugin configured within mkdocs.yml results in a FileNotFoundError.

❯ mkdocs build
INFO     -  [macros] - Macros arguments: {'module_name': 'macros', 'modules': [], 'render_by_default': True, 'include_dir': '',
            'include_yaml': [], 'j2_block_start_string': '', 'j2_block_end_string': '', 'j2_variable_start_string': '', 'j2_variable_end_string': '',
            'on_undefined': 'strict', 'on_error_fail': True, 'verbose': False}
INFO     -  [macros] - Found local Python module 'macros' in: /Users/senkwich/projects/distant.dev
INFO     -  [macros] - Found external Python module 'macros' in: /Users/senkwich/projects/distant.dev
INFO     -  [macros] - Extra filters (module): ['pretty']
INFO     -  Cleaning site directory
INFO     -  Building documentation to directory: /Users/senkwich/projects/distant.dev/site

  Optimizations:
Traceback (most recent call last):
  File "/Users/senkwich/Library/Python/3.9/bin/mkdocs", line 8, in <module>
    sys.exit(cli())
  File "/Users/senkwich/Library/Python/3.9/lib/python/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/senkwich/Library/Python/3.9/lib/python/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/senkwich/Library/Python/3.9/lib/python/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/senkwich/Library/Python/3.9/lib/python/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/senkwich/Library/Python/3.9/lib/python/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/senkwich/Library/Python/3.9/lib/python/site-packages/mkdocs/__main__.py", line 250, in build_command
    build.build(cfg, dirty=not clean)
  File "/Users/senkwich/Library/Python/3.9/lib/python/site-packages/mkdocs/commands/build.py", line 332, in build
    config.plugins.run_event('post_build', config=config)
  File "/Users/senkwich/Library/Python/3.9/lib/python/site-packages/mkdocs/plugins.py", line 522, in run_event
    result = method(**kwargs)
  File "/Users/senkwich/Library/Python/3.9/lib/python/site-packages/material/plugins/optimize/plugin.py", line 113, in on_post_build
    size_opt += os.path.getsize(target)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/genericpath.py", line 50, in getsize
    return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '.cache/plugin/optimize/assets/favicon.png'

Related links

• Optimize Plugin

Reproduction

example.zip

I've removed everything but a blank index.md page as it appears unrelated to which images are actual referenced. Additionally, I cleared out the majority of images and other assets alongside all css and javascript.

mkdocs.yml

nav:
  - 'index.md'

theme:
  name: material
  favicon: assets/favicon.png
  logo: assets/logo.svg

plugins:
  - optimize

Steps to reproduce

1. Create a couple of PNGs in the assets directory
   • For me, a minimal layout is

     assets/
         favicon.png
         logo.svg
         images/
             architecture/
                 picture.png
     

2. Attempt to run mkdocs build or mkdocs serve
3. Observe error about a file not existing in the .cache directory
   • In my case, there is a path to .cache/plugin/optimize/assets/images/architecture without anything inside, nor anything else in the parent directories.

Browser

No response

Before submitting

• I have read and followed the bug reporting guidelines.
• I have attached links to the documentation, and possibly related issues and discussions.
• I assure that I have removed all customizations before submitting this bug report.
• I have attached a .zip file with a minimal reproduction.

[squidfunk]

Thanks for reporting! Unfortunately, I'm not able to see the error you reported (multiple runs, same result):

INFO     -  Cleaning site directory
INFO     -  Building documentation to directory: /Users/squidfunk/Downloads/example 6/site
INFO     -  Optimized media file: assets/images/favicon.png 1.9 kB
INFO     -  Optimized media file: assets/favicon.png 1.7 kB
INFO     -  Optimized media file: assets/images/architecture/authenticate-with-server.png 93.5 kB ↓ 242.8 kB [72.2%]

  Optimizations:
    *.png 97.0 kB ↓ 242.8 kB [71.4%]

INFO     -  Documentation built in 1.24 seconds

My gut tells me it might be a race condition. Does the error occur when you set concurrency to 1?

plugins:
  - optimize:
      concurrency: 1

[squidfunk]

Additionally, please check if #5098 (comment) fixes it.

[squidfunk]

Okay, so 647614b27 is an attempt to mitigate the issue. After diving into concurrent futures, my understanding was that ThreadPoolExecutor.shutdown would automatically wait for all futures to be resolved or cancelled. However, I'm not sure whether this is implementation dependent. Thus, I've opted for making awaiting all futures more explicit by using wait. Could you please check the tip of master and see if this mitigates the issue? Unfortunately, I'm not able to reproduce, but making things more explicit is likely a good idea.

[squidfunk]

/cc @yrashk, @uPagge ⤴️

[chipsenkbeil]

My gut tells me it might be a race condition. Does the error occur when you set concurrency to 1?

plugins:
 - optimize:
     concurrency: 1

Additionally, please check if #5098 (comment) fixes it.

@squidfunk setting concurrency: 1 for optimize did not seem to impact the issue, but print_gain_summary: false enabled it to succeed. I'm not seeing any actual images in .cache though. Is that to be expected?

If it is a race condition, my machine hits it consistently.

---

I also tried https://github.com/squidfunk/mkdocs-material-insiders/commit/647614b2710e544009555fa3a7c358aa712bf5c4 and using concurrency: 1 yielded same error and print_gain_summary: false continued to succeed with nothing in the cache.

[JuLeeAtPlanet]

Just to share: this allowed the pipeline to succeed, but many images missing.

[squidfunk]

Hmm, I'm a little bit lost here, as I'm not able to observe the issue. I also tried to run it under the exact same Python version as reported in the OP + I'm also on macOS, but no success – everything works as expected 😐 If somebody that can observe the issue can assist, I'd be more than happy to collaborate in a PR on the Insiders repository.

[squidfunk]

Wait, I think I know what might be happening: do you have pngquant installed? The plugin needs pngquant to optimize PNG images. The problem is that it will currently silently fail when pngquant is not available. I'll fix that and improve error reporting, so it's apparent what's happening.

Just to share: this allowed the pipeline to succeed, but many images missing.

Are all of those images PNGs?

[squidfunk]

I've improved the resilience of the optimize plugin in 523653774:

• Upon start, the plugin checks for pngquant and notifies the user if it is not installed
• The plugin should keep running in serve mode when it encounters errors, similar to the social plugin

I'm pretty convinced this should fix the reported errors, and/or point the author into the right direction, i.e., if pngquant is not installed. Our documentation explains how to install pngquant.

[JuLeeAtPlanet]

I didn't install your latest, but I did add pngquant via brew to the pipeline and successfully built. The images I dynamically populate with a main.py macro on landing pages aren't showing up, but that could be another issue? At any rate, this ticket is about the gitlab pipeline failing and it no longer fails once I added brew and installed pngquant that way.

[squidfunk]

I didn't install your latest, but I did add pngquant via brew to the pipeline and successfully built.

Thanks for testing! Perfect, sounds like we found the issue! The latest commit only improves error reporting and checks if pngquant is installed, the logic itself is not changed (only refactored). Thus, I think, we can consider this resolved.

[chipsenkbeil]

@squidfunk I don't believe I had pngquant installed, so that's definitely a problem! I pulled the latest commit and tried running this on another, fresh computer, and it seems like installing pngquant didn't get detected. I know that my pip installation is working because I also had to install pillow, which did work fine.

I can also verify that running python3, I am able to import pngquant:

[chipsenkbeil]

Ah, strike that! I didn't realize your instructions were to run brew install pngquant instead of using pip install pngquant. Using the homebrew method, it appears to work!

Closing this out as resolved.

[squidfunk]

Perfect, thanks for testing! We use pngquant directly, because pip install pngquant will basically only install a thin Python wrapper that doesn't add much value, and you would need to install pngquant on your system anyway. Thus, we decided to use subprocess to run it directly. Let's leave this open until it is released.

[JuLeeAtPlanet]

Oops. Forgot to remove "print_gain_summary: false" when I did, it failed on a later image:

.
.
.
INFO     -  Optimized media file: assets/images/layers/1-landscape.png 1.4 MB ↓ 1.9 MB [56.4%]
  Optimizations:
    *.png 137.7 MB ↓ 211.1 MB [60.5%]
Traceback (most recent call last):
  File "/builds/code/devrel/devcenter_v2_i/venv/bin/mkdocs", line 8, in <module>
    sys.exit(cli())
  File "/builds/code/devrel/devcenter_v2_i/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/builds/code/devrel/devcenter_v2_i/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/builds/code/devrel/devcenter_v2_i/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/builds/code/devrel/devcenter_v2_i/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/builds/code/devrel/devcenter_v2_i/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/builds/code/devrel/devcenter_v2_i/venv/lib/python3.10/site-packages/mkdocs/__main__.py", line 250, in build_command
    build.build(cfg, dirty=not clean)
  File "/builds/code/devrel/devcenter_v2_i/venv/lib/python3.10/site-packages/mkdocs/commands/build.py", line 332, in build
    config.plugins.run_event('post_build', config=config)
  File "/builds/code/devrel/devcenter_v2_i/venv/lib/python3.10/site-packages/mkdocs/plugins.py", line 522, in run_event
    result = method(**kwargs)
  File "/builds/code/devrel/devcenter_v2_i/venv/src/mkdocs-material/material/plugins/optimize/plugin.py", line 113, in on_post_build
    size_opt += os.path.getsize(target)
  File "/usr/local/lib/python3.10/genericpath.py", line 50, in getsize
    return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '.cache/plugin/optimize/platform/order-monitor/tasking/images/v2_flexible_state.jpeg'
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: command terminated with exit code 1

[chipsenkbeil]

@JuLeeAtPlanet and you installed pngquant using a package manager and pillow using pip? I just added optimize with no special configuration settings and it worked for me. I only had PNGs, so I tried adding a jpeg to test, and it still works.

[squidfunk]

@JuLeeAtPlanet could you install the latest version of Insiders and try again? If it still happens, I'll look into it asap.

[JuLeeAtPlanet]

Can do this in 2 hours. Can I upgrade?

[squidfunk]

@JuLeeAtPlanet just go through the installation steps, pip should pull the latest changes then:

pip install git+https://${GH_TOKEN}@github.com/squidfunk/mkdocs-material-insiders.git

If that doesn't work, try:

pip install --upgrade --force-reinstall git+https://${GH_TOKEN}@github.com/squidfunk/mkdocs-material-insiders.git

[squidfunk]

Released as part of 9.1.18+insiders-4.37.0 – I'm sufficiently confident that this should fix all reported problems, i.e., instruct the user to install pngquant and pillow, if they're not available.

[JuLeeAtPlanet]

Still not getting to a successful job. One of the reasons: when I pip freeze > requirements.txt, the ${GH_TOKEN} wasn't automatically inserted:

mkdocs-material @ git+https://github.com/squidfunk/mkdocs-material-insiders.git@fe2d494410df15d6b9edbd58bac068de1529654b

Will continue to work on this over the weekend...

[squidfunk]

Please see #4074.

[mjtobias]

Apologies for tossing this on an existing closed issue, I'm happy to open another issue if it would be more appropriate.

I was also experiencing this issue on site build with insiders mkdocs_material-9.3.1+insiders.4.41.0. I had all of the correct packages installed but it kept choking on a specific image. I realized that I wasn't actually using that image on my site so figured that might be an issue, when I removed that image it choked on another image. I realized that what it was choking on was jpeg images, jpeg specifically. I had jpg images that worked fine, png worked fine. Once I converted my jpeg images to png the site built correctly. Not sure if this is a known issue but if you are running into the issue of the Optimize plugin failing with this error, consider if you have jpeg images and try converting them to jpg or png.

[squidfunk]

@mjtobias thanks for the detailed description. I was able to reproduce the problem by renaming the extension of a file from .jpg to .jpeg. This should be fixed in 1b61060. If not, please create a new issue with a minimal reproduction and we're happy to look into it ☺️

[mjtobias]

@squidfunk Can confirm this looks fixed for me. Added a .jpeg file back to the project, pulling from latest insiders, and it is now correctly optimized and successfully builds. Thank you for the super fast response as always!