⚠️ Social cards broken – update to 9.5.17

[aronmolnar]

Note

Edit by @squidfunk: Please update to the latest version before raising another issue or opening a discussion. The problem has been fixed and released as part of 9.5.17, and is reported to fix the BadZipFile error.

---

Context

The social plugin tries to fetch fonts from Google and to unzip them:
https://github.com/squidfunk/mkdocs-material-insiders/blob/4ff6a572151f684fe2e21527582e1096a88805d0/src/plugins/social/plugin.py#L857

Bug description

Google no longer provides direct access to the zip file, so building sites fails.

  File "/usr/local/lib/python3.12/site-packages/material/plugins/social/plugin.py", line 522, in _render_typography
    path = self._resolve_font(family, style, variant)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/material/plugins/social/plugin.py", line 809, in _resolve_font
    self._fetch_font_from_google_fonts(family)
  File "/usr/local/lib/python3.12/site-packages/material/plugins/social/plugin.py", line 876, in _fetch_font_from_google_fonts
    archive = ZipFile(f)
              ^^^^^^^^^^
  File "/usr/local/lib/python3.12/zipfile/__init__.py", line 1341, in __init__
    self._RealGetContents()
  File "/usr/local/lib/python3.12/zipfile/__init__.py", line 1408, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file

Workaround was disabling the social plugin.

Related links

• Reporting a bug

Reproduction

Using the code referenced above shows that the link provides HTML data.

Steps to reproduce

import requests
family= "Noto Sans"
url = f"https://fonts.google.com/download?family={family}"
res = requests.get(url, stream = True)
for chunk in res.iter_content(chunk_size = 8192):
    print(chunk)

<script data-id="_gd" nonce="lai0FBQz_50-Cv_k8L_woA">window.WIZ_global_data
...snip...

Browser

No response

Before submitting

• I have read and followed the bug reporting guidelines.
• I have attached links to the documentation, and possibly related issues and discussions.
• I assure that I have removed all customizations before submitting this bug report.
• I have attached a .zip file with a minimal reproduction using the built-in info plugin.

[alexvoss]

This has also been observed by others: #6982 (comment)

[kcgthb]

Confirming the issue. It's also being reported upstream: google/fonts#7481

[alexvoss]

Assuming that Google don't reverse their change, what would be a sensible solution? I could imagine downloading the needed fonts manually and pointing the social plugin at them. That way I could also use fonts that are not on GF. If they did reverse it we'd still win that.

[rathboma]

I'm also having this issue building my docs - https://github.com/beekeeper-studio/beekeeper-studio/actions/runs/8469768189

This only started when I upgraded to 9.5.15, but seems like that is unrelated?

[alexvoss]

Yes, this is a change on Google's side that broke the automated font downloading.

[alexvoss]

@kamilkrzyskow should one of us do a PR that introduces a "I have the fonts locally" option? Should not be too difficult to bypass the downloading and unzipping from Google but it does not make sense for us to do this in parallel. Squidfunk can then do what he likes with it - incorporate it, provide an alternative, fix it, whatever. Seems important to fix this asap as it will affect a lot of people?

[peterschmidt85]

Any workaround folks? I can't deploy a new release.

[alexvoss]

For right now the only thing I can suggest is to turn off the social plugin. There should be a fix tomorrow, I think. IMHO downloading the fonts from Google Fonts was a convenience function that now no longer works. I will start looking at a PR to fix this issue with a local font option. That leaves open the possibility of automating it should that become feasible again but gives us a plan B.

[discdiver]

Affecting us also.

[alexvoss]

Ok, here is a workaround of sorts in the hope that this helps some people in the short term. Edit: this is for the Insider Edition, instructions for the public version see post #6983 (comment) by @jbott below.

Download the font(s) you need from Google Fonts manually and install them in the .cache directory following this recipe, adjusting it as necessary:

export FONT_FAMILY=Inter
mkdir -p .cache/plugin/social/fonts/$FONT_FAMILY
cd .cache/plugin/social/fonts/$FONT_FAMILY
unzip ~/Downloads/$FONT_FAMILY.zip .
mv static/*.ttf . 
mv static/*.otf .

Now, rename the fonts to have only the variant in the name, i.e., get rid of the family name in it. That should leave them in the cache in the way the plugin expects them and it will skip the trip to Google Fonts.

Would be good if someone could try this and give feedback in case I made a mistake. I constructed the cache layout from what I got after running the code in my PR, which I am about to send off to @squidfunk. Might take a few hours before he sees it, please bear with us.

Edit: simplified by unpacking from Downloads directory

Also, this is what the result looks like for the Inter font family:

$ ls -l .cache/plugin/social/fonts/Inter          
total 5544
-rw-rw-r--@ 1 avoss  staff  316848 Sep 15  2023 Black.ttf
-rw-rw-r--@ 1 avoss  staff  316584 Sep 15  2023 Bold.ttf
-rw-rw-r--@ 1 avoss  staff  317184 Sep 15  2023 ExtraBold.ttf
-rw-rw-r--@ 1 avoss  staff  311232 Sep 15  2023 ExtraLight.ttf
-rw-rw-r--@ 1 avoss  staff  310832 Sep 15  2023 Light.ttf
-rw-rw-r--@ 1 avoss  staff  315132 Sep 15  2023 Medium.ttf
-rw-rw-r--@ 1 avoss  staff  310252 Sep 15  2023 Regular.ttf
-rw-rw-r--@ 1 avoss  staff  316220 Sep 15  2023 SemiBold.ttf
-rw-rw-r--@ 1 avoss  staff  310984 Sep 15  2023 Thin.ttf

[alexvoss]

Just a note that the PR I am working on will not require manual manipulations of the fonts beyond the downloading and putting the .zip in a directory somewhere that can be configured with a fonts_dir option.

Edit: PR has just gone in. Hope it is of some use and we'll see a fix released soon. Might take a little because of timezones.

[kcgthb]

The issue reported to the Google Fonts repo has been closed as "completed" with the following comment:

fonts.google.com/download (and all other endpoints under fonts.google.com/*) are internal and not meant for external use as they may change at any time without warning. If you need access to font metadata and the font binaries you can instead use the public developer api: https://developers.google.com/fonts/docs/developer_api

[alexvoss]

Does anyone know if there are any rate limits on the use of the API? That is one possible problem that I could see. It would be super important that caching is used to prevent fonts being downloaded again and again. The example Github Actions workflow provided in the documentation contains settings to prevent this.

Edit: ok, so the API key needed to access the API is a secret? That ain't gonna work. Hope I am holding the wrong end of the stick there.

[squidfunk]

Wow, what just happened while I was sleeping? I'm currently on vacation, and albeit I try to stay on top of things, answer questions, fix bugs, and be responsive nonetheless, I will likely not be able to resolve this in the next few days, as we need to think carefully here before we decide on what to do, as it is a very, very fundamental change impacting DX big time. I'm confident that we can find other ways to cope with this while I'm gone, maybe even work towards a solution.

Context

Before we started using the downloads endpoint, we fetched the font files from Google Fonts' CSS files, but that led to some problems where characters were not available, as those files are heavily optimized. I can't really find the commit now, but a user suggested that we could switch to the downloads endpoint for a better experience, and it served us well for years.

However, I wasn't aware that it was internal. Google switching this endpoint off is inconvenient for us, to put it mildly, but I think we can't really complain if we have used an unofficial API until now. Thus, we need to look for other solutions.

Directions

We cannot instruct the social plugin to use the developer API, as it requires an API key. This means that every user of Material for MkDocs that wants to use the social plugin would need to register for a Google API key prior to using it. It would also mean this key would need to be passed along as a secret everywhere the project is built, which for Open Source projects would mean that contributors would not be able to build your documentation, since the API key must be kept secret and considered sensitive, or it would defeat its purpose. Thus, this is a dead end from my perspective.

Before we decide how we proceed, we should evaluate multiple possibilites. A first PR by @alexvoss is up that might help to remedy the situation, so you can help us collect feedback on this PR. We're open to other suggestions, including entirely different font catalogs (i.e. switching away from Google Fonts), etc., but we can't just bandaid this, since everybody using the social plugin is impacted.

Mitigation

The only save mitigation that does not rely on caching is to disable the social plugin. If you're using Insiders, you could theoretically just download the font via Google Fonts web interface and unpack it in .cache/plugins/social/fonts. This does not work for the community edition, since the Insiders edition is a more elaborate rewrite. We can match folder locations if that helps, though.

I'll try to find some time here and there during my vacation and be as helpful as possible, but please understand that right now, I might be a little slow to respond.

[capcom6]

Why not use fonts directly from https://github.com/google/fonts ?

The main difference is that this repository contains multi-variant fonts in single files, while fonts.google.com provided archives of single-variant files.

[squidfunk]

@ssbarnea other users report that social cards are building again, so as always, we need a minimal reproduction to assist you. @discdiver I don't think it's a cached font, because fonts are now stored in another location in the community edition (since you're referring to 9.5.17), so the plugin will definitely download them. Same thing: we need a reproduction to fix it.

We're happy to fix things immediately, once we receive reproductions ☺️

[ssbarnea]

First, sorry for what Google bought you last week, I feel the pressure on you. mkdocs-material is so popular, so... you pay the celebrity price. For the moment the easiest thing for me to do is to disable the social plugin as it was never an essential bit.

In addition to the default Roboto fonts, I am also using the RedHat ones, but boths are "google fonts". As I do have my own theme on top of material-theme for reusability across projects, I am really looking for a solution to embed the fonts inside the theme package, so we no longer need to download fonts from google in order to build. Still, there is not documentation about how to achieve this.

Another change that could prove useful, if the time allows, would be to catch zipfile.BadZipFile exception and throw a clear error message when it happens instead of one where we would need to guess which download might have had failed. Important to display the original URL and hopefully http code received too.

[vedranmiletic]

Command was killed with SIGBUS (Bus error due to misaligned, non-existing address or paging error)

This is a low-level error and could indicate a bug in the Python interpreter that might or might not be related to the operations of Material for MkDocs.

In addition to the default Roboto fonts, I am also using the RedHat ones, but boths are "google fonts".

This is an interesting edge case that presumably not many people in the Material for MkDocs community have. Does the minimal example break as well?

[squidfunk]

First, sorry for what Google bought you last week, I feel the pressure on you. mkdocs-material is so popular, so... you pay the celebrity price. For the moment the easiest thing for me to do is to disable the social plugin as it was never an essential bit.

Thanks for acknowledging! Yeah, I tried to fix this as fast as possible during my vacation, and yes, there's a lot of pressure on me, but I can handle that. The funding we have makes that possible, as I can put icreasing weight on more shoulders. I also always try to provide mitigations as fast as possible, albeit it was "disable the plugin" this time.

In addition to the default Roboto fonts, I am also using the RedHat ones, but boths are "google fonts". As I do have my own theme on top of material-theme for reusability across projects, I am really looking for a solution to embed the fonts inside the theme package, so we no longer need to download fonts from google in order to build. Still, there is not documentation about how to achieve this.

Yes, we'll consider adding the ability to provide a custom font directory in the Insiders version of the social plugin after we sorted out the best way how to deal with Google Fonts. What we released with 9.5.17 is our first take on it, and we need to see if this fixes all problems or if we need to change things. After the social plugin can be considered stable again, we'll look into providing more options. Note that the fonts will likely need to have specific names, so we keep code duplication at a minimum, but I guess this is a reasonable requirement for custom fonts that you control anyway.

The simplest way to achieve what you want is to whitelist the .cache/plugins/social/fonts directory in your .gitignore, and build the project locally which will download the fonts, and check the fonts into version control.

Another change that could prove useful, if the time allows, would be to catch zipfile.BadZipFile exception and throw a clear error message when it happens instead of one where we would need to guess which download might have had failed. Important to display the original URL and hopefully http code received too.

Yes, we're going to improve error reporting when we settle on the final font loading logic. We'll first collect some feedback on the current draft, so we can iterate more quickly. Then we're going to improve resilience.

[janaka]

I'm facing this also with 9.5.17 in @docqai/docq docs for whatever reason. Local build works in CI it doesn't.

Just a +1 report for now but maybe the above action log helps.

[kamilkrzyskow]

@janaka I never worked with poetry too deeply, so I'm not sure if there are any missing logs etc., but the failed CI says it installed version 9.5.13:
https://github.com/docqai/docq/actions/runs/8590257742/job/23537555949#step:6:296

And in the other working CI where you disabled the social plugin it says it installed 9.5.17:
https://github.com/docqai/docq/actions/runs/8590353932/job/23537780963#step:6:296

Quite bizarre, perhaps the issue lies in the 2 different version definitions?
https://github.com/docqai/docq/blob/8c88f4c7019654167d6d1075414ee79fdf393cfe/pyproject.toml#L51
https://github.com/docqai/docq/blob/8c88f4c7019654167d6d1075414ee79fdf393cfe/pyproject.toml#L77

pip with requirements.txt is so much simpler 😸
perhaps you could clear the caches too 🤔

[vedranmiletic]

Just noticed today that my naive requirements.txt without version pins:

mkdocs[i18n]
mkdocs-material[recommended, imaging]

results in Material for MkDocs version 9.5.2 getting installed. This could be related to MkDocs 1.6.0 now available. Anyhow, I pinned Material to >=9.5.17 and it works fine, but reporting here just in case other use unpinned requirements and stumble upon this issue.

Edit: FWIW, specifying only mkdocs-material[recommended, imaging] also works.

[squidfunk]

We don't yet support MkDocs 1.6, because MkDocs has had a history of breaking changes in minor releases, which is why we've limited to <1.6 for now. We'll try to upgrade to 1.6 quickly, but I currently need to finish other things for Material for MkDocs, so I'm happy to collaborate if somebody wants to go ahead and do the legwork ☺️

Edit: This is now tracked in #7076.