feat: add <lastmod> to sitemap #9234

pmarschik · 2023-08-16T15:21:53Z

Pre-flight checklist

I have read the Contributing Guidelines on pull requests.
If this is a code change: I have written unit tests and/or added dogfooding pages to fully verify the new behavior.
If this is a new API or substantial change: the PR has an accompanying issue (closes #0000) and the maintainers have approved on my working plan.

Motivation

Add <lastmod> to sitemap, should fix #2604.

Test Plan

Extend existing sitemap tests to expect generated lastmod date.

Test links

Deploy preview: https://deploy-preview-_____--docusaurus-2.netlify.app/sitemap.xml

Related issues/PRs

Add lastmod to sitemap #2604

facebook-github-bot · 2023-08-16T15:21:58Z

Hi @pmarschik!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

netlify · 2023-08-16T15:27:06Z

✅ [V2]

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`ddba059`
🔍 Latest deploy log	https://app.netlify.com/sites/docusaurus-2/deploys/65d602d27e9e3f0008b202bb
😎 Deploy Preview	https://deploy-preview-9234--docusaurus-2.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

github-actions · 2023-08-16T15:27:47Z

⚡️ Lighthouse report for the deploy preview of this PR

URL	Performance	Accessibility	Best Practices	SEO	PWA	Report
/	🟠 59	🟢 98	🟢 100	🟢 100	🟠 89	Report
/docs/installation	🟠 89	🟢 98	🟢 100	🟢 100	🟠 89	Report
/docs/category/getting-started	🟠 77	🟢 100	🟢 100	🟢 90	🟠 89	Report
/blog	🟠 75	🟢 100	🟢 100	🟢 90	🟠 89	Report
/blog/preparing-your-site-for-docusaurus-v3	🟠 64	🟢 97	🟢 100	🟢 100	🟠 89	Report
/blog/tags/release	🟠 76	🟢 100	🟢 100	🟠 80	🟠 89	Report
/blog/tags	🟠 77	🟢 100	🟢 100	🟢 90	🟠 89	Report

Co-authored-by: Joshua Chen <sidachen2003@gmail.com>

ardeche07 · 2024-02-20T09:18:06Z

Thanks for submitting this @pmarschik ! Have you gotten any feedback on this PR? We use Docusaurus at Unleash and were just trying to add last mod to our docs since it is becoming increasingly important for Google. Would love to see this merged.

pmarschik · 2024-02-20T09:20:51Z

@ardeche07 I've not gotten any feedback, however I've been using the changes for our published docusaurus page for some time. When I get around it, I'll update the PR with the latest changes (its not actually 170 files changed, this PR was based on docusaurus v2)

slorber

Thanks for your contribution @pmarschik and sorry for the late review

Your PR is not really in a state I could merge, and try to implement things that are not really needed according to Google recommendations.

It should also target main and not docusaurus-v3 branch

Let me know if you want to give it another try, otherwise I'll create another PR.

More details on what we want to implement here:
#2604 (comment)

slorber · 2024-03-15T13:21:13Z

packages/docusaurus-plugin-sitemap/src/createSitemap.ts

+      lastmod:
+        routes !== undefined && options.lastmod !== false
+          ? matchRoutes(rrRoutes, routePath).find(
+            ({route}) => route?.exact === true,
+          )?.route?.lastModified
+          : undefined,
+    });
+  });


that's not ideal to use matchRoute this way.

matchRoute can be expensive particularly for very large sites with thousands of routes

We should create the sitemap directly from the new routes attribute.

We have an util somewhere in core to do exactly that, and is what we use to create the routesPaths attribute.

export function getAllFinalRoutes(routeConfig: RouteConfig[]): RouteConfig[] { function getFinalRoutes(route: RouteConfig): RouteConfig[] { return route.routes ? route.routes.flatMap(getFinalRoutes) : [route]; } return routeConfig.flatMap(getFinalRoutes); }

Honestly I'm not a fan of the routesPaths attribute, and will likely try to remove it in favor of just routes in the future, so I'll take this opportunity to refactor the sitemap plugin to not use routesPaths and use routes instead.

slorber · 2024-03-15T13:23:31Z

packages/docusaurus-plugin-sitemap/src/options.ts

+   * truncated to date and not include the timestamp.
+   * @see https://www.sitemaps.org/protocol.html#xmlTagDefinitions
+   */
+  lastmod: boolean | 'date';


Wonder if we really need that option? 🤔

Maybe it could be useful but afaik nobody asked for this yet

slorber · 2024-03-15T13:26:23Z

packages/docusaurus-types/src/routing.d.ts

+  /** Used for sitemap generation. Will be exposed in the lastmod tag. */
+  lastModified?: Date;


I would probably add this to a metadata subattribute because we may assign other attributes to routes in the future and also let plugin authors assign custom ones (using an interface to allow typesafe extension through TS interface declaration merging).

These attributes should only be available on the Node side and not be part of the .docusaurus/routes.js file generated

slorber · 2024-03-15T13:27:06Z

packages/docusaurus-plugin-content-pages/src/index.ts

@@ -176,6 +176,9 @@ export default function pluginContentPages(
              modules: {
                content: source,
              },
+              // currently no way to get last modified date for pages
+              // we would have to refactor lastUpdate from docs
+              lastModified: undefined,


doesn't look like necessary to add

slorber · 2024-03-15T13:27:11Z

packages/docusaurus-plugin-content-pages/src/index.ts

+              // currently no way to get last modified date for pages
+              // would have to refactor lastUpdate from docs
+              lastModified: undefined,


doesn't look like necessary to add

slorber · 2024-03-15T13:38:26Z

packages/docusaurus-plugin-content-blog/src/blogUtils.ts

+        latestModified: _.maxBy(blogPostsOnPage, (item) => item.metadata.date)
+          ?.metadata?.date,


slorber · 2024-03-15T13:39:04Z

packages/docusaurus-plugin-content-blog/src/index.ts

+          lastModified: _.maxBy(blogPosts, (item) => item.metadata.date)
+            ?.metadata?.date,


not needed?

slorber · 2024-03-15T13:39:30Z

packages/docusaurus-plugin-content-blog/src/index.ts

@@ -299,6 +303,7 @@ export default async function pluginContentBlog(
              items: blogPostItemsModule(items),
              metadata: aliasedSource(pageMetadataPath),
            },
+            lastModified: metadata.latestModified,


slorber · 2024-03-15T13:40:14Z

packages/docusaurus-plugin-content-blog/src/index.ts

@@ -274,6 +277,7 @@ export default async function pluginContentBlog(
              sidebar: aliasedSource(sidebarProp),
              content: metadata.source,
            },
+            lastModified: metadata.date,


we have metadata.lastUpdatedAt on blog posts too now

slorber · 2024-03-15T13:40:27Z

packages/docusaurus-plugin-content-blog/src/plugin-content-blog.d.ts

+    /** Latest modified date of posts on the current page. */
+    readonly latestModified?: Date;


pmarschik · 2024-03-15T14:08:39Z

Thanks for your contribution @pmarschik and sorry for the late review

Thank you for reviewing the MR @slorber.

Your PR is not really in a state I could merge, and try to implement things that are not really needed according to Google recommendations.

It should also target main and not docusaurus-v3 branch

I'm aware, this is just what I use to pnpm patch our build and have a working lastmod until the feature is available upstream.

Let me know if you want to give it another try, otherwise I'll create another PR.

I'm currently occupied with other projects and I don't know when I next have time to circle back to this.
So I think it would be best if you create another PR.

slorber · 2024-03-15T15:38:41Z

Thanks

I'll close this one in favor of a new one, track #9954

feat: add <lastmod> to sitemap

088e807

pmarschik requested review from slorber, lex111 and Josh-Cena as code owners August 16, 2023 15:21

facebook-github-bot added the CLA Signed Signed Facebook CLA label Sep 19, 2023

homotechsual and others added 6 commits October 31, 2023 23:00

fix(website): Fix typo in announcement bar (facebook#9484)

9a425bc

Co-authored-by: Joshua Chen <sidachen2003@gmail.com>

fix(create-docusaurus): fix readme docusaurus 2 ref (facebook#9487)

791da2e

Merge branch 'main' into lastmod-sitemap

3a2b5bf

Merge remote-tracking branch 'origin/docusaurus-v3' into lastmod-sitemap

63f8f3c

fix build for me

334c4b3

Merge remote-tracking branch 'origin/docusaurus-v3' into lastmod-sitemap

3e0b37b

pmarschik changed the base branch from main to docusaurus-v3 February 21, 2024 14:03

Merge remote-tracking branch 'origin/docusaurus-v3' into lastmod-sitemap

ddba059

slorber mentioned this pull request Mar 15, 2024

feat(blog): add LastUpdateAuthor & LastUpdateTime #9912

Merged

3 tasks

slorber requested changes Mar 15, 2024

View reviewed changes

slorber mentioned this pull request Mar 15, 2024

feat(sitemap): add support for "lastmod" #9954

Merged

slorber closed this Mar 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add <lastmod> to sitemap #9234

feat: add <lastmod> to sitemap #9234

pmarschik commented Aug 16, 2023

facebook-github-bot commented Aug 16, 2023

netlify bot commented Aug 16, 2023 •

edited

github-actions bot commented Aug 16, 2023 •

edited

ardeche07 commented Feb 20, 2024

pmarschik commented Feb 20, 2024

slorber left a comment

slorber Mar 15, 2024

slorber Mar 15, 2024

slorber Mar 15, 2024

slorber Mar 15, 2024

slorber Mar 15, 2024

slorber Mar 15, 2024

slorber Mar 15, 2024

slorber Mar 15, 2024

slorber Mar 15, 2024

slorber Mar 15, 2024

pmarschik commented Mar 15, 2024

slorber commented Mar 15, 2024

		/** Used for sitemap generation. Will be exposed in the lastmod tag. */
		lastModified?: Date;

		latestModified: _.maxBy(blogPostsOnPage, (item) => item.metadata.date)
		?.metadata?.date,

		lastModified: _.maxBy(blogPosts, (item) => item.metadata.date)
		?.metadata?.date,

		/** Latest modified date of posts on the current page. */
		readonly latestModified?: Date;

feat: add <lastmod> to sitemap #9234

feat: add <lastmod> to sitemap #9234

Conversation

pmarschik commented Aug 16, 2023

Pre-flight checklist

Motivation

Test Plan

Test links

Related issues/PRs

facebook-github-bot commented Aug 16, 2023

Action Required

Process

netlify bot commented Aug 16, 2023 • edited

✅ [V2]

github-actions bot commented Aug 16, 2023 • edited

⚡️ Lighthouse report for the deploy preview of this PR

ardeche07 commented Feb 20, 2024

pmarschik commented Feb 20, 2024

slorber left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmarschik commented Mar 15, 2024

slorber commented Mar 15, 2024

netlify bot commented Aug 16, 2023 •

edited

github-actions bot commented Aug 16, 2023 •

edited