RFC: Headless Sitemap

[colinmurphy]

Abstract

This RFC explores the challenges of incorporating sitemaps into a WordPress headless application and the broader issues faced by the WordPress headless community. After reviewing and assessing various integration approaches, this RFC propose solutions to offer flexibility and a better experience for a developer when adding a sitemap to their WordPress headless application.

Motivation

Our motivation is to provide developers with easy to follow examples and documentation on how to integrate a sitemap into their headless application. We want to offer different solutions when building your sitemap. Additionally we would like to address how to integrate some of the most SEO WordPress plugins that our community uses.

We also want to foster a deeper understanding of the best practices for sitemaps in a headless WordPress aoplication. This means continuously updating and expanding our documentation and examples. We welcome community contributions and aim to build documentation and examples that is intuitive and beneficial to our community.

Research

We have conducted research in a few different aspects relating to sitemaps

• We reviewed how Faust.js currently fetches and parses sitemaps and also we have reviewed feedback from the community through Github and Discord.
• We have experimented with Next.js on how you can generate sitemaps out of the box and different approaches (whether fetching the sitemap directly or using WP GraphQL to generate your own array of data)
• We also reviewed best practices around sitemaps and identified other sitemap features such as sitemap extensions.

Proposal

We propose to offer documentation and examples of how to build a sitemap in this toolkit repository.

From our research we determined that there was two main approaches to building a sitemap in headless WordPress. The first method was fetching and then parsing an existing sitemap from WordPress and the second approach was to create a custom sitemap by fetching the data through an API endpoint such as WP GraphQL.

For our initial set of examples and documentation we plan to document:

• How to fetch and parse a sitemap from WordPress (including examples for some of the common WP SEO plugins)
• How to create your own custom sitemap from a data API endpoint
• How to proxy the sitemaps from the back end through to the front end

Within these examples and documentation we also plan to offer solutions around caching or adding/combining sitemap extensions and multisite.

Alternatives Considered

We have decided not to create a CLI package at this time for sitemaps as we felt that it wasn't the best approach. We also think that adding documentation and examples would be a a better way approach to serve the community as it gives optionality for the developer on how best to build their sitemap for their headless application.

Adoption Strategy

We aim to create the extensive documentation and examples for the WordPress headless community in relation to sitemaps. We will promote and encourage community contributions and reviews to foster a collaborative learning environment.

How Do We Teach This?

We will ensure that our documentation and example projects are easy to follow and provide strong community support.

How Can You Help?

We are excited about this new toolkit and the opportunities it presents for the WordPress headless community.

We’d love your feedback on the following:

• Is this the best approach for the community regarding sitemaps? Are there better approaches we should consider?
• Are we missing any major topics or examples for sitemaps?
• What other topics would you like us to cover? We would love to hear from you!

RFC Changelog

• January 31, 2025 - Initial RFC for sitemaps.
• February 5, 2025 - Updated examples for sitemap.

[Fran-A-Dev]

This looks great to me! Thanks @colinmurphy !

[colinmurphy]

Thanks @Fran-A-Dev 😄

[moonmeister]

Another option is to proxy the sitemaps from the back end through to the front end. This is what I did on https://wpdecoupled.dev. https://github.com/wpdecoupled/site/blob/main/src/routes/%5Bfile%3Dsitemap%5D/%2Bserver.ts

• low overhead - very little implementation
• Sitemaps on WP must be correct. You could modify the contents of the sitemap but this adds latency.
• Without additional work/mods pages outside of WP aren't in the sitemap. But if your site is entirely CMs driven for sitemap content, this works just fine.

[colinmurphy]

Thanks @moonmeister

I think we can definitely add this as an example into our docs. I hadn't considered that you could just fetch and render a current WP sitemap as I thought it would require changes. It also as you said reduces overhead as you are just rendering the data in the response.

[theodesp]

When considering the approach of proxying sitemap.xml from WordPress in a Next.js project as part of this ticket
#48
several challenges arise:

• CORS and security Issues: Directly proxying resources from WordPress can lead to CORS errors due to domain mismatches.
• Complexity with Next.js Routing: Issues arise while proxying sitemap indexes for each type/category/tag etc that WordPress generates. For example WordPress generates sitemap indexes for post and pages like the following:

http://localhost:8888/wp-sitemap-posts-post-1.xml
--
http://localhost:8888/wp-sitemap-posts-page-1.xml

If directly proxying this url to Next.js it would require 2 separate files wp-sitemap-posts-post-1.xml.js and wp-sitemap-posts-page-1.xml to serve the index.

Unless we actually have a single route for sitemaps (just like Faust.js with the hybrid approach) and while this method works well in frameworks like Svelte, it might not be the most viable solution for Next.js due to these complexities.

[moonmeister]

So my svelte implementation uses regex to match any sitemap or xsl file. This requires the domains in those files on the backend to already be correct, no modifying these in the frontend. Next.js rewrites support regex matches. happy to jump on a call with someone and see what we can get working. The one thing I did in transit in Svelte was modify the CORS header.

regex /^(?:.*sitemap.*).(xml|xsl)$/

[ahuseyn]

A Note on WPGraphQL Sitemap Implementation

When implementing a custom sitemap with data fetched from WPGraphQL, the pagination limit should be taken into account. Since WPGraphQL limits the maximum number of nodes to 100, fetching all the data requires paginated data fetching. The viability of this approach depends on the size of the website, update frequency, and available resources.

Alternatively, extending WPGraphQL to add extra fields for the sitemap could be considered. As the most flexible approach, it allows sitemaps to be fully customized to meet the website's requirements. However, this approach requires PHP implementation. We will provide an example using this method to demonstrate a custom sitemap in a headless environment with the data fetched through the custom WPGraphQL fields.