Chatbot Icon

May 29, 2023

Next.js: How to implement a dynamic sitemap

Written By

Adam Lyth


How to

Read time

4 minute

Sitemaps are an essential element of SEO best practices, allowing search engines to efficiently crawl and index a website. For new websites looking to have their new blogs, articles or content picked up by Google and other search engines, they are essential. A sitemap allows search engines to to easily identify pages and updates, which ultimately can lead to better SEO and site traffic.
However, maintaining a sitemap can be challenging, especially for websites built with Next.js that use a headless CMS with dynamic content. The challenge lies in keeping the sitemap up to date with the dynamically changing content, if you create a new article for example, you’ll want search engines to pick it up as fast as possible. In this article, we'll explore the benefits and drawbacks of sitemaps and provide a step-by-step guide on how to set up a dynamic sitemap for a Next.js website using a headless CMS.

The Benefits of Sitemaps for SEO

  1. Improved Crawlability: Sitemaps are even more vital for large websites, as they may have pages that Google's web crawlers could overlook. A sitemap provides a comprehensive guide, ensuring no page is left uncrawled.
  2. Better Indexing for Isolated Pages: Sometimes, certain pages on your site may not be well-linked, creating isolated pockets of content. In such cases, a sitemap becomes essential to ensure that these pages are not missed by Google's indexing bots.
  3. Promotion of New Sites: New websites often struggle with visibility due to a lack of external links. A sitemap helps search engine bots discover and index all the pages on your site, thus overcoming this hurdle.
  4. Enhanced Visibility for Rich Media Content: If your Next.js website features rich media content like videos and images, or is featured in Google News, having a sitemap allows Google to take additional information into account for search, improving the visibility of this content.

The Drawbacks of Sitemaps

  1. Not a Guarantee: Submitting a sitemap doesn't guarantee that all pages will be indexed by Google. Numerous other factors, like the quality and relevance of the page content, come into play.
  2. Doesn’t Improve Rankings: While a sitemap helps search engines find and index your pages, it doesn't directly enhance your rankings. You still need to focus on content quality and SEO optimisation for high search rankings.

Building a Dynamic Sitemap in Next.js

Considering the challenge of maintaining a sitemap, especially for dynamic websites built with Next.js, automating the sitemap generation process becomes critical. Here's how to build a dynamic sitemap in Next.js:
First we need to create a sitemap baseline which will be computed at build time. We are going to do this by crawling the pages folder which automatically updates a json file in our project. First let’s install globby and create a script to generate the baseline sitemap:
# Install Globby so we can interface with the files system. yarn add -D globby
// scripts/generate-sitemap.mjs import { writeFileSync } from 'fs'; import { globby } from 'globby'; async function generate() { const pages = await globby([ 'pages/*.tsx', 'data/**/*.mdx', '!data/*.mdx', '!pages/_*.tsx', '!pages/api', '!pages/404.tsx', '!pages/sitemap.xml.tsx', ]); const formatted = { paths: pages .map((page) => { const path = page .replace('pages', '') .replace('data', '') .replace('.tsx', '') .replace('.mdx', ''); const route = path === '/index' ? '' : path; return `${route}`; }) }; writeFileSync('pages/sitemap.xml/base.json', JSON.stringify(formatted)); } generate();
Once the script is set up we need to add a line to our package.json
"scripts": { ... "prebuild": "node ./scripts/generate-sitemap.mjs" }
The above will now crawl the pages folder creating a list of paths which don’t change until more code is committed. This will form our sitemap baseline. We now need to dynamically add in any dynamic content controlled by a headless CMS such as blog posts.
To do this we are going to create a new page which will automatically pick up our base.json and add in additional paths on demand, taking advantage of getServerSideProps to perform any api requests required.
//pages/sitemap.xml/index.tsx import { NextPageContext } from 'next'; import base from './base.json'; const createSitemap = ( blogs: string[] ) => { return `<?xml version="1.0" encoding="UTF-8"?> <urlset xmlns=""> ${base.paths .map( (path) => ` <url> <loc>${path}</loc> </url> ` ) .join('')} ${blogs .map((path) => { return ` <url> <loc>${`${path}`}</loc> </url> `; }) .join('')} </urlset> `; }; function SiteMap() { // getServerSideProps will do the heavy lifting } export async function getServerSideProps({ res }: NextPageContext) { // Add code here to connect to your third party service and get your additional paths res?.setHeader('Content-Type', 'text/xml'); res?.write(createSitemap(paths?.filter((path) => !!path) ?? [])); res?.end(); return { props: {}, }; } export default SiteMap;
//pages/sitemap.xml/base.json { "paths": [] }
From above we can see that getServerSideProps is performing an api request to get the additional paths the website is expecting to host, it then passes it into createSitemap which dynamically renders the different options. The base.json will be ran at build-time and will populate with all non-dynamic paths from the pages folder.
Lastly we need to create a robots.txt to allow Google and other search engines to find our sitemap, copy this into public/robots.txt
User-agent: *
Don’t forget to test your sitemap, once deployed (or locally), if you head to you should now find a location list of pages your website is hosting!
In conclusion, sitemaps are not just vital tools for SEO, they're an indispensable asset when optimising your website. By aiding in more efficient indexing by search engines, sitemaps significantly boost your site's discoverability. Here's where Next.js steps up to the challenge - with its capability to automate and dynamically generate a sitemap, your sitemap is always kept current, accurately reflecting your site's content. This is more than a convenience; it's a powerful strategy that can greatly enhance your SEO ranking, streamline the user experience by improving site navigation, and ultimately increase your site's traffic and conversion rates.


Related posts

How did you find your chatbot experience today?

Thank you for your feedback

There has been an unexpected error, please try again later