Adding a Sitemap and Robots.txt to a Next.js Site

Overview

Sitemaps serve as a directory of the pages and content on your website, and creating these files is one of the most basic SEO steps to take. A sitemap is essentially a roadmap to the pages on your website. It can be in either HTML or XML format, with the latter used in the next-sitemap package that I’ll discuss here.

robots.txt communicates which pages your site permits bots to crawl. While this considered a terms of use agreement, it does not restrict actors from accessing your site. This file is not sufficient for ensuring operational security. Additional measures must be implemented to prevent unauthorized access to URL paths and files. The robots.txt file is a tool to assist search engines in crawling your site.

Generating sitemaps and a robots.txt file is incredibly simple with next-sitemap. Prioritizing efficiency, I used the most basic configuration for this package, which met my needs. As the blog grows, I may revisit some of the configuration options. I just needed these files, and I did not want to manually create them. It’s also great that the postbuild script ensures that the files are updated whenever the application is built for production. This means that anytime I add a new post and rebuild, the sitemap will be updated with the new URL.

Using next-sitemap

By default, the build process generates an index sitemap along with other sitemaps containing website URLs. To generate a robots.txt file, you must add generateRobotsTxt: true as an option in the configuration file. The project’s README contains information on splitting sitemap files when you have a large number of URLs (over 7000). That information can be found in this section.

  1. Install the package using your package manager of choice:
yarn add next-sitemap
  1. Next, you’ll need to create a configuration file. This should go in the root directory of your Next.js project. The file must be called next-sitemap.config.js.
// next-sitemap.config.js

/** @type {import('next-sitemap').IConfig} */
module.exports = {
    siteUrl: process.env.SITE_URL || 'https://paul-schick.com',
    generateRobotsTxt: true,
}
  1. I added a postbuild script to my package.json to generate these files when the application is built for production.
// package.json scripts
{
  "scripts": {
    "dev": "next",
    "build": "next build",
    "postbuild": "next-sitemap"
  }
}

The command run by node is simply next-sitemap. I’ll be covering more postbuild stuff in a later post while I document my Next.js blog strategy. I’m using static files rather than a database (for now), so I want to make use of some command line builds to reduce indexing time and generate static files.