Skip to the code

A robots.txt that keeps staging out of search

Leaf and Kettle's code runs in more than one place: the shop at https://leafandkettle.example, a staging site at https://staging.leafandkettle.example, and a preview deploy for every pull request. Search engines found the staging site and started showing its half-finished pages. Write app/robots.ts. Its default export returns a MetadataRoute.Robots object, and Next.js turns it into /robots.txt.

env.SITE_URL in @/lib/env is the address of the deploy being built. When its hostname is exactly PRODUCTION_HOSTNAME, crawlers are welcome: return one rule for every crawler (userAgent: "*") that allows / and disallows /account and /checkout, and a sitemap with the sitemap's full URL, built from env.SITE_URL. A SITE_URL may end with a slash, and the sitemap's URL still has just one before sitemap.xml.

On any other host, whether staging, a preview, or localhost, return one rule for every crawler that disallows /, and no sitemap, so nothing on that deploy is crawled.

The tests change env.SITE_URL to play each deploy, so read it inside robots().

SITE_URL https://leafandkettle.example → /robots.txt
User-Agent: *
Allow: /
Disallow: /account
Disallow: /checkout
 
Sitemap: https://leafandkettle.example/sitemap.xml
 
SITE_URL https://staging.leafandkettle.example → /robots.txt
User-Agent: *
Disallow: /

The app's other files

lib/env.ts, read-only. Import it as @/lib/env.

/**
 * This deploy's settings. A real app reads them from process.env, which each deploy sets for itself:
 * SITE_URL is https://leafandkettle.example in production, https://staging.leafandkettle.example on
 * staging, and the preview's own address on every pull request.
 */
export const env = {
  SITE_URL: "https://leafandkettle.example",
};
 
/** The hostname of the one deploy search engines may crawl: the shop itself. */
export const PRODUCTION_HOSTNAME = "leafandkettle.example";

What the tests check

  1. production lets every crawler in, except to /account and /checkout
    // Each rule as lists. A rule without a userAgent is for every crawler, as Next.js writes it.
    const rulesOf = (robots) =>
      [].concat(robots.rules).map((rule) => ({
        userAgent: [].concat(rule.userAgent ?? "*"),
        allow: [].concat(rule.allow ?? []),
        disallow: [].concat(rule.disallow ?? []).sort(),
      }));
    const sitemapsOf = (robots) => [].concat(robots.sitemap ?? []);
    expect(rulesOf(await Robots())).toEqual([{ userAgent: ["*"], allow: ["/"], disallow: ["/account", "/checkout"] }]);
    
  2. production names the sitemap by its full URL
    // Each rule as lists. A rule without a userAgent is for every crawler, as Next.js writes it.
    const rulesOf = (robots) =>
      [].concat(robots.rules).map((rule) => ({
        userAgent: [].concat(rule.userAgent ?? "*"),
        allow: [].concat(rule.allow ?? []),
        disallow: [].concat(rule.disallow ?? []).sort(),
      }));
    const sitemapsOf = (robots) => [].concat(robots.sitemap ?? []);
    expect(sitemapsOf(await Robots())).toEqual(["https://leafandkettle.example/sitemap.xml"]);
    
  3. a SITE_URL ending in a slash still gives one slash before sitemap.xml
    // Each rule as lists. A rule without a userAgent is for every crawler, as Next.js writes it.
    const rulesOf = (robots) =>
      [].concat(robots.rules).map((rule) => ({
        userAgent: [].concat(rule.userAgent ?? "*"),
        allow: [].concat(rule.allow ?? []),
        disallow: [].concat(rule.disallow ?? []).sort(),
      }));
    const sitemapsOf = (robots) => [].concat(robots.sitemap ?? []);
    env.SITE_URL = "https://leafandkettle.example/";
    const robots = await Robots();
    expect(sitemapsOf(robots)).toEqual(["https://leafandkettle.example/sitemap.xml"]);
    expect(rulesOf(robots)).toEqual([{ userAgent: ["*"], allow: ["/"], disallow: ["/account", "/checkout"] }]);
    
  4. staging blocks every crawler from every page, and names no sitemap
    // Each rule as lists. A rule without a userAgent is for every crawler, as Next.js writes it.
    const rulesOf = (robots) =>
      [].concat(robots.rules).map((rule) => ({
        userAgent: [].concat(rule.userAgent ?? "*"),
        allow: [].concat(rule.allow ?? []),
        disallow: [].concat(rule.disallow ?? []).sort(),
      }));
    const sitemapsOf = (robots) => [].concat(robots.sitemap ?? []);
    env.SITE_URL = "https://staging.leafandkettle.example";
    const robots = await Robots();
    expect(rulesOf(robots)).toEqual([{ userAgent: ["*"], allow: [], disallow: ["/"] }]);
    expect(sitemapsOf(robots)).toEqual([]);
    
  5. previews, localhost, and look-alike hosts are blocked too
    // Each rule as lists. A rule without a userAgent is for every crawler, as Next.js writes it.
    const rulesOf = (robots) =>
      [].concat(robots.rules).map((rule) => ({
        userAgent: [].concat(rule.userAgent ?? "*"),
        allow: [].concat(rule.allow ?? []),
        disallow: [].concat(rule.disallow ?? []).sort(),
      }));
    const sitemapsOf = (robots) => [].concat(robots.sitemap ?? []);
    for (const url of ["https://leaf-and-kettle-git-new-teas.vercel.app", "http://localhost:3000", "https://leafandkettle.example.preview.dev", "https://shop-leafandkettle.example"]) {
      env.SITE_URL = url;
      const robots = await Robots();
      expect(rulesOf(robots)).toEqual([{ userAgent: ["*"], allow: [], disallow: ["/"] }]);
      expect(sitemapsOf(robots)).toEqual([]);
    }
    

Learn more

app/robots.tsSaved in this browser

Run tests runs every test on your code, here in your browser. The problem is solved when they all pass.

Ctrl + Enter runs the tests (⌘ + Enter on a Mac).