Leaf and Kettle's code runs in more than one place: the shop at https://leafandkettle.example, a staging site at https://staging.leafandkettle.example, and a preview deploy for every pull request. Search engines found the staging site and started showing its half-finished pages. Write app/robots.ts. Its default export returns a MetadataRoute.Robots object, and Next.js turns it into /robots.txt.
env.SITE_URL in @/lib/env is the address of the deploy being built. When its hostname is exactly PRODUCTION_HOSTNAME, crawlers are welcome: return one rule for every crawler (userAgent: "*") that allows / and disallows /account and /checkout, and a sitemap with the sitemap's full URL, built from env.SITE_URL. A SITE_URL may end with a slash, and the sitemap's URL still has just one before sitemap.xml.
On any other host, whether staging, a preview, or localhost, return one rule for every crawler that disallows /, and no sitemap, so nothing on that deploy is crawled.
The tests change env.SITE_URL to play each deploy, so read it inside robots().
SITE_URL https://leafandkettle.example → /robots.txt
User-Agent: *
Allow: /
Disallow: /account
Disallow: /checkout
Sitemap: https://leafandkettle.example/sitemap.xml
SITE_URL https://staging.leafandkettle.example → /robots.txt
User-Agent: *
Disallow: /
lib/env.ts, read-only. Import it as @/lib/env.
/**
* This deploy's settings. A real app reads them from process.env, which each deploy sets for itself:
* SITE_URL is https://leafandkettle.example in production, https://staging.leafandkettle.example on
* staging, and the preview's own address on every pull request.
*/
export const env = {
SITE_URL: "https://leafandkettle.example",
};
/** The hostname of the one deploy search engines may crawl: the shop itself. */
export const PRODUCTION_HOSTNAME = "leafandkettle.example";
// Each rule as lists. A rule without a userAgent is for every crawler, as Next.js writes it.
const rulesOf = (robots) =>
[].concat(robots.rules).map((rule) => ({
userAgent: [].concat(rule.userAgent ?? "*"),
allow: [].concat(rule.allow ?? []),
disallow: [].concat(rule.disallow ?? []).sort(),
}));
const sitemapsOf = (robots) => [].concat(robots.sitemap ?? []);
expect(rulesOf(await Robots())).toEqual([{ userAgent: ["*"], allow: ["/"], disallow: ["/account", "/checkout"] }]);
// Each rule as lists. A rule without a userAgent is for every crawler, as Next.js writes it.
const rulesOf = (robots) =>
[].concat(robots.rules).map((rule) => ({
userAgent: [].concat(rule.userAgent ?? "*"),
allow: [].concat(rule.allow ?? []),
disallow: [].concat(rule.disallow ?? []).sort(),
}));
const sitemapsOf = (robots) => [].concat(robots.sitemap ?? []);
expect(sitemapsOf(await Robots())).toEqual(["https://leafandkettle.example/sitemap.xml"]);
// Each rule as lists. A rule without a userAgent is for every crawler, as Next.js writes it.
const rulesOf = (robots) =>
[].concat(robots.rules).map((rule) => ({
userAgent: [].concat(rule.userAgent ?? "*"),
allow: [].concat(rule.allow ?? []),
disallow: [].concat(rule.disallow ?? []).sort(),
}));
const sitemapsOf = (robots) => [].concat(robots.sitemap ?? []);
env.SITE_URL = "https://leafandkettle.example/";
const robots = await Robots();
expect(sitemapsOf(robots)).toEqual(["https://leafandkettle.example/sitemap.xml"]);
expect(rulesOf(robots)).toEqual([{ userAgent: ["*"], allow: ["/"], disallow: ["/account", "/checkout"] }]);
// Each rule as lists. A rule without a userAgent is for every crawler, as Next.js writes it.
const rulesOf = (robots) =>
[].concat(robots.rules).map((rule) => ({
userAgent: [].concat(rule.userAgent ?? "*"),
allow: [].concat(rule.allow ?? []),
disallow: [].concat(rule.disallow ?? []).sort(),
}));
const sitemapsOf = (robots) => [].concat(robots.sitemap ?? []);
env.SITE_URL = "https://staging.leafandkettle.example";
const robots = await Robots();
expect(rulesOf(robots)).toEqual([{ userAgent: ["*"], allow: [], disallow: ["/"] }]);
expect(sitemapsOf(robots)).toEqual([]);
// Each rule as lists. A rule without a userAgent is for every crawler, as Next.js writes it.
const rulesOf = (robots) =>
[].concat(robots.rules).map((rule) => ({
userAgent: [].concat(rule.userAgent ?? "*"),
allow: [].concat(rule.allow ?? []),
disallow: [].concat(rule.disallow ?? []).sort(),
}));
const sitemapsOf = (robots) => [].concat(robots.sitemap ?? []);
for (const url of ["https://leaf-and-kettle-git-new-teas.vercel.app", "http://localhost:3000", "https://leafandkettle.example.preview.dev", "https://shop-leafandkettle.example"]) {
env.SITE_URL = url;
const robots = await Robots();
expect(rulesOf(robots)).toEqual([{ userAgent: ["*"], allow: [], disallow: ["/"] }]);
expect(sitemapsOf(robots)).toEqual([]);
}
Run tests runs every test on your code, here in your browser. The problem is solved when they all pass.
Ctrl + Enter runs the tests (⌘ + Enter on a Mac).
I'm here to help with A robots.txt that keeps staging out of search. I give hints, never the solution, so you get to solve it yourself. Ask for a hint, select code to ask about it, or press Ask Tuto why on a failing test.
Your first 3 answers need no account. for free to keep asking.