ReviseAlgo Logo

SEO

Robots.txt

Learn how robots.txt works in Next.js, how to generate crawl rules, and why robots directives are not security controls.

1. Learning Objectives

By the end of this lesson, you will be able to generate robots.txt, allow and disallow crawler paths, link your sitemap, and explain why robots rules do not protect private data.

Difficulty: Beginner.

2. Prerequisites

  • Sitemap basics.
  • Public vs private routes.
  • SEO crawling basics.

    • 3. Overview

    robots.txt is a crawler instruction file. In Next.js App Router, app/robots.ts can generate the file at /robots.txt.

    4. Why This Topic Matters

    Robots rules guide crawlers away from low-value or duplicate areas and point them toward your sitemap. But they are public hints, not access control.

    5. Real-World Analogy

    robots.txt is like a polite sign saying "Visitors, please do not enter this hallway." It helps cooperative visitors, but it is not a locked door.

    6. Core Concepts

    ConceptMeaning
    User agentCrawler identity the rule applies to.
    AllowPath crawler may crawl.
    DisallowPath crawler should avoid.
    SitemapURL to sitemap file.
    Crawl directivePublic instruction, not security.

    7. Syntax & API Reference

    8. Visual Diagram

    9. Live Example - Full Working Code

    What just happened? Next.js generates /robots.txt from a typed route metadata file.

    10. Interactive Playground

    Try this:

  • Add app/robots.ts.
  • Visit /robots.txt.
  • Add a sitemap URL.

    • 11. Common Mistakes

    MistakeWhy It HappensCorrect Approach
    Treating robots as securityDisallow sounds like blocking.Use authentication for private data.
    Blocking important public pagesRules are too broad.Test crawl rules before release.
    Forgetting sitemap linkSitemap lives separately.Add sitemap URL in robots output.

    12. Best Practices

  • Disallow private and low-value crawl paths.
  • Never rely on robots to protect sensitive information.
  • Include your sitemap URL.
  • Review rules before major SEO releases.
  • Keep staging robots restrictive if publicly reachable.

    • 13. Browser Compatibility

    FeatureBrowser ImpactNotes
    robots.txtCrawler-facingBrowser can display text.
    Disallow rulesCooperative crawler behaviorNot access control.
    Sitemap fieldCrawler discoveryPoints to sitemap URL.

    14. Interview Questions

    Easy: What file generates /robots.txt in App Router?

    Answer: app/robots.ts.

    Medium: Is robots.txt a security feature?

    Answer: No. It is a crawler instruction file.

    Hard: Why might you disallow search result pages?

    Answer: Internal search pages can create duplicate, low-value, or infinite crawl spaces.

    15. Debugging Exercise

    Bug report: "Google is not crawling our blog."

    Solution

    Check robots.txt for overly broad disallow rules, verify sitemap URLs, and confirm pages are indexable and return 200.

    16. Practice Exercises

  • Easy: Allow all crawlers.
  • Medium: Disallow /admin and /account.
  • Hard: Create different rules for a specific bot.

    • 17. Scenario-Based Challenge

    Your staging site is public but should not be indexed. What should you do?

    Walkthrough

    Use authentication or network restriction first. Add restrictive robots and noindex headers as crawl/indexing signals, but do not rely on robots alone.

    18. Quick Quiz

    1. Robots file route? Answer: /robots.txt. 2. Next.js generator file? Answer: app/robots.ts. 3. Is robots a lock? Answer: No. 4. What links crawlers to URL lists? Answer: sitemap. 5. Should admin pages be public? Answer: No.

    19. Summary & Key Takeaways

  • robots.txt guides cooperative crawlers.
  • Next.js can generate it with app/robots.ts.
  • Include sitemap location.
  • Do not treat disallow as security.
  • Test rules to avoid blocking valuable pages.

    • 20. Cheat Sheet

    NeedRobots Field
    All crawlersuserAgent: '*'
    Crawl public siteallow: '/'
    Avoid admindisallow: '/admin'
    Link sitemapsitemap
    Host hinthost

    21. Further Reading

  • Next.js Docs: robots.txt.
  • Google Search Central: robots.txt.
  • robots.org.

    • 22. Next Lesson Preview

    Next, you will learn Open Graph metadata for rich social previews.