Robots.txt

## 1. Learning Objectives By the end of this lesson, you will be able to generate `robots.txt`, allow and disallow crawler paths, link your sitemap, and explain why robots rules do not protect private data. Difficulty: Beginner. ## 2. Prerequisites - Sitemap basics. - Public vs private routes. - SEO crawling basics. ## 3. Overview `robots.txt` is a crawler instruction file. In Next.js App Router, `app/robots.ts` can generate the file at `/robots.txt`. ## 4. Why This Topic Matters Robots rules guide crawlers away from low-value or duplicate areas and point them toward your sitemap. But they are public hints, not access control. ## 5. Real-World Analogy `robots.txt` is like a polite sign saying "Visitors, please do not enter this hallway." It helps cooperative visitors, but it is not a locked door. ## 6. Core Concepts | Concept | Meaning | |---|---| | User agent | Crawler identity the rule applies to. | | Allow | Path crawler may crawl. | | Disallow | Path crawler should avoid. | | Sitemap | URL to sitemap file. | | Crawl directive | Public instruction, not security. | ## 7. Syntax & API Reference

## 8. Visual Diagram

## 9. Live Example - Full Working Code

What just happened? Next.js generates `/robots.txt` from a typed route metadata file. ## 10. Interactive Playground Try this: - Add `app/robots.ts`. - Visit `/robots.txt`. - Add a sitemap URL. ## 11. Common Mistakes | Mistake | Why It Happens | Correct Approach | |---|---|---| | Treating robots as security | Disallow sounds like blocking. | Use authentication for private data. | | Blocking important public pages | Rules are too broad. | Test crawl rules before release. | | Forgetting sitemap link | Sitemap lives separately. | Add sitemap URL in robots output. | ## 12. Best Practices - Disallow private and low-value crawl paths. - Never rely on robots to protect sensitive information. - Include your sitemap URL. - Review rules before major SEO releases. - Keep staging robots restrictive if publicly reachable. ## 13. Browser Compatibility | Feature | Browser Impact | Notes | |---|---|---| | `robots.txt` | Crawler-facing | Browser can display text. | | Disallow rules | Cooperative crawler behavior | Not access control. | | Sitemap field | Crawler discovery | Points to sitemap URL. | ## 14. Interview Questions **Easy:** What file generates `/robots.txt` in App Router? Answer: `app/robots.ts`. **Medium:** Is `robots.txt` a security feature? Answer: No. It is a crawler instruction file. **Hard:** Why might you disallow search result pages? Answer: Internal search pages can create duplicate, low-value, or infinite crawl spaces. ## 15. Debugging Exercise Bug report: "Google is not crawling our blog." Solution Check `robots.txt` for overly broad disallow rules, verify sitemap URLs, and confirm pages are indexable and return 200. ## 16. Practice Exercises - Easy: Allow all crawlers. - Medium: Disallow `/admin` and `/account`. - Hard: Create different rules for a specific bot. ## 17. Scenario-Based Challenge Your staging site is public but should not be indexed. What should you do? Walkthrough Use authentication or network restriction first. Add restrictive robots and noindex headers as crawl/indexing signals, but do not rely on robots alone. ## 18. Quick Quiz 1. Robots file route? Answer: `/robots.txt`. 2. Next.js generator file? Answer: `app/robots.ts`. 3. Is robots a lock? Answer: No. 4. What links crawlers to URL lists? Answer: `sitemap`. 5. Should admin pages be public? Answer: No. ## 19. Summary & Key Takeaways - `robots.txt` guides cooperative crawlers. - Next.js can generate it with `app/robots.ts`. - Include sitemap location. - Do not treat disallow as security. - Test rules to avoid blocking valuable pages. ## 20. Cheat Sheet | Need | Robots Field | |---|---| | All crawlers | `userAgent: '*'` | | Crawl public site | `allow: '/'` | | Avoid admin | `disallow: '/admin'` | | Link sitemap | `sitemap` | | Host hint | `host` | ## 21. Further Reading - Next.js Docs: robots.txt. - Google Search Central: robots.txt. - robots.org. ## 22. Next Lesson Preview Next, you will learn Open Graph metadata for rich social previews.