juyung/fuwari
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix: disallow crawling from /_astro/ (#399)

Browse source 
* fix: disallow crawling from `/_astro/`

* fix: add missing newline at end of robots.txt.ts file

---------

Co-authored-by: Joey Chen <JoeyC-Dev@users.noreply.github.com>
Co-authored-by: L4Ph <4ranci0ne@gmail.com>
This commit is contained in:
Joey Chen 2025-04-15 17:28:24 +08:00 committed by GitHub
parent 8cceb83e81
commit 174c5d8efc
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
src/pages/robots.txt.ts
View file

@ -2,7 +2,7 @@ import type { APIRoute } from "astro";
const robotsTxt = ` const robotsTxt = `
User-agent: * User-agent: *
Allow: / Disallow: /_astro/
Sitemap: ${new URL("sitemap-index.xml", import.meta.env.SITE).href} Sitemap: ${new URL("sitemap-index.xml", import.meta.env.SITE).href}
`.trim(); `.trim();