robots.txt

Same filename in this branch
  1. 11.x core/tests/Drupal/Tests/Composer/Plugin/Scaffold/fixtures/drupal-assets-fixture/assets/robots.txt
  2. 11.x robots.txt
Same filename in other branches
  1. 7.x robots.txt
  2. 9 core/tests/Drupal/Tests/Composer/Plugin/Scaffold/fixtures/drupal-assets-fixture/assets/robots.txt
  3. 9 core/assets/scaffold/files/robots.txt
  4. 9 robots.txt
  5. 8.9.x core/tests/Drupal/Tests/Composer/Plugin/Scaffold/fixtures/drupal-assets-fixture/assets/robots.txt
  6. 8.9.x core/assets/scaffold/files/robots.txt
  7. 8.9.x robots.txt
  8. 10 core/tests/Drupal/Tests/Composer/Plugin/Scaffold/fixtures/drupal-assets-fixture/assets/robots.txt
  9. 10 core/assets/scaffold/files/robots.txt
  10. 10 robots.txt
#
# robots.txt
#
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.
#
# This file will be ignored unless it is at the root of your host:
# Used:    http://example.com/robots.txt
# Ignored: http://example.com/site/robots.txt
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/robotstxt.html

User-agent: *
# CSS, JS, Images
Allow: /core/*.css$
Allow: /core/*.css?
Allow: /core/*.js$
Allow: /core/*.js?
Allow: /core/*.gif
Allow: /core/*.jpg
Allow: /core/*.jpeg
Allow: /core/*.png
Allow: /core/*.svg
Allow: /profiles/*.css$
Allow: /profiles/*.css?
Allow: /profiles/*.js$
Allow: /profiles/*.js?
Allow: /profiles/*.gif
Allow: /profiles/*.jpg
Allow: /profiles/*.jpeg
Allow: /profiles/*.png
Allow: /profiles/*.svg
# Directories
Disallow: /core/
Disallow: /profiles/
# Files
Disallow: /README.md
Disallow: /composer/Metapackage/README.txt
Disallow: /composer/Plugin/ProjectMessage/README.md
Disallow: /composer/Plugin/Scaffold/README.md
Disallow: /composer/Plugin/VendorHardening/README.txt
Disallow: /composer/Template/README.txt
Disallow: /modules/README.txt
Disallow: /sites/README.txt
Disallow: /themes/README.txt
# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register
Disallow: /user/password
Disallow: /user/login
Disallow: /user/logout
Disallow: /media/oembed
Disallow: /*/media/oembed
# Paths (no clean URLs)
Disallow: /index.php/admin/
Disallow: /index.php/comment/reply/
Disallow: /index.php/filter/tips
Disallow: /index.php/node/add/
Disallow: /index.php/search/
Disallow: /index.php/user/password
Disallow: /index.php/user/register
Disallow: /index.php/user/login
Disallow: /index.php/user/logout
Disallow: /index.php/media/oembed
Disallow: /index.php/*/media/oembed

File

core/assets/scaffold/files/robots.txt

View source 
  1. #

  2. # robots.txt

  3. #

  4. # This file is to prevent the crawling and indexing of certain parts

  5. # of your site by web crawlers and spiders run by sites like Yahoo!

  6. # and Google. By telling these "robots" where not to go on your site,

  7. # you save bandwidth and server resources.

  8. #

  9. # This file will be ignored unless it is at the root of your host:

  10. # Used:    http://example.com/robots.txt

  11. # Ignored: http://example.com/site/robots.txt

  12. #

  13. # For more information about the robots.txt standard, see:

  14. # http://www.robotstxt.org/robotstxt.html

  15. 

  16. User-agent: *

  17. # CSS, JS, Images

  18. Allow: /core/*.css$

  19. Allow: /core/*.css?

  20. Allow: /core/*.js$

  21. Allow: /core/*.js?

  22. Allow: /core/*.gif

  23. Allow: /core/*.jpg

  24. Allow: /core/*.jpeg

  25. Allow: /core/*.png

  26. Allow: /core/*.svg

  27. Allow: /profiles/*.css$

  28. Allow: /profiles/*.css?

  29. Allow: /profiles/*.js$

  30. Allow: /profiles/*.js?

  31. Allow: /profiles/*.gif

  32. Allow: /profiles/*.jpg

  33. Allow: /profiles/*.jpeg

  34. Allow: /profiles/*.png

  35. Allow: /profiles/*.svg

  36. # Directories

  37. Disallow: /core/

  38. Disallow: /profiles/

  39. # Files

  40. Disallow: /README.md

  41. Disallow: /composer/Metapackage/README.txt

  42. Disallow: /composer/Plugin/ProjectMessage/README.md

  43. Disallow: /composer/Plugin/Scaffold/README.md

  44. Disallow: /composer/Plugin/VendorHardening/README.txt

  45. Disallow: /composer/Template/README.txt

  46. Disallow: /modules/README.txt

  47. Disallow: /sites/README.txt

  48. Disallow: /themes/README.txt

  49. # Paths (clean URLs)

  50. Disallow: /admin/

  51. Disallow: /comment/reply/

  52. Disallow: /filter/tips

  53. Disallow: /node/add/

  54. Disallow: /search/

  55. Disallow: /user/register

  56. Disallow: /user/password

  57. Disallow: /user/login

  58. Disallow: /user/logout

  59. Disallow: /media/oembed

  60. Disallow: /*/media/oembed

  61. # Paths (no clean URLs)

  62. Disallow: /index.php/admin/

  63. Disallow: /index.php/comment/reply/

  64. Disallow: /index.php/filter/tips

  65. Disallow: /index.php/node/add/

  66. Disallow: /index.php/search/

  67. Disallow: /index.php/user/password

  68. Disallow: /index.php/user/register

  69. Disallow: /index.php/user/login

  70. Disallow: /index.php/user/logout

  71. Disallow: /index.php/media/oembed

  72. Disallow: /index.php/*/media/oembed

Buggy or inaccurate documentation? Please file an issue. Need support? Need help programming? Connect with the Drupal community.