When your websites are ready for publishing, don’t forget to remove the header
add_header X-Robots-Tag "noindex, nofollow, nosnippet, noarchive";from the server block of all the websites that you want to be indexed by search engines.
Change also the content of the robots.txt file for all the websites that you want to allow public access to. First open the file:
nano /var/www/example.com/robots.txtChange its content to make it look like this:
User-agent: *
Disallow:
Sitemap: https://www.example.com/wp-sitemap.xmlThe Sitemap parameter is optional. Yet, mentioning the URL of the XML sitemap that is automatically generated by WordPress, or the URL of a different sitemap generated by a plugin (like Yoast SEO), can improve SEO.
It’s recommended to stop the well-known AI bots from crawling your website, so, you can add the following lines before the ones mentioned above, to make the content of robots.txt look like this:
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: *
Disallow:
Sitemap: https://www.example.com/wp-sitemap.xmlIf you configured Nginx to serve an ‘under construction’ page to outside visitors, you have to reverse those changes, to make the website accessible to everyone. Therefore, the lines that restricted access should be commented out like this:
# location = /underconstruction.html {}
# error_page 403 =200 /underconstruction.html;
location / {
# allow 123.123.123.123;
# allow 124.124.124.124;
# allow 2a05:6ef0:407::c3b4:d1e5;
# deny all;
try_files $uri $uri/ /index.php?$args;
} 