In this tutorial “Configuring Robots.txt – SEO WordPress“, I explain how to configure robots.txt for your WordPress website the right way.
Robots.txt is an interesting file. It lives in the root directory of your website and it’s sole purpose is to communicate with search engine crawlers. It tells the crawlers which areas of your website should and shouldn’t be indexed.
The activities in this tutorial require the Yoast SEO plugin to be installed, and its advanced features to be enabled.
Go to: WP Admin Menu → SEO → Tools → File Editor
Use the button “Create robots.txt file”, if you don’t already have one.
This is what our robots.txt file looks like, but you should edit yours as needed, then save changes.
User-agent: * Sitemap: https://opensite.com.au/sitemap_index.xml
You can test your robots.txt file with this tester tool from Google.
Note: It’s also possible to edit this file from the root directory of your website via FTP.
Here’s a good article for further reading.
This article is part of a series, explaining how to configure WordPress so that it’s search engine and social media ready, “SEO WordPress: The Definitive Guide“.