Configure Robots.txt - SEO WordPress – Opensite - Digital Solutions Agency

0422 374 731 info@opensite.com.au

Configure Robots.txt – SEO WordPress

Last update:

Tutorials ,

The Goal

In this tutorial “Configuring Robots.txt – SEO WordPress“, I explain how to configure robots.txt for your WordPress website the right way.

Robots.txt is an interesting file. It lives in the root directory of your website and it’s sole purpose is to communicate with search engine crawlers. It tells the crawlers which areas of your website should and shouldn’t be indexed.

Requirements

The activities in this tutorial require the Yoast SEO plugin to be installed, and its advanced features to be enabled.

The Method

Go to: WP Admin Menu → SEO → Tools → File Editor

Use the button “Create robots.txt file”, if you don’t already have one.

This is what our robots.txt file looks like, but you should edit yours as needed, then save changes.

User-agent: *
Sitemap: https://opensite.com.au/sitemap_index.xml

You can test your robots.txt file with this tester tool from Google.

Note: It’s also possible to edit this file from the root directory of your website via FTP.

Here’s a good article for further reading.

Summary

This article is part of a series, explaining how to configure WordPress so that it’s search engine and social media ready, “SEO WordPress: The Definitive Guide“.