Magento 2 has an inbuilt feature to edit Robots.txt from admin panel. It can be accessed from Content -> Design > Configuration. Click on ‘Edit’ button of the store you want to make changes to and go to ‘Search Engine Robots’ tab. You can make the required changes in this tab. Robots.txt is one of the most important and most overlooked parts of SEO. A lot of people make mistakes while implementing robots.txt on an eCommerce store. Magento 2 has made it easier to edit it right from the admin panel. There are various things which should be taken care of while implementing robots.txt to your eCommerce store.

In most cases, you don’t want your layered navigation pages to be indexed on search engines, but blocking them by the way of robots.txt is not an optimal solution. If you don’t want layered navigation pages to get indexed, use meta no index tag as I discussed earlier.

One more important to configure in robots.txt is to block site search results. You don’t want search engines to index search result with a lot of thin content. Make sure you disallow catalog search results.