If you don't want a site to be indexed by search engines you must place a robots.txt file. Typically you don't want anything indexed except production sites.

Today Spatie released a new package called laravel-robots-middleware. It was coded up by my colleague Sebastian. Instead of you having to create a robots.txt this Laravel package will add a x-robots-tag to every request. If you want to limit a search bot to only crawl your production enviroment you can extend the Spatie\RobotsMiddleware\RobotsMiddleware -class like this:

use Illuminate \ Http \ Request ; use Spatie \ RobotsMiddleware \ RobotsMiddleware ; class MyRobotsMiddleware extends RobotsMiddleware { protected function shouldIndex (Request $request) { return app()->environment( 'production' ); } }

To learn all the options read the documentation on GitHub. Like this package? Then maybe some of our other Laravel packages might be useful to you as well.