The robots.txt file is an incredibly simplistic part of a website that for the most part never causes any issues.
However, some platforms don’t allow you to modify the robots.txt file (such as Shopify), and in some cases modifying the Robots.txt file is not a simple job thanks to infrastructure or development backlogs.
Through Sloth, and Cloudflare Workers, this is something you can modify and override with ease.
In fact, all you need to do is go to the Robots.txt option on the left-sidebar of your Sloth dashboard…
… input the desired robots.txt file contents you want in the text field (which can be expanded by dragging the bottom right corner), and then hitting save.
Leaving this blank or toggling the sidebar icon from green to grey will disable the robots.txt override on the next deployment.