Fix robots.txt feature

[sneridagh]

It seems that the robot.txt feature is in a sorry state right now, so I'd like to be able to expose the robots.txt feature from the control panel as in classic.

In addition, to fix these issues that we have opened.

I would remove (at least in docs and code) the feature that the one in "public" folder is used (if everybody agrees), so we reduce the way of doing things in Volto instead of exposing yet another one. It seems it's not working since long time ago either way. One could still make it work if the middleware is disabled.

• robots.txt management #2166
• robots.txt issue in Volto 16 #5386
• naked file and image urls no longer download the file (causing SEO problems) #4934
• Update VOLTO_ROBOTSTXT

[robgietema]

The robot.txt is already a setting in Plone and part of the site configuration controlpanel but it is blacklisted in the settings in Volto. We can enable this again to make it configurable in the volto controlpanel.

The robots.txt is fetched from the Plone backend but not via the restapi. This code needs to be chnaged to fetch the robots.txt value from the @site endpoint.

@sneridagh: do we want to keep the environment variable option?

[robgietema]

#5584

[ericof]

Related issue plone.volto