Add X-Robots-Tag header to response for items marked as no_index

[sneridagh]

No description provided.

[davisagli]

@sneridagh I updated this to add the header for any item that has the behavior enabled and seo_noindex checked (not just images).

But I tested it in our project and I think it needs more work:

1. The header is added to backend responses, but Volto's proxy and SSR does not pass it through.
2. The header is added for the default view of the item, but not for related paths like ++api++ traversal or downloading image scales. I think we want to add it if an item with seo_noindex is the closest content item in request["PARENTS"], even if it's not the first item.
3. This gets a bit more complicated, so I think some functional tests would be good.

[sneridagh]

@davisagli

1. fixed and released: https://github.com/plone/volto/releases/tag/16.31.9
2. Why do we need the header in ++api++? the bots won't never search in there, nor parse the JSON in search for images.

[davisagli]

@sneridagh Are you sure? In the logs from our project, I see traffic from Bingbot on API URLs.

[davisagli]

@sneridagh Anyway, we definitely need it for image scales.