Empty room history on new session after message redaction and purge history call

[szilardx]

Description

I am writing a "message removal" script for a non federated server that will be used for compliance reasons.
Let's say we want 3 month old messages removed from the server and also from the clients. I found that the only sure and convenient way to remove messages form clients is to get a user token through Admin API and redact messages in the given room. The mass redact script works fine, but after running Purge History API call on that room, new sessions find that room empty, not containing the remaining message history.

Thank you!

Steps to reproduce

• take a config and db backup from the production server
• restore config and db to local development copy (Docker)
• login with an admin user - rooms, room history looks okay
• select a room for testing - one to one message room with the admin and an another user
• get a user auth token for the other user POST /_synapse/admin/v1/users/<user_id>/login
• list all messages in the room _synapse/admin/v1/rooms/${roomId}/messages?limit=10000&filter={%22types%22:[%22m.room.message%22]}&dir=b
• redact messages older than 3 month _matrix/client/v3/rooms/${roomId}/redact/${eventId}/${randomUUID()
• up to this point everything is okay. Element clients already logged in show the redactions. New logins will load room history with the redaction info
• run Purge History API call on the room, with the 'purge_up_to_ts' parameter being the timestamp 3 months ago _synapse/admin/v1/purge_history/${roomId}
• checking the given room with existing Element login show no difference from the previous state. (I guess this is expected, as Purge History API call is a server cleanup call, not a redaction call, so Element clients are NOT expected to remove messages)
• New Element login, or simply running "Clear cache and reload" is causing the purged room to behave weirdly:
  • room history is totally blank
  • when hitting search in the room, there is an up arrow with a green dot in the upper right corner. Pressing that will make the room show proper history. But navigation to an another room and coming back will make the room blank again.
  • New messages are showing, but they disappear when history is loaded with the previous method. So either history (as before the purge) or new messages (after the new login) are showing.
• Doing this whole process without the redactions part - so only running Purge History API call results in proper operation:
  • already logged in Element client shows the purged messages in the room history
  • new login / "Clear cache and reload" results in purged messages missing from room history - as expected

Homeserver

local

Synapse Version

1.99.0

Installation Method

Docker (matrixdotorg/synapse)

Database

PostgreSQL

Workers

Single process

Platform

Docker on MacOS as a testing environment.

Configuration

I had to change this to enable mass redaction:
rc_message:
per_second: 40
burst_count: 99999

First I was trying this, but was still rate limited:
rc_admin_redaction:
per_second: 40
burst_count: 99999

Relevant log output

Lots of log. Can send privately.

Anything else that would be useful to know?

I could not really figure out if this was an Element client or server issue. Maybe both. Client seems to get an empty response when getting messages for the given room on the new login.

Edit: I just tested that enabling purge in the server config after the mass redaction results in the same behavior.

[reivilibre]

Is the room encrypted? That is another reason why new clients can find the room empty (they need to have the keys shared with them or imported, before they can show the old messages).

I wonder if this is because there are lots of redactions in the timeline that the client needs to 'scroll past' (even if they are invisible) before it can get to the historical messages.

[szilardx]

The rooms are not encrypted.

[reivilibre]

Sorry for the delay.

My suspicion is that you have swamped the timeline with redaction events and the client has to metaphorically 'scroll up' past those invisible events. Possibly it doesn't do that after a certain limit.

(I assume you would have mentioned if Element still shows the loading spinner on screen, which would suggest it's still loading history?)

If you, in Element Web/Desktop, use /devtools and enable 'Show hidden events in timeline', you'll should now be able to see those hidden redaction events.

If you can do this, potentially scrolling up above the wave of redactions manually and confirm that's the issue, that sort of confirms it as a client-side issue. But as you can maybe appreciate, the design of using redactions for this makes it tricky to do anything else; a client doesn't really want to be forced to make a very large number of requests to load any visible history.

A workaround I'd consider is to try to find some way to space out your redactions rather than send them all at once, e.g. intersperse them with other messages in the room so that every 'page' of history contains at least one client-visible event (message).
If you have a room that used to be very active and now isn't, then I guess you'll struggle trying to do this, so it's not perfect by any means.

The long term solution here would be to use time-based retention policies and have the clients enforce this themselves, without need to send explicit redaction events to them. I don't know if or when that will be available though :).

[szilardx]

Hi,

I just tested again, with newer server - client verisons etc.
Checked the devtools too. It did not show the redaction events in the purged room. The problem persisted, only showed the remaining messages when using search feature in the room. When I clicked on an another room, then went be to the specific room, old messages did not show again.

Right now even redacting thousands of messages by a room admin from the start of the chat history (and not using purge room) is causing message display problems too.

Did a lot of experimenting, and the results are the following:

• redacting 70 messages by a user from the beginning of the room history is okay, new user sessions sees it
• running purge room up to the timestamp of the first 49 redacted message is okay too, new user session sees the message history properly
• adding an another purge (ie.: up to the timestamp of the next 10 messages) to the previous is causing the message history display problem described in the opening comment
• In some rooms I was able to redact, than purge messages, throwing in some new messages between them, and then message history display on new login worked. Element seemed to "page" through the batches of redactions.

So my conclusion is that this is an issue between the Element client and the server.

[reivilibre]

In some rooms I was able to redact, than purge messages, throwing in some new messages between them, and then message history display on new login worked. Element seemed to "page" through the batches of redactions.

I think this is probably an Element client problem.

Likely it has some logic like 'if no visible message received, stop paging'. I don't know for sure, this is just speculation, but it seems like they are the only ones that can help here.

I don't suspect Synapse to be at fault here because I can't think of any reason why we'd treat these 'invisible' redaction events any differently from other events — they'd still get sent down to clients.

It is a pity that we can't just skip over those for new sessions, since they never received the original message anyway, they don't really need to care about the redactions — but such a thing would be more of a spec problem and some clients can/do in fact show redactions.

I am going to close this issue because I don't think there's more we can do here as Synapse. If you want to open an issue on Element Web to discuss it, please do link to this issue though, so it's cross-referenced.

Thanks