Requests to /rooms/.../event and /rooms/.../relations stuck in "stalled" without ever completing

[Johennes]

Description

As part of investigating element-hq/element-web#25479 / https://github.com/matrix-org/element-web-rageshakes/issues/21713, I've noticed that requests to fetch events and relations in the affected room appear are failing for no apparent reason.

Here are examples of requests that failed:

https://element.ems.host/_matrix/client/r0/rooms/!aREzkHGCDRZFBtnnxx%3Aone.ems.host/event/%242IxstC5MW8GXED7OJFFoW9d9njGHL13JsSqGq42ejYA
https://element.ems.host/_matrix/client/v1/rooms/!aREzkHGCDRZFBtnnxx%3Aone.ems.host/relations/%24BIGtDHogyvgyOEOoJV0rCGYaBj1v1RMfs3y9E9kbH8M/m.replace/m.room.encrypted?limit=1

In the browser's dev tools it looks like the request didn't actually complete and is locked into a "Stalled" state:

The request is separately also displayed as "failed" though:

instrument.js:144     GET https://element.ems.host/_matrix/client/v1/rooms/!aREzkHGCDRZFBtnnxx%3Aone.ems.host/relations/%24Y0_WGya6JZSpb98hrg9iqP6inbHHIrdST8TuenbmyKU/m.replace/m.room.encrypted?limit=1 net::ERR_FAILED
(anony

Steps to reproduce

• Open https://matrix.to/#/!aREzkHGCDRZFBtnnxx:one.ems.host?via=matrix.org&via=element.io&via=one.ems.host
• Open one of the threads

I wasn't able to consistently reproduce it but it does happen fairly frequently on retrying.

Homeserver

element.io / element.ems.host

Synapse Version

1.84.1

Installation Method

I don't know

Database

I don't know

Workers

I don't know

Platform

I don't know

Configuration

I don't know

Relevant log output

No response

Anything else that would be useful to know?

No response

[erikjohnston]

Hi @Johennes, do you have some example timestamps (and the server name) so we can look up these requests in the server logs?

[erikjohnston]

Oh, its worth noting that the workers on matrix.org HS that service those requests have been overloaded recently, so that could be a cause of slow requests.

[Johennes]

This was on element.io but I'm not able to reproduce it again this morning. The timestamps would have been shortly before the time I opened this issue. Unless that's helpful, feel free to close this issue and I'll come back with timestamps if I see it again.

[erikjohnston]

(Wrong button, but oh well).

I can't see anything in the graphs to suggest that anything was going wrong at that point. If you do have a specific request + timestamp combo then we can go and have a look at that specific request.