synapse_port_db: IncorrectDatabaseSetup after port was interrupted

[margau]

Description

I've been getting an error message after synapse_port_db is interrupted (e.g. due to network issues) during the first run.

I'd expect that it is possible to resume by starting port_db again.

Steps to reproduce

Run synapse_port_db:
synapse_port_db --sqlite-database /var/lib/matrix-synapse/homeserver.db.snap --postgres-config homeserver-postgres.yaml

After something broke (e.g. connection issue), reconnect again.

Postgres is accessed over the network.

Homeserver

selfhosted

Synapse Version

matrix-synapse-py3/unknown,unknown,now 1.113.0+bullseye1 amd64

Installation Method

Debian packages from packages.matrix.org

Database

Both/Migration state

Workers

Single process

Platform

Debian 11

Configuration

database:
name: psycopg2
args:
(removed user/password/host)
# seconds of inactivity after which TCP should send a keepalive message to the server
keepalives_idle: 10

# the number of seconds after which a TCP keepalive message that is not
# acknowledged by the server should be retransmitted
keepalives_interval: 10

# the number of TCP keepalives that can be lost before the client's connection
# to the server is considered dead
keepalives_count: 3

Relevant log output

Preparing psycopg2...
2024-08-21 10:23:54,009 - synapse.storage.prepare_database - 135 - INFO - ('main', 'state'): Checking existing schema version
2024-08-21 10:23:54,024 - synapse.storage.prepare_database - 139 - INFO - ('main', 'state'): Existing schema is 86 (+1 deltas)
2024-08-21 10:23:54,024 - synapse.storage.databases.main - 391 - INFO - Checking database for consistency with configuration...
2024-08-21 10:23:54,027 - synapse.storage.prepare_database - 433 - INFO - Applying schema deltas for v86
2024-08-21 10:23:54,034 - synapse.storage.prepare_database - 568 - INFO - Schema now up to date
2024-08-21 10:23:54,040 - synapse.metrics - 121 - WARNING - synapse_background_update_status already registered, reregistering
2024-08-21 10:23:54,380 - synapse.storage.util.id_generators - 98 - INFO - Initialising stream generator for push_rules(id): 25
2024-08-21 10:23:54,383 - synapse.storage.util.id_generators - 98 - INFO - Initialising stream generator for push_rules_enable(id): 28
2024-08-21 10:23:54,445 - synapse_port_db - 926 - ERROR - 
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py", line 726, in run
    self.postgres_store = self.build_db_store(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py", line 642, in build_db_store
    store = Store(DatabasePool(hs, db_config, engine), db_conn, hs)  # type: ignore[arg-type]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1896, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 263, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/client_ips.py", line 84, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/deviceinbox.py", line 975, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/devices.py", line 1613, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/events_bg_updates.py", line 107, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/media_repository.py", line 114, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/registration.py", line 2104, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/registration.py", line 183, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/room.py", line 1772, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/room.py", line 152, in __init__
    self._un_partial_stated_rooms_stream_id_gen = MultiWriterIdGenerator(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/util/id_generators.py", line 295, in __init__
    self._sequence_gen.check_consistency(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/util/sequence.py", line 192, in check_consistency
    raise IncorrectDatabaseSetup(
synapse.storage.engines._base.IncorrectDatabaseSetup: 
Postgres sequence 'un_partial_stated_room_stream_sequence' is inconsistent with associated stream position
of 'un_partial_stated_room_stream' in the 'stream_positions' table.

This is likely a programming error and should be reported at
https://github.com/matrix-org/synapse.

A temporary workaround to fix this error is to shut down Synapse (including
any and all workers) and run the following SQL:

    DELETE FROM stream_positions WHERE stream_name = 'un_partial_stated_room_stream';

This will need to be done every time the server is restarted.

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py", line 726, in run
    self.postgres_store = self.build_db_store(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py", line 642, in build_db_store
    store = Store(DatabasePool(hs, db_config, engine), db_conn, hs)  # type: ignore[arg-type]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1896, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 263, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/client_ips.py", line 84, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/deviceinbox.py", line 975, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/devices.py", line 1613, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/events_bg_updates.py", line 107, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/media_repository.py", line 114, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/registration.py", line 2104, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/registration.py", line 183, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/room.py", line 1772, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/room.py", line 152, in __init__
    self._un_partial_stated_rooms_stream_id_gen = MultiWriterIdGenerator(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/util/id_generators.py", line 295, in __init__
    self._sequence_gen.check_consistency(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/util/sequence.py", line 192, in check_consistency
    raise IncorrectDatabaseSetup(
synapse.storage.engines._base.IncorrectDatabaseSetup: 
Postgres sequence 'un_partial_stated_room_stream_sequence' is inconsistent with associated stream position
of 'un_partial_stated_room_stream' in the 'stream_positions' table.

This is likely a programming error and should be reported at
https://github.com/matrix-org/synapse.

A temporary workaround to fix this error is to shut down Synapse (including
any and all workers) and run the following SQL:

    DELETE FROM stream_positions WHERE stream_name = 'un_partial_stated_room_stream';

This will need to be done every time the server is restarted.


Postgres sequence 'un_partial_stated_room_stream_sequence' is inconsistent with associated stream position
of 'un_partial_stated_room_stream' in the 'stream_positions' table.

This is likely a programming error and should be reported at
https://github.com/matrix-org/synapse.

A temporary workaround to fix this error is to shut down Synapse (including
any and all workers) and run the following SQL:

    DELETE FROM stream_positions WHERE stream_name = 'un_partial_stated_room_stream';

This will need to be done every time the server is restarted.

Anything else that would be useful to know?

No response

[nielsk]

In my case the migration went through, I still get this message in the logs