Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/1870 historic state rotation #2063

Open
wants to merge 68 commits into
base: v4.0.0
Choose a base branch
from

Conversation

olehnikolaiev
Copy link
Contributor

@olehnikolaiev olehnikolaiev commented Jan 2, 2025

fixes #1870

introduce rotating historic state - skaled writes to a historic db until its size less than HISTORIC_STATE_DB_LIMIT_BYTES bytes. then skaled closes that db for write and opens another one.

  • Added BatchedRotatingHistoricDbIO class to manage the new historic state architecture. It rotates the databases, closes the dbs that are opend but not used and allow access to the specific db by block number.
  • Added RotatingHistoricState class to access historic state db. It rotates the database pieces, performs read, write, add and delete operations through the element of BatchedRotatingHistoricDbIO class.
  • Added rootBlockNumber parameter to the TrieDB for optimized scan of DBs in BatchedRotatingHistoricDbIO.
  • MemoryDB and libdevcore/OverlayDB are merged together and stored under libskaled/OverlayDB directory in ClassicOverlayDB class.
  • Skaled checks whether the historic state db needs to be rotated before executing block.
  • BatchedRotatingHistoricDbIO allows to have up to MAX_OPENED_DB_COUNT open dbs at the same time. Any db might be open for up to OPENED_DB_CHECK_INTERVAL ms. Once one of these limits is reached skaled closes all dbs that are open but not in use. This is done to avoid huge resource consummation.

tests - added unit tests for the new historic state mechanism. tested it on the mainnet - spun up a Nebula Archive node and verified that it catches up blocks faster than they are produced.

docs: https://github.com/skalenetwork/internal-support/blob/main/docs/specifications/skaled/rotating-historic-state.md

olehnikolaiev and others added 30 commits August 12, 2024 19:45
…lenetwork/skaled into feature/1870-historic-state-rotation
…lenetwork/skaled into feature/1870-historic-state-rotation
…alenetwork/skaled into feature/1870-historic-state-rotation
Copy link

github-actions bot commented Jan 2, 2025

CLA Assistant Lite bot:
Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request

@olehnikolaiev olehnikolaiev linked an issue Jan 13, 2025 that may be closed by this pull request
@DmytroNazarenko DmytroNazarenko added this to the SKALE 4.0 milestone Jan 16, 2025
@olehnikolaiev olehnikolaiev marked this pull request as ready for review January 20, 2025 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Investigate archive node performance
3 participants