Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fast sync algorithm. #2763

Closed
wants to merge 50 commits into from
Closed
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
f2c9704
Add fast-sync engine.
shamil-gadelshin May 9, 2024
eca1522
Add raw block importer.
shamil-gadelshin May 9, 2024
00c3d65
node: Add fast-sync algorithm.
shamil-gadelshin Mar 29, 2024
8a5e758
node: Modify CLI configuration to support fast-sync.
shamil-gadelshin Mar 29, 2024
011f4a5
Integrate fast-sync algorithm into dsn-sync
shamil-gadelshin May 13, 2024
042ae07
Apply review suggestions.
shamil-gadelshin May 16, 2024
f8c4600
Change peer source for fast-sync.
shamil-gadelshin May 17, 2024
ae590ca
Update fast sync method.
shamil-gadelshin May 17, 2024
1fb2c0e
Only sync state from peers that have a higher best block number than …
nazar-pc May 21, 2024
14408db
Import lists of blocks together instead of individually to leverage v…
nazar-pc May 21, 2024
90b0ec9
Remove incorrect of last block from second last segment and add TODOs
nazar-pc May 21, 2024
9685fe9
Merge pull request #2779 from subspace/add-fast-sync-algorithm-fixes
shamil-gadelshin May 22, 2024
9d384f5
Refactor fast-sync.
shamil-gadelshin May 21, 2024
34b5a14
Move fast-sync out of DSN-sync scope.
shamil-gadelshin May 22, 2024
7e71097
Modify sync CLI arguments.
shamil-gadelshin May 22, 2024
c6f0240
Simplify fast-sync-engine.
shamil-gadelshin May 14, 2024
08510d2
Merge branch 'main' into add-fast-sync-algorithm
shamil-gadelshin May 23, 2024
f9d544f
Inline both `import_raw_block` and `create_raw_block` into the place …
nazar-pc May 23, 2024
70de7e5
Merge pull request #2785 from subspace/inline-import-raw-block
shamil-gadelshin May 24, 2024
5938323
Remove peer-management elements from fast-sync.
shamil-gadelshin May 24, 2024
72820c2
Add some comments.
shamil-gadelshin May 24, 2024
9a8778d
Revert unnecessary changes to DSN sync code
nazar-pc May 27, 2024
81b011d
Remove `FastSyncResult` that is no longer necessary
nazar-pc May 27, 2024
17601c2
Remove unnecessary retrieval of last segment header
nazar-pc May 27, 2024
f335461
Rename `download_segment_headers` to `sync_segment_headers` and simpl…
nazar-pc May 27, 2024
83228fc
Make fast sync not assume block size
nazar-pc May 27, 2024
73c0d8d
Merge pull request #2795 from subspace/revert-dsn-sync-changes
shamil-gadelshin May 28, 2024
ac4db9e
Toggle skip_proof variable for state downloading.
shamil-gadelshin May 28, 2024
96c20cf
Merge pull request #2796 from subspace/fast-sync-improvements
nazar-pc May 28, 2024
7e416f1
Make fast sync import block atomically, simplify code
nazar-pc May 27, 2024
3771a0a
Update Subspace block import to be better suited for custom sync mech…
nazar-pc May 27, 2024
1ad08d9
Small corrections and cleanups
nazar-pc May 27, 2024
2005b4b
Merge pull request #2797 from subspace/atomic-fast-sync-block-import
nazar-pc May 28, 2024
fa89440
Remove unnecessary `Mutex` and unused generics
nazar-pc May 27, 2024
350dfbc
Simplify fast sync engine API, add support for `fork_id`
nazar-pc May 27, 2024
f247877
Simplify networking in fast sync engine, simplify generics further
nazar-pc May 27, 2024
3b5ab82
Merge pull request #2798 from subspace/fast-sync-cleanup
nazar-pc May 28, 2024
95d17ce
Correct check for fast sync start and fix sync mode resetting after (…
nazar-pc May 28, 2024
5b39dd1
Simplify block import without resorting to `apply_block` low-level API
nazar-pc May 28, 2024
0edf1f7
Do not finalize blocks that do not exist
nazar-pc May 29, 2024
ae8a54c
Fix condition for clearing block gap
nazar-pc May 29, 2024
ccea056
Print info after fast sync
nazar-pc May 29, 2024
c14a7b2
Merge pull request #2799 from subspace/simplify-fast-sync
nazar-pc May 29, 2024
7b39a9d
Merge branch 'main' into add-fast-sync-algorithm
shamil-gadelshin May 30, 2024
0dafb27
Merge branch 'main' into add-fast-sync-algorithm
shamil-gadelshin May 30, 2024
c175ca8
Update fast-sync after latest polkadot-sdk
shamil-gadelshin May 30, 2024
1940fe4
Rename fast-sync -> snap-sync
shamil-gadelshin May 30, 2024
677c662
Refactor fast-sync.
shamil-gadelshin May 30, 2024
2e0a3d7
Remove redundant `ConfigSyncMode` type
shamil-gadelshin May 30, 2024
b216a18
Remove substrate sync option.
shamil-gadelshin May 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions crates/subspace-service/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ pallet-transaction-payment-rpc = { git = "https://github.com/subspace/polkadot-s
parity-scale-codec = "3.6.9"
parking_lot = "0.12.2"
prometheus-client = "0.22.2"
prost = "0.12"
sc-basic-authorship = { git = "https://github.com/subspace/polkadot-sdk", rev = "808269708cf5375526755797e8f9a9986016727d" }
sc-chain-spec = { git = "https://github.com/subspace/polkadot-sdk", rev = "808269708cf5375526755797e8f9a9986016727d" }
sc-client-api = { git = "https://github.com/subspace/polkadot-sdk", rev = "808269708cf5375526755797e8f9a9986016727d" }
Expand All @@ -52,6 +53,7 @@ sc-telemetry = { git = "https://github.com/subspace/polkadot-sdk", rev = "808269
sc-tracing = { git = "https://github.com/subspace/polkadot-sdk", rev = "808269708cf5375526755797e8f9a9986016727d" }
sc-transaction-pool = { git = "https://github.com/subspace/polkadot-sdk", rev = "808269708cf5375526755797e8f9a9986016727d" }
sc-transaction-pool-api = { git = "https://github.com/subspace/polkadot-sdk", rev = "808269708cf5375526755797e8f9a9986016727d" }
sc-utils = { git = "https://github.com/subspace/polkadot-sdk", rev = "808269708cf5375526755797e8f9a9986016727d" }
schnorrkel = "0.11.4"
sp-api = { git = "https://github.com/subspace/polkadot-sdk", rev = "808269708cf5375526755797e8f9a9986016727d" }
sp-blockchain = { git = "https://github.com/subspace/polkadot-sdk", rev = "808269708cf5375526755797e8f9a9986016727d" }
Expand Down
2 changes: 2 additions & 0 deletions crates/subspace-service/src/sync_from_dsn.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
mod fast_sync_engine;
mod import_blocks;
pub(super) mod piece_validator;
mod raw_block_import;
mod segment_header_downloader;

use crate::sync_from_dsn::import_blocks::import_blocks_from_dsn;
Expand Down
Loading