-
Notifications
You must be signed in to change notification settings - Fork 56
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add test reporting doc to benchmarks dir
- Loading branch information
Showing
1 changed file
with
63 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,63 @@ | ||
--- | ||
title: Performance Benchmarks and Test Reports | ||
--- | ||
|
||
|
||
## Introduction | ||
This page summarizes key performance metrics for nwaku and provides links to detailed test reports. | ||
|
||
> ## Quick reference | ||
> | ||
> - **10%** decrease in bandwidth usage in larger networks (1000 vs 2000 relay nodes) | ||
> - **0.4s** average, **5.5s** max message propogation latency (2000 relay nodes) | ||
> - Relevancy to Status App: TODO | ||
|
||
|
||
## Insights | ||
*Some metrics for specific protocols are currently unavailable due to reporting and logging limitations. Improvements are in progress.* | ||
|
||
### Bandwidth Usage: nwaku v0.34 | ||
Average `libp2p` bandwidth usage for 1KB message size in a 1000 relay nodes network | ||
|
||
| Message Rate | libp2p-in (KB/s) | libp2p-out (KB/s) | | ||
|--------------|------------------|-------------------| | ||
| 1 msg/s | ~10.1 | ~10.3 | | ||
| 1 msg/10s | ~1.8 | ~1.9 | | ||
|
||
### Message Latency: nwaku v0.34.0-rc1 | ||
Latency results for 1KB messages (1msg/s, 600 total). Click on config to see detailed report: | ||
|
||
|
||
| Config | Avg Latency | Max Latency | | ||
|------------------------|-------------|-------------| | ||
| [Relay](https://www.notion.so/Waku-regression-testing-v0-34-1618f96fb65c803bb7bad6ecd6bafff9) (1000 nodes) | 0.05s | 1.6s | | ||
| [Mixed](https://www.notion.so/Mixed-environment-analysis-1688f96fb65c809eb235c59b97d6e15b) (210 nodes) | 0.0125s | 0.007s | | ||
| [Non-persistent Relay](https://www.notion.so/High-Churn-Relay-Store-Reliability-16c8f96fb65c8008bacaf5e86881160c) | 0.0125s | 0.25s | | ||
|
||
|
||
## Testing | ||
### DST | ||
The VAC DST team performs regression testing on all new **nwaku** releases, comparing performance with previous versions. They simulate large Waku networks with a variety of network and protocol configurations that are representative of real-world usage. | ||
|
||
**Test Reports**: [DST Reports](https://www.notion.so/DST-Reports-1228f96fb65c80729cd1d98a7496fe6f) | ||
|
||
|
||
### QA | ||
The VAC QA team performs interoperability tests for **Nim Waku** and **Go Waku** using the latest main branch images. These tests run daily and verify protocol functionality by targeting specific features of each protocol. | ||
|
||
**Test Reports**: [QA Reports](https://discord.com/channels/1110799176264056863/1196933819614363678) | ||
|
||
### nwaku | ||
The **nwaku** team follows a structured release procedure for all release candidates. This involves deploying RCs to `status.staging` fleet for validation and performing sanity checks. | ||
|
||
**Release Process**: [nwaku Release Procedure](https://github.com/waku-org/nwaku/blob/master/.github/ISSUE_TEMPLATE/prepare_release.md) | ||
|
||
|
||
### Research | ||
The Waku Research team conducts a variety of benchmarking, performance testing, proof-of-concept validations and debugging efforts. They also maintaining a Waku simulator designed for small-scale, single-purpose, on-demand testing. | ||
|
||
|
||
**Test Reports**: [Waku Research Reports](https://www.notion.so/Miscellaneous-2c02516248db4a28ba8cb2797a40d1bb) | ||
|
||
**Waku Simulator**: [Waku SImulator Book](https://waku-org.github.io/waku-simulator/) |