Basic Stress Test

[JulianGCalderon]

related to #682

Adds cairo-native-stress that generates, compiles, caches, and runs contracts in order to stress test cairo native.

• Receives by argument the amount rounds to execute.
• In each round it generates, compiles, caches, and runs a new contract
• The compiled dynamic shared libraries are stored on disk
• The cache keeps in memory every executor for every compiled contract
• After each round, it prints a report

Program Generation

At the start of the program, a dummy program is generated with a hardcoded value (835 in this case) in the return value:

#[starknet::contract]
mod Contract {
    #[storage]
    struct Storage {}

    #[external(v0)]
    fn main(self: @ContractState) -> felt252 {
        return 835;
    }
}

At the start of each round, the program is traversed to find the hardcoded value and is replaced with a new one. This allows to generate a new valid program with a different return value each round

Examples

With info logging (20 rounds):

With debug logging (5 rounds):

Next Steps

• Export the logs to a JSON or a CSV and plot it
• Try different implementations of the Cache

Checklist

• Linked to Github Issue
• This change requires new documentation.
  • Documentation has been added/updated.

[github-actions]

Benchmarking results

Benchmark for program factorial_2M

Open benchmarks

Command	Mean [s]	Min [s]	Max [s]	Relative
Cairo-vm (Rust, Cairo 1)	11.371 ± 0.075	11.290	11.514	23.68 ± 0.16
cairo-native (embedded AOT)	1.525 ± 0.011	1.512	1.542	3.18 ± 0.02
cairo-native (embedded JIT using LLVM's ORC Engine)	1.613 ± 0.013	1.596	1.634	3.36 ± 0.03
cairo-native (standalone AOT)	0.653 ± 0.001	0.652	0.655	1.36 ± 0.00
cairo-native (standalone AOT with -march=native)	0.480 ± 0.001	0.479	0.482	1.00

Benchmark for program fib_2M

Open benchmarks

Command	Mean [s]	Min [s]	Max [s]	Relative
Cairo-vm (Rust, Cairo 1)	10.893 ± 0.064	10.782	10.997	1340.35 ± 16.81
cairo-native (embedded AOT)	1.151 ± 0.017	1.121	1.173	141.69 ± 2.60
cairo-native (embedded JIT using LLVM's ORC Engine)	1.133 ± 0.013	1.116	1.151	139.44 ± 2.19
cairo-native (standalone AOT)	0.008 ± 0.000	0.008	0.009	1.04 ± 0.02
cairo-native (standalone AOT with -march=native)	0.008 ± 0.000	0.008	0.009	1.00

Benchmark for program logistic_map

Open benchmarks

Command	Mean [s]	Min [s]	Max [s]	Relative
Cairo-vm (Rust, Cairo 1)	1.916 ± 0.022	1.880	1.938	28.94 ± 0.36
cairo-native (embedded AOT)	1.258 ± 0.014	1.237	1.278	19.00 ± 0.23
cairo-native (embedded JIT using LLVM's ORC Engine)	1.442 ± 0.020	1.419	1.490	21.78 ± 0.32
cairo-native (standalone AOT)	0.107 ± 0.000	0.107	0.107	1.62 ± 0.01
cairo-native (standalone AOT with -march=native)	0.066 ± 0.000	0.066	0.068	1.00

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 204 lines in your changes missing coverage. Please review.

Project coverage is 90.97%. Comparing base (8a3502b) to head (a2e010c).

Files	Patch %	Lines
src/bin/cairo-native-stress.rs	0.00%	204 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #683      +/-   ##
==========================================
- Coverage   91.53%   90.97%   -0.56%     
==========================================
  Files         116      117       +1     
  Lines       33202    33406     +204     
==========================================
  Hits        30391    30391              
- Misses       2811     3015     +204     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.