[ci] Scrape and verify metrics at the end of e2e tests

[chahatsagarmain]

Which problem is this PR solving?

• Part of Implement framework to validate backwards compatibility of metrics #6278

Description of the changes

• scrape script and usage in cit workflow
• diff calculating script
• cache save and restore from main workflow runs
• A sample diff (txt)

--- 
+++ 
@@ -303,2 +303,2 @@
-rpc_server_requests_per_rpc{le="+Inf",rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TestService",rpc_system="grpc",service_name="jaeger",service_version=""}
-rpc_server_requests_per_rpc{le="0",rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_name="jaeger",service_version=""}
+rpc_server_requests_per_rpc{le="+Inf",rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_name="jaeger",service_version=""}
+rpc_server_requests_per_rpc{le="0",rpc_grpc_status_code="1",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_name="jaeger",service_version=""}

@@ -321 +321 @@
-rpc_server_response_size{le="+Inf",rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_name="jaeger",service_version=""}
+rpc_server_response_size{le="+Inf",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_name="jaeger",service_version="",test_change="Export"}

@@ -338 +338 @@
-rpc_server_response_size{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_name="jaeger",service_version=""}
+rpc_server_response_size{rpc_method="Export",rpc_service="opentelemetry.proto.collector.trace.v1.TraceService",rpc_system="grpc",service_name="test-jaeger",service_version=""}

How was this change tested?

Checklist

• I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
• I have signed all commits
• I have added unit tests for the new functionality
• I have run lint and test steps successfully
  • for jaeger: make lint test
  • for jaeger-ui: yarn lint and yarn test

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.21%. Comparing base (2ee8e4c) to head (9d232bb).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6330      +/-   ##
==========================================
- Coverage   96.26%   96.21%   -0.06%     
==========================================
  Files         372      372              
  Lines       21371    21371              
==========================================
- Hits        20573    20562      -11     
- Misses        608      616       +8     
- Partials      190      193       +3     

Flag	Coverage Δ
badger_v1	10.65% <ø> (ø)
badger_v2	2.78% <ø> (ø)
cassandra-4.x-v1-manual	16.55% <ø> (ø)
cassandra-4.x-v2-auto	2.71% <ø> (ø)
cassandra-4.x-v2-manual	2.71% <ø> (ø)
cassandra-5.x-v1-manual	16.55% <ø> (ø)
cassandra-5.x-v2-auto	2.71% <ø> (ø)
cassandra-5.x-v2-manual	2.71% <ø> (ø)
elasticsearch-6.x-v1	20.33% <ø> (-0.01%)	⬇️
elasticsearch-7.x-v1	20.40% <ø> (ø)
elasticsearch-8.x-v1	20.57% <ø> (+<0.01%)	⬆️
elasticsearch-8.x-v2	2.77% <ø> (ø)
grpc_v1	12.17% <ø> (+<0.01%)	⬆️
grpc_v2	9.04% <ø> (ø)
kafka-3.x-v1	10.33% <ø> (ø)
kafka-3.x-v2	2.78% <ø> (ø)
memory_v2	2.78% <ø> (ø)
opensearch-1.x-v1	20.46% <ø> (ø)
opensearch-2.x-v1	20.45% <ø> (-0.01%)	⬇️
opensearch-2.x-v2	2.77% <ø> (-0.01%)	⬇️
tailsampling-processor	0.51% <ø> (ø)
unittests	95.09% <ø> (-0.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[yurishkuro]

We changed the meaning of this cache items, it's no longer "release" snapshot. We can call it baseline_*

[yurishkuro]

any idea why the log mentions the name twice?

Cache not found for input keys: badger-metrics-v2.txt, badger-metrics-v2.txt

[chahatsagarmain]

@yurishkuro I guess its the exact key that needs to be matched followed by restore key .

[yurishkuro]

lgtm, final cleanup

[yurishkuro]

Suggested change

	cache_key:
	description: 'Cache key used for uploading and fetching the correct cached metric'
	required: true