Worker crashes with connection error

[adzialocha]

Describe the bug

Logs in api-worker running on production:

/usr/src/app/build/services/web3.js:62
      throw new Error(error);
      ^

Error: Error: CONNECTION ERROR: The connection got closed with the close code `1006` and the following reason string `Connection dropped by remote peer.`
    at Subscription.callback (/usr/src/app/build/services/web3.js:62:13)
    at Object.callback (/usr/src/app/node_modules/web3-core-subscriptions/lib/subscription.js:248:27)
    at /usr/src/app/node_modules/web3-core-requestmanager/lib/index.js:108:34
    at Map.forEach (<anonymous>)
    at WebsocketProvider.close (/usr/src/app/node_modules/web3-core-requestmanager/lib/index.js:107:37)
    at WebsocketProvider.emit (/usr/src/app/node_modules/eventemitter3/index.js:181:35)
    at WebsocketProvider._onClose (/usr/src/app/node_modules/web3-providers-ws/lib/index.js:152:10)
    at W3CWebSocket._dispatchEvent [as dispatchEvent] (/usr/src/app/node_modules/yaeti/lib/EventTarget.js:115:12)
    at W3CWebSocket.onClose (/usr/src/app/node_modules/websocket/lib/W3CWebSocket.js:228:10)
    at WebSocketConnection.<anonymous> (/usr/src/app/node_modules/websocket/lib/W3CWebSocket.js:201:17)
    at WebSocketConnection.emit (events.js:314:20)
    at WebSocketConnection.EventEmitter.emit (domain.js:483:12)
    at WebSocketConnection.handleSocketClose (/usr/src/app/node_modules/websocket/lib/WebSocketConnection.js:389:14)
    at TLSSocket.emit (events.js:326:22)
    at TLSSocket.EventEmitter.emit (domain.js:483:12)
    at net.js:675:12

[adzialocha]

Still failing with:

2021-03-16T07:35:51.464Z [error]: Web3 subscription error: Error: connection not open on send()
2021-03-16T07:35:51.464Z [error]: Web3 subscription error: Error: connection not open on send()
2021-03-16T07:35:51.464Z [error]: Web3 subscription error: Error: connection not open on send()
2021-03-16T07:35:51.464Z [error]: Web3 subscription error: Error: connection not open on send()

<--- Last few GCs --->

[64:0x3196e10] 64253398 ms: Mark-sweep 4068.7 (4103.0) -> 4067.0 (4101.0) MB, 5315.6 / 0.0 ms  (average mu = 0.094, current mu = 0.022) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2ed301b008d1 <JSObject>
    0: builtin exit frame: stringify(this=0x2ed301b1ee79 <Object map = 0xc7e18d03639>,0x01fc9cc004b1 <undefined>,0x01fc9cc004b1 <undefined>,0x11af6e12f4c9 <JSArray[96]>,0x2ed301b1ee79 <Object map = 0xc7e18d03639>)

    1: arguments adaptor frame: 1->3
    2: _saveAttempt [0x362bddca3a39] [/usr/src/app/node_modules/bull/lib/job.js:577] [bytecode=0x3490d507a349 offset=120](this=0x11af6e10ac19 <Job...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xa18150 node::Abort() [node]
 2: 0xa1855c node::OnFatalError(char const*, char const*) [node]
 3: 0xb9705e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb973d9 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd54655  [node]
 6: 0xd54ce6 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xd615a5 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xd62455 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xd64f0c v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xd32dac v8::internal::Factory::NewRawOneByteString(int, v8::internal::AllocationType) [node]
11: 0x1102495 v8::internal::IncrementalStringBuilder::Extend() [node]
12: 0xe49148 v8::internal::JsonStringifier::SerializeString(v8::internal::Handle<v8::internal::String>) [node]
13: 0xe4de96 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
14: 0xe4eabf v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
15: 0xe4f764 v8::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [node]
16: 0xc487eb v8::internal::Builtin_JsonStringify(int, unsigned long*, v8::internal::Isolate*) [node]
17: 0x140a8f9  [node]
Aborted (core dumped)

[llunaCreixent]

We also see these errors in the worker when connecting to poa-sokol (from staging environment and from local dev env).

[error]: Web3 subscription error: Error: CONNECTION ERROR: The connection got closed with the close code `1006` and the following reason string `Connection dropped by remote peer.`
[error]: Web3 subscription error: Error: connection not open on send()
[error]: Web3 subscription error: Error: connection not open on send()
[...]

The error messages originate in https://github.com/CirclesUBI/circles-api/blob/main/src/services/web3.js#L23

[llunaCreixent]

Note: Sokol validators run the Nethermind client only. Source: https://www.poa.network/v/master-1/for-developers/developer-resourses#poa-sokol-testnet