Make downloads more efficient

[jspijker]

In current setup, multiple instances of the application might download the same data. Create some sort of communication between processes. Prevent blocking of data requests due to crashed downloads

here: {https://github.com/rivm-syso/Analyse-Together/blob/79f9ff9f0a6cb07a41b1eb505622bfb324b344b9/scripts/queue_manager.R#L83C3-L84C75)

[jspijker]

This is hard to prevent since equal station/range combo's are allowed in the queue. Simple solution is to randomize the station list to lower the change that the same station/range is downloaded twice.

[jspijker]

The following error may be related to this:

Error in ATdatabase::add_doc(type = "data_req_done", ref = job_id, doc = j_done,  : 
  Error add_doc: doc exists and overwrite is FALSE

[jspijker]

This is only relevant for LML/KNMI stations, since they can occur multiple times in a data request. Individual sensors should by unique. This can be optimised to get all LML/KNMI stations from within a data req first, and then the sensors.

LML and KNMI station are also downloaded individually, but doesn't work because the dl_station function is aimed at sensors, hence resulting in empty streams.

[jspijker]

again error: Happens when on heavy load.

Listening on http://0.0.0.0:3838
Error in ATdatabase::add_doc(type = "data_req_done", ref = job_id, doc = j_done, :
Error add_doc: doc exists and overwrite is FALSE
Execution halted

[jspijker]

Implement the solution of #392 into the dev branch, add it to the queue_manager and make it independent of the app