[FEATURE] Revisit execution logs

[jugdemon]

• Feature name: Access data of completed executions in Dashboard

Description:

The logs of an execution are only available at the time of running. If the window is closed and revisited, there is only a table of executions but the executions themselves can no longer be inspected. It would be great if the log-UI available when pressing run is also available when selecting an execution from the table.

Importance Level

(Medium)

Users that leave the ODTP Dashboard and revisits and can no longer look at older execution. The information is in the file system (but our Dashboard runs on a server where the user has no CLI).

Origin

The demo of ODTP at ETHZ should be accessible by Dashboard only to all interested members of CSFM/ETHZ. Not all features of ODTP need to be available but we need to demostrate the benefit of ODTP.

User Impact

Users cannot leave the run page which is unfortunate.

Mockups or Diagrams

---

Affected Components (examples: components, modules, … )

All executions.

Technical Requirements (if possible, otherwise completed by SDSC)

Detailed technical specifications or requirements needed to implement the feature. This could include algorithms, data structures, APIs, or third-party services.

Related Documents/Links:

References to any related documentation, user stories, tickets, or external resources that provide additional context.

Dependencies (if possible, otherwise completed by SDSC):

Identification of any other features, systems, or processes that the proposed feature depends on or interacts with. This can be considered a “ready if” field and it will define what’s needed to have in order to start the development.

Acceptance criteria:

Specific criteria or metrics for evaluating the success or effectiveness of the feature once implemented.

[sabinem]

@jugdemon To keep the logs accessible, we have the following choices:

Option A: memorize the project path on the server

• when a execution runs, store the project path with the execution (anyway it is never cleaned up so far on the server)
• provide access to the logs in the project path from the execution detail page
• Optional: provide the option to cleanup the execution with the project path on the server directly from the execution detail page with a "cleanup" button

Option B: store the logfile in the s3 and get it from there.

Option C: get the logs from the mongodb, where they can be stored, but this optional as they were sometimes extensive and bloating the mongodb up to an unreasonable size

My suggestion would be Option A, as it seem straightforward to implement and the least disruptive to our concept of execution runs. @caviri what is your opinion on this?

@jugdemon: Would Option A be okay for you?

[jugdemon]

I think option A is sufficient. That would resolve this nicely.

[caviri]

Hello @sabinem and @jugdemon,

Originally logs were stored in mongodb in order to provide this feature and I still think it's the easier. But as Sabine's mentioned it can be overwhelming for MongoDB. Solutions as log pagination should solve this issue.

Regarding Option A, I think it's good too but I'm less friend on creating too many dependencies of local files considering that this won't be compatible with the API components (We can discuss about this late @sabinem). The option of deleting an execution content is available in the CLI from v0.4.0: https://github.com/odtp-org/odtp/blob/main/odtp/cli/execution.py This deletes the content + db entries related to an execution.

[jugdemon]

@caviri That makes sense and in the long-term is certainly better. I cannot assess how much time is left on the project and which option should be taken. I see option A as a quick intermediate solution to being able to demo it but if time permits, I think than C seems to be the logical choice. If I remember correctly the S3 bucket is for twin content and the mongodb for ODTP state, right?

[sabinem]

@jugdemon The S3 has an output.zip for each step that produced output. It could also hold a log file in the component directory. I will check options A and B, but I am strongly against C. I will probably go for A as a intermediate solution.