Make pipeline version per Organism

[fhennig]

resolves #1728

preview URL:

Summary

• TODO: adapt current_processing_pipeline Table
• TODO: adapt external_metadata_view (?)
• TODO: Docs: Document pipeline version concept better. Relevant link: https://github.com/loculus-project/loculus/blob/main/preprocessing/specification.md#reprocessing

Screenshot

PR Checklist

• All necessary documentation has been adapted.
• The implemented feature is covered by an appropriate test.

[fengelniederhammer]

Wow, the pipeline version in the backend is not per organism? But in the values.yml it's per organism, isn't it?

[fhennig]

What do you mean? The pipeline version is internal, so it doesn't appear in the values.yaml if I'm not mistaken.

[chaoran-chen]

The pipeline version also occurs in the Values.yaml because it's also used to deploy the pipelines.

[fengelniederhammer]

loculus/kubernetes/loculus/values.yaml

Lines 1148 to 1150 in 9aa28ab

	preprocessing:
	- &preprocessing
	version: 1

This is the pipeline version, isn't it?

[theosanderson]

Currently for the backend (/in the database) there is a single "latest pipeline version". So if you make a new pipeline version for one organism in the values yaml you have to do so for all organisms. This issue is (intended to be) about addressing that.

[fhennig]

Note to self: The table now starts out empty, somehow it needs to be filled. (I don't think there is a way around the table starting empty now? Because we don't know which organisms there are)

[fhennig]

Update: I found a way to initialize the table on backend-startup.

Now I marked some places in the schema that still need to be adjusted, that currently lead to: "ERROR: more than one row returned by a subquery used as an expression"

[fhennig]

From all the places I had to touch, I think the general handling of this table isn't the best unfortunately. It is different to all the other tables because it starts out with data, and all the others don't.