github,tmt: run the bib integration tests

[mvo5]

Here is a draft for my idas how we could run integration tests
for the bootc containers.

---

This commit runs parametrized bootc-image-builder integration
tests as part of the testing. The parametrization requires [0].
The test matrix right now is to generated a "qcow2" and "ami" image
and for both:

1. test_image_is_generated(): just checks that the image can be build
2. test_image_boots(): the image boots in qemu and can be accessed
   via ssh [1]
3. test_image_build_without_se_linux_denials(): ensure that the
   image is build without selinux denials

We can run arbitrary post-boot tests in step (2) if we want, at
this point we have a ssh connection to the booted image so we
can do any post-boot checks we consider important (code for this
needs to be written though but should be easy).

Caveats:

• bootc-image-builder cannot build quay.io/centos-bootc/centos-bootc:stream9
  right now [2]. This is being worked on and we could start with
  testing the quay.io/centos-bootc/fedora-bootc:eln images until bib
  supports centos.

Some ideas for further improvements:

• tests could be extracted from bootc-image-builder into a shared
  bootc-testing repo
• post-boot tests
• upgrade tests, e.g. by building an image from the current "stable"
  bootc first and then upgrading to the bootc container that is
  tested

I hope this is helpful!

[0] osbuild/bootc-image-builder#87
[1] https://github.com/osbuild/bootc-image-builder/blob/main/test/test_smoke.py#L128
[2] osbuild/bootc-image-builder#20

[cgwalters]

Thanks for starting this! There's a lot of unfamiliar-to-me YAML going on here...looks sane overall, just some questions

[cgwalters]

This is a virtual machine under which we run podman, it's not really system-under-test no? (Other than I guess it contains the podman version)

[lmilbaum]

@mvo5 My PR #120 was merged

[rhatdan]

@mvo5 Still working on this?

[mvo5]

@mvo5 Still working on this?

Yes! I'm stuck right now because I don't see where the testing farm tests are actually run (and if they run where the output goes), I will poke at this more, when I inspect Red Hat Trusted App Pipeline / centos-bootc-on-pull-request Failing after 263m — Failed I see failures on s390x and ppc64el (which look unrelated). Click on the amd64 build I get to console but seem to be not allow-listed yet for the service (I requested access now).

[lmilbaum]

/retest

[shi2wei3]

I'll help to check the rhtap failure once I got the access, currently it's block by a bug in order to add me to the bootc workspace.
I can see the two failed jobs are all building tasks, it's not related to this PR, and the integration test task has the name "fedora-bootc-enterprise-contract", I guess it's from the demo template, so we need to setup the integration test on RHTAP to run the code in this PR.
We have our bootc test in https://gitlab.com/bootc-org/tests/bootc-workflow-test, and also have the rhtap integration test pipeline definition there.

[shi2wei3]

/retest

[shi2wei3]

/retest

[shi2wei3]

/retest

[shi2wei3]

I've got the access to rhtap instance and added bib integration test to fedora-bootc and centos-bootc projects.

I checked the build error message is:
Updating metadata for 'baseos-devel'...done
error: Updating rpm-md repo 'baseos-devel': cannot update repo 'baseos-devel': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried; Last error: Status code: 404 for https://composes.stream.centos.org/development/CentOS-Stream-9-20240117.d.0/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 18.67.76.31)
error: compose tree failed: ExitStatus(unix_wait_status(256))

I think a rebase is needed to run the build tasks successfully.

And I just realized that currently this PR (the changes related to tmt metadata) is not be tested by Konflux pipeline job on pull-request, that's because the tmt metadata is fetched from a fixed url and branch, in order to cover the changes introduced by PR, we need to change the testing farm git-ref param to a revision number, but I have not try that and I'm not sure if this also works for push trigger or manual trigger from rhtap UI.

[cgwalters]

Thanks so much for this! Looks like we're passing CI here now. But...as far as I understand it we're not actually testing the container image built as part of a PR, right?

[shi2wei3]

Thanks so much for this! Looks like we're passing CI here now. But...as far as I understand it we're not actually testing the container image built as part of a PR, right?

The CI integration test DOES use the bootc image build as part of the PR, RHTAP have a SNAPSHOT variable which contains the container image url built for PR and will pass it over to integration test.

But if the PR (like this one) contains the code change related to the tests (test case / tmt metadata / pipeline definition), it will not be run in integration test, because the integration test is configured to fetch from a fixed git branch, afterwards the testing farm fetch the tmt metadata from the git repo configured by a fixed param.

bib use rhtap yaml and tmt metadata in this bootc repo, and test cases in bib repo.
Our bootc test store all these in bootc-workflow-test.

I've drafted a PR #341 for this, it will work for future PRs after it's merged.

[shi2wei3]

@mvo5 integration test failed due to missing python3-boto3, there are also other diffs compare to the bib repo you may want to add.

[mvo5]

I just had a (very nice) chat with @shi2wei3 and we talked about the existing testsuite and how our bib tests can help. During that he pointed me to https://github.com/virt-s1/bootc-workflow-test/blob/main/bib-image.sh#L155 which is essentially doing what our bib tests are also doing so adding bib at this level seems to not add any value at this point and I will close this PR as it's redundant with the already existing testing/gating.

We (bib team) will keep in touch because when we add e.g. azure uploads we will need to extend this side of the testsuite as well (but that should be straightforward).