Missing files in on-demand mode

[vogll]

Version
"versions": [
{
"component": "core",
"version": "3.69.0",
"package": "pulpcore",
"module": "pulpcore.app",
"domain_compatible": true
},
{
"component": "deb",
"version": "3.4.0",
"package": "pulp_deb",
"module": "pulp_deb.app",
"domain_compatible": false
},
{
"component": "ostree",
"version": "2.4.4",
"package": "pulp-ostree",
"module": "pulp_ostree.app",
"domain_compatible": true
},
{
"component": "certguard",
"version": "3.69.0",
"package": "pulpcore",
"module": "pulp_certguard.app",
"domain_compatible": true
},
{
"component": "file",
"version": "3.69.0",
"package": "pulpcore",
"module": "pulp_file.app",
"domain_compatible": true
},
{
"component": "container",
"version": "2.22.0",
"package": "pulp-container",
"module": "pulp_container.app",
"domain_compatible": false
},
{
"component": "ansible",
"version": "0.23.1",
"package": "pulp-ansible",
"module": "pulp_ansible.app",
"domain_compatible": false
},
{
"component": "python",
"version": "3.12.5",
"package": "pulp-python",
"module": "pulp_python.app",
"domain_compatible": true
},
{
"component": "maven",
"version": "0.8.1",
"package": "pulp-maven",
"module": "pulp_maven.app",
"domain_compatible": false
},
{
"component": "rpm",
"version": "3.27.2",
"package": "pulp-rpm",
"module": "pulp_rpm.app",
"domain_compatible": true
}
],

Describe the bug
We have been using Pulp in on-demand mode for the Ubuntu archives (noble, noble-updates, noble-security) and only for the amd64 architecture. After several months of use, we observed that some artifacts previously used, reports 404 even though the files are present in /var/lib/pulp.

We haven't performed any resynchronizations other than the initial sync. For all repositories (noble, noble-updates, noble-security), we have created their own remote.

Pulp has been deployed in our Kubernetes cluster using the Pulp Operator. After the first occurence of the missing files we performed a redeploy.

Then I tested the archive with only noble updates in on-demand mode and created one remote version and one repository version. Then I updated the test machine with Pulp repository to get the files in Pulp repository (on-demand mode). The behavior was the same, out of 187 artifacts 38 were missing. Everything as in the first deployment.

I did a dB and storage check

https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/c/cryptsetup/cryptsetup_2.7.0-1ubuntu4.1_amd64.deb

log

pulp [None]: pulpcore.content:DEBUG: Content App '22@pulp-content-6b4d88fb97-vk679' heartbeat written, sleeping for '7' seconds
::ffff:10.42.1.40 [09/Jan/2025:10:30:07 +0000] "GET /pulp/content/ubuntu-noble-updates-amd64/pool/main/c/cryptsetup/cryptsetup_2.7.0-1ubuntu4.1_amd64.deb HTTP/1.1" 404 174 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:133.0) Gecko/20100101 Firefox/133.0"
pulp [None]: pulpcore.content:DEBUG: Content App '23@pulp-content-6b4d88fb97-vk679' heartbeat written, sleeping for '7' seconds

We have only one version of the repository.
Now i found package

(venv) vogll@deployer:/opt/pulp$ pulp deb content list --repository-version /pulp/api/v3/repositories/deb/apt/0193daab-70bb-7a1e-aec6-394e48640fa2/versions/1/ --package cryptsetup
[
  {
    "pulp_href": "/pulp/api/v3/content/deb/packages/0193daab-9832-7872-b726-ada5f2c159f2/",
    "prn": "prn:deb.package:0193daab-9832-7872-b726-ada5f2c159f2",
    "pulp_created": "2024-12-18T16:48:30.580629Z",
    "pulp_last_updated": "2024-12-18T16:48:30.580665Z",
    "artifact": "/pulp/api/v3/artifacts/01944140-e46d-717d-b2d3-5cdae9fa476b/",
    "relative_path": "pool/main/c/cryptsetup/cryptsetup_2.7.0-1ubuntu4.1_amd64.deb",
    "md5": null,
    "sha1": null,
    "sha224": "5bc5ca57f1a0676dccc8a2d12b1f2c2193d182f8b96326f227557c2f",
    "sha256": "d6eac1d471955e33252a366c2faa34ab3fc492cbf194383e123ef1faca7fe9f8",
    "sha384": "1ececa6753663b9c14e053bdb2537020f886461bbeaff304ce1fb54b5b8dd8f8e468d92cebf13b05a54ad868a8fa21c6",
    "sha512": "c266ad3c33e08e653ca55361b72bf365c820be677896d97cef8817c99e5b73d06ea56a8f22c2c0c70fa2b463780a15c23a65e0ba5141982e2515af73165fcbbf",
    "package": "cryptsetup",
    "source": null,
    "version": "2:2.7.0-1ubuntu4.1",

Artifact sha224 to identify file

(venv) vogll@deployer:/opt/pulp$ pulp artifact list --sha224 5bc5ca57f1a0676dccc8a2d12b1f2c2193d182f8b96326f227557c2f
[
  {
    "pulp_href": "/pulp/api/v3/artifacts/01944140-e46d-717d-b2d3-5cdae9fa476b/",
    "prn": "prn:core.artifact:01944140-e46d-717d-b2d3-5cdae9fa476b",
    "pulp_created": "2025-01-07T14:52:39.408868Z",
    "pulp_last_updated": "2025-01-07T14:52:39.408910Z",
    "file": "artifact/d6/eac1d471955e33252a366c2faa34ab3fc492cbf194383e123ef1faca7fe9f8",
    "size": 207834,
    "md5": null,
    "sha1": null,
    "sha224": "5bc5ca57f1a0676dccc8a2d12b1f2c2193d182f8b96326f227557c2f",

Open pulp content (kubectl exec)

bash-5.1$ ls -la /var/lib/pulp/media/artifact/d6/eac1d471955e33252a366c2faa34ab3fc492cbf194383e123ef1faca7fe9f8
-rw-r--r-- 1 pulp pulp 207834 Jan  7 14:52 d6/eac1d471955e33252a366c2faa34ab3fc492cbf194383e123ef1faca7fe9f8

(venv) vogll@deployer:/opt/pulp$ pulp artifact show --href /pulp/api/v3/artifacts/01944140-e46d-717d-b2d3-5cdae9fa476b/
{
  "pulp_href": "/pulp/api/v3/artifacts/01944140-e46d-717d-b2d3-5cdae9fa476b/",
  "prn": "prn:core.artifact:01944140-e46d-717d-b2d3-5cdae9fa476b",
  "pulp_created": "2025-01-07T14:52:39.408868Z",
  "pulp_last_updated": "2025-01-07T14:52:39.408910Z",
  "file": "artifact/d6/eac1d471955e33252a366c2faa34ab3fc492cbf194383e123ef1faca7fe9f8",
  "size": 207834,
  "md5": null,
  "sha1": null,
  "sha224": "5bc5ca57f1a0676dccc8a2d12b1f2c2193d182f8b96326f227557c2f",
  "sha256": "d6eac1d471955e33252a366c2faa34ab3fc492cbf194383e123ef1faca7fe9f8",
  "sha384": "1ececa6753663b9c14e053bdb2537020f886461bbeaff304ce1fb54b5b8dd8f8e468d92cebf13b05a54ad868a8fa21c6",
  "sha512": "c266ad3c33e08e653ca55361b72bf365c820be677896d97cef8817c99e5b73d06ea56a8f22c2c0c70fa2b463780a15c23a65e0ba5141982e2515af73165fcbbf"
}

check sha

sha256sum /var/lib/pulp/media/artifact/d6/eac1d471955e33252a366c2faa34ab3fc492cbf194383e123ef1faca7fe9f8
d6eac1d471955e33252a366c2faa34ab3fc492cbf194383e123ef1faca7fe9f8

To Reproduce

#!/bin/bash -x
REPOSITORY_NAME='ubuntu-noble-updates-amd64'
REMOTE_NAME="${REPOSITORY_NAME}"
DISTRIBUTION_NAME="${REPOSITORY_NAME}"
SIGNING_SERVICE="deb-signing-service"
REMOTE_OPTIONS=(
  --url=http://archive.ubuntu.com/ubuntu/
  --distribution=noble-updates
  --component=main
  --component=restricted
  --component=universe
  --component=multiverse
  --architecture=amd64
  --policy on_demand
)

pulp deb remote create --name=${REMOTE_NAME} ${REMOTE_OPTIONS[@]}
pulp deb repository create --name=${REPOSITORY_NAME} --remote=${REMOTE_NAME}
pulp deb repository sync --name=${REPOSITORY_NAME}
pulp deb publication create --repository=${REPOSITORY_NAME}
pulp deb distribution create --name=${DISTRIBUTION_NAME} --base-path=${REPOSITORY_NAME} --repository=${REPOSITORY_NAME}

pulp deb remote create --name=${REMOTE_NAME} ${REMOTE_OPTIONS[@]}
pulp deb repository create --name=${REPOSITORY_NAME} --remote=${REMOTE_NAME}
pulp deb repository sync --name=${REPOSITORY_NAME}
pulp deb publication create --repository=${REPOSITORY_NAME}
pulp deb distribution create --name=${DISTRIBUTION_NAME} --base-path=${REPOSITORY_NAME} --repository=${REPOSITORY_NAME}

Additional context
I have discussed problem in the pulp_deb pulp/pulp_deb#1214

Here is the list of the files reported unavailable

Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/z/zlib/zlib1g_1.3.dfsg-3.1ubuntu2.1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/v/vim/xxd_9.1.0016-1ubuntu7.5_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/v/vim/vim_9.1.0016-1ubuntu7.5_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/v/vim/vim-tiny_9.1.0016-1ubuntu7.5_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/v/vim/vim-runtime_9.1.0016-1ubuntu7.5_all.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/v/vim/vim-common_9.1.0016-1ubuntu7.5_all.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/u/update-manager/update-manager-core_24.04.9_all.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/u/update-manager/python3-update-manager_24.04.9_all.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/s/shadow/passwd_4.13+dfsg1-4ubuntu3.2_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/p/procps/procps_4.0.4-4ubuntu3.2_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/o/openssh/openssh-sftp-server_9.6p1-3ubuntu13.5_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/o/openssh/openssh-server_9.6p1-3ubuntu13.5_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/o/openssh/openssh-client_9.6p1-3ubuntu13.5_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/s/shadow/login_4.13+dfsg1-4ubuntu3.2_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/o/open-vm-tools/open-vm-tools_12.4.5-1~ubuntu0.24.04.1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/p/procps/libproc2-0_4.0.4-4ubuntu3.2_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/l/llvm-toolchain-18/libllvm18_18.1.3-1ubuntu1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/l/lvm2/libdevmapper1.02.1_1.02.185-3ubuntu3.1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/l/lvm2/libdevmapper-event1.02.1_1.02.185-3ubuntu3.1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/l/llvm-toolchain-18/libclang1-18_18.1.3-1ubuntu1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/l/llvm-toolchain-18/libclang-cpp18_18.1.3-1ubuntu1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/c/cryptsetup/libcryptsetup12_2.7.0-1ubuntu4.1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/a/audit/libaudit1_3.1.2-2.1build1.1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/a/audit/libaudit-common_3.1.2-2.1build1.1_all.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/g/git/git_2.43.0-1ubuntu7.1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/g/git/git-man_2.43.0-1ubuntu7.1_all.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/u/util-linux/bsdutils_2.39.3-9ubuntu6.1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/l/lvm2/dmsetup_1.02.185-3ubuntu3.1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/l/lvm2/dmeventd_1.02.185-3ubuntu3.1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/d/dhcpcd/dhcpcd-base_10.0.6-1ubuntu3.1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/c/cryptsetup/cryptsetup_2.7.0-1ubuntu4.1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/c/cryptsetup/cryptsetup-initramfs_2.7.0-1ubuntu4.1_all.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/c/cryptsetup/cryptsetup-bin_2.7.0-1ubuntu4.1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/b/busybox/busybox-static_1.36.1-6ubuntu3.1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/b/busybox/busybox-initramfs_1.36.1-6ubuntu3.1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/b/bind9/bind9-libs_9.18.30-0ubuntu0.24.04.1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/b/bind9/bind9-host_9.18.30-0ubuntu0.24.04.1_amd64.deb
Not Found: https://pulp.cloud.prz/pulp/content/ubuntu-noble-updates-amd64/pool/main/b/bind9/bind9-dnsutils_9.18.30-0ubuntu0.24.04.1_amd64.deb