postgresql: fix build if pythonSupport is enabled

[hrdinka]

Hi everyone,

this is a fix for #372333

It includes two commits:

postgresql: fix build if pythonSupport is enabled

Since 445371f (which added strictDeps = true;) the build of PostgreSQL is broken if python support is enabled (via argument pythonSupport = true). The problem seems to be that PostgreSQL’s build system is unable to find the supplied Python installation.

This commit fixes the build by setting the PYTHON environment variable within the build environment, as document here:

https://www.postgresql.org/docs/current/install-make.html#CONFIGURE-ENVVARS-PYTHON

postgresql: add PYTHONPATH via wrapProgram

Currently PYTHONPATH is not set when building PostgreSQL. This results in Python modules not being available within pl/python3u.

This commit adds PYTHONPATH of the supplied python3 via wrapProgram, which is especially useful when supplying a custom Python env.

Here is a minimal flake to test everything:

{
  # offending commit, which does not build
  # inputs.nixpkgs.url = "github:nixos/nixpkgs?ref=445371f309a62e9107ad78fb3c65fff768efb43c";

  # this pull request
  inputs.nixpkgs.url = "github:nixos/nixpkgs?ref=pull/372655/head";

  outputs =
    { self, nixpkgs }:
    let
      pkgs = nixpkgs.legacyPackages.x86_64-linux;
    in
    {
      packages.x86_64-linux.default = pkgs.postgresql_16.override {
        pythonSupport = true;

        // optionally supply a custom python env:
        python3 = pkgs.python3.withPackages (ps: with ps; [ base58 ]);
      };
    };
}

Note

I have tested the compilation on both NixOS (x86_64-linux) and aarch64-darwin and noticed that several extensions (pg_cron, pgvector, postgis) don’t build on Darwin due linking errors. Others (pgjwt, pgtap) do, so presumably this is unrelated to this commit. Further I only tested the functionality on NixOS. I only have a Darwin machine for testing since a few days, will try to look into it, but everything is still very complicated on MacOS for me… so any help is appreciated.

@wolfgangwalther (author of offending commit)
@thoughtpolice
@Ma27
@eclairevoyant

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[wolfgangwalther]

I added the following passthru test and confirmed the fix works:

            // lib.optionalAttrs pythonSupport {
              postgresql-python = stdenvNoCC.mkDerivation {
                name = "plpython-test-extension";
                dontUnpack = true;
                doCheck = true;
                nativeCheckInputs = [
                  postgresqlTestHook
                  (this.override { python3 = python3.withPackages (ps: [ ps.base58 ]); })
                ];
                failureHook = "postgresqlStop";
                postgresqlTestUserOptions = "LOGIN SUPERUSER";
                sql = ''
                  CREATE EXTENSION plpython3u;
                  DO LANGUAGE plpython3u $$
                    import base58
                  $$;
                '';
                passAsFile = [ "sql" ];
                checkPhase = ''
                  runHook preCheck
                  psql -a -v ON_ERROR_STOP=1 -f "$sqlPath"
                  runHook postCheck
                '';
                installPhase = "touch $out";
              };

But.. this needs to rebuild postgresql entirely, just for this test. And that's the case every time we change any of the provided python modules as well.

I wonder whether we can find a solution where we avoid the rebuilds?

Some ideas:

• plpython is just an extension actually - can we treat it as such and compile it independently from the main package?
• if so, can we provide additional packages as part of the environment we're creating with postgresql.withPackages, which we then need to call for plpython anyway?

[wolfgangwalther]

• plpython is just an extension actually - can we treat it as such and compile it independently from the main package?

• if so, can we provide additional packages as part of the environment we're creating with postgresql.withPackages, which we then need to call for plpython anyway?

Or maybe.. if we can pull off building plpython separately, we could have that have a withPackages (for python).

So we'd do this in the end:

postgresql.withPackages (pgps: [ pgps.plpython3u.withPackages (pyps: [ pyps.base58 ]) ])

That would be ❤️

[Ma27]

Personally I'd be fine with merging this as-is and implementing @wolfgangwalther's suggestion later on (I can add it to my pile of things, but no guarantees it'll happen soon then).

[hrdinka]

@Ma27, yes I agree.

It is broken now and the fix is ready with no changes to the API. Splitting plpython3u into a separate package is a breaking change, that will take some extra time, caution and requires changes on the user side. It is also not related to the fix, but a new feature.

I will also try to look into it, but it is the same for me – no guarantees. I hoped to find some time today, but the world has different plan for me…

Not sure if plpython3u can be built on its own or if it requires to build the whole of postgres to just extract the module in the end. Then we would not have gained anything, it would actually be worse. Other distros do have it packaged separately, but since they are binary distributions (e.g. debian) they don’t have the problem of long build times on user machines. Gentoo for example does not package it separately. However, I haven’t look at the build scripts yet, so this is just some uninformed rumbling at this point.

[nixpkgs-ci]

Successfully created backport PR for release-24.11:

• [Backport release-24.11] postgresql: fix build if pythonSupport is enabled #374807

[nixpkgs-ci]

Git push to origin failed for release-24.11 with exitcode 1

[wolfgangwalther]

Not sure if plpython3u can be built on its own or if it requires to build the whole of postgres to just extract the module in the end.

Yeah, that's exactly what happens, just tried. So that part won't work.

I'd still be great if we could find a way to add available python packages to the whole thing without needing to rebuild all of postgresql all the time. Maybe something like postgresql.with { packages = ps: [ ps.pg_cron ...]; python = ps: [ ps.pysomething ]; }. We could probably default to enable python once python = .. is passed in there.

[hrdinka]

In general I don’t think that rebuilding postgres because of changed python package changes is that bad. After all this is not a normal python installation where you need to add three new packages a day from AI libraries to web frameworks. This runs in the DB and everyone should do their best to keep it to a minimum. It also should not do any long running resource intensive tasks to not block the executing worker, which can easily lead to DoS. To not end up with a partly dysfunctional DB in the future only packages that can either be easily replaced or are well maintained and supported should be used. The package should not add potentially big attack surfaces to not compromise security. All this already boils down the package selection process.

So at best this is set once to 1–4 packages and never edited again. The base58 package from my test flake is a good example for packages added. More advanced functionally should always be done async from outside services that can communicate with the DB via exchange tables and notifications.

If you really want to circumvent the additional builds, the only way might be to build postgres in one package and wrap it in a second one. The second package only sets the PYTHONPATH in a wrapper around postgres and the first one does not need to react to any python env changes.

[wolfgangwalther]

So at best this is set once to 1–4 packages and never edited again.

Not for us at $work. We use plpython frequently. Especially during development, it would be handy to just add a new python dependency to test something, without having to rebuild the whole of postgresql.