nixos/k3s: add nftables to Path of k3s service

[RMTT]

kubernetes 1.31 has nftables support, and k3s now support kubernetes 1.31, so it needs nftables binary nft.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[superherointj]

Please, to follow up with nixpkgs guidelines, properly scope the NixOS module name and rename title of this PR and commits. Such as:

nixos/k3s: add nftables to path of k3s service

(Note: It is in lowercase.)

[github-actions]

Eval summary

• Added packages: 0
• Removed packages: 0
• Changed packages: 2
• Rebuild Linux: 2
• Rebuild Darwin: 0

[Mic92]

@superherointj happy with that one?

[superherointj]

@superherointj happy with that one?

Yes.

[RMTT]

pls dont merge now, i'm testing it.

[Mic92]

Just undraft when ready.

[RMTT]

After searched kube-proxy code, i found the kubeconfig param(k3s will add it automatically: https://github.com/k3s-io/k3s/blob/4ec261733e241e2baf77684ad5d87ff071d0f00a/pkg/daemons/agent/agent_linux.go#L43) will be override if we use extraKubeProxyConfig(https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-proxy/app/options.go#L221). That says, if user want to use extraKubeProxyConfig, the user must set kubeconfig (which it's a file generated by k3s):

  services.k3s = {
    enable = true;
    role = "server";
    extraKubeProxyConfig = {
      clientConnection = { kubeconfig = "/etc/rancher/k3s/k3s.yaml"; };
    };
  };

otherwise, the kube-proxy will throw an error to indicate it miss the kubernetes server and port configuration.

Do you think it's acceptable? Or just still use the --kube-proxy-args as before. @superherointj

[superherointj]

if user want to use extraKubeProxyConfig, the user must set kubeconfig (which it's a file generated by k3s):

services.k3s.extraKubeProxyConfig.clientConnection.kubeconfig = "/etc/rancher/k3s/k3s.yaml";

otherwise, the kube-proxy will throw an error to indicate it miss the kubernetes server and port configuration.

Do you think it's acceptable? Or just still use the --kube-proxy-args as before. @superherointj

Since that is required, clientConnection.kubeconfig can be a default value, that can be overridden.
We should briefly documented this behavior in extraKubeProxyConfig description or k3s docs in nixpkgs.

Unfortunately iptables/nftables switch adds another testing dimension. iptables, nftables, kube-proxy-args features needs tests or may break silently on upgrades since K3s maintainers trust tests for upgrades.

As this is extra work, I'm not requesting you to do that. Just adding the extra nftables package is fine. Feel free to only go as far as you are comfortable with. Whatever you prefer.

[superherointj]

All git commits should have a scoped title (like, nixos/k3s: xxxx). Since nixpkgs is a monorepo.

[RMTT]

All git commits should have a scoped title (like, nixos/k3s: xxxx). Since nixpkgs is a monorepo.

i know, but i prefer to format commit message via squash merging

[rorosen]

Other packages that k3s needs are wrapped with the binaries. We should add nftables also there to be consistent with other dependencies.

See

nixpkgs/pkgs/applications/networking/cluster/k3s/builder.nix

Line 361 in 1d7a088

k3sRuntimeDeps = [

[rorosen]

I'm not sure whether I get your comments right, but I think this is a wrong default. K3s actually doesn't use this kubeconfig for kube-proxy. By default it passes kube-proxy the argument --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig, which is another file. I suggest to make the default {}, so that the --kube-proxy-arg=config=... gets only passed if a user explicitly changes this option.

[RMTT]

k3s will pass kubeconfig to kube-proxy in https://github.com/k3s-io/k3s/blob/4ec261733e241e2baf77684ad5d87ff071d0f00a/pkg/daemons/agent/agent_linux.go#L43 via command line arg --kube-config , then kube-proxy will save this arg to o.config.ClientConnection.Kubeconfig in https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-proxy/app/options.go#L106 . However, the o.config object will be override in https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-proxy/app/options.go#L221 by the kubeProxyConfigurataion object which specified via the config=path option.

[RMTT]

You can firstly run k3s with this pr to observe behaviors with different args.

[rorosen]

k3s will pass kubeconfig to kube-proxy in https://github.com/k3s-io/k3s/blob/4ec261733e241e2baf77684ad5d87ff071d0f00a/pkg/daemons/agent/agent_linux.go#L43 via command line arg --kube-config

I agree, but notice that the variable used here is cfg.KubeConfigKubeProxy, which gets constructed here . K3s generates multiple kubeconfigs with different certificates for different components, e.g. kubelet, kubeconfig, k3s-controller, etc. I don't know the internals but we shouldn't change which kubeconfig is used without good reason. The default kubeconfig for kube-proxy is at /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig and not the admin kubeconfig at /etc/rancher/k3s/k3s.yaml.

You can see which args are passed to kube-proxy with journalctl -u k3s | grep kube-proxy, on the current master without this PR and default settings I see:

k3s[778]: time="2024-12-08T14:05:20Z" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=foo --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"

The actual default for extraKubeProxyConfig would change the kubeconfig that kube-proxy uses. Of course we could change the default, but this would lead to other problems, for example if k3s changes its defaults or the k3s data dir is changed. I think it would be best to not generate a config for kube-proxy by default, so that k3s can decide which kubeconfig it passes to kube-proxy. However, if somebody configures a custom extraKubeProxyConfig, he has to take care of setting clientConnection.kubeconfig as mentioned in the option description.

[RMTT]

The actual default for extraKubeProxyConfig would change the kubeconfig that kube-proxy uses. Of course we could change the default, but this would lead to other problems, for example if k3s changes its defaults or the k3s data dir is changed. I think it would be best to not generate a config for kube-proxy by default, so that k3s can decide which kubeconfig it passes to kube-proxy. However, if somebody configures a custom extraKubeProxyConfig, he has to take care of setting clientConnection.kubeconfig as mentioned in the option description.

I agree that.

[rorosen]

Thanks a lot for this!

I can't resolve my own comments but they are resolved. From my side this is ready to go once you squashed the commits.

[marcusramberg]

Looks great! Thanks for this.