Merge branch 'main' into add-cuttlefish-(v1.3.0)

Krulknul · Nov 26, 2024 · 7661899 · 7661899
2 parents 7853e60 + 5250024
commit 7661899
Show file tree

Hide file tree

Showing 10 changed files with 330 additions and 72 deletions.
diff --git a/.gitignore b/.gitignore
@@ -1 +1,3 @@
-result
+result
+ssh-config
+keystore.ks
diff --git a/README.md b/README.md
@@ -12,25 +12,25 @@ The new easiest way to spin up a node on [RadixDLT](https://www.radixdlt.com)'s
 { config, pkgs, ... }:
 
 {
-  # Create a "babylon_node" user to run the process as
-  users.users.babylon_node = {
+  # Create a "babylon-node" user to run the process as
+  users.users.babylon-node = {
     # Give the user a home directory
     isNormalUser = true;
-    group = "babylon_node";
-    home = "/home/babylon_node";
+    group = "babylon-node";
+    home = "/home/babylon-node";
   };
 
   # Add a group corresponding to the user
-  users.groups.babylon_node = { };
+  users.groups.babylon-node = { };
 
-  # Enable the babylon_node service
-  services.babylon_node.enable = true;
+  # Enable the babylon-node service
+  services.babylon-node.enable = true;
 
   # Configure the service.
-  services.babylon_node.config = {
-    db.location = "/home/babylon_node/db";
+  services.babylon-node.config = {
+    db.location = "/home/babylon-node/db";
     node.key = {
-      path = "/home/babylon_node/keystore.ks";
+      path = "/home/babylon-node/keystore.ks";
       create_if_missing = true;
     };
   };
@@ -60,7 +60,7 @@ NixOS is a Linux distribution that uses the Nix language for declaratively build
 
 ## Why Nix for your Babylon node?
 
-While it's possible to run a babylon node in a docker container, some node runners prefer the performance of running without docker. Setting this up often comes down to having to manually set up a system by ssh-ing in and imperatively running commands. If we instead use NixOS, we can add the `babylon_node` service configuration to the `configuration.nix`, run `sudo nixos-rebuild switch` and it will simply start working. In addition to easily being able to start and configure the Babylon node service, we can also configure all other things about our node deployment in `configuration.nix`, making it very simple to deploy our node on another machine with exactly the same firewall settings / executables / services / packages etc.
+While it's possible to run a babylon node in a docker container, some node runners prefer the performance of running without docker. Setting this up often comes down to having to manually set up a system by ssh-ing in and imperatively running commands. If we instead use NixOS, we can add the `babylon-node` service configuration to the `configuration.nix`, run `sudo nixos-rebuild switch` and it will simply start working. In addition to easily being able to start and configure the Babylon node service, we can also configure all other things about our node deployment in `configuration.nix`, making it very simple to deploy our node on another machine with exactly the same firewall settings / executables / services / packages etc.
 
 Even if you're not interested in using NixOS to run your node, this project can still come in handy for you. It packages the Babylon node binaries in a reproducible manner, and with runtime dependencies baked in.
 Usually, to run the node software on Ubuntu or another linux distribution, you will have to roughly go through these steps (excluding most configuration):
@@ -78,7 +78,7 @@ Usually, to run the node software on Ubuntu or another linux distribution, you w
 However, with the nix package from this repository:
 * Install the Nix package manager
 * `nix-build babylon-node.nix`, which produces the `result` directory as build output.
-* `./result/bin/babylon_node`
+* `./result/bin/babylon-node`
 
 No need to manually install Java - it's managed by Nix. It should also work across Linux distributions because even glibc is an explicit dependency to this build. And if you use Nix [flakes](https://zero-to-nix.com/concepts/flakes), the versions are pinned to make this package completely reproducible.
 
@@ -117,11 +117,11 @@ Please read and understand the `configuration.nix` to see all the configurations
 
 Now run `nixos-rebuild switch --flake /etc/nixos#x86_64-linux`. (replacing `x86_64-linux` with your system) The system will rebuild the configuration, and after a few seconds it should be finished and the Babylon node service should start running.
 
-Just `journalctl -fu babylon_node` to confirm that it's running.
+Just `journalctl -fu babylon-node` to confirm that it's running.
 
 Note: You may need to adjust the security group on AWS to allow port 30000, because I believe it blocks all but 22 by default.
 
-You now have a minimal node installation running on your machine. To configure the node further, the babylon_node service exposes a number of options that you can declaratively set. See `options.nix` for all currently available options. Most of these options are a wrapper around the regular Babylon node configuration files. In fact, such a file is generated by Nix based on our configuration in `configuration.nix` and is written to `/etc/radixdlt/babylon_node.config` when you rebuild the system.
+You now have a minimal node installation running on your machine. To configure the node further, the babylon-node service exposes a number of options that you can declaratively set. See `options.nix` for all currently available options. Most of these options are a wrapper around the regular Babylon node configuration files. In fact, such a file is generated by Nix based on our configuration in `configuration.nix` and is written to `/etc/radixdlt/babylon-node.config` when you rebuild the system.
 Some required configurations however, are not possible to configure using that config file. This includes things like the user that is used by the systemd service, and the environment variable that contains the keystore password. See the `run_with` suboption in `options.nix` for these configurations.
 
 #### Advanced configurations
@@ -137,22 +137,22 @@ Some required configurations however, are not possible to configure using that c
     # Your other imports...
   ];
 
-  # Create a "babylon_node" user to run the process as
-  users.users.babylon_node = {
+  # Create a "babylon-node" user to run the process as
+  users.users.babylon-node = {
     # Give the user a home directory
     isNormalUser = true;
-    group = "babylon_node";
-    home = "/home/babylon_node";
+    group = "babylon-node";
+    home = "/home/babylon-node";
   };
 
   # Add a group corresponding to the user
-  users.groups.babylon_node = { };
+  users.groups.babylon-node = { };
 
-  # Enable the babylon_node service
-  services.babylon_node.enable = true;
+  # Enable the babylon-node service
+  services.babylon-node.enable = true;
 
   # Configure the service.
-  services.babylon_node.config = {
+  services.babylon-node.config = {
     network = {
       # Use ID of 1 for Mainnet
       id = 1;
@@ -172,7 +172,7 @@ Some required configurations however, are not possible to configure using that c
       };
     };
     db = {
-      location = "/home/babylon_node/db";
+      location = "/home/babylon-node/db";
       # We can turn off the transaction execution index
       local_transaction_execution_index.enable = false;
       # But leave the account change index enabled
@@ -185,28 +185,37 @@ Some required configurations however, are not possible to configure using that c
       system.bind_address = "0.0.0.0"
     };
     node.key = {
-      path = "/home/babylon_node/keystore.ks";
+      path = "/home/babylon-node/keystore.ks";
       create_if_missing = false;
     };
     run_with = {
       # Explicitly set the user for the service
-      user = "babylon_node";
+      user = "babylon-node";
       # And the group
-      group = "babylon_node";
+      group = "babylon-node";
       # Override java options, which allows you to set things like memory allocation
       java_option_overrides = "-Xms12g -Xmx12g"
       # A file which contains environment variables to expose to the service.
       # One particularly important one if you're using a keystore with a password is RADIX_NODE_KEYSTORE_PASSWORD
       environment_file = "/directory/with/secrets/environment";
       # The directory where log files are written
-      working_directory = "/home/babylon_node";
+      working_directory = "/home/babylon-node";
     };
   };
 
   # Any other NixOS configurations, like firewalls or services...
 }
 ```
 
+### Deploying to multiple hosts a the same time using Colmena
+In the ecosystem of Nix community tools, there are a few tools that allow you to automatically deploy configurations to multiple nodes over SSH. There is [deploy-rs](https://github.com/serokell/deploy-rs), [NixOps](https://github.com/NixOS/nixops), and [colmena](https://github.com/zhaofengli/colmena) to name a few. I chose to use Colmena because it can easily deploy files with secrets.
+
+#### Quick note on secrets
+To run a Radix node, you will at least need a keystore file, which contains the private keys associated with the node. A keystore file may also have a password. It is not possible to store these kinds of secrets in the Nix configuration. Configurations stored in `configuration.nix` may end up in the Nix store (`/nix`), which is the immutable cache of packages on the device. This directory is readable by all users, meaning we should avoid storing secrets in this manner.
+
+There are multiple different secret management schemes for Nix: [nix-sops](https://github.com/Mic92/sops-nix), [agenix](https://github.com/ryantm/agenix), and more. But in this example, we will use a built in feature of the Colmena deployment tool which is to simply deploy secrets at a specific file path on the hosts without storing them in the nix store. This way we don't have to manually ssh into the hosts to deploy these files.
+
+
 ## Versioning
 
 For this project, I am trying out the following versioning scheme:

diff --git a/babylon-node.nix b/babylon-node.nix
@@ -34,14 +34,16 @@ let
   binary = systemToBinary.${stdenv.hostPlatform.system};
 in
 stdenv.mkDerivation rec {
-  pname = "babylon_node";
+  pname = "babylon-node";
   version = "1.2.3";
 
   srcs = [
+
     (fetchurl {
       url = "https://github.com/radixdlt/babylon-node/releases/download/v1.3.0/babylon-node-v1.3.0.zip";
       sha256 = "0h0mj6y23ldmb783m0rq9zzczb3pgslar577jfy7xflpz7cmbxf3";
       name = "babylon_node";
+
     })
     (fetchurl {
       url = binary.url;
@@ -56,6 +58,7 @@ stdenv.mkDerivation rec {
     unzip
   ];
 
+
   unpackPhase = ''
     array=($srcs)
     babylon_node=''${array[0]}
@@ -69,6 +72,7 @@ stdenv.mkDerivation rec {
 
   sourceRoot = "babylon_node";
 
+
   installPhase = ''
     mkdir -p $out/jni
     cp ../library/libcorerust.${binary.libraryExtension} $out/jni/
@@ -81,7 +85,7 @@ stdenv.mkDerivation rec {
       --run 'export JAVA_OPTS="$JAVA_OPTS $OVERRIDE_JAVA_OPTS"' \
       --set LD_PRELOAD "$out/jni/libcorerust.${binary.libraryExtension}" \
       --set LD_LIBRARY_PATH "${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH"
-    mv $out/bin/core $out/bin/babylon_node
+    mv $out/bin/core $out/bin/babylon-node
 
     wrapProgram $out/bin/keygen \
       --set JAVA_HOME "${jdk}"

diff --git a/babylon-service.nix b/babylon-service.nix
@@ -11,7 +11,7 @@ let
   pkgsFixed = import nixpkgs { system = pkgs.system; };
   babylon-node = import ./babylon-node.nix { pkgs = pkgsFixed; };
   options = import ./options.nix { inherit lib; };
-  cfg = config.services.babylon_node;
+  cfg = config.services.babylon-node;
   cfgfile = pkgsFixed.writeText "babylon.config" ''
     network.id=${toString cfg.config.network.id}
     network.host_ip=${cfg.config.network.host_ip}
@@ -31,29 +31,52 @@ let
     api.prometheus.port=${toString cfg.config.api.prometheus.port}
     api.core.flags.enable_unbounded_endpoints=${boolToString cfg.config.api.core.flags.enable_unbounded_endpoints}
   '';
+  ledger-snapshot = import ./snapshot/snapshot.nix {
+    pkgs = pkgsFixed;
+    dbDir = cfg.config.db.location;
+    user = cfg.config.run_with.user;
+    group = cfg.config.run_with.group;
+  };
 
 in
 {
-  options.services.babylon_node = options;
-
-  config = lib.mkIf cfg.enable {
-    environment.etc."radixdlt/babylon_node.config".source = cfgfile;
+  options.services.babylon-node = options;
+  config = lib.mkMerge [
+    (lib.mkIf cfg.enable {
+      environment.etc."radixdlt/babylon-node.config".source = cfgfile;
 
-    systemd.services.babylon_node = {
-      wantedBy = [ "multi-user.target" ];
-      after = [ "network.target" ];
-      description = "RadixDLT Babylon Node Service";
-      serviceConfig = {
-        User = cfg.config.run_with.user;
-        Group = cfg.config.run_with.group;
-        ExecStart = "${babylon-node}/bin/babylon_node -config /etc/radixdlt/babylon_node.config";
-        Restart = "always";
-        WorkingDirectory = cfg.config.run_with.working_directory;
-        EnvironmentFile = cfg.config.run_with.environment_file;
+      systemd.services.babylon-node = {
+        wantedBy = [ "multi-user.target" ];
+        after = [
+        "network-online.target"
+        "local-fs.target"
+        "nns-lookup.target"
+        "time-sync.target"
+        "systemd-journald-dev-log.socket"
+        ];
+        wants = [ "network-online.target" ];
+        description = "RadixDLT Babylon Node Service";
+        serviceConfig = {
+          User = cfg.config.run_with.user;
+          Group = cfg.config.run_with.group;
+          ExecStart = "${pkgs.bash}/bin/bash -c 'RADIX_NODE_KEYSTORE_PASSWORD=\$(cat ${cfg.config.run_with.keystore_password_file}) exec ${babylon-node}/bin/babylon-node -config /etc/radixdlt/babylon-node.config'";
+          Restart = "always";
+          WorkingDirectory = cfg.config.run_with.working_directory;
+          LimitNOFILE = 65536;
+          LimitNPROC = 65536;
+          LimitMEMLOCK = "infinity";
+          SuccessExitStatus = "143";
+          TimeoutStopSec = 10;
+        };
+        environment = {
+          OVERRIDE_JAVA_OPTS = cfg.config.run_with.java_option_overrides;
+        };
       };
-      environment = {
-        OVERRIDE_JAVA_OPTS = cfg.config.run_with.java_option_overrides;
-      };
-    };
-  };
+    })
+    {
+      environment.systemPackages = with pkgs; [
+        ledger-snapshot
+      ];
+    }
+  ];
 }
diff --git a/examples/ec2/configuration.nix b/examples/ec2/configuration.nix
@@ -13,10 +13,9 @@
     vim
     htop
     git
+    magic-wormhole # A secure way to transfer files between computers
   ];
 
-  networking.hostName = "babylon-node";
-
   # We need to enable these experimental features
   # for some features of NixOS - like flakes - to work
   nix.settings.experimental-features = [
@@ -36,24 +35,24 @@
     22
   ];
 
-  # Create a "babylon_node" user to run the process as
-  users.users.babylon_node = {
+  # Create a "babylon-node" user to run the process as
+  users.users.babylon-node = {
     isNormalUser = true; # Give the user a home directory to allow node to store some files there
-    group = "babylon_node"; # Add the user to the group
-    home = "/home/babylon_node";
+    group = "babylon-node"; # Add the user to the group
+    home = "/home/babylon-node";
   };
 
   # Add a group corresponding to the user
-  users.groups.babylon_node = { };
+  users.groups.babylon-node = { };
 
-  # Enable the babylon_node service
-  services.babylon_node.enable = true;
+  # Enable the babylon-node service
+  services.babylon-node.enable = true;
 
   # Configure the service.
-  services.babylon_node.config = {
-    db.location = "/home/babylon_node/db";
+  services.babylon-node.config = {
+    db.location = "/home/babylon-node/db";
     node.key = {
-      path = "/home/babylon_node/keystore.ks";
+      path = "/home/babylon-node/keystore.ks";
       create_if_missing = true; # Create a new key if the key file is missing
     };
   };

diff --git a/examples/ec2/flake.nix b/examples/ec2/flake.nix
@@ -2,8 +2,8 @@
   description = "NixOS Flake example for babylon-node-nix usage";
 
   inputs = {
-    nixpkgs.url = github:nixos/nixpkgs/nixos-unstable;
-    babylon-node-nix.url = git+https://github.com/Krulknul/babylon-node-nix?tag=1.2.3+1;
+    nixpkgs.url = "github:nixos/nixpkgs/nixos-24.05";
+    babylon-node-nix.url = "git+https://github.com/Krulknul/babylon-node-nix?tag=1.2.3+1";
   };
 
   outputs =
@@ -21,7 +21,7 @@
               inherit system;
               modules = [
                 ./configuration.nix
-                babylon-node-nix.nixosModules.babylon_node
+                babylon-node-nix.nixosModules.babylon-node
               ];
             };
         in

diff --git a/flake.nix b/flake.nix
@@ -20,7 +20,7 @@
     in
     {
       packages = allPackages;
-      nixosModules.babylon_node = import ./babylon-service.nix {
+      nixosModules.babylon-node = import ./babylon-service.nix {
         inherit nixpkgs;
       };
     };

diff --git a/options.nix b/options.nix
@@ -173,12 +173,12 @@ in
             options = {
               user = mkOption {
                 type = types.str;
-                default = "babylon_node";
+                default = "babylon-node";
                 description = "The user to run the service as";
               };
               group = mkOption {
                 type = types.str;
-                default = "babylon_node";
+                default = "babylon-node";
                 description = "The group to run the service as";
               };
               java_option_overrides = mkOption {
@@ -187,18 +187,18 @@ in
                 description = "Java option overrides";
                 example = "-Xms12g -Xmx12g";
               };
-              environment_file = mkOption {
+              keystore_password_file = mkOption {
                 type = types.str;
                 default = "";
-                description = "The environment file that contains the environment variable RADIX_NODE_KEYSTORE_PASSWORD.";
+                description = "A file that contains the keystore password in cleartext.";
               };
               # I'm not sure whether it is possible to override the directory where log files
               # are written, so it is not currently an option you can set.
               # For that reason, it is important to set the working directory to a directory where the service has write permissions,
               # else the service will not be able to write log files and it will fail to start.
               working_directory = mkOption {
                 type = types.str;
-                default = "/home/babylon_node";
+                default = "/home/babylon-node";
                 description = "The working directory for the service. The service writes log files to this directory.";
               };
             };