Failed to create chainstate manager options.

[Sjors]

On Intel macOS 14.6.1

Built Bitcoin Core from bitcoin/bitcoin#30595 @ acca9ee21652d8cffac14926511c2d7a3e40043d using:

mkdir ~/local-dev
cmake -B build -DBUILD_KERNEL_LIB=ON -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX=$HOME/local-dev -DBUILD_DAEMON=OFF -DBUILD_CLI=OFF -DBUILD_TX=OFF -DBUILD_UTIL=OFF -DENABLE_EXTERNAL_SIGNER=OFF -DBUILD_TESTS=OFF -DENABLE_WALLET=OFF
cmake --build build -j 17
cmake --install build

Built this repo master @ a4649f9 using:

PKG_CONFIG_PATH=$HOME/local-dev/lib/pkgconfig cargo b

And then run it:

RUST_BACKTRACE=1 PKG_CONFIG_PATH=$HOME/local-dev/lib/pkgconfig DYLD_LIBRARY_PATH=$HOME/local-dev/lib cargo run
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.08s
     Running `target/debug/rust-bitcoinkernel`
[2024-09-13T12:47:04Z INFO  libbitcoinkernel] 2024-09-13T12:47:04Z Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
[2024-09-13T12:47:04Z INFO  libbitcoinkernel] 2024-09-13T12:47:04Z Using RdSeed as an additional entropy source
[2024-09-13T12:47:04Z INFO  libbitcoinkernel] 2024-09-13T12:47:04Z Using RdRand as an additional entropy source
thread 'main' panicked at src/main.rs:231:60:
called `Result::unwrap()` on an `Err` value: Internal("Failed to create chainstate manager options.")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
sjors@EzmjP43G rust-bitcoinkernel % RUST_BACKTRACE=1 PKG_CONFIG_PATH=$HOME/local-dev/lib/pkgconfig DYLD_LIBRARY_PATH=$HOME/local-dev/lib cargo run
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.10s
     Running `target/debug/rust-bitcoinkernel`
[2024-09-13T12:49:05Z INFO  libbitcoinkernel] 2024-09-13T12:49:05Z Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
[2024-09-13T12:49:05Z INFO  libbitcoinkernel] 2024-09-13T12:49:05Z Using RdSeed as an additional entropy source
[2024-09-13T12:49:05Z INFO  libbitcoinkernel] 2024-09-13T12:49:05Z Using RdRand as an additional entropy source
thread 'main' panicked at src/main.rs:231:60:
called `Result::unwrap()` on an `Err` value: Internal("Failed to create chainstate manager options.")
stack backtrace:
   0: rust_begin_unwind
             at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panicking.rs:652:5
   1: core::panicking::panic_fmt
             at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/result.rs:1679:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/result.rs:1102:23
   4: rust_bitcoinkernel::main
             at ./src/main.rs:231:9
   5: core::ops::function::FnOnce::call_once
             at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[TheCharlatan]

I think you are the first person to test this on a mac - thank you for doing so! This executable built by cargo run is still a dummy example. It assumes a data directory on my own machine. I will probably move it into an /example directory and make it take a directory argument in the next few days. Does running cargo test work for you?

I am working on making the packaging here a bit easier, but for the meantime did you see the installation command that hebasto posted here: bitcoin/bitcoin#30835 ? Might be a bit easier to wield than switching everything else off:

$ rm -rf build && cmake -B build -DBUILD_KERNEL_LIB=ON
$ cmake --build build --target bitcoinkernel
$ cmake --install build --component Kernel --prefix /home/hebasto/INSTALL
-- Install configuration: "RelWithDebInfo"
-- Installing: /home/hebasto/INSTALL/lib/libbitcoinkernel.so

[Sjors]

That is an easier workflow indeed (also having --prefix in the install phase).

Somehow cargo build doesn't work anymore though:

% cmake --install build --component Kernel --prefix $HOME/local-dev
-- Install configuration: "RelWithDebInfo"
-- Installing: /Users/sjors/local-dev/lib/libbitcoinkernel.dylib

Probably because this doesn't install pkgconfig.

% PKG_CONFIG_PATH=$HOME/local-dev/lib/pkgconfig cargo b
    Updating crates.io index
     Locking 92 packages to latest compatible versions
      Adding bech32 v0.9.1 (latest: v0.11.0)
...
   Compiling serde_derive v1.0.210
   Compiling secp256k1 v0.28.2
   Compiling libbitcoinkernel-sys v0.1.0 (/Users/sjors/dev/rust-bitcoinkernel/libbitcoinkernel-sys)
error: failed to run custom build command for `libbitcoinkernel-sys v0.1.0 (/Users/sjors/dev/rust-bitcoinkernel/libbitcoinkernel-sys)`

Caused by:
  process didn't exit successfully: `/Users/sjors/dev/rust-bitcoinkernel/target/debug/build/libbitcoinkernel-sys-67ee716d1a17b1ac/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=LIBBITCOINKERNEL_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-apple-darwin
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_apple_darwin
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=LIBBITCOINKERNEL_STATIC
  cargo:rerun-if-env-changed=LIBBITCOINKERNEL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-apple-darwin
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_apple_darwin
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-apple-darwin
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_apple_darwin
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-apple-darwin
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_apple_darwin
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR

  --- stderr
  thread 'main' panicked at libbitcoinkernel-sys/build.rs:12:10:
  Failed to find the 'bitcoinkernel' library with pkg-config: 
  pkg-config exited with status code 1
  > PKG_CONFIG_PATH=/Users/sjors/local-dev/lib/pkgconfig PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 pkg-config --libs --cflags libbitcoinkernel

  The system library `libbitcoinkernel` required by crate `libbitcoinkernel-sys` was not found.
  The file `libbitcoinkernel.pc` needs to be installed and the PKG_CONFIG_PATH environment variable must contain its parent directory.
  PKG_CONFIG_PATH contains the following:
      - /Users/sjors/local-dev/lib/pkgconfig

  HINT: you may need to install a package such as libbitcoinkernel, libbitcoinkernel-dev or libbitcoinkernel-devel.

  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

[TheCharlatan]

I just force-pushed a patch to my kernelApi branch to fix this, can you fetch, reset and try again?

[Sjors]

Yup, now it installs pkgconfig.

But building still fails:

   Compiling libbitcoinkernel-sys v0.1.0 (/Users/sjors/dev/rust-bitcoinkernel/libbitcoinkernel-sys)
error: failed to run custom build command for `libbitcoinkernel-sys v0.1.0 (/Users/sjors/dev/rust-bitcoinkernel/libbitcoinkernel-sys)`

Caused by:
  process didn't exit successfully: `/Users/sjors/dev/rust-bitcoinkernel/target/debug/build/libbitcoinkernel-sys-67ee716d1a17b1ac/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=LIBBITCOINKERNEL_NO_PKG_CONFIG
...
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR

  --- stderr
  thread 'main' panicked at libbitcoinkernel-sys/build.rs:19:10:
  Unable to generate bindings: NotExist("/usr/local/include/bitcoinkernel.h")
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

It's looking in the wrong place for bitcoinkernel.h, but that file is also missing from --prefix

[TheCharlatan]

It's looking in the wrong place for bitcoinkernel.h, but that file is also missing from --prefix

Yeah, I missed that 🫠
I force-pushed again.

Edit: Mmh, maybe I recently also broke the way it is looking for headers.

[Sjors]

20e0b8178ff931672717ba401c083d03712b3f28 now puts bitcoinkernel.h in $prefix/include, so that's good. But doesn't find it because it's looking the wrong place.

NotExist("/usr/local/include/bitcoinkernel.h")

libbitcoinkernel.pc says prefix=/usr/local so that seems wrong.

[TheCharlatan]

libbitcoinkernel.pc says prefix=/usr/local so that seems wrong.

Seems like the prefix directory needs to be specified at configure time after all. So adding back -DCMAKE_INSTALL_PREFIX=/path/to/install should fix it. I'll add it to the docs.

[Sjors]

That results in a more reasonable libbitcoinkernel.pc.

cargo b succeeds now, but test doesn't:

% PKG_CONFIG_PATH=$HOME/local-dev/lib/pkgconfig cargo test
warning: fields `prevout`, `script_sig`, and `witness` are never read
   --> tests/test.rs:286:13
    |
285 |         struct Input {
    |                ----- fields in this struct
286 |             prevout: Vec<u8>,
    |             ^^^^^^^
287 |             script_sig: Vec<u8>,
    |             ^^^^^^^^^^
288 |             witness: Vec<Vec<u8>>,
    |             ^^^^^^^
    |
    = note: `Input` has a derived impl for the trait `Debug`, but this is intentionally ignored during dead code analysis
    = note: `#[warn(dead_code)]` on by default

warning: field `outs` is never read
   --> tests/test.rs:294:13
    |
292 |         struct ScanTxHelper {
    |                ------------ field in this struct
293 |             ins: Vec<Input>,
294 |             outs: Vec<Vec<u8>>,
    |             ^^^^
    |
    = note: `ScanTxHelper` has a derived impl for the trait `Debug`, but this is intentionally ignored during dead code analysis

warning: `rust-bitcoinkernel` (test "test") generated 2 warnings
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.11s
     Running unittests src/main.rs (target/debug/deps/rust_bitcoinkernel-d0c4478d859b58d5)
dyld[88278]: Library not loaded: @rpath/libbitcoinkernel.dylib
  Referenced from: <C276E166-3122-3E44-A972-CCC57F924D9D> /Users/sjors/dev/rust-bitcoinkernel/target/debug/deps/rust_bitcoinkernel-d0c4478d859b58d5
  Reason: tried: '/usr/local/lib/libbitcoinkernel.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/lib/libbitcoinkernel.dylib' (no such file), '/usr/local/lib/libbitcoinkernel.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/lib/libbitcoinkernel.dylib' (no such file), '/Users/sjors/dev/rust-bitcoinkernel/target/debug/build/secp256k1-sys-2878a1968f63c986/out/libbitcoinkernel.dylib' (no such file), '/Users/sjors/dev/rust-bitcoinkernel/target/debug/deps/libbitcoinkernel.dylib' (no such file), '/Users/sjors/dev/rust-bitcoinkernel/target/debug/libbitcoinkernel.dylib' (no such file), '/Users/sjors/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libbitcoinkernel.dylib' (no such file), '/Users/sjors/.rustup/toolchains/stable-x86_64-apple-darwin/lib/libbitcoinkernel.dylib' (no such file), '/Users/sjors/lib/libbitcoinkernel.dylib' (no such file), '/usr/local/lib/libbitcoinkernel.dylib' (no such file), '/usr/lib/libbitcoinkernel.dylib' (no such file, not in dyld cache)
error: test failed, to rerun pass `--bin rust-bitcoinkernel`

Caused by:
  process didn't exit successfully: `/Users/sjors/dev/rust-bitcoinkernel/target/debug/deps/rust_bitcoinkernel-d0c4478d859b58d5` (signal: 6, SIGABRT: process abort signal)

[TheCharlatan]

You probably have to set DYLD_LIBRARY_PATH.

[Sjors]

Yes that works! All tests pass.

[TheCharlatan]

Bitcoin Core is now integrated as a git subtree in this project, which makes building and linking much easier. There is no need to set environment variables anymore. It also makes it clear which version of my various branches are actually working with this library.

[Sjors]

Nice! I was able to cargo build and cargo test on the latest master, much simpler.
Does it automatically use the right number of CPU cores for building Bitcoin Core, or should you document NUM_JOBS?

cargo run is still broken of course:

[2024-09-17T11:29:17Z INFO  libbitcoinkernel] 2024-09-17T11:29:17Z Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
[2024-09-17T11:29:17Z INFO  libbitcoinkernel] 2024-09-17T11:29:17Z Using RdSeed as an additional entropy source
[2024-09-17T11:29:17Z INFO  libbitcoinkernel] 2024-09-17T11:29:17Z Using RdRand as an additional entropy source
thread 'main' panicked at src/main.rs:231:60:
called `Result::unwrap()` on an `Err` value: Internal("Failed to create chainstate manager options.")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[TheCharlatan]

Found some time to clean up the example binary. It now takes as an argument a path to a previously used Bitcoin Core regtest directory and then scans for silent payments to a specific hard-coded address. I will completely rework these examples in the future and they probably won't include a silent payment scanner anymore. The current example can be used with:

cargo run -- /home/user/.bitcoin/regtest

[Sjors]

I might wait for that refactor, because I only use silent payments on mainnet :-)