🚀 Replace vcpkg with conan #641

AnotherFoxGuy · 2023-09-24T14:28:32Z

This PR is still a work in progress, but the interface should built and run on both Windows and Linux

TODO

How to build on Windows with conan
Add the remote
conan remote add overte https://git.anotherfoxguy.com/api/packages/overte/conan
Install the deps with conan
conan install . -b missing -pr:b=tools/conan-profiles/vs-19-release -pr=tools/conan-profiles/vs-19-release -of build
Generate the VS project files
cmake --preset conan-default
(Note: Currently only the Release build can be built)

How to build on Linux with conan
Let conan detect the build environment
conan profile detect
Add the remote
conan remote add overte https://git.anotherfoxguy.com/api/packages/overte/conan
Install the deps with conan
conan install . -s build_type=Release -b missing -of build
Generate the make files
cmake --preset conan-default
Build the interface
cd build && make

JulianGro

A little comment on vhacd-util: As far as I know this is somewhat legacy. Our engine will automatically generate collision hulls and doesn't use vhacd-util for that. vhacd-util could be very useful for generating performant collision hulls for more complicated objects, but I don't think anyone does that. In fact, I have never even tried using it, and don't even know if it works.

JulianGro · 2023-09-30T11:49:22Z

conan-recipes/nvidia-texture-tools/all/conanfile.py

+        tc.variables["BUILD_TOOLS"] = "OFF"
+        tc.variables["USE_CUDA"] = "OFF"
+        if self.settings.os == "Linux":
+            tc.variables["CMAKE_CXX_FLAGS"] = "-march=msse3 -fPIC"


This would break on aarch64, which technically Overte can currently run on.

cmake/macros/AutoScribeShader.cmake

JulianGro · 2023-09-30T11:53:51Z

cmake/macros/PackageLibrariesForDeployment.cmake

-    set(QTAUDIO_PATH "$<TARGET_FILE_DIR:${TARGET_NAME}>/audio")
-    set(QTAUDIO_WIN7_PATH "$<TARGET_FILE_DIR:${TARGET_NAME}>/audioWin7/audio")
-    set(QTAUDIO_WIN8_PATH "$<TARGET_FILE_DIR:${TARGET_NAME}>/audioWin8/audio")
+    # TODO: Is this still needed? needs testing


Windows 7 and Windows 8(.1) are End Of Life and therefore also not supported by us anymore.

JulianGro · 2023-09-30T11:56:27Z

cmake/macros/TargetJson.cmake

@@ -7,5 +7,9 @@
 # 
 macro(TARGET_JSON)
 	# We use vcpkg for both json and glm, so we just re-use the target_glm macro here


We will have to remember to remove this.

JulianGro · 2023-09-30T12:00:15Z

CMakeLists.txt

@@ -15,67 +15,32 @@ endif()
 # 3.14 is the minimum version that supports symlinks on Windows
 cmake_minimum_required(VERSION 3.14)


Don't be shy about bumping this version if there is a reason to. VCPKG required a much higher CMake version than this anyway.

JulianGro · 2023-09-30T12:11:15Z

conan-recipes/libnode/config.yml

@@ -0,0 +1,3 @@
+versions:
+  "2019.02":


Does this version the conan-recipe?

JulianGro · 2023-09-30T12:20:27Z

conanfile.py

+        "qt*:qtxmlpatterns": "True",
+        "glad*:spec": "gl",
+        "glad*:gl_profile": "core",
+        "glad*:gl_version": "4.6",


Not sure if you know, but Overte supports multiple OpenGL versions. For example, it will fall back to OpenGL 4.1 if OpenGL 4.6 is not available. It can also be built for OpenGL ES 3.2.
I don't know how this part of the Conan code works though.

This is just the targeted version for glad, I just copied the settings from the vcpkg package

JulianGro · 2023-09-30T12:21:54Z

interface/CMakeLists.txt

@@ -68,8 +68,8 @@ foreach(EXTERNAL ${OPTIONAL_EXTERNALS})
  endif ()
 endforeach()

-find_package(Qt5LinguistTools REQUIRED)
-find_package(Qt5LinguistToolsMacros)
+# find_package(Qt5LinguistTools REQUIRED)


As you might know, we would like to add (more) localization support to Overte, but this is currently unused.

JulianGro · 2023-09-30T12:25:00Z

tools/conan-profiles/vs-19-debug

@@ -0,0 +1,8 @@
+[settings]
+os=Windows
+arch=x86_64


We don't currently have any plans for an aarch64 Windows version, but would it make sense to have the profile name reflect the architecture?

JulianGro · 2023-09-30T12:35:03Z

tools/shadergen.py

@@ -217,7 +217,7 @@ def processCommand(line):
        executeSubprocess(scribeArgs)

        # Generate the un-optimized output
-        executeSubprocess([glslangExec, '-V110', '-o', upoptSpirvFile, unoptGlslFile])
+        executeSubprocess([glslangExec, '-V100', '-o', upoptSpirvFile, unoptGlslFile])


Could you explain what this change is doing? Looking it up online it seems like this switches to an older Visual Studio toolchain?

The glslangValidator from the conan package is too old to support that version, it needs to be updated

JulianGro · 2023-10-01T05:54:51Z

A comment on building this on Linux:
Since Conan is not available as a distribution package (at least not on Debian), but rather a pip package, the installation procedure is not that trivial.

# Install python3 virtual environment
sudo apt install python3-venv
# Create a virtual environment in the Overte codebase
python3 -m venv .venv
# Install pip dependencies
.venv/bin/pip3 install conan
# Now one can run conan
.venv/bin/conan profile detect

JulianGro · 2023-10-01T16:53:18Z

Am I understanding it right that Conan is supposed to only get dependencies of its own if the system doesn't already provide them?
If yes, we should probably always depend on a version range, from the oldest version that we know works, to the newest version that we know works. With libopus we would get version 1.3.1 from the system on Ubuntu 20.04.
Or does Conan "only" provide the tools for us to conditionally use system packages on our own?

AnotherFoxGuy · 2023-10-01T17:16:34Z

Am I understanding it right that Conan is supposed to only get dependencies of its own if the system doesn't already provide them?

That is really dependent on how the conan recipes are written.
The recipes provided by the conan-center are always using conan packages to make packaging easier and more reproducible.
But it is possible to script conan to install system packages, for example the xorg recipe just installs the appropriate system packages https://github.com/conan-io/conan-center-index/blob/master/recipes/xorg/all/conanfile.py#L26

JulianGro · 2023-10-02T08:30:31Z

It would be nice if you could put a README into the conan-recipes directory on how to consume those and how to update them.

It seems like I can build node for example using a command like

../../.venv/bin/conan create all/conanfile.py --version 18.17.1

JulianGro · 2023-10-02T21:04:59Z

Something to figure out later: The resulting binary seems to be searching for the Qt platform plugins in the wrong directory.

juliangro@x299-desktop:~/git/overte/build$ QT_DEBUG_PLUGINS=1 ./interface/interface --disable-displays "OpenVR (Vive)" --disable-inputs "OpenVR (Vive)","Sdl2" --allowMultipleInstances
[10/02 23:02:35] [DEBUG] [hifi.shared] Settings file: "/home/juliangro/.config/Overte - dev/Interface.json"
[10/02 23:02:35] [DEBUG] [settings.manager] Initializing settings write thread
[10/02 23:02:35] [WARNING] [default] QEventLoop: Cannot be used without QApplication
[10/02 23:02:35] [DEBUG] [default] UserActivityLogger is enabled: false
[10/02 23:02:35] [INFO] [default] Disabling following display plugins: ("OpenVR (Vive)")
[10/02 23:02:35] [WARNING] [default] DependencyManager::get(): No instance available for 13PluginManager
[10/02 23:02:35] [INFO] [default] Disabling following input plugins: ("OpenVR (Vive)", "Sdl2")
[10/02 23:02:35] [WARNING] [default] DependencyManager::get(): No instance available for 13PluginManager
[10/02 23:02:35] [DEBUG] [default] QFactoryLoader::QFactoryLoader() checking directory path "/home/juliangro/git/overte/build/interface/platforms" ...
[10/02 23:02:35] [WARNING] [qt.qpa.plugin] Could not find the Qt platform plugin "xcb" in ""
[10/02 23:02:35] [FATAL] [default] This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.
[10/02 23:02:35] [FATAL] [default] 
Abgebrochen (Speicherabzug geschrieben)
juliangro@x299-desktop:~/git/overte/build$

JulianGro · 2023-10-02T21:41:03Z

Building libnode through conan fails on Ubuntu 20.04.
profile detect seems to have chosen C++14, while libnode seems to require C++17 or higher to build.
Log: https://bin.linux.pizza/?3ad49c3deaebf232#5NxEbsMUNagxtaMvnFJxm3dpWXTt2Ja5CJQrYio89wAR

JulianGro · 2023-10-07T18:33:22Z

Just tried on aarch64 postmarketOS. (I cannot select text or pipe it to file for some reason.)

It looks like it is trying to install gl through the Alpine Package manager, which is not a valid package name. Unfortunately, Conan Center is having issues, so I don't know how to take a look at the source code.

AnotherFoxGuy · 2023-10-07T18:58:14Z

I don't know how to take a look at the source code.

The source for the package is here:
https://github.com/conan-io/conan-center-index/blob/master/recipes/opengl/all/conanfile.py

JulianGro · 2023-10-07T20:14:48Z

Installing a bunch of packages fixed the gl issue.

This way we can continue to override CGLTF_ATOF to fix floating point conversion issues on systems with `,` decimal separators.

ksuprynowicz · 2025-01-22T17:26:11Z

I built it from source and tested, Interface works perfectly, and build time is so much faster than vcpkg :)

ksuprynowicz · 2025-01-22T17:41:36Z

I tested server too and works well

ksuprynowicz · 2025-01-22T17:42:58Z

What else should we do before merging?

ksuprynowicz · 2025-01-22T17:44:48Z

Maybe test Windows version once github actions work on this PR?

JulianGro · 2025-01-22T18:53:12Z

Well, technically there is a lot:

Make sure AppImages still build (fix them if needed)
Make sure server packages still build (fix them if needed)
Update GitHub Actions to use Conan (shouldn't take much work at all)
Make sure that aarch64 still builds (fix if needed)
Fix libraries without searchpaths being copied next to the Interface binary (thought, as long as Interface doesn't link to them, this shouldn't be causing problems)
Figure out where to get Qt from

JulianGro · 2025-01-22T18:58:42Z

Also, I am perfectly fine with doing the GitHub Actions stuff. Last time I just waited because I saw Edgar experimenting with some cool stuff, like calling Conan from CMake (instead of calling Conan before CMake).

JulianGro · 2025-01-23T05:42:27Z

So, there is the problem of Qt:
In master, we get Qt as part of the pre-build step on systems where it isn't available through a package manager.
On Windows, I wouldn't be against having people just run the Qt installer, but on Linux this sounds like a mess. People are already going to have Qt, so installing it will make that multiple versions which we have to pick from. Not to mention the Qt installer requiring a Qt account to use, which is a deal breaker in my opinion.

Personally, the best solution I can think of is to do exactly what we already do in master: Run a pre-build step, which gets Qt for us if necessary. This also allows us to get and run Conan automagically, same as we do with VCPKG on master.

This still fails as upstream Qt includes an old bug in qtlocation/src/3rdparty/mapbox-gl-native/include/mbgl/util/geometry.hpp; We should switch to KDE's Qt patchset as it has issues like this patched and we already use it on VCPKG. (https://invent.kde.org/qt/qt/qt5/-/tree/kde/5.15)

AnotherFoxGuy · 2025-01-23T09:47:36Z

Run a pre-build step, which gets Qt for us if necessary

I think it would be better to create a conan package for the prebuild QT, otherwise you can run into weird issues with other conan packages that depend on QT

JulianGro · 2025-01-23T09:52:23Z

Yeah, right now I am thinking we should use Conan's Qt package on any system where it is necessary.
I was thinking we should have a prebuild step to decide which package to use, but I think right now we should probably just get this out of the door and think about a prebuild step later.

JulianGro · 2025-01-23T09:56:35Z

@AnotherFoxGuy I tried getting Qt to build, but upstream isn't compatible with GCC13 and above. As mentioned in 992221b I believe we should use KDE's Qt patchset instead of Trolltech's Qt upstream. That would entail us adding yet another recipe. I have to go for now though, so if you have time and want to give it a try…

JulianGro · 2025-01-24T22:15:42Z

I need a break.
Current issues:

Building with system Qt causes some ssl issues later down the line, both on my machine and on GitHub Actions:

-- Conan: Target declared 'openssl::openssl'
CMake Error at build/generators/cmakedeps_macros.cmake:67 (message):
  Library 'ssl' not found in package.  If 'ssl' is a system library, declare
  it with 'cpp_info.system_libs' property
Call Stack (most recent call first):
  build/generators/openssl_system-Target-release.cmake:23 (conan_package_library_targets)
  build/generators/openssl_systemTargets.cmake:24 (include)
  build/generators/openssl_system-config.cmake:16 (include)
  /usr/share/cmake-3.31/Modules/CMakeFindDependencyMacro.cmake:76 (find_package)
  build/generators/libnode-config.cmake:24 (find_dependency)
  cmake/macros/TargetV8.cmake:10 (find_package)
  libraries/script-engine/CMakeLists.txt:10 (target_v8)


-- Configuring incomplete, errors occurred!

Building with Conan Qt fails on GitHub Actions with some weird undefined symbol error in cpython. This doesn't happen on my local machine.
Windows has issues with Python when building Qt.
Building locally with Qt fails for me during the Overte build process. It throws tons of undefined references. Looking at libscript-engine.so, it doesn't link against libnode! I double checked and it is definitely supposed to link against libnode (at least it does so outside of the conan branch).

[ 38%] Linking CXX executable skeleton-dump
/usr/bin/ld: ../../libraries/script-engine/libscript-engine.so: undefined reference to `v8::CpuProfileNode::GetChildrenCount() const'
/usr/bin/ld: ../../libraries/script-engine/libscript-engine.so: undefined reference to `v8::PropertyDescriptor::~PropertyDescriptor()'
/usr/bin/ld: ../../libraries/script-engine/libscript-engine.so: undefined reference to `v8::Locker::Initialize(v8::Isolate*)'
…

AnotherFoxGuy force-pushed the conan branch from f10011e to c7a34e6 Compare September 24, 2023 15:02

This comment was marked as resolved.

Sign in to view

AnotherFoxGuy force-pushed the conan branch 2 times, most recently from 1f6bfe1 to 705a375 Compare September 27, 2023 11:55

AnotherFoxGuy marked this pull request as ready for review September 27, 2023 14:04

AnotherFoxGuy requested a review from JulianGro as a code owner September 27, 2023 14:04

JulianGro reviewed Sep 30, 2023

View reviewed changes

This comment was marked as resolved.

Sign in to view

AnotherFoxGuy force-pushed the conan branch from 7c7d9d8 to e2fadc3 Compare October 2, 2023 21:18

JulianGro force-pushed the conan branch from 114c5d2 to 2bec5c7 Compare October 9, 2023 07:34

AnotherFoxGuy force-pushed the conan branch from 2bec5c7 to 752e0f2 Compare November 8, 2023 10:02

AnotherFoxGuy force-pushed the conan branch 3 times, most recently from becf99d to 8d93025 Compare November 24, 2023 09:16

daleglass mentioned this pull request Dec 4, 2023

Upgrade openexr #752

Merged

AnotherFoxGuy force-pushed the conan branch 2 times, most recently from e98cdb9 to c699930 Compare December 7, 2023 10:32

AnotherFoxGuy and others added 10 commits January 21, 2025 09:51

🔧 Added artery-font-format

dec4acb

⬆️ Updated google-webrtc to 124

c6d4c8d

🔧 Use system OpenSSL on Linux

afe77e0

🔧 Switch back to prebuild WebRTC for now

7480b2a

🐛 Fixed VS 2022 build scripts

a300379

📝 Updated documentation

9a7a5de

🔧 Also generate RelWithDebInfo config on Linux

eaeb583

Target only TBB, not the conan specific meta target onetbb:onetbb.

e6f5165

Package cgltf as header-only library.

d72a615

This way we can continue to override CGLTF_ATOF to fix floating point conversion issues on systems with `,` decimal separators.

🔧 Updated conan overte remote url

c83838a

AnotherFoxGuy force-pushed the conan branch from d890c35 to c83838a Compare January 21, 2025 08:52

This comment was marked as resolved.

Sign in to view

JulianGro force-pushed the conan branch from fea1b49 to 992221b Compare January 23, 2025 09:48

JulianGro added 5 commits January 24, 2025 22:35

Reenable GitHub Actions

45847b1

Update GitHub Actions for use with Conan.

d4265de

Add KDE patchset Qt.

6cc0ff7

Fix libnode building on Linux.

15c2aa0

Update libnode.

9da5100

		@@ -15,67 +15,32 @@ endif()
		# 3.14 is the minimum version that supports symlinks on Windows
		cmake_minimum_required(VERSION 3.14)

🚀 Replace vcpkg with conan #641

Are you sure you want to change the base?

🚀 Replace vcpkg with conan #641

Conversation

AnotherFoxGuy commented Sep 24, 2023 • edited Loading

This comment was marked as resolved.

JulianGro left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JulianGro commented Oct 1, 2023

This comment was marked as resolved.

This comment was marked as resolved.

JulianGro commented Oct 1, 2023

AnotherFoxGuy commented Oct 1, 2023

This comment was marked as resolved.

JulianGro commented Oct 2, 2023 • edited Loading

This comment was marked as resolved.

This comment was marked as resolved.

JulianGro commented Oct 2, 2023

JulianGro commented Oct 2, 2023

JulianGro commented Oct 7, 2023

AnotherFoxGuy commented Oct 7, 2023

JulianGro commented Oct 7, 2023

ksuprynowicz commented Jan 22, 2025

ksuprynowicz commented Jan 22, 2025

ksuprynowicz commented Jan 22, 2025

ksuprynowicz commented Jan 22, 2025

JulianGro commented Jan 22, 2025 • edited Loading

JulianGro commented Jan 22, 2025

JulianGro commented Jan 23, 2025 • edited Loading

This comment was marked as resolved.

AnotherFoxGuy commented Jan 23, 2025

JulianGro commented Jan 23, 2025

JulianGro commented Jan 23, 2025

JulianGro commented Jan 24, 2025 • edited Loading

AnotherFoxGuy commented Sep 24, 2023 •

edited

Loading

JulianGro commented Oct 2, 2023 •

edited

Loading

JulianGro commented Jan 22, 2025 •

edited

Loading

JulianGro commented Jan 23, 2025 •

edited

Loading

JulianGro commented Jan 24, 2025 •

edited

Loading