-
Notifications
You must be signed in to change notification settings - Fork 26
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* wip * wip * wip * wip * wip * wip * wip * wip * wip * wip * wip * review * review * wip * wip * Update _posts/2024-04-23-Introducing-local-recipes-index-remote.markdown Co-authored-by: Rubén Rincón Blanco <[email protected]> * Update _posts/2024-04-23-Introducing-local-recipes-index-remote.markdown * Update _posts/2024-04-23-Introducing-local-recipes-index-remote.markdown * Update _posts/2024-04-23-Introducing-local-recipes-index-remote.markdown Co-authored-by: Daniel <[email protected]> * Update _posts/2024-04-23-Introducing-local-recipes-index-remote.markdown Co-authored-by: Daniel <[email protected]> * Update _posts/2024-04-23-Introducing-local-recipes-index-remote.markdown Co-authored-by: Rubén Rincón Blanco <[email protected]> * Update _posts/2024-04-23-Introducing-local-recipes-index-remote.markdown Co-authored-by: Rubén Rincón Blanco <[email protected]> * Update _posts/2024-04-23-Introducing-local-recipes-index-remote.markdown Co-authored-by: Rubén Rincón Blanco <[email protected]> * Update _posts/2024-04-23-Introducing-local-recipes-index-remote.markdown Co-authored-by: Rubén Rincón Blanco <[email protected]> * Update _posts/2024-04-23-Introducing-local-recipes-index-remote.markdown Co-authored-by: Rubén Rincón Blanco <[email protected]> --------- Co-authored-by: Rubén Rincón Blanco <[email protected]> Co-authored-by: Daniel <[email protected]>
- Loading branch information
3 people
authored
Apr 23, 2024
1 parent
f221508
commit cfcd256
Showing
2 changed files
with
323 additions
and
0 deletions.
There are no files selected for viewing
323 changes: 323 additions & 0 deletions
323
_posts/2024-04-23-Introducing-local-recipes-index-remote.markdown
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,323 @@ | ||
| --- | ||
| layout: post | ||
| comments: false | ||
| title: "Introducing the Local-Recipes-Index: A New Repository Type in Conan" | ||
| meta_title: "Understanding Conan's Local-Recipes-Index to improve package management for C/C++ - Conan Blog" | ||
| description: "Explore the capabilities of Conan's local-recipes-index feature and how it enhances dependency management for C and C++ libraries." | ||
| --- | ||
|
|
||
| ConanCenter, the central open-source repository for C and C++ libraries, is a success | ||
| story, containing a vast collection of C and C++ open-source packages. It processes hundreds | ||
| of thousands of download requests daily, building its packages [from a central GitHub | ||
| repository](https://github.com/conan-io/conan-center-index), which is organized in a | ||
| specific folder layout and received nearly 6,000 pull requests from contributors in | ||
| 2023 alone. | ||
|
|
||
| However, ConanCenter does not fit all use cases. For instance, it cannot include | ||
| closed-source libraries or tools that might still be beneficial to the community. | ||
| Additionally, some organizations, particularly large enterprises, prefer not to use | ||
| binaries downloaded from the internet. Instead, they build their own binaries in-house | ||
| using the `conan-center-index` recipes. These organizations often need to customize these | ||
| recipes to meet unique requirements that are not applicable to the broader community, | ||
| making such contributions unsuitable for the upstream repository. The Conan documentation | ||
| [acknowledges this | ||
| practice](https://docs.conan.io/2/devops/conancenter/hosting_binaries.html), recommending | ||
| working from a fork of `conan-center-index` for such needs. | ||
|
|
||
| For all those reasons in version 2.2.0, Conan introduced a new repository type called | ||
| `local-recipes-index`, designed to offer more flexibility by allowing a Conan remote to | ||
| point to a local copy of Conan recipes with a specific layout. | ||
|
|
||
| The `local-recipes-index` allows users to maintain a local folder with the same structure | ||
| as the `conan-center-index` GitHub repository, using it as a source for package recipes. | ||
| This new type of repository is recipes-only, necessitating the construction of package | ||
| binaries from source on each machine where the package is used. For sharing binaries | ||
| across teams, we continue to recommend [using a Conan remote server like | ||
| Artifactory](https://docs.conan.io/2/tutorial/conan_repositories/setting_up_conan_remotes/artifactory/artifactory_ce_cpp.html) | ||
| for production purposes. | ||
|
|
||
| <p class="centered"> | ||
| <img src="{{ site.baseurl }}/assets/post_images/2024-04-23/general-flow-diagram.png" style="display: block; margin-left: auto; margin-right: auto;" alt="Diagram of the general workflow"/> | ||
| </p> | ||
|
|
||
| In this post, we will explore how this feature facilitates the following: | ||
|
|
||
| - This feature enables contributors to share package recipes with the community for | ||
| libraries that might not be suitable for ConanCenter due to various reasons, such as | ||
| licensing constraints or binary distribution policies. An example of this could be CUDA | ||
| or other proprietary libraries, which are distributed as precompiled closed-source | ||
| binaries. | ||
|
|
||
| - It simplifies the adoption of best practices outlined [in the Conan | ||
| documentation](https://docs.conan.io/2/devops/conancenter/hosting_binaries.html) for | ||
| organizations requiring custom-built binaries or modified ConanCenter recipes to meet | ||
| unique requirements. This approach grants users complete control over their third-party | ||
| dependencies, ensuring they are both robust and fully customizable. | ||
|
|
||
| Next, we'll delve into practical examples to demonstrate these two use cases of the | ||
| `local-recipes-index` repository. | ||
|
|
||
| ## Using a local-recipes-index repository with your own recipes | ||
|
|
||
| In this section, we will illustrate how to use the `local-recipes-index` feature for | ||
| scenarios where certain libraries or tools, due to licensing restrictions or proprietary | ||
| nature, are not suitable for ConanCenter. | ||
|
|
||
| For demonstration purposes, let's create a `local-recipes-index` repository for a hypothetical | ||
| `hello` closed-source library using the `local_recipes_index` template for the `conan new` | ||
| command: | ||
|
|
||
| $ mkdir repo && cd repo | ||
| $ conan new local_recipes_index -d name=hello -d version=0.1 \ | ||
| -d url=https://github.com/conan-io/libhello/archive/refs/tags/0.0.1.zip \ | ||
| -d sha256=1dfb66cfd1e2fb7640c88cc4798fe25853a51b628ed9372ffc0ca285fe5be16b | ||
| $ cd .. | ||
|
|
||
| The `conan new local_recipes_index` command creates a template that assumes CMake as the | ||
| build system alongside other heavy assumptions. In practice, it will require customizing | ||
| it, but for this demo, it works as-is. It will create a folder layout equal to the | ||
| `conan-center-index` GitHub repository: | ||
|
|
||
| . | ||
| └── repo | ||
| └── recipes | ||
| └── hello | ||
| ├── all | ||
| │ ├── conandata.yml | ||
| │ ├── conanfile.py | ||
| │ └── test_package | ||
| │ ├── CMakeLists.txt | ||
| │ ├── conanfile.py | ||
| │ └── src | ||
| │ └── example.cpp | ||
| └── config.yml | ||
|
|
||
| After setting up the repository, we add it as a local remote to Conan: | ||
|
|
||
| $ conan remote add mylocalrepo ./repo --allowed-packages="hello/*" | ||
|
|
||
| Please pay special attention to the `--allowed-packages` argument. This argument ensures | ||
| that all packages other than `hello` are discarded by Conan. This can be used to minimize | ||
| the surface area for a potential supply chain attack. | ||
|
|
||
| Now you can list and install packages from this new repository: | ||
|
|
||
| $ conan list "*" -r=mylocalrepo | ||
| $ conan install --requires=hello/0.1 -r=mylocalrepo --build=missing | ||
|
|
||
| At this point, you could push this repository to your GitHub account and share it with the | ||
| community. Please be aware that, as we commented earlier, this feature is specifically | ||
| tailored for scenarios where certain libraries are not suitable for ConanCenter. Remember, | ||
| a "local-recipes-index" repository has limitations: it is not fully reproducible as it | ||
| models only versions and not revisions, and it does not provide binaries. Therefore, | ||
| outside of these cases, it is advised to use a remote package server such as Artifactory. | ||
|
|
||
| Now, users simply need to clone the GitHub repository and add the cloned folder as a local | ||
| repository themselves. | ||
|
|
||
| ## Building Binaries from a private `conan-center-index` fork | ||
|
|
||
| As outlined in [the Conan DevOps | ||
| Guide](https://docs.conan.io/2/devops/using_conancenter.html), there are many cases where | ||
| organizations need to operate independently of ConanCenter by building their own binaries. | ||
| Being decoupled from the public upstream ConanCenter server and building your own binaries | ||
| from a fork of ``conan-center-index`` as suggested in the linked documentation page can have | ||
| many advantages, including absolute control and possibility to customize recipes, giving | ||
| us the ability for the repository to act as a snapshot of versions, be completely robust | ||
| against possible continuous changes and new releases in upstream ConanCenter, etc. | ||
|
|
||
| The `local-recipes-index` repository allows you to easily build binaries from a fork of | ||
| `conan-center-index`, and then hosting them on a Conan remote repository like Artifactory. | ||
| The main difference with the process explained [in the Conan DevOps | ||
| guide](https://docs.conan.io/2/devops/conancenter/hosting_binaries.html) is the ability | ||
| to immediately test multiple local changes without the need to export each time a recipe | ||
| is modified. | ||
|
|
||
| Note that in this case, mixing binaries from ConanCenter with locally built binaries is | ||
| not recommended for several reasons: | ||
|
|
||
| - Binary compatibility: There may be small differences in setup between the ConanCenter CI | ||
| and the user's CI. Maintaining a consistent setup for all binaries can mitigate some issues. | ||
|
|
||
| - Full control over builds: Building all binaries yourself ensures you have complete control | ||
| over the compilation environment and dependency versions. | ||
|
|
||
| Instead, it's recommended to build all your direct and transitive dependencies from the fork. | ||
| To begin, remove the upstream ConanCenter as it will not be used, everything will come | ||
| from our own fork: | ||
|
|
||
| $ conan remote remove conancenter | ||
|
|
||
| Then we will clone our fork (in this case, we are cloning directly the upstream for demo | ||
| purposes, but you would be cloning your fork instead): | ||
|
|
||
| $ git clone https://github.com/conan-io/conan-center-index | ||
|
|
||
| Add this as our `mycenter` remote: | ||
|
|
||
| # Add the mycenter remote pointing to the local folder | ||
| $ conan remote add mycenter ./conan-center-index | ||
|
|
||
| And that’s all! Now you're set to list and use packages from your `conan-center-index` local folder: | ||
|
|
||
| $ conan list "zlib/*" -r=mycenter | ||
| mycenter | ||
| zlib | ||
| zlib/1.2.11 | ||
| zlib/1.2.12 | ||
| zlib/1.2.13 | ||
| zlib/1.3 | ||
| zlib/1.3.1 | ||
|
|
||
| We can also install packages from this repo, for example we can do: | ||
|
|
||
| $ conan install --requires=zlib/1.3 | ||
| ... | ||
| ======== Computing dependency graph ======== | ||
| zlib/1.3: Not found in local cache, looking in remotes... | ||
| zlib/1.3: Checking remote: mycenter | ||
| zlib/1.3: Downloaded recipe revision 5c0f3a1a222eebb6bff34980bcd3e024 | ||
| Graph root | ||
| cli | ||
| Requirements | ||
| zlib/1.3#5c0f3a1a222eebb6bff34980bcd3e024 - Downloaded (mycenter) | ||
|
|
||
| ======== Computing necessary packages ======== | ||
| Requirements | ||
| zlib/1.3#5c0f3a1a222eebb6bff34980bcd3e024:72c852c5f0ae27ca0b1741e5fd7c8b8be91a590a - Missing | ||
| ERROR: Missing binary: zlib/1.3:72c852c5f0ae27ca0b1741e5fd7c8b8be91a590a | ||
|
|
||
| As we can see, Conan managed to get the recipe for ``zlib/1.3`` from ``mycenter``, but | ||
| then it failed because there is no binary. This is expected, **the repository only contains | ||
| the recipes, but not the binaries**. We can build the binary from source with | ||
| ``--build=missing`` argument: | ||
|
|
||
| $ conan install --requires=zlib/1.3 --build=missing | ||
| ... | ||
| zlib/1.3: package(): Packaged 2 '.h' files: zconf.h, zlib.h | ||
| zlib/1.3: package(): Packaged 1 file: LICENSE | ||
| zlib/1.3: package(): Packaged 1 '.a' file: libz.a | ||
| zlib/1.3: Created package revision 0466b3475bcac5c2ce37bb5deda835c3 | ||
| zlib/1.3: Package '72c852c5f0ae27ca0b1741e5fd7c8b8be91a590a' created | ||
| zlib/1.3: Full package reference: zlib/1.3#5c0f3a1a222eebb6bff34980bcd3e024:72c852c5f0ae27ca0b1741e5fd7c8b8be91a590a#0466b3475bcac5c2ce37bb5deda835c3 | ||
| zlib/1.3: Package folder /home/conan/.conan2/p/b/zlib1ed9fe13537a2/p | ||
| WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X: | ||
| WARN: deprecated: 'cpp_info.names' used in: zlib/1.3 | ||
|
|
||
| ======== Finalizing install (deploy, generators) ======== | ||
| cli: Generating aggregated env files | ||
| cli: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh'] | ||
| Install finished successfully | ||
|
|
||
| We can see now the binary package in our local cache: | ||
|
|
||
| $ conan list zlib:* | ||
| Local Cache | ||
| zlib | ||
| zlib/1.3 | ||
| revisions | ||
| 5c0f3a1a222eebb6bff34980bcd3e024 (2024-04-10 11:50:34 UTC) | ||
| packages | ||
| 72c852c5f0ae27ca0b1741e5fd7c8b8be91a590a | ||
| info | ||
| settings | ||
| arch: x86_64 | ||
| build_type: Release | ||
| compiler: gcc | ||
| compiler.version: 9 | ||
| os: Linux | ||
| options | ||
| fPIC: True | ||
| shared: False | ||
|
|
||
| Finally, upload the binary package to our Artifactory repository to make it available for | ||
| our organization, users and CI jobs: | ||
|
|
||
| $ conan remote add myartifactoryrepo <artifactory_url> | ||
| $ conan upload zlib* -r=myartifactoryrepo -c | ||
|
|
||
| This way, consumers of the packages will not only enjoy the pre-compiled binaries and | ||
| avoid having to always re-build from source all dependencies, but that will also provide | ||
| stronger guarantees that the dependencies build and work correctly, that all dependencies | ||
| and transitive dependencies play well together, etc. Decoupling the binary creation | ||
| process from the binary consumption process is the way to achieve faster and more reliable | ||
| usage of dependencies. | ||
|
|
||
| Remember, in a production setting, the `conan upload` command should be executed by CI, | ||
| not developers, following the [Conan | ||
| guidelines](https://docs.conan.io/2/knowledge/guidelines.html). This approach ensures that | ||
| package consumers enjoy pre-compiled binaries and consistency across dependencies. | ||
|
|
||
| ### Modifying the local-recipes-index repository files | ||
|
|
||
| One of the advantages of this approach is that all the changes that we do in every single | ||
| recipe are automatically available for the Conan client. For example, changes to the | ||
| `recipes/zlib/config.yml` file are immediately recognized by the Conan client. If you | ||
| edit that file and remove all versions but the latest and then we `list` the recipes: | ||
|
|
||
| $ conan list "zlib/*" -r=mycenter | ||
| mycenter | ||
| zlib | ||
| zlib/1.3.1 | ||
|
|
||
| When some of the recipes change, then note that the current Conan home already contains a | ||
| cached copy of the package, so it will not update it unless we explicitly use the | ||
| ``--update``, as any other Conan remote. | ||
|
|
||
| So if we do a change in the ``zlib`` recipe in ``recipes/zlib/all/conanfile.py`` and | ||
| repeat: | ||
|
|
||
| $ conan install --requires=zlib/1.3.1 -r=mycenter --update --build=missing | ||
|
|
||
| We will immediately have the new package binary locally built from source from the new | ||
| modified recipe in our Conan home. | ||
|
|
||
| ### Using local-recipes-index Repositories in Production | ||
|
|
||
| Several important points should be considered when using this new feature: | ||
|
|
||
| - It is designed for **third-party packages**, where recipes in one repository are creating | ||
| packages with sources located elsewhere. To package your own code, the standard practice | ||
| of adding `conanfile.py` recipes along with the source code and using the standard | ||
| `conan create` flow is recommended. | ||
|
|
||
| - The `local-recipes-index` repositories point to **local folders in the filesystem**. | ||
| While users may choose to sync that folder with a git repository or other version | ||
| control mechanisms, Conan is agnostic to this, as it is only aware of the folder in the | ||
| filesystem that points to the (current) state of the repository. Users may choose to run | ||
| git commands directly to switch branches/commit/tags and Conan will automatically | ||
| recognise the changes | ||
|
|
||
| - This approach operates at the source level and does not generate package binaries. For | ||
| deployment for development and production environments, the use of a remote package | ||
| server such as Artifactory is crucial. It's important to note that this feature is not a | ||
| replacement for Conan's remote package servers, which play a vital role in hosting | ||
| packages for regular use. | ||
|
|
||
| - Also, note that a server remote can retain a history of changes storing multiple recipe | ||
| revisions. In contrast, a `local-recipes-index` remote can only represent a single | ||
| snapshot at any given time. | ||
|
|
||
| Furthermore, this feature does not support placing server URLs directly in recipes; remote | ||
| repositories must be explicitly added with `conan remote add`. Decoupling abstract package | ||
| requirements, such as "zlib/1.3.1", from their specific origins is crucial to resolving | ||
| dependencies correctly and leveraging Conan's graph capabilities, including version | ||
| conflict detection and resolution, version-ranges resolution, [opting into | ||
| pre-releases](https://docs.conan.io/2/devops/versioning/resolve_prereleases.html), | ||
| [platform_requires](https://docs.conan.io/2/reference/config_files/profiles.html#platform-requires), | ||
| [replace_requires](https://docs.conan.io/2/reference/config_files/profiles.html#replace-requires), | ||
| etc. This separation also facilitates the implementation of modern DevOps practices, such | ||
| as package immutability, full relocatability and package promotions. | ||
|
|
||
| ## Conclusions | ||
|
|
||
| The `local-recipes-index` repository type introduces a new tool that enables workflows | ||
| previously not possible with Conan 1.X: | ||
|
|
||
| - It allows the easy creation of packages from forks of the `conan-center-index` GitHub | ||
| repository. Many enterprises require this due to policies necessitating private | ||
| customizations in recipes that are unsuitable for merging into the upstream repository. | ||
|
|
||
| - It provides a solution for packaging closed-source libraries and tools within the C/C++ | ||
| ecosystem that cannot be included in ConanCenter, enabling their recipes to be shared | ||
| and conveniently used within the Conan community. |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.