C++ Development With Conda
Conda is a general purpose package manager widely used in the scientific community. We have shown in another post how it can install many assets, including managing binary compatibility. Conda-Forge is a community driven Conda channel1 and set of tools to build and update packages. Together, they make a wide variety of packages available that we can leverage for C and C++ development. Without further ado, let us have a look at some of them.
Package managers usually focus on delivering the libraries that we use in our project, but there are actually many more tools that we use in our development. They are needed at different time and for different purposes (non exclusively):
- to build (compile) our project,
- to run our project (our executable our library),
- to test our project,
- to benchmark our project,
- to generate the documentation for our project,
- to perform quality analysis of our project (e.g. code check and formatting).
Following are useful packages, all available on Conda-Forge, that we may need to work on our C/C++ project.
- A C, C++, and Fortran compiler, along with their standard libraries, and binary toolchain (linker, archiver…).
- Various build tools, such as CMake, Make, Ninja, and Bazel.
- CCache and SCCache compiler caches.
- Header only libraries, such as Xtensor, or dynamic2 libraries such as TBB, multiple BLAS and LAPACK implementations,3 OpenMP, GMP, Zlib, Boost, and Fmtlib.
- The Conan C/C++ package manager, if you need an additional source of libraries, or if you are working to make yours compatible with it.
- Testing frameworks, such as GoogleTest and Catch2.
- Benchmark framework such as Google Benchmark.
- Doxygen, possibly with Sphinx and Breathe if you prefer that toolset.4
- The Clangd5 C/C++ language server for your editor.
- Clang-Tidy5 and CppCheck to verify code; Clang-Format5 to format it.
- Pre-Commit to run check automatically. Notice that this is a Python package, but because Conda is a general purpose package manager, it will autmatically get a compatible Python interpreter without additional input.
- Many more utilities, such as Git, Hub, Wget, AWSCLI.
- Scripting langages, including Python, Perl, Bash if you have custom scripts.
- Language binding libraries, such as Cython, Pybind11, Rcpp, and LibCxxWrap.
- The Singularity containerization solution.
If you are missing something, you can submit it to Conda-Forge. The process happens entirely on Github, and all the building infrastructure will be set up automatically.
Advantages and Drawbacks
Many of the packages mentioned above are redundant with what is already available on our computer or
easily installable with the OS package manager (
Why then choose Conda?
- Conda is cross platform, which means you can easily share the package requirements with collaborators and contributors.
- Conda provides isolated environments, which means you do not have to worry about conflicting dependencies in different projects (one environment per project is a good rule of thumb).
- Conda is reproducible, with environment and lock files. You can share them with collaborators, use them to switch to another machine, or to rebuild your environment later on if you decide to free up some space.
- Conda does not need admin rights to install packages. This is useful if you (or you collaborators) need to work on a university or company server.
- Over language specific package managers, Conda is not limited to libraries or a single language.
- Everything is pre-compiled, which means fast installtion and update times.6
However it is true that Conda has some downsides.
- Conda can be slow to find compatible libraries.6
- Conda is very picky about packages compatibility and can sometimes refuse to install packages,6 all while giving cryptic error messages.
- Conda environments can be hard to use at first.
We will be running the example inside the continuumio/miniconda3 Docker image. This image does not have much except the Miniconda installation. In particular, there is no compiler, C/C++ standard library, or linker.
We run a shell inside the image with
docker run -it --rm continuumio/miniconda3
Then create our environment from the following
It is a good practice to keep all our dependencies there, rather than installing them with
conda install, so that we can easily rebuild our environment if needed.
channels: - conda-forge dependencies: - cxx-compiler - make - cmake - boost-cpp
Finally, we create the environment and activate it.
conda env create --name myproject --file environment.yaml conda activate --name myproject
We can now compile a simple C++ project and keep adding any other packages that we need.
Note on the Compiler Packages
cxx-compiler set many environment variables to
control the compilation process.
AR are necessary as they help tools like CMake find the
compilers and other utilities.
LDFLAGS control compilation options and
are also picked up by CMake.
These are the flags used by Conda when building the packages, so they are very reasonable.
However, they are not all well-suited for development.
For instance, even when configuring with
CMAKE_BUILD_TYPE=Debug, CMake ends up compiling with the
The first solution that we have is to not use the compilers packages but rather the one on our machine if that is an option.
The second option is to leave it as it is. The flags can make it a bit harder to debug, but they are not weakening the build in any way, on the contrary. If we are not using this environment for debugging (e.g. if this is used in continuous integration, or to deploy on a university server), then we can use the flags as is. It is also possible to unset these environment variables manually when we need to run the debugger.
Finally, we can adapt the way our build tool processes these environment varaible flags.
A CMake example is given in this toolchain file
It defines two addtional
CondaRelease that uses the Conda
flags, and reset the other build types to their default value.
We can use it wihtout modifying our source code:
cmake -B build CMAKE_TOOLCHAIN_FILE="path/to/conda-toolchain.cmake -D CMAKE_BUILD_TYPE=..."
Is Conda still relevant for other languages? I belive so. We can use a language’s usual package manager to build our library, and still complete our development environment with Conda to get all the other tools needed, including the language itself.
We can get Rust, Node.js, Haskell, and Julia7 on Conda-Forge. Even for Python and C/C++ that have many libraries in Conda-Forge, we can also use them through other package managers (Pip and Conan) also available on Conda-Forge.
Looking in the Future: The Mamba Org
Many of the issues of Conda are being tackled in a community led project: the Mamba Organization. It includes:
- Mamba, a fast replacement to Conda. It is also fully written in C++, which means that it can be installed very easily without Python (which Conda needs). Micromanba, a standalone distribution, is also under development as a Microconda replacement.
- Boa, a replacement to
conda-buildthe tool used to build Conda packages.
- Quetz, the first open-source server for hosting Conda packages. Up to now, hosting packages has been exclusively done by Ananconda.
All the details and acknowledgments are available in the release announcement
On top of making the Conda experience better for users, this also opens exciting possiblilies. Mamba speed makes it a promising candidate to resolve dependencies in continuous integration. This means that with a single requirement file, we can get the same dependencies in continuous integration that we use for development, and this for all OS!
The open source Quetz server will make it possible to host Conda packages with fine-grained access, including in private networks. This is a possibility available with most major package managers, and which fits the more complex needs of companies.
I would like to thank Wolf Vollprecht, Chris Burr, and Isuru Fernando for helping me understand the Conda-Forge compiler packages.
A collection of Conda packages. ↩︎
Conda encourages shipping dynamic libraries over static. ↩︎
Conda has strict requirements to manage (binary) packages compatibility, as explained in this other post. This can make it slow (or impossible) to find compatible versions of libraries. However, a package manager that would need to compile some libraries would take significantly more time. ↩︎