C++ Development With Conda

Posted at — Jan 17, 2021

Conda is a general purpose package manager widely used in the scientific community. We have shown in another post how it can install many assets, including managing binary compatibility. Conda-Forge is a community driven Conda channel1 and set of tools to build and update packages. Together, they make a wide variety of packages available that we can leverage for C and C++ development. Without further ado, let us have a look at some of them.

Useful Packages

Package managers usually focus on delivering the libraries that we use in our project, but there are actually many more tools that we use in our development. They are needed at different time and for different purposes (non exclusively):

  • to build (compile) our project,
  • to run our project (our executable our library),
  • to test our project,
  • to benchmark our project,
  • to generate the documentation for our project,
  • to perform quality analysis of our project (e.g. code check and formatting).

Following are useful packages, all available on Conda-Forge, that we may need to work on our C/C++ project.

If you are missing something, you can submit it to Conda-Forge. The process happens entirely on Github, and all the building infrastructure will be set up automatically.

Advantages and Drawbacks

Many of the packages mentioned above are redundant with what is already available on our computer or easily installable with the OS package manager (apt-get, HomeBrew, WinGet, Chocolatey…). Why then choose Conda?

  • Conda is cross platform, which means you can easily share the package requirements with collaborators and contributors.
  • Conda provides isolated environments, which means you do not have to worry about conflicting dependencies in different projects (one environment per project is a good rule of thumb).
  • Conda is reproducible, with environment and lock files. You can share them with collaborators, use them to switch to another machine, or to rebuild your environment later on if you decide to free up some space.
  • Conda does not need admin rights to install packages. This is useful if you (or you collaborators) need to work on a university or company server.
  • Over language specific package managers, Conda is not limited to libraries or a single language.
  • Everything is pre-compiled, which means fast installtion and update times.6

However it is true that Conda has some downsides.

  • Conda can be slow to find compatible libraries.6
  • Conda is very picky about packages compatibility and can sometimes refuse to install packages,6 all while giving cryptic error messages.
  • Conda environments can be hard to use at first.

Example

We will be running the example inside the continuumio/miniconda3 Docker image. This image does not have much except the Miniconda installation. In particular, there is no compiler, C/C++ standard library, or linker.

We run a shell inside the image with

docker run -it --rm continuumio/miniconda3

Then create our environment from the following environment.yaml. It is a good practice to keep all our dependencies there, rather than installing them with conda install, so that we can easily rebuild our environment if needed.

channels:
  - conda-forge
dependencies:
  - cxx-compiler
  - make
  - cmake
  - boost-cpp

Finally, we create the environment and activate it.

conda env create --name myproject --file environment.yaml
conda activate --name myproject

We can now compile a simple C++ project and keep adding any other packages that we need.

Note on the Compiler Packages

The packages fortran-compiler, c-compiler, and cxx-compiler set many environment variables to control the compilation process. Some, like FC, CC, CXX, LD, AR are necessary as they help tools like CMake find the compilers and other utilities. Other like FFLAGS, CFLAGS, CXXFLAGS, CPPFLAGS, and LDFLAGS control compilation options and are also picked up by CMake. These are the flags used by Conda when building the packages, so they are very reasonable. However, they are not all well-suited for development. For instance, even when configuring with CMAKE_BUILD_TYPE=Debug, CMake ends up compiling with the -O2 option.

The first solution that we have is to not use the compilers packages but rather the one on our machine if that is an option.

The second option is to leave it as it is. The flags can make it a bit harder to debug, but they are not weakening the build in any way, on the contrary. If we are not using this environment for debugging (e.g. if this is used in continuous integration, or to deploy on a university server), then we can use the flags as is. It is also possible to unset these environment variables manually when we need to run the debugger.

Finally, we can adapt the way our build tool processes these environment varaible flags. A CMake example is given in this toolchain file conda-toolchain.cmake. It defines two addtional CMAKE_BUILD_TYPE, CondaDebug and CondaRelease that uses the Conda flags, and reset the other build types to their default value. We can use it wihtout modifying our source code:

cmake -B build CMAKE_TOOLCHAIN_FILE="path/to/conda-toolchain.cmake -D CMAKE_BUILD_TYPE=..."

Other Languages

Is Conda still relevant for other languages? I belive so. We can use a language’s usual package manager to build our library, and still complete our development environment with Conda to get all the other tools needed, including the language itself.

We can get Rust, Node.js, Haskell, and Julia7 on Conda-Forge. Even for Python and C/C++ that have many libraries in Conda-Forge, we can also use them through other package managers (Pip and Conan) also available on Conda-Forge.

Looking in the Future: The Mamba Org

Many of the issues of Conda are being tackled in a community led project: the Mamba Organization. It includes:

  • Mamba, a fast replacement to Conda. It is also fully written in C++, which means that it can be installed very easily without Python (which Conda needs). Micromanba, a standalone distribution, is also under development as a Microconda replacement.
  • Boa, a replacement to conda-build the tool used to build Conda packages.
  • Quetz, the first open-source server for hosting Conda packages. Up to now, hosting packages has been exclusively done by Ananconda.

All the details and acknowledgments are available in the release announcement

On top of making the Conda experience better for users, this also opens exciting possiblilies. Mamba speed makes it a promising candidate to resolve dependencies in continuous integration. This means that with a single requirement file, we can get the same dependencies in continuous integration that we use for development, and this for all OS!

The open source Quetz server will make it possible to host Conda packages with fine-grained access, including in private networks. This is a possibility available with most major package managers, and which fits the more complex needs of companies.

Acknowledgements

I would like to thank Wolf Vollprecht, Chris Burr, and Isuru Fernando for helping me understand the Conda-Forge compiler packages.


  1. A collection of Conda packages. ↩︎

  2. Conda encourages shipping dynamic libraries over static. ↩︎

  3. See the knowledge base to use the different libraries. ↩︎

  4. Breathe connects Doxygen outputs to Sphinx. Check out Xtensor for an example of the output. ↩︎

  5. In the clang-tools package. ↩︎

  6. Conda has strict requirements to manage (binary) packages compatibility, as explained in this other post. This can make it slow (or impossible) to find compatible versions of libraries. However, a package manager that would need to compile some libraries would take significantly more time. ↩︎

  7. At the time or writting Julia is stuck on version 1.1 in Conda-Forge, while the latest available is 1.5. This is due to a change in the Julia installation and is being addressed. ↩︎