How to compile C project

This is an introductory example whose purpose is to show what is needed to compile a C project with dependencies, and what technologies we use to simplify that.

Goal

Write program sum-a-b that sums two floating-point numbers stored in sum-input.json file. To parse the JSON file, we will use the json-c library.

In the following sections, we describe how to build the project on Linux, solving the task above from scratch, what are the difficulties with this approach, and how to use Meson build system and Nix package manager to simplify the build.

Access to the ritchie server

You can work on your computer with Linux (or Windows with WSL). Almost any Linux distribution should be suitable. If unsure, choose Ubuntu.

We will require you to use the ritchie server at least for the semestral work. However, you can use it for other tasks, such as this one, too. You can access the server via SSH:

ssh login@ritchie.ciirc.cvut.cz

For the first time, use the main CTU password (like in KOS). Then, we recommend setting up passwordless login via SSH keys. Run the following commands on your computer:

ssh-keygen
ssh-copy-id login@ritchie.ciirc.cvut.cz

Building manually

Prepare application code

The source code of our application is as follows:

#include <stdio.h>
#include "json-c/json.h"

int
main(void)
{
    json_object *root = json_object_from_file("sum-input.json");
    if (!root) {
        perror("sum-input.json");
        exit(1);
    }

    json_object *obj_a, *obj_b;
    obj_a = json_object_object_get(root, "a");
    obj_b = json_object_object_get(root, "b");

    double a, b;
    a = json_object_get_double(obj_a);
    b = json_object_get_double(obj_b);

    printf("%f + %f = %f\n", a, b, (a + b));
    return 0;
}

Store it under the esw-intro directory to sum-a-b.c file:

mkdir esw-intro
cd esw-intro
$EDITOR sum-a-b.c

Also prepare a JSON input file sum-input.json with the following content:

{
    "a": 2.5,
    "b": 3
}

Building a 3rd-party library

Our program depends on the json-c library to parse JSON files, so we need to get that library first. Most Linux distribution allow you to install the library via its package system, but we build the library from the source code.

From the library's README, the prerequisites are gcc and cmake. We are lucky enough to have them installed on our system.

To build the library, run the following commands:

cd ..
git clone https://github.com/json-c/json-c.git
mkdir json-c-build
mkdir json-c-install
cd json-c-build
cmake ../json-c -DCMAKE_INSTALL_PREFIX=../json-c-install
make           # or cmake --build .
make install   # or cmake --install .

Line-by-line, these commands do:

  1. Go outside of the esw-intro directory
  2. Copy the library source code using git.
  3. Create build directory for the library.
  4. Create install directory for the library.
  5. Change to the build directory.
  6. Call the cmake program with the arguments where to find the source code and where to install the compiled library. CMake is a build system used to automate steps for building the project. In the default configuration, it generates a Makefile used in the next step.
  7. Build the library by running make.
  8. Copy the files needed for using the library to the install location defined above.

The last step installs the following files:

Install the project...
-- Install configuration: "debug"
-- Installing: /esw/1/json-c-install/lib/libjson-c.so.5.3.0
-- Installing: /esw/1/json-c-install/lib/libjson-c.so.5
-- Installing: /esw/1/json-c-install/lib/libjson-c.so
-- Installing: /esw/1/json-c-install/lib/libjson-c.a
-- Installing: /esw/1/json-c-install/lib/cmake/json-c/json-c-targets.cmake
-- Installing: /esw/1/json-c-install/lib/cmake/json-c/json-c-targets-debug.cmake
-- Installing: /esw/1/json-c-install/lib/cmake/json-c/json-c-config.cmake
-- Installing: /esw/1/json-c-install/lib/pkgconfig/json-c.pc
-- Installing: /esw/1/json-c-install/include/json-c/json.h
-- Installing: /esw/1/json-c-install/include/json-c/json_config.h
...

As you can see, static .a and dynamic .so libraries are installed into the lib subdirectory, header files into the include subdirectory and then there are cmake and pkgconfig files, which simplify the use of the library from other projects, e.g. our program. Below, we look at pkg-config in more details.

pkg-config

pkg-config is a tool to simplify compiling and linking applications (and libraries) that need to use other libraries. To illustrate the the main principle, we will investigate the content of json-c.pc installed in the previous step:

prefix=/esw/1/json-c-install
exec_prefix=/esw/1/json-c-install
libdir=/esw/1/json-c-install/lib
includedir=/esw/1/json-c-install/include

Name: json-c
Description: A JSON implementation in C
Version: 0.17.99
Requires:
Libs.private: -lm
Libs: -L${libdir} -ljson-c
Cflags: -I${includedir} -I${includedir}/json-c

The most important are the last two lines telling us which command switches we should pass to the C compiler and linker when we want to use that library.

We can use the pkg-config command to show these switches in the expanded form:

$ pkg-config --with-path ../json-c-install/lib/pkgconfig --cflags json-c
-I/esw/1/json-c-install/include -I/esw/1/json-c-install/include/json-c
$ pkg-config --with-path ../json-c-install/lib/pkgconfig --libs json-c
-L/esw/1/json-c-install/lib -ljson-c

Build the program

Now, we can compile our application by the gcc compiler and get the sum-a-b executable:

cd ../esw-intro
gcc -osum-a-b -O2 -Wall sum-a-b.c

This will fail because the compiler cannot find our library (unless you have it installed globally, e.g. from a Linux package). We have to tell the compiler (and the linker invoked internally) where to find json-c header files and libraries. Either copy the output of the pkg-config command or simply use the following:

gcc -osum-a-b -O2 -Wall sum-a-b.c -I../json-c-install/include -L../json-c-install/lib -ljson-c

Let us briefly discuss the gcc's arguments:

  • -osum-a-b specifies the name of the resulting executable (o as output).
  • -O2 switches on optimization
  • -Wall print (almost) all possible warnings
  • sum-a-b.c is the file with the source code
  • -I../json-c-install/include specifies where to look for header files; header files of standard libraries are searched automatically, usually in /usr/include
  • -L../json-c-install/lib specifies where to look for libraries; standard libraries (e.g., libc) are linked automatically, when needed
  • -ljson-c specifies to link the program with the json-c library. The library is found as libjson-c.so.5.

Finally, run the executable:

$ ./sum-a-b
2.500000 + 3.000000 = 5.500000

If it works, you're lucky, because you have the json-c library installed globally in your system. You can verify this by running:

$ ldd ./sum-a-b
linux-vdso.so.1 (0x00007fff77de8000)
libjson-c.so.5 => /lib/x86_64-linux-gnu/libjson-c.so.5 (0x00007ff10ed14000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff10eb33000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff10ed58000)

If some library is missing, you have two options:

  • compiling the program statically (add -static to the command line):

    gcc -static -osum-a-b -O2 -Wall sum-a-b.c -I../json-c-install/include -L../json-c-install/lib -ljson-c

  • telling the dynamic linker where to find our library via an environment variable:

    LD_LIBRARY_PATH=../json-c-install/lib ./sum-a-b

Issues with compiling the program manually

There are two main issues with the approach described above. First is the distribution of dependencies. The most common solution is distributing them via packages within an operating system's package manager (at least on Linux). We can also use language-specific package managers like Conan for C/C++ or Pip for Python. In our labs, we will use Nix, modern approach to package management promising reproducible builds and working in all Linux distributions.

The second issue corresponds to the arguments of gcc in the example above. Writing compilation commands manually to build the project is not scalable as the number of the arguments grows really fast. The solution is a high-level build system. Well-known high-level build systems are CMake and Meson. In the following labs, we will use Meson, which is conceptually similar to CMake, but is more user-friendly.

Meson build system

To build our program with the Meson build system, we need to create a meson.build file that describes how to build our project:

project('esw-intro', 'c')
jsonc_dep = dependency('json-c')
executable('sum-a-b', 'sum-a-b.c', dependencies: [jsonc_dep])

Then, you can compile the project with Meson:

meson setup builddir
meson compile -C builddir

If you don't have the development files for the json-c library installed globally in your system, the setup fails, because Meson doesn't know where to find the json-c library:

meson.build:2:0: ERROR: Dependency "json-c" not found, tried pkgconfig and cmake

You have to tell it where to find it. Since Meson can use pkg-config to find the dependencies, we just need to tell it, where to look for additional .pc files:

meson setup --pkg-config-path ../json-c-install/lib/pkgconfig/ builddir
meson compile -C builddir

Now, the build should succeed and you can run the program:

./builddir/sum-a-b

You can notice that you don't even need to set LD_LIBRARY_PATH variable, as we did above, because Meson links the programs so that the correct version of the shared library is found automatically:

$ ldd ./builddir/sum-a-b
linux-vdso.so.1 (0x00007fff471f3000)
libjson-c.so.5 => /esw/1/json-c-install/lib/libjson-c.so.5 (0x00007fe74d1cd000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe74cfc2000)
/lib64/ld-linux-x86-64.so.2 (0x00007fe74d1eb000)

Nix package manager

Nix is a revolutionary package manager that allows to install the needed dependencies automatically, independently of your Linux distribution and programming language used and without the administrator permissions. First, you need to install Nix and then create a shell.nix file that contains a list of packages our project depends on:

{ pkgs ? import <nixpkgs> {} }:
pkgs.mkShell {
    packages = with pkgs; [
        pkg-config
        meson
        ninja
        json_c
    ];
}

You can choose from over 90 000 packages available in the nixpkgs repository. Names of available packages can be found on https://search.nixos.org/packages.

Now, you can run nix-shell, which will start a new bash shell with the requested packages magically available. From there, you can run Meson as before:

meson setup nixbuilddir
meson compile -C nixbuilddir

It will automatically use the compiler and dependencies provided by Nix.

Note

Nix does not relay on container technology – it works only by modifying your environment variables. See for example the following:

$ echo $PATH | tr : \\n
/nix/store/1zslabm02hi75anb2w8zjrqwzgs0vrs3-bash-interactive-5.2p26/bin
/nix/store/knxv5h4hsh86c649rabd6dqfd97kwp5d-pkg-config-wrapper-0.29.2/bin
/nix/store/xiy08ppl1znxs87rbjydbrmih5aamwwr-meson-1.3.1/bin
/nix/store/0xns1gyza3sm938pc7klvzrjk5iqam9l-ninja-1.11.1/bin
/nix/store/v3b4la4kh5l7dqzdyraqb1lyfrajfl5w-patchelf-0.15.0/bin
/nix/store/4cjqvbp1jbkps185wl8qnbjpf8bdy8j9-gcc-wrapper-13.2.0/bin
/nix/store/qs1nwzbp2ml3cxzsxihn82hl0w73snr0-gcc-13.2.0/bin
/nix/store/36wymklsa60bigdhb0p3139ws02r46lw-glibc-2.38-44-bin/bin
/nix/store/bicmg5gd50q6igk0y5mga1v0p1lk8f26-coreutils-9.4/bin
/nix/store/c53f8hagyblvx52zylsnqcc0b3nxbrcl-binutils-wrapper-2.40/bin
/nix/store/2ab5740x0cy1d74qvbpl5s28qikmppl5-binutils-2.40/bin
/nix/store/bicmg5gd50q6igk0y5mga1v0p1lk8f26-coreutils-9.4/bin
/nix/store/p6fd7piqrin2h0mqxzmvyxyr6pyivndj-findutils-4.9.0/bin
/nix/store/2d582qba31ii28nyrww9bzb00aq06d1g-diffutils-3.10/bin
/nix/store/vd92lhcxs39hbdnzj8ycak5wvj466s3l-gnused-4.9/bin
/nix/store/mn911d51n5lklwr3zy4mdhxa77wzancb-gnugrep-3.11/bin
/nix/store/h53ycc406fmbq3ff0n0rjxdzb6lk9zcn-gawk-5.2.2/bin
/nix/store/1ds6c0i7z4advdr0z210sxgvmq786h09-gnutar-1.35/bin
/nix/store/nf4fhdqgjka360nkibx1yg14gybwb018-gzip-1.13/bin
/nix/store/v3hp6kidlb9yz6j51a0wlbnpclqpi94f-bzip2-1.0.8-bin/bin
/nix/store/15xrks0frcgils8qxfkhspyg6gi9rxdh-gnumake-4.4.1/bin
/nix/store/5l50g7kzj7v0rdhshld1vx46rf2k5lf9-bash-5.2p26/bin
/nix/store/2pi9hb31np2vhy8r9lfih47rf9n51crz-patch-2.7.6/bin
/nix/store/h8vfiwhq6kmvrnj96w52n36c6qm4lbyl-xz-5.4.6-bin/bin
/nix/store/rn6yfzxwp12z0zqavxx1841mh0ypr7jg-file-5.45/bin
/home/user/.nix-profile/bin
/nix/var/nix/profiles/default/bin
/usr/local/bin
/usr/bin
/bin

Using different versions of dependencies

Besides availability of big amount of packages for any Linux distribution, another advantage of Nix is that it's easy to use different versions of the packages. Let's say we want to test our program with older version 0.15 of the json-c library. We can use this site to find which nixpkgs commit corresponds to which json-c version. Then run nix-shell with that version:

nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/f597e7e9fcf37d8ed14a12835ede0a7d362314bd.tar.gz

Now, you can build a version of your program with different version of json-c (and other libraries):

meson setup nixbuilddir-0.15
meson compile -C nixbuilddir-0.15

Compare the output of the following two commands:

$ ldd ./nixbuilddir/sum-a-b
    linux-vdso.so.1 (0x00007ffcfcdbc000)
    libjson-c.so.5 => /nix/store/gi98mlif4q34c2yc0q7jk7qxikbb1yg9-json-c-0.17/lib/libjson-c.so.5 (0x00007f5d49897000)
    libc.so.6 => /nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib/libc.so.6 (0x00007f5d496ae000)
    libm.so.6 => /nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib/libm.so.6 (0x00007f5d495cc000)
    /nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib/ld-linux-x86-64.so.2 => /nix/store/fz33c1mfi2krpg1lwzizfw28kj705yg0-glibc-2.34-210/lib64/ld-linux-x86-64.so.2 (0x00007f5d498ac000)
$ ldd ./nixbuilddir-0.15/sum-a-b
    linux-vdso.so.1 (0x00007ffe1452a000)
    libjson-c.so.5 => /nix/store/vvy33znvfhd6kd7kz2ikkmhzbvy4q8rw-json-c-0.15/lib/libjson-c.so.5 (0x00007f4074012000)
    libc.so.6 => /nix/store/fz33c1mfi2krpg1lwzizfw28kj705yg0-glibc-2.34-210/lib/libc.so.6 (0x00007f4073e14000)
    /nix/store/fz33c1mfi2krpg1lwzizfw28kj705yg0-glibc-2.34-210/lib/ld-linux-x86-64.so.2 => /nix/store/fz33c1mfi2krpg1lwzizfw28kj705yg0-glibc-2.34-210/lib64/ld-linux-x86-64.so.2 (0x00007f4074027000)

There are many more ways, how you can use Nix to influence versions of used packages, but this is out of scope of this class. It's important, that for this course, we will provide you shell.nix files with needed programs and libraries.

Using IDEs with Nix

If you want to use your favorite IDE with dependencies provided by Nix, it's necessary that your IDE uses the all environment variables set by Nix. The easiest way to achieve that is to run your IDE from within the nix-shell environment. I.e. don't use icons or menus to start the IDE, but run it from the command line.

Alternatively, you can install additional Nix-related tools and/or editor/IDE integrations, such as direnv.

README

Finally, when creating your projects, it is also a good idea to put all the information needed into the README file:

To build the project, install dependencies or use Nix shell:

    nix-shell

Then build the project with meson:

    meson setup builddir
    meson compile -C builddir

And finally run the project:

    ./builddir/sum-a-b