Compliance automation with Yocto

Note: this post was posted 2020-10-24 and modified 2020-10-25

Let’s begin with what I want. I would like to automate the task of verifying license compliance of a component (including dependencies), that is I want to make sure that the combination of software is OK. For this I use flict. Flict requires a component specified in JSON - let’s look at how flict specifies itself:

$ curl https://raw.githubusercontent.com/vinland-technology/flict/primary/meta/flict.json
{
    "component": {
        "name": "FOSS License Compliance Tool",
        "license": "GPL-3.0-only",
        "dependencies": [
            {
                "name": "gson-2.2.2",
                "license":"Apache-2.0",
                "dependencies": []
            },
            {
                "name": "Apache Commons CLI",
                "license": "Apache-2.0",
                "dependencies": []
            }
        ]
    }
}

So this is the structure of the package and dependency data we want from every component out of a Yocto build. The Yocto version I am using is Dunfell (3.1). In this blog post the focus will be on getting the dependencies for package. In coming posts we will have a look at how to get hold of the license.

Yocto build setup

I have built a small vanilla yocto image with a local.conf thats the following interesting lines added:

IMAGE_INSTALL_append = " epiphany  "
IMAGE_INSTALL_append = " elfutils "

INHERIT += "archiver"
ARCHIVER_MODE[src] = "original"
COPY_LIC_MANIFEST = "1"
COPY_LIC_DIRS = "1"
LICENSE_CREATE_PACKAGE = "1"

INHERIT += "buildhistory"
BUILDHISTORY_COMMIT = "1"
BUILDHISTORY_FEATURES = "image"

And then we do:

$ bitbake core-image-minimal
$ bitbake -g core-image-minimal

Getting the dependencies

Using depends.dot and library names

How do we get this? So far We’ve tried using depends.dot According to the manual:

depends.dot: Dependency graph for the SDK that is compatible with graphviz.

Let’s try this out and have a look at what Yocto is producing assuming we want to check out the library libcairo, a part of Cairo, and what it dependes on (excluding the libs from GLIBC). We do this by using a small tool we’ve written, yocto-plot-package.sh.

$ yocto-plot-package.sh -pdf -df ../build/buildhistory/images/qemux86_64/glibc/core-image-minimal/depends.dot libcairo2 && cat /home/hesa/.vinland-compliance-utils/plot-package/libcairo2.dot
Created /home/hesa/.vinland-compliance-utils/plot-package/libcairo2.dot.png
Created /home/hesa/.vinland-compliance-utils/plot-package/libcairo2.dot.pdf
digraph depends {
    node [shape=plaintext]
"libcairo2" -> "libfontconfig.so.1"
"libcairo2" -> "libfreetype.so.6"
"libcairo2" -> "libGL.so.1"
"libcairo2" -> "libm.so.6"
"libcairo2" -> "libpixman-1.so.0"
"libcairo2" -> "libpng16.so.16"
"libcairo2" -> "libX11.so.6"
"libcairo2" -> "libxcb-render.so.0"
"libcairo2" -> "libxcb-shm.so.0"
"libcairo2" -> "libxcb.so.1"
"libcairo2" -> "libXext.so.6"
"libcairo2" -> "libXrender.so.1"
"libcairo2" -> "libz.so.1"
}

or a graphical view of the same:

*Note: let’s not get into details about why the Yocto package produces a library ending with 2. *

Using depends.dot and package names

Still using depends.dot, but this name we’re going to use the package names in depends.dot. And again, let’s use yocto-plot-package.sh but this time with the option -pn (as in package name):

$ yocto-plot-package.sh -pn -pdf  -df ../build/buildhistory/images/qemux86_64/glibc/core-image-minimal/depends.dot libcairo2  && cat /home/hesa/.vinland-compliance-utils/plot-package/libcairo2.dot
Created /home/hesa/.vinland-compliance-utils/plot-package/libcairo2.dot.png
Created /home/hesa/.vinland-compliance-utils/plot-package/libcairo2.dot.pdf
digraph depends {
    node [shape=plaintext]
"libcairo2" -> "libfontconfig1" 
"libcairo2" -> "libfreetype6" 
"libcairo2" -> "libgl-mesa" 
"libcairo2" -> "libpixman-1-0" 
"libcairo2" -> "libpng16-16" 
"libcairo2" -> "libx11-6" 
"libcairo2" -> "libxcb1" 
"libcairo2" -> "libxcb-render0" 
"libcairo2" -> "libxcb-shm0" 
"libcairo2" -> "libxext6" 
"libcairo2" -> "libxrender1" 
"libcairo2" -> "libz1" 
}

or a graphical view of the same:

*Note: let’s not get into details about why the Yocto package produces a library ending with 2. *

Using the build tree

Now, let’s try using readelf on the library (libcairo.so.2) to find what it dependes on (excluding the libs from GLIBC). We can use readelf directly:

$ readelf -d ./tmp/work/core2-64-poky-linux/cairo/1.16.0-r0/package/usr/lib/libcairo.so | grep NEEDED | cut -d ":" -f 2 | sed -e 's,^[ ]*\[,,g' -e 's,\]$,,g' | grep -v -e libpthread -e librt -e libc.so -e libdl
libpixman-1.so.0so -e libdl
libfontconfig.so.1
libfreetype.so.6
libpng16.so.16
libxcb-shm.so.0
libxcb.so.1
libxcb-render.so.0
libXrender.so.1
libX11.so.6
libXext.so.6
libz.so.1
libGL.so.1
libm.so.6

or by using a small util tool lib-plot.sh

$ lib-plot.sh ./tmp/work/core2-64-poky-linux/cairo/1.16.0-r0/package/usr/lib/libcairo.so && cat libcairo.so.dot
Created libcairo.so.dot.pdf
Created libcairo.so.dot.jpg
Created libcairo.so.dot.png
digraph depends {
  node [shape=plaintext]"libcairo.so" -> "libfontconfig.so.1"
"libcairo.so" -> "libfreetype.so.6"
"libcairo.so" -> "libGL.so.1"
"libcairo.so" -> "libm.so.6"
"libcairo.so" -> "libpixman-1.so.0"
"libcairo.so" -> "libpng16.so.16"
"libcairo.so" -> "libX11.so.6"
"libcairo.so" -> "libxcb-render.so.0"
"libcairo.so" -> "libxcb-shm.so.0"
"libcairo.so" -> "libxcb.so.1"
"libcairo.so" -> "libXext.so.6"
"libcairo.so" -> "libXrender.so.1"
"libcairo.so" -> "libz.so.1"
}

or a graphical view of the same:

Using .shlibdeps

Yes, you’re right, there’s more files to look in to. Another file produced by Yocto ends with .shlibdeps. In the case of Cairo we can find it and view it like this:

$ find . -name "cairo.shlibdeps"
./tmp/work/core2-64-poky-linux/cairo/1.16.0-r0/packages-split/cairo.shlibdeps
$ cat ./tmp/work/core2-64-poky-linux/cairo/1.16.0-r0/packages-split/cairo.shlibdeps
fontconfig (>= 2.13.1)
freetype (>= 2.10.1)
glibc (>= 2.31+git0+1094741224)
libgl-mesa (>= 20.0.2)
libpng (>= 1.6.37)
libx11 (>= 1.6.9)
libxcb (>= 1.13.1)
libxcb-render (>= 1.13.1)
libxcb-shm (>= 1.13.1)
libxext (>= 1.3.4)
libxrender (>= 0.9.10)
pixman (>= 0.38.4)
zlib (>= 1.2.11)

Let’s try finding the coresponding shlibdpes file for each of them:

$ for i in fontconfig freetype libgl-mesa libpng libx11 libxcb libxcb-render libxcb-shm libxext libxrender pixman zlib; do echo -n "$i:  "; find . -name "$i.shlibdeps"  | grep packages-split; done
fontconfig:  ./tmp/work/core2-64-poky-linux/fontconfig/2.13.1-r0/packages-split/fontconfig.shlibdeps
freetype:  ./tmp/work/core2-64-poky-linux/freetype/2.10.1-r0/packages-split/freetype.shlibdeps
libgl-mesa:  ./tmp/work/core2-64-poky-linux/mesa/2_20.0.2-r0/packages-split/libgl-mesa.shlibdeps
libpng:  ./tmp/work/core2-64-poky-linux/libpng/1.6.37-r0/packages-split/libpng.shlibdeps
libx11:  ./tmp/work/core2-64-poky-linux/libx11/1_1.6.9-r0/packages-split/libx11.shlibdeps
libxcb:  ./tmp/work/core2-64-poky-linux/libxcb/1.13.1-r0/packages-split/libxcb.shlibdeps
libxcb-render:  ./tmp/work/core2-64-poky-linux/libxcb/1.13.1-r0/packages-split/libxcb-render.shlibdeps
libxcb-shm:  ./tmp/work/core2-64-poky-linux/libxcb/1.13.1-r0/packages-split/libxcb-shm.shlibdeps
libxext:  ./tmp/work/core2-64-poky-linux/libxext/1_1.3.4-r0/packages-split/libxext.shlibdeps
libxrender:  ./tmp/work/core2-64-poky-linux/libxrender/1_0.9.10-r0/packages-split/libxrender.shlibdeps
pixman:  ./tmp/work/core2-64-poky-linux/pixman/1_0.38.4-r0/packages-split/pixman.shlibdeps
zlib:  ./tmp/work/core2-64-poky-linux/zlib/1.2.11-r0/packages-split/zlib.shlibdeps

It works, but it is a tedious task. Yeah, I could have done with a bit of formating… but what the heck.

Note: yeah, I could have gone for i in $(cat ./tmp/work/core2-64-poky-linux/cairo/1.16.0-r0/packages-split/cairo.shlibdeps | awk '{ print $1 }') ; do echo -n "$i: "; find . -name "$i.shlibdeps" | grep packages-split; done instead, but that would not make for easy reading.

Thoughts on the results above

So, the approaches with depends.dot and build dir seem to produce a valid JSON file for use with flict. Let’s discuss them a bit.

What does the numbers in the table mean? It is the approaches used.

1. Using depends.dot and library names
2. Using depends.dot and package names
3. Using the build tree
4. Using .shlibdeps

Ok, so (ecluding the libc related libraries) we can see that all the 4 approaches find all the libs. Good.

Find the dependencies recursively

All fine so far. But this is not all the information we need. Some of the dependencies might have, fact is they do, their own dependencies. So how do we get them?

We can use readelf to find the dependencies for libfreetype:

$ find -name "libfreetype*.so" | grep "packages-split"
./tmp/work/core2-64-poky-linux/freetype/2.10.1-r0/packages-split/freetype-dev/usr/lib/libfreetype.so
$ readelf -d ./tmp/work/core2-64-poky-linux/freetype/2.10.1-r0/packages-split/freetype/usr/lib/libfreetype.so.6.17.1 | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libpng16.so.16]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

OK, so libfreetype depends on libpng and libz (we’re excluding libc). The task now is to see how we can go from what we found in the first round and get the resursive dpenendencies.

Using depends.dot and library names

Let’s go back to what we found for libcairo2 and focus on the freetype data:

 "libcairo2" -> "libfreetype.so.6"

Can we find the deps from this freetype expression (libfreetype.so.6)? Let’s try:

$ yocto-plot-package.sh -pdf  -df ../build/buildhistory/images/qemux86_64/glibc/core-image-minimal/depends.dot libfreetype.so.6 

Oh, no graph created. So we can’t use libfreetype.so.6 directly. Let’s skip this approach then and continue with trying package names.

Using depends.dot and package names

As before, lets look at the data produced earlier and focus in the freetype part:

"libcairo2" -> "libfreetype6" 

The question now is if we can use this expression (libfreetype6) to find the dependencies. Let’s check:

$ yocto-plot-package.sh -pdf  -df ../build/buildhistory/images/qemux86_64/glibc/core-image-minimal/depends.dot libfreetype6
Created /home/hesa/.vinland-compliance-utils/plot-package/libfreetype6.dot.png
Created /home/hesa/.vinland-compliance-utils/plot-package/libfreetype6.dot.pdf
$ cat /home/hesa/.vinland-compliance-utils/plot-package/libfreetype6.dot
digraph depends {
    node [shape=plaintext]
"libfreetype6" -> "libpng16.so.16"
"libfreetype6" -> "libz.so.1"
}

So a graph was created and we can see the two libraries we expected to find. This means we can probably use the package name to find dependencies recursively.

Since we’ve created a graph, let’s have a look at it:

Using the build tree

We’ve already seen this in action, when we did

$ find -name "libfreetype*.so" | grep "packages-split"
./tmp/work/core2-64-poky-linux/freetype/2.10.1-r0/packages-split/freetype-dev/usr/lib/libfreetype.so
$ readelf -d ./tmp/work/core2-64-poky-linux/freetype/2.10.1-r0/packages-split/freetype/usr/lib/libfreetype.so.6.17.1 | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libpng16.so.16]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

So this works. However, using find is most likely slower than using a file (depends.dot) that we can keep in memory.

Using .shlibdeps

This is similar to build tree approach above so we need not dive in to this any further.

Producing recursive dependencies

Using depends.dot and package names

We believe it is possible to loop through the dependencies for a pcakge. So let’s try this with libcairo2. What we need to do is:

1. find the dependencis for libcairo2

2. for each dependency package, find the dependencies

… and so on.

We’ve written (yet another) util for this, yocto-depends-to-flict.sh. Using this script we get:

$ yocto-depends-to-flict.sh -df ../build/buildhistory/images/qemux86_64/glibc/core-image-minimal/depends.dot libcairo2 > libcairo.json

Let’s verify the JSON file (libcairo.json) with jq:

$ jq '.' libcairo.json
{
  "component": {
    "name": "libcairo2",
    "license": "unknown",
    "dependencies": [
      {
        "name": "libfontconfig1",
        "license": "unknown",
        "dependencies": [
          {
            "name": "libexpat1",
            "license": "unknown",
            "dependencies": []
          },
          {
            "name": "libfreetype6",
            "license": "unknown",
            "dependencies": [
              {
                "name": "libpng16-16",
                "license": "unknown",
                "dependencies": [
                  {
                    "name": "libz1",
                    "license": "unknown",
                    "dependencies": []
                  }
                ]
              },
...... snip
      {
        "name": "libz1",
        "license": "unknown",
        "dependencies": []
      }
    ]
  }
}

Using the build tree

It’s a bit more complicated to find the dependencies for a package using the build tree. Let’s continue with the example of libcairo2.

First of all, we need to find the path to the library. There are roughly 7221547 files (around 7 million) files in my build. So we need to limit our search a bit. Let’s search only in the tmp tree:

$ find tmp/work/core2-64-poky-linux/ -type f | wc -l
4820340

A bit better. Let’s see how many files we can find matching libcairo.so*:

$ find tmp/work/core2-64-poky-linux/ -type f -name "libcairo.so*" | wc -l
22

Uh oh. Some colleagues of mine (I ve many great colleague developers) told us that the library produced by the package cairo should have a library called packages-split in it and that I can ignore files with .debug in the path, so

$ find tmp/work/core2-64-poky-linux/ -type f -name "libcairo.so*" | grep packages-split | grep -v '\.debug'
tmp/work/core2-64-poky-linux/cairo/1.16.0-r0/packages-split/cairo/usr/lib/libcairo.so.2.11600.0

So using this recursively we should be able to find all dependencies recursively. And, yes you guessed right, we do have a script that does this, yocto-build-to-flict.sh. The script takes a package name (e.g cairo) and produces JSON files (in flict format) for every sub directory with artefacts (e.g. libcairo.so):

$ yocto-build-to-flict.sh --no-libc cairo 
Calculating dependencies for: cairo / cairo:  - created cairo__cairo.json
Calculating dependencies for: cairo / cairo-dbg:  - nothing to report
Calculating dependencies for: cairo / cairo-dev:  - nothing to report
Calculating dependencies for: cairo / cairo-doc:  - nothing to report
Calculating dependencies for: cairo / cairo-gobject:  - created cairo__cairo-gobject.json
Calculating dependencies for: cairo / cairo-locale:  - nothing to report
Calculating dependencies for: cairo / cairo-perf-utils:  - created cairo__cairo-perf-utils.json
Calculating dependencies for: cairo / cairo-script-interpreter:  - created cairo__cairo-script-interpreter.json
Calculating dependencies for: cairo / cairo-src:  - nothing to report
Calculating dependencies for: cairo / cairo-staticdev:  - nothing to report

The interesting JSON file is (cairo__cairo.json). Which has a component specified like this:

{
  "component": {
    "name": "libcairo.so.2.11600.0",
    "license": "unknown",
    "dependencies": [
      {
        "name": "libcairo.so.2.11600.0",
        "license": "unknown",
        "dependencies": [
          {
            "name": "libpixman-1.so.0.38.4",
            "license": "unknown",
            "dependencies": []
          },
          {
            "name": "libfontconfig.so.1.12.0",
            "license": "unknown",
            "dependencies": [
              {
                "name": "libfreetype.so.6.17.1",
                "license": "unknown",
                "dependencies": [
                  {
                    "name": "libpng16.so.16.37.0",
                    "license": "unknown",
                    "dependencies": [
                      {
                        "name": "libz.so.1.2.11",
                        "license": "unknown",
                        "dependencies": []
                      }
                    ]
                  },

 .... snip
               {
                "name": "libxshmfence.so.1.0.0",
                "license": "unknown",
                "dependencies": []
              }
            ]
          }
        ]
      }
    ]
  }
}

Proceeding

Using depends.dot and package names

This approach is quick. It can most likely be done faster if we read up the dot file (depends.dot) either as it is or as JSON (after converting it). This will limit the number of reads but still, this solution was fast and easy.

Using the build tree

This approach is slow. Takes about 20-25 seconds for the entire cairo package. We could look in to caching etc to speed things up. But for now, we know that it works albeit a bit slow.

Using .shlibdeps

Same as mentioned earlier, this approach is similar to the build tree approach.

Graph over libcairo’s dependencies (recursively)

With the already mentioned tool, yocto-plot-package.sh, we can plot the dependencies (recursively) for a package. Let’s create a graph over libcairo so we know a bit of what we’ve looked into in this post.

$ yocto-plot-package.sh -r -pdf  -df ../build/buildhistory/images/qemux86_64/glibc/core-image-minimal/depends.dot libcairo2
Created /home/hesa/.vinland-compliance-utils/plot-package/libcairo2.dot.png
Created /home/hesa/.vinland-compliance-utils/plot-package/libcairo2.dot.pdf

Next post

In the coming post we will look at how to retrieve the license from a Yocto build. Stay tuned :)

About the cover image

Entering the pyramid from flickr, (c) 2017 Henrik Sandklef released under Attribution-ShareAlike 2.0 Generic (CC BY-SA 2.0)

The idea behind using this picture is, of course, the link between Egypt and Cairo.