Gathering compliance information from Yocto
Background
The overall and long term goal is to have an automated compliance tooling for Yocto based distributions. I’ve previously looked at how to generate dependencies etc. This time I will show a “new” piece of tool that gets us a bit on our way towards automation.
Vanilla++ build
I have built a small vanilla yocto image with a local.conf that
has the following interesting lines added:
IMAGE_INSTALL_append = " epiphany "
IMAGE_INSTALL_append = " elfutils "
INHERIT += "archiver"
ARCHIVER_MODE[src] = "original"
COPY_LIC_MANIFEST = "1"
COPY_LIC_DIRS = "1"
LICENSE_CREATE_PACKAGE = "1"
INHERIT += "buildhistory"
BUILDHISTORY_COMMIT = "1"
BUILDHISTORY_FEATURES = "image"
And then we do:
$ bitbake core-image-minimal
$ bitbake -g core-image-minimal
About the tools (Compliance Utils)
Over at compliance-utils there are some scripts and stuff we’re going to use:
-
yoda - python script to find packages, describe packages, detect license and find dependencies (for binaries)
-
yoga - bash script to “orchestrate” (buzzword bingo!) the yoda functionaliy and to collect license texts
You don’t have to spend any time on the above since all you need to do is to run yoga
About the script names:
-
yoda is short for “Yocto Dependencies Analyser”
-
yoda is short for “Yoda’s Generic Aggregator”
Analysing the build
Enter the build directory of a Yocto build. By default the script we’re going to use (find-yoga-settings.sh probably could chose the latest build date it can find (more on this some other tome) but in this example we’re going to check the build from 20201024110850. So we execute the script like this:
$ DATE=20201024110850 find-yoga-settings.sh core-image-minimal
# yoga -d 20201024110850 -mtd ../ -m qemux86-64 -bd tmp/work/all-poky-linux tmp/work/core2-64-poky-linux tmp/work/qemux86_64-poky-linux -i core-image-minimal-qemux86-64
This tells use how to invoke yoga (yeah, we should probably include this in yoga itself).
Before we do this we need to disable license compatibility checks (-ncc)
(hey, we can do them but we’re going to do it right now).
$ yoga -d 20201024110850 -mtd ../ -m qemux86-64 -bd tmp/work/all-poky-linux tmp/work/core2-64-poky-linux tmp/work/qemux86_64-poky-linux -i core-image-minimal-qemux86-64 -ncc
Creating imagepackages file compliance-results//imagepackage-core-image-minimal-qemux86-64-20201024110850.json: OK
Extracting information about packages (in imagepackage list): OK
Listing all packages
* kmod / 1 sub packages
* Checking if package already done: not done, continuing
* Getting license type: FOSS licenses only, continuing
* Creating component JSON: OK
* Splitting component file OK
* Verifying JSON files for kmod
* Creating flict component JSON: OK
* Creating dot and graph files ()
* kmod-pile-flict.json: OK ( pdf png svg)
* kmod_libkmod.so.2.3.4-tree-flict.json: OK ( pdf png svg)
* Checking license compliance (flict)
PASS, ignored by user request
* Collecting copyright and license information
* Trying tmp/work/core2-64-poky-linux/kmod/26-r0/packages-split/kmod-lic/usr/share/licenses/kmod/: OK
* Creating zip file: OK
The script will run for quite a while and also print out a lot of text. We will exclude this from you and instead focus on the results.
To make your and my life easier we will use a demo site for this: http://rameau.sandklef.com:8080/yoga-demo/compliance-results-2021-01-24/. It is not easy to find your way through all these files and directories, so we will have a look at a limited scope. If you’ve read this blog before you what package we’re going to look at, yes it’s time to look at the project Cairo and the files created when analysing Cairo build by Yocto: http://rameau.sandklef.com:8080/yoga-demo/compliance-results-2021-01-24/cairo/1.16.0-r0/.
Definitions of installed files
For each installed binary file yoga creates a definition. Let’s look at the file cairo_libcairo.so.2.11600.0-tree-flict.json
$ jq . cairo_libcairo.so.2.11600.0-tree-flict.json | head -30
{
"component": {
"name": "libcairo.so.2.11600.0",
"package": "cairo",
"subPackage": "cairo",
"license": " MPL-1.1 | LGPLv2.1",
"version": "1.16.0",
"dependencies": [
{
"name": "pixman",
"component": "libpixman-1.so.0",
"version": "1_0.38.4",
"license": " MIT & MIT-style & PD",
"valid": true,
"dependencies": []
},
{
"name": "fontconfig",
"component": "libfontconfig.so.1",
"version": "2.13.1",
"license": " MIT-style & MIT & PD",
"valid": true,
"dependencies": [
{
"name": "freetype",
"component": "libfreetype.so.6",
"version": "2.10.1",
"license": " FreeType | GPLv2+",
"valid": true,
"dependencies": [
The file contains information about the package name, version, licese (as declared by Yocto), the installed and if the file is a binary it contains the dependencies (tree structure).
Component definition
The file cairo-component.json contains a definition if the cairo component. Looking at the first couple of lines we get an idea of what’s in there:
$ jq . cairo-component.json | head -30
{
"package": "cairo",
"version": "1.16.0",
"license": " (MPL-1.1 | LGPLv2.1) & GPLv3+",
"packageFiles": [
{
"file": "libcairo-gobject.so.2.11600.0",
"subPackage": "cairo-gobject",
"package": "cairo",
"version": "1.16.0",
"license": " MPL-1.1 | LGPLv2.1",
"dependencies": [
{
"file": "libcairo.so.2",
"subPackage": "cairo",
"package": "cairo",
"version": "1.16.0",
"license": " MPL-1.1 | LGPLv2.1",
"dependencies": [
{
"file": "libpixman-1.so.0",
"subPackage": "pixman",
"package": "pixman",
"version": "1_0.38.4",
"license": " MIT & MIT-style & PD",
"dependencies": []
},
{
"file": "libfontconfig.so.1",
"subPackage": "fontconfig",
This file is a cummulative file consisting of the information from all of the above files.
The file cairo-pile-flict.json also contains a definition if the cairo component. Looking at the first couple of lines we get an idea of what’s in there:
$ jq . cairo-pile-flict.json | head -30
{
"component": {
"name": "cairo",
"top_project_license": " (MPL-1.1 | LGPLv2.1) & GPLv3+",
"license": " MPL-1.1 | LGPLv2.1",
"version": "1.16.0",
"dependencies": [
{
"name": "libxshmfence",
"license": " MIT-style",
"version": "1.3",
"dependencies": []
},
{
"name": "libxdmcp",
"license": " MIT-style",
"version": "1_1.1.3",
"dependencies": []
},
{
"name": "libxau",
"license": " MIT-style",
"version": "1_1.0.9",
"dependencies": []
},
{
"name": "libxcb",
"license": " MIT",
"version": "1.13.1",
"dependencies": []
The difference from the previous file is that this time the
dependencies are flattened out (we call it pile). Since the
combinations of a package and its dependencies might be really big
(see dependencies.sh - listing dependencies recursively, pt
II
we need this pile definition when checking license compatibility
(which we disabled using -ncc by the way. In case you still want
to see the compliance report, check out:
cairo-pile-flict-compliance-report.json).
Graphs
In the graphs directory you’ll find graphical representations of the installed files, such as
This is, as you can see, the tree structure version of libcairo’s dependencies. If you insted want to see the flat pile of the cummulative binaries check out: cairo-pile-flict.json.dot.png
Yoga configuraiton
Yoga creates a file called yoga.conf for us. If you want to run yoga again, you don’t have to supply the arguments again, simply invoke yoga. Let’s briefly look at the configuation file.
$ cat yoga.conf
#####################################################
#
# Automatically created onfiguration file for Yoga
#
# License: MIT
#
# Created: lör 13 feb 2021 09:58:34 CET
# OS: Linux gubaidulina 5.8.0-41-generic #46~20.04.1-Ubuntu SMP Mon Jan 18 17:52:23 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
# Host: gubaidulina
# User: hesa
#
#
# Yoga
# project page: https://github.com/vinland-technology/compliance-utils/
# bug report: https://github.com/vinland-technology/compliance-utils/issues
#
#####################################################
# DATE - build date (leave this and yoda will find the latest)
# DATE="20201024110850"
# MACHINE - machine this image was built for
MACHINE="qemux86-64"
# IMAGE - name of the image
IMAGE="core-image-minimal-qemux86-64"
# BUILD_DIRS - list of directories where the build results are stored
BUILD_DIRS=" tmp/work/all-poky-linux tmp/work/core2-64-poky-linux tmp/work/qemux86_64-poky-linux"
# META_TOP_DIR - top directory for the meta files (e.g recipes/bb)
META_TOP_DIR="../"
# TRANSLATION_FILE - file with license translations
#TRANSLATION_FILE=fill in your translation file name here
# POLICY_FILE - file with license policy
#POLICY_FILE=fill in your policy file name here
About the cover image
Henrik Sandklef
Cranes and graffiti from
flickr, (c) 2015 Henrik
Sandklef released under
Attribution-ShareAlike 2.0 Generic (CC BY-SA
2.0)
Hmmm, well…it’s a graphical view?