The new README file adds information to build a
target on a local machine, explains the scripts,
parameters, and configuration files in more detail,
and provides some notes on limitations and possible
improvements that can be made to the autobuilder-helper
scripts.

Signed-off-by: Peiran Hong <peiran1997@...>

 README.md | 317 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 317 insertions(+)
 create mode 100644 README.md

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..b50061d
--- /dev/null
+++ b/README.md
@@ -0,0 +1,317 @@
+## Directories
+
+- `git/trash`: Directory for storing successfully finished and not
+  published builds.
+
+- `git/mirror`: *Optional*, directory for storing repo mirrors. If
+  repos are present in this directory, `git config` will do a local
+  clone on the repo instead of fetching from remote.
+
+## Json File Containing Repository Data
+
+In order to build a target, user needs to create a json file to hold
+the data of the repos that are required by the target (e.g. poky,
+bitbake, meta-openembbeded, etc.). All available repo data can be
+found in the `repo-defaults` entry in the `config.json` file. User can
+then copy the desired repo data and create a new json file consisting
+of only the repo data. An example of this would be:
+
+```json
+{
+	"poky" : {
+		"url" : "git://git.yoctoproject.org/poky",
+		"branch" : "master",
+		"revision" : "HEAD",
+		"checkout-dirname" : ".",
+		"no-layer-add" : true,
+		"call-init" : true
+	},
+	"bitbake" : {
+		"url" : "git://git.openembedded.org/bitbake",
+		"branch" : "master",
+		"revision" : "HEAD",
+		"no-layer-add" : true
+	}
+}
+```
+
+In the subsequent sections, this file will be referred to as **`repos.json`**.
+
+## Entry Point Scripts and Usage
+
+### `scripts/prepare-shared-repos`
+
+This script will fetch repos specified by `repos.json` and cache them
+in a custom directory. The cached repos can then be reused for future
+builds.
+
+Args:
+
+1. `repojson`: The path to `repos.json` containing repository data
+   explained in the previous section.
+
+2. `sharedsrcdir`: The directory where the repos will be cached.
+
+3. `-p`, `--publish-dir`: *Optional*, where to publish artefacts
+   to. Fetched repos would be archived as tarball and stored in this
+   directory. This procedure will not happen if `publish-dir` is not
+   specified.
+
+### `scripts/shared-repo-unpack`
+
+This script first fetches repos specified by `repos.json` *as well as*
+specified in the `NEEDREPOS` entry of the target in `config.json` to
+the autobuilder working directory given by user, then calls another
+script `scripts/layer-config`.
+
+`scripts/layer-config` will source `oe-init-build-env` file and
+execute `bitbake-layers add-layer` to add necessary layers (the
+`call-init` entry in `repos.json` for each repository will determine
+if `oe-init-build-env` is sourced, the `no-layer-add` entry will
+determine if `bitbake add-layer` should be run on this repo). Note
+that if `NEEDREPOS` is not set for the specific build target,
+`NEEDREPOS` in target `defaults` is checked and added. refer to
+`getconfigvar()` function in `scripts/utils.py` for details.
+
+Args:
+
+1. `repojson`: Same as in `scripts/prepare-shared-repos`.
+
+2. `abworkdir`: The autobuilder working directory which will resemble
+   the look of `poky` or `openembbeded-core` repo with build
+   directory, `oe-init-build-env`, etc.
+
+3. `target`: The target name defined under the `overrides` entry in
+   `config.json`.
+
+4. `-c`, `--cache-dir`: *Optional*, path to the directory caching the
+   repos fetch by `scripts/prepare-shared-repos`. If this argument is
+   not set, fetched repos will be stored in the `repos` directory
+   inside `abworkdir`.
+
+5. `-p`, `--publish-dir`: *Optional*, same as in
+   `scripts/prepare-shared-repos`.
+
+### `scripts/run-config`
+
+This script is the command responsible for sourcing the
+`oe-init-build-env` file, preparing for and executing `bitbake` to
+start the build, and running any sanity checks and extra commands
+specified by the target.
+
+First, build history related tasks are done. Build history related
+entries in `config.json` are checked. Specifically, it checks if the
+`BUILDHISTORY` entry is set to `true` for the target, and if the
+`reponame:branchname` argument passed into `run-config` is present in
+the `BUILD_HISTORY_DIRECTPUSH` or the `BUILD_HISTORY_FORKPUSH` entry
+in `config.json`. Then `scripts/buildhistory-init` script is executed
+to fetch the build history repo specified in the `BUILD_HISTORY_REPO`
+entry in `config.json`. However, the default build history repo given
+in `config.json` requires permission to clone.
+
+Then for each `step` of the target, the following tasks are run in
+sequence:
+
+1. `bitbake-layers addlayer` is run for all the layers present in the
+   `ADDLAYER` entry of the target in `config.json`. Note that these
+   layers are different from the ones added in the previous step by
+   `scripts/layer-config`.
+
+2. `scripts/setup-config` generates an `auto.conf` under
+   `abworkdir/build/conf` and writes to it the extra configs specified
+   by the `extravars` entry of the target in `config.json`. It also
+   generates a `sdk-extra.conf` file and writes to it the items
+   specified by the `SDKEXTRAS` entry of the target in `config.json`.
+
+3. `bitbake` command is run on the `BBTARGET` specified in the current
+   step with `-k` option.
+
+4. `SANITYTARGETS` of the step is run.
+
+5. `EXTRACMDS` and `EXTRAPLAINCMDS` of the step is run.
+
+6. Remove all layers added in reverse order using `bitbake-layers
+   remove-layer`.
+
+After the build is finished, publish artefacts (copy generated images
+and sdks to the user-defined publish directory. Refer to
+`scripts/publish-artefacts`), collect build results (copy
+`build_directory/tmp/log/oeqa/testresults.json` to the user-defined
+results directory and also compare the current build with previous
+ones. `buildhistory` is being run, refer to `scripts/collect-results`)
+and send error report if and error occurred and the `SENDERRORS` entry
+for the target is set in `config.json` (`send-error-report` is being
+run, refer to `scripts/upload-error-reports`).
+
+Lastly, everything is cleaned up. If there is any error during the
+build or the build is published (i.e. the `--publish-dir` argument is
+specified), the build directory is renamed by appending `-renamed`
+after the original build directory name and the script
+exits. Otherwise, `janitor/clobberdir` script is run to try to move
+the build directory to the trash directory set in `config.json` by the
+`TRASH_DIR` entry.  `janitor/clobberdir` first checks if trash
+directory is present and is valid, then it checks if the trash
+directory is on the same file system with the build directory. If it
+is, move the build directory to trash directory, else delete it.
+
+Args:
+
+1. `target`: Same as in `scripts/shared-repo-unpack`
+
+2. `builddir`: The build directory created by the sourcing of
+   `oe-init-build-env`, **This directory is not the autobuilder
+   working `abworkdir`**, but rather the `build` directory inside it,
+   i.e. it is something like `abworkdir/build`.
+
+3. `branchname`: The branch the build is running on. However, **This
+   does not set the branch the build is running**, it is only used for
+   build history. To change the branch of the build, modify the
+   `branch` entry in `repos.json`.
+
+4. `reponame`: The repo the build is running on. Again, **This does
+   not set the repo the build is running**, it is only used for build
+   history.
+
+5. `-s`, `--sstateprefix`: *Defaults to empty*, the directory prefix
+   to publish sstate into.
+
+6. `-b`, `--buildappsrcrev`: *Defaults to empty*, a build appliance
+   SRCREV to use.
+
+7. `-p`, `--publish-dir`: *Optional*, sets the directory
+   `scripts/publish-artefacts` script will be copying to.
+
+8. `-r`, `--results-dir`: *Optional*, sets the directory
+   `scripts/collect-results` script will be copying to.
+
+9. `-u`, `--build-url`: *Optional*, sets the url to link back to this
+   build from the error report server.
+
+10. `--build-type`: *Defaults to `quick`*, can be either `quick` or
+    `full`. For `quick` type, toolchain tests are skipped.
+
+11. `-t`, `--test`: *Defaults to `false`*, if set to `true`, only
+    `scripts/setup-config` is executed and the commands are dry-run.
+
+12. `-q`, `--quietlogging`: *Defaults to `false`*, prevents the
+    flushing of `bitbake` stdout output.
+
+## Steps to build a target
+
+Suppose we are now in the `scripts` directory. We want to build the
+target `poky-tiny` and work in the directory `$HOME/workdir` and want
+to cache all the repos in the directory `$HOME/cache` for future use.
+
+1. Modify the `BASE_HOMEDIR` in `config.json` to a valid directory for the build to happen. Here we use `$HOME`
+
+```json
+{
+    "BASE_HOMEDIR" : "/home/usr"
+    ...
+}
+```
+
+2. Create required directory `git/trash`. Here we create it under `$HOME`:
+
+```bash
+$ mkdir -p $HOME/git/trash
+```
+
+3. Check the `NEEDREPOS` entry for corresponding build targets in
+   `config.json` and create the `repos.json` file according to the
+   [Json File Containing Repository
+   Data](##{Json-File-Containing-Repository-Data}) section above. In
+   this case `poky` and `bitbake` are needed, so the `repos.json` file
+   should look like:
+
+```json
+{
+	"poky" : {
+		"url" : "git://git.yoctoproject.org/poky",
+		"branch" : "",
+		"revision" : "HEAD",
+		"checkout-dirname" : ".",
+		"no-layer-add" : true,
+		"call-init" : true
+	},
+	"bitbake" : {
+		"url" : "git://git.openembedded.org/bitbake",
+		"branch" : "master",
+		"revision" : "HEAD",
+		"no-layer-add" : true
+	}
+}
+```
+
+4. Run `scripts/prepare-shared-repos` and point to the `repos.json`
+   file as well as the cache directory. In this case we want:
+
+```bash
+$ ./prepare-shared-repos $HOME/repos.json $HOME/cache
+```
+
+5. Run `scripts/shared-repo-unpack` and point to the `repos.json` file
+   as well as the autobuilder's working directory, `-c` or the
+   `--cache-dir` argument can be set since we have already fetched the
+   repos in the previous step:
+
+```bash
+$ ./shared-repo-unpack -c $HOME/cache $HOME/repos.json $HOME/workdir poky-tiny
+```
+
+Now the `$HOME/workdir` should be created and `oe-init-build-env` should be
+sourced, which creates a `build` directory inside `$HOME/workdir`.
+
+6. Run `scripts/run-config` with the target to build, the build
+   directory (**Not the autobuilder working directory**), the branch
+   and the repo we are building on ([**The last two arguments does not
+   affect build**](##`scripts/run-config`)).
+
+```bash
+$ ./run-config poky-tiny $HOME/workdir/build master poky
+```
+
+7. If the build finished without error and is not published, the build
+   directory should be rsynced to the `git/trash` directory. If the
+   build failed, the `build` directory will be renamed to
+   `build-renamed` and stay inside the working directory.
+
+## Notes
+
+1. Instead of asking user for the `config.json` file, all the entry
+   point scripts referred to the file through function
+   `util.loadconfig()`, which first finds out the absolute path of the
+   current executing script (i.e. `scripts/prepare-shared-repos`) and
+   then append `../config.json` to it. This is probably not a good
+   idea.
+
+2. All the copy of the repos are done using `rsync`, `git clone
+   --reference or git clone --shared` may be better.
+
+3. Using `rsync` also caused `prepare-shared-repos` and
+   `shared-repo-unpack` script to do `git clone` and overwrite repo
+   content every time they are being called, even when the repo
+   content is exactly the same. We should probably only be doing `git
+   clone` for new repos and do `git checkout` or `git pull` for
+   existing repos.
+
+4. After a failed or published `scripts/run-config`, the build
+   directory is simply appended with `-renamed` to avoid distracting
+   other builds, this is probably not a good idea (Maybe append random
+   string or build ID type of string after each build directory?).
+
+5. The path specified for `git clone` in function `fetchgitrepo()` in
+   `scripts/utils.py` append `/` directly and causes the path to have
+   two `/` if 'abworkdir' is passed in with a `/` at the end
+   (e.g. `/home/workdir/` instead of `/home/workdir`), this does not
+   have any effect on `git clone` but maybe an extra parsing/formating
+   of the path is better.
+
+6. Feeding an incorrect target name into `scripts/shared-repo-unpack`
+   and `scripts/run-config` will not result in an error. The scripts
+   will instead use configurations in the `defaults` target and
+   `scripts/run-config` will simply skip the bitbake command and exit
+   normally with an exit code 0.
+
+7. `checkvnc` script tries to call `-kill` only on the first display,
+   there should be more flexibility on this.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#47784): https://lists.yoctoproject.org/g/yocto/message/47784
Mute This Topic: https://lists.yoctoproject.org/mt/68855528/1024635
Group Owner: yocto+owner@...
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub  [akuster@...]
-=-=-=-=-=-=-=-=-=-=-=-

Thank you both very much for your quick answers !

I will definitely give a shot at Ross' contrib. I also thought about mounting a sshFS when the container starts and trick Yocto by using the FILE protocol.
I will compare the performances.

Thanks again !

Hi Yi,

I've merged the rest of this series, but this one fails to apply.  It looks like your tree didn't contain:

commit 5fd3c5b71edb99659aeb5cb5903088d84517382e (relabel, master)

Author: Christophe PRIOUZEAU <christophe.priouzeau@...>

Date:   Tue Nov 5 14:47:09 2019 +0000

    autorelabel: only selinux-autorelabel need autorelabel file

    With previous implementation, several packages provided

     .autorelabel file while only selinux-autorelabel manage it.

    If there is several packages which try to install .autorelabel

    file, an issue occur during installation of packagegroup-core-selinux.

    Signed-off-by: Christophe Priouzeau <christophe.priouzeau@...>

    Signed-off-by: Joe MacDonald <joe_macdonald@...>

Can you take a look at it and verify which parts of the change are still necessary given the above change?

Sure. I will send V2.

//Yi

Thanks.

-J.

On Wed, Nov 13, 2019 at 8:50 PM Yi Zhao <yi.zhao@...> wrote:

* Switch to python3

* Drop patches:
  Fix-build-failure-with-GCC-7-due-to-possible-truncat.patch
  setools4-fix-cross-compiling-errors-for-powerpc-mips.patch

Signed-off-by: Yi Zhao <yi.zhao@...>
---
 ...e-with-GCC-7-due-to-possible-truncat.patch | 105 ------------------
 ...ss-compiling-errors-for-powerpc-mips.patch |  35 ------
 .../setools4-fixes-for-cross-compiling.patch  |  34 +++---
 .../{setools_4.1.1.bb => setools_4.2.2.bb}    |  14 +--
 4 files changed, 19 insertions(+), 169 deletions(-)
 delete mode 100644 recipes-security/setools/setools/Fix-build-failure-with-GCC-7-due-to-possible-truncat.patch
 delete mode 100644 recipes-security/setools/setools/setools4-fix-cross-compiling-errors-for-powerpc-mips.patch
 rename recipes-security/setools/{setools_4.1.1.bb => setools_4.2.2.bb} (61%)

diff --git a/recipes-security/setools/setools/Fix-build-failure-with-GCC-7-due-to-possible-truncat.patch b/recipes-security/setools/setools/Fix-build-failure-with-GCC-7-due-to-possible-truncat.patch
deleted file mode 100644
index a5af041..0000000
--- a/recipes-security/setools/setools/Fix-build-failure-with-GCC-7-due-to-possible-truncat.patch
+++ /dev/null
@@ -1,105 +0,0 @@
-Upstream-Status: Backport [https://github.com/TresysTechnology/setools/commit/e41adf0]
-
-Signed-off-by: Kai Kang <kai.kang@...>
-
-From e41adf01647c695b80b112b337e76021bb9f30c3 Mon Sep 17 00:00:00 2001
-From: Laurent Bigonville <bigon@...>
-Date: Tue, 26 Sep 2017 15:15:30 +0200
-Subject: [PATCH] Fix build failure with GCC 7 due to possible truncation of
- snprintf output
-
-setools fails to build under GCC7 -Wformat -Werror with the following error:
-
-x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -Wno-sign-compare -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Ilibqpol -Ilibqpol/include -I/usr/include/python3.6m -c libqpol/policy_extend.c -o build/temp.linux-amd64-3.6/libqpol/policy_extend.o -Werror -Wextra -Waggregate-return -Wfloat-equal -Wformat -Wformat=2 -Winit-self -Wmissing-format-attribute -Wmissing-include-dirs -Wnested-externs -Wold-style-definition -Wpointer-arith -Wredundant-decls -Wstrict-prototypes -Wunknown-pragmas -Wwrite-strings -Wno-missing-field-initializers -Wno-unused-parameter -Wno-cast-qual -Wno-shadow -Wno-unreachable-code -fno-exceptions
-libqpol/policy_extend.c: In function 'policy_extend':
-libqpol/policy_extend.c:161:27: error: '%04zd' directive output may be truncated writing between 4 and 10 bytes into a region of size 5 [-Werror=format-truncation=]
-    snprintf(buff, 9, "@ttr%04zd", i + 1);
-                           ^~~~~
-libqpol/policy_extend.c:161:22: note: directive argument in the range [1, 4294967295]
-    snprintf(buff, 9, "@ttr%04zd", i + 1);
-                      ^~~~~~~~~~~
-
-Increase the size of the buffer to avoid collisions
-
-Closes: https://github.com/TresysTechnology/setools/issues/174
-Signed-off-by: Laurent Bigonville <bigon@...>
----
- libqpol/policy_extend.c | 16 ++++++++--------
- 1 file changed, 8 insertions(+), 8 deletions(-)
-
-diff --git a/libqpol/policy_extend.c b/libqpol/policy_extend.c
-index 742819b..739e184 100644
---- a/libqpol/policy_extend.c
-+++ b/libqpol/policy_extend.c
-@@ -110,7 +110,7 @@ static int qpol_policy_remove_bogus_aliases(qpol_policy_t * policy)
-  *  Builds data for the attributes and inserts them into the policydb.
-  *  This function modifies the policydb. Names created for attributes
-  *  are of the form @ttr<value> where value is the value of the attribute
-- *  as a four digit number (prepended with 0's as needed).
-+ *  as a ten digit number (prepended with 0's as needed).
-  *  @param policy The policy from which to read the attribute map and
-  *  create the type data for the attributes. This policy will be altered
-  *  by this function.
-@@ -125,7 +125,7 @@ static int qpol_policy_build_attrs_from_map(qpol_policy_t * policy)
-       uint32_t bit = 0, count = 0;
-       ebitmap_node_t *node = NULL;
-       type_datum_t *tmp_type = NULL, *orig_type;
--      char *tmp_name = NULL, buff[10];
-+      char *tmp_name = NULL, buff[16];
-       int error = 0, retv;
-
-       INFO(policy, "%s", "Generating attributes for policy. (Step 4 of 5)");
-@@ -137,7 +137,7 @@ static int qpol_policy_build_attrs_from_map(qpol_policy_t * policy)
-
-       db = &policy->p->p;
-
--      memset(&buff, 0, 10 * sizeof(char));
-+      memset(&buff, 0, 16 * sizeof(char));
-
-       for (i = 0; i < db->p_types.nprim; i++) {
-               /* skip types */
-@@ -158,7 +158,7 @@ static int qpol_policy_build_attrs_from_map(qpol_policy_t * policy)
-                * with this attribute */
-               /* Does not exist */
-               if (db->p_type_val_to_name[i] == NULL){
--                      snprintf(buff, 9, "@ttr%04zd", i + 1);
-+                      snprintf(buff, 15, "@ttr%010zd", i + 1);
-                       tmp_name = strdup(buff);
-                       if (!tmp_name) {
-                               error = errno;
-@@ -240,7 +240,7 @@ static int qpol_policy_build_attrs_from_map(qpol_policy_t * policy)
-  *  Builds data for empty attributes and inserts them into the policydb.
-  *  This function modifies the policydb. Names created for the attributes
-  *  are of the form @ttr<value> where value is the value of the attribute
-- *  as a four digit number (prepended with 0's as needed).
-+ *  as a ten digit number (prepended with 0's as needed).
-  *  @param policy The policy to which to add type data for attributes.
-  *  This policy will be altered by this function.
-  *  @return Returns 0 on success and < 0 on failure; if the call fails,
-@@ -251,7 +251,7 @@ static int qpol_policy_build_attrs_from_map(qpol_policy_t * policy)
- static int qpol_policy_fill_attr_holes(qpol_policy_t * policy)
- {
-       policydb_t *db = NULL;
--      char *tmp_name = NULL, buff[10];
-+      char *tmp_name = NULL, buff[16];
-       int error = 0, retv = 0;
-       ebitmap_t tmp_bmap = { NULL, 0 };
-       type_datum_t *tmp_type = NULL;
-@@ -265,12 +265,12 @@ static int qpol_policy_fill_attr_holes(qpol_policy_t * policy)
-
-       db = &policy->p->p;
-
--      memset(&buff, 0, 10 * sizeof(char));
-+      memset(&buff, 0, 16 * sizeof(char));
-
-       for (i = 0; i < db->p_types.nprim; i++) {
-               if (db->type_val_to_struct[i])
-                       continue;
--              snprintf(buff, 9, "@ttr%04zd", i + 1);
-+              snprintf(buff, 15, "@ttr%010zd", i + 1);
-               tmp_name = strdup(buff);
-               if (!tmp_name) {
-                       error = errno;
---
-2.20.1
-
diff --git a/recipes-security/setools/setools/setools4-fix-cross-compiling-errors-for-powerpc-mips.patch b/recipes-security/setools/setools/setools4-fix-cross-compiling-errors-for-powerpc-mips.patch
deleted file mode 100644
index 9a6b818..0000000
--- a/recipes-security/setools/setools/setools4-fix-cross-compiling-errors-for-powerpc-mips.patch
+++ /dev/null
@@ -1,35 +0,0 @@
-From dc86d880ae0d66233679112a2bf0115c39df68f1 Mon Sep 17 00:00:00 2001
-From: Wenzong Fan <wenzong.fan@...>
-Date: Fri, 17 Feb 2017 08:57:35 +0000
-Subject: [meta-selinux][PATCH] setools4: fix cross-compiling errors for powerpc, mips
-
-Fix build errors:
-| libqpol/policy.c: In function 'qpol_binpol_version':
-| libqpol/policy.c:95:24: error: implicit declaration of function 'bswap_32' [-Werror=implicit-function-declaration]
-| #define le32_to_cpu(x) bswap_32(x)
-
-Upstream-Status: Pending
-
-Signed-off-by: Wenzong Fan <wenzong.fan@...>
----
- libqpol/policy.c | 4 ++++
- 1 file changed, 4 insertions(+)
-
-diff --git a/libqpol/policy.c b/libqpol/policy.c
-index ae3acb5..b5b87f9 100644
---- a/libqpol/policy.c
-+++ b/libqpol/policy.c
-@@ -45,6 +45,10 @@
- # include <asm/types.h>
- #endif
-
-+#if defined(_ARCH_PPC) || defined(mips)
-+#include <byteswap.h>
-+#endif
-+
- #include <sepol/debug.h>
- #include <sepol/handle.h>
- #include <sepol/policydb/flask_types.h>
---
-2.11.0
-
diff --git a/recipes-security/setools/setools/setools4-fixes-for-cross-compiling.patch b/recipes-security/setools/setools/setools4-fixes-for-cross-compiling.patch
index 5c43c49..51c7603 100644
--- a/recipes-security/setools/setools/setools4-fixes-for-cross-compiling.patch
+++ b/recipes-security/setools/setools/setools4-fixes-for-cross-compiling.patch
@@ -1,7 +1,7 @@
-From a104374147b398838edc04e937c92e762ea3f5d9 Mon Sep 17 00:00:00 2001
+From e0a112874d10dbf741c27b107edddedd86d51529 Mon Sep 17 00:00:00 2001
 From: Wenzong Fan <wenzong.fan@...>
 Date: Tue, 14 Feb 2017 06:32:35 +0000
-Subject: [meta-selinux][PATCH] setools4: fixes for cross compiling
+Subject: [PATCH] setools4: fixes for cross compiling

 * search libsepol from $STAGING_LIBDIR
 * fix manual install path as '/usr/share/man/man1'
@@ -9,32 +9,24 @@ Subject: [meta-selinux][PATCH] setools4: fixes for cross compiling
 Upstream-Status: Inappropriate [embedded specific]

 Signed-off-by: Wenzong Fan <wenzong.fan@...>
+Signed-off-by: Yi Zhao <yi.zhao@...>
 ---
- setup.py | 4 ++--
- 1 file changed, 2 insertions(+), 2 deletions(-)
+ setup.py | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)

 diff --git a/setup.py b/setup.py
-index 2ca44c9..300ff70 100644
+index ac8a876..df7ebbc 100644
 --- a/setup.py
 +++ b/setup.py
-@@ -77,7 +77,7 @@ class BuildExtCommand(build_ext):
-         build_ext.run(self)
+@@ -78,7 +78,7 @@ class QtHelpCommand(Command):


--base_lib_dirs = ['.', '/usr/lib64', '/usr/lib', '/usr/local/lib']
-+base_lib_dirs = [os.environ["STAGING_LIBDIR"]]
- include_dirs = ['libqpol', 'libqpol/include']
+ # Library linkage
+-lib_dirs = ['.', '/usr/lib64', '/usr/lib', '/usr/local/lib']
++lib_dirs = [os.environ["STAGING_LIBDIR"]]
+ include_dirs = []

- try:
-@@ -182,7 +182,7 @@ setup(name='setools',
-                 'build_qhc': QtHelpCommand},
-       packages=['setools', 'setools.diff', 'setools.policyrep', 'setoolsgui', 'setoolsgui.apol'],
-       scripts=['apol', 'sediff', 'seinfo', 'seinfoflow', 'sesearch', 'sedta'],
--      data_files=[(join(sys.prefix, 'share/man/man1'), glob.glob("man/*.1"))],
-+      data_files=[('/usr/share/man/man1', glob.glob("man/*.1"))],
-       package_data={'': ['*.ui', '*.qhc', '*.qch'], 'setools': ['perm_map']},
-       ext_modules=ext_py_mods,
-       test_suite='tests',
+ with suppress(KeyError):
 --
-2.13.0
+2.7.4

diff --git a/recipes-security/setools/setools_4.1.1.bb b/recipes-security/setools/setools_4.2.2.bb
similarity index 61%
rename from recipes-security/setools/setools_4.1.1.bb
rename to recipes-security/setools/setools_4.2.2.bb
index db529f4..6e5a950 100644
--- a/recipes-security/setools/setools_4.1.1.bb
+++ b/recipes-security/setools/setools_4.2.2.bb
@@ -9,26 +9,24 @@ SECTION = "base"
 LICENSE = "GPLv2 & LGPLv2.1"

 S = "${WORKDIR}/git"
-SRC_URI = "git://github.com/SELinuxProject/${BPN}.git;branch=4.1 \
+SRC_URI = "git://github.com/SELinuxProject/${BPN}.git;branch=4.2 \
            file://setools4-fixes-for-cross-compiling.patch \
-           file://setools4-fix-cross-compiling-errors-for-powerpc-mips.patch \
-           file://Fix-build-failure-with-GCC-7-due-to-possible-truncat.patch \
 "

-SRCREV = "e03617eb7ab5a035633bff66500b95d25232e331"
+SRCREV = "15bffa7823b9a999f9d51533785ade18fe44df08"

 LIC_FILES_CHKSUM = "file://${S}/COPYING;md5=83a5eb6974c11f30785e90d0eeccf40c \
                     file://${S}/COPYING.GPL;md5=b234ee4d69f5fce4486a80fdaf4a4263 \
                     file://${S}/COPYING.LGPL;md5=4fbd65380cdd255951079008b364516c"

-DEPENDS += "bison-native flex-native swig-native python libsepol"
+DEPENDS += "bison-native flex-native swig-native python3 python3-cython-native libsepol"

-RDEPENDS_${PN} += "python-networkx python-enum34 python-decorator python-setuptools \
-                   python-logging python-json python-argparse libselinux-python"
+RDEPENDS_${PN} += "python3-networkx python3-decorator python3-setuptools \
+                   python3-logging python3-json libselinux-python"

 RPROVIDES_${PN} += "${PN}-console"

-inherit setuptools
+inherit setuptools3

 do_install_append() {
        # Need PyQt5 support, disable gui tools
--
2.17.1

--

Joe MacDonald
:wq

Hi,

i'm new here and not really shure this is the right place for this issue.

I try to get apparmor working using yocto. I tried using thud and zeus with the same result on apparmor_status: apparmor module is not loaded
Checking the kernel flags manually with bitbake -c menuconfig virtual/kernel shows that none of the relevant flags like CONFIG_SECURITY_APPARMOR are set.
Setting these manually i see that the kernel parameters

"apparmor=1 security=apparmor" are missing.

My assumption was that adding the apparmor recipe would do all these things.

Do i add the recipe somewhere completely wrong?
Do i possibly override apparmor with other recipes?

Any help would be greatly appreciated.

Best regards

hu






Should these 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#47775): https://lists.yoctoproject.org/g/yocto/message/47775
Mute This Topic: https://lists.yoctoproject.org/mt/68834537/1024635
Mute #yocto: https://lists.yoctoproject.org/mk?hashtag=yocto&subid=6691541
Group Owner: yocto+owner@...
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub  [akuster@...]
-=-=-=-=-=-=-=-=-=-=-=-

Hi all,

My layer has a bunch of "development" recipes that contain something
like:

PV = "1.1.2+git${SRCPV}"
SRCREV = "${AUTOREV}"
DEFAULT_PREFERENCE = "-1"


In these cases, my recipes have a "known good" version that has a
specific known-good version number (recipe_1.1.2.bb), and then the
"recipe_git.bb" file that I only enable using PREFERRED_VERSION when
I'm wanting to use the latest source code.

My problem with this is that it seems to defeat BB_NO_NETWORK, and
I'm not sure how to handle it. I would like my layer to work
correctly with no network, but the mere presence of these development
recipes causes my build to fail during parsing if the network is not
available.

My last failed attempt was to create a "skip-git" class that I
inherit in local.conf with the following:

python () {
no_network = d.getVar('BB_NO_NETWORK')
pn = d.getVar('PN')
pv = d.getVar('PV')

if no_network and 'git' in pv:
raise bb.parse.SkipRecipe("Recipe requires git, but no
network: %s-%s" % (pn, pv))
}


Unfortunately, bitbake seems to fail in data_smart.py before my class
actually has a chance to skip the recipe even in this case.

Is there a better way to handle this?

Thanks,
Aaron
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
You automatically follow any topics you start or reply to.

View/Reply Online (#47773):
https://lists.yoctoproject.org/g/yocto/message/47773
Mute This Topic: https://lists.yoctoproject.org/mt/68832469/1997914
Group Owner: yocto+owner@...
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [raj.khem@
gmail.com]
-=-=-=-=-=-=-=-=-=-=-=-

Hi,

when enable readonly rootfs mc fails to start because cannot write to
directory: /home/root/.config/mc

I've created volatile-bindd to this directory (to be able to write
something) and extend mc using below bbappend:

do_install_append () {
install -d ${D}/home/root/.config/mc
}

FILES_${PN} += "/home/root/.config/mc/*"

But now I got QA issue:

ERROR: mc-4.8.22-r0 do_package: QA Issue: mc: Files/directories were
installed but not shipped in any package:
/home
/home/root
/home/root/.config
/home/root/.config/mc
Please set FILES such that these items are packaged. Alternatively if
they are unneeded, avoid installing them or delete them within
do_install.

Checked FILES_mc and path is there. Am I missing anything obvious? Thanks.

BR,

marek

--
as simple and primitive as possible
-------------------------------------------------
Marek Belisko - OPEN-NANDRA
Freelance Developer

Ruska Nova Ves 219 | Presov, 08005 Slovak Republic
Tel: +421 915 052 184
skype: marekwhite
twitter: #opennandra
web: http://open-nandra.com
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#47771): https://lists.yoctoproject.org/g/yocto/message/47771
Mute This Topic: https://lists.yoctoproject.org/mt/68830517/900721
Group Owner: yocto+owner@...
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [marek.belisko@...]
-=-=-=-=-=-=-=-=-=-=-=-

Do you have any idea on how to use these fetchers (or other SSH keys-
based secure fetcher) to share sstate cache ?

I think this is a "not implemented" problem.
If you look at bitbake/lib/bb/fetch2/ssh.py or sftp.py, they don't have
a checkstatus() method that local.py or wget.py has.
I think the sstate code relies on the checkstatus method working to
know know if an sstate artefact is present.
Performance of that function is key and creating a new ssh connection
to check for each sstate artefact may make the build prohibitively
slow. There are some connection caching mechanisms present which may
help with that.

Hi everyone,

Recently, I tried to use the SSTATE_MIRRORS variable to get back
sstate cache from a build machine shared amongst developers.
When I tried with the simple FILE fetcher, it worked great and
resumed the work where the build machine stopped.
But when I tried to replace it with SSH and/or SFTP fetchers (setup
to connect on my own machine), I had no errors or warnings, but the
steps started from scratch, ignoring the sstate cache mirror.

The thing is that we can't use unauthenticated HTTP/FTP fetchers for
corporate security reasons.
SSH and SFTP are convenients as we use crops and mount ssh keys
inside to authenticate ourselves on corporate git servers.

Environment:

I'm using Yocto 2.6 (Thud) and CROPS.
I've set up my machine as SSH/SFTP server, and I tried it manually,
it works: the connection is established (even in CROPS).

I defined the SSTATE_MIRRORS variable in the build/conf/local.conf
file.
Working: SSTATE_MIRRORS = "file://.*
file://{COMMON_PATH}/sstate-cache/PATH"
Not working: SSTATE_MIRRORS = "file://.*
ssh://user@my-machine-ip/{COMMON_PATH}/sstate-cache/PATH"
Notworking: SSTATE_MIRRORS = "file://.*
sftp://user@my-machine-ip/{COMMON_PATH}/sstate-cache/PATH"

Do you have any idea on how to use these fetchers (or other SSH keys-
based secure fetcher) to share sstate cache ?

On 12/18/19 2:46 AM, Richard Purdie wrote:

On Wed, 2019-12-18 at 03:32 +0000, Jain, Sangeeta wrote:

Hello all,

This is the full report for 3.1_M1.rc8:
https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/tree/?h=intel-yocto-testresults


======= Summary ========
No high milestone defects.
No new defects are found in this cycle.
glib and valgrind ptest failed (BUG id:13713).

Hi everyone,

Recently, I tried to use the SSTATE_MIRRORS variable to get back sstate cache from a build machine shared amongst developers.
When I tried with the simple FILE fetcher, it worked great and resumed the work where the build machine stopped.
But when I tried to replace it with SSH and/or SFTP fetchers (setup to connect on my own machine), I had no errors or warnings, but the steps started from scratch, ignoring the sstate cache mirror.

The thing is that we can't use unauthenticated HTTP/FTP fetchers for corporate security reasons.
SSH and SFTP are convenients as we use crops and mount ssh keys inside to authenticate ourselves on corporate git servers.

Environment:

I'm using Yocto 2.6 (Thud) and CROPS.
I've set up my machine as SSH/SFTP server, and I tried it manually, it works: the connection is established (even in CROPS).

I defined the SSTATE_MIRRORS variable in the build/conf/local.conf file.
Working:        SSTATE_MIRRORS = "file://.* file://{COMMON_PATH}/sstate-cache/PATH"
Not working:  SSTATE_MIRRORS = "file://.* ssh://user@my-machine-ip/{COMMON_PATH}/sstate-cache/PATH"
Notworking:  SSTATE_MIRRORS = "file://.* sftp://user@my-machine-ip/{COMMON_PATH}/sstate-cache/PATH"

Do you have any idea on how to use these fetchers (or other SSH keys-based secure fetcher) to share sstate cache ?

Don't hesitate to ask for more information/tests from me.
Regards,
Anthony