Date   

Re: [PATCH yocto-autobuilder-helper] Add a detailed README file for yocto-autobuilder-helper scripts

Armin Kuster
 



On 12/20/19 12:11 PM, Peiran wrote:
The new README file adds information to build a
target on a local machine, explains the scripts,
parameters, and configuration files in more detail,
and provides some notes on limitations and possible
improvements that can be made to the autobuilder-helper
scripts.

Signed-off-by: Peiran Hong <peiran1997@...>

Thank you very much for taking the time to documenting this.

- armin
 README.md | 317 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 317 insertions(+)
 create mode 100644 README.md

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..b50061d
--- /dev/null
+++ b/README.md
@@ -0,0 +1,317 @@
+## Directories
+
+- `git/trash`: Directory for storing successfully finished and not
+  published builds.
+
+- `git/mirror`: *Optional*, directory for storing repo mirrors. If
+  repos are present in this directory, `git config` will do a local
+  clone on the repo instead of fetching from remote.
+
+## Json File Containing Repository Data
+
+In order to build a target, user needs to create a json file to hold
+the data of the repos that are required by the target (e.g. poky,
+bitbake, meta-openembbeded, etc.). All available repo data can be
+found in the `repo-defaults` entry in the `config.json` file. User can
+then copy the desired repo data and create a new json file consisting
+of only the repo data. An example of this would be:
+
+```json
+{
+	"poky" : {
+		"url" : "git://git.yoctoproject.org/poky",
+		"branch" : "master",
+		"revision" : "HEAD",
+		"checkout-dirname" : ".",
+		"no-layer-add" : true,
+		"call-init" : true
+	},
+	"bitbake" : {
+		"url" : "git://git.openembedded.org/bitbake",
+		"branch" : "master",
+		"revision" : "HEAD",
+		"no-layer-add" : true
+	}
+}
+```
+
+In the subsequent sections, this file will be referred to as **`repos.json`**.
+
+## Entry Point Scripts and Usage
+
+### `scripts/prepare-shared-repos`
+
+This script will fetch repos specified by `repos.json` and cache them
+in a custom directory. The cached repos can then be reused for future
+builds.
+
+Args:
+
+1. `repojson`: The path to `repos.json` containing repository data
+   explained in the previous section.
+
+2. `sharedsrcdir`: The directory where the repos will be cached.
+
+3. `-p`, `--publish-dir`: *Optional*, where to publish artefacts
+   to. Fetched repos would be archived as tarball and stored in this
+   directory. This procedure will not happen if `publish-dir` is not
+   specified.
+
+### `scripts/shared-repo-unpack`
+
+This script first fetches repos specified by `repos.json` *as well as*
+specified in the `NEEDREPOS` entry of the target in `config.json` to
+the autobuilder working directory given by user, then calls another
+script `scripts/layer-config`.
+
+`scripts/layer-config` will source `oe-init-build-env` file and
+execute `bitbake-layers add-layer` to add necessary layers (the
+`call-init` entry in `repos.json` for each repository will determine
+if `oe-init-build-env` is sourced, the `no-layer-add` entry will
+determine if `bitbake add-layer` should be run on this repo). Note
+that if `NEEDREPOS` is not set for the specific build target,
+`NEEDREPOS` in target `defaults` is checked and added. refer to
+`getconfigvar()` function in `scripts/utils.py` for details.
+
+Args:
+
+1. `repojson`: Same as in `scripts/prepare-shared-repos`.
+
+2. `abworkdir`: The autobuilder working directory which will resemble
+   the look of `poky` or `openembbeded-core` repo with build
+   directory, `oe-init-build-env`, etc.
+
+3. `target`: The target name defined under the `overrides` entry in
+   `config.json`.
+
+4. `-c`, `--cache-dir`: *Optional*, path to the directory caching the
+   repos fetch by `scripts/prepare-shared-repos`. If this argument is
+   not set, fetched repos will be stored in the `repos` directory
+   inside `abworkdir`.
+
+5. `-p`, `--publish-dir`: *Optional*, same as in
+   `scripts/prepare-shared-repos`.
+
+### `scripts/run-config`
+
+This script is the command responsible for sourcing the
+`oe-init-build-env` file, preparing for and executing `bitbake` to
+start the build, and running any sanity checks and extra commands
+specified by the target.
+
+First, build history related tasks are done. Build history related
+entries in `config.json` are checked. Specifically, it checks if the
+`BUILDHISTORY` entry is set to `true` for the target, and if the
+`reponame:branchname` argument passed into `run-config` is present in
+the `BUILD_HISTORY_DIRECTPUSH` or the `BUILD_HISTORY_FORKPUSH` entry
+in `config.json`. Then `scripts/buildhistory-init` script is executed
+to fetch the build history repo specified in the `BUILD_HISTORY_REPO`
+entry in `config.json`. However, the default build history repo given
+in `config.json` requires permission to clone.
+
+Then for each `step` of the target, the following tasks are run in
+sequence:
+
+1. `bitbake-layers addlayer` is run for all the layers present in the
+   `ADDLAYER` entry of the target in `config.json`. Note that these
+   layers are different from the ones added in the previous step by
+   `scripts/layer-config`.
+
+2. `scripts/setup-config` generates an `auto.conf` under
+   `abworkdir/build/conf` and writes to it the extra configs specified
+   by the `extravars` entry of the target in `config.json`. It also
+   generates a `sdk-extra.conf` file and writes to it the items
+   specified by the `SDKEXTRAS` entry of the target in `config.json`.
+
+3. `bitbake` command is run on the `BBTARGET` specified in the current
+   step with `-k` option.
+
+4. `SANITYTARGETS` of the step is run.
+
+5. `EXTRACMDS` and `EXTRAPLAINCMDS` of the step is run.
+
+6. Remove all layers added in reverse order using `bitbake-layers
+   remove-layer`.
+
+After the build is finished, publish artefacts (copy generated images
+and sdks to the user-defined publish directory. Refer to
+`scripts/publish-artefacts`), collect build results (copy
+`build_directory/tmp/log/oeqa/testresults.json` to the user-defined
+results directory and also compare the current build with previous
+ones. `buildhistory` is being run, refer to `scripts/collect-results`)
+and send error report if and error occurred and the `SENDERRORS` entry
+for the target is set in `config.json` (`send-error-report` is being
+run, refer to `scripts/upload-error-reports`).
+
+Lastly, everything is cleaned up. If there is any error during the
+build or the build is published (i.e. the `--publish-dir` argument is
+specified), the build directory is renamed by appending `-renamed`
+after the original build directory name and the script
+exits. Otherwise, `janitor/clobberdir` script is run to try to move
+the build directory to the trash directory set in `config.json` by the
+`TRASH_DIR` entry.  `janitor/clobberdir` first checks if trash
+directory is present and is valid, then it checks if the trash
+directory is on the same file system with the build directory. If it
+is, move the build directory to trash directory, else delete it.
+
+Args:
+
+1. `target`: Same as in `scripts/shared-repo-unpack`
+
+2. `builddir`: The build directory created by the sourcing of
+   `oe-init-build-env`, **This directory is not the autobuilder
+   working `abworkdir`**, but rather the `build` directory inside it,
+   i.e. it is something like `abworkdir/build`.
+
+3. `branchname`: The branch the build is running on. However, **This
+   does not set the branch the build is running**, it is only used for
+   build history. To change the branch of the build, modify the
+   `branch` entry in `repos.json`.
+
+4. `reponame`: The repo the build is running on. Again, **This does
+   not set the repo the build is running**, it is only used for build
+   history.
+
+5. `-s`, `--sstateprefix`: *Defaults to empty*, the directory prefix
+   to publish sstate into.
+
+6. `-b`, `--buildappsrcrev`: *Defaults to empty*, a build appliance
+   SRCREV to use.
+
+7. `-p`, `--publish-dir`: *Optional*, sets the directory
+   `scripts/publish-artefacts` script will be copying to.
+
+8. `-r`, `--results-dir`: *Optional*, sets the directory
+   `scripts/collect-results` script will be copying to.
+
+9. `-u`, `--build-url`: *Optional*, sets the url to link back to this
+   build from the error report server.
+
+10. `--build-type`: *Defaults to `quick`*, can be either `quick` or
+    `full`. For `quick` type, toolchain tests are skipped.
+
+11. `-t`, `--test`: *Defaults to `false`*, if set to `true`, only
+    `scripts/setup-config` is executed and the commands are dry-run.
+
+12. `-q`, `--quietlogging`: *Defaults to `false`*, prevents the
+    flushing of `bitbake` stdout output.
+
+## Steps to build a target
+
+Suppose we are now in the `scripts` directory. We want to build the
+target `poky-tiny` and work in the directory `$HOME/workdir` and want
+to cache all the repos in the directory `$HOME/cache` for future use.
+
+1. Modify the `BASE_HOMEDIR` in `config.json` to a valid directory for the build to happen. Here we use `$HOME`
+
+```json
+{
+    "BASE_HOMEDIR" : "/home/usr"
+    ...
+}
+```
+
+2. Create required directory `git/trash`. Here we create it under `$HOME`:
+
+```bash
+$ mkdir -p $HOME/git/trash
+```
+
+3. Check the `NEEDREPOS` entry for corresponding build targets in
+   `config.json` and create the `repos.json` file according to the
+   [Json File Containing Repository
+   Data](##{Json-File-Containing-Repository-Data}) section above. In
+   this case `poky` and `bitbake` are needed, so the `repos.json` file
+   should look like:
+
+```json
+{
+	"poky" : {
+		"url" : "git://git.yoctoproject.org/poky",
+		"branch" : "",
+		"revision" : "HEAD",
+		"checkout-dirname" : ".",
+		"no-layer-add" : true,
+		"call-init" : true
+	},
+	"bitbake" : {
+		"url" : "git://git.openembedded.org/bitbake",
+		"branch" : "master",
+		"revision" : "HEAD",
+		"no-layer-add" : true
+	}
+}
+```
+
+4. Run `scripts/prepare-shared-repos` and point to the `repos.json`
+   file as well as the cache directory. In this case we want:
+
+```bash
+$ ./prepare-shared-repos $HOME/repos.json $HOME/cache
+```
+
+5. Run `scripts/shared-repo-unpack` and point to the `repos.json` file
+   as well as the autobuilder's working directory, `-c` or the
+   `--cache-dir` argument can be set since we have already fetched the
+   repos in the previous step:
+
+```bash
+$ ./shared-repo-unpack -c $HOME/cache $HOME/repos.json $HOME/workdir poky-tiny
+```
+
+Now the `$HOME/workdir` should be created and `oe-init-build-env` should be
+sourced, which creates a `build` directory inside `$HOME/workdir`.
+
+6. Run `scripts/run-config` with the target to build, the build
+   directory (**Not the autobuilder working directory**), the branch
+   and the repo we are building on ([**The last two arguments does not
+   affect build**](##`scripts/run-config`)).
+
+```bash
+$ ./run-config poky-tiny $HOME/workdir/build master poky
+```
+
+7. If the build finished without error and is not published, the build
+   directory should be rsynced to the `git/trash` directory. If the
+   build failed, the `build` directory will be renamed to
+   `build-renamed` and stay inside the working directory.
+
+## Notes
+
+1. Instead of asking user for the `config.json` file, all the entry
+   point scripts referred to the file through function
+   `util.loadconfig()`, which first finds out the absolute path of the
+   current executing script (i.e. `scripts/prepare-shared-repos`) and
+   then append `../config.json` to it. This is probably not a good
+   idea.
+
+2. All the copy of the repos are done using `rsync`, `git clone
+   --reference or git clone --shared` may be better.
+
+3. Using `rsync` also caused `prepare-shared-repos` and
+   `shared-repo-unpack` script to do `git clone` and overwrite repo
+   content every time they are being called, even when the repo
+   content is exactly the same. We should probably only be doing `git
+   clone` for new repos and do `git checkout` or `git pull` for
+   existing repos.
+
+4. After a failed or published `scripts/run-config`, the build
+   directory is simply appended with `-renamed` to avoid distracting
+   other builds, this is probably not a good idea (Maybe append random
+   string or build ID type of string after each build directory?).
+
+5. The path specified for `git clone` in function `fetchgitrepo()` in
+   `scripts/utils.py` append `/` directly and causes the path to have
+   two `/` if 'abworkdir' is passed in with a `/` at the end
+   (e.g. `/home/workdir/` instead of `/home/workdir`), this does not
+   have any effect on `git clone` but maybe an extra parsing/formating
+   of the path is better.
+
+6. Feeding an incorrect target name into `scripts/shared-repo-unpack`
+   and `scripts/run-config` will not result in an error. The scripts
+   will instead use configurations in the `defaults` target and
+   `scripts/run-config` will simply skip the bitbake command and exit
+   normally with an exit code 0.
+
+7. `checkvnc` script tries to call `-kill` only on the first display,
+   there should be more flexibility on this.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#47784): https://lists.yoctoproject.org/g/yocto/message/47784
Mute This Topic: https://lists.yoctoproject.org/mt/68855528/1024635
Group Owner: yocto+owner@...
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub  [akuster@...]
-=-=-=-=-=-=-=-=-=-=-=-


[PATCH yocto-autobuilder-helper] Add a detailed README file for yocto-autobuilder-helper scripts

Peiran
 

The new README file adds information to build a
target on a local machine, explains the scripts,
parameters, and configuration files in more detail,
and provides some notes on limitations and possible
improvements that can be made to the autobuilder-helper
scripts.

Signed-off-by: Peiran Hong <peiran1997@...>
---
README.md | 317 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 317 insertions(+)
create mode 100644 README.md

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..b50061d
--- /dev/null
+++ b/README.md
@@ -0,0 +1,317 @@
+## Directories
+
+- `git/trash`: Directory for storing successfully finished and not
+ published builds.
+
+- `git/mirror`: *Optional*, directory for storing repo mirrors. If
+ repos are present in this directory, `git config` will do a local
+ clone on the repo instead of fetching from remote.
+
+## Json File Containing Repository Data
+
+In order to build a target, user needs to create a json file to hold
+the data of the repos that are required by the target (e.g. poky,
+bitbake, meta-openembbeded, etc.). All available repo data can be
+found in the `repo-defaults` entry in the `config.json` file. User can
+then copy the desired repo data and create a new json file consisting
+of only the repo data. An example of this would be:
+
+```json
+{
+ "poky" : {
+ "url" : "git://git.yoctoproject.org/poky",
+ "branch" : "master",
+ "revision" : "HEAD",
+ "checkout-dirname" : ".",
+ "no-layer-add" : true,
+ "call-init" : true
+ },
+ "bitbake" : {
+ "url" : "git://git.openembedded.org/bitbake",
+ "branch" : "master",
+ "revision" : "HEAD",
+ "no-layer-add" : true
+ }
+}
+```
+
+In the subsequent sections, this file will be referred to as **`repos.json`**.
+
+## Entry Point Scripts and Usage
+
+### `scripts/prepare-shared-repos`
+
+This script will fetch repos specified by `repos.json` and cache them
+in a custom directory. The cached repos can then be reused for future
+builds.
+
+Args:
+
+1. `repojson`: The path to `repos.json` containing repository data
+ explained in the previous section.
+
+2. `sharedsrcdir`: The directory where the repos will be cached.
+
+3. `-p`, `--publish-dir`: *Optional*, where to publish artefacts
+ to. Fetched repos would be archived as tarball and stored in this
+ directory. This procedure will not happen if `publish-dir` is not
+ specified.
+
+### `scripts/shared-repo-unpack`
+
+This script first fetches repos specified by `repos.json` *as well as*
+specified in the `NEEDREPOS` entry of the target in `config.json` to
+the autobuilder working directory given by user, then calls another
+script `scripts/layer-config`.
+
+`scripts/layer-config` will source `oe-init-build-env` file and
+execute `bitbake-layers add-layer` to add necessary layers (the
+`call-init` entry in `repos.json` for each repository will determine
+if `oe-init-build-env` is sourced, the `no-layer-add` entry will
+determine if `bitbake add-layer` should be run on this repo). Note
+that if `NEEDREPOS` is not set for the specific build target,
+`NEEDREPOS` in target `defaults` is checked and added. refer to
+`getconfigvar()` function in `scripts/utils.py` for details.
+
+Args:
+
+1. `repojson`: Same as in `scripts/prepare-shared-repos`.
+
+2. `abworkdir`: The autobuilder working directory which will resemble
+ the look of `poky` or `openembbeded-core` repo with build
+ directory, `oe-init-build-env`, etc.
+
+3. `target`: The target name defined under the `overrides` entry in
+ `config.json`.
+
+4. `-c`, `--cache-dir`: *Optional*, path to the directory caching the
+ repos fetch by `scripts/prepare-shared-repos`. If this argument is
+ not set, fetched repos will be stored in the `repos` directory
+ inside `abworkdir`.
+
+5. `-p`, `--publish-dir`: *Optional*, same as in
+ `scripts/prepare-shared-repos`.
+
+### `scripts/run-config`
+
+This script is the command responsible for sourcing the
+`oe-init-build-env` file, preparing for and executing `bitbake` to
+start the build, and running any sanity checks and extra commands
+specified by the target.
+
+First, build history related tasks are done. Build history related
+entries in `config.json` are checked. Specifically, it checks if the
+`BUILDHISTORY` entry is set to `true` for the target, and if the
+`reponame:branchname` argument passed into `run-config` is present in
+the `BUILD_HISTORY_DIRECTPUSH` or the `BUILD_HISTORY_FORKPUSH` entry
+in `config.json`. Then `scripts/buildhistory-init` script is executed
+to fetch the build history repo specified in the `BUILD_HISTORY_REPO`
+entry in `config.json`. However, the default build history repo given
+in `config.json` requires permission to clone.
+
+Then for each `step` of the target, the following tasks are run in
+sequence:
+
+1. `bitbake-layers addlayer` is run for all the layers present in the
+ `ADDLAYER` entry of the target in `config.json`. Note that these
+ layers are different from the ones added in the previous step by
+ `scripts/layer-config`.
+
+2. `scripts/setup-config` generates an `auto.conf` under
+ `abworkdir/build/conf` and writes to it the extra configs specified
+ by the `extravars` entry of the target in `config.json`. It also
+ generates a `sdk-extra.conf` file and writes to it the items
+ specified by the `SDKEXTRAS` entry of the target in `config.json`.
+
+3. `bitbake` command is run on the `BBTARGET` specified in the current
+ step with `-k` option.
+
+4. `SANITYTARGETS` of the step is run.
+
+5. `EXTRACMDS` and `EXTRAPLAINCMDS` of the step is run.
+
+6. Remove all layers added in reverse order using `bitbake-layers
+ remove-layer`.
+
+After the build is finished, publish artefacts (copy generated images
+and sdks to the user-defined publish directory. Refer to
+`scripts/publish-artefacts`), collect build results (copy
+`build_directory/tmp/log/oeqa/testresults.json` to the user-defined
+results directory and also compare the current build with previous
+ones. `buildhistory` is being run, refer to `scripts/collect-results`)
+and send error report if and error occurred and the `SENDERRORS` entry
+for the target is set in `config.json` (`send-error-report` is being
+run, refer to `scripts/upload-error-reports`).
+
+Lastly, everything is cleaned up. If there is any error during the
+build or the build is published (i.e. the `--publish-dir` argument is
+specified), the build directory is renamed by appending `-renamed`
+after the original build directory name and the script
+exits. Otherwise, `janitor/clobberdir` script is run to try to move
+the build directory to the trash directory set in `config.json` by the
+`TRASH_DIR` entry. `janitor/clobberdir` first checks if trash
+directory is present and is valid, then it checks if the trash
+directory is on the same file system with the build directory. If it
+is, move the build directory to trash directory, else delete it.
+
+Args:
+
+1. `target`: Same as in `scripts/shared-repo-unpack`
+
+2. `builddir`: The build directory created by the sourcing of
+ `oe-init-build-env`, **This directory is not the autobuilder
+ working `abworkdir`**, but rather the `build` directory inside it,
+ i.e. it is something like `abworkdir/build`.
+
+3. `branchname`: The branch the build is running on. However, **This
+ does not set the branch the build is running**, it is only used for
+ build history. To change the branch of the build, modify the
+ `branch` entry in `repos.json`.
+
+4. `reponame`: The repo the build is running on. Again, **This does
+ not set the repo the build is running**, it is only used for build
+ history.
+
+5. `-s`, `--sstateprefix`: *Defaults to empty*, the directory prefix
+ to publish sstate into.
+
+6. `-b`, `--buildappsrcrev`: *Defaults to empty*, a build appliance
+ SRCREV to use.
+
+7. `-p`, `--publish-dir`: *Optional*, sets the directory
+ `scripts/publish-artefacts` script will be copying to.
+
+8. `-r`, `--results-dir`: *Optional*, sets the directory
+ `scripts/collect-results` script will be copying to.
+
+9. `-u`, `--build-url`: *Optional*, sets the url to link back to this
+ build from the error report server.
+
+10. `--build-type`: *Defaults to `quick`*, can be either `quick` or
+ `full`. For `quick` type, toolchain tests are skipped.
+
+11. `-t`, `--test`: *Defaults to `false`*, if set to `true`, only
+ `scripts/setup-config` is executed and the commands are dry-run.
+
+12. `-q`, `--quietlogging`: *Defaults to `false`*, prevents the
+ flushing of `bitbake` stdout output.
+
+## Steps to build a target
+
+Suppose we are now in the `scripts` directory. We want to build the
+target `poky-tiny` and work in the directory `$HOME/workdir` and want
+to cache all the repos in the directory `$HOME/cache` for future use.
+
+1. Modify the `BASE_HOMEDIR` in `config.json` to a valid directory for the build to happen. Here we use `$HOME`
+
+```json
+{
+ "BASE_HOMEDIR" : "/home/usr"
+ ...
+}
+```
+
+2. Create required directory `git/trash`. Here we create it under `$HOME`:
+
+```bash
+$ mkdir -p $HOME/git/trash
+```
+
+3. Check the `NEEDREPOS` entry for corresponding build targets in
+ `config.json` and create the `repos.json` file according to the
+ [Json File Containing Repository
+ Data](##{Json-File-Containing-Repository-Data}) section above. In
+ this case `poky` and `bitbake` are needed, so the `repos.json` file
+ should look like:
+
+```json
+{
+ "poky" : {
+ "url" : "git://git.yoctoproject.org/poky",
+ "branch" : "",
+ "revision" : "HEAD",
+ "checkout-dirname" : ".",
+ "no-layer-add" : true,
+ "call-init" : true
+ },
+ "bitbake" : {
+ "url" : "git://git.openembedded.org/bitbake",
+ "branch" : "master",
+ "revision" : "HEAD",
+ "no-layer-add" : true
+ }
+}
+```
+
+4. Run `scripts/prepare-shared-repos` and point to the `repos.json`
+ file as well as the cache directory. In this case we want:
+
+```bash
+$ ./prepare-shared-repos $HOME/repos.json $HOME/cache
+```
+
+5. Run `scripts/shared-repo-unpack` and point to the `repos.json` file
+ as well as the autobuilder's working directory, `-c` or the
+ `--cache-dir` argument can be set since we have already fetched the
+ repos in the previous step:
+
+```bash
+$ ./shared-repo-unpack -c $HOME/cache $HOME/repos.json $HOME/workdir poky-tiny
+```
+
+Now the `$HOME/workdir` should be created and `oe-init-build-env` should be
+sourced, which creates a `build` directory inside `$HOME/workdir`.
+
+6. Run `scripts/run-config` with the target to build, the build
+ directory (**Not the autobuilder working directory**), the branch
+ and the repo we are building on ([**The last two arguments does not
+ affect build**](##`scripts/run-config`)).
+
+```bash
+$ ./run-config poky-tiny $HOME/workdir/build master poky
+```
+
+7. If the build finished without error and is not published, the build
+ directory should be rsynced to the `git/trash` directory. If the
+ build failed, the `build` directory will be renamed to
+ `build-renamed` and stay inside the working directory.
+
+## Notes
+
+1. Instead of asking user for the `config.json` file, all the entry
+ point scripts referred to the file through function
+ `util.loadconfig()`, which first finds out the absolute path of the
+ current executing script (i.e. `scripts/prepare-shared-repos`) and
+ then append `../config.json` to it. This is probably not a good
+ idea.
+
+2. All the copy of the repos are done using `rsync`, `git clone
+ --reference or git clone --shared` may be better.
+
+3. Using `rsync` also caused `prepare-shared-repos` and
+ `shared-repo-unpack` script to do `git clone` and overwrite repo
+ content every time they are being called, even when the repo
+ content is exactly the same. We should probably only be doing `git
+ clone` for new repos and do `git checkout` or `git pull` for
+ existing repos.
+
+4. After a failed or published `scripts/run-config`, the build
+ directory is simply appended with `-renamed` to avoid distracting
+ other builds, this is probably not a good idea (Maybe append random
+ string or build ID type of string after each build directory?).
+
+5. The path specified for `git clone` in function `fetchgitrepo()` in
+ `scripts/utils.py` append `/` directly and causes the path to have
+ two `/` if 'abworkdir' is passed in with a `/` at the end
+ (e.g. `/home/workdir/` instead of `/home/workdir`), this does not
+ have any effect on `git clone` but maybe an extra parsing/formating
+ of the path is better.
+
+6. Feeding an incorrect target name into `scripts/shared-repo-unpack`
+ and `scripts/run-config` will not result in an error. The scripts
+ will instead use configurations in the `defaults` target and
+ `scripts/run-config` will simply skip the bitbake command and exit
+ normally with an exit code 0.
+
+7. `checkvnc` script tries to call `-kill` only on the first display,
+ there should be more flexibility on this.
--
2.24.0


Re: SSTATE_MIRRORS with ssh/sftp fetchers #yocto

angeal1105@...
 

Thank you both very much for your quick answers !

I will definitely give a shot at Ross' contrib. I also thought about mounting a sshFS when the container starts and trick Yocto by using the FILE protocol.
I will compare the performances.

Thanks again !


Re: [meta-selinux][PATCH 18/19] setools: upgrade 4.1.1 -> 4.2.2

Yi Zhao
 


On 12/20/19 1:32 AM, Joe MacDonald wrote:
Hi Yi,

I've merged the rest of this series, but this one fails to apply.  It looks like your tree didn't contain:

commit 5fd3c5b71edb99659aeb5cb5903088d84517382e (relabel, master)
Author: Christophe PRIOUZEAU <christophe.priouzeau@...>
Date:   Tue Nov 5 14:47:09 2019 +0000

    autorelabel: only selinux-autorelabel need autorelabel file

    With previous implementation, several packages provided
     .autorelabel file while only selinux-autorelabel manage it.
    If there is several packages which try to install .autorelabel
    file, an issue occur during installation of packagegroup-core-selinux.

    Signed-off-by: Christophe Priouzeau <christophe.priouzeau@...>
    Signed-off-by: Joe MacDonald <joe_macdonald@...>

Can you take a look at it and verify which parts of the change are still necessary given the above change?


Sure. I will send V2.


//Yi


Thanks.
-J.

On Wed, Nov 13, 2019 at 8:50 PM Yi Zhao <yi.zhao@...> wrote:
* Switch to python3

* Drop patches:
  Fix-build-failure-with-GCC-7-due-to-possible-truncat.patch
  setools4-fix-cross-compiling-errors-for-powerpc-mips.patch

Signed-off-by: Yi Zhao <yi.zhao@...>
---
 ...e-with-GCC-7-due-to-possible-truncat.patch | 105 ------------------
 ...ss-compiling-errors-for-powerpc-mips.patch |  35 ------
 .../setools4-fixes-for-cross-compiling.patch  |  34 +++---
 .../{setools_4.1.1.bb => setools_4.2.2.bb}    |  14 +--
 4 files changed, 19 insertions(+), 169 deletions(-)
 delete mode 100644 recipes-security/setools/setools/Fix-build-failure-with-GCC-7-due-to-possible-truncat.patch
 delete mode 100644 recipes-security/setools/setools/setools4-fix-cross-compiling-errors-for-powerpc-mips.patch
 rename recipes-security/setools/{setools_4.1.1.bb => setools_4.2.2.bb} (61%)

diff --git a/recipes-security/setools/setools/Fix-build-failure-with-GCC-7-due-to-possible-truncat.patch b/recipes-security/setools/setools/Fix-build-failure-with-GCC-7-due-to-possible-truncat.patch
deleted file mode 100644
index a5af041..0000000
--- a/recipes-security/setools/setools/Fix-build-failure-with-GCC-7-due-to-possible-truncat.patch
+++ /dev/null
@@ -1,105 +0,0 @@
-Upstream-Status: Backport [https://github.com/TresysTechnology/setools/commit/e41adf0]
-
-Signed-off-by: Kai Kang <kai.kang@...>
-
-From e41adf01647c695b80b112b337e76021bb9f30c3 Mon Sep 17 00:00:00 2001
-From: Laurent Bigonville <bigon@...>
-Date: Tue, 26 Sep 2017 15:15:30 +0200
-Subject: [PATCH] Fix build failure with GCC 7 due to possible truncation of
- snprintf output
-
-setools fails to build under GCC7 -Wformat -Werror with the following error:
-
-x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -Wno-sign-compare -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Ilibqpol -Ilibqpol/include -I/usr/include/python3.6m -c libqpol/policy_extend.c -o build/temp.linux-amd64-3.6/libqpol/policy_extend.o -Werror -Wextra -Waggregate-return -Wfloat-equal -Wformat -Wformat=2 -Winit-self -Wmissing-format-attribute -Wmissing-include-dirs -Wnested-externs -Wold-style-definition -Wpointer-arith -Wredundant-decls -Wstrict-prototypes -Wunknown-pragmas -Wwrite-strings -Wno-missing-field-initializers -Wno-unused-parameter -Wno-cast-qual -Wno-shadow -Wno-unreachable-code -fno-exceptions
-libqpol/policy_extend.c: In function 'policy_extend':
-libqpol/policy_extend.c:161:27: error: '%04zd' directive output may be truncated writing between 4 and 10 bytes into a region of size 5 [-Werror=format-truncation=]
-    snprintf(buff, 9, "@ttr%04zd", i + 1);
-                           ^~~~~
-libqpol/policy_extend.c:161:22: note: directive argument in the range [1, 4294967295]
-    snprintf(buff, 9, "@ttr%04zd", i + 1);
-                      ^~~~~~~~~~~
-
-Increase the size of the buffer to avoid collisions
-
-Closes: https://github.com/TresysTechnology/setools/issues/174
-Signed-off-by: Laurent Bigonville <bigon@...>
----
- libqpol/policy_extend.c | 16 ++++++++--------
- 1 file changed, 8 insertions(+), 8 deletions(-)
-
-diff --git a/libqpol/policy_extend.c b/libqpol/policy_extend.c
-index 742819b..739e184 100644
---- a/libqpol/policy_extend.c
-+++ b/libqpol/policy_extend.c
-@@ -110,7 +110,7 @@ static int qpol_policy_remove_bogus_aliases(qpol_policy_t * policy)
-  *  Builds data for the attributes and inserts them into the policydb.
-  *  This function modifies the policydb. Names created for attributes
-  *  are of the form @ttr<value> where value is the value of the attribute
-- *  as a four digit number (prepended with 0's as needed).
-+ *  as a ten digit number (prepended with 0's as needed).
-  *  @param policy The policy from which to read the attribute map and
-  *  create the type data for the attributes. This policy will be altered
-  *  by this function.
-@@ -125,7 +125,7 @@ static int qpol_policy_build_attrs_from_map(qpol_policy_t * policy)
-       uint32_t bit = 0, count = 0;
-       ebitmap_node_t *node = NULL;
-       type_datum_t *tmp_type = NULL, *orig_type;
--      char *tmp_name = NULL, buff[10];
-+      char *tmp_name = NULL, buff[16];
-       int error = 0, retv;
-
-       INFO(policy, "%s", "Generating attributes for policy. (Step 4 of 5)");
-@@ -137,7 +137,7 @@ static int qpol_policy_build_attrs_from_map(qpol_policy_t * policy)
-
-       db = &policy->p->p;
-
--      memset(&buff, 0, 10 * sizeof(char));
-+      memset(&buff, 0, 16 * sizeof(char));
-
-       for (i = 0; i < db->p_types.nprim; i++) {
-               /* skip types */
-@@ -158,7 +158,7 @@ static int qpol_policy_build_attrs_from_map(qpol_policy_t * policy)
-                * with this attribute */
-               /* Does not exist */
-               if (db->p_type_val_to_name[i] == NULL){
--                      snprintf(buff, 9, "@ttr%04zd", i + 1);
-+                      snprintf(buff, 15, "@ttr%010zd", i + 1);
-                       tmp_name = strdup(buff);
-                       if (!tmp_name) {
-                               error = errno;
-@@ -240,7 +240,7 @@ static int qpol_policy_build_attrs_from_map(qpol_policy_t * policy)
-  *  Builds data for empty attributes and inserts them into the policydb.
-  *  This function modifies the policydb. Names created for the attributes
-  *  are of the form @ttr<value> where value is the value of the attribute
-- *  as a four digit number (prepended with 0's as needed).
-+ *  as a ten digit number (prepended with 0's as needed).
-  *  @param policy The policy to which to add type data for attributes.
-  *  This policy will be altered by this function.
-  *  @return Returns 0 on success and < 0 on failure; if the call fails,
-@@ -251,7 +251,7 @@ static int qpol_policy_build_attrs_from_map(qpol_policy_t * policy)
- static int qpol_policy_fill_attr_holes(qpol_policy_t * policy)
- {
-       policydb_t *db = NULL;
--      char *tmp_name = NULL, buff[10];
-+      char *tmp_name = NULL, buff[16];
-       int error = 0, retv = 0;
-       ebitmap_t tmp_bmap = { NULL, 0 };
-       type_datum_t *tmp_type = NULL;
-@@ -265,12 +265,12 @@ static int qpol_policy_fill_attr_holes(qpol_policy_t * policy)
-
-       db = &policy->p->p;
-
--      memset(&buff, 0, 10 * sizeof(char));
-+      memset(&buff, 0, 16 * sizeof(char));
-
-       for (i = 0; i < db->p_types.nprim; i++) {
-               if (db->type_val_to_struct[i])
-                       continue;
--              snprintf(buff, 9, "@ttr%04zd", i + 1);
-+              snprintf(buff, 15, "@ttr%010zd", i + 1);
-               tmp_name = strdup(buff);
-               if (!tmp_name) {
-                       error = errno;
---
-2.20.1
-
diff --git a/recipes-security/setools/setools/setools4-fix-cross-compiling-errors-for-powerpc-mips.patch b/recipes-security/setools/setools/setools4-fix-cross-compiling-errors-for-powerpc-mips.patch
deleted file mode 100644
index 9a6b818..0000000
--- a/recipes-security/setools/setools/setools4-fix-cross-compiling-errors-for-powerpc-mips.patch
+++ /dev/null
@@ -1,35 +0,0 @@
-From dc86d880ae0d66233679112a2bf0115c39df68f1 Mon Sep 17 00:00:00 2001
-From: Wenzong Fan <wenzong.fan@...>
-Date: Fri, 17 Feb 2017 08:57:35 +0000
-Subject: [meta-selinux][PATCH] setools4: fix cross-compiling errors for powerpc, mips
-
-Fix build errors:
-| libqpol/policy.c: In function 'qpol_binpol_version':
-| libqpol/policy.c:95:24: error: implicit declaration of function 'bswap_32' [-Werror=implicit-function-declaration]
-| #define le32_to_cpu(x) bswap_32(x)
-
-Upstream-Status: Pending
-
-Signed-off-by: Wenzong Fan <wenzong.fan@...>
----
- libqpol/policy.c | 4 ++++
- 1 file changed, 4 insertions(+)
-
-diff --git a/libqpol/policy.c b/libqpol/policy.c
-index ae3acb5..b5b87f9 100644
---- a/libqpol/policy.c
-+++ b/libqpol/policy.c
-@@ -45,6 +45,10 @@
- # include <asm/types.h>
- #endif
-
-+#if defined(_ARCH_PPC) || defined(mips)
-+#include <byteswap.h>
-+#endif
-+
- #include <sepol/debug.h>
- #include <sepol/handle.h>
- #include <sepol/policydb/flask_types.h>
---
-2.11.0
-
diff --git a/recipes-security/setools/setools/setools4-fixes-for-cross-compiling.patch b/recipes-security/setools/setools/setools4-fixes-for-cross-compiling.patch
index 5c43c49..51c7603 100644
--- a/recipes-security/setools/setools/setools4-fixes-for-cross-compiling.patch
+++ b/recipes-security/setools/setools/setools4-fixes-for-cross-compiling.patch
@@ -1,7 +1,7 @@
-From a104374147b398838edc04e937c92e762ea3f5d9 Mon Sep 17 00:00:00 2001
+From e0a112874d10dbf741c27b107edddedd86d51529 Mon Sep 17 00:00:00 2001
 From: Wenzong Fan <wenzong.fan@...>
 Date: Tue, 14 Feb 2017 06:32:35 +0000
-Subject: [meta-selinux][PATCH] setools4: fixes for cross compiling
+Subject: [PATCH] setools4: fixes for cross compiling

 * search libsepol from $STAGING_LIBDIR
 * fix manual install path as '/usr/share/man/man1'
@@ -9,32 +9,24 @@ Subject: [meta-selinux][PATCH] setools4: fixes for cross compiling
 Upstream-Status: Inappropriate [embedded specific]

 Signed-off-by: Wenzong Fan <wenzong.fan@...>
+Signed-off-by: Yi Zhao <yi.zhao@...>
 ---
- setup.py | 4 ++--
- 1 file changed, 2 insertions(+), 2 deletions(-)
+ setup.py | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)

 diff --git a/setup.py b/setup.py
-index 2ca44c9..300ff70 100644
+index ac8a876..df7ebbc 100644
 --- a/setup.py
 +++ b/setup.py
-@@ -77,7 +77,7 @@ class BuildExtCommand(build_ext):
-         build_ext.run(self)
+@@ -78,7 +78,7 @@ class QtHelpCommand(Command):


--base_lib_dirs = ['.', '/usr/lib64', '/usr/lib', '/usr/local/lib']
-+base_lib_dirs = [os.environ["STAGING_LIBDIR"]]
- include_dirs = ['libqpol', 'libqpol/include']
+ # Library linkage
+-lib_dirs = ['.', '/usr/lib64', '/usr/lib', '/usr/local/lib']
++lib_dirs = [os.environ["STAGING_LIBDIR"]]
+ include_dirs = []

- try:
-@@ -182,7 +182,7 @@ setup(name='setools',
-                 'build_qhc': QtHelpCommand},
-       packages=['setools', 'setools.diff', 'setools.policyrep', 'setoolsgui', 'setoolsgui.apol'],
-       scripts=['apol', 'sediff', 'seinfo', 'seinfoflow', 'sesearch', 'sedta'],
--      data_files=[(join(sys.prefix, 'share/man/man1'), glob.glob("man/*.1"))],
-+      data_files=[('/usr/share/man/man1', glob.glob("man/*.1"))],
-       package_data={'': ['*.ui', '*.qhc', '*.qch'], 'setools': ['perm_map']},
-       ext_modules=ext_py_mods,
-       test_suite='tests',
+ with suppress(KeyError):
 --
-2.13.0
+2.7.4

diff --git a/recipes-security/setools/setools_4.1.1.bb b/recipes-security/setools/setools_4.2.2.bb
similarity index 61%
rename from recipes-security/setools/setools_4.1.1.bb
rename to recipes-security/setools/setools_4.2.2.bb
index db529f4..6e5a950 100644
--- a/recipes-security/setools/setools_4.1.1.bb
+++ b/recipes-security/setools/setools_4.2.2.bb
@@ -9,26 +9,24 @@ SECTION = "base"
 LICENSE = "GPLv2 & LGPLv2.1"

 S = "${WORKDIR}/git"
-SRC_URI = "git://github.com/SELinuxProject/${BPN}.git;branch=4.1 \
+SRC_URI = "git://github.com/SELinuxProject/${BPN}.git;branch=4.2 \
            file://setools4-fixes-for-cross-compiling.patch \
-           file://setools4-fix-cross-compiling-errors-for-powerpc-mips.patch \
-           file://Fix-build-failure-with-GCC-7-due-to-possible-truncat.patch \
 "

-SRCREV = "e03617eb7ab5a035633bff66500b95d25232e331"
+SRCREV = "15bffa7823b9a999f9d51533785ade18fe44df08"

 LIC_FILES_CHKSUM = "file://${S}/COPYING;md5=83a5eb6974c11f30785e90d0eeccf40c \
                     file://${S}/COPYING.GPL;md5=b234ee4d69f5fce4486a80fdaf4a4263 \
                     file://${S}/COPYING.LGPL;md5=4fbd65380cdd255951079008b364516c"

-DEPENDS += "bison-native flex-native swig-native python libsepol"
+DEPENDS += "bison-native flex-native swig-native python3 python3-cython-native libsepol"

-RDEPENDS_${PN} += "python-networkx python-enum34 python-decorator python-setuptools \
-                   python-logging python-json python-argparse libselinux-python"
+RDEPENDS_${PN} += "python3-networkx python3-decorator python3-setuptools \
+                   python3-logging python3-json libselinux-python"

 RPROVIDES_${PN} += "${PN}-console"

-inherit setuptools
+inherit setuptools3

 do_install_append() {
        # Need PyQt5 support, disable gui tools
--
2.17.1



--
Joe MacDonald
:wq


[ANNOUNCEMENT] Milestone 1 for Yocto Project 3.1 (yocto-3.1_M1) now available

Vineela
 

Hello,

We are pleased to announce the first milestone release for Yocto Project 3.1 (yocto-3.1_M1) is now available for download.

Download:

http://downloads.yoctoproject.org/releases/yocto/milestones/yocto-3.1_M1

bitbake: 99d46107ccfcec576238d32cfe7903440857038d
meta-gplv2: a67a5fd160f97129a6afd65f107cd2cbdfec41e7
meta-intel: e1b373f3cbb9acd71efe84d782675025d59b6035
meta-mingw: 126efdad243e1a3bead5b695df6656e94353dd46
oecore: 0f04e81c797d5d337ece4e8638a6b71c75bc0a00
poky: 1abffc542a0571f0d1512b92c1a59d138cf3ea6a

Full Test Report:

http://downloads.yoctoproject.org/releases/yocto/milestones/yocto-3.1_M1/testreport.txt

Thank you.
Vineela Tummalapalli
vineela.tummalapalli@...
Yocto Project Build and Release


Re: Raspberry pi 4 recipe and layer issues.

Ed Vidal
 

Hi All,

This is the error, that I was getting with meta-yosys-tools/recipes-yosys/yosys/yosys_0.1.bb,
found at github.com/develone.

The strip command fails, both on raspberrypi4-64 target and during the "bitbake yosys" build.
During the do_install which returns an error.
strip: Unable to recognise the format of the input file `yosys'

This above error was happening during do_install.
Made some chgs to the Makefile 
chg'ed prefix from /usr/local to /opt
added a variable 
YOCTO := "--input-target=elf64-littleaarch64"
added $(YOCTO) to lines 763, 765, and 769 of yosys Makefile

Now, my build for the rpi4-64 gets to the do_package step.
NOTE: Skipping file /opt/bin/yosys-abc from yosys for already-stripped QA test
NOTE: Skipping file /opt/bin/yosys-filterlib from yosys for already-stripped QA test
ERROR: Fatal errors occurred in subprocesses:

ls tmp/work/aarch64-poky-linux/yosys/0.1+gitAUTOINC+d406f2ffd7-r0/image/opt/
bin  share

Where the desired, files are located, in the /opt/bin & /opt/share/yosys folders.

I tested these files on rpi4-64 target.  I am getting an error on rpi4-64 target.
/opt/bin/yosys: cannot execute binary file: Exec format error.

All help is appreciated.
Best Regards

 


Edward Vidal Jr. e-mail develone@... 915-595-1613


Re: apparmor setup #yocto

Armin Kuster
 



On 12/19/19 8:36 AM, hu.schlieben@... wrote:
Hi,

i'm new here and not really shure this is the right place for this issue.

First you need to enable the distro feature

DISTRO_FEATURES_append = " apparmor"

otherwise the kernel fragments wont be included
I try to get apparmor working using yocto. I tried using thud and zeus with the same result on apparmor_status: apparmor module is not loaded
Checking the kernel flags manually with bitbake -c menuconfig virtual/kernel shows that none of the relevant flags like CONFIG_SECURITY_APPARMOR are set.
Setting these manually i see that the kernel parameters
These you need to append to your kernel commend line. I think it is:
KERNEL_CMDLINE_APPEND_append =

"apparmor=1 security=apparmor" are missing.

My assumption was that adding the apparmor recipe would do all these things.
Actually the recipe should have failed with a DISTRO check message.

I should add that.

This should get you on your way.
- armin




Do i add the recipe somewhere completely wrong?
Do i possibly override apparmor with other recipes?

Any help would be greatly appreciated.

Best regards

hu






Should these 











-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#47775): https://lists.yoctoproject.org/g/yocto/message/47775
Mute This Topic: https://lists.yoctoproject.org/mt/68834537/1024635
Mute #yocto: https://lists.yoctoproject.org/mk?hashtag=yocto&subid=6691541
Group Owner: yocto+owner@...
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub  [akuster@...]
-=-=-=-=-=-=-=-=-=-=-=-


[layerindex-web][PATCH] requirements.txt: bump Django version to fix CVE-2019-19844

Paul Eggleton
 

Fixes a vulnerability in the password reset process due to
insufficiently stringent validation of unicode email addresses.

https://www.djangoproject.com/weblog/2019/dec/18/security-releases/
https://nvd.nist.gov/vuln/detail/CVE-2019-19844

(The existing version specification would have selected the fixed
version of Django already for new installs, but bumping the minimum
ensures that it will be installed for upgrades with
./dockersetup.py -u as well.)

Signed-off-by: Paul Eggleton <paul.eggleton@...>
---
requirements.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/requirements.txt b/requirements.txt
index 07d8495e..4ba53971 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -3,7 +3,7 @@ beautifulsoup4=3D=3D4.8.1
billiard=3D=3D3.6.1.0
celery=3D=3D4.3.0
confusable-homoglyphs=3D=3D3.2.0
-Django>=3D1.11.24,<1.12
+Django>=3D1.11.27,<1.12
django-appconf=3D=3D1.0.3
django-axes=3D=3D4.5.4
django-bootstrap-pagination=3D=3D1.7.1
--=20
2.20.1


Re: [meta-selinux][PATCH 18/19] setools: upgrade 4.1.1 -> 4.2.2

Joe MacDonald
 

Hi Yi,

I've merged the rest of this series, but this one fails to apply.  It looks like your tree didn't contain:

commit 5fd3c5b71edb99659aeb5cb5903088d84517382e (relabel, master)
Author: Christophe PRIOUZEAU <christophe.priouzeau@...>
Date:   Tue Nov 5 14:47:09 2019 +0000

    autorelabel: only selinux-autorelabel need autorelabel file

    With previous implementation, several packages provided
     .autorelabel file while only selinux-autorelabel manage it.
    If there is several packages which try to install .autorelabel
    file, an issue occur during installation of packagegroup-core-selinux.

    Signed-off-by: Christophe Priouzeau <christophe.priouzeau@...>
    Signed-off-by: Joe MacDonald <joe_macdonald@...>

Can you take a look at it and verify which parts of the change are still necessary given the above change?

Thanks.
-J.

On Wed, Nov 13, 2019 at 8:50 PM Yi Zhao <yi.zhao@...> wrote:
* Switch to python3

* Drop patches:
  Fix-build-failure-with-GCC-7-due-to-possible-truncat.patch
  setools4-fix-cross-compiling-errors-for-powerpc-mips.patch

Signed-off-by: Yi Zhao <yi.zhao@...>
---
 ...e-with-GCC-7-due-to-possible-truncat.patch | 105 ------------------
 ...ss-compiling-errors-for-powerpc-mips.patch |  35 ------
 .../setools4-fixes-for-cross-compiling.patch  |  34 +++---
 .../{setools_4.1.1.bb => setools_4.2.2.bb}    |  14 +--
 4 files changed, 19 insertions(+), 169 deletions(-)
 delete mode 100644 recipes-security/setools/setools/Fix-build-failure-with-GCC-7-due-to-possible-truncat.patch
 delete mode 100644 recipes-security/setools/setools/setools4-fix-cross-compiling-errors-for-powerpc-mips.patch
 rename recipes-security/setools/{setools_4.1.1.bb => setools_4.2.2.bb} (61%)

diff --git a/recipes-security/setools/setools/Fix-build-failure-with-GCC-7-due-to-possible-truncat.patch b/recipes-security/setools/setools/Fix-build-failure-with-GCC-7-due-to-possible-truncat.patch
deleted file mode 100644
index a5af041..0000000
--- a/recipes-security/setools/setools/Fix-build-failure-with-GCC-7-due-to-possible-truncat.patch
+++ /dev/null
@@ -1,105 +0,0 @@
-Upstream-Status: Backport [https://github.com/TresysTechnology/setools/commit/e41adf0]
-
-Signed-off-by: Kai Kang <kai.kang@...>
-
-From e41adf01647c695b80b112b337e76021bb9f30c3 Mon Sep 17 00:00:00 2001
-From: Laurent Bigonville <bigon@...>
-Date: Tue, 26 Sep 2017 15:15:30 +0200
-Subject: [PATCH] Fix build failure with GCC 7 due to possible truncation of
- snprintf output
-
-setools fails to build under GCC7 -Wformat -Werror with the following error:
-
-x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -Wno-sign-compare -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Ilibqpol -Ilibqpol/include -I/usr/include/python3.6m -c libqpol/policy_extend.c -o build/temp.linux-amd64-3.6/libqpol/policy_extend.o -Werror -Wextra -Waggregate-return -Wfloat-equal -Wformat -Wformat=2 -Winit-self -Wmissing-format-attribute -Wmissing-include-dirs -Wnested-externs -Wold-style-definition -Wpointer-arith -Wredundant-decls -Wstrict-prototypes -Wunknown-pragmas -Wwrite-strings -Wno-missing-field-initializers -Wno-unused-parameter -Wno-cast-qual -Wno-shadow -Wno-unreachable-code -fno-exceptions
-libqpol/policy_extend.c: In function 'policy_extend':
-libqpol/policy_extend.c:161:27: error: '%04zd' directive output may be truncated writing between 4 and 10 bytes into a region of size 5 [-Werror=format-truncation=]
-    snprintf(buff, 9, "@ttr%04zd", i + 1);
-                           ^~~~~
-libqpol/policy_extend.c:161:22: note: directive argument in the range [1, 4294967295]
-    snprintf(buff, 9, "@ttr%04zd", i + 1);
-                      ^~~~~~~~~~~
-
-Increase the size of the buffer to avoid collisions
-
-Closes: https://github.com/TresysTechnology/setools/issues/174
-Signed-off-by: Laurent Bigonville <bigon@...>
----
- libqpol/policy_extend.c | 16 ++++++++--------
- 1 file changed, 8 insertions(+), 8 deletions(-)
-
-diff --git a/libqpol/policy_extend.c b/libqpol/policy_extend.c
-index 742819b..739e184 100644
---- a/libqpol/policy_extend.c
-+++ b/libqpol/policy_extend.c
-@@ -110,7 +110,7 @@ static int qpol_policy_remove_bogus_aliases(qpol_policy_t * policy)
-  *  Builds data for the attributes and inserts them into the policydb.
-  *  This function modifies the policydb. Names created for attributes
-  *  are of the form @ttr<value> where value is the value of the attribute
-- *  as a four digit number (prepended with 0's as needed).
-+ *  as a ten digit number (prepended with 0's as needed).
-  *  @param policy The policy from which to read the attribute map and
-  *  create the type data for the attributes. This policy will be altered
-  *  by this function.
-@@ -125,7 +125,7 @@ static int qpol_policy_build_attrs_from_map(qpol_policy_t * policy)
-       uint32_t bit = 0, count = 0;
-       ebitmap_node_t *node = NULL;
-       type_datum_t *tmp_type = NULL, *orig_type;
--      char *tmp_name = NULL, buff[10];
-+      char *tmp_name = NULL, buff[16];
-       int error = 0, retv;
-
-       INFO(policy, "%s", "Generating attributes for policy. (Step 4 of 5)");
-@@ -137,7 +137,7 @@ static int qpol_policy_build_attrs_from_map(qpol_policy_t * policy)
-
-       db = &policy->p->p;
-
--      memset(&buff, 0, 10 * sizeof(char));
-+      memset(&buff, 0, 16 * sizeof(char));
-
-       for (i = 0; i < db->p_types.nprim; i++) {
-               /* skip types */
-@@ -158,7 +158,7 @@ static int qpol_policy_build_attrs_from_map(qpol_policy_t * policy)
-                * with this attribute */
-               /* Does not exist */
-               if (db->p_type_val_to_name[i] == NULL){
--                      snprintf(buff, 9, "@ttr%04zd", i + 1);
-+                      snprintf(buff, 15, "@ttr%010zd", i + 1);
-                       tmp_name = strdup(buff);
-                       if (!tmp_name) {
-                               error = errno;
-@@ -240,7 +240,7 @@ static int qpol_policy_build_attrs_from_map(qpol_policy_t * policy)
-  *  Builds data for empty attributes and inserts them into the policydb.
-  *  This function modifies the policydb. Names created for the attributes
-  *  are of the form @ttr<value> where value is the value of the attribute
-- *  as a four digit number (prepended with 0's as needed).
-+ *  as a ten digit number (prepended with 0's as needed).
-  *  @param policy The policy to which to add type data for attributes.
-  *  This policy will be altered by this function.
-  *  @return Returns 0 on success and < 0 on failure; if the call fails,
-@@ -251,7 +251,7 @@ static int qpol_policy_build_attrs_from_map(qpol_policy_t * policy)
- static int qpol_policy_fill_attr_holes(qpol_policy_t * policy)
- {
-       policydb_t *db = NULL;
--      char *tmp_name = NULL, buff[10];
-+      char *tmp_name = NULL, buff[16];
-       int error = 0, retv = 0;
-       ebitmap_t tmp_bmap = { NULL, 0 };
-       type_datum_t *tmp_type = NULL;
-@@ -265,12 +265,12 @@ static int qpol_policy_fill_attr_holes(qpol_policy_t * policy)
-
-       db = &policy->p->p;
-
--      memset(&buff, 0, 10 * sizeof(char));
-+      memset(&buff, 0, 16 * sizeof(char));
-
-       for (i = 0; i < db->p_types.nprim; i++) {
-               if (db->type_val_to_struct[i])
-                       continue;
--              snprintf(buff, 9, "@ttr%04zd", i + 1);
-+              snprintf(buff, 15, "@ttr%010zd", i + 1);
-               tmp_name = strdup(buff);
-               if (!tmp_name) {
-                       error = errno;
---
-2.20.1
-
diff --git a/recipes-security/setools/setools/setools4-fix-cross-compiling-errors-for-powerpc-mips.patch b/recipes-security/setools/setools/setools4-fix-cross-compiling-errors-for-powerpc-mips.patch
deleted file mode 100644
index 9a6b818..0000000
--- a/recipes-security/setools/setools/setools4-fix-cross-compiling-errors-for-powerpc-mips.patch
+++ /dev/null
@@ -1,35 +0,0 @@
-From dc86d880ae0d66233679112a2bf0115c39df68f1 Mon Sep 17 00:00:00 2001
-From: Wenzong Fan <wenzong.fan@...>
-Date: Fri, 17 Feb 2017 08:57:35 +0000
-Subject: [meta-selinux][PATCH] setools4: fix cross-compiling errors for powerpc, mips
-
-Fix build errors:
-| libqpol/policy.c: In function 'qpol_binpol_version':
-| libqpol/policy.c:95:24: error: implicit declaration of function 'bswap_32' [-Werror=implicit-function-declaration]
-| #define le32_to_cpu(x) bswap_32(x)
-
-Upstream-Status: Pending
-
-Signed-off-by: Wenzong Fan <wenzong.fan@...>
----
- libqpol/policy.c | 4 ++++
- 1 file changed, 4 insertions(+)
-
-diff --git a/libqpol/policy.c b/libqpol/policy.c
-index ae3acb5..b5b87f9 100644
---- a/libqpol/policy.c
-+++ b/libqpol/policy.c
-@@ -45,6 +45,10 @@
- # include <asm/types.h>
- #endif
-
-+#if defined(_ARCH_PPC) || defined(mips)
-+#include <byteswap.h>
-+#endif
-+
- #include <sepol/debug.h>
- #include <sepol/handle.h>
- #include <sepol/policydb/flask_types.h>
---
-2.11.0
-
diff --git a/recipes-security/setools/setools/setools4-fixes-for-cross-compiling.patch b/recipes-security/setools/setools/setools4-fixes-for-cross-compiling.patch
index 5c43c49..51c7603 100644
--- a/recipes-security/setools/setools/setools4-fixes-for-cross-compiling.patch
+++ b/recipes-security/setools/setools/setools4-fixes-for-cross-compiling.patch
@@ -1,7 +1,7 @@
-From a104374147b398838edc04e937c92e762ea3f5d9 Mon Sep 17 00:00:00 2001
+From e0a112874d10dbf741c27b107edddedd86d51529 Mon Sep 17 00:00:00 2001
 From: Wenzong Fan <wenzong.fan@...>
 Date: Tue, 14 Feb 2017 06:32:35 +0000
-Subject: [meta-selinux][PATCH] setools4: fixes for cross compiling
+Subject: [PATCH] setools4: fixes for cross compiling

 * search libsepol from $STAGING_LIBDIR
 * fix manual install path as '/usr/share/man/man1'
@@ -9,32 +9,24 @@ Subject: [meta-selinux][PATCH] setools4: fixes for cross compiling
 Upstream-Status: Inappropriate [embedded specific]

 Signed-off-by: Wenzong Fan <wenzong.fan@...>
+Signed-off-by: Yi Zhao <yi.zhao@...>
 ---
- setup.py | 4 ++--
- 1 file changed, 2 insertions(+), 2 deletions(-)
+ setup.py | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)

 diff --git a/setup.py b/setup.py
-index 2ca44c9..300ff70 100644
+index ac8a876..df7ebbc 100644
 --- a/setup.py
 +++ b/setup.py
-@@ -77,7 +77,7 @@ class BuildExtCommand(build_ext):
-         build_ext.run(self)
+@@ -78,7 +78,7 @@ class QtHelpCommand(Command):


--base_lib_dirs = ['.', '/usr/lib64', '/usr/lib', '/usr/local/lib']
-+base_lib_dirs = [os.environ["STAGING_LIBDIR"]]
- include_dirs = ['libqpol', 'libqpol/include']
+ # Library linkage
+-lib_dirs = ['.', '/usr/lib64', '/usr/lib', '/usr/local/lib']
++lib_dirs = [os.environ["STAGING_LIBDIR"]]
+ include_dirs = []

- try:
-@@ -182,7 +182,7 @@ setup(name='setools',
-                 'build_qhc': QtHelpCommand},
-       packages=['setools', 'setools.diff', 'setools.policyrep', 'setoolsgui', 'setoolsgui.apol'],
-       scripts=['apol', 'sediff', 'seinfo', 'seinfoflow', 'sesearch', 'sedta'],
--      data_files=[(join(sys.prefix, 'share/man/man1'), glob.glob("man/*.1"))],
-+      data_files=[('/usr/share/man/man1', glob.glob("man/*.1"))],
-       package_data={'': ['*.ui', '*.qhc', '*.qch'], 'setools': ['perm_map']},
-       ext_modules=ext_py_mods,
-       test_suite='tests',
+ with suppress(KeyError):
 --
-2.13.0
+2.7.4

diff --git a/recipes-security/setools/setools_4.1.1.bb b/recipes-security/setools/setools_4.2.2.bb
similarity index 61%
rename from recipes-security/setools/setools_4.1.1.bb
rename to recipes-security/setools/setools_4.2.2.bb
index db529f4..6e5a950 100644
--- a/recipes-security/setools/setools_4.1.1.bb
+++ b/recipes-security/setools/setools_4.2.2.bb
@@ -9,26 +9,24 @@ SECTION = "base"
 LICENSE = "GPLv2 & LGPLv2.1"

 S = "${WORKDIR}/git"
-SRC_URI = "git://github.com/SELinuxProject/${BPN}.git;branch=4.1 \
+SRC_URI = "git://github.com/SELinuxProject/${BPN}.git;branch=4.2 \
            file://setools4-fixes-for-cross-compiling.patch \
-           file://setools4-fix-cross-compiling-errors-for-powerpc-mips.patch \
-           file://Fix-build-failure-with-GCC-7-due-to-possible-truncat.patch \
 "

-SRCREV = "e03617eb7ab5a035633bff66500b95d25232e331"
+SRCREV = "15bffa7823b9a999f9d51533785ade18fe44df08"

 LIC_FILES_CHKSUM = "file://${S}/COPYING;md5=83a5eb6974c11f30785e90d0eeccf40c \
                     file://${S}/COPYING.GPL;md5=b234ee4d69f5fce4486a80fdaf4a4263 \
                     file://${S}/COPYING.LGPL;md5=4fbd65380cdd255951079008b364516c"

-DEPENDS += "bison-native flex-native swig-native python libsepol"
+DEPENDS += "bison-native flex-native swig-native python3 python3-cython-native libsepol"

-RDEPENDS_${PN} += "python-networkx python-enum34 python-decorator python-setuptools \
-                   python-logging python-json python-argparse libselinux-python"
+RDEPENDS_${PN} += "python3-networkx python3-decorator python3-setuptools \
+                   python3-logging python3-json libselinux-python"

 RPROVIDES_${PN} += "${PN}-console"

-inherit setuptools
+inherit setuptools3

 do_install_append() {
        # Need PyQt5 support, disable gui tools
--
2.17.1



--
Joe MacDonald
:wq


Re: Skipping network-required recipes automatically?

Khem Raj
 

On Thu, 2019-12-19 at 10:03 -0500, Joel A Cohen wrote:
Hi all,

My layer has a bunch of "development" recipes that contain something
like:

PV = "1.1.2+git${SRCPV}"
SRCREV = "${AUTOREV}"
DEFAULT_PREFERENCE = "-1"


In these cases, my recipes have a "known good" version that has a
specific known-good version number (recipe_1.1.2.bb), and then the
"recipe_git.bb" file that I only enable using PREFERRED_VERSION when
I'm wanting to use the latest source code.

My problem with this is that it seems to defeat BB_NO_NETWORK, and
I'm not sure how to handle it. I would like my layer to work
correctly with no network, but the mere presence of these development
recipes causes my build to fail during parsing if the network is not
available.

My last failed attempt was to create a "skip-git" class that I
inherit in local.conf with the following:

python () {
no_network = d.getVar('BB_NO_NETWORK')
pn = d.getVar('PN')
pv = d.getVar('PV')

if no_network and 'git' in pv:
raise bb.parse.SkipRecipe("Recipe requires git, but no
network: %s-%s" % (pn, pv))
}


Unfortunately, bitbake seems to fail in data_smart.py before my class
actually has a chance to skip the recipe even in this case.

Is there a better way to handle this?
you might add PREFERRED_VERSION to select the given recipe, in that
case recipe in your layer should have different version than the
original one. Then you can do something like

python () {
if d.getVar("PREFERRED_VERSION_<pn>") != d.getVar("PV"):
d.delVar("BB_DONT_CACHE")
raise bb.parse.SkipRecipe("Skip it")
}


not too sure if it will ignore it.

another option is to look at devupstream.bbclass it might have some
ideas for you.

Thanks,
Aaron
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
You automatically follow any topics you start or reply to.

View/Reply Online (#47773):
https://lists.yoctoproject.org/g/yocto/message/47773
Mute This Topic: https://lists.yoctoproject.org/mt/68832469/1997914
Group Owner: yocto+owner@...
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [raj.khem@
gmail.com]
-=-=-=-=-=-=-=-=-=-=-=-


apparmor setup #yocto

Hans-Ulrich Schlieben
 

Hi,

i'm new here and not really shure this is the right place for this issue.

I try to get apparmor working using yocto. I tried using thud and zeus with the same result on apparmor_status: apparmor module is not loaded
Checking the kernel flags manually with bitbake -c menuconfig virtual/kernel shows that none of the relevant flags like CONFIG_SECURITY_APPARMOR are set.
Setting these manually i see that the kernel parameters "apparmor=1 security=apparmor" are missing.

My assumption was that adding the apparmor recipe would do all these things.

Do i add the recipe somewhere completely wrong?
Do i possibly override apparmor with other recipes?

Any help would be greatly appreciated.

Best regards

hu






Should these 











Re: variable and task/function timing

Joel A Cohen
 

As a user, I much prefer having one MESON_BUILD_TYPE variable that I can modify using the pn-operator, even if the acceptable values are a little less pretty. The alternative is a variable that I have to guess that name of and what valid values are per-recipe. Sure it's just MESA_BUILD_TYPE now, but next week is there going to be GLIB_BUILD_TYPE and GTK_BUILD_TYPE too?

Also, if "debugoptimized" is the accepted name from the meson world, I feel it's better to keep it rather than second-guessing them with your own name.

--Aaron

On Wed, Dec 11, 2019 at 10:40 AM Trevor Woerner <twoerner@...> wrote:
On Wed 2019-12-11 @ 11:06:44 AM, Nicolas Dechesne wrote:
> On Wed, Dec 11, 2019 at 8:18 AM Trevor Woerner <twoerner@...> wrote:
... 
> You are
> mixing BUILD_TYPE and BUILDTYPE in your email.. maybe you are mixing
> that in your testing as well? What is set in local.conf will be parsed
> first, so if you set a variable there, it will be 'set' when you parse
> the recipe, so when you use ?= it should be a no-op.

Notice, though, that I'm using two different variables:

1) MESA_BUILD_TYPE: which can be set to "debug" or "release"
2) MESON_BUILDTYPE: which can be set to "plain" or "debugoptimized"

MESA_BUILD_TYPE is how the user tweaks the MESON_BUILDTYPE from the mesa
recipe. Given what you've said above, however, I can see now that (as you say)
setting:

        MESON_BUILDTYPE_pn-mesa = ...

in conf/local.conf will do what I want since this variable is recipe-specific
and setting it in one recipe this way won't affect it in others.

I'm sure some will say "there's your solution, do it that way, problem
solved". However, I think "debug/release" are much more natural than
"plain/debugoptimized". I can't change the MESON_BUILDTYPE strings,
they have to be one of those two, so I introduced MESA_BUILD_TYPE as a
level-of-indirection above MESON_BUILDTYPE to allow the user to use the more
natural "debug/release" wording.

So my real question is (and maybe I'm just yak shaving at this point): given
the row/column view of variable setting, how do we factor in the element of
time? For example, *when* do the variables referenced by anonymous python
functions and by tasks get set?

This works (but doesn't allow me the space for nice error checking):

        MESON_BUILDTYPE = "${@bb.utils.contains('MESA_BUILD_TYPE', 'debug', 'debugoptimized', 'plain', d)}"

and this doesn't:

        python do_check_build_type() {
            _buildtype = d.getVar('MESA_BUILD_TYPE')
            if _buildtype not in ['release', 'debug']:
                bb.fatal("unknown build type (%s), please set to either 'release' or 'debug'" % _buildtype)
            if _buildtype == 'debug':
                d.setVar('MESON_BUILDTYPE', 'debugoptimized')
                bb.plain("setting meson build type to debugoptimized")
        }
        addtask check_build_type before do_configure
-=-=-=-=-=-=-=-=-=-=-=-


Skipping network-required recipes automatically?

Joel A Cohen
 

Hi all,

My layer has a bunch of "development" recipes that contain something like:

PV = "1.1.2+git${SRCPV}"
SRCREV = "${AUTOREV}"
DEFAULT_PREFERENCE = "-1"


In these cases, my recipes have a "known good" version that has a specific known-good version number (recipe_1.1.2.bb), and then the "recipe_git.bb" file that I only enable using PREFERRED_VERSION when I'm wanting to use the latest source code.

My problem with this is that it seems to defeat BB_NO_NETWORK, and I'm not sure how to handle it. I would like my layer to work correctly with no network, but the mere presence of these development recipes causes my build to fail during parsing if the network is not available.

My last failed attempt was to create a "skip-git" class that I inherit in local.conf with the following:

python () {
    no_network = d.getVar('BB_NO_NETWORK')
    pn = d.getVar('PN')
    pv = d.getVar('PV')

    if no_network and 'git' in pv:
        raise bb.parse.SkipRecipe("Recipe requires git, but no network: %s-%s" % (pn, pv))
}


Unfortunately, bitbake seems to fail in data_smart.py before my class actually has a chance to skip the recipe even in this case.

Is there a better way to handle this?

Thanks,
Aaron


Re: mc fails to start with readonly rootfs

Marek Belisko
 

On Thu, Dec 19, 2019 at 2:04 PM Marek Belisko via
Lists.Yoctoproject.Org
<marek.belisko=gmail.com@...> wrote:

Hi,

when enable readonly rootfs mc fails to start because cannot write to
directory: /home/root/.config/mc

I've created volatile-bindd to this directory (to be able to write
something) and extend mc using below bbappend:

do_install_append () {
install -d ${D}/home/root/.config/mc
}

FILES_${PN} += "/home/root/.config/mc/*"

But now I got QA issue:

ERROR: mc-4.8.22-r0 do_package: QA Issue: mc: Files/directories were
installed but not shipped in any package:
/home
/home/root
/home/root/.config
/home/root/.config/mc
Please set FILES such that these items are packaged. Alternatively if
they are unneeded, avoid installing them or delete them within
do_install.
QA issue is resolved by adding: FILES_${PN} += "/home/root/.config/mc/"
Is this patch welcomed to upstream to overcome same problem?

Checked FILES_mc and path is there. Am I missing anything obvious? Thanks.

BR,

marek

--
as simple and primitive as possible
-------------------------------------------------
Marek Belisko - OPEN-NANDRA
Freelance Developer

Ruska Nova Ves 219 | Presov, 08005 Slovak Republic
Tel: +421 915 052 184
skype: marekwhite
twitter: #opennandra
web: http://open-nandra.com
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#47771): https://lists.yoctoproject.org/g/yocto/message/47771
Mute This Topic: https://lists.yoctoproject.org/mt/68830517/900721
Group Owner: yocto+owner@...
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [marek.belisko@...]
-=-=-=-=-=-=-=-=-=-=-=-
BR,

marek


mc fails to start with readonly rootfs

Marek Belisko
 

Hi,

when enable readonly rootfs mc fails to start because cannot write to
directory: /home/root/.config/mc

I've created volatile-bindd to this directory (to be able to write
something) and extend mc using below bbappend:

do_install_append () {
install -d ${D}/home/root/.config/mc
}

FILES_${PN} += "/home/root/.config/mc/*"

But now I got QA issue:

ERROR: mc-4.8.22-r0 do_package: QA Issue: mc: Files/directories were
installed but not shipped in any package:
/home
/home/root
/home/root/.config
/home/root/.config/mc
Please set FILES such that these items are packaged. Alternatively if
they are unneeded, avoid installing them or delete them within
do_install.

Checked FILES_mc and path is there. Am I missing anything obvious? Thanks.

BR,

marek

--
as simple and primitive as possible
-------------------------------------------------
Marek Belisko - OPEN-NANDRA
Freelance Developer

Ruska Nova Ves 219 | Presov, 08005 Slovak Republic
Tel: +421 915 052 184
skype: marekwhite
twitter: #opennandra
web: http://open-nandra.com


Re: SSTATE_MIRRORS with ssh/sftp fetchers #yocto

Ross Burton <ross.burton@...>
 

On 19/12/2019 11:16, Richard Purdie wrote:
Do you have any idea on how to use these fetchers (or other SSH keys-
based secure fetcher) to share sstate cache ?
I think this is a "not implemented" problem.
If you look at bitbake/lib/bb/fetch2/ssh.py or sftp.py, they don't have
a checkstatus() method that local.py or wget.py has.
I think the sstate code relies on the checkstatus method working to
know know if an sstate artefact is present.
Performance of that function is key and creating a new ssh connection
to check for each sstate artefact may make the build prohibitively
slow. There are some connection caching mechanisms present which may
help with that.
I actually started implementing this a while ago:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=ross/mut2&id=42bdcd8181d2826a5a5dff6803958a900c0c4a2d

I *think* that worked but I can't be sure, it was a while ago and typically I forgot to put notes to my future self in the commit message.

As RP says, you'll want to use connection sharing: this is configured using ControlMaster and friends in the ssh configuration. That should make the performance reasonable.

Ross


Re: SSTATE_MIRRORS with ssh/sftp fetchers #yocto

Richard Purdie
 

On Thu, 2019-12-19 at 02:37 -0800, angeal1105@... wrote:
Hi everyone,

Recently, I tried to use the SSTATE_MIRRORS variable to get back
sstate cache from a build machine shared amongst developers.
When I tried with the simple FILE fetcher, it worked great and
resumed the work where the build machine stopped.
But when I tried to replace it with SSH and/or SFTP fetchers (setup
to connect on my own machine), I had no errors or warnings, but the
steps started from scratch, ignoring the sstate cache mirror.

The thing is that we can't use unauthenticated HTTP/FTP fetchers for
corporate security reasons.
SSH and SFTP are convenients as we use crops and mount ssh keys
inside to authenticate ourselves on corporate git servers.

Environment:

I'm using Yocto 2.6 (Thud) and CROPS.
I've set up my machine as SSH/SFTP server, and I tried it manually,
it works: the connection is established (even in CROPS).

I defined the SSTATE_MIRRORS variable in the build/conf/local.conf
file.
Working: SSTATE_MIRRORS = "file://.*
file://{COMMON_PATH}/sstate-cache/PATH"
Not working: SSTATE_MIRRORS = "file://.*
ssh://user@my-machine-ip/{COMMON_PATH}/sstate-cache/PATH"
Notworking: SSTATE_MIRRORS = "file://.*
sftp://user@my-machine-ip/{COMMON_PATH}/sstate-cache/PATH"

Do you have any idea on how to use these fetchers (or other SSH keys-
based secure fetcher) to share sstate cache ?
I think this is a "not implemented" problem.

If you look at bitbake/lib/bb/fetch2/ssh.py or sftp.py, they don't have
a checkstatus() method that local.py or wget.py has.

I think the sstate code relies on the checkstatus method working to
know know if an sstate artefact is present.

Performance of that function is key and creating a new ssh connection
to check for each sstate artefact may make the build prohibitively
slow. There are some connection caching mechanisms present which may
help with that.

Probably not the answer you wanted, sorry but at least it might explain
why its not working.

Cheers,

Richard


Re: QA Cycle report for build (yocto-3.1_M1.rc8)

Richard Purdie
 

On Wed, 2019-12-18 at 07:38 -0800, akuster808 wrote:

On 12/18/19 2:46 AM, Richard Purdie wrote:
On Wed, 2019-12-18 at 03:32 +0000, Jain, Sangeeta wrote:
Hello all,

This is the full report for 3.1_M1.rc8:
https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/tree/?h=intel-yocto-testresults


======= Summary ========
No high milestone defects.
No new defects are found in this cycle.
glib and valgrind ptest failed (BUG id:13713).
Vineela,

We have TSC approval so could we release this please!

Thanks,

Richard


SSTATE_MIRRORS with ssh/sftp fetchers #yocto

angeal1105@...
 

Hi everyone,

Recently, I tried to use the SSTATE_MIRRORS variable to get back sstate cache from a build machine shared amongst developers.
When I tried with the simple FILE fetcher, it worked great and resumed the work where the build machine stopped.
But when I tried to replace it with SSH and/or SFTP fetchers (setup to connect on my own machine), I had no errors or warnings, but the steps started from scratch, ignoring the sstate cache mirror.

The thing is that we can't use unauthenticated HTTP/FTP fetchers for corporate security reasons.
SSH and SFTP are convenients as we use crops and mount ssh keys inside to authenticate ourselves on corporate git servers.

Environment:

I'm using Yocto 2.6 (Thud) and CROPS.
I've set up my machine as SSH/SFTP server, and I tried it manually, it works: the connection is established (even in CROPS).

I defined the SSTATE_MIRRORS variable in the build/conf/local.conf file.
Working:        SSTATE_MIRRORS = "file://.* file://{COMMON_PATH}/sstate-cache/PATH"
Not workingSSTATE_MIRRORS = "file://.* ssh://user@my-machine-ip/{COMMON_PATH}/sstate-cache/PATH"
NotworkingSSTATE_MIRRORS = "file://.* sftp://user@my-machine-ip/{COMMON_PATH}/sstate-cache/PATH"

Do you have any idea on how to use these fetchers (or other SSH keys-based secure fetcher) to share sstate cache ?

Don't hesitate to ask for more information/tests from me.
Regards,
Anthony


Re: Inverse of BBFILES_DYNAMIC

Konrad Weihmann <kweihmann@...>
 

 

Thanks for your input – That‘s what I used as well in the past, but I have to admit it’s hard to read and to debug, that’s why I want to omit such constructs.

So is there a less hacky way (maybe without the conditional syntax, which I still think is a readability nightmare)?

 

Von: Yann Dirson [mailto:yann.dirson@...]
Gesendet: Mittwoch, 18. Dezember 2019 22:59
An: Konrad Weihmann
Cc: yocto@...
Betreff: Re: [yocto] Inverse of BBFILES_DYNAMIC

 

 

 

Le mer. 18 déc. 2019 à 21:10, Konrad Weihmann <kweihmann@...> a écrit :

Hi All,

I was wondering if there is already a mechanism which kind of mimics the inverse of BBFILES_DYNAMIC.
Like I want a recipe to be put into tree only if a layer is absent.

Example: I need a fancy python-lib, which isn't part of the core-lib, but according to the layer index can be obtained by some 3rd party layer (let's call fancy-layer).
For some projects I worked for, it might be challenging to use "fancy-layer" as it doesn't comply with some corporate rules or whatsoever, while others don't see any issues here.

What I have in mind is a mechanism like BBFILES_DYNAMIC, but inverse, like

BBFILES_DYNAMIC += "!fancy-layer: ${LAYERDIR}/dynamic-layers/fancy-fallback/*/*.bb"

So in case fancy-layer is absent, I could put in a recipe which isn't that well maintained (also might including a warning that this only a fallback solution), but does make my layer usable for a wider audience.

So back to the original question, is that already existing functionality? Or is there a workaround hack existing?
If both not, is that something, that could be considered as valuable by the project (If so I could craft some patches for that)?

 

Before BBFILES_DYNAMIC the funtionnality could already be achieved with code like the following:

 

BBFILES += "${@' '.join('${LAYERDIR}/dynamic-layers/%s/recipes*/*/*.bbappend' % layer \
               for layer in BBFILE_COLLECTIONS.split())}"

 

You're only limited by what you can express using such python expressions :)

 


Regards
Konrad

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#47760): https://lists.yoctoproject.org/g/yocto/message/47760
Mute This Topic: https://lists.yoctoproject.org/mt/68804558/3618661
Group Owner: yocto+owner@...
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub  [yann.dirson@...]
-=-=-=-=-=-=-=-=-=-=-=-



--

Yann Dirson <yann@...>
Blade / Shadow -- http://shadow.tech

9601 - 9620 of 57384