Date   

[yocto-autobuilder-helper][PATCH] config.json: set max load in PARALLEL_MAKE

Trevor Gamblin
 

Add "-l 52" to PARALLEL_MAKE in config.json to limit Make and Ninja
builds based on the detected system load. With this option added, if
either tool has at least one job running and detects that the system
load exceeds the given value, it will wait until either the system load
average drops below that limit, or until all other jobs are finished
before starting additional jobs.

Since most autobuilder machines have 56 cores, this should help keep the
system from being overloaded during builds.

Reference: https://www.gnu.org/software/make/manual/html_node/Parallel.html

Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
---
config.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/config.json b/config.json
index 7be0df7..3709b40 100644
--- a/config.json
+++ b/config.json
@@ -44,7 +44,7 @@
"PREMIRRORS = ''",
"BB_GENERATE_MIRROR_TARBALLS = '1'",
"BB_NUMBER_THREADS = '16'",
- "PARALLEL_MAKE = '-j 16'",
+ "PARALLEL_MAKE = '-j 16 -l 52'",
"XZ_MEMLIMIT = '5%'",
"XZ_THREADS = '8'",
"BB_TASK_NICE_LEVEL = '5'",
--
2.31.1


Yocto Autobuilder: Latency Monitor and AB-INT - Meeting notes: July 15, 2021

Randy MacLeod
 

YP AB Intermittent failures meeting
===================================
July 15, 2021, 9 AM ET
https://windriver.zoom.us/j/3696693975

Attendees: Tony, Richard, Trevor, Randy


Summary:
========

ptest failures, somewhat improved but still seeing
problems particularly on the arm builder.


Add Michael Halstead, see questions below in section 4.


If anyone wants to help, we could use more eyes on the logs,
particularly the summary logs and understanding iostat #
when the dd test times out.



Plans for the week:
===================

Richard: maybe bitbake server
Alex:
Sakib: hook more responsive load average in to latency test.
Trevor: patch to set PARALLEL_MAKE : -l 50
Tony: tony to drop AB INT flags for bugs that we have worked around.
Saul: on vacation
Randy: organize, rust


Meeting Notes:
==============

1, runqemu (same as last week so I'll drop this comment next week)

Tony having trouble with runqemu on some Wind River machine.
Richard has a fix for a race in runqemu in master-next.
These might be related but if not Tony should debug the
issue/ collect logs.

2. job server

- Trevor has submitted changes to use -l for both make and ninja
with a value of '50', in master-next, The auto-builders are 56 core
machines. Sometimes the load average is still around 65 and that's
likely because ninja uses the 1 minute load average and it can
start 10s of compiles before that limit is set.

- ninja could be patched with make's more responsive algorithm
next or is this good enough?

- Richard suggested that we extract make's code for measuring the load
average to a separate binary and run it in the periodic io latency
test. Also can we translate it to python?


3. AB status

ptest cases are improving but still need work.



progress on tcl, and other tests thanks to Ross.

parted is still failing frequently,

Ross is not able to reproduce it locally.
gdb test failing still. - Randy!

4. Nothing new on this item this week:
Richard reported
- something really flaky going on with serial ports.
- particularly bad on qemuppc but also x86.
- related to Saul's QMP data dump?

5. Sakib's improvements to the logging are merged.
We think Michael needs to update the script that generates the
web page. Randy/Sakib to talk with Michael.

6. (From July 8)
Richard says that we may need to redesign the data collection system
that Sakib's AB INT tests are based on.



Still relevant parts of
Previous Meeting Notes:
=======================


4. bitbake server timeout.

"Timeout while waiting for a reply from the bitbake server (60s)"

Randy mentioned that the bitbake server timeouts seen in the
Wind River build cluster have gone away after upgrading to
a newer version of docker.

Old: Docker Version: Docker version 18.09.4, build d14af54266
New: Docker Version: Docker version 20.10.7, build f0df350


Clearly the YP ABs aren't running in docker but what
about firmware and kernel tunings.

Michael,

Is the BIOS/firmware kept up to date on most nodes?

It seems that we are running stock kernels which makes sense but
given that we don't have concerns about privacy since system access
is controlled and the nodes are being used to test open
source software, we might consider optimizing for performance
rather than security.

Alex pointed at: https://make-linux-fast-again.com/
Which just lists a set of kernel boot options:
noibrs noibpb nopti nospectre_v2 nospectre_v1 \
l1tf=off nospec_store_bypass_disable no_stf_barrier \
mds=off tsx=on tsx_async_abort=off mitigations=off

Michael,
Can we enable some or all of these on a node to see what the
performance difference is?


5. io stalls

Richard said that it would make sense to write an ftrace utility
/ script to monitor io latency and we could install it with sudo
Ch^W mentioned ftrace on IRC.
Sakib and Randy will work on that but not for a week or two.





../Randy


[meta-zephyr][PATCH v4 2/2] zephyr-openthread-echo-client: Add new echo-client variant for OpenThread

Stefan Schmidt
 

From: Stefan Schmidt <stefan.schmidt@...>

Similar to the normal echo-client example it demonstrates socket usage,
but in this variant we enable the OpenThread config overlay and add the
needed module to the build.

Signed-off-by: Stefan Schmidt <stefan.schmidt@...>
---
Since v3: Added COMPATIBLE_MACHINE filter and comment

.../zephyr-kernel/zephyr-openthread-echo-client.bb | 13 +++++++++++++
1 file changed, 13 insertions(+)
create mode 100644 recipes-kernel/zephyr-kernel/zephyr-openthread-echo-c=
lient.bb

diff --git a/recipes-kernel/zephyr-kernel/zephyr-openthread-echo-client.b=
b b/recipes-kernel/zephyr-kernel/zephyr-openthread-echo-client.bb
new file mode 100644
index 0000000..49f5565
--- /dev/null
+++ b/recipes-kernel/zephyr-kernel/zephyr-openthread-echo-client.bb
@@ -0,0 +1,13 @@
+include zephyr-sample.inc
+
+ZEPHYR_SRC_DIR =3D "${S}/samples/net/sockets/echo_client"
+
+ZEPHYR_MODULES_append =3D "\;${S}/modules/lib/mbedtls"
+ZEPHYR_MODULES_append =3D "\;${S}/modules/lib/openthread"
+
+EXTRA_OECMAKE +=3D "-DOVERLAY_CONFIG=3Doverlay-ot.conf"
+
+# The overlay config and OpenThread itself imposes some specific require=
ments
+# towards the boards (e.g. flash layout and ieee802154 radio) so we need=
to
+# limit to known working machines here.
+COMPATIBLE_MACHINE =3D "(arduino-nano-33-ble)"
--=20
2.31.1


[meta-zephyr][PATCH v4 1/2] zephyr-kernel-src.inc: Add backport patch for storage partition

Stefan Schmidt
 

From: Stefan Schmidt <stefan.schmidt@...>

Patch already merged upstream, but after the 2.6 release we are based
on. Taking the backport in here until we can figure out if that can go
to the Zephyr 2.6 LTS branch.

The storage partition itself is needed on the Arduino Nano e.g. for
OpenThread or applications needed a storage space.

Signed-off-by: Stefan Schmidt <stefan.schmidt@...>
---
since v3: Added backported patch for flash layout

...rduino-nano-33-ble-storage-partition.patch | 46 +++++++++++++++++++
.../zephyr-kernel/zephyr-kernel-src.inc | 1 +
2 files changed, 47 insertions(+)
create mode 100644 recipes-kernel/zephyr-kernel/files/arduino-nano-33-bl=
e-storage-partition.patch

diff --git a/recipes-kernel/zephyr-kernel/files/arduino-nano-33-ble-stora=
ge-partition.patch b/recipes-kernel/zephyr-kernel/files/arduino-nano-33-b=
le-storage-partition.patch
new file mode 100644
index 0000000..fa0f27f
--- /dev/null
+++ b/recipes-kernel/zephyr-kernel/files/arduino-nano-33-ble-storage-part=
ition.patch
@@ -0,0 +1,46 @@
+commit 6c9945aafa00c09149e2052a9c2bccad16dd1d8a
+Author: Stefan Schmidt <stefan.schmidt@...>
+Date: Fri May 7 11:47:44 2021 +0200
+
+ boards/arduino_nano_33_ble: add storage partition at end of flash
+ =20
+ Change default partition table to allow for application which need
+ storage. One use case is running the OpenThread integration which ha=
s
+ a dependency on this.
+ =20
+ Signed-off-by: Stefan Schmidt <stefan.schmidt@...>
+
+diff --git a/boards/arm/arduino_nano_33_ble/arduino_nano_33_ble.dts b/bo=
ards/arm/arduino_nano_33_ble/arduino_nano_33_ble.dts
+index d09b66ec43..d11d800eb5 100644
+--- a/boards/arm/arduino_nano_33_ble/arduino_nano_33_ble.dts
++++ b/boards/arm/arduino_nano_33_ble/arduino_nano_33_ble.dts
+@@ -44,15 +44,27 @@
+=20
+ boot_partition: partition@0 {
+ label =3D "sam-ba";
+- reg =3D <0x0 0x10000>;
++ reg =3D <0x00000000 0x00010000>;
+ read-only;
+ };
+=20
+ code_partition: partition@10000 {
+ label =3D "code";
+- reg =3D <0x10000 0xf0000>;
++ reg =3D <0x00010000 0x000e8000>;
+ read-only;
+ };
++
++ /*
++ * The flash starting at 0x000f8000 and ending at
++ * 0x000fffff is reserved for use by the application.
++ *
++ * Storage partition will be used by FCB/LittleFS/NVS
++ * if enabled.
++ */
++ storage_partition: partition@f8000 {
++ label =3D "storage";
++ reg =3D <0x000f8000 0x00008000>;
++ };
+ };
+ };
+=20
diff --git a/recipes-kernel/zephyr-kernel/zephyr-kernel-src.inc b/recipes=
-kernel/zephyr-kernel/zephyr-kernel-src.inc
index a0004ed..227c7f4 100644
--- a/recipes-kernel/zephyr-kernel/zephyr-kernel-src.inc
+++ b/recipes-kernel/zephyr-kernel/zephyr-kernel-src.inc
@@ -18,6 +18,7 @@ SRC_URI =3D "\
git://github.com/zephyrproject-rtos/openthread.git;protocol=3Dhttps;=
branch=3Dzephyr;destsuffix=3Dgit/modules/lib/openthread;name=3Dopenthread=
\
git://github.com/zephyrproject-rtos/libmetal.git;protocol=3Dhttps;de=
stsuffix=3Dgit/modules/hal/libmetal;name=3Dlibmetal \
git://github.com/zephyrproject-rtos/tinycrypt.git;protocol=3Dhttps;d=
estsuffix=3Dgit/modules/crypto/tinycrypt;name=3Dtinycrypt \
+ file://arduino-nano-33-ble-storage-partition.patch \
"
S =3D "${WORKDIR}/git"
=20
--=20
2.31.1


[meta-security][PATCH] Do not use clang toolchain in Parsec recipes

Anton Antonov
 

Signed-off-by: Anton Antonov <Anton.Antonov@...>
---
.../recipes-parsec/parsec-service/parsec-service_0.7.0.bb | 3 +--
meta-parsec/recipes-parsec/parsec-tool/parsec-tool_0.3.0.bb | 2 --
2 files changed, 1 insertion(+), 4 deletions(-)

diff --git a/meta-parsec/recipes-parsec/parsec-service/parsec-service_0.7.0.bb b/meta-parsec/recipes-parsec/parsec-service/parsec-service_0.7.0.bb
index 0e14955..d57a43a 100644
--- a/meta-parsec/recipes-parsec/parsec-service/parsec-service_0.7.0.bb
+++ b/meta-parsec/recipes-parsec/parsec-service/parsec-service_0.7.0.bb
@@ -10,8 +10,7 @@ SRC_URI += "crate://crates.io/parsec-service/${PV} \
file://parsec-tmpfiles.conf \
"

-DEPENDS = "tpm2-tss"
-TOOLCHAIN = "clang"
+DEPENDS = "tpm2-tss clang-native"

CARGO_BUILD_FLAGS += " --features all-providers,cryptoki/generate-bindings,tss-esapi/generate-bindings"

diff --git a/meta-parsec/recipes-parsec/parsec-tool/parsec-tool_0.3.0.bb b/meta-parsec/recipes-parsec/parsec-tool/parsec-tool_0.3.0.bb
index 35c65c0..881f8d8 100644
--- a/meta-parsec/recipes-parsec/parsec-tool/parsec-tool_0.3.0.bb
+++ b/meta-parsec/recipes-parsec/parsec-tool/parsec-tool_0.3.0.bb
@@ -7,8 +7,6 @@ inherit cargo
SRC_URI += "crate://crates.io/parsec-tool/${PV} \
"

-TOOLCHAIN = "clang"
-
do_install() {
install -d ${D}/${bindir}
install -m 755 "${B}/target/${TARGET_SYS}/release/parsec-tool" "${D}${bindir}/parsec-tool"
--
2.25.1


Re: Sysvinit recipe with custom package group Build ERROR #dunfell

Abdelrahman El-Gammal <a.elgammal2019@...>
 

here is the package group custom recipe:
DESCRIPTION = "My Custom Package groups"
inherit packagegroup
 
PROVIDES = "${PACKAGES}"
PACKAGES = " \
           ${PN}-apps \
           ${PN}-tools \
           "
 
RDEPENDS_${PN}-apps = " \
                       \
                      demoapp \
                      demoappstartup \
                      "
 
RDEPENDS_${PN}-tools = " \
                      \   
                      "
 
RECOMMENDS_${PN}-apps = " \
                       \
                      "
#RDEPENDS_qt-kiosk-browser += "qtbase ogl-runtime qtwebengine"
#demoappstartup


Sysvinit recipe with custom package group Build ERROR #dunfell

Abdelrahman El-Gammal <a.elgammal2019@...>
 

Hello there,
I am building a sysvinit recipe for a startup app. And add into the image using custom package groups. Both the package group and recipe when built on their own, the build succeeds. But, when building the image, I got error in do_rootfs task.
Here is the error msg:

The following packages have unmet dependencies:
 packagegroup-custom-apps : Depends: demoappstartup but it is not installable
E: Unable to correct problems, you have held broken packages.
Here is the Sysvinit recipe:
SUMMARY = "startup_script"
DESCRIPTION = "This recipe builds startup applications"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
 
 
 
 
SRC_URI = "file://demoapp_startup.sh"
           
 
 
INITSCRIPT_NAME = "demoapp_startup.sh"
INITSCRIPT_PARAMS = "start 99 5 2 . stop 20 0 1 6 ."
 
inherit update-rc.d
S = "${WORKDIR}"
 
 
do_install () {
   install -d ${D}${sysconfdir}/init.d/
   install -c -m 755 ${WORKDIR}/${INITSCRIPT_NAME} ${D}${sysconfdir}/init.d/${INITSCRIPT_NAME}
  }

Here is the shell script for the app, demoapp_startup.sh:
#! /bin/sh
# /etc/init.d/start_sample_app
#
 
# Carry out specific functions when asked to by the system
case "$1" in
start)
   echo "Starting sampleApp "
   exec /opt/trial1/bin/trial1 -s &
      ;;
stop)
   echo "Stopping sampleApp"
   start-stop-daemon --stop --name trial1 --quiet
   ;;
(*))
   echo "Usage: /etc/init.d/start_sample_app {start|stop}"
   exit 1
   ;;
esac
exit 0


[meta-rockchip][PATCH] rock-pi-e: update preferred kernel

Trevor Woerner
 

The latest updates to linux-yocto-dev now include support for the rock-pi-e so
do away with our custom recipe and use the one from oe-core.

Signed-off-by: Trevor Woerner <twoerner@...>
---
conf/machine/rock-pi-e.conf | 2 +-
.../linux/linux-stable-bleeding_5.11.bb | 18 ------------------
recipes-kernel/linux/linux-yocto%.bbappend | 1 +
3 files changed, 2 insertions(+), 19 deletions(-)
delete mode 100644 recipes-kernel/linux/linux-stable-bleeding_5.11.bb

diff --git a/conf/machine/rock-pi-e.conf b/conf/machine/rock-pi-e.conf
index 7afe143..9cd3ed4 100644
--- a/conf/machine/rock-pi-e.conf
+++ b/conf/machine/rock-pi-e.conf
@@ -7,7 +7,7 @@ require conf/machine/include/rk3328.inc

MACHINEOVERRIDES =. "rock-pi-e:"

-PREFERRED_PROVIDER_virtual/kernel = "linux-stable-bleeding"
+PREFERRED_PROVIDER_virtual/kernel = "linux-yocto-dev"
KERNEL_DEVICETREE = "rockchip/rk3328-rock-pi-e.dtb"
MACHINE_EXTRA_RRECOMMENDS += "kernel-modules"

diff --git a/recipes-kernel/linux/linux-stable-bleeding_5.11.bb b/recipes-kernel/linux/linux-stable-bleeding_5.11.bb
deleted file mode 100644
index 508ddae..0000000
--- a/recipes-kernel/linux/linux-stable-bleeding_5.11.bb
+++ /dev/null
@@ -1,18 +0,0 @@
-# the rock-pi-e is very new
-# it's exciting that support has already been added upstream
-# but it's so new that even linux-yocto-dev doesn't have it yet
-#
-# in time we should see the need for this recipe going away
-
-inherit kernel
-require recipes-kernel/linux/linux-yocto.inc
-
-LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
-
-SRC_URI = "git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git"
-SRCREV = "c03c21ba6f4e95e406a1a7b4c34ef334b977c194"
-LINUX_VERSION = "5.11"
-LINUX_VERSION_EXTENSION = ""
-PV = "${LINUX_VERSION}"
-
-COMPATIBLE_MACHINE_rock-pi-e = "rock-pi-e"
diff --git a/recipes-kernel/linux/linux-yocto%.bbappend b/recipes-kernel/linux/linux-yocto%.bbappend
index 3789c72..c2fe9ad 100644
--- a/recipes-kernel/linux/linux-yocto%.bbappend
+++ b/recipes-kernel/linux/linux-yocto%.bbappend
@@ -9,6 +9,7 @@ COMPATIBLE_MACHINE_rock-pi-4 = "rock-pi-4"
COMPATIBLE_MACHINE_nanopi-m4 = "nanopi-m4"
COMPATIBLE_MACHINE_nanopi-m4-2gb = "nanopi-m4-2gb"
COMPATIBLE_MACHINE_rock64 = "rock64"
+COMPATIBLE_MACHINE_rock-pi-e = "rock-pi-e"

FILESEXTRAPATHS_prepend := "${THISDIR}/files:"

--
2.30.0.rc0


[meta-rockchip][PATCH] remove LINUX_VERSION_EXTENSION

Trevor Woerner
 

Adding "-rockchip" to the Linux kernel name implies, to me anyway, that this
is a vendor kernel. The PREFERRED_PROVIDERs of all kernels specified in this
BSP are upstream linux-yocto kernels, not vendor kernels. Therefore remove the
version name extension to avoid confusion.

Signed-off-by: Trevor Woerner <twoerner@...>
---
conf/machine/include/rockchip-defaults.inc | 1 -
1 file changed, 1 deletion(-)

diff --git a/conf/machine/include/rockchip-defaults.inc b/conf/machine/include/rockchip-defaults.inc
index 0666119..905b2a6 100644
--- a/conf/machine/include/rockchip-defaults.inc
+++ b/conf/machine/include/rockchip-defaults.inc
@@ -3,7 +3,6 @@
# kernel
PREFERRED_PROVIDER_virtual/kernel ?= "linux-yocto"
KCONFIG_MODE ?= "alldefconfig"
-LINUX_VERSION_EXTENSION ?= "-rockchip"

# xserver
PREFERRED_PROVIDER_virtual/xserver = "xserver-xorg"
--
2.30.0.rc0


Yocto Technical Team Minutes, Engineering Sync, for July 13, 2021

Trevor Woerner
 

Yocto Technical Team Minutes, Engineering Sync, for July 13, 2021
archive: https://docs.google.com/document/d/1ly8nyhO14kDNnFcW2QskANXW3ZT7QwKC5wWVDg9dDH4/edit

== disclaimer ==
Best efforts are made to ensure the below is accurate and valid. However,
errors sometimes happen. If any errors or omissions are found, please feel
free to reply to this email with any corrections.

== attendees ==
Trevor Woerner
Stephen Jolley
Scott Murray
Joshua Watt
Peter Kjellerstedt
Michael Halstead
Jasper Orschulko
Steve Sakoman
Tony Tascioglu
Trevor Gamblin
Saul Wold
Armin Kuster
Randy MacLeod
Richard Purdie
Angolini
Ross Burton
Bruce Ashfield

== project status ==
- an eSDK issue has been resolved and this should resolve a number of eSDK
bugs
- the prserv rewrite is still pending on resolving the issues with python
asyncio
- continuing to see AB-INT improvements
- multiconfig needs simpler test cases to reproduce issues

== discussion ==
RP: it was a good week for AB improvements! ptests are growing in number


RP: we are lacking official maintainers for various subsystems. there are a
number of unofficial ones (e.g. Bruce - kernels, Khem - toolchains, etc)
but nothing official. we have a “bus” problem.
Randy: people can adopt packages, but i’m curious about an outline of what
you think is lacking
RP: pseudo, devtool, extensible sdk, eclipse plugin (maybe not), recipetool,
a backup maintainer for bitbake, test suites, QA frameworks, reporting
system…
Randy: it could be a question of corporate mandate for some
RP: i’ll be raising this with the members
TrevorW: in one case there was a problem with a certain subsystem that had
been going on for a long time with no signs of life from the “active
maintainer”. after a period of time i decided i would get involved
and step forward and say i’d take it over. then suddenly the AWOL
maintainer shows up, declares the subsystem has a maintainer and no change
in maintainership is needed. in another case i found what i thought was
a bug in an unmaintained subsystem, worked on a fix/patch but when i
submitted it, the previous maintainer (who had stepped down at this point)
spoke up to say there was no bug, pointed out that my “fix” didn’t
preserve the existing behaviour (which i felt was broken, so of course it
didn’t maintain it) therefore my patch shouldn’t go in. so we have
subsystems whose active maintainers aren’t heard from for long periods
but then show up once someone steps forward to take over, and we have
other subsystems where previous maintainers who have given up their roles
are still controlling the subsystem. if i were to maintain a subsystem, i
would expect some sort of control over the subsystem i was maintaining.
For example with U-Boot a person agrees to maintain, for example, an SoC
so Tom gives them full control over the code related to that SoC. they
don’t wake up the next day to find a whole bunch of SoC-specific patches
have been merged by the benevolent dictator while they were sleeping and
then have to deal with the consequences. The same happens in the kernel:
people are given an area to maintain and they do so.
RP: a maintainer doesn’t get to make all decisions. for example: recently
i wanted to make a change to bitbake but was shot down, so even the
benevelent dictator can be ruled down ;-) there’s no free choice. being
a maintainer isn’t a binary switch, you don’t become a maintainer
then suddenly get full control of some part of the project and get to
make any decisions you want. another example: a kernel submaintainer
isn’t absolute either, when i maintained a kernel subsystem sometimes
you win some discussions and sometimes you lose some. one has to work into
a maintainer role, not just take it over. and it’s good to retain a
connection to a former maintainer for continuance.
TrevorW: having code change “underneath the maintainer” can get
frustrating
RP: we will never have a system whereby i will only grab a patch after
maintainer sign-off. the maintainer leaves for a 2 week vacation and the
patch queue comes to a halt for that subsystem, that’s not right
TrevorW: having more distributed maintainership might require fundamental
changes to how the project is managed (e.g. how we do releases). some
other projects with more distributed maintainership will have times when
things get added (-rc1, -rc2) with integration windows and during that
time it’s up to maintainers to do their own things with the patches.
with our project (yocto) patches that show up on the mailing list today
are added to next and even master in huge batches and run on the AB. so I
don’t see what the role of a maintainer would be in that case if patches
are being moved about without their intervention
Scott: yocto is different because we’re dealing with the timelines of lots
of sub projects so it might not fit into easy merge windows like it does
for individual projects (-rc1, -rc2 etc merge windows)
RP: getting patches in quickly helps keep people engaged. how would people
feel if they submitted a patch then 3 weeks later got a reply saying
“your patch failed on the AB, please re-spin”?
TrevorW: if you submitted a patch for the linux kernel today it would show
up… 3 (?) releases from now (provided it was “immediately”
accepted)? so that would be the same as other projects
Randy: we do have maintainers for every layer and we have Bruce and Khem, but
there is a gap
RP: wrt pseudo, i still haven’t merged things to master, so i’ve held off
because the original maintainer doesn’t like some of my changes. it’s
a difficult balancing act
Randy: TrevorW would you still be interested in taking over pseudo
TrevorW: I think i’d aim for devtool instead, i think pseudo is
“complicated” (not from a technical point of view, but from a social
standpoint)
RP: the problem with pseudo is that anyone taking it over would most likely
start over and solve the problem in a completely different way. there are
a lot of newer kernel features that we would use instead
JPEW: i would rather have a devtool maintainer than a pseudo maintainer. as a
project we strongly push people towards using devtool on a daily basis, i
would prefer to see that subproject have an active maintainer


Randy: we’re a week away from -m2, anyone have anything else to discuss?
Randy: i’m hoping to update things i’m working on.
RP: there are some things that are broken in sato (icons not generated
correctly) so i’m looking forward to those updates (icons depends on
librsvg, which depends on rust)
Ross: maybe we should revert now until the fix is ready?
RP: if you know which bits to reverse then go ahead
Ross: oh… i think the reverse is already in
RP: okay, maybe there’s something else that needs revert. every week the AUH
reports not being able to move forward on this
Scott: i have a work tree setup for read-only PR server. so i’m hoping to be
able to take that over at some point
RP: ping me if you need help, i can help with historical context
JPEW: i can help too, with some technical issues as well
RP: MichaelH is probably not excited about the webserver stuff
JPEW: where is it running
MH: google compute
JPEW: do we have a kubernetes cluster
MH: no
RP: we would like to share hash equiv on a read-only port, which might give MH
a heart-attack ;-)
MH: currently we’re running on a google computer node, but based on it’s
resources, we could move it to something cheaper
Randy: JPEW: do you need a kubernetes cluster?
JPEW: no. i brought it up because it might be a way to load balance the
demand, but that would require a re-write on some of the internals of the
hash equiv server (e.g. move to a full, separate SQL database for example)


MH: btw, irc logs now being saved again


JPEW: my sstate patch is now ready
RP: does it use compression?
JPEW: originally yes, but then i noticed that it makes sstate not
reproducible, so i dropped it
RP: why? that shouldn’t be the case?
JPEW: it wouldn’t if it were single thread compression, but the way zstd
does parallel compression makes it give different results based on the
compression factor
PR: we use parallel compression for a lot of other stuff and i’ve never seen
such an issue
JPEW: i’ll do more investigation. i think there might be an option to pass
to make it reproducible even under parallel compression
RP: yes, that sounds right
JPEW: also if we’re going to use zstd and pzstd they require those tools to
be available in the host
RP: cmdline (i.e. not libraries)?
JPEW: yes, so we can include it in the buildtools tarball
RP: yes and we can detect it in the host easily
JPEW: okay, i will work up the patch
RP: you can send the patch as-is now and we can add compression later


Scott: any updates to “make jobserver”
TrevorG: are you referring to the patch based on some changes RP made a couple
years back
Scott: yes
TrevorG: i don’t think that patch is doing much for us. i didn’t find
anything conclusive to say that patch is helping. we moved to another
approach (passing a “-l” option to ninja to limit parallel). we’ve
run a couple builds and collected some data, but we don’t have any
results to publish yet. i have a couple hundred log files to look at to
see if we’re on the right track. on the surface i think we’ve already
seen a couple instances of things taking more parallel than they should
(despite our changes) so we still have more things to dig into


Re: Per recipes simple smoke tests

Khem Raj
 

Hi Samuel

Please look at insane.bbclass and perhaps a new check can be added there ?

On Tue, Jul 13, 2021 at 12:42 AM Samuel Dolt <samuel.dolt@...> wrote:

Hi everyone,

I was asked to add some basic tests to an internal recipe in order to improve our CI system.

The purpose would be to produce an error if some check doesn't pass like
- a file is not present
- the generated file is of size 0
- the generated file is bigger than x bytes

I'm thinking to add the functionality to a class, in order to be able to use these tests in more than one recipe. Something like that: https://gist.github.com/samdolt/cf1c557f73f4f2098a19ba7ad6bc092f

Does something like that already exists? And if not, would it be useful to have this contributed upstream? Maybe as an addition to insane.bbclass?

Cheers,
Samuel


Re: [meta-spdxscanner] Question about meta-spdxscanner

Marco Cavallini
 

On 13/07/21 10:57, leimaohui@... wrote:
Hi Marco
.snip.
By the way, why not try fossology? It is really good that you can browse the compliance information on fossology server after you get spdx files by bitbake of YP.
Best regards
Lei

Hi Lei,
I tried the latest meta-spdxscanner with a Docker version cof Fossology but the entire process fails with errors on the Yocto bitbake side.

I'd like to have a minimal proof of work of this toolkit.
Would you mind sharing the version and the configuration you are using successfully (for both meta-spdxscanner and Fossology)?

Thank you
--
Marco


Yocto Project Status WW28`21

Stephen Jolley
 

Current Dev Position: YP 3.4 M2

Next Deadline: 12th July 2021 YP 3.4 M2 build

 

Next Team Meetings:

 

Key Status/Updates:

  • An issue with eSDK being host specific due to pseudo-native host dependencies has been resolved and this should resolve a number of bugs being reported by the community around use of the eSDK.
  • The prserv rewrite is still pending on resolving the issues with python asyncio.
  • We are continuing to make progress on the intermittent autobuilder failures. A race was identified in the qemu networking lock handling and there were some reproducibility fixes. Tweaks have been made to the autobuilder configuration such as avoiding key generation in more cases which it is believed may have been running into entropy shortages on the hosts, or just hitting CPU load issues. Some ptest improvements have also been made.
  • Despite this progress, we continue to see intermittent issues, particularly ptest ones. Help is very much welcome on these issues. You can see the list of failures we’re continuing to see by searching for the “AB-INT” tag in bugzilla: https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=AB-INT
  • The multiconfig changes in bitbake continue to cause problems, we still need simpler test cases to reproduce issues rather than huge builds. The existing patches seem to fix some workloads and break others and current test cases are very slow to work with.

 

Ways to contribute:

 

YP 3.4 Milestone Dates:

  • YP 3.4 M2 build date 2021/07/12
  • YP 3.4 M2 Release date 2021/07/23
  • YP 3.4 M3 build date 2021/08/23 (Feature Freeze)
  • YP 3.4 M3 Release date 2021/09/03
  • YP 3.4 M4 build date 2021/10/04
  • YP 3.4 M4 Release date 2021/10/29

 

Planned upcoming dot releases:

  • YP 3.3.2 build date 2021/07/19
  • YP 3.3.2 release date 2021/07/30
  • YP 3.1.10 build date 2021/07/26
  • YP 3.1.10 release date 2021/08/06
  • YP 3.1.11 build date 2021/09/13
  • YP 3.1.11 release date 2021/9/24

 

Tracking Metrics:

 

The Yocto Project’s technical governance is through its Technical Steering Committee, more information is available at:

https://wiki.yoctoproject.org/wiki/TSC

 

The Status reports are now stored on the wiki at: https://wiki.yoctoproject.org/wiki/Weekly_Status

 

[If anyone has suggestions for other information you’d like to see on this weekly status update, let us know!]

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


M+ & H bugs with Milestone Movements WW27

Stephen Jolley
 

All,

YP M+ or high bugs which moved to a new milestone in WW28 are listed below:

Priority

Bug ID

Short Description

Changer

Owner

Was

Became

Medium+

11906

rpmbuild: Can not build packages on qemu target

randy.macleod@...

unassigned@...

3.4 M1

3.4 M3

 

13934

Apparent duplication of libtirpc package causes failure in "bitbake linux-yocto -c menuconfig"

randy.macleod@...

unassigned@...

3.4 M1

3.4 M3

 

14123

License manifest loss when switching MACHINE

randy.macleod@...

unassigned@...

3.4

3.4 M3

 

14120

pkg_postinst_ontarget_${PN}

randy.macleod@...

unassigned@...

3.4 M1

3.4 M2

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Enhancements/Bugs closed WW28!

Stephen Jolley
 

All,

The below were the owners of enhancements or bugs closed during the last week!

Who

Count

richard.purdie@...

3

akuster808@...

2

ross@...

2

thomas.perrot@...

1

mshah@...

1

Grand Total

9

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Current high bug count owners for Yocto Project 3.4

Stephen Jolley
 

All,

Below is the list as of top 50 bug owners as of the end of WW28 of who have open medium or higher bugs and enhancements against YP 3.4.   There are 77 possible work days left until the final release candidates for YP 3.4 needs to be released.

Who

Count

ross@...

31

richard.purdie@...

29

michael.opdenacker@...

26

david.reyna@...

22

bruce.ashfield@...

18

bluelightning@...

12

sakib.sajal@...

11

timothy.t.orling@...

11

JPEWhacker@...

11

akuster808@...

11

trevor.gamblin@...

11

randy.macleod@...

10

tony.tascioglu@...

9

kai.kang@...

7

raj.khem@...

5

Qi.Chen@...

5

yi.zhao@...

5

mingli.yu@...

5

chee.yang.lee@...

5

hongxu.jia@...

4

mostthingsweb@...

3

jaewon@...

2

ydirson@...

2

yf3yu@...

2

mshah@...

2

alexandre.belloni@...

2

alejandro@...

2

mister_rs@...

1

florian.amstutz@...

1

devendra.tewari@...

1

pokylinux@...

1

shachar@...

1

Martin.Jansa@...

1

john.kaldas.enpj@...

1

douglas.royds@...

1

diego.sueiro@...

1

tonyb@...

1

mhalstead@...

1

kergoth@...

1

stacygaikovaia@...

1

nicolas.dechesne@...

1

yoctoproject@...

1

jon.mason@...

1

naveen.kumar.saini@...

1

kexin.hao@...

1

alex.kanavin@...

1

jeanmarie.lemetayer@...

1

liezhi.yang@...

1

aehs29@...

1

mark.hatle@...

1

matthewzmd@...

1

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Yocto Project Newcomer & Unassigned Bugs - Help Needed

Stephen Jolley
 

All,

 

The triage team is starting to try and collect up and classify bugs which a newcomer to the project would be able to work on in a way which means people can find them. They're being listed on the triage page under the appropriate heading:

https://wiki.yoctoproject.org/wiki/Bug_Triage#Newcomer_Bugs  Also please review: https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded and how to create a bugzilla account at: https://bugzilla.yoctoproject.org/createaccount.cgi

The idea is these bugs should be straight forward for a person to help work on who doesn't have deep experience with the project.  If anyone can help, please take ownership of the bug and send patches!  If anyone needs help/advice there are people on irc who can likely do so, or some of the more experienced contributors will likely be happy to help too.

 

Also, the triage team meets weekly and does its best to handle the bugs reported into the Bugzilla. The number of people attending that meeting has fallen, as have the number of people available to help fix bugs. One of the things we hear users report is they don't know how to help. We (the triage team) are therefore going to start reporting out the currently 360 unassigned or newcomer bugs.

 

We're hoping people may be able to spare some time now and again to help out with these.  Bugs are split into two types, "true bugs" where things don't work as they should and "enhancements" which are features we'd want to add to the system.  There are also roughly four different "priority" classes right now, “3.2”, “3.3, "3.99" and "Future", the more pressing/urgent issues being in "3.2" and then “3.3”.

 

Please review this link and if a bug is something you would be able to help with either take ownership of the bug, or send me (sjolley.yp.pm@...) an e-mail with the bug number you would like and I will assign it to you (please make sure you have a Bugzilla account).  The list is at: https://wiki.yoctoproject.org/wiki/Bug_Triage_Archive#Unassigned_or_Newcomer_Bugs

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


[yocto-autobuilder-helper] [PATCH] config.json: Use pregen-hostkeys on all qemu targets

Richard Purdie
 

Rather than just ppc/mips, use the pregen-hostkeys on all the qemu targets
since this is using a lot of time on the autobuilders when we don't really
need to. This should avoid some of the testing failures seen on qemuarm
recently.

Signed-off-by: Richard Purdie <richard.purdie@...>
---
config.json | 18 ++++++------------
1 file changed, 6 insertions(+), 12 deletions(-)

diff --git a/config.json b/config.json
index 4f6044e..1c52d60 100644
--- a/config.json
+++ b/config.json
@@ -71,6 +71,9 @@
"arch-qemu" : {
"BUILDINFO" : true,
"BUILDHISTORY" : true,
+ "extravars" : [
+ "IMAGE_INSTALL_append = ' ssh-pregen-hostkeys'"
+ ],
"step1" : {
"BBTARGETS" : "core-image-sato core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk",
"SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk"
@@ -91,6 +94,9 @@
"DISTRO" : "poky-altcfg",
"BUILDINFO" : true,
"BUILDHISTORY" : true,
+ "extravars" : [
+ "IMAGE_INSTALL_append = ' ssh-pregen-hostkeys'"
+ ],
"step1" : {
"BBTARGETS" : "core-image-full-cmdline core-image-sato core-image-sato-sdk",
"SANITYTARGETS" : "core-image-full-cmdline:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage"
@@ -383,16 +389,10 @@
},
"qemumips" : {
"MACHINE" : "qemumips",
- "extravars" : [
- "IMAGE_INSTALL_append = ' ssh-pregen-hostkeys'"
- ],
"TEMPLATE" : "arch-qemu"
},
"qemumips-alt" : {
"MACHINE" : "qemumips",
- "extravars" : [
- "IMAGE_INSTALL_append = ' ssh-pregen-hostkeys'"
- ],
"TEMPLATE" : "altcfg-qemu"
},
"edgerouter" : {
@@ -409,16 +409,10 @@
},
"qemuppc" : {
"MACHINE" : "qemuppc",
- "extravars" : [
- "IMAGE_INSTALL_append = ' ssh-pregen-hostkeys'"
- ],
"TEMPLATE" : "arch-qemu"
},
"qemuppc-alt" : {
"MACHINE" : "qemuppc",
- "extravars" : [
- "IMAGE_INSTALL_append = ' ssh-pregen-hostkeys'"
- ],
"TEMPLATE" : "altcfg-qemu"
},
"qemux86" : {
--
2.30.2


Re: [meta-spdxscanner] Question about meta-spdxscanner

Marco Cavallini
 

On 13/07/21 10:57, leimaohui@... wrote:
Hi Marco

I see that meta-spdxscanner has been moved to http://git.yoctoproject.org,
and doesn't maintained on github
Yes, you can contact me or contribute to meta-spdxscanner to this ML.

I'd like to have advice from you to understand if is it possible to test it without
any external service and discover what kind of artefacts are generated into
deploy/spdx.
I have not maintained scancode-tk for long time. And recently there are somebody else asked me about scancode-tk.
Yes, I planned to make scancode-tk.bbclass work without external service. But there are always issues because environment.
The problem of scancode-tk.bbclass that scancode must work with specify a version of python(latest requires 3.6), but you know that for YP or your host, it is hard to meet the requirement.
I found the newest version of scancode-tk support running in docker. So I plan to make scancode-tk.bbclass work with scancode command by docker next.
Of course, if you have good ideas, please tell me, or contribute to meta-spdxscanner directlly.
By the way, why not try fossology? It is really good that you can browse the compliance information on fossology server after you get spdx files by bitbake of YP.
Best regards
Lei

Hi Lei,
thank you for answering.

Considering the problems I encounter with scancode-tk and that the artifacts it produces are simple text files that need further analysis, I was just deciding to migrate to Fossology with "fossology-rest".
The only drawback is having to install the server, but I don't think it will be a problem.

Thanks again, I will contact you again if you have any problems with this mode.

Best,
--
Marco Cavallini | KOAN sas
Bergamo - Italia
embedded software engineering
https://KoanSoftware.com


Re: [meta-spdxscanner] Question about meta-spdxscanner

leimaohui
 

Hi Marco

I see that meta-spdxscanner has been moved to http://git.yoctoproject.org,
and doesn't maintained on github
Yes, you can contact me or contribute to meta-spdxscanner to this ML.

I'd like to have advice from you to understand if is it possible to test it without
any external service and discover what kind of artefacts are generated into
deploy/spdx.
I have not maintained scancode-tk for long time. And recently there are somebody else asked me about scancode-tk.
Yes, I planned to make scancode-tk.bbclass work without external service. But there are always issues because environment.
The problem of scancode-tk.bbclass that scancode must work with specify a version of python(latest requires 3.6), but you know that for YP or your host, it is hard to meet the requirement.
I found the newest version of scancode-tk support running in docker. So I plan to make scancode-tk.bbclass work with scancode command by docker next.
Of course, if you have good ideas, please tell me, or contribute to meta-spdxscanner directlly.

By the way, why not try fossology? It is really good that you can browse the compliance information on fossology server after you get spdx files by bitbake of YP.

Best regards
Lei


-----Original Message-----
From: yocto@... <yocto@...> On Behalf
Of Marco Cavallini
Sent: Monday, July 12, 2021 9:09 PM
To: yocto@...
Cc: Lei, Maohui <leimaohui@...>
Subject: [yocto] [meta-spdxscanner] Question about meta-spdxscanner

Hello,
I see that meta-spdxscanner has been moved to http://git.yoctoproject.org,
and doesn't maintained on github
anymore.https://github.com/dl9pf/meta-spdxscanner/issues/21

Therefore the only way to get in contact to developers is writing to this ML

I am trying to understand how meta-spdxscanner works.
I'd like to test it without any external service (no Fossology) therefore I am
trying the INHERIT += "scancode-tk" approach.

I am testing it with a dunfell version and I noticed a lot of errors so I switched to
master and the bitbake build started but again I am facing to a problem

ERROR: bash-completion-2.10-r0 do_get_report: Could not invoke scancode
Command

I'd like to have advice from you to understand if is it possible to test it without
any external service and discover what kind of artefacts are generated into
deploy/spdx.

Thank you

--
Marco

3721 - 3740 of 57813