refactoring existing (legacy) code base into more bite-sized
bitbake recipes to speed up build by taking advantage of 6-core
(12-thread) dell laptop ... end result is that i get so much
parallelism that laptop shuts down from overheating.

le *sigh* ...

rday

Hi Frédéric,

Thanks for your suggestion,

you can customize QB_OPT_APPEND by removing options you don't want, e.g.

QB_OPT_APPEND_remove = "-device usb-tablet"

and add your own after.

I suppose this would be a bit less maintainable, as it's effectively
duplicating the entry in the base config... but I think it will work
for me for now. Thank you.

Regards,
--Mike

Greetings Yocto,

I'm interested in assigning custom config settings for machine type qemuarm.

I tried a method suggested in this thread
https://lists.yoctoproject.org/g/yocto/message/38359 by Ayoub Zaki, he
suggested defining the following in local.conf

include conf/machine/${MACHINE}-extra.conf

However, this doesn't work for me because quemuarm.conf in poky/meta
**assigns** its variables, e.g.,

QB_OPT_APPEND = "-device VGA,edid=on"
QB_OPT_APPEND += "-device qemu-xhci -device usb-tablet -device usb-kbd"

So that if I define a value for this, it will be overwritten.

Is there another method for achieving this? If not I could submit a
patch to change poky's definitions to "??" assignments.

Cheers,
--Mike

Now that there is a second rk3328-based MACHINE (rock64) switch rock-pi-e to
use the common rk3328 include.

Signed-off-by: Trevor Woerner <twoerner@...>
---
conf/machine/rock-pi-e.conf | 21 +++------------------
1 file changed, 3 insertions(+), 18 deletions(-)

diff --git a/conf/machine/rock-pi-e.conf b/conf/machine/rock-pi-e.conf
index 035a950..38362a0 100644
--- a/conf/machine/rock-pi-e.conf
+++ b/conf/machine/rock-pi-e.conf
@@ -3,29 +3,16 @@
#@DESCRIPTION: ROCK Pi E is a Rockchip RK3328-based SBC by Radxa. E is for Ethernets.
#https://wiki.radxa.com/RockpiE

-MACHINEOVERRIDES =. "rock-pi-e:"
-SOC_FAMILY = "rk3328"
-DEFAULTTUNE = "cortexa53-crypto"
+require conf/machine/include/rk3328.inc

-require conf/machine/include/soc-family.inc
-require conf/machine/include/tune-cortexa53.inc
-require conf/machine/include/rockchip-defaults.inc
+MACHINEOVERRIDES =. "rock-pi-e:"

PREFERRED_PROVIDER_virtual/kernel = "linux-stable-bleeding"
KERNEL_DEVICETREE = "rockchip/rk3328-rock-pi-e.dtb"
-KBUILD_DEFCONFIG = "defconfig"
-KERNEL_CLASSES = "kernel-fitimage"
-KERNEL_IMAGETYPE = "fitImage"
MACHINE_EXTRA_RRECOMMENDS += "kernel-modules"

-TFA_PLATFORM = "rk3328"
-TFA_BUILD_TARGET = "bl31"
-
-UBOOT_MACHINE = "rock-pi-e-rk3328_defconfig"
-UBOOT_SUFFIX = "itb"
-UBOOT_ENTRYPOINT = "0x06000000"
PREFERRED_PROVIDER_virtual/bootloader = "u-boot"
-SPL_BINARY = "idbloader.img"
+UBOOT_MACHINE = "rock-pi-e-rk3328_defconfig"

WKS_FILE = "rock-pi-e.wks"
IMAGE_FSTYPES += "wic.xz wic.bmap"
@@ -38,5 +25,3 @@ WKS_FILE_DEPENDS = " \
IMAGE_BOOT_FILES ?= " \
${KERNEL_IMAGETYPE} \
"
-
-SERIAL_CONSOLES = "1500000;ttyS2"
--
2.30.0.rc0

Applied to meta-rockchip master. Thanks! :-)

On 6/18/21 5:16 AM, Sekine Shigeki wrote:

CVE-2014-0363, CVE-2014-0364, CVE-2016-10027 are not for smack of smack-team(https://github.com/smack-team/smack) but other project.

Thanks. So this is for meta-security layer based on version.

- armin

Signed-off-by: Sekine Shigeki <sekine.shigeki@...>
---
recipes-mac/smack/smack_1.3.1.bb | 5 +++++
1 file changed, 5 insertions(+)

diff --git a/recipes-mac/smack/smack_1.3.1.bb b/recipes-mac/smack/smack_1.3.1.bb
index b1ea4e9..6ae715e 100644
--- a/recipes-mac/smack/smack_1.3.1.bb
+++ b/recipes-mac/smack/smack_1.3.1.bb
@@ -13,6 +13,11 @@ SRC_URI = " \

PV = "1.3.1"

+# CVE-2014-0363, CVE-2014-0364, CVE-2016-10027 is valnerble for other product.
+CVE_CHECK_WHITELIST += "CVE-2014-0363"
+CVE_CHECK_WHITELIST += "CVE-2014-0364"
+CVE_CHECK_WHITELIST += "CVE-2016-10027"
+
inherit autotools update-rc.d pkgconfig ptest
inherit ${@bb.utils.contains('VIRTUAL-RUNTIME_init_manager','systemd','systemd','', d)}
inherit features_check

CVE-2014-0363, CVE-2014-0364, CVE-2016-10027 are not for smack of smack-team(https://github.com/smack-team/smack) but other project.

Signed-off-by: Sekine Shigeki <sekine.shigeki@...>
---
recipes-mac/smack/smack_1.3.1.bb | 5 +++++
1 file changed, 5 insertions(+)

diff --git a/recipes-mac/smack/smack_1.3.1.bb b/recipes-mac/smack/smack_1.3.1.bb
index b1ea4e9..6ae715e 100644
--- a/recipes-mac/smack/smack_1.3.1.bb
+++ b/recipes-mac/smack/smack_1.3.1.bb
@@ -13,6 +13,11 @@ SRC_URI = " \

PV = "1.3.1"

+# CVE-2014-0363, CVE-2014-0364, CVE-2016-10027 is valnerble for other product.
+CVE_CHECK_WHITELIST += "CVE-2014-0363"
+CVE_CHECK_WHITELIST += "CVE-2014-0364"
+CVE_CHECK_WHITELIST += "CVE-2016-10027"
+
inherit autotools update-rc.d pkgconfig ptest
inherit ${@bb.utils.contains('VIRTUAL-RUNTIME_init_manager','systemd','systemd','', d)}
inherit features_check
--
2.25.1

On 17/06/2021 10:05:48-0400, Randy MacLeod wrote:

5. On the ubuntu-18.04 builders, we seem to see issues there,
we dont' know why, maybe only that we have more of those workers...
Alex, could you possibly get failures per worker statistics?

Specifically for bug 14273 (the rcu stall is seen on the logs):

ubuntu1804-ty-1 6 23.1%
ubuntu1804-ty-2 5 19.2%
ubuntu1804-ty-3 12 38.5%
fedora31-ty-1 2 7.7%
debian8-ty-1 3 11.5%

6. discussed Sakib's summary script. It's coming along.
TO DO:
- special activities: rm (of trash), tar, qemu*
- report all zombies
(The current hoard are due to Paul Barker's patch)
7.
make: job server
- the fifo was being re-created by the wrapper on each call
so Trevor will fix that.



8. From last week, I don't think we've increased the timeouts:

- qemu-runner? timeout increase 120 -> 240
- ptest timeouts 300 -> 450?




8.
Plans for the week:

Richard: RCU stall
Alex:
Sakib: task summary
Trevor: make job server
Tony: ptests and work with upstream valgrind on fixing bugs.
Saul: (1 week) have QMP deal with sigusr1 to close the QMP socket
Randy: coffee, herd cats!!


../Randy

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

It it can help to understand, I rechecked the full content of source packages and glib-2.0 / networkamanager are not ok like I said earlire.
There are not empty but they seem to contain only build generated files (for example glib/glibconfig.h which, in the glib source repository, is glib/glibconfig.h.in).

Hello,

I currently used yocto warrior and I encounter an issue during packaging of recipes.
I would like to embed some source package (to be able to run coverage on target) , this work without issues and places the source of the package in ${ROOTFS}/usr/src/debug, but when I 'devtool modify' a recipe to work on , the source package is totally empty.

First, I suspected that was my various recipe customization so I tried the same method (devtool modify then packing it and look at content of -src package) for more standard package, here are my results:
     - glib-2.0 -> ok
     - networkmanager -> ok
     - dnsmasq -> empty source
     - zlib -> empty source

I try to look at log.do_package but I didn't see any warning nor error there.

Since warrior is EOL, I tried a fresh poky on dunfell and hardknott, I got the exact same results. Is anybody here have experienced this ? Is this bug or did I miss some configuration ?

Thank you very much in advance for any help you can bring.

Yocto Technical Team Minutes, Engineering Sync, for June 15 2021
archive: https://docs.google.com/document/d/1ly8nyhO14kDNnFcW2QskANXW3ZT7QwKC5wWVDg9dDH4/edit

== disclaimer ==
Best efforts are made to ensure the below is accurate and valid. However,
errors sometimes happen. If any errors or omissions are found, please feel
free to reply to this email with any corrections.

== attendees ==
Trevor Woerner, Stephen Jolley, Jan-Simon Möller, Steve Sakoman, Randy
MacLeod, Joshua Watt (JPEW), Michael Opdenacker, Peter Kjellesrstedt,
Richard Purdie, Tim Orling, Tony Tascioglu, Trevor Gamblin, Denys
Dmytriyenko, Bruce Ashfield, Michael Halstead, Ross Burton, Saul Wold

== notes ==
- 3.4 m1 built and in QA (honister)
- m1 unblocked, we thought it was caused by kernel stuff, but it’s going to
take longer
- still working on AB-INT centos8 issue
- multiconfig issues continue, need simpler test cases
- ongoing discussion of potential new bitbake assignment operator (see
architecture mailing list)
- pr-serv updates from PaulB are revealing issues with shutdown, hanging
threads, etc
- uptick in CVEs (up to 17, was down to 4 at one point) thanks to RossB for
help (please join in)

== general ==
TimO: i was looking at one of the CVEs for dunfell, there was a collision in the patch that we’re working on


Randy: Tony, did you send the email regarding ffmpeg
Tony: not yet, but i sent the 2nd part of valgrind fixes
Randy: there are many CVEs out for ffmpeg that Tony is looking at


TrevorG: we’re using kia as the test package, looking at how the job server
works, collecting data. if you run with X jobs but add the debug flag it
spits out messages about needing a token
Randy: 2 builds, both building kia, we should see a difference in the amount
of time taken but we’re not seeing that
TrevorG: working on gathering more data
Randy: do you want the patch today
RP: seems to be confusion over the results so lets sort that out. we’ve
been looking at how make does its job server stuff and we’re looking to
integrating it into our builds (it sets up pipes and writes tokens to the
pipe and the sub builders read a token, work on stuff, then put the token
back)
RP: hopefully this helps with the AB-INT issues, and if we can solve the
kernel stuff wrt AB-INT then we’ll be in a good position


Randy: can we disable sstate generation?
RP: not that simple, sstate does a bunch of stuff, recipe-specific sysroots,
for example, so some parts of the sstate generation have to be used
regardless, the only bit you can disable is the final creation of the
sstate object, or we could zero out the final function call. what would
blow up if we did? don’t know, won’t look into it. i wont be accepting
patches for that. the gains you think you might see won’t be worth it
Randy: okay, i’ll look into it. if it’s less than 1 or 2% then i won’t
care. my use-case is that we clean up everything after each build, so why
create it in the first place
RP: you’ll see problems with multiconfig and in other places, and i’m not
convinced it’s worth the pain
JPEW: doesn’t he also have to disable siginfodata generating?
RP: probably, this is why i don’t want to go there


JPEW: been working on SPDX stuff. we have our own view of the world and
that’s what we need to include in the SPDX.
RP: i think we’re going to see more announcements soon (external tools etc)
JPEW: we are going to the plugfest in 2 weeks
RP: but you’ve missed the deadline
JPEW: i submitted one
RP: (needs to follow up)
JPEW: maybe i didn’t put it in the right place. i’ll double-check.
RP: i’ll give you Kate’s email and coordinate with her. getting involved
would be good
JPEW: looking at making meta-doubleopen better integrated into oe-core
RP: it feels like a lot of patches and hacks, which worries me
JPEW: a lot of it was making it do sstate properly. the big thing is the
archiving of the sources which i think will make use of your sources
changes. the changes to oe-core were minimal (debug sources)
RP: if we could pipe that through lz4 and compress it
JPEW: we could just add it as a host package (dependency)
RP: that would be simpler than doing it through python APIs
JPEW: agreed
RP: maybe we could save off base hash
MichaelO: why lz4?
JPEW: it’s the fastest of the ones we’ve looked at for this type of data.
zstd is more configurable, but lz4 is just pinned to be fast
MichaelO: if speed matters then i agree lz4 is the fastest
RP: we’ve looked at a couple and had better compression with some others,
but lz4 always ended up being the fastest. we even looked at having
bitbake start the next task before finishing the compression, but that
didn’t work out
JPEW: i think zstd has parallel compressing so we could use multiple threads.
and with configurability we can change between size and speed
RP: with the testing i did i also found gz worked well in some cases. i’m
open to any ideas here if there’s data to support it. i’m not sure
what’s in centos7, but we use buildtools tarball anyway
MichaelO: i can do some testing, were is it set?
RP: it’s not parameterized, but look in the sstate bbclass and do a
search+replace of tgz, then do a test and compare size and time
JPEW: we use pigz if it’s available
RP: right. it’s smart enough to start with gz and then use pigz if available
JPEW: i think zstd has the same, there’s pzstd for the parallel version


JPEW: compressed package data, debuginfo, do you want it debuginfo or do
you want to be able to put other things in there, or replicate what’s in
the package data
RP: i assume you’re going to create a new file
JPEW: yes
RP: then let’s have an api that says “this file has this data in it”
Ross: is it time to consider a package data v2 (e.g. with compressed json)
because i was looking at adding more info into package data too and it was
bad
Saul: it could be configurable. it could be packagedata but all in one file
RP: the “configurable” thing is problematic. a 2nd build with a different
config won’t be able to reuse sstate info. but adding debuginfo is large
(hence compression). i can see what Ross would like because internally
using that information to regenerate its own internal state is quite fast
Saul: sorry, i meant what’s output could be configurable
RP: but what JPEW wants is adding to the packagedata. there’s the top-level
file but there’s also data in the runtime directory and the two are
linked. i’m fine with moving it to json (or some other format) but we
need to make sure we can access the subsets so it doesn’t become 1 huge
json file, but a file with sets. i’m thinking out loud (these might not
work). it goes into sstate as individual entries but then gets splattered
like a sysroot. the nice thing is it is lockless, but with individual
files maybe locking becomes an issue? although with package data being
recipe-specific it might avoid any issues.
JPEW: then there’s the hard links too
RP: it only hardlinks to things it depends on
JPEW: knowing what we need from the packagedata is the first step
RP: some of that is obsolete because of recipe-specific sysroots
RP: what do people think of making a hardlink of the sources? the downside is
having hardlinked directory trees that take a long time to delete.
JPEW: how are we deleting them?
RP: we’ve tried a couple things, but they’re all slow because fs people
don’t care about that usecase
JPEW: are you thinking of moving the place where we checkout to in order to
make cleanup easier?
RP: there are a number of reasons, currently there is no simple place to point
to to find the sources for a build (think devtool)
JPEW: the archiver is very configurable, would you save off the copy before or
after do_configure?
RP: good question. i think we might have to ask people what they want. it
would depend on what their legal departments might think. are there any
preferences?
Randy: we’ve gone back and forth, but i think we save off the patched ones
now
RP: i do have patches for playing with saving off the sources, and they do
produce a build, but as soon as you touch it with devtool (for example) it
explodes, so if anyone wants to play with it they are available

oe-core has mpg123 already, so just use that recipe.

Ross

Greetings !

If anyone achieved the task of integrating the "mpg321" in yocto , kindly requesting to share the procedure/steps to be followed .

Thanks

On 2021-06-16 1:33 p.m., Richard Purdie wrote:

[Please note: This e-mail is from an EXTERNAL e-mail address]

On Wed, 2021-06-16 at 04:43 -0400, sakib.sajal@... wrote:

summarize_top_output.py is used to summarize the top
output that is captured during autobuilder intermittent
failures.

Use the script to summarize the host top output and
publish the summary that is created instead of
the raw logfile.

[...]

if jcfg:
diff --git a/scripts/summarize_top_output.py b/scripts/summarize_top_output.py
new file mode 100755
index 0000000..0606a34
--- /dev/null
+++ b/scripts/summarize_top_output.py
@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+
+import os, sys, glob
+
+# constants
+top_header = 7
+top_summary = 5
+max_cols = 11
+
+# string substitution to make things easier to read
+subs = {
+ "/home/pokybuild/yocto-worker/" : "~/",
+ "/build/build/tmp/work/core2-32-poky-linux/" : "/.../POKY_32/.../",
+ "/build/build/tmp/work/core2-64-poky-linux/" : "/.../POKY_64/.../",
+ "/recipe-sysroot-native/usr/bin/x86_64-poky-linux/../../libexec/x86_64-poky-linux/gcc/x86_64-poky-linux/" : "/...GCC.../"
+}

One quick question - the above assumes an x86 target machine using those two tunes.
Should that be wildcarded?

Cheers,

Richard

Yes it should be. We've looked at a number of logs and I will send a V2 with wild-carding for the cross-compiler paths.

After the V2, I plan on submitting patches for the following in roughly the presented order:

1) any process that is listed once (singleton) is not reported in the summary, but we are planning to make exceptions for "special" commands such as "rm" and "tar"

2) i noticed some zombie processes and it might be worthwhile to report them. Right now they are not indicated in the summary. Should we report all zombies or ignore the singletons?

For example I'd collapse [Parser-31], [Parser-32] into [Parser-N]


3) It would be nice to specify other builds going on and the top level directory

4) currently top does not show PPID, should we include the information in the output? This is useful for more context in the raw logfile.

5) show top 5 or 10 virtual memory users


From the yp-ab index page, there are no logs for arm host. I will run a build and make sure the script work well there as well.

Any ideas on why the arm host builds are not being shown on the index page?

Sakib

Hello All,

I'm trying to get a Perl module (specifically
Google-ProtocolBuffers-Dynamic @
https://metacpan.org/pod/Google::ProtocolBuffers::Dynamic) cross
compiled for an imx8 target; and having a bit of difficulty doing so.

I was hoping someone could provide some clues/examples/direction on cross
compiling Perl modules under Yocto (specifically those that interface
with underlining c/c++ code like this one).

As a note, I manually futzed around with it and I was able to cross
compile the Perl module manually with things like `perl Build.PL
--config cc=.. --config ld=..` and got a library that I placed on the
target. Attempting to use it though, gives a error of "loadable library
and Perl binaries are mismatched".

Thank you in advance for any help!

Marco

Join Zoom Meeting - 9 AM ET
https://windriver.zoom.us/j/3696693975

Attendees: Alex, Richard, Saul, Randy, Tony, Trevor, Sakib


Summary: Things are improving somewhat on the autobuilder,
RCU stalls are the top problem now.


1.
LTP kernel BUG:
Many thanks to Paul Gortmaker for his work on this!


2. The most common problem now is the qemu RCU hang.

For example these builds:

https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/3541/steps/13/logs/stdio

https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/3541/steps/13/logs/stdio

Richards links on RCU stall detection, and tuning parameters:
https://www.kernel.org/doc/Documentation/RCU/stallwarn.txt
https://lwn.net/Articles/777214/

Next:
- Ask around for advice on qemu debugging.
- RP thinks that the underlying system has a problem:
CPU or other overload.
We do see that there are two qemus that are using lots of CPU in
the links above.
Richard says that the likely activity is:
- core-image-sato-sdk, compiler tests
- core-image-sato lighter general tests
Alex thinks that the particular workload is not significant.
- run two qemu in a controlled env, with stress-ng.
- iostat will help - Sakib.

3. Valgrind ptest results are getting better.

4. ptest issues are coming along, with util-linux being the next
thing to be merged today likely.

5. On the ubuntu-18.04 builders, we seem to see issues there,
we dont' know why, maybe only that we have more of those workers...
Alex, could you possibly get failures per worker statistics?


6. discussed Sakib's summary script. It's coming along.
TO DO:
- special activities: rm (of trash), tar, qemu*
- report all zombies
(The current hoard are due to Paul Barker's patch)
7.
make: job server
- the fifo was being re-created by the wrapper on each call
so Trevor will fix that.



8. From last week, I don't think we've increased the timeouts:

- qemu-runner? timeout increase 120 -> 240
- ptest timeouts 300 -> 450?




8.
Plans for the week:

Richard: RCU stall
Alex:
Sakib: task summary
Trevor: make job server
Tony: ptests and work with upstream valgrind on fixing bugs.
Saul: (1 week) have QMP deal with sigusr1 to close the QMP socket
Randy: coffee, herd cats!!


../Randy