YP AB Intermittent failures meeting
===================================
Sept 9, 2021, 9 AM ET
https://windriver.zoom.us/j/3696693975

Attendees: Kiran, Richard, Trevor, Randy!


Summary:
========

Ptest results continue to improve but there's still room
for even more improvement.

The make/ninja load average limit is in but it's not clear
if it's effective yet and it breaks dunfell. Trevor investigating
(Aug 26!, Sept 9). Trevor is going to get back to this!


If anyone wants to help, we could use more eyes on the logs,
particularly the summary logs and understanding iostat #
when the dd test times out.



Plans for the week:
===================

Richard: QA results for M3, etc.
Alex: SWAT plans. September email, training. libevent monotonic fix!
Sakib: hook more responsive load average in to latency test. (v3)
Trevor: patch to set PARALLEL_MAKE : -l 50
-> dunfell, gatesgarth, hardknott (Aug 5 - it's a priority)
Investigate dunfell which failed with this change.
- data on WR AB load average.
Saul: SBOM
Randy: Vacation
Kiran: probably work on one of the ptest failures.


Nothing much new below here. Keeping the list since it's still to-do.

../Randy

Meeting Notes:
==============

1. job server

- ninja could be patched with make's more responsive algorithm
next or is this good enough?

Aug 26:
Randy made some graphs that show that the -l NUM results
in the number of compile jobs oscillates *wildly* between 0 and 200
on a 192 core builder compiling chromium. What I did was:
$ bitbake -c cleansstate chromium-x11
$ bitbake -c configure chromium-x11
$ bitbake -c compile chromium-x11
and while that compile was running:
$ while [ ! -f /tmp/compiling-chromium-is-done ]; do \
cat /proc/loadavg >> procs-load.log ; sleep 0.5 ;
done
Results so far:
https://postimg.cc/gallery/3hjfYfG/f8f46c97
Next step is either:
a. collect data as above for an image build and see if the sub-optimal
ninja behaviour makes a difference
and/or
b. patch ninja with make's more responsive load avg
algorithm:
https://git.savannah.gnu.org/cgit/make.git/commit/?id=d8728efc8


- Richard suggested that we extract make's code for measuring the load
average to a separate binary and run it in the periodic io latency
test. Also can we translate it to python?
- Trevor is working on this and had some problems so next week.
(Aug 19 - Trevor is back from vaction so maybe next week.)

- Trevor to see if the load average change really did reduce load
on WR build systems. (Aug 19)

2. AB status

Trevor is learning about buildbot and working on a scheduling bug
(CentOS worker?)

bitbake layer setup tool should allow multiple backends:
eg: kas, a y-a-helper.

ptest cases are improving, we may be close to done!
Let's wait a week to see how things go.
(July29, Aug 5, Aug 19, we're not done...)

- lttng-tools ptest is failing. RP is working on it with upstream.
The timeout (done on Aug 5) increase hasn't helped.


3. Sakib's improvements to the logging are merged.

Sakib generated a summary of all high latency 'top' logs from
~July 23->July 29 by just running his summary script on the
merged raw top logs.

More analysis required....


Still relevant parts of
Previous Meeting Notes:
=======================


4. bitbake server timeout ( no change july 29, Aug 19)

"Timeout while waiting for a reply from the bitbake server (60s)"

5. io stalls (no update: July 29)

Richard said that it would make sense to write an ftrace utility
/ script to monitor io latency and we could install it with sudo
Ch^W mentioned ftrace on IRC.
Sakib and Randy will work on that but not for a week or two
or longer! (Aug 19).

Randy collected iostat data on 3 build server:
https://postimg.cc/gallery/8cN6LYB
We agreed that having -ty-2 be ~ 100 utilization for many hours
in a row is not acceptable and that a threshold of ~ 10 minutes
at 100% utilization may be a reasonable limt. I need to figure out
if I can get data on the fraction of IO done per IO clas since
we do use ionice to do clean-up and other activities.


../Randy

- Randy to move AB defects to M4 after call

- Tim to file a bug about improving ptest runner timeout warnings

- Stephen to handle 3.4 M3 unassigned bugs (move to M4 or 3.5 M1 if enhancements)

- Trevor to ask Alex Kanavin if he could help on 14518

This prioritizeBuilders function sorts builders by the length of their
associated worker lists, so that builders that can only be assigned to a
small number of workers are assigned to those workers before other
builds that don't have specific needs when resources are limited. An
example might be when a slot is available on an Ubuntu-based worker, and
"oe-selftest-ubuntu" and "genericx86-64" build requests exist in the
queue. Since oe-selftest-ubuntu requires an Ubuntu-based worker and
genericx86-64 does not, genericx86-64 will be assigned a higher value
(lower priority) so that oe-selftest-ubuntu is assigned to that worker
first.

Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
---
builders.py | 13 +++++++++++++
master.cfg | 6 ++++++
2 files changed, 19 insertions(+)

diff --git a/builders.py b/builders.py
index 53c5f0e..94fb561 100644
--- a/builders.py
+++ b/builders.py
@@ -135,6 +135,19 @@ for builder in config.subbuilders:
workernames=workers, nextWorker=nextWorker, nextBuild=nextBuild,
factory=f, env=extra_env))

+# prioritize assigning builders to available workers based on the length
+# of the worker lists they are associated with. Builders that have fewer
+# valid worker options should always be assigned first
+def prioritizeBuilders(buildmaster, builders):
+ # re-use the builder_to_workers list
+ builder_to_workers = config.builder_to_workers
+
+ # sort builders by the length of their worker lists. Since not all
+ # builders are explicitly listed in builder_to_workers, make sure to
+ # default to the len() of the "default" value
+ builders.sort(key=lambda b: len(builder_to_workers.get(b.name)) if b.name in builder_to_workers.keys() else len(builder_to_workers.get("default")))
+ return builders
+
def create_parent_builder_factory(buildername, waitname):
factory = util.BuildFactory()
# NOTE: Assumes that yocto-autobuilder repo has been cloned to home
diff --git a/master.cfg b/master.cfg
index a7c151f..4f7d74e 100644
--- a/master.cfg
+++ b/master.cfg
@@ -88,6 +88,12 @@ c['www'] = www.www
# These items are specific to an individual AB deployment
c['workers'] = workers.workers

+# This enables our prioritizeBuilders function from builders.py.
+# Builders such as buildperf-centos7, buildperf-ubuntu1604,
+# oe-selftest-*, and reproducible-* will be assigned (if possible)
+# before other builders since their possible worker lists are smaller
+c['prioritizeBuilders'] = builders.prioritizeBuilders
+
c['title'] = "Yocto Autobuilder"
c['titleURL'] = "https://autobuilder.yoctoproject.org/main/"
# visible location for internal web server
--
2.31.1

Hi,

1. I'm aware that yocto picks up packages from respective open source community. For e.g. dbus package is picked from freedesktop.org etc. In this regard, what kind of tests does yocto runs to ensure that the package that is getting picked-up is stable enough?
2. Is it Yocto, that writes pTest's to ensure that the package that is getting picked up is stable enough?
3. I could find pTest report for e.g. on Dunfell @ http://downloads.yoctoproject.org/releases/yocto/yocto-3.1.1/testreport.txt (Refer "qemuarm64 PTest Result Summary" or "qemux86-64 PTest Result Summary" section for details). The test report only details the number of pTest's that passed, failed or skipped for respective package. Is there any report that details the pTest's, which failed, passed or skipped?

Additionally, I'm looking for pTest's for systemd, udev and glibc packages.
Below are my observations on these packages w.r.t pTests.

1. pTest for systemd package has been removed with http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/meta/recipes-core/systemd?h=dunfell&id=40db00818e3b1cce8aa85ee30f3081ae9a5f8e66 commit. 
2. Similarly pTest for udev has been removed with http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/meta/recipes-core?h=dunfell&id=3e5e540513665105b963262c2eaf33f197a0a36c.
3. I don't find ptest's for glibc package either.

As there aren't any pTests for these three packages, are there any other existing tests that can be run to validate these packages?

Thanks,
Ravi

Hi All,

This is the full report for yocto-3.4_M3.rc1:
https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/tree/?h=intel-yocto-testresults

======= Summary ========
No high milestone defects.

No new issue found.


Thanks,
Jay

Hello, 
I cant find how to add maybe graphic conf for the yocto image I already have for the raspberrypi, 
Any help? 

--
Regards,
Sateesh

Hi,

I'm really not used to this e-mail process :-D
forwording my original answer to the mailing list...

BR Max

On Tue, 2021-09-07 at 10:43 -0700, Rudolf J Streif wrote:

What does your 'Hello World' service do? Just print 'Hello World' to
the console and then exit?

If so that would be the reason why it is inactive. Systemd starts it
and then it exits. Typically, services keep running (except for some
special cases) once they are started. That's what makes them a service.
To stop them 'systemctl stop <service>' is used. By default systemctl
sends SIGTERM to tell the service to clean up and terminate itself.

To piggyback on that, it is possible to set Type=oneshot[1] for the
service so that systemd considers the service to be up after it exited.
I find it useful for init or migration scripts that another service can
then depend upon.

[1]:
https://www.freedesktop.org/software/systemd/man/systemd.service.html#Type=

:rjs


On Tue, Sep 7, 2021, 10:20 <yasminebenghozzi6@...> wrote:

Hi everyone, 

I wanted to run the helloworld service, but it keeps telling that its
inactive , I really cant find why, I changed the service many times
but no solution.
I m sorry I ant copy paste from putty

For the error,  i have tried installed below package in aws container
libgdk-pixbuf2.0-0 in container,
and added DEPENDS += "qemuwrapper-cross" in bb file.

Will try the option building "core-image-minimal" or "core-image-sato"

Regards,
Vijay

Raj thanks for the response. Couldnt able to find error from the logs..

Attached detailed logs..

On Tue, 2021-09-07 at 06:10 +0000, Groeting, Claus wrote:

Hi all!

I have a problem with this commit, applied in poky:dunfell on file toolchain-
shar-relocate.sh.
sha1: d6f40be29bf56a835f5825692a22365f04aeb6c3

Author: hongxu <hongxu.jia@...>  2021-08-10 15:33:56
Committer: Richard Purdie <richard.purdie@...>  2021-08-26
09:32:18
Parent: dad6dd12fd6e2512f3bd5591327ee35ee22b913e (toolchain-shar-relocate.sh:
Fix handling files with colons)
Branches: dunfell, remotes/origin/dunfell
Follows: dunfell-23.0.10, yocto-3.1.10
Precedes:

    sdk: fix relocate symlink failed
   
   
    (From OE-Core rev: a155c4d4ca841ef492213a5f0b9121703c62e2b4)
   
    Signed-off-by: Hongxu Jia <hongxu.jia@...>
    Signed-off-by: Alexandre Belloni <alexandre.belloni@...>
    Signed-off-by: Richard Purdie <richard.purdie@...>
    (cherry picked from commit 942c06a7348070b92f722fa5c439c8c4404485b7)
    Signed-off-by: Steve Sakoman <steve@...>
    Signed-off-by: Richard Purdie <richard.purdie@...>

-------------------- meta/files/toolchain-shar-relocate.sh --------------------
index 5433741296..ba873373e2 100644
@@ -72,7 +72,7 @@ fi
 
 # change all symlinks pointing to @SDKPATH@
 for l in $($SUDO_EXEC find $native_sysroot -type l); do
- $SUDO_EXEC ln -sfn $(readlink $l|$SUDO_EXEC sed -e
"s:$DEFAULT_INSTALL_DIR:$target_sdk_dir:") $l
+ $SUDO_EXEC ln -sfn $(readlink $l|$SUDO_EXEC sed -e
"s:$SDK_BUILD_PATH:$target_sdk_dir:") $l
  if [ $? -ne 0 ]; then
  echo "Failed to setup symlinks. Relocate script failed. Abort!"
  exit 1

it looks to me that SDK_BUILD_PATH is not defined in the sed instruction. When
installing the generated SDK shell script, we see a lot of:
sed: -e expression #1, char 0: no previous regular expression
sed: -e expression #1, char 0: no previous regular expression
sed: -e expression #1, char 0: no previous regular expression
sed: -e expression #1, char 0: no previous regular expression
sed: -e expression #1, char 0: no previous regular expression
sed: -e expression #1, char 0: no previous regular expression

Without that commit, everything looks ok. On master-branch SDK_BUILD_PATH is
preset with "@SDKPATH@" but not on dunfell-branch this is missing in the
file toolchain-shar-relocate.sh.

Hi,

I think there is a patch queued in the testing branch for dunfell:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=stable/dunfell-nut


http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=stable/dunfell-nut&id=ed0f6236ceb6a127903241f783a526bcdefc66ce

that should address this.

Cheers,

Richard

Hi everyone, 

I wanted to run the helloworld service, but it keeps telling that its inactive , I really cant find why, I changed the service many times but no solution.
I m sorry I ant copy paste from putty

That is exactly what I thought and why I insisted that much.17:38, 7 September 2021, Ross Burton <ross@...>:

On Tue, 7 Sept 2021 at 11:22, KL <deco33000@...> wrote:

Thanks does that mean that I can reshape the whole "everything is a file" under unix?

I don't want /etc /var /usr (and any other premade structure you find in every linux distro...) but manage it differently via our db.

You confirm that doable?

Doable? Yes, everything is open source.

Sensible? Not really. I'm not sure you've considered the scale of
what you're asking. For example, /etc/passwd and /etc/hosts are deeply
tied into large amounts of software.

You can make *your* software use whatever configuration database you
want, but you'll have a lot of work to do if you want to move the user
database or DNS resolver information out of /etc.

Ross