Join Zoom Meeting - 9 AM ET
https://windriver.zoom.us/j/3696693975

Attendees: Alex, Richard, Saul, Randy, Tony, Trevor, Sakib


Summary: Things are improving somewhat on the autobuilder,
RCU stalls are the top problem now.


1.
LTP kernel BUG:
Many thanks to Paul Gortmaker for his work on this!


2. The most common problem now is the qemu RCU hang.

For example these builds:

https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/3541/steps/13/logs/stdio

https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/3541/steps/13/logs/stdio

Richards links on RCU stall detection, and tuning parameters:
https://www.kernel.org/doc/Documentation/RCU/stallwarn.txt
https://lwn.net/Articles/777214/

Next:
- Ask around for advice on qemu debugging.
- RP thinks that the underlying system has a problem:
CPU or other overload.
We do see that there are two qemus that are using lots of CPU in
the links above.
Richard says that the likely activity is:
- core-image-sato-sdk, compiler tests
- core-image-sato lighter general tests
Alex thinks that the particular workload is not significant.
- run two qemu in a controlled env, with stress-ng.
- iostat will help - Sakib.

3. Valgrind ptest results are getting better.

4. ptest issues are coming along, with util-linux being the next
thing to be merged today likely.

5. On the ubuntu-18.04 builders, we seem to see issues there,
we dont' know why, maybe only that we have more of those workers...
Alex, could you possibly get failures per worker statistics?


6. discussed Sakib's summary script. It's coming along.
TO DO:
- special activities: rm (of trash), tar, qemu*
- report all zombies
(The current hoard are due to Paul Barker's patch)
7.
make: job server
- the fifo was being re-created by the wrapper on each call
so Trevor will fix that.



8. From last week, I don't think we've increased the timeouts:

- qemu-runner? timeout increase 120 -> 240
- ptest timeouts 300 -> 450?




8.
Plans for the week:

Richard: RCU stall
Alex:
Sakib: task summary
Trevor: make job server
Tony: ptests and work with upstream valgrind on fixing bugs.
Saul: (1 week) have QMP deal with sigusr1 to close the QMP socket
Randy: coffee, herd cats!!


../Randy

Hello all,

In some other build systems the generated SDK can be used to also
generate an image. Thus the SDK allows development against the target
sysroot using the prebuilt host and target tools, as well as
(re)generate the target images in quick iteration cycles.

What approaches are recommended with Yocto to achieve the same benefits?

My need is to regenerate an initramfs as well as compose this into an
image using WIC.

I am aware of shared state, and using prebuilt toolchains, but my
question is whether the prebuilt SDK allows to generate images?

Regards,

Leon.

When the log name contains a space, the generated URL is not correct. Thi=
s
later also breaks parsing in swatbot. This was triggered by "property
changes" and the correct URL is indeed logs/property_changes.

Signed-off-by: Alexandre Belloni <alexandre.belloni@...>
---

This is untested because I don't have the setup handy but, I hope, it is
trivial enough to work properly


reporters/swatbot.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/reporters/swatbot.py b/reporters/swatbot.py
index 4a5a04eb1809..41ae762359a0 100644
--- a/reporters/swatbot.py
+++ b/reporters/swatbot.py
@@ -202,7 +202,7 @@ class SwatBotURI(object):
logs =3D list(logs)
urls =3D []
for l in logs:
- urls.append('%s/steps/%s/logs/%s' % (build['url'], step_=
number, l['name']))
+ urls.append('%s/steps/%s/logs/%s' % (build['url'], step_=
number, l['name'].replace(" ", "_")))
if urls:
urls =3D " ".join(urls)
else:
--=20
2.31.1

Hi all,

This is the full report for yocto-3.4_M1.rc1:
https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/tree/?h=intel-yocto-testresults

======= Summary ========
No high milestone defects.

new issue found

BUG id:14434 - [3.4 M1] dmesg: proc: Bad value for 'hidepid' with poky-altcfg distro

BUG id:14435 - [3.4 M1 beaglebone] Some drm error messages in dmesg


======= Bugs ========
https://bugzilla.yoctoproject.org/show_bug.cgi?14434
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14435

Thanks,
Sangeeta

From: Bruce Ashfield <bruce.ashfield@...>

Updating linux-yocto/5.10 to the latest korg -stable release, and to
match the qemu BSPs in oe-core

Signed-off-by: Bruce Ashfield <bruce.ashfield@...>
---
.../linux/linux-yocto_5.10.bbappend | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/meta-yocto-bsp/recipes-kernel/linux/linux-yocto_5.10.bbappend b/meta-yocto-bsp/recipes-kernel/linux/linux-yocto_5.10.bbappend
index bc2b3bf576..f8362b6635 100644
--- a/meta-yocto-bsp/recipes-kernel/linux/linux-yocto_5.10.bbappend
+++ b/meta-yocto-bsp/recipes-kernel/linux/linux-yocto_5.10.bbappend
@@ -7,17 +7,17 @@ KMACHINE_genericx86 ?= "common-pc"
KMACHINE_genericx86-64 ?= "common-pc-64"
KMACHINE_beaglebone-yocto ?= "beaglebone"

-SRCREV_machine_genericx86 ?= "8c516ced69f41563404ada0bea315a55bcf1df6f"
-SRCREV_machine_genericx86-64 ?= "8c516ced69f41563404ada0bea315a55bcf1df6f"
-SRCREV_machine_edgerouter ?= "965ab3ab746ae8a1158617b6302d9c218ffbbb66"
-SRCREV_machine_beaglebone-yocto ?= "8c516ced69f41563404ada0bea315a55bcf1df6f"
+SRCREV_machine_genericx86 ?= "ab49d2db98bdee2c8c6e17fb59ded9e5292b0f41"
+SRCREV_machine_genericx86-64 ?= "ab49d2db98bdee2c8c6e17fb59ded9e5292b0f41"
+SRCREV_machine_edgerouter ?= "274d63799465eebfd201b3e8251f16d29e93a978"
+SRCREV_machine_beaglebone-yocto ?= "ab49d2db98bdee2c8c6e17fb59ded9e5292b0f41"

COMPATIBLE_MACHINE_genericx86 = "genericx86"
COMPATIBLE_MACHINE_genericx86-64 = "genericx86-64"
COMPATIBLE_MACHINE_edgerouter = "edgerouter"
COMPATIBLE_MACHINE_beaglebone-yocto = "beaglebone-yocto"

-LINUX_VERSION_genericx86 = "5.10.21"
-LINUX_VERSION_genericx86-64 = "5.10.21"
-LINUX_VERSION_edgerouter = "5.10.21"
-LINUX_VERSION_beaglebone-yocto = "5.10.21"
+LINUX_VERSION_genericx86 = "5.10.43"
+LINUX_VERSION_genericx86-64 = "5.10.43"
+LINUX_VERSION_edgerouter = "5.10.43"
+LINUX_VERSION_beaglebone-yocto = "5.10.43"
--
2.19.1

On 2021-06-16 11:41 a.m., Randy MacLeod wrote:

On 2021-06-16 4:43 a.m., sakib.sajal@... wrote:

summarize_top_output.py is used to summarize the top
output that is captured during autobuilder intermittent
failures.

Use the script to summarize the host top output and
publish the summary that is created instead of
the raw logfile.

Looks good Sakib,

Can you show people what the typical output looks like?
Is the raw top output still published?

../Randy

The script goes over the raw logfile (for example, foo.txt) which consists of multiple top outputs and summarizes each top output and writes them to foo_summary.txt

Some improvements that can be made:

1) create a separate file for each top summary, ie, if foo.txt has n top outputs, create foo_summary_[1...n].txt

2) since python3 is used to run many different kinds of jobs, it is difficult to generalize python3 threads. the script can be modified to log all the different jobs run under python3, right now it only counts the processes that are running python3.

Any feedback is welcome!

Sample output:

NOTE: program names have been shortened for better readability.
Substitutions are as follows:
~/ = /home/pokybuild/yocto-worker/
/.../POKY_32/.../ = /build/build/tmp/work/core2-32-poky-linux/
/.../POKY_64/.../ = /build/build/tmp/work/core2-64-poky-linux/
/...GCC.../ = /recipe-sysroot-native/usr/bin/x86_64-poky-linux/../../libexec/x86_64-poky-linux/gcc/x86_64-poky-linux/

top was invoked 12 times.

top - 16:51:30 up 4 days, 19:26,  1 user,  load average: 93.84, 57.26, 48.58
Tasks: 1084 total,  80 running, 664 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.1 us,  2.0 sy,  9.8 ni, 83.1 id,  2.9 wa,  0.0 hi, 0.0 si,  0.0 st
KiB Mem : 13192559+total,  5571436 free, 14307780 used, 11204637+buff/cache
KiB Swap:  8388604 total,  8239384 free,   149220 used. 11455193+avail Mem

Summary:
85  /bin/sh
41  python3
39  /bin/bash
30  x86_64-poky-linux-g++
29  make
16 ~/meta-oe/.../POKY_64/.../cpprest/2.10.18-r0/...GCC.../11.1.0/cc1plus
16 ~/meta-oe/.../POKY_64/.../libvpx/1.8.2-r0/...GCC.../11.1.0/cc1plus
16  ~/meta-oe/.../POKY_64/.../libvpx/1.8.2-r0/...GCC.../11.1.0/as
16 ~/meta-oe/.../POKY_64/.../cpprest/2.10.18-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux/x86_64-poky-linux-g++
16 ~/meta-oe/.../POKY_64/.../cpprest/2.10.18-r0/...GCC.../11.1.0/as
14 ~/meta-oe/.../POKY_64/.../rtorrent/0.9.8-r0/...GCC.../11.1.0/cc1plus
14 ~/meta-oe/.../POKY_64/.../rtorrent/0.9.8-r0/...GCC.../11.1.0/as
13 ~/meta-oe/.../POKY_64/.../vulkan-cts/1.2.6.0-r0/...GCC.../11.1.0/cc1plus
13 ~/meta-oe/.../POKY_64/.../vulkan-cts/1.2.6.0-r0/...GCC.../11.1.0/as
13 ~/meta-oe/.../POKY_64/.../vulkan-cts/1.2.6.0-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux/x86_64-poky-linux-g++
11  x86_64-poky-linux-gcc
10 ~/meta-oe/.../POKY_64/.../dovecot/2.3.14-r0/...GCC.../11.1.0/cc1
10 ~/meta-oe/.../POKY_64/.../fluentbit/1.3.5-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux/x86_64-poky-linux-gcc
10 ~/meta-oe/.../POKY_64/.../dovecot/2.3.14-r0/...GCC.../11.1.0/as
7 ~/meta-oe/.../POKY_64/.../fluentbit/1.3.5-r0/...GCC.../11.1.0/cc1
7 ~/meta-oe/.../POKY_64/.../fluentbit/1.3.5-r0/...GCC.../11.1.0/as
4  /usr/bin/python3
3  cmake
2 ~/meta-oe/.../POKY_64/.../vulkan-cts/1.2.6.0-r0/...GCC.../11.1.0/ar
2  sh
2  /lib/systemd/systemd
2  (sd-pam)
2  ninja
2  bitbake-server
2 ~/meta-oe/.../POKY_64/.../vulkan-cts/1.2.6.0-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux/x86_64-poky-linux-gcc-ar

Kernel Summary:
293  kworker
56  ksoftirqd
56  migration
56  watchdog
56  cpuhp
3  jbd2
3  ext4-rsv-conver
2  kdmflush
2  bioset
....

Sakib

Signed-off-by: Sakib Sajal <sakib.sajal@...>
---
  scripts/collect-results              |   2 +-
  scripts/generate-testresult-index.py |   2 +-
  scripts/run-config                   |   1 +
  scripts/summarize_top_output.py      | 132 +++++++++++++++++++++++++++
  4 files changed, 135 insertions(+), 2 deletions(-)
  create mode 100755 scripts/summarize_top_output.py

diff --git a/scripts/collect-results b/scripts/collect-results
index 7474e36..7178380 100755
--- a/scripts/collect-results
+++ b/scripts/collect-results
@@ -19,7 +19,7 @@ if [ -e $WORKDIR/buildhistory ]; then
  fi
    HSFILE=$WORKDIR/tmp/buildstats/*/host_stats
-d=`date +%Y-%m-%d--%H-%M`
+d="intermittent_failure_host_data"
    mkdir -p $DEST/$target/$d
  diff --git a/scripts/generate-testresult-index.py b/scripts/generate-testresult-index.py
index 7fdc17c..d85d606 100755
--- a/scripts/generate-testresult-index.py
+++ b/scripts/generate-testresult-index.py
@@ -154,7 +154,7 @@ for build in sorted(os.listdir(path), key=keygen, reverse=True):
      hd = []
      counter = 0
      # do we really need the loop?
-    for p in glob.glob(buildpath + "/*/*/host_stats*top.txt"):
+    for p in glob.glob(buildpath + "/*/*/host_stats*top_summary.txt"):
          n_split = p.split(build)
          res = reldir[0:-1] + n_split[1]
          hd.append((res, str(counter)))
diff --git a/scripts/run-config b/scripts/run-config
index 8ed88cf..82de91f 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -327,6 +327,7 @@ elif args.phase == "finish" and args.stepname == "collect-results":
      if args.results_dir:
          hp.printheader("Running results collection")
          runcmd([scriptsdir + "/collect-results", args.builddir, args.results_dir, args.target])
+        runcmd([scriptsdir + "/summarize_top_output.py", args.results_dir, args.target])
      sys.exit(0)
    if jcfg:
diff --git a/scripts/summarize_top_output.py b/scripts/summarize_top_output.py
new file mode 100755
index 0000000..0606a34
--- /dev/null
+++ b/scripts/summarize_top_output.py
@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+
+import os, sys, glob
+
+# constants
+top_header = 7
+top_summary = 5
+max_cols = 11
+
+# string substitution to make things easier to read
+subs = {
+    "/home/pokybuild/yocto-worker/" : "~/",
+    "/build/build/tmp/work/core2-32-poky-linux/" : "/.../POKY_32/.../",
+    "/build/build/tmp/work/core2-64-poky-linux/" : "/.../POKY_64/.../",
+ "/recipe-sysroot-native/usr/bin/x86_64-poky-linux/../../libexec/x86_64-poky-linux/gcc/x86_64-poky-linux/" : "/...GCC.../"
+}
+
+def usage():
+    print("Usage: " + sys.argv[0] + " <dest> <target>")
+
+def list_top_outputs(logfile):
+    # top delimiter
+    top_start = "start: top output"
+    top_end = "end: top output"
+
+    # list of top outputs
+    top_outputs = []
+
+    # flag
+    collect = False
+    with open(logfile) as log:
+        top_output = []
+        for line in log:
+            lstrip = line.strip()
+            if collect:
+                if lstrip.startswith(top_end):
+                    collect = False
+                    top_outputs.append(top_output)
+                    top_output = []
+                else:
+                    top_output.append(lstrip)
+            if lstrip.startswith(top_start):
+                collect = True
+
+    return top_outputs
+
+def summarize_top(top_outs):
+    summaries = []
+    kernel_summaries = []
+    short_summaries = []
+    for top_out in top_outs:
+        summary = {}
+        kernel_summary = {}
+        short_summary = top_out[:top_summary]
+        for line in top_out[top_header:]:
+            cmd = line.split(maxsplit=max_cols)[-1]
+            # kernel processes
+            if cmd[0] == "[" and cmd[-1] == "]":
+                kproc = cmd[1:-1].split("/")[0]
+                if kproc not in kernel_summary:
+                    kernel_summary[kproc] = 1
+                else:
+                    kernel_summary[kproc] += 1
+                continue
+            cmd_split = cmd.split()
+            prog = cmd_split[0]
+            if prog not in summary:
+                summary[prog] = 1
+            else:
+                summary[prog] += 1
+        summary = dict(sorted(summary.items(), key=lambda item: item[1], reverse=True))
+        kernel_summary = dict(sorted(kernel_summary.items(), key=lambda item: item[1], reverse=True))
+
+        summaries.append(summary)
+        kernel_summaries.append(kernel_summary)
+        short_summaries.append(short_summary)
+
+    return (short_summaries, summaries, kernel_summaries)
+
+def summarize_path(path):
+    p = path
+    for k, v in subs.items():
+        p = p.replace(k, v)
+    return p
+
+def write_summary(short_summary, summary, kernel_summary, logfile):
+    dirname = os.path.dirname(logfile)
+    fname = os.path.basename(logfile)
+    report_name = fname.split(".")[0] + "_summary.txt"
+    outfile = os.path.join(dirname, report_name)
+    out = "NOTE: program names have been shortened for better readability.\nSubstitutions are as follows:\n"
+    for k, v in subs.items():
+        out += (v + " = " + k + "\n")
+    out += "\n"
+
+    out += "top was invoked " + str(len(short_summary)) + " times.\n\n"
+
+    for i in range(len(short_summary)):
+        for l in short_summary[i]:
+            out += (l + "\n")
+
+        out += ("\nSummary: " + "\n")
+        for k, v in summary[i].items():
+            if v > 1:
+                r = summarize_path(k)
+                out += (str(v) + "  " + r + "\n")
+
+        out += ("\nKernel Summary: " + "\n")
+        for k, v in kernel_summary[i].items():
+            if v > 1:
+                r = summarize_path(k)
+                out += (str(v) + "  " + r + "\n")
+        out += ("\n")
+
+    with open(outfile, "w") as of:
+        of.write(out)
+
+def main():
+    if len(sys.argv) != 3:
+        usage()
+        sys.exit()
+
+    dest = sys.argv[1]
+    target = sys.argv[2]
+    host_data_dir = "intermittent_failure_host_data"
+    directory = os.path.join(dest, target, host_data_dir)
+    for f in glob.glob(directory + "/*_top.txt"):
+        outputs = list_top_outputs(f)
+        short_summary, summary, kernel_summary = summarize_top(outputs)
+        write_summary(short_summary, summary, kernel_summary, f)
+
+main()

On 2021-06-16 1:24 pm, Alexander Kanavin wrote:

On Wed, 16 Jun 2021 at 16:58, simon <simon@...> wrote:

- There's a warning about not finding the IRIS driver:
MESA_LOADER: failed to open iris: /usr/lib/dri/iris_dri.so: cannot open
shared object file: No such file or directory
failed to load driver: iris

But from what I was able to understand, I was expecting it would be
available with mesa 21.0.3 since I've seen some commit about fixing
issue with it in the change logs.

I've tried to force the driver to both i965 and i915 but I either got a
warning that this GEN cannot use it or still got the same fps.
Am I right to assume that it goes to a default driver that uses the CPU
instead of the GPU?

Hello Simon,

 

I honestly don't know how Intel folks have overlooked this, but mesa in yocto not only does not enable iris, the mesa recipe does not even have an option to enable it.

 

However, it should be pretty easy to add. Please take a look at poky/meta/recipes-graphics/mesa/mesa.inc, specificially any lines that mention GALLIUMDRIVERS and matching PACKAGECONFIG,

and you should be able to add iris there, and rebuild your image. Then you can submit your first patch to yocto via oe-core mailing list ;)

 

Alex

Hello Alex,

I was able to fix the iris driver issue with the following added to mesa.inc

GALLIUMDRIVERS_append = "${@bb.utils.contains('PACKAGECONFIG', 'iris', ',iris', '', d)}"

I will look into sending the patch

thanks a lot for your help

Simon

On Wed, 2021-06-16 at 04:43 -0400, sakib.sajal@... wrote:

summarize_top_output.py is used to summarize the top
output that is captured during autobuilder intermittent
failures.

Use the script to summarize the host top output and
publish the summary that is created instead of
the raw logfile.

[...]

 if jcfg:
diff --git a/scripts/summarize_top_output.py b/scripts/summarize_top_output.py
new file mode 100755
index 0000000..0606a34
--- /dev/null
+++ b/scripts/summarize_top_output.py
@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+
+import os, sys, glob
+
+# constants
+top_header = 7
+top_summary = 5
+max_cols = 11
+
+# string substitution to make things easier to read
+subs = {
+ "/home/pokybuild/yocto-worker/" : "~/",
+ "/build/build/tmp/work/core2-32-poky-linux/" : "/.../POKY_32/.../",
+ "/build/build/tmp/work/core2-64-poky-linux/" : "/.../POKY_64/.../",
+ "/recipe-sysroot-native/usr/bin/x86_64-poky-linux/../../libexec/x86_64-poky-linux/gcc/x86_64-poky-linux/" : "/...GCC.../"
+}

One quick question - the above assumes an x86 target machine using those two tunes. 
Should that be wildcarded?

Cheers,

Richard

On 2021-06-16 4:43 a.m., sakib.sajal@... wrote:

summarize_top_output.py is used to summarize the top
output that is captured during autobuilder intermittent
failures.
Use the script to summarize the host top output and
publish the summary that is created instead of
the raw logfile.

Looks good Sakib,

Can you show people what the typical output looks like?
Is the raw top output still published?

../Randy

Signed-off-by: Sakib Sajal <sakib.sajal@...>
---
scripts/collect-results | 2 +-
scripts/generate-testresult-index.py | 2 +-
scripts/run-config | 1 +
scripts/summarize_top_output.py | 132 +++++++++++++++++++++++++++
4 files changed, 135 insertions(+), 2 deletions(-)
create mode 100755 scripts/summarize_top_output.py
diff --git a/scripts/collect-results b/scripts/collect-results
index 7474e36..7178380 100755
--- a/scripts/collect-results
+++ b/scripts/collect-results
@@ -19,7 +19,7 @@ if [ -e $WORKDIR/buildhistory ]; then
fi
HSFILE=$WORKDIR/tmp/buildstats/*/host_stats
-d=`date +%Y-%m-%d--%H-%M`
+d="intermittent_failure_host_data"
mkdir -p $DEST/$target/$d
diff --git a/scripts/generate-testresult-index.py b/scripts/generate-testresult-index.py
index 7fdc17c..d85d606 100755
--- a/scripts/generate-testresult-index.py
+++ b/scripts/generate-testresult-index.py
@@ -154,7 +154,7 @@ for build in sorted(os.listdir(path), key=keygen, reverse=True):
hd = []
counter = 0
# do we really need the loop?
- for p in glob.glob(buildpath + "/*/*/host_stats*top.txt"):
+ for p in glob.glob(buildpath + "/*/*/host_stats*top_summary.txt"):
n_split = p.split(build)
res = reldir[0:-1] + n_split[1]
hd.append((res, str(counter)))
diff --git a/scripts/run-config b/scripts/run-config
index 8ed88cf..82de91f 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -327,6 +327,7 @@ elif args.phase == "finish" and args.stepname == "collect-results":
if args.results_dir:
hp.printheader("Running results collection")
runcmd([scriptsdir + "/collect-results", args.builddir, args.results_dir, args.target])
+ runcmd([scriptsdir + "/summarize_top_output.py", args.results_dir, args.target])
sys.exit(0)
if jcfg:
diff --git a/scripts/summarize_top_output.py b/scripts/summarize_top_output.py
new file mode 100755
index 0000000..0606a34
--- /dev/null
+++ b/scripts/summarize_top_output.py
@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+
+import os, sys, glob
+
+# constants
+top_header = 7
+top_summary = 5
+max_cols = 11
+
+# string substitution to make things easier to read
+subs = {
+ "/home/pokybuild/yocto-worker/" : "~/",
+ "/build/build/tmp/work/core2-32-poky-linux/" : "/.../POKY_32/.../",
+ "/build/build/tmp/work/core2-64-poky-linux/" : "/.../POKY_64/.../",
+ "/recipe-sysroot-native/usr/bin/x86_64-poky-linux/../../libexec/x86_64-poky-linux/gcc/x86_64-poky-linux/" : "/...GCC.../"
+}
+
+def usage():
+ print("Usage: " + sys.argv[0] + " <dest> <target>")
+
+def list_top_outputs(logfile):
+ # top delimiter
+ top_start = "start: top output"
+ top_end = "end: top output"
+
+ # list of top outputs
+ top_outputs = []
+
+ # flag
+ collect = False
+ with open(logfile) as log:
+ top_output = []
+ for line in log:
+ lstrip = line.strip()
+ if collect:
+ if lstrip.startswith(top_end):
+ collect = False
+ top_outputs.append(top_output)
+ top_output = []
+ else:
+ top_output.append(lstrip)
+ if lstrip.startswith(top_start):
+ collect = True
+
+ return top_outputs
+
+def summarize_top(top_outs):
+ summaries = []
+ kernel_summaries = []
+ short_summaries = []
+ for top_out in top_outs:
+ summary = {}
+ kernel_summary = {}
+ short_summary = top_out[:top_summary]
+ for line in top_out[top_header:]:
+ cmd = line.split(maxsplit=max_cols)[-1]
+ # kernel processes
+ if cmd[0] == "[" and cmd[-1] == "]":
+ kproc = cmd[1:-1].split("/")[0]
+ if kproc not in kernel_summary:
+ kernel_summary[kproc] = 1
+ else:
+ kernel_summary[kproc] += 1
+ continue
+ cmd_split = cmd.split()
+ prog = cmd_split[0]
+ if prog not in summary:
+ summary[prog] = 1
+ else:
+ summary[prog] += 1
+ summary = dict(sorted(summary.items(), key=lambda item: item[1], reverse=True))
+ kernel_summary = dict(sorted(kernel_summary.items(), key=lambda item: item[1], reverse=True))
+
+ summaries.append(summary)
+ kernel_summaries.append(kernel_summary)
+ short_summaries.append(short_summary)
+
+ return (short_summaries, summaries, kernel_summaries)
+
+def summarize_path(path):
+ p = path
+ for k, v in subs.items():
+ p = p.replace(k, v)
+ return p
+
+def write_summary(short_summary, summary, kernel_summary, logfile):
+ dirname = os.path.dirname(logfile)
+ fname = os.path.basename(logfile)
+ report_name = fname.split(".")[0] + "_summary.txt"
+ outfile = os.path.join(dirname, report_name)
+ out = "NOTE: program names have been shortened for better readability.\nSubstitutions are as follows:\n"
+ for k, v in subs.items():
+ out += (v + " = " + k + "\n")
+ out += "\n"
+
+ out += "top was invoked " + str(len(short_summary)) + " times.\n\n"
+
+ for i in range(len(short_summary)):
+ for l in short_summary[i]:
+ out += (l + "\n")
+
+ out += ("\nSummary: " + "\n")
+ for k, v in summary[i].items():
+ if v > 1:
+ r = summarize_path(k)
+ out += (str(v) + " " + r + "\n")
+
+ out += ("\nKernel Summary: " + "\n")
+ for k, v in kernel_summary[i].items():
+ if v > 1:
+ r = summarize_path(k)
+ out += (str(v) + " " + r + "\n")
+ out += ("\n")
+
+ with open(outfile, "w") as of:
+ of.write(out)
+
+def main():
+ if len(sys.argv) != 3:
+ usage()
+ sys.exit()
+
+ dest = sys.argv[1]
+ target = sys.argv[2]
+ host_data_dir = "intermittent_failure_host_data"
+ directory = os.path.join(dest, target, host_data_dir)
+ for f in glob.glob(directory + "/*_top.txt"):
+ outputs = list_top_outputs(f)
+ short_summary, summary, kernel_summary = summarize_top(outputs)
+ write_summary(short_summary, summary, kernel_summary, f)
+
+main()

--
# Randy MacLeod
# Wind River Linux

Hi all,

I'm having trouble figuring out how to fix my issues with the new intel NUC gen 11 (NUC11TNBi5).

At the moment I'm testing with the following meta-layer
poky (hardknott)
meta-openembedded (hardknott)
meta-intel (master)
meta-qt5 (hardknott)

I'm currently using EGLFS to display our Qt-QML GUI

My first issue is with the our GUI, it is launching as expected but we barely have 1 fps

- There's a warning about not finding the IRIS driver:
MESA_LOADER: failed to open iris: /usr/lib/dri/iris_dri.so: cannot open shared object file: No such file or directory
failed to load driver: iris

But from what I was able to understand, I was expecting it would be available with mesa 21.0.3 since I've seen some commit about fixing issue with it in the change logs.

I've tried to force the driver to both i965 and i915 but I either got a warning that this GEN cannot use it or still got the same fps.
Am I right to assume that it goes to a default driver that uses the CPU instead of the GPU?

My second issue is with the nvme slot, it does not detect the partition on the SSD, but it does when I use a NVME to USB adaptor

- There's a few warning about ACPI bios error AE_NOT_FOUND too, and again with my little knowledge I thought this would have been fixed with the most recent update in master with the new microcode since the Core gen 11 was added as new device.
But again, I'm not entirely sure that I'm doing the right thing to update the microcode. I've installed manually the microcode.cpio in /etc/firmware/intel-ucode/

The ouput of "dmesg | grep microcode":
microcode: sig=0x806c1, pf=0x80, revision=0x72
microcode: Microcode Update Driver: v2.2.

I don't know if I'm too early to be using the latest NUC11, any information about when it should be supported would be very appreciated.

I'm willing to help but my knowledge is pretty limited so I might need assistance.

thanks a lot

Simon

Disables debuginfod when using MingGW. This feature brings in
unbuildable dependencies and can't be used.

Signed-off-by: Joshua Watt <JPEWhacker@...>
---
conf/machine-sdk/include/mingw32-common.inc | 2 ++
recipes-devtools/gdb/gdb-cross-canadian_%.bbappend | 1 +
2 files changed, 3 insertions(+)

diff --git a/conf/machine-sdk/include/mingw32-common.inc b/conf/machine-sdk/include/mingw32-common.inc
index 0109e75..3997a26 100644
--- a/conf/machine-sdk/include/mingw32-common.inc
+++ b/conf/machine-sdk/include/mingw32-common.inc
@@ -15,6 +15,8 @@ USE_NLS_mingw32 = "no"
FILES_${PN}-staticdev_append_mingw32 = " ${libdir}/*.lib"
ALLOW_EMPTY_${PN}_mingw32 = "1"

+DISTRO_FEATURES_FILTER_NATIVESDK_remove_mingw32 = "debuginfod"
+
# Do what amounts to a NOOP
SDK_PACKAGING_FUNC = "do_compile"

diff --git a/recipes-devtools/gdb/gdb-cross-canadian_%.bbappend b/recipes-devtools/gdb/gdb-cross-canadian_%.bbappend
index c33a9ce..096fc63 100644
--- a/recipes-devtools/gdb/gdb-cross-canadian_%.bbappend
+++ b/recipes-devtools/gdb/gdb-cross-canadian_%.bbappend
@@ -4,3 +4,4 @@ RDEPENDS_${PN}_remove_sdkmingw32 = "nativesdk-python-core nativesdk-python-lang
EXTRA_OECONF_append_sdkmingw32 = " --without-curses --without-system-readline --with-python=no"
PACKAGECONFIG_remove_sdkmingw32 = "readline"
PACKAGECONFIG_remove_sdkmingw32 = "python"
+PACKAGECONFIG_remove_sdkmingw32 = "debuginfod"
--
2.31.1

On Tue, 2021-06-15 at 17:21 +0200, Alexander Kanavin wrote:

So the memory usage is actually *better* without prelink. And any timing 
benefits are lost in statistical noise, in these tests at least.

The numbers certainly don't look convincing, thanks for running the tests.

So I do not think it is wrong to question the usefulness of this feature. 
I'd like to hear Mark's take on this, as prelink-cross is primarily his 
work, he's put a lot of effort into it, and I would want to know where 
the benefits are. Note that Red Hat abandoned prelink in 2013, and 
prelink-cross likewise hasn't seen any commits for two years.

Mark and I did briefly talk yesterday and whilst Mark can/will speak
up, the summary was "drop it".

What makes me sad is that this is an embedded focused feature which 
the project should be caring about, i.e. memory footprint and speed.
I don't know why the numbers don't show it, Mark and I have theories
but it would take work to explore them and neither of us have that time,
nor are we experts on prelink able to maintain it with the time we have
available.

In my view the project should be caring about things like this, instead 
it looks more like a race to look like 'any other desktop distro' and 
drop anything which looks different.

I've tried to say no. Nobody else would appear to care, I'm hearing 
silence. I can't fight this.

Cheers,

Richard

summarize_top_output.py is used to summarize the top
output that is captured during autobuilder intermittent
failures.

Use the script to summarize the host top output and
publish the summary that is created instead of
the raw logfile.

Signed-off-by: Sakib Sajal <sakib.sajal@...>
---
scripts/collect-results | 2 +-
scripts/generate-testresult-index.py | 2 +-
scripts/run-config | 1 +
scripts/summarize_top_output.py | 132 +++++++++++++++++++++++++++
4 files changed, 135 insertions(+), 2 deletions(-)
create mode 100755 scripts/summarize_top_output.py

diff --git a/scripts/collect-results b/scripts/collect-results
index 7474e36..7178380 100755
--- a/scripts/collect-results
+++ b/scripts/collect-results
@@ -19,7 +19,7 @@ if [ -e $WORKDIR/buildhistory ]; then
fi

HSFILE=$WORKDIR/tmp/buildstats/*/host_stats
-d=`date +%Y-%m-%d--%H-%M`
+d="intermittent_failure_host_data"

mkdir -p $DEST/$target/$d

diff --git a/scripts/generate-testresult-index.py b/scripts/generate-testresult-index.py
index 7fdc17c..d85d606 100755
--- a/scripts/generate-testresult-index.py
+++ b/scripts/generate-testresult-index.py
@@ -154,7 +154,7 @@ for build in sorted(os.listdir(path), key=keygen, reverse=True):
hd = []
counter = 0
# do we really need the loop?
- for p in glob.glob(buildpath + "/*/*/host_stats*top.txt"):
+ for p in glob.glob(buildpath + "/*/*/host_stats*top_summary.txt"):
n_split = p.split(build)
res = reldir[0:-1] + n_split[1]
hd.append((res, str(counter)))
diff --git a/scripts/run-config b/scripts/run-config
index 8ed88cf..82de91f 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -327,6 +327,7 @@ elif args.phase == "finish" and args.stepname == "collect-results":
if args.results_dir:
hp.printheader("Running results collection")
runcmd([scriptsdir + "/collect-results", args.builddir, args.results_dir, args.target])
+ runcmd([scriptsdir + "/summarize_top_output.py", args.results_dir, args.target])
sys.exit(0)

if jcfg:
diff --git a/scripts/summarize_top_output.py b/scripts/summarize_top_output.py
new file mode 100755
index 0000000..0606a34
--- /dev/null
+++ b/scripts/summarize_top_output.py
@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+
+import os, sys, glob
+
+# constants
+top_header = 7
+top_summary = 5
+max_cols = 11
+
+# string substitution to make things easier to read
+subs = {
+ "/home/pokybuild/yocto-worker/" : "~/",
+ "/build/build/tmp/work/core2-32-poky-linux/" : "/.../POKY_32/.../",
+ "/build/build/tmp/work/core2-64-poky-linux/" : "/.../POKY_64/.../",
+ "/recipe-sysroot-native/usr/bin/x86_64-poky-linux/../../libexec/x86_64-poky-linux/gcc/x86_64-poky-linux/" : "/...GCC.../"
+}
+
+def usage():
+ print("Usage: " + sys.argv[0] + " <dest> <target>")
+
+def list_top_outputs(logfile):
+ # top delimiter
+ top_start = "start: top output"
+ top_end = "end: top output"
+
+ # list of top outputs
+ top_outputs = []
+
+ # flag
+ collect = False
+ with open(logfile) as log:
+ top_output = []
+ for line in log:
+ lstrip = line.strip()
+ if collect:
+ if lstrip.startswith(top_end):
+ collect = False
+ top_outputs.append(top_output)
+ top_output = []
+ else:
+ top_output.append(lstrip)
+ if lstrip.startswith(top_start):
+ collect = True
+
+ return top_outputs
+
+def summarize_top(top_outs):
+ summaries = []
+ kernel_summaries = []
+ short_summaries = []
+ for top_out in top_outs:
+ summary = {}
+ kernel_summary = {}
+ short_summary = top_out[:top_summary]
+ for line in top_out[top_header:]:
+ cmd = line.split(maxsplit=max_cols)[-1]
+ # kernel processes
+ if cmd[0] == "[" and cmd[-1] == "]":
+ kproc = cmd[1:-1].split("/")[0]
+ if kproc not in kernel_summary:
+ kernel_summary[kproc] = 1
+ else:
+ kernel_summary[kproc] += 1
+ continue
+ cmd_split = cmd.split()
+ prog = cmd_split[0]
+ if prog not in summary:
+ summary[prog] = 1
+ else:
+ summary[prog] += 1
+ summary = dict(sorted(summary.items(), key=lambda item: item[1], reverse=True))
+ kernel_summary = dict(sorted(kernel_summary.items(), key=lambda item: item[1], reverse=True))
+
+ summaries.append(summary)
+ kernel_summaries.append(kernel_summary)
+ short_summaries.append(short_summary)
+
+ return (short_summaries, summaries, kernel_summaries)
+
+def summarize_path(path):
+ p = path
+ for k, v in subs.items():
+ p = p.replace(k, v)
+ return p
+
+def write_summary(short_summary, summary, kernel_summary, logfile):
+ dirname = os.path.dirname(logfile)
+ fname = os.path.basename(logfile)
+ report_name = fname.split(".")[0] + "_summary.txt"
+ outfile = os.path.join(dirname, report_name)
+ out = "NOTE: program names have been shortened for better readability.\nSubstitutions are as follows:\n"
+ for k, v in subs.items():
+ out += (v + " = " + k + "\n")
+ out += "\n"
+
+ out += "top was invoked " + str(len(short_summary)) + " times.\n\n"
+
+ for i in range(len(short_summary)):
+ for l in short_summary[i]:
+ out += (l + "\n")
+
+ out += ("\nSummary: " + "\n")
+ for k, v in summary[i].items():
+ if v > 1:
+ r = summarize_path(k)
+ out += (str(v) + " " + r + "\n")
+
+ out += ("\nKernel Summary: " + "\n")
+ for k, v in kernel_summary[i].items():
+ if v > 1:
+ r = summarize_path(k)
+ out += (str(v) + " " + r + "\n")
+ out += ("\n")
+
+ with open(outfile, "w") as of:
+ of.write(out)
+
+def main():
+ if len(sys.argv) != 3:
+ usage()
+ sys.exit()
+
+ dest = sys.argv[1]
+ target = sys.argv[2]
+ host_data_dir = "intermittent_failure_host_data"
+ directory = os.path.join(dest, target, host_data_dir)
+ for f in glob.glob(directory + "/*_top.txt"):
+ outputs = list_top_outputs(f)
+ short_summary, summary, kernel_summary = summarize_top(outputs)
+ write_summary(short_summary, summary, kernel_summary, f)
+
+main()
--
2.25.1

From: Kai Kang <kai.kang@...>

The patch fix-ldblibdir.patch has been dropped when update sssd to
2.5.0. But it fails to start sssd without this patch. So add it back.

Signed-off-by: Kai Kang <kai.kang@...>
---
.../sssd/files/fix-ldblibdir.patch | 25 +++++++++++++++++++
recipes-security/sssd/sssd_2.5.0.bb | 1 +
2 files changed, 26 insertions(+)
create mode 100644 recipes-security/sssd/files/fix-ldblibdir.patch

diff --git a/recipes-security/sssd/files/fix-ldblibdir.patch b/recipes-security/sssd/files/fix-ldblibdir.patch
new file mode 100644
index 0000000..e350baf
--- /dev/null
+++ b/recipes-security/sssd/files/fix-ldblibdir.patch
@@ -0,0 +1,25 @@
+When calculate value of ldblibdir, it checks whether the directory of
+$ldblibdir exists. If not, it assigns ldblibdir with ${libdir}/ldb. It is not
+suitable for cross compile. Fix it that only re-assign ldblibdir when its value
+is empty.
+
+Upstream-Status: Inappropriate [cross compile specific]
+
+Signed-off-by: Kai Kang <kai.kang@...>
+---
+ src/external/libldb.m4 | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/src/external/libldb.m4 b/src/external/libldb.m4
+index c400add..5e5f06d 100644
+--- a/src/external/libldb.m4
++++ b/src/external/libldb.m4
+@@ -19,7 +19,7 @@ if test x"$with_ldb_lib_dir" != x; then
+ ldblibdir=$with_ldb_lib_dir
+ else
+ ldblibdir="`$PKG_CONFIG --variable=modulesdir ldb`"
+- if ! test -d $ldblibdir; then
++ if test -z $ldblibdir; then
+ ldblibdir="${libdir}/ldb"
+ fi
+ fi
diff --git a/recipes-security/sssd/sssd_2.5.0.bb b/recipes-security/sssd/sssd_2.5.0.bb
index 4c92519..c5c870c 100644
--- a/recipes-security/sssd/sssd_2.5.0.bb
+++ b/recipes-security/sssd/sssd_2.5.0.bb
@@ -21,6 +21,7 @@ SRC_URI = "https://github.com/SSSD/sssd/releases/download/2.5.0/sssd-2.5.0.tar.g
file://no_gen.patch \
file://fix_gid.patch \
file://drop_ntpdate_chk.patch \
+ file://fix-ldblibdir.patch \
"
SRC_URI[sha256sum] = "afa62d7d8d23fca3aba093abe4ec0d14e7d9346c5b28ceb7c2c624bed98caa06"

--
2.17.1

Signed-off-by: Federico Pellegrin <fede@...>
---
.../{aircrack-ng_1.3.bb => aircrack-ng_1.6.bb} | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
rename recipes-security/aircrack-ng/{aircrack-ng_1.3.bb => aircrack-ng_1.6.bb} (82%)

diff --git a/recipes-security/aircrack-ng/aircrack-ng_1.3.bb b/recipes-security/aircrack-ng/aircrack-ng_1.6.bb
similarity index 82%
rename from recipes-security/aircrack-ng/aircrack-ng_1.3.bb
rename to recipes-security/aircrack-ng/aircrack-ng_1.6.bb
index d739227..8d3b531 100644
--- a/recipes-security/aircrack-ng/aircrack-ng_1.3.bb
+++ b/recipes-security/aircrack-ng/aircrack-ng_1.6.bb
@@ -9,8 +9,8 @@ DEPENDS = "libnl openssl sqlite3 libpcre libpcap"

SRC_URI = "http://download.aircrack-ng.org/${BP}.tar.gz"

-SRC_URI[md5sum] = "c7c5b076dee0c25ee580b0f56f455623"
-SRC_URI[sha256sum] = "8ae08a7c28741f6ace2769267112053366550e7f746477081188ad38410383ca"
+SRC_URI[md5sum] = "22ddc85549b51ed0da0931d01ef215e5"
+SRC_URI[sha256sum] = "4f0bfd486efc6ea7229f7fbc54340ff8b2094a0d73e9f617e0a39f878999a247"

inherit autotools-brokensep pkgconfig

@@ -29,6 +29,8 @@ do_install () {
make DESTDIR=${D} ${OEMAKE_EXTRA} ext_scripts=true install
}

-FILES_${PN} += "/usr/local/"
+FILES_${PN} += "${libdir}/*.so"
+FILES_SOLIBSDEV = ""
+INSANE_SKIP_${PN} += "dev-so"

RDEPENDS_${PN} = "libpcap"
--
2.31.1

Hi all,

Intel and WR YP QA is planning for QA execution for YP build yocto-3.4_M1.rc1. We are planning to execute following tests for this cycle:

OEQA-manual tests for following module:
1. OE-Core
2. BSP-hw

Runtime auto test for following platforms:
1. MinnowTurbot 32-bit
2. Coffee Lake
3. NUC 7
4. NUC 6
5. Edgerouter
6. Beaglebone

ETA for completion is next Thursday, June 17.

Thanks,
Sangeeta