Summary:

Build: 20210420-1 had 23 triggers and isn't really worth
analyzing. Stay tuned for what is hopefully better data
with a higher timeout threshold.

There is a nice looking pair of graph attached though! :)

../Randy

On 2021-04-15 4:48 p.m., Randy MacLeod wrote:

On 2021-04-15 1:55 p.m., Randy MacLeod wrote:

Should we  increase the interval to 30, 60, ore more seconds?

I've bumped the interval to 60 seconds.


I spent some time looking at the first bit of data along with
Sakib and Saul from time to time.

General conclusions:
1. It seems like ALL triggers involve oe-selftest being active.

Still true.

2. xz might be a problem but we're not sure yet.

Added to graph attached.

3. We need more data and tools and time to think about it.

To Do:
1. increase top cmdline length from 512 to 16K

Still to do, maybe. It's not clear that it's useful yet.

2. sometimes we see:
Command '['oe-time-dd-test.sh', '100']' timed out after 10.0 seconds
That should not happen so we should understand why and either increase
the time between runs or fix the tooling. This seems to happen under load
so it's hiding the interesting data that we are looking for!

With a 60 second interval, this does NOT happen.

3. tail the cooker console in addition to top. Present that before top.
It would be nice to have a top equivalent for bitbake.

Still to do.
This can be done via the cooker log.
We can parse the file and do:
1. list recent tasks regardless of whether they have completed:
tail -20?
2. list tasks that have started but not completed.



This email is about:
https://autobuilder.yocto.io/pub/non-release/20210420-1/
It was a 'quick' build. There was only 1 log file produced.
It used the master so the timeout was still 5 seconds.

There were 23 (!!) times that the dd time exceeded the
15 (ACTUALLY 5) second limit out of a total of 1504 invocations
and those triggers were captured by 1 log file:
testresults/qemux86-world/2021-04-20--03-43/host_stats_0_top.txt


Splitting each top output into a separate file as before,
now we have 23 log files:

How big are these files, ie how many process/kernel threads were running
when top ran?

$ wc -l host-stats-0* | sort -n
699 host-stats-03:42:40--23.log
704 host-stats-02:20:41--5.log
720 host-stats-03:33:41--21.log
733 host-stats-02:19:40--4.log
743 host-stats-03:32:41--20.log
752 host-stats-02:25:45--6.log
753 host-stats-02:02:46--3.log
760 host-stats-03:36:07--22.log
784 host-stats-02:01:48--2.log
784 host-stats-03:30:43--18.log
802 host-stats-02:45:55--10.log
807 host-stats-03:31:42--19.log
816 host-stats-03:29:49--17.log
829 host-stats-03:19:50--15.log
845 host-stats-02:41:40--8.log
851 host-stats-02:00:57--1.log
899 host-stats-02:32:47--7.log
906 host-stats-03:18:41--14.log
925 host-stats-03:28:24--16.log
947 host-stats-02:42:41--9.log
1084 host-stats-03:16:50--13.log
1204 host-stats-02:54:43--11.log
1314 host-stats-03:03:41--12.log
19661 total



I noticed that several but not all log files were running xz:

$ for i in `ls host-stats-*`; do echo -n $i ": "; grep "xz " $i | wc -l; done
host-stats-02:00:57--1.log : 0
host-stats-02:01:48--2.log : 0
host-stats-02:02:46--3.log : 0
host-stats-02:19:40--4.log : 2
host-stats-02:20:41--5.log : 0
host-stats-02:25:45--6.log : 2
host-stats-02:32:47--7.log : 0
host-stats-02:41:40--8.log : 1
host-stats-02:42:41--9.log : 2
host-stats-02:45:55--10.log : 1
host-stats-02:54:43--11.log : 0
host-stats-03:03:41--12.log : 0
host-stats-03:16:50--13.log : 47
host-stats-03:18:41--14.log : 0
host-stats-03:19:50--15.log : 9
host-stats-03:28:24--16.log : 15
host-stats-03:29:49--17.log : 0
host-stats-03:30:43--18.log : 0
host-stats-03:31:42--19.log : 7
host-stats-03:32:41--20.log : 7
host-stats-03:33:41--21.log : 11
host-stats-03:36:07--22.log : 15
host-stats-03:42:40--23.log : 9


I've plotted the number of xz processes along with the load average
and the dd time in the attached graph.


All of the top output logs seems to be running oe-selftest:

for i in host-stats-0*; do grep -H -c "DISPLAY.*oe-selftest " $i ; done
host-stats-02:00:57--1.log:1
host-stats-02:01:48--2.log:1
host-stats-02:02:46--3.log:1
host-stats-02:19:40--4.log:1
host-stats-02:20:41--5.log:1
host-stats-02:25:45--6.log:1
host-stats-02:32:47--7.log:1
host-stats-02:41:40--8.log:2
host-stats-02:42:41--9.log:2
host-stats-02:45:55--10.log:2
host-stats-02:54:43--11.log:2
host-stats-03:03:41--12.log:2
host-stats-03:16:50--13.log:2
host-stats-03:18:41--14.log:2
host-stats-03:19:50--15.log:2
host-stats-03:28:24--16.log:2
host-stats-03:29:49--17.log:2
host-stats-03:30:43--18.log:2
host-stats-03:31:42--19.log:2
host-stats-03:32:41--20.log:2
host-stats-03:33:41--21.log:2
host-stats-03:36:07--22.log:2
host-stats-03:42:40--23.log:2



The logs DO seem to be clustered in that there are several 1 minute adjacent intervals where the dd time exceeded the threshold.

Data here but you can see this in the attached graph.

$ for i in host-stats-*; do echo -n $i ": "; head -1 $i | cut -c -15; done
host-stats-02:00:57--1.log : top - 02:00:57
host-stats-02:01:48--2.log : top - 02:01:48
host-stats-02:02:46--3.log : top - 02:02:46
host-stats-02:19:40--4.log : top - 02:19:40
host-stats-02:20:41--5.log : top - 02:20:41
host-stats-02:25:45--6.log : top - 02:25:45
host-stats-02:32:47--7.log : top - 02:32:47
host-stats-02:41:40--8.log : top - 02:41:40
host-stats-02:42:41--9.log : top - 02:42:41
host-stats-02:45:55--10.log : top - 02:45:55
host-stats-02:54:43--11.log : top - 02:54:43
host-stats-03:03:41--12.log : top - 03:03:41
host-stats-03:16:50--13.log : top - 03:16:50
host-stats-03:18:41--14.log : top - 03:18:41
host-stats-03:19:50--15.log : top - 03:19:50
host-stats-03:28:24--16.log : top - 03:28:24
host-stats-03:29:49--17.log : top - 03:29:49
host-stats-03:30:43--18.log : top - 03:30:43
host-stats-03:31:42--19.log : top - 03:31:42
host-stats-03:32:41--20.log : top - 03:32:41
host-stats-03:33:41--21.log : top - 03:33:41
host-stats-03:36:07--22.log : top - 03:36:07
host-stats-03:42:40--23.log : top - 03:42:40



*** Add time distribution of dd.
Oddly enough, the output of dd does not seem to be consistent:

102400 bytes (102 kB) copied, 0.0163975 s, 6.2 MB/s
102400 bytes (102 kB) copied, 0.0054732 s, 18.7 MB/s
102400 bytes (102 kB, 100 KiB) copied, 0.0984687 s, 1.0 MB/s
102400 bytes (102 kB, 100 KiB) copied, 0.304233 s, 337 kB/s

I'm not sure what's going on there, yet. Sakib?


I think the cooker output will be quite useful.
I'm not going to put any more time into this data set
since the time threshold is still 5 seconds.

All for now.

../Randy

../Randy

Cheers,

Richard

The template is not used anywhere, yet, the initial patchset enables the
data collection by default.

I have left the template in case the data collection is removed from
defaults and need to be used on a case by case basis.

I am not entirely sure if nesting templates work. I have not seen any
examples of it, neither did i try it myself. If nesting does work, the
template should be useful.

I had a quick look at the code and sadly, it doesn't appear I implemented
nesting so this wouldn't be that useful as things stand.

Cheers,

Richard

--
# Randy MacLeod
# Wind River Linux

On Tue, Apr 20, 2021 at 12:18 PM Monsees, Steven C (US)
<steven.monsees@...> wrote:

There appears to be an issue with build zeus based meta-clang under zeus platform...

I followed steps off meta-clang page, and from the work I did with meta-clang under "Rocko" about a year ago, a lot has changed under the hood...

Working under zeus 3.0.4... EXT SDK appears fully functional, I require CLANG/LLVM for future development...

I think for future development you should look into dunfell or newer
since those are supported releases.

Never seen this error before, did I miss a patch or possibly a step ?

clang toolchain has evolved rapidly in past few years and so you might
find major changes from release to release and also SDK builds are
relatively
new, I think dunfell and newer is where its properly working without
much issues.

Clean build area...

(1) Add meta-clang to bblayers.conf

(2) Added EXT SDK Settings for meta-clang :

SDKIMAGE_FEATURES_append = " staticdev-pkgs"

SDK_EXTRA_TOOLS = " \
nativesdk-cmake \
nativesdk-clang \ <---Only appears to download/build llvm when present
"

TOOLCHAIN_HOST_TASK_append = "${SDK_EXTRA_TOOLS}"

The build is much slower due to "native-clang-9.0.1-r0 do_compile"...


EXT SDK build error seen:
=========================

10:12 smonsees@yix490038 /disk0/scratch/smonsees/yocto/workspace_3/builds2/sbcb-default> bitbake sbcb-defaultfs-full -c populate_sdk_ext
Parsing recipes: 100% |#################################################################################| Time: 0:01:25
Parsing of 2463 .bb files complete (0 cached, 2463 parsed). 3671 targets, 91 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION = "1.44.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "rhel-7.9"
TARGET_SYS = "x86_64-poky-linux"
MACHINE = "sbcb-default"
DISTRO = "limws"
DISTRO_VERSION = "3.0.4"
TUNE_FEATURES = "m64 corei7"
TARGET_FPU = ""
meta
meta-poky = "my_yocto_3.0.4:f2eb22a8783f1eecf99bd4042695bab920eed00e"
meta-perl
meta-python
meta-filesystems
meta-networking
meta-initramfs
meta-oe = "zeus:2b5dd1eb81cd08bc065bc76125f2856e9383e98b"
meta-clang = "zeus:f5355ca9b86fb5de5930132ffd95a9b352d694f9"
meta = "master:a32ddd2b2a51b26c011fa50e441df39304651503"
meta-intel = "zeus:d9942d4c3a710406b051852de7232db03c297f4e"
meta-intel = "v2019.02:f635a364c55f1fb12519aff54924a0a5b947091e"

NOTE: Fetching uninative binary shim from file:///ede/tms/yocto/zeus/downloads/intel/x86_64-nativesdk-libc.tar.xz;sha256sum=a09922172c3a439105e0ae6b943daad2d83505b17da0aba97961ff433b8c21ab
Initialising tasks: 100% |##############################################################################| Time: 0:00:12
Checking sstate mirror object availability: 100% |######################################################| Time: 0:00:00
Sstate summary: Wanted 2490 Found 2157 Missed 333 Current 0 (86% match, 0% complete)
NOTE: Executing Tasks
NOTE: Setscene tasks completed
ERROR: sbcb-defaultfs-full-1.0-r0 do_sdk_depends: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:extend_recipe_sysroot(d)
0003:
File: '/disk0/scratch/smonsees/yocto/workspace_3/poky/meta/classes/staging.bbclass', lineno: 551, function: extend_recipe_sysroot
0547: dest = newmanifest[l]
0548: if l.endswith("/"):
0549: staging_copydir(l, targetdir, dest, seendirs)
0550: continue
*** 0551: staging_copyfile(l, targetdir, dest, postinsts, seendirs)
0552:
0553: bb.note("Installed into sysroot: %s" % str(msg_adding))
0554: bb.note("Skipping as already exists in sysroot: %s" % str(msg_exists))
0555:
File: '/disk0/scratch/smonsees/yocto/workspace_3/poky/meta/classes/staging.bbclass', lineno: 144, function: staging_copyfile
0140: if os.path.islink(c):
0141: linkto = os.readlink(c)
0142: if os.path.lexists(dest):
0143: if not os.path.islink(dest):
*** 0144: raise OSError(errno.EEXIST, "Link %s already exists as a file" % dest, dest)
0145: if os.readlink(dest) == linkto:
0146: return dest
0147: raise OSError(errno.EEXIST, "Link %s already exists to a different location? (%s vs %s)" % (dest, os.readlink(dest), linkto), dest)
0148: os.symlink(linkto, dest)
Exception: FileExistsError: [Errno 17] Link /disk0/scratch/smonsees/yocto/workspace_3/builds2/sbcb-default/tmp/work/sbcb_default-poky-linux/sbcb-defaultfs-full/1.0-r0/recipe-sysroot/lib already exists as a file: '/disk0/scratch/smonsees/yocto/workspace_3/builds2/sbcb-default/tmp/work/sbcb_default-poky-linux/sbcb-defaultfs-full/1.0-r0/recipe-sysroot/lib'

ERROR: Logfile of failure stored in: /disk0/scratch/smonsees/yocto/workspace_3/builds2/sbcb-default/tmp/work/sbcb_default-poky-linux/sbcb-defaultfs-full/1.0-r0/temp/log.do_sdk_depends.24732
ERROR: Task (/disk0/scratch/smonsees/yocto/workspace_3/poky/../meta-bae/meta-limws/meta-intel/recipes-core/images/sbcb-defaultfs-full.bb:do_sdk_depends) failed with exit code '1'
NOTE: Tasks Summary: Attempted 6892 tasks of which 5539 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
/disk0/scratch/smonsees/yocto/workspace_3/poky/../meta-bae/meta-limws/meta-intel/recipes-core/images/sbcb-defaultfs-full.bb:do_sdk_depends
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
13:01 smonsees@yix490038 /disk0/scratch/smonsees/yocto/workspace_3/builds2/sbcb-default>



Note: Modifying SDK_EXTRA_TOOLS by removing "nativesdk-clang", i.e.:

SDK_EXTRA_TOOLS = " \
nativesdk-cmake \
"

If I remove, the build is much faster, and it builds without the error, also appears not to require/include
"llvm" component in build.
I do not believe the full package was actually built/enabled, but no errors and image is functional.


13:09 smonsees@yix490038 /disk0/scratch/smonsees/yocto/workspace_3/builds2/sbcb-default> bitbake sbcb-defaultfs-full -c populate_sdk_ext
Parsing recipes: 100% |#################################################################################| Time: 0:01:26
Parsing of 2463 .bb files complete (0 cached, 2463 parsed). 3671 targets, 91 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION = "1.44.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "rhel-7.9"
TARGET_SYS = "x86_64-poky-linux"
MACHINE = "sbcb-default"
DISTRO = "limws"
DISTRO_VERSION = "3.0.4"
TUNE_FEATURES = "m64 corei7"
TARGET_FPU = ""
meta
meta-poky = "my_yocto_3.0.4:f2eb22a8783f1eecf99bd4042695bab920eed00e"
meta-perl
meta-python
meta-filesystems
meta-networking
meta-initramfs
meta-oe = "zeus:2b5dd1eb81cd08bc065bc76125f2856e9383e98b"
meta-clang = "zeus:f5355ca9b86fb5de5930132ffd95a9b352d694f9"
meta = "master:a32ddd2b2a51b26c011fa50e441df39304651503"
meta-intel = "zeus:d9942d4c3a710406b051852de7232db03c297f4e"
meta-intel = "v2019.02:f635a364c55f1fb12519aff54924a0a5b947091e"

NOTE: Fetching uninative binary shim from file:///ede/tms/yocto/zeus/downloads/intel/x86_64-nativesdk-libc.tar.xz;sha256sum=a09922172c3a439105e0ae6b943daad2d83505b17da0aba97961ff433b8c21ab
Initialising tasks: 100% |##############################################################################| Time: 0:00:12
Checking sstate mirror object availability: 100% |######################################################| Time: 0:00:00
Sstate summary: Wanted 2436 Found 2407 Missed 29 Current 0 (98% match, 0% complete)
NOTE: Executing Tasks
NOTE: Setscene tasks completed
NOTE: Tasks Summary: Attempted 6767 tasks of which 6223 didn't need to be rerun and all succeeded.
13:31 smonsees@yix490038 /disk0/scratch/smonsees/yocto/workspace_3/builds2/sbcb-default>


-----Original Message-----
From: Khem Raj <raj.khem@...>
Sent: Tuesday, April 20, 2021 12:02 PM
To: Monsees, Steven C (US) <steven.monsees@...>; Anton Antonov <anton.antonov@...>; yocto@...
Subject: Re: [yocto] #yocto llvm support

External Email Alert

This email has been sent from an account outside of the BAE Systems network.

Please treat the email with caution, especially if you are requested to click on a link, decrypt/open an attachment, or enable macros. For further information on how to spot phishing, access "Cybersecurity OneSpace Page" and report phishing by clicking the button "Report Phishing" on the Outlook toolbar.



On 4/20/21 7:00 AM, Monsees, Steven C (US) via lists.yoctoproject.org wrote:

I noticed similar behavior.

I am running zeus 3.0.4, "devtool sdk-install llvm" will get llvm
8.0.1.

When I build meta-clang, and I set TOOLCHAIN?="clang" in local.conf it
appears to grab llvm-project-source-9.0.1-9.0.1.

this is intentional, when you use meta-clang, then llvm is preferred from LLVM since them we have consistent version of llvm for clang and others.

*From:*yocto@... <yocto@...> *On
Behalf Of *Anton Antonov
*Sent:* Tuesday, April 20, 2021 9:51 AM
*To:* yocto@...
*Subject:* Re: [yocto] #yocto llvm support

*_External Email Alert_*

*This email has been sent from an account outside of the BAE Systems
network.*

Please treat the email with caution, especially if you are requested
to click on a link, decrypt/open an attachment, or enable macros. For
further information on how to spot phishing, access "Cybersecurity
OneSpace Page" and report phishing by clicking the button "Report
Phishing" on the Outlook toolbar.


Hi Steven,

I used meta-clang in my recipes and I noticed that:
1. The current release of poky uses LLVM v11.1.0 by default
(poky/meta/recipes-devtools/llvm/llvm_git.bb)
2. Meta-clang requires LLVM v12.0.0 (meta-clang/conf/layer.conf
defines LLVMVERSION = "12.0.0")

As a result just including meta-clang into bblayers.conf will require
bitbake to build a new version of LLVM and rebuild everything which
uses it

Anton

There appears to be an issue with build zeus based meta-clang under zeus platform...

I followed steps off meta-clang page, and from the work I did with meta-clang under "Rocko" about a year ago, a lot has changed under the hood...

Working under zeus 3.0.4... EXT SDK appears fully functional, I require CLANG/LLVM for future development...

Never seen this error before, did I miss a patch or possibly a step ?

Clean build area...

(1) Add meta-clang to bblayers.conf

(2) Added EXT SDK Settings for meta-clang :

SDKIMAGE_FEATURES_append = " staticdev-pkgs"

SDK_EXTRA_TOOLS = " \
nativesdk-cmake \
nativesdk-clang \ <---Only appears to download/build llvm when present
"

TOOLCHAIN_HOST_TASK_append = "${SDK_EXTRA_TOOLS}"

The build is much slower due to "native-clang-9.0.1-r0 do_compile"...


EXT SDK build error seen:
=========================

10:12 smonsees@yix490038 /disk0/scratch/smonsees/yocto/workspace_3/builds2/sbcb-default> bitbake sbcb-defaultfs-full -c populate_sdk_ext
Parsing recipes: 100% |#################################################################################| Time: 0:01:25
Parsing of 2463 .bb files complete (0 cached, 2463 parsed). 3671 targets, 91 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION = "1.44.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "rhel-7.9"
TARGET_SYS = "x86_64-poky-linux"
MACHINE = "sbcb-default"
DISTRO = "limws"
DISTRO_VERSION = "3.0.4"
TUNE_FEATURES = "m64 corei7"
TARGET_FPU = ""
meta
meta-poky = "my_yocto_3.0.4:f2eb22a8783f1eecf99bd4042695bab920eed00e"
meta-perl
meta-python
meta-filesystems
meta-networking
meta-initramfs
meta-oe = "zeus:2b5dd1eb81cd08bc065bc76125f2856e9383e98b"
meta-clang = "zeus:f5355ca9b86fb5de5930132ffd95a9b352d694f9"
meta = "master:a32ddd2b2a51b26c011fa50e441df39304651503"
meta-intel = "zeus:d9942d4c3a710406b051852de7232db03c297f4e"
meta-intel = "v2019.02:f635a364c55f1fb12519aff54924a0a5b947091e"

NOTE: Fetching uninative binary shim from file:///ede/tms/yocto/zeus/downloads/intel/x86_64-nativesdk-libc.tar.xz;sha256sum=a09922172c3a439105e0ae6b943daad2d83505b17da0aba97961ff433b8c21ab
Initialising tasks: 100% |##############################################################################| Time: 0:00:12
Checking sstate mirror object availability: 100% |######################################################| Time: 0:00:00
Sstate summary: Wanted 2490 Found 2157 Missed 333 Current 0 (86% match, 0% complete)
NOTE: Executing Tasks
NOTE: Setscene tasks completed
ERROR: sbcb-defaultfs-full-1.0-r0 do_sdk_depends: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:extend_recipe_sysroot(d)
0003:
File: '/disk0/scratch/smonsees/yocto/workspace_3/poky/meta/classes/staging.bbclass', lineno: 551, function: extend_recipe_sysroot
0547: dest = newmanifest[l]
0548: if l.endswith("/"):
0549: staging_copydir(l, targetdir, dest, seendirs)
0550: continue
*** 0551: staging_copyfile(l, targetdir, dest, postinsts, seendirs)
0552:
0553: bb.note("Installed into sysroot: %s" % str(msg_adding))
0554: bb.note("Skipping as already exists in sysroot: %s" % str(msg_exists))
0555:
File: '/disk0/scratch/smonsees/yocto/workspace_3/poky/meta/classes/staging.bbclass', lineno: 144, function: staging_copyfile
0140: if os.path.islink(c):
0141: linkto = os.readlink(c)
0142: if os.path.lexists(dest):
0143: if not os.path.islink(dest):
*** 0144: raise OSError(errno.EEXIST, "Link %s already exists as a file" % dest, dest)
0145: if os.readlink(dest) == linkto:
0146: return dest
0147: raise OSError(errno.EEXIST, "Link %s already exists to a different location? (%s vs %s)" % (dest, os.readlink(dest), linkto), dest)
0148: os.symlink(linkto, dest)
Exception: FileExistsError: [Errno 17] Link /disk0/scratch/smonsees/yocto/workspace_3/builds2/sbcb-default/tmp/work/sbcb_default-poky-linux/sbcb-defaultfs-full/1.0-r0/recipe-sysroot/lib already exists as a file: '/disk0/scratch/smonsees/yocto/workspace_3/builds2/sbcb-default/tmp/work/sbcb_default-poky-linux/sbcb-defaultfs-full/1.0-r0/recipe-sysroot/lib'

ERROR: Logfile of failure stored in: /disk0/scratch/smonsees/yocto/workspace_3/builds2/sbcb-default/tmp/work/sbcb_default-poky-linux/sbcb-defaultfs-full/1.0-r0/temp/log.do_sdk_depends.24732
ERROR: Task (/disk0/scratch/smonsees/yocto/workspace_3/poky/../meta-bae/meta-limws/meta-intel/recipes-core/images/sbcb-defaultfs-full.bb:do_sdk_depends) failed with exit code '1'
NOTE: Tasks Summary: Attempted 6892 tasks of which 5539 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
/disk0/scratch/smonsees/yocto/workspace_3/poky/../meta-bae/meta-limws/meta-intel/recipes-core/images/sbcb-defaultfs-full.bb:do_sdk_depends
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
13:01 smonsees@yix490038 /disk0/scratch/smonsees/yocto/workspace_3/builds2/sbcb-default>



Note: Modifying SDK_EXTRA_TOOLS by removing "nativesdk-clang", i.e.:

SDK_EXTRA_TOOLS = " \
nativesdk-cmake \
"

If I remove, the build is much faster, and it builds without the error, also appears not to require/include
"llvm" component in build.
I do not believe the full package was actually built/enabled, but no errors and image is functional.


13:09 smonsees@yix490038 /disk0/scratch/smonsees/yocto/workspace_3/builds2/sbcb-default> bitbake sbcb-defaultfs-full -c populate_sdk_ext
Parsing recipes: 100% |#################################################################################| Time: 0:01:26
Parsing of 2463 .bb files complete (0 cached, 2463 parsed). 3671 targets, 91 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION = "1.44.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "rhel-7.9"
TARGET_SYS = "x86_64-poky-linux"
MACHINE = "sbcb-default"
DISTRO = "limws"
DISTRO_VERSION = "3.0.4"
TUNE_FEATURES = "m64 corei7"
TARGET_FPU = ""
meta
meta-poky = "my_yocto_3.0.4:f2eb22a8783f1eecf99bd4042695bab920eed00e"
meta-perl
meta-python
meta-filesystems
meta-networking
meta-initramfs
meta-oe = "zeus:2b5dd1eb81cd08bc065bc76125f2856e9383e98b"
meta-clang = "zeus:f5355ca9b86fb5de5930132ffd95a9b352d694f9"
meta = "master:a32ddd2b2a51b26c011fa50e441df39304651503"
meta-intel = "zeus:d9942d4c3a710406b051852de7232db03c297f4e"
meta-intel = "v2019.02:f635a364c55f1fb12519aff54924a0a5b947091e"

NOTE: Fetching uninative binary shim from file:///ede/tms/yocto/zeus/downloads/intel/x86_64-nativesdk-libc.tar.xz;sha256sum=a09922172c3a439105e0ae6b943daad2d83505b17da0aba97961ff433b8c21ab
Initialising tasks: 100% |##############################################################################| Time: 0:00:12
Checking sstate mirror object availability: 100% |######################################################| Time: 0:00:00
Sstate summary: Wanted 2436 Found 2407 Missed 29 Current 0 (98% match, 0% complete)
NOTE: Executing Tasks
NOTE: Setscene tasks completed
NOTE: Tasks Summary: Attempted 6767 tasks of which 6223 didn't need to be rerun and all succeeded.
13:31 smonsees@yix490038 /disk0/scratch/smonsees/yocto/workspace_3/builds2/sbcb-default>

Yocto Technical Team Minutes, Engineering Sync, for April 20, 2021
archive: https://docs.google.com/document/d/1ly8nyhO14kDNnFcW2QskANXW3ZT7QwKC5wWVDg9dDH4/edit

== announcements ==
The upcoming Yocto Project Summit is taking place May 25-26 2021
details: https://www.yoctoproject.org/yocto-project-virtual-summit-2021/
CfP: https://pretalx.com/yocto-project-summit-2021/cfp
registration: https://www.cvent.com/d/yjq4dr/4W?ct=868bfddd-ca91-46bb-aaa5-62d2b61b2501

== disclaimer ==
Best efforts are made to ensure the below is accurate and valid. However,
errors sometimes happen. If any errors or omissions are found, please feel
free to reply to this email with any corrections.

== attendees ==
Trevor Woerner, Stephen Jolley, Armin Kuster, Steve Sakoman, Peter
Kjellerstedt, Alexandre Belloni, Michael Opdenacker, Scott Murray, Jon
Mason, Michael Halstead, Richard Purdie, Bruce Ashfield, Ross Burton, Randy
MacLeod, Saul Wold, Tim Orling, Daiane Angolini

== notes ==
- 3.3 (hardknott) released!!
- 3.1.7 (dunfell) through QA, pending TSC approval
- YP TSC: 3 seats re-elected, 2 seats up for re-election
- master branch open for 3.4 (honister) and things are starting to merge
- new command proposed: bitbake-getvar
- AB issues: 59 (going up)

== general ==
RP: the re-election for OE TSC is coming up in August, but the re-election for
the YP TSC is in May. a call for candidates will be going out soon
TrevorW: do we want to align the voting with the conference?
RP: probably not


MichaelO: is there a 3.3 celebration?
RP: usually we do it at the conferences, but we’ll have to do it at the
virtual one this year
MichaelO: i didn’t see anything on lwn
RP: we don’t have a mechanism for this
MichaelO: there are usually release notes
RP: yocto-announce mailing list
TrevorW: release notes?
RP: yes, PaulE put together some notes
TrevorW: is someone putting a release note together?
MichaelO: okay, i’ll put it together and email internally first
RP: advocacy is less that it was, if people can think of places to
help/advertise that would be great. sometime i publish something in the LF
newsletter


RP: bitbake-getvar, instead of “bitbake -e | grep”. saves people some
keystrokes
RP: uses tinfoil, acts as a very good, very small example of how to write a
small tinfoil tool (e.g. c larson’s bb tool). maybe a way of expanding
this infrastructure to make it easier to add tools in the future, perhaps
expandable in layers itself
RP: also perhaps could be used for layer and config setup (e.g. kas,
combo-layer)
AlexB: sounds great! i often show students “bitbake -e | grep” and
they’re often surprised and happy to see it
ScottM: +1
RP: it’s in master-next. there are probably a lot of tutorials that will
need updating after this :-) and there are places in selftest where we do
“bitbake -e | grep” (it runs it once then builds a dictionary of the
results to avoid having to run it multiple times throughout the test run)
RossB: it’s similar to bbvars
RP: i’d like to keep it simple, single variable, i don’t want to get too
much scope creep which makes it more complicated. not meant to solve all
problems, just solve one elegantly. if we add something to take multiple
vars then we’d have to return JSON (?) or something else that would be
machine-parsable
RP: i think a separate command to do multiple vars would be best


PeterK: don't forget to move poky-conf back to master
RP: ah, thanks


Saul: CentOS 8 issues, grrr
RP: i haven’t looked into it
Saul: i think qemu is dying and qmp is trying to read the linux socket and
getting nothing. maybe i could take out the centos builder for a while
RP: sounds reasonable. if you want to pull one builder out to work on it, that
would be fine
Saul: i’ll talk to MichaelH about it


Saul: any cmake experts out there?
<crickets>
Saul: i have a nasty cmake + python issue wrt ceph: it doesn’t find the
core gcc libraries, sysroot being set to /not/exist. i think there’s an
environment passing issue between cmake and the python
RP: that /not/exist is something *we* hardcode into the compiler to make sure
sysroot gets set
Bruce: have a look at what we did in libvirt recently. we had a similar issue
with meson whereby it couldn’t find getent from libc. we had to create a
patch. meson+cmake couldn’t find getent from libc
RossB: i’ll take a look at your patch Bruce, meson acts strange in some
ways, but sometimes there’s a way to straighten it out


ScottM: Steve: 3.1.7 this week?
SteveS: up to TSC to approve it
RP: it’s through QA, QA didn’t flag any issues (there was one failure but
it succeeded on a subsequent build). the TSC meeting is later today, so if
they approve it’ll be out by tomorrow


Randy: latency monitoring. turned down from 10 to 6 seconds, tuned timeout
from 5 to 15 seconds. new column in AB console to link to this new data.
generates graphs of “time to dd 100 kb periodically” to hopefully help
correlate failures with load. we usually see 3-5 seconds but sometimes 50s
for unexplained reasons
RP: that’s very interesting. but we still don’t know what the system is
doing at those times when we see the 20. it’s one thing to know it’s
happening, it’s another to know why. i’ve noticed that webkit builds
add a lot of load, also compression adds a lot of load (xz)
Randy: behind on the rust things since i’ve been working on this


RP: speaking of languages, Go also worries me. Go has 2 issues: the fetcher
is messy and builds are not reproducible. are there any Go experts in our
community who can help?
TimO: Bruce and I have been struggling over a bunch of things, but it’s not
fun
Bruce: working on something; i’m currently up to 37 fetches, but it’s not
just fetching but also a layout issue. if you fetch something to “c/a”
but Go expects to find it at “a” then it won’t use it
Randy: someone wrote a cargo module for Rust to do something similar (i.e. a
translation between Rust and bitbake), would that pattern be useful here?
Bruce: probably, hard to know at what point we need to make deep changes vs
patches on the surface. the “make” infrastructure doesn’t help, too
many hard-coded paths and options in the Go  “make” files
RP: and that’s only ½ the problem (still have build reproducibility issues)
Bruce: Khem can bump the Go version, but then we need to chase after all these
fixes


Randy: suricata?
Armin: yes, those are in meta-security now, donated by ARM
Randy: did you use a cargo build
Armin: it was a couple things, had to start with one but then switch over
partway


TrevorW: is there any update on the “layout of where patches go” work?
RP: you mean “work directory” vs “source directory”?
RP: pretty invasive things, put it aside for now. a change like this will end
up causing lots of follow-on issues for weeks to come. the cleanup would
be nice, don’t know if the cost is worth it and would eat up my time for
a while. although i agree the time would be best now to do it (start of
new release). i think this is the second time i’ve tried


RP: parallel make job server (another such example of something I’ve looked
at and think would be nice, but have had to put aside for the time-being)
RP: the todo list isn’t too bad, but it’s something that fell off to the
side.
RP: i get the feeling that some of the things Randy is trying to track down
(load on AB servers) might end up being helped by having a parallel job
server, but we just don’t know
ScottM: is that branch still around. i’m interested in tinkering with it
RP: ignore the branch, just take the commit.
RP: i had hard-coded the fifo to one place (/tmp) on the machine, which means
every build on that machine is tied together. is that good or bad?
ScottM: is that something you could see being global for all bitbake on the
machine
RP: as long as all the builds are being run by the same user. probably need to
create a separate fifo, give it a relevant name, and place it somewhere
where each user can access it themselves. maybe i should try it on the AB
and see what blows up

On 4/20/21 7:41 AM, Armin Kuster wrote:

Signed-off-by: Armin Kuster <akuster808@...>
---
recipes-core/packagegroup/packagegroup-core-security.bb | 3 +++
1 file changed, 3 insertions(+)
diff --git a/recipes-core/packagegroup/packagegroup-core-security.bb b/recipes-core/packagegroup/packagegroup-core-security.bb
index 9ac0d2c..a6142a8 100644
--- a/recipes-core/packagegroup/packagegroup-core-security.bb
+++ b/recipes-core/packagegroup/packagegroup-core-security.bb
@@ -80,6 +80,9 @@ RDEPENDS_packagegroup-security-mac = " \
${@bb.utils.contains("DISTRO_FEATURES", "smack", "smack", "",d)} \
"
+RDEPENDS_packagegroup-security-mac_remove_mips64 = "apparmor"
+RDEPENDS_packagegroup-security-mac_remove_mips64le = "apparmor"
+

this should be mips64el

RDEPENDS_packagegroup-meta-security-ptest-packages = "\
ptest-runner \
samhain-standalone-ptest \

On 4/20/21 7:00 AM, Monsees, Steven C (US) via lists.yoctoproject.org wrote:

I noticed similar behavior…
I am running zeus 3.0.4, “devtool sdk-install llvm” will get llvm 8.0.1…
When I build meta-clang, and I set TOOLCHAIN?=”clang” in local.conf it appears to grab llvm-project-source-9.0.1-9.0.1…

this is intentional, when you use meta-clang, then llvm is preferred from LLVM since them we have consistent version of llvm for clang and others.

*From:*yocto@... <yocto@...> *On Behalf Of *Anton Antonov
*Sent:* Tuesday, April 20, 2021 9:51 AM
*To:* yocto@...
*Subject:* Re: [yocto] #yocto llvm support
*_External Email Alert_*
*This email has been sent from an account outside of the BAE Systems network.*
Please treat the email with caution, especially if you are requested to click on a link, decrypt/open an attachment, or enable macros.  For further information on how to spot phishing, access “Cybersecurity OneSpace Page” and report phishing by clicking the button “Report Phishing” on the Outlook toolbar.
Hi Steven,
  I used meta-clang in my recipes and I noticed that:
1. The current release of poky uses LLVM v11.1.0 by default (poky/meta/recipes-devtools/llvm/llvm_git.bb)
2. Meta-clang requires LLVM v12.0.0 (meta-clang/conf/layer.conf defines LLVMVERSION = "12.0.0")
As a result just including meta-clang into bblayers.conf will require bitbake to build a new version of LLVM and rebuild everything which uses it
Anton

Signed-off-by: Armin Kuster <akuster808@...>
---
recipes-core/packagegroup/packagegroup-core-security.bb | 3 +++
1 file changed, 3 insertions(+)

diff --git a/recipes-core/packagegroup/packagegroup-core-security.bb b/recipes-core/packagegroup/packagegroup-core-security.bb
index 9ac0d2c..a6142a8 100644
--- a/recipes-core/packagegroup/packagegroup-core-security.bb
+++ b/recipes-core/packagegroup/packagegroup-core-security.bb
@@ -80,6 +80,9 @@ RDEPENDS_packagegroup-security-mac = " \
${@bb.utils.contains("DISTRO_FEATURES", "smack", "smack", "",d)} \
"

+RDEPENDS_packagegroup-security-mac_remove_mips64 = "apparmor"
+RDEPENDS_packagegroup-security-mac_remove_mips64le = "apparmor"
+
RDEPENDS_packagegroup-meta-security-ptest-packages = "\
ptest-runner \
samhain-standalone-ptest \
--
2.17.1

Hi Steven,

  I used meta-clang in my recipes and I noticed that:
1. The current release of poky uses LLVM v11.1.0 by default (poky/meta/recipes-devtools/llvm/llvm_git.bb)
2. Meta-clang requires LLVM v12.0.0 (meta-clang/conf/layer.conf defines LLVMVERSION = "12.0.0")

As a result just including meta-clang into bblayers.conf will require bitbake to build a new version of LLVM and rebuild everything which uses it

Anton

Obviously this is to the wrong list!

It escaped before I could stop it :D

I've already resent to oe-core.

Bruce

On Tue, Apr 20, 2021 at 8:31 AM Bruce Ashfield via
lists.yoctoproject.org
<bruce.ashfield=gmail.com@...> wrote:

--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

From: Bruce Ashfield <bruce.ashfield@...>

A recent fix to the kern-tools promoted some previously unseen
issues to warnings. This commit fixes them by tagging some BT
options as non-hardware so they won't generate warnings if they
don't appear in the final .config. These are sub BT options and
shouldn't warn when/if their controlling option is disabled by
a fragment.

d7fd0213b75 base: exclude some BT options as non-hardware

Signed-off-by: Bruce Ashfield <bruce.ashfield@...>
---
meta/recipes-kernel/linux/linux-yocto-rt_5.4.bb | 2 +-
meta/recipes-kernel/linux/linux-yocto-tiny_5.4.bb | 2 +-
meta/recipes-kernel/linux/linux-yocto_5.4.bb | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-kernel/linux/linux-yocto-rt_5.4.bb b/meta/recipes-kernel/linux/linux-yocto-rt_5.4.bb
index e52c9ebe0e..a802a570a1 100644
--- a/meta/recipes-kernel/linux/linux-yocto-rt_5.4.bb
+++ b/meta/recipes-kernel/linux/linux-yocto-rt_5.4.bb
@@ -12,7 +12,7 @@ python () {
}

SRCREV_machine ?= "324e77d816cf6434507ab29140beb24044009efa"
-SRCREV_meta ?= "5017570a130f142365dfdcd043d84143fdd6e255"
+SRCREV_meta ?= "d7fd0213b75ce9b6206f63dbdd435ab326598642"

SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;branch=${KBRANCH};name=machine \
git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.4;destsuffix=${KMETA}"
diff --git a/meta/recipes-kernel/linux/linux-yocto-tiny_5.4.bb b/meta/recipes-kernel/linux/linux-yocto-tiny_5.4.bb
index f07375c761..1edc632de7 100644
--- a/meta/recipes-kernel/linux/linux-yocto-tiny_5.4.bb
+++ b/meta/recipes-kernel/linux/linux-yocto-tiny_5.4.bb
@@ -17,7 +17,7 @@ KCONF_BSP_AUDIT_LEVEL = "2"

SRCREV_machine_qemuarm ?= "8463db325b93f0669446f68c19334cfe11ffb9c2"
SRCREV_machine ?= "5f54b437b6502d3febee553100b2cb2a9e0c5f8a"
-SRCREV_meta ?= "5017570a130f142365dfdcd043d84143fdd6e255"
+SRCREV_meta ?= "d7fd0213b75ce9b6206f63dbdd435ab326598642"

PV = "${LINUX_VERSION}+git${SRCPV}"

diff --git a/meta/recipes-kernel/linux/linux-yocto_5.4.bb b/meta/recipes-kernel/linux/linux-yocto_5.4.bb
index 4f056d6d82..3a9463af6a 100644
--- a/meta/recipes-kernel/linux/linux-yocto_5.4.bb
+++ b/meta/recipes-kernel/linux/linux-yocto_5.4.bb
@@ -23,7 +23,7 @@ SRCREV_machine_qemux86 ?= "5f54b437b6502d3febee553100b2cb2a9e0c5f8a"
SRCREV_machine_qemux86-64 ?= "5f54b437b6502d3febee553100b2cb2a9e0c5f8a"
SRCREV_machine_qemumips64 ?= "996fe040c8d8d01a9af6be42dae3844d127471bf"
SRCREV_machine ?= "5f54b437b6502d3febee553100b2cb2a9e0c5f8a"
-SRCREV_meta ?= "5017570a130f142365dfdcd043d84143fdd6e255"
+SRCREV_meta ?= "d7fd0213b75ce9b6206f63dbdd435ab326598642"

# remap qemuarm to qemuarma15 for the 5.4 kernel
# KMACHINE_qemuarm ?= "qemuarma15"
--
2.19.1

I tend to use a reasonably standard image, usually the one that will evolve into the product, and once the network (usb, ethernet or wifi) is up, use "opkg install" to grab whatever else I need from my build machine on demand.




Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@...
W: www.topic.nl

Please consider the environment before printing this e-mail

--
Mike Looijmans

Hi All,

This is the full report for yocto-3.1.7.rc1:
https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/tree/?h=intel-yocto-testresults

======= Summary ========
No high milestone defects.

no new issue found

Thanks,
Sangeeta

Hi,

On Tue, Apr 20, 2021 at 03:24:25AM -0700, anthony.marchand@... wrote:

Hello,
I'm actually migrating a yocto project zeus version to dunfell LTS to build my imx6 Linux system on this LTS release.

Almost all works fine except the following message I try to understand when I try to compile the image:

---------------------------------------------------------------------------------------------------------------------------------------------------------
ERROR: Multiple .bb files are due to be built which each provide u-boot-fw-utils:                       | ETA:  0:00:01
Dunfell/build/../meta-mymeta/recipes-bsp/u-boot/u-boot-fw-utils_2018.09.bb
Dunfell/build/../openembedded-core/meta/recipes-bsp/u-boot/libubootenv_0.3.1.bb
A list of tasks depending on these providers is shown and may help explain where the dependency comes from.
Dunfell/build/../meta-mymeta/recipes-bsp/u-boot/u-boot-fw-utils_2018.09.bb has unique dependees:

Dunfell/build/../openembedded-core/meta/recipes-bsp/u-boot/libubootenv_0.3.1.bb has unique dependees:
Dunfell/build/../meta-swupdate/recipes-support/swupdate/swupdate_2020.11.bb:do_build
Dunfell/build/../meta-swupdate/recipes-support/swupdate/swupdate_2020.11.bb:do_package
Dunfell/build/../meta-swupdate/recipes-support/swupdate/swupdate_2020.11.bb:do_prepare_recipe_sysroot
It could be that one recipe provides something the other doesn't and should. The following provider and runtime provider differences may be helpful.
Dunfell/build/../meta-mymeta/recipes-bsp/u-boot/u-boot-fw-utils_2018.09.bb has unique provides:

Dunfell/build/../meta-mymeta/recipes-bsp/u-boot/u-boot-fw-utils_2018.09.bb has unique rprovides:
u-boot-fw-utils-dev
u-boot-fw-utils-locale
u-boot-fw-utils-dbg
u-boot-fw-utils-staticdev
u-boot-fw-utils-doc
^u-boot-fw-utils-locale-.*
u-boot-fw-utils-src
Dunfell/build/../openembedded-core/meta/recipes-bsp/u-boot/libubootenv_0.3.1.bb has unique provides:
libubootenv
Dunfell/build/../openembedded-core/meta/recipes-bsp/u-boot/libubootenv_0.3.1.bb has unique rprovides:
libubootenv
libubootenv-src
libubootenv-bin
libubootenv-dbg
libubootenv-doc
^libubootenv-locale-.*
libubootenv-locale
libubootenv-dev
libubootenv-staticdev
---------------------------------------------------------------------------------------------------------------------------------------------------------
Does someone already meet this problem? Do you know what I should look in for to solve it?

Multiple ways to fix this. One is to switch to using dunfell branches for the
BSP SW stack, if they are available. Alternatively you can use the old
zeus based BSP SW stack but for that you need to copy the files which
BSP SW stack depends on poky side from zeus branch to the BSP SW meta layer.

Then things will work out with minor bug fixes or API changes.

For IMX*, I copied the old u-boot recipe files from poky zeus branch over to
the BSP so that meta-freescale* and meta-imx find them and prefer over
the newer u-boot version from poky dunfell branch.

Hope this helps,

-Mikko

Thanks by advance, best reguards.

Hello,
I'm actually migrating a yocto project zeus version to dunfell LTS to build my imx6 Linux system on this LTS release.

Almost all works fine except the following message I try to understand when I try to compile the image:

---------------------------------------------------------------------------------------------------------------------------------------------------------