Yocto Autobuilder: Latency Monitor and AB-INT - Meeting notes: June 10, 2021
(Buried on my desktop, better late than never)
Attendees: Alex, Richard, Saul, Randy, Tony, Trevor 1. valgrind: - taskset work-around really helps. - working on stack_changes bug for qemuarm43 - reproduced on master, hardknott but not gatesgarth - Let's drop it for now to reduce the AB noise. 2. task summary script: - procpath compared to top -bn1 - about the same number of syscalls (strace) - about 5x more cpu and 2x more ram. - could be a problem in that we'd need to boostrap since procpath-native would need to be built at the beginning of each bitbake run or be added to buildtools or hacked into y-ab-helper. - it might be simpler to screen-scrape top output... 3. make: job server - working and able to build images now. - testing to confirm that it actually limits the load - we'll submit for master-next once that's confirmed. 4. There's a problem in qemu/kernel corruption apparently from an ltp test. Richard has done lots of work to go back to the near pristine 5.10 kernel and the problem still happens so that seems to eliminate the possibility that it's a linux-yocto specific issue. Richard has a patch in here: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=d7d65aae104caa03afc28837b0abe0b486d5a8b8 and to reproduce the problem you should be able to just run: IMAGE_INSTALL_append = ' ltp' TEST_SUITES = 'ping ssh ltp' then bitbake core-image-sato; bitbake core-image-sato -c testimage ( qemux86-64, with kvm) then look for BUG: in log in core-image-sato WORKDIR/testimage/qemu* i.e.: tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/testimage/ Reproducible about 70% of the time. RP: revert qemu-6.0 to 5.x: The problem persists. using ubu-20.04 5. Remaining more frequent issues: valgrind: https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2168 6. There are 3 patches in master-next that we need to decide on: - library preload: read qemu libs into memory. - helpful but maybe treat as a bandaid until we are able to reduce the IO and cpu spikes. - qemu-runner? timeout increase 120 -> 240 - ptest timeouts 300 -> 450? 7. The additional (top and qemu log) output when qemu hangs is merged. Thanks Sakib. 8. Plans for the week: Richard: build M1, debug ltp issue. Alex: full day on ltp tomorrow? Sakib: task summary Trevor: make job server Tony: more valgrind bugs/work-arounds. Saul: have QMP deal with sigusr1 to close the QMP socket Randy: test ltp failure
|
|
Re: bitbake controlling memory use
Ferry Toth
Hi
Op 13-06-2021 om 02:38 schreef Randy MacLeod: On 2021-06-12 12:31 p.m., Ferry Toth wrote:I watched running processes from KDE ksysguard and I believe the number of compilers was actually restricted with the patch. On the other hand the server I tried was running munin which logs #processes over time and there I didn't see a difference. Confused..HiHi Ferry, the next thing we try will be to patch 'make' to log when the jobserverI'm available for testing. A little RTFM / UTSL may also be required.I do like the jobserver idea a lot. Especially if it would take memory restrictions into account as well. The problem with building nodejs (and probably rust as well), there is lots to compile so you really want -j 16. But then during linking ld uses 4GB per instance, and there are 5 started. So on my machine I wouldn't want to start more then 3. - Trevor
|
|
Re: bitbake controlling memory use
On 2021-06-12 12:31 p.m., Ferry Toth wrote:
HiHi Ferry, Thanks for the update. Trevor and I saw similar (lack of ) results. Trevor even trying getting kea, which uses 'make' to be done the 'configure' stage, for two builds in differect dirs. Then to run the two 'bitbake -c compile kea' with and with out the patch with the expectation that with the job server patch and the right number of jobs, the two builds would take longer. I don't know the exact timing but there was no noticeable difference. We did strace things to confirm that the make wrapper was being called and the actual make was being called by the wrapper. I suspect that the next thing we try will be to patch 'make' to log when the jobserver kicks in or to play with some make jobserver demo such as: https://github.com/olsner/jobclient to get some experience with how things are supposed to work and to be able to strace a successful use of the job server feature. A little RTFM / UTSL may also be required. ../Randy This is on 4 core/8ht i7-3770 CPU @ 3.40GHz with 16Gb RAM and nodejs restricted to -j 2 (so that alone takes ~ 60min to build).- Trevor -- # Randy MacLeod # Wind River Linux
|
|
Re: bitbake controlling memory use
Ferry Toth
Hi
Op 10-06-2021 om 22:35 schreef Ferry Toth: Hi,It works fine. To measure time I first built https://github.com/htot/meta-intel-edison (gatesgarth), so everything needed is downloaded and cached. Then prior to each run I `rm -rf out` and `rm -rf bbcache/sstate-cache/*` to force everything to rebuild. And then `time bitbake -k edison-image` With patch: real 218m12,686s user 0m24,058s sys 0m4,379s Without: real 219m36,944s user 0m24,770s sys 0m4,266s Strange, I expected more. This is on 4 core/8ht i7-3770 CPU @ 3.40GHz with 16Gb RAM and nodejs restricted to -j 2 (so that alone takes ~ 60min to build). - Trevor
|
|
QA notification for completed autobuilder build (yocto-3.4_M1.rc1)
Pokybuild User <pokybuild@...>
A build flagged for QA (yocto-3.4_M1.rc1) was completed on the autobuilder and is available at:
https://autobuilder.yocto.io/pub/releases/yocto-3.4_M1.rc1 Build hash information: bitbake: 398a1686176c695d103591089a36e25173f9fd6e meta-arm: 6c3d62c776fc45b4bae47d178899e84b17248b31 meta-gplv2: 1ee1a73070d91e0c727f9d0db11943a61765c8d9 meta-intel: 0937728bcd98dd13d2c6829e1cd604ea2e53e5cd meta-mingw: bfd22a248c0db4c57d5a72d675979d8341a7e9c1 oecore: 3b2903ccc791d5dedd84c75227f38ae4c8d29251 poky: 59d93693bf24e02ca0f05fe06d96a46f4f0f1bf8 This is an automated message from the Yocto Project Autobuilder Git: git://git.yoctoproject.org/yocto-autobuilder2 Email: richard.purdie@...
|
|
Re: Extensible SDK of core-image-minimal fails to install
Hi,
On 12/06/2021 10:47, Richard Purdie wrote: I think it may be this bug:Not a fix, but a workaround, if you are willing to use containers is to create a build container and a similar sdk container. I added more information to the bug above. Regards, Robert
|
|
Re: Extensible SDK of core-image-minimal fails to install
Richard Purdie
On Fri, 2021-06-11 at 22:30 +0200, Hendrik wrote:
I was able to reproduce the error even with plain poky (without kas):I think it may be this bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14428 We don't have a fix as yet, it looks difficult to solve :( Cheers, Richard
|
|
Re: Extensible SDK of core-image-minimal fails to install
Hendrik
I was able to reproduce the error even with plain poky (without kas):
toggle quoted messageShow quoted text
```bash git clone -b dunfell https://git.yoctoproject.org/git/poky cd poky source oe-init-build-env bitbake -c populate_sdk_ext core-image-minimal ``` After successful build, copy the `build/tmp/deploy/sdk/poky....sh` to another machine (with different OS) and run it there. I tried several different installation machines and they fail consistently to install the extensible SDK if they run a different OS than the build machine (e.g. Debian 10 vs. Ubuntu 18.04 vs. 20.04). See the exakt error and complete log output in my previous message. What is happening here? Did I use the extensible SDK wrongly? Is it expected to be incompatible? Thanks in advance! Hendrik
On 09.06.21 17:58, Hendrik wrote:
Hello World,
|
|
Re: [meta-java] icedtea7 fetching error
Alexander Kanavin
I have reconfigured our build system to use an internal server via bbappend: ICEDTEA_HG_URL = "https://internal.artifact.server/icedtea/" placed all the tarballs there according to expected locations, and it works just fine. Alex
On Fri, 11 Jun 2021 at 12:44, <stefano.fiore84@...> wrote:
|
|
Re: QEMU Size Increase from Yocto Thud to Zeus
Ross Burton <ross@...>
Are you installing qemu into your image though?
toggle quoted messageShow quoted text
Qemu did get larger as it is built with more architectures enabled, but unless you're installing it in your image it won't make a difference. Ross
On Fri, 11 Jun 2021 at 11:40, Aashik Aswin <thisisaash9698@...> wrote:
|
|
Re: [meta-java] icedtea7 fetching error
stefano.fiore84@...
I have also the tarball but the build will fail later in the configure phase stating: Is there something more I can try ?
|
|
QEMU Size Increase from Yocto Thud to Zeus
Aashik Aswin
Hi Experts, I am upgrading my Linux from Yocto Thud to Zeus (5.4 Kernel) . After building I could see a significant increase in the size of the image. On checking with buildhistory enabled, I could see that qemu has nearly doubled in size. Thud (4.19) - 223084 KiB qemu Zeus (5.4) - 474757 KiB qemu Is this size increase expected or are there some additional configs that might have been added as a part of the upgrade ? Appreciate your help. TIA, Aashik
|
|
[meta-zephyr][PATCH 3/3] zephyr-flash-bossac.bbclass: Enable board flashing via bossac
Wojciech Zmuda
From: Nagesh Shamnur <nagesh.shamnur@...>
Currently only dfu and pyocd flashing are supported. Some boards such as Arduino Nano 33 BLE can be flashed via bossac which is released by Arduino repo. Find the installed Arudino version of bossac and flash using that tool. Signed-off-by: Nagesh Shamnur <nagesh.shamnur@...> --- classes/zephyr-flash-bossac.bbclass | 50 +++++++++++++++++++++++++++ conf/machine/arduino-nano-33-ble.conf | 1 + 2 files changed, 51 insertions(+) create mode 100644 classes/zephyr-flash-bossac.bbclass diff --git a/classes/zephyr-flash-bossac.bbclass b/classes/zephyr-flash-bossac.bbclass new file mode 100644 index 0000000..50222d5 --- /dev/null +++ b/classes/zephyr-flash-bossac.bbclass @@ -0,0 +1,50 @@ +#@DESCRIPTION: class file to flash boards like Arduino Nano BLE which depends on bossac for flashing + +python do_flash_usb() { + import shutil + import subprocess + import serial.tools.list_ports + + # Note: make sure the installed bossac is set to PATH before running flash_usb() + # Check if bossac is avaiable for flashing + origbbenv = d.getVar("BB_ORIGENV", False) + bossac_path = shutil.which("bossac", path=origbbenv.getVar('PATH')) + + if not bossac_path: + bb.fatal("ERROR: bossac not found, please install first and add to PATH") + + board = d.getVar('BOARD') + + if board == 'arduino_nano_33_ble': + # find the serial port to which board is connected to + for port in serial.tools.list_ports.comports(): + if 'Arduino Nano 33 BLE' in port.description: + serial_port = port.device + break + else: + bb.fatal("ERROR: board not connected for flashing. Connect via USB and enable permission to connected port") + + image = "%s/%s.bin" % (d.getVar('DEPLOY_DIR_IMAGE'), d.getVar('PN')) + + command = [bossac_path, '-p', serial_port , '-R', '-e', '-w', '-v', '-b', image] + else: + bb.fatal("ERROR: Unsupported board %s" % board) + + bb.note("command: %s" % command) + bb.plain("Attempting to flash board: %s" % board) + + # Random failure are a possibility here, retry till there is a success for finite times + for _ in range(10, 0, -1): + try: + subprocess.check_call(command) + bb.plain("Bossac Flashing board: %s Success " % board) + break + except subprocess.CalledProcessError as e: + bb.warn("Failed to flash %s (error code: %s). Retrying after 1 second..." % (board, e.returncode)) + time.sleep(1) +} + +addtask do_flash_usb after do_deploy + +do_flash_usb[nostamp] = "1" +do_flash_usb[vardepsexclude] = "BB_ORIGENV" diff --git a/conf/machine/arduino-nano-33-ble.conf b/conf/machine/arduino-nano-33-ble.conf index e45cfc2..18ba056 100644 --- a/conf/machine/arduino-nano-33-ble.conf +++ b/conf/machine/arduino-nano-33-ble.conf @@ -4,5 +4,6 @@ #@DESCRIPTION: Machine configuration for Arudino nano 33 ble and Arduino nano 33 ble sense require conf/machine/include/nrf52.inc +ZEPHYR_INHERIT_CLASSES += "zephyr-flash-bossac" ARCH_arduino-nano-33-ble = "arm" -- 2.25.1
|
|
[meta-zephyr][PATCH 2/3] arduino-nano-33-ble.conf: Add Arduino Nano 33 BLE and BLE Sense support
Wojciech Zmuda
From: Nagesh Shamnur <nagesh.shamnur@...>
Signed-off-by: Nagesh Shamnur <nagesh.shamnur@...> --- conf/machine/arduino-nano-33-ble.conf | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 conf/machine/arduino-nano-33-ble.conf diff --git a/conf/machine/arduino-nano-33-ble.conf b/conf/machine/arduino-nano-33-ble.conf new file mode 100644 index 0000000..e45cfc2 --- /dev/null +++ b/conf/machine/arduino-nano-33-ble.conf @@ -0,0 +1,8 @@ +#@TYPE: Machine +#@NAME: arduino-nano-33-ble + +#@DESCRIPTION: Machine configuration for Arudino nano 33 ble and Arduino nano 33 ble sense + +require conf/machine/include/nrf52.inc +ARCH_arduino-nano-33-ble = "arm" + -- 2.25.1
|
|
[meta-zephyr][PATCH 1/3] zephyr-kernel: install .bin image if available
Wojciech Zmuda
From: Nagesh Shamnur <nagesh.shamnur@...>
Some boards (e.g. Arduino Nano 33 BLE) require image in bin format for flashing with `-c flash_usb`. Provide that image along with ELF image on do_deploy step. Signed-off-by: Nagesh Shamnur <nagesh.shamnur@...> Signed-off-by: Stefan Schmidt <stefan.schmidt@...> --- recipes-kernel/zephyr-kernel/zephyr-kernel-common.inc | 1 + recipes-kernel/zephyr-kernel/zephyr-sample.inc | 4 ++++ 2 files changed, 5 insertions(+) diff --git a/recipes-kernel/zephyr-kernel/zephyr-kernel-common.inc b/recipes-kernel/zephyr-kernel/zephyr-kernel-common.inc index 330fe59..46f19e2 100644 --- a/recipes-kernel/zephyr-kernel/zephyr-kernel-common.inc +++ b/recipes-kernel/zephyr-kernel/zephyr-kernel-common.inc @@ -12,6 +12,7 @@ ZEPHYR_GCC_VARIANT="yocto" ZEPHYR_SYSROOT="${STAGING_DIR_TARGET}" ZEPHYR_MAKE_OUTPUT = "zephyr.elf" +ZEPHYR_MAKE_BIN_OUTPUT = "zephyr.bin" EXTRA_OECMAKE = "\ -DZEPHYR_BASE=${S} \ diff --git a/recipes-kernel/zephyr-kernel/zephyr-sample.inc b/recipes-kernel/zephyr-kernel/zephyr-sample.inc index f7621d1..7b49611 100644 --- a/recipes-kernel/zephyr-kernel/zephyr-sample.inc +++ b/recipes-kernel/zephyr-kernel/zephyr-sample.inc @@ -9,5 +9,9 @@ do_install[noexec] = "1" do_deploy () { install -D ${B}/zephyr/${ZEPHYR_MAKE_OUTPUT} ${DEPLOYDIR}/${PN}.elf + if [ -f ${B}/zephyr/${ZEPHYR_MAKE_BIN_OUTPUT} ] + then + install -D ${B}/zephyr/${ZEPHYR_MAKE_BIN_OUTPUT} ${DEPLOYDIR}/${PN}.bin + fi } addtask deploy after do_compile -- 2.25.1
|
|
[meta-zephyr][PATCH 0/3] Add Arduino Nano 33 BLE and BLE Sense support
Wojciech Zmuda
From: Wojciech Zmuda <wojciech.zmuda@...>
This patch set adds support for nRF52-based Arduino boards - Nano 33 BLE and BLE Sense - to both build and flash Zephyr applications. The board support itself is a trivial MACHINE config. Flashing is done with a new bbclass, that is a wrapper around `bossac` flashing tool. One general kernel recipe fix was necessary to produce and copy over the .bin file, as the flasher requires input in that format. This does not affect scenarios where .elf file is required. Tested on an actual Arduino Nano 33 BLE dev board. Nagesh Shamnur (3): zephyr-kernel: install .bin image if available arduino-nano-33-ble.conf: Add Arduino Nano 33 BLE and BLE Sense support zephyr-flash-bossac.bbclass: Enable board flashing via bossac classes/zephyr-flash-bossac.bbclass | 50 +++++++++++++++++++ conf/machine/arduino-nano-33-ble.conf | 9 ++++ .../zephyr-kernel/zephyr-kernel-common.inc | 1 + .../zephyr-kernel/zephyr-sample.inc | 4 ++ 4 files changed, 64 insertions(+) create mode 100644 classes/zephyr-flash-bossac.bbclass create mode 100644 conf/machine/arduino-nano-33-ble.conf -- 2.25.1
|
|
[meta-zephyr][PATCH 1/1] zephyr-kernel-test: fix Cortex-M tests failure with 2.6.0 kernel
Wojciech Zmuda
From: Nagesh shamnur <nagesh.shamnur@...>
Edit the test recipe removing obj_tracing tests that have been removed from Zephyr 2.6.0 release. Signed-off-by: Nagesh Shamnur <nagesh.shamnur@...> Signed-off-by: Wojciech Zmuda <wojciech.zmuda@...> --- recipes-kernel/zephyr-kernel/zephyr-kernel-test.inc | 1 - 1 file changed, 1 deletion(-) diff --git a/recipes-kernel/zephyr-kernel/zephyr-kernel-test.inc b/recipes-kernel/zephyr-kernel/zephyr-kernel-test.inc index b6b4766..78747f9 100644 --- a/recipes-kernel/zephyr-kernel/zephyr-kernel-test.inc +++ b/recipes-kernel/zephyr-kernel/zephyr-kernel-test.inc @@ -36,7 +36,6 @@ ZEPHYRTESTS = " \ mp \ msgq \ mutex \ - obj_tracing \ pending \ pipe \ poll \ -- 2.25.1
|
|
[meta-zephyr][PATCH 0/1] Partially fix tests failure with 2.6.0 kernel
Wojciech Zmuda
From: Wojciech Zmuda <wojciech.zmuda@...>
Transition to Zephyr 2.6.0 broke meta-zephyr test suite. This patch fixes QEMU Cortex-M3 test suite (MACHINE=qemu-cortex-m3 bitbake zephyr-kernel-test-all -c testimage). The reason of failure is that the `obj_tracing` test suite has been removed from Zephyr sources, but it still is referenced in zephyr-kernel-test.inc. Unfortunately, there are other problems in this area not solved in this path. We'd like to take the opportunity to report it anyway. The QEMU x86 test suite build fails because of unimplemented instructions: | /build/work/i586-yocto-elf/interrupt/2.6.0+gitAUTOINC+79a6c07536_c3bd2094f9-r0/git/tests/kernel/interrupt/src/direct_isr.c: In function 'direct_isr1': | /build/work/i586-yocto-elf/interrupt/2.6.0+gitAUTOINC+79a6c07536_c3bd2094f9-r0/git/tests/kernel/interrupt/src/direct_isr.c:47:1: sorry, unimplemented: 80387 instructions aren't allowed in an interrupt service routine | 47 | ISR_DIRECT_DECLARE(direct_isr1) | | ^~~~~~~~~~~~~~~~~~ | /build/work/i586-yocto-elf/interrupt/2.6.0+gitAUTOINC+79a6c07536_c3bd2094f9-r0/git/tests/kernel/interrupt/src/direct_isr.c: In function 'direct_isr2': | /build/work/i586-yocto-elf/interrupt/2.6.0+gitAUTOINC+79a6c07536_c3bd2094f9-r0/git/tests/kernel/interrupt/src/direct_isr.c:54:1: sorry, unimplemented: 80387 instructions aren't allowed in an interrupt service routine | 54 | ISR_DIRECT_DECLARE(direct_isr2) | | ^~~~~~~~~~~~~~~~~~ The problem above does not occur in case of Cortex-M3 QEMU tests - `interrupt` builds and passes just fine. We have not investigated this one further. Nagesh shamnur (1): zephyr-kernel-test: fix Cortex-M tests failure with 2.6.0 kernel recipes-kernel/zephyr-kernel/zephyr-kernel-test.inc | 1 - 1 file changed, 1 deletion(-) -- 2.25.1
|
|
[meta-zephyr][PATCH] zephyr-blinky: add sample app recipe
Wojciech Zmuda
From: Davide Ricci <davide.ricci@...>
Blinky is the most referenced sample in Zephyr's documentation and recall Arduino's first sketch example. This .bb file allows to build such example. Signed-off-by: Davide Ricci <davide.ricci@...> --- recipes-kernel/zephyr-kernel/zephyr-blinky.bb | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 recipes-kernel/zephyr-kernel/zephyr-blinky.bb diff --git a/recipes-kernel/zephyr-kernel/zephyr-blinky.bb b/recipes-kernel/zephyr-kernel/zephyr-blinky.bb new file mode 100644 index 0000000..bd5ce4f --- /dev/null +++ b/recipes-kernel/zephyr-kernel/zephyr-blinky.bb @@ -0,0 +1,3 @@ +include zephyr-sample.inc + +ZEPHYR_SRC_DIR = "${S}/samples/basic/blinky" -- 2.25.1
|
|
Re: [meta-zephyr][PATCH v2 1/2] zephyr-kernel: Add OpenThread add module to build
Naveen Saini
toggle quoted messageShow quoted text
-----Original Message-----This is not required. It already listed in required sample recipe. Please remove it.
It would cause build failure with v2.5.0. So add SRCREV_openthread in zephyr-kernel-src-2.5.0.inc too. rtos/libmetal.git;protocol=https;destsuffix=git/modules/hal/libmetal;name=l
|
|