|
Re: nodejs do_compile eats all resources
Oliver Westermann <oliver.westermann@...>
Am Dienstag, 27. September 2022 10:50 schrieb Alexander Kanavin <alex.kanavin@...>:Yeah, I'm currently using PARALLEL_MAKE = "-j ${@int(oe.utils.cpu_count() / 4)}" In a .bbappend to limit it, but keep it dynamic. One question out of curiosity or lack of understanding: Bitbake offers BB_NUMBER_THREADS to limit (IMHO) the number of bitbake tasks, eg parallel do_compile() tasks. Each of these however spawn the compiler with several parallel threads oe.utils.cpu_count (PARALLEL_MAKE defaults to cpu_count()). Why isn't it more common that we end up in issues due to cpu_count^2 threads being spawned? Shouldn't (roughly) BB_NUMBER_THREADS * PARALLEL_MAKE fully load the CPU? Best regards, a curious Olli |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Re: nodejs do_compile eats all resources
Alexander Kanavin
Do keep in mind that PARALLEL_MAKE can and should be set per recipe,
so you can make-limit only the worst items. Alex On Tue, 27 Sept 2022 at 10:34, Westermann, Oliver <Oliver.Westermann@...> wrote: Am Montag, 26. September 2022 19:33 schrieb Alexander Kanavin <alex.kanavin@...>I expected something like this. Thanks for the confirmation and I will try to play with PARALLEL_MAKE to reduce the impact. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Re: nodejs do_compile eats all resources
Oliver Westermann <oliver.westermann@...>
Am Montag, 26. September 2022 19:33 schrieb Alexander Kanavin <alex.kanavin@...>I expected something like this. Thanks for the confirmation and I will try to play with PARALLEL_MAKE to reduce the impact. - Olli |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
[ANNOUNCEMENT] Yocto Project 4.0.4 is Released
Lee Chee Yang
Hi
We are pleased to announce the Yocto Project 4.0.4 Release is now available for download.
A gpg signed version of these release notes is available at:
http://downloads.yoctoproject.org/releases/yocto/yocto-4.0.4/RELEASENOTES Full Test Report:
http://downloads.yoctoproject.org/releases/yocto/yocto-4.0.4/testreport.txt Thank you for everyone's contributions to this release.
Chee Yang Yocto Project Build and Release - -------------------------- yocto-4.0.4 Release Notes - --------------------------
- -------------------------- Repositories/Downloads - --------------------------
Repository Name: poky Repository Location: https://git.yoctoproject.org/git/poky Branch: kirkstone Tag: yocto-4.0.4 Git Revision: d64bef1c7d713b92a51228e5ade945835e5a94a4 Release Artefact: poky-d64bef1c7d713b92a51228e5ade945835e5a94a4 sha: b5e92506b31f88445755bad2f45978b747ad1a5bea66ca897370542df5f1e7db Download Locations: http://downloads.yoctoproject.org/releases/yocto/yocto-4.0.4/poky-d64bef1c7d713b92a51228e5ade945835e5a94a4.tar.bz2 http://mirrors.kernel.org/yocto/yocto/yocto-4.0.4/poky-d64bef1c7d713b92a51228e5ade945835e5a94a4.tar.bz2
Repository Name: openembedded-core Repository Location: https://git.openembedded.org/openembedded-core Branch: kirkstone Tag: yocto-4.0.4 Git Revision: f7766da462905ec67bf549d46b8017be36cd5b2a Release Artefact: oecore-f7766da462905ec67bf549d46b8017be36cd5b2a sha: ce0ac011474db5e5f0bb1be3fb97f890a02e46252a719dbcac5813268e48ff16 Download Locations: http://downloads.yoctoproject.org/releases/yocto/yocto-4.0.4/oecore-f7766da462905ec67bf549d46b8017be36cd5b2a.tar.bz2 http://mirrors.kernel.org/yocto/yocto/yocto-4.0.4/oecore-f7766da462905ec67bf549d46b8017be36cd5b2a.tar.bz2
Repository Name: meta-mingw Repository Location: https://git.yoctoproject.org/git/meta-mingw Branch: kirkstone Tag: yocto-4.0.4 Git Revision: a90614a6498c3345704e9611f2842eb933dc51c1 Release Artefact: meta-mingw-a90614a6498c3345704e9611f2842eb933dc51c1 sha: 49f9900bfbbc1c68136f8115b314e95d0b7f6be75edf36a75d9bcd1cca7c6302 Download Locations: http://downloads.yoctoproject.org/releases/yocto/yocto-4.0.4/meta-mingw-a90614a6498c3345704e9611f2842eb933dc51c1.tar.bz2 http://mirrors.kernel.org/yocto/yocto/yocto-4.0.4/meta-mingw-a90614a6498c3345704e9611f2842eb933dc51c1.tar.bz2
Repository Name: meta-gplv2 Repository Location: https://git.yoctoproject.org/git/meta-gplv2 Branch: kirkstone Tag: yocto-4.0.4 Git Revision: d2f8b5cdb285b72a4ed93450f6703ca27aa42e8a Release Artefact: meta-gplv2-d2f8b5cdb285b72a4ed93450f6703ca27aa42e8a sha: c386f59f8a672747dc3d0be1d4234b6039273d0e57933eb87caa20f56b9cca6d Download Locations: http://downloads.yoctoproject.org/releases/yocto/yocto-4.0.4/meta-gplv2-d2f8b5cdb285b72a4ed93450f6703ca27aa42e8a.tar.bz2 http://mirrors.kernel.org/yocto/yocto/yocto-4.0.4/meta-gplv2-d2f8b5cdb285b72a4ed93450f6703ca27aa42e8a.tar.bz2
Repository Name: bitbake Repository Location: https://git.openembedded.org/bitbake Branch: 2.0 Tag: yocto-4.0.4 Git Revision: ac576d6fad6bba0cfea931883f25264ea83747ca Release Artefact: bitbake-ac576d6fad6bba0cfea931883f25264ea83747ca sha: 526c2768874eeda61ade8c9ddb3113c90d36ef44a026d6690f02de6f3dd0ea12 Download Locations: http://downloads.yoctoproject.org/releases/yocto/yocto-4.0.4/bitbake-ac576d6fad6bba0cfea931883f25264ea83747ca.tar.bz2 http://mirrors.kernel.org/yocto/yocto/yocto-4.0.4/bitbake-ac576d6fad6bba0cfea931883f25264ea83747ca.tar.bz2
Repository Name: yocto-docs Repository Location: https://git.yoctoproject.org/git/yocto-docs Branch: kirkstone Tag: yocto-4.0.4 Git Revision: f632dad24c39778f948014029e74db3c871d9d21
- --------------- Contributors - --------------- Alejandro Hernandez Samaniego Alex Stewart Alexander Kanavin Alexandre Belloni Andrei Gherzan Anuj Mittal Aryaman Gupta Awais Belal Beniamin Sandu Bertrand Marquis Bruce Ashfield Changqing Li Chee Yang Lee Daiane Angolini Enrico Scholz Ernst Sjöstrand Gennaro Iorio Hitendra Prajapati Jacob Kroon Jon Mason Jose Quaresma Joshua Watt Kai Kang Khem Raj Kristian Amlie LUIS ENRIQUEZ Mark Hatle Martin Beeger Martin Jansa Mateusz Marciniec Michael Opdenacker Mihai Lindner Mikko Rapeli Ming Liu Ola x Nilsson Otavio Salvador Paul Eggleton Pavel Zhukov Peter Bergin Peter Kjellerstedt Peter Marko Rajesh Dangi Randy MacLeod Rasmus Villemoes Richard Purdie Robert Joslyn Roland Hieber Ross Burton Sakib Sajal Shubham Kulkarni Steve Sakoman Ulrich Ölmann Yang Xu Yongxin Liu ghassaneben niko.mauno@... pgowda wangmy
- --------------- Known Issues - --------------- N/A
- --------------- Security Fixes - --------------- binutils : fix CVE-2022-38533 curl: fix CVE-2022-35252 sqlite: fix CVE-2022-35737 grub2: fix CVE-2021-3695 CVE-2021-3696 CVE-2021-3697 CVE-2022-28733 CVE-2022-28734 CVE-2022-28735 u-boot: fix CVE-2022-30552 CVE-2022-33967 libxml2: Ignore CVE-2016-3709 libtiff: fix CVE-2022-34526 zlib: fix CVE-2022-37434 gnutls: fix CVE-2022-2509 u-boot: fix CVE-2022-33103 qemu: fix CVE-2021-3507 CVE-2021-3929 CVE-2021-4158 CVE-2022-0216 CVE-2022-0358
- --------------- Fixes - --------------- apr: Cache configure tests which use AC_TRY_RUN apr: Use correct strerror_r implementation based on libc type apt: fix nativesdk-apt build failure during the second time build archiver.bbclass: remove unsed do_deploy_archives[dirs] archiver.bbclass: some recipes that uses the kernelsrc bbclass uses the shared source autoconf: Fix strict prototype errors in generated tests autoconf: Update K & R stype functions bind: upgrade to 9.18.5 bitbake.conf: set BB_DEFAULT_UMASK using ??= bitbake: ConfHandler/BBHandler: Improve comment error messages and add tests bitbake: ConfHandler: Remove lingering close bitbake: bb/utils: movefile: use the logger for printing bitbake: bb/utils: remove: check the path again the expand python glob bitbake: bitbake-user-manual: Correct description of the ??= operator bitbake: bitbake-user-manual: npm fetcher: improve description of SRC_URI format bitbake: bitbake: bitbake-user-manual: hashserv can be accessed on a dedicated domain bitbake: bitbake: runqueue: add cpu/io pressure regulation bitbake: bitbake: runqueue: add memory pressure regulation bitbake: cooker: Drop sre_constants usage bitbake: doc: bitbake-user-manual: add explicit target for crates fetcher bitbake: doc: bitbake-user-manual: document npm and npmsw fetchers bitbake: event.py: ignore exceptions from stdout and sterr operations in atexit bitbake: fetch2: Ensure directory exists before creating symlink bitbake: fetch2: gitsm: fix incorrect handling of git submodule relative urls bitbake: runqueue: Change pressure file warning to a note bitbake: runqueue: Fix unihash cache mismatch issues bitbake: toaster: fix kirkstone version bitbake: utils: Pass lock argument in fileslocked bluez5: upgrade to 5.65 boost: fix install of fiber shared libraries cairo: Adapt the license information based on what is being built classes: cve-check: Get shared database lock cmake: remove CMAKE_ASM_FLAGS variable in toolchain file connman: Backports for security fixes core-image.bbclass: Exclude openssh complementary packages cracklib: Drop using register keyword cracklib: upgrade to 2.9.8 create-spdx: Fix supplier field create-spdx: handle links to inaccessible locations create-spdx: ignore packing control files from ipk and deb cve-check: Don't use f-strings cve-check: close cursors as soon as possible devtool/upgrade: catch bb.fetch2.decodeurl errors devtool/upgrade: correctly clean up when recipe filename isn't yet known devtool: error out when workspace is using old override syntax ell: upgrade to 0.50 epiphany: upgrade to 42.4 externalsrc: Don't wipe out src dir when EXPORT_FUNCTIONS is used. gcc-multilib-config: Fix i686 toolchain relocation issues gcr: Define _GNU_SOURCE gdk-pixbuf: upgrade to 2.42.9 glib-networking: upgrade to 2.72.2 go: upgrade to v1.17.13 insane.bbclass: Skip patches not in oe-core by full path iso-codes: upgrade to 4.11.0 kernel-fitimage.bbclass: add padding algorithm property in config nodes kernel-fitimage.bbclass: only package unique DTBs kernel: Always set CC and LD for the kernel build kernel: Use consistent make flags for menuconfig lib:npm_registry: initial checkin libatomic-ops: upgrade to 7.6.14 libcap: upgrade to 2.65 libjpeg-turbo: upgrade to 2.1.4 libpam: use /run instead of /var/run in systemd tmpfiles libtasn1: upgrade to 4.19.0 liburcu: upgrade to 0.13.2 libwebp: upgrade to 1.2.4 libwpe: upgrade to 1.12.3 libxml2: Port gentest.py to Python-3 lighttpd: upgrade to 1.4.66 linux-yocto/5.10: update genericx86* machines to v5.10.135 linux-yocto/5.10: update to v5.10.137 linux-yocto/5.15: update genericx86* machines to v5.15.59 linux-yocto/5.15: update to v5.15.62 linux-yocto: Fix COMPATIBLE_MACHINE regex match linux-yocto: prepend the the value with a space when append to KERNEL_EXTRA_ARGS lttng-modules: fix 5.19+ build lttng-modules: fix build against mips and v5.19 kernel lttng-modules: fix build for kernel 5.10.137 lttng-modules: replace mips compaction fix with upstream change lz4: upgrade to 1.9.4 maintainers: update opkg maintainer meta: introduce UBOOT_MKIMAGE_KERNEL_TYPE migration guides: add missing release notes mobile-broadband-provider-info: upgrade to 20220725 nativesdk: Clear TUNE_FEATURES npm: replace 'npm pack' call by 'tar czf' npm: return content of 'package.json' in 'npm_pack' npm: take 'version' directly from 'package.json' npm: use npm_registry to cache package oeqa/gotoolchain: put writable files in the Go module cache oeqa/gotoolchain: set CGO_ENABLED=1 oeqa/parselogs: add qemuarmv5 arm-charlcd masking oeqa/qemurunner: add run_serial() comment oeqa/selftest: rename git.py to intercept.py oeqa: qemurunner: Report UNIX Epoch timestamp on login package_rpm: Do not replace square brackets in %files packagegroup-self-hosted: update for strace parselogs: Ignore xf86OpenConsole error perf: Fix reproducibility issues with 5.19 onwards pinentry: enable _XOPEN_SOURCE on musl for wchar usage in curses poky.conf: add ubuntu-22.04 to tested distros poky.conf: bump version for 4.0.4 pseudo: Update to include recent upstream minor fixes python3-pip: Fix RDEPENDS after the update ref-manual: add numa to machine features relocate_sdk.py: ensure interpreter size error causes relocation to fail rootfs-postcommands.bbclass: avoid moving ssh host keys if etc is writable rootfs.py: dont try to list installed packages for baremetal images rootfspostcommands.py: Cleanup subid backup files generated by shadow-utils ruby: drop capstone support runqemu: Add missing space on default display option runqemu: display host uptime when starting sanity: add a comment to ensure CONNECTIVITY_CHECK_URIS is correct scripts/oe-setup-builddir: make it known where configurations come from scripts/runqemu.README: fix typos and trailing whitespaces selftest/wic: Tweak test case to not depend on kernel size shadow: Avoid nss warning/error with musl shadow: Enable subid support system-requirements.rst: Add Ubuntu 22.04 to list of supported distros systemd: Add 'no-dns-fallback' PACKAGECONFIG option systemd: Fix unwritable /var/lock when no sysvinit handling sysvinit-inittab/start_getty: Fix respawn too fast tcp-wrappers: Fix implicit-function-declaration warnings tzdata: upgrade to 2022b util-linux: Remove --enable-raw from EXTRA_OECONF vala: upgrade to 0.56.3 vim: Upgrade to 9.0.0453 watchdog: Include needed system header for function decls webkitgtk: upgrade to 2.36.5 weston: upgrade to 10.0.2 wic/bootimg-efi: use cross objcopy when building unified kernel image wic: add target tools to PATH when executing native commands wic: depend on cross-binutils wireless-regdb: upgrade to 2022.08.12 wpebackend-fdo: upgrade to 1.12.1 xinetd: Pass missing -D_GNU_SOURCE xz: update to 5.2.6* |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
M+ & H bugs with Milestone Movements WW39
Stephen Jolley
All,
Thanks,
Stephen K. Jolley Yocto Project Program Manager ( Cell: (208) 244-4460 * Email: sjolley.yp.pm@...
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Enhancements/Bugs closed WW39!
Stephen Jolley
All,
Thanks,
Stephen K. Jolley Yocto Project Program Manager ( Cell: (208) 244-4460 * Email: sjolley.yp.pm@...
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Current high bug count owners for Yocto Project 4.1
Stephen Jolley
All,
Thanks,
Stephen K. Jolley Yocto Project Program Manager ( Cell: (208) 244-4460 * Email: sjolley.yp.pm@...
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yocto Project Newcomer & Unassigned Bugs - Help Needed
Stephen Jolley
All,
The triage team is starting to try and collect up and classify bugs which a newcomer to the project would be able to work on in a way which means people can find them. They're being listed on the triage page under the appropriate heading: https://wiki.yoctoproject.org/wiki/Bug_Triage#Newcomer_Bugs Also please review: https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded and how to create a bugzilla account at: https://bugzilla.yoctoproject.org/createaccount.cgi The idea is these bugs should be straight forward for a person to help work on who doesn't have deep experience with the project. If anyone can help, please take ownership of the bug and send patches! If anyone needs help/advice there are people on irc who can likely do so, or some of the more experienced contributors will likely be happy to help too.
Also, the triage team meets weekly and does its best to handle the bugs reported into the Bugzilla. The number of people attending that meeting has fallen, as have the number of people available to help fix bugs. One of the things we hear users report is they don't know how to help. We (the triage team) are therefore going to start reporting out the currently 408 unassigned or newcomer bugs.
We're hoping people may be able to spare some time now and again to help out with these. Bugs are split into two types, "true bugs" where things don't work as they should and "enhancements" which are features we'd want to add to the system. There are also roughly four different "priority" classes right now, “4.1”, “4.2”, “4.3”, "4.99" and "Future", the more pressing/urgent issues being in "4.1" and then “4.2”.
Please review this link and if a bug is something you would be able to help with either take ownership of the bug, or send me (sjolley.yp.pm@...) an e-mail with the bug number you would like and I will assign it to you (please make sure you have a Bugzilla account). The list is at: https://wiki.yoctoproject.org/wiki/Bug_Triage_Archive#Unassigned_or_Newcomer_Bugs
Thanks,
Stephen K. Jolley Yocto Project Program Manager ( Cell: (208) 244-4460 * Email: sjolley.yp.pm@...
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Re: nodejs do_compile eats all resources
On 9/26/22 10:20 AM, Oliver Westermann via lists.yoctoproject.org wrote:
Hey,how many cores do you have on the dual Epyc sockets. for Mmeory., usually 4GB/per-core is a safe mark for worst case scenario. nodejs build does run some target binaries using qemu-usermode eg. mksnapshot during build, which in a threaded mode will put more pressure on memory. I only observed this on the nodejs build, is there a good approach to debug this (or even a known issue)? |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
[meta-zephyr] new qemu version breaking qemu-cortex-a53
Jon Mason
The recently updated version of qemu (v7.1) is causing qemu-cortex-a53
tests to fail. See https://gitlab.com/jonmason00/meta-zephyr/-/jobs/3082086509 I was able to bisect the issue to that commit. I'm not certain of the cause beyond that, but other issues are being seen with the new qemu and meta-arm. Let me know if you want me to open a bug. Thanks, Jon |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
[meta-zephyr][PATCH] ci: add entry for nrf52840-mdk-usb-dongle
Jon Mason
Signed-off-by: Jon Mason <jon.mason@...>
--- .gitlab-ci.yml | 3 +++ ci/nrf52840-mdk-usb-dongle.yml | 6 ++++++ 2 files changed, 9 insertions(+) create mode 100644 ci/nrf52840-mdk-usb-dongle.yml diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 082cd44..c185477 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -74,6 +74,9 @@ arduino-nano-33-ble: intel-x86-64: extends: .build +nrf52840-mdk-usb-dongle: + extends: .build + nrf52840dk-nrf52840: extends: .build diff --git a/ci/nrf52840-mdk-usb-dongle.yml b/ci/nrf52840-mdk-usb-dongle.yml new file mode 100644 index 0000000..0dc3433 --- /dev/null +++ b/ci/nrf52840-mdk-usb-dongle.yml @@ -0,0 +1,6 @@ +header: + version: 11 + includes: + - ci/base.yml + +machine: nrf52840-mdk-usb-dongle -- 2.17.1 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Re: [meta-zephyr][PATCHv2] nrf52840-mdk-usb-dongle.conf: Add new machine from makerdiary
Jon Mason
On Fri, Sep 23, 2022 at 12:45:09PM +0200, philippe.coval@... wrote:
From: Philippe Coval <philippe.coval@...>This needs a Gitlab CI entry to test. I did a quick one and ran CI on it here: https://gitlab.com/jonmason00/meta-zephyr/-/pipelines/651094923 I'll email the patch to add CI shortly. Thanks, Jon --- |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Re: meta-swupdate integration with the custom Yocto image
#dunfell
Mahendra Sondagar
Hi... Quentin Thanks for the reply...Regards Mahendra On Mon, Sep 26, 2022 at 1:44 PM Quentin Schulz <quentin.schulz@...> wrote: Hi Mahendra Sondagar, |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Re: nodejs do_compile eats all resources
Alexander Kanavin
Anything written in C++ tends to consume 1-2 Gb of ram per compiler
toggle quoted message
Show quoted text
process. If that lands you in OOM, you probably should limit that with PARALLEL_MAKE:pn-nodejs, but otherwise that is the sad reality of C++ builds. Alex On Mon, 26 Sept 2022 at 19:20, Oliver Westermann via lists.yoctoproject.org <oliver.westermann=cognex.com@...> wrote:
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
nodejs do_compile eats all resources
Oliver Westermann <oliver.westermann@...>
Hey,
We recently added nodejs to our images and noticed that it's do compile process is a real memory hog. Since we planned to update the recipe anyway we didn't pay much attention, but today we updated the recipe to the current version from openembedded-core (16.14.2). Still I see that it consumes nearly all resources it can get, way more than any other package in our build process (and crashing some VMs even). On my build machine (dual Epyc, 256GB of RAM) it manages to produce nearly 100% CPU load AND eats ~90GB of RAM. Some other machines fail the build due to OOM issues. I only observed this on the nodejs build, is there a good approach to debug this (or even a known issue)? Best regards, Olli |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Setting up layers in Yocto, the introduction
Alexander Kanavin
TL;DR version
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D If you want to get everything needed to run a yocto build: - clone the bootstrap repository (ask your distribution maintainer where = it is): $ git clone bootstrap_repo_uri - run 'setup-layers' from that repo: $ /path/to/bootstrap_repo/setup-layers If you want to save the layers setup for replication elsewhere: - save the config file and the replication script into a 'bootstrap repos= itory') (this can be an organization-specific yocto layer, or a standalone reposi= tory): $ bitbake-layers create-layers-setup /path/to/bootstrap_repo - document where the bootstrap repository can be found for your users What is this layer setup tooling? (the long version) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Once you have a working build with the correct set of layers, it is benef= icial to capture the layer setup --- what they are, which repositories they com= e from and which SCM revisions they're at --- into a configuration file, so that= this setup can be easily replicated later, perhaps on a different machine. Her= e's how to do this:: $ bitbake-layers create-layers-setup /srv/work/alex/meta-alex/ NOTE: Starting bitbake server... NOTE: Created /srv/work/alex/meta-alex/setup-layers.json NOTE: Created /srv/work/alex/meta-alex/setup-layers The tool needs a single argument which tells where to place the output, c= onsisting of a json formatted layer configuration, and a ``setup-layers`` script th= at can use that configuration to restore the layers in a different location, or on a different host mac= hine. The argument can point to a custom layer (which is then deemed a "bootstrap" layer tha= t needs to be checked out first), or into a completely independent location. The replication of the layers is performed by running the ``setup-layers`= ` script provided above: 1. Clone the bootstrap layer or some other repository to obtain the json config and the setup script that can use it. 2. Run the script directly with no options:: alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers Note: not checking out source meta-alex, use --force-bootstraplayer= -checkout to override. Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96= edae, branch master Running 'git init -q /srv/work/alex/my-build/meta-intel' Running 'git remote remove origin > /dev/null 2>&1; git remote add = origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/= meta-intel Running 'git fetch -q origin || true' in /srv/work/alex/my-build/me= ta-intel Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' = in /srv/work/alex/my-build/meta-intel Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch aka= navin/setup-layers Running 'git init -q /srv/work/alex/my-build/poky' Running 'git remote remove origin > /dev/null 2>&1; git remote add = origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky Running 'git fetch -q origin || true' in /srv/work/alex/my-build/po= ky Running 'git remote remove poky-contrib > /dev/null 2>&1; git remot= e add poky-contrib ssh://git@.../poky-contrib' in /srv/= work/alex/my-build/poky Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-bu= ild/poky Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' = in /srv/work/alex/my-build/poky .. note:: This will work to update an existing checkout as well. .. note:: The script is self-sufficient and requires only python3 and git on the build machine. .. note:: Both the ``create-layers-setup`` and the ``setup-layers`` provided sev= eral additional options that customize their behavior - you are welcome to study them via ``--= help`` command line parameter. Questions and answers =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 1. Why JSON, and not YAML? JSON is a part of the python standard library, while YAML is not. This me= ans you can bootstrap a build on just about any machine that has python and its core library, without first having to ins= tall any dependencies. JSON not quite as easy on the eye, but with decent indentation it's totally ok. Before anyone asks, XML is appallingly difficult to read by modern standa= rds of readability. No thanks. 2. I don't want the script in the bootstrap repo, just the config. I will= provide my own tools. How? Use --json-only to generate only the config, and not the script: $ bitbake-layers create-layers-setup -json-only /srv/work/alex/meta-a= lex/ 3. I want to generate the compatible json with my own tools. Is there a s= chema to validate it? There is: https://git.yoctoproject.org/poky/tree/meta/files/layers.schema.json You can use python3-jsonschema package to check the validity. 4. I don't want the json or the script, and would want to use an entirely= different format and tools. Is 'create-layers-setup' hardcoded to the OE json format? Actually not! Other formats can be added through a plugin mechanism. OE j= son writer is just the default plugin but others can be implemented and selected: --writer {oe-setup-layers}, -w {oe-setup-layers} Choose the output format (defaults to oe-setup-la= yers). 5. I want to use something else than hardcoded revisions in the json (e.g= . refer to branch names or tags). No problem! Edit the json to say 'main' or 'release-x.y'. Anything that '= git checkout' can take will work. Just be careful that these modification are not overwritten later b= y 'create-layers-setup' by e.g. renaming the json file first. 6. I already have an existing checkout and want to update it to the lates= t greatest. How? First, update the bootstrap repository to the latest (or otherwise desire= d) revision to obtain the latest versions of the script, and the json config. Ask the di= stro maintainers for the recommended practice to do so. Then, run the script again - it should be able to handle a layers checkou= t that already exists, and update everything accoriding to the latest json, obtained in = the first step. 7. Ok, I have checked out all the layers. Now what? The next step is to actually set up a yocto build from an existing config= uration template, and start a bitbake session. This is made separate from handling the laye= r repositories and will be described separately. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Re: meta-spdxscanner
Martin Leduc <martin.leduc@...>
Hi Lei,
Yes, the issue is fixed by doing the actions defined into https://stackoverflow.com/questions/69858963/how-can-one-fully-replace-distutils-which-is-deprecated-in-3-10 :
Replace from distutils.core import setup with from setuptools import setup
into ../tmp/work/x86_64-linux/python3-patch-native/1.16-r0/patch-1.16/setup.py
The output is not really useful at the moment because it crash after hours of computing for the kernel and some other packages.
Actually, I’ve made a workaround to my problems by making a PHP script parsing the Yocto SPDX output but it’s not enough. INHERIT += "create-spdx"
My idea is: I do not use the scancode-tk as expected.
I’ve also built one Fossology server but it run hours and hours and hours without finishing the produce the report and the reports are not, for my needs, complete.
If you have any input else than it’s defined in the README, it will be really interesting.
Thank you for your help.
De : yocto@... <yocto@...>
De la part de leimaohui
Hi,
I’m sorry for late reply.
Is this issue resolved? If not, can you tell me which class you are using? I can’t get enough information from your log. It will be better if you supply the setting about enable meta-spdxscanner in local.conf.
Best regards Lei
Hi Community,
I’m facing to an issue with meta-spdxscanner.bbclass. A Python Warning fail the build. This is the error:
ERROR: python3-patch-native-1.16-r0 do_compile: 'python3 setup.py bdist_wheel ' execution failed. ERROR: python3-patch-native-1.16-r0 do_compile: ExecutionError('/home/mleduc/1087/1087-NIM/build-sdk/tmp/work/x86_64-linux/python3-patch-native/1.16-r0/temp/run.do_compile.3058985', 1, None, None) ERROR: Logfile of failure stored in: /home/mleduc/1087/1087-NIM/build-sdk/tmp/work/x86_64-linux/python3-patch-native/1.16-r0/temp/log.do_compile.3058985 Log data follows: | DEBUG: Executing shell function do_compile | /home/mleduc/1087/1087-NIM/build-sdk/tmp/work/x86_64-linux/python3-patch-native/1.16-r0/patch-1.16/setup.py:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives | from distutils.core import setup | usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] | or: setup.py --help [cmd1 cmd2 ...] | or: setup.py --help-commands | or: setup.py cmd --help | | error: invalid command 'bdist_wheel' | ERROR: 'python3 setup.py bdist_wheel ' execution failed. | WARNING: /home/mleduc/1087/1087-NIM/build-sdk/tmp/work/x86_64-linux/python3-patch-native/1.16-r0/temp/run.do_compile.3058985:177 exit 1 from 'exit 1' | WARNING: Backtrace (BB generated script): | #1: bbfatal_log, /home/mleduc/1087/1087-NIM/build-sdk/tmp/work/x86_64-linux/python3-patch-native/1.16-r0/temp/run.do_compile.3058985, line 177 | #2: setuptools3_do_compile, /home/mleduc/1087/1087-NIM/build-sdk/tmp/work/x86_64-linux/python3-patch-native/1.16-r0/temp/run.do_compile.3058985, line 167 | #3: do_compile, /home/mleduc/1087/1087-NIM/build-sdk/tmp/work/x86_64-linux/python3-patch-native/1.16-r0/temp/run.do_compile.3058985, line 156 | #4: main, /home/mleduc/1087/1087-NIM/build-sdk/tmp/work/x86_64-linux/python3-patch-native/1.16-r0/temp/run.do_compile.3058985, line 181 ERROR: Task (virtual:native:/home/mleduc/1087/1087-NIM/sources/meta-spdxscanner/recipes-devtools/python/python3-patch_1.16.bb:do_compile) failed with exit code '1'
Any patch or approach to fix this?
I’m using Yocto Kirkstone
Any help will be appreciated
Martin Leduc Superviseur logiciel platforme, Rail Luminator Technology Group T : 1 418 856-6896
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Re: [qa-build-notification] QA notification for completed autobuilder build (yocto-4.0.4.rc1)
Richard Purdie
On Mon, 2022-09-26 at 09:22 +0000, Teoh, Jay Shen wrote:
Hi All,Thanks! Since this is a point release with a clean build and clean QA report, we can move straight to releasing it and don't need to obtain approval from the whole TSC. Cheers, Richard |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Re: [Openembedded-architecture] [yocto-announce] [ANNOUNCEMENT] Milestone 3 for Yocto Project 4.1 (yocto-4.1_M3) Now Available
Konrad Weihmann <kweihmann@...>
On 25.09.22 11:41, Richard Purdie wrote:
On Sun, 2022-09-25 at 10:53 +0200, Konrad Weihmann wrote:Bug filed: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14916Hi all,Someone needs to do the work involved and I guess that hasn't happened. Not sure if I put it in the right category, so feel free to move it around if needed I still don't know who will do it but that way we at least remember we |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Re: [qa-build-notification] QA notification for completed autobuilder build (yocto-4.0.4.rc1)
Teoh, Jay Shen
Hi All,
toggle quoted message
Show quoted text
QA for yocto-4.0.4.rc1 is completed. This is the full report for this release: https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/tree/?h=intel-yocto-testresults ======= Summary ======== No high milestone defects. No new issue found. Thanks, Jay -----Original Message----- |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|