Hi,

On 10-01-2023 02:57, Randy MacLeod wrote:

Add Zheng.
Switch to Trevor's gmail since he might still be interested in this.

On 2023-01-09 05:49, Ferry Toth via lists.yoctoproject.org wrote:

Hi

On 09-01-2023 11:39, Alexander Kanavin wrote:

On Sun, 8 Jan 2023 at 23:13, Ferry Toth <fntoth@...> wrote:

Now it works I'm not sure what to do. Richard marked the original patch
as WIP. Maybe it's not appropriate for including into poky? If so I
wouldn't mind carrying it in meta-intel-edison.

Or may be with a little work (from me) and a run on the CI servers we
could make it go in?

I'd rather teach make/ninja upstream to watch memory consumption
and/or host pressure directly (e.g. similar to how it handles -l). And
offer any resulting patches to upstream first.

Zheng and I *started* on that for make over the Xmas holiday.
See the (poorly formatted) thread:
  Add support for limiting CPU pressure, contrib, 2022/12/20
in:
https://lists.gnu.org/archive/html/bug-make/2022-12/threads.html

There were mixed reviews upstream with the maintainer, Paul Smith,
seeming to prefer that we investigate the job server approach and the current
load averaging. I'll happily try to find time to play with Ferry's job server work.

The work is actually Richard's. I only fixed what broke over time.

I've been thinking about the various workflows and as Richard said, it seems
that for many people who only do one build at a time, the job server and maybe
a little linker restraint, would be all that's needed. For activities such
as YP AB, we could teach the main job server to also look at /proc/pressure
as a way to limit the pool size we could make a meta-jobserver for those who don't
want/need to worry about non-compile tasks such as tests and build clean-up.


Note that cargo has the notion of a job server:
   https://github.com/rust-lang/cargo/issues/1744
   https://github.com/rust-lang/cargo/pull/4110


and ninja has an open issue:
   https://github.com/ninja-build/ninja/issues/1139
and an initial implementation:
   https://github.com/stefanb2/ninja/tree/topic-jobserver-fifo

What other build tools are in need of regulation and/or job server patches?

What I read, gcc has already -flto=jobserver.

Other (single threaded CPU intensive) might just be started from jobclient?

../Randy

Alex

Yeah, I'd rather teach the kernel to consider thrashing when scheduling jobs. As is now any user process can slow down any other users process and even the kernel itself to a standstill.

But kernel developers consider those issues "orthogonal" (i.e. they don't want to make the scheduler aware of io).

Add Zheng.
Switch to Trevor's gmail since he might still be interested in this.

On 2023-01-09 05:49, Ferry Toth via lists.yoctoproject.org wrote:

Hi
On 09-01-2023 11:39, Alexander Kanavin wrote:

On Sun, 8 Jan 2023 at 23:13, Ferry Toth <fntoth@...> wrote:

Now it works I'm not sure what to do. Richard marked the original patch
as WIP. Maybe it's not appropriate for including into poky? If so I
wouldn't mind carrying it in meta-intel-edison.

Or may be with a little work (from me) and a run on the CI servers we
could make it go in?

I'd rather teach make/ninja upstream to watch memory consumption
and/or host pressure directly (e.g. similar to how it handles -l). And
offer any resulting patches to upstream first.

Zheng and I *started* on that for make over the Xmas holiday.
See the (poorly formatted) thread:
Add support for limiting CPU pressure, contrib, 2022/12/20
in:
https://lists.gnu.org/archive/html/bug-make/2022-12/threads.html

There were mixed reviews upstream with the maintainer, Paul Smith,
seeming to prefer that we investigate the job server approach and the current
load averaging. I'll happily try to find time to play with Ferry's job server work.

I've been thinking about the various workflows and as Richard said, it seems
that for many people who only do one build at a time, the job server and maybe
a little linker restraint, would be all that's needed. For activities such
as YP AB, we could teach the main job server to also look at /proc/pressure
as a way to limit the pool size we could make a meta-jobserver for those who don't
want/need to worry about non-compile tasks such as tests and build clean-up.


Note that cargo has the notion of a job server:
https://github.com/rust-lang/cargo/issues/1744
https://github.com/rust-lang/cargo/pull/4110


and ninja has an open issue:
https://github.com/ninja-build/ninja/issues/1139
and an initial implementation:
https://github.com/stefanb2/ninja/tree/topic-jobserver-fifo

What other build tools are in need of regulation and/or job server patches?

../Randy

Alex

Yeah, I'd rather teach the kernel to consider thrashing when scheduling jobs. As is now any user process can slow down any other users process and even the kernel itself to a standstill.
But kernel developers consider those issues "orthogonal" (i.e. they don't want to make the scheduler aware of io).

--
# Randy MacLeod
# Wind River Linux

On 09/01/2023 19:30:36+0100, Alexander Kanavin wrote:

On Mon, 9 Jan 2023 at 19:23, Alexandre Belloni
<alexandre.belloni@...> wrote:

No, please send patches against the correct git tree, else you are
putting more load on the maintainers than necessary.
This was the cause of:
https://lore.kernel.org/all/Y5Sr9HSWIc5LVzl9@mail.local/

This can silently fail and then you need to wait for 7hours for the AB
to fail and this delays testing for one more day.

We never got to the bottom of what happened to that one. Part of the
patch applied correctly, the other part did not.

Just try to apply your patch on a bitbake repository, it will reproduce.


--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

On Mon, 9 Jan 2023 at 19:23, Alexandre Belloni
<alexandre.belloni@...> wrote:

No, please send patches against the correct git tree, else you are
putting more load on the maintainers than necessary.
This was the cause of:
https://lore.kernel.org/all/Y5Sr9HSWIc5LVzl9@mail.local/

This can silently fail and then you need to wait for 7hours for the AB
to fail and this delays testing for one more day.

We never got to the bottom of what happened to that one. Part of the
patch applied correctly, the other part did not.

Alex

On 09/01/2023 10:29:13+0100, Alexander Kanavin wrote:

I've been making patches against an integrated poky checkout for many
years, including bitbake patches. They're usually correctly picked up
and applied; you only need to ensure you route them to correct mailing
lists.

No, please send patches against the correct git tree, else you are
putting more load on the maintainers than necessary.
This was the cause of:
https://lore.kernel.org/all/Y5Sr9HSWIc5LVzl9@mail.local/

This can silently fail and then you need to wait for 7hours for the AB
to fail and this delays testing for one more day.

Alex

On Sat, 7 Jan 2023 at 14:05, Yoann Congal <yoann.congal@...> wrote:

Hi,

Since I plan to work regularly on upstream Yocto and contributing, I
wonder about the best way to best setup my git repos to do that...

I've started with the upstream poky repo but when I make a commit on,
for example, bitbake, my patch applies on a bitbake/some-path file which
does not exist in the bitbake repo (only "some-path" exists in the
bitbake repo).

Is this something I should address ? I found some patches for bitbake in
the mailing list that are for "bitbake/..." paths... So, maybe, basing
my patches on the poky repo is totally OK for now...?

To the Yocto maintainers, how do you manage this (poky repo, combo-layer
and individual bitbake/doc/meta-yocto repo) ?

Note: I have found the "combo-layer splitpatch" tool but it does not
look like a full solution... Maybe I'm missing something?

Thanks!
--
Yoann Congal ("yocton" on IRC)
Smile ECS - Tech Expert

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Hi all,

Intel and WR YP QA is planning for QA execution for YP build yocto-4.1.2.rc1. We are planning to execute following tests for this cycle:

OEQA-manual tests for following module:
1. OE-Core
2. BSP-hw

Runtime auto test for following platforms:
1. MinnowTurbot 32-bit
2. NUC 7
3. ADL
4. TGL NUC 11
5. Edgerouter
6. Beaglebone

ETA for completion Thursday, 12th Jan 2023.

Best regards,
Jing Hui

On 1/9/23 10:29, Alexander Kanavin wrote:

I've been making patches against an integrated poky checkout for many
years, including bitbake patches. They're usually correctly picked up
and applied; you only need to ensure you route them to correct mailing
lists.

Ok, I'll keep it that way.

Thanks!

Alex
On Sat, 7 Jan 2023 at 14:05, Yoann Congal <yoann.congal@...> wrote:

Hi,

Since I plan to work regularly on upstream Yocto and contributing, I
wonder about the best way to best setup my git repos to do that...

I've started with the upstream poky repo but when I make a commit on,
for example, bitbake, my patch applies on a bitbake/some-path file which
does not exist in the bitbake repo (only "some-path" exists in the
bitbake repo).

Is this something I should address ? I found some patches for bitbake in
the mailing list that are for "bitbake/..." paths... So, maybe, basing
my patches on the poky repo is totally OK for now...?

To the Yocto maintainers, how do you manage this (poky repo, combo-layer
and individual bitbake/doc/meta-yocto repo) ?

Note: I have found the "combo-layer splitpatch" tool but it does not
look like a full solution... Maybe I'm missing something?

Thanks!
--
Yoann Congal ("yocton" on IRC)
Smile ECS - Tech Expert

--
Yoann Congal
Smile ECS - Tech Expert

Hi

On Mon, 2023-01-09 at 11:39 +0100, Alexander Kanavin wrote:

On Sun, 8 Jan 2023 at 23:13, Ferry Toth <fntoth@...> wrote:

Now it works I'm not sure what to do. Richard marked the original patch
as WIP. Maybe it's not appropriate for including into poky? If so I
wouldn't mind carrying it in meta-intel-edison.

Or may be with a little work (from me) and a run on the CI servers we
could make it go in?

I'd rather teach make/ninja upstream to watch memory consumption
and/or host pressure directly (e.g. similar to how it handles -l). And
offer any resulting patches to upstream first.

Whilst teaching make/ninja about pressure is probably a nice idea, I
think there is a use case for sharing the job pool between bitbake
tasks rather than having a job pool per task since that scales badly.

One issue that comes to mind with the patch is that it is currently
writing into /tmp/ and we likely need to work out a better location for
the job server fifo. Currently that would break things on the
autobuilder as there are multiple builds on a given host.

Ha, I didn't consider multiple bitbakes running simultaneously.

Maybe the location /tmp is fine but we would need a unique (random?) filename instead of makefifo?

Cheers,

Richard

Hi,

I'm moving my layers from dunfell to kirkstone and for some recipes
I'm hitting an issue with following fetch issue:
Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch
without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use
SRCPV in PV for OE).", None)

My SRC_URI looks like this:

SRC_URI = "git://mysupersecretgitproject.git;protocol=https;branch=master;tag=v${PV}"
and PV is properly set to tag name.

By adding this to beginning of recipe it seems it start working:
python() {
bb.warn("SRCPV:%s"%d.getVar("SRCPV"))
}

Any ideas what am I missing?

Thanks and regards,

marek

--
as simple and primitive as possible
-------------------------------------------------
Marek Belisko - OPEN-NANDRA
Freelance Developer

Ruska Nova Ves 219 | Presov, 08005 Slovak Republic
Tel: +421 915 052 184
skype: marekwhite
twitter: #opennandra
web: http://open-nandra.com

Hi

On 09-01-2023 11:39, Alexander Kanavin wrote:

On Sun, 8 Jan 2023 at 23:13, Ferry Toth <fntoth@...> wrote:

Now it works I'm not sure what to do. Richard marked the original patch
as WIP. Maybe it's not appropriate for including into poky? If so I
wouldn't mind carrying it in meta-intel-edison.

Or may be with a little work (from me) and a run on the CI servers we
could make it go in?

I'd rather teach make/ninja upstream to watch memory consumption
and/or host pressure directly (e.g. similar to how it handles -l). And
offer any resulting patches to upstream first.

Alex

Yeah, I'd rather teach the kernel to consider thrashing when scheduling jobs. As is now any user process can slow down any other users process and even the kernel itself to a standstill.

But kernel developers consider those issues "orthogonal" (i.e. they don't want to make the scheduler aware of io).

On Mon, 2023-01-09 at 11:39 +0100, Alexander Kanavin wrote:

On Sun, 8 Jan 2023 at 23:13, Ferry Toth <fntoth@...> wrote:

Now it works I'm not sure what to do. Richard marked the original patch
as WIP. Maybe it's not appropriate for including into poky? If so I
wouldn't mind carrying it in meta-intel-edison.

Or may be with a little work (from me) and a run on the CI servers we
could make it go in?

I'd rather teach make/ninja upstream to watch memory consumption
and/or host pressure directly (e.g. similar to how it handles -l). And
offer any resulting patches to upstream first.

Whilst teaching make/ninja about pressure is probably a nice idea, I
think there is a use case for sharing the job pool between bitbake
tasks rather than having a job pool per task since that scales badly.

One issue that comes to mind with the patch is that it is currently
writing into /tmp/ and we likely need to work out a better location for
the job server fifo. Currently that would break things on the
autobuilder as there are multiple builds on a given host.

Cheers,

Richard

On Sun, 8 Jan 2023 at 23:13, Ferry Toth <fntoth@...> wrote:

Now it works I'm not sure what to do. Richard marked the original patch
as WIP. Maybe it's not appropriate for including into poky? If so I
wouldn't mind carrying it in meta-intel-edison.

Or may be with a little work (from me) and a run on the CI servers we
could make it go in?

I'd rather teach make/ninja upstream to watch memory consumption
and/or host pressure directly (e.g. similar to how it handles -l). And
offer any resulting patches to upstream first.

Alex

I've been making patches against an integrated poky checkout for many
years, including bitbake patches. They're usually correctly picked up
and applied; you only need to ensure you route them to correct mailing
lists.

Alex

There is; the source code for wic and its tests are fully available to
you, so you are welcome to implement the missing bits and submit them
for review and inclusion.

Also, we do not have a wic maintainer.

Alex

On Sat, 7 Jan 2023 at 00:29, Benjamin Mordaunt
<crawford.benjamin15@...> wrote:

Hi,

I am trying to configure an image to conform to the VHD size alignment
requirements of Azure. Azure requires that the filesize of a VHD be a
multiple of 1 MiB. The '--align' option works to align the _start_ of a
partition to a particular multiple, but I need the _end_ to be aligned.
As a workaround, I tried:

part /fake --fixed-size=0 --no-table --align=1024

I figured this would create a zero-sized partition which would be
appropriately aligned at the end of the image, but wic complains that
zero-sized partitions are not allowed.

Is there a way to achieve this, or extend the capabilities of wic to
handle this use case?

Best,
Ben

It helps if you show your build configuration, e.g. the output of
'bitbake -e core-image-full-cmdline'.

Alex

After bitbaking core-image-full-cmdline, my root file system is readable only. How to make it writable? Is there something I have to change in local.conf?

Hi,

Just in case you are interested in the result.

Op 03-01-2023 om 18:33 schreef Alexander Kanavin:

On Tue, 3 Jan 2023 at 16:58, Ferry Toth <fntoth@...> wrote:

I know you don't want to spend time on this. So don't. Others may have
an interest or a tip to improve Yocto in this area.

I'm just trying to get an old patch from Richard Purdie to work
(http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wipqueue4&id=d66a327fb6189db5de8bc489859235dcba306237)
with later fixes by Trevor Gamblin.

I got Richards patch "Add shared make jobserver support <https://github.com/htot/poky/commit/28fd4b9e6e60bdd07352827ec713510ae86a39c6>" working again. It was a bit more work then last time, it was broken in 3 places (poky, make and python). Find it here: https://github.com/htot/poky/commits/kirkstone

It works by making each make believe it is a submake, while the named pipe holding the execution tokens is created by bitbake.

Of course as is it only limits number of threads started by make, not ninja. On my machine (16 threads) while compiling (linux, nodejs, nodejs-native etc.) simultaneously this limits memory consumption to ~14GB RAM. A little less then 1GB/thread.

Linking nodejs is another story. Each thread uses 4-5GB, and the recipe allows 5 threads simultaneously. With a nodejs_%.bbappend:

EXTRA_OEMAKE:prepend = "\
LINK='flock /tmp ${CXX}' \
"
only one thread will be linked simultaneously.

This brought my image build time from 3h > 1u47m (that is after removing out and sstate, not bbcache so excluding download time for sources).

Or alternatively something based on https://github.com/olsner/jobclient.

In principle we could use jobclient to start other programs then make, but for meta-intel-edison I didn't see any that would benefit from this.

It would be nice if ninja could use the same named pipe for the execution tokens as make, that might be for another holiday.

By all means please do. Everyone will benefit from teaching make to
abstain from starting new processes if RAM is tight, even people with
xeon/epyc grade hardware. It's not that I don't want to spend time on
this, it's simply that I do not have the time.

Ah, yes, that is a known problem. Like I said, this was for me just a little holiday project.

Now it works I'm not sure what to do. Richard marked the original patch as WIP. Maybe it's not appropriate for including into poky? If so I wouldn't mind carrying it in meta-intel-edison.

Or may be with a little work (from me) and a run on the CI servers we could make it go in?

Alex