On Thu, Mar 19, 2020 at 11:04:26AM +0000, Richard Purdie wrote:

On Thu, 2020-03-19 at 08:05 +0000, Mikko Rapeli wrote:

Once this is done, IO still happens when anything calls sync() and
fsync() and worst offenders are package management tools. In yocto
builds, package manager actions to flush to disk are always useless
since rootfs images are going to be compressed and original ones
wiped by rm_work anyway.
I've tried to hook eatmydata library into the build which makes
sync() and fsync() calls no-ops but I've still failed to fix all the
tools and processes called during build from python code. For shell
based tasks this does it:

$ export LD_LIBRARY_PATH=/usr/lib/libeatmydata
$ export LD_PRELOAD=libeatmydata.so
$ grep -rn LD_PRELOAD conf/local.conf
conf/local.conf:305:BB_HASHBASE_WHITELIST_append = " LD_PRELOAD"
conf/local.conf:306:BB_HASHCONFIG_WHITELIST_append = " LD_PRELOAD"

Doesn't pseudo intercept and stop these sync calls already? Its
supposed to so if its not, we should fix that.

I will double check, but I'm sure I see IO going to disk when plenty of RAM
is still available in page cache.

The effect is clearly visible during build time using Performance Co-
Pilot (pcp) or similar tools to monitor CPU, memory, IO and network
IO. The usage of RAM as page cache grows until limits are hit and
only then writes to disk start, except for the python image
classes... Hints to fix this are welcome!

To my knowledge of monitoring our builds, there is a lot of
optimization
potential to better build times. CPU are under utilized during
bitbake recipe parsing

Recipe parsing should hit 100% CPU, its one of the few places we can do
that.

I'm not fully aware what bitbake does before starting task execution.
With sumo, there is an initial spike in CPU use and then a long single
thread wait where log shows "Initialising tasks..." and Cooker process
is using a single core. For me this takes at least one minutes for
every build. Same is visible with zeus too.

Example graph from pmchart:

https://mcfrisk.kapsi.fi/temp/bitbake_start_to_task_execution.png

, fetch, configure, package and rootfs tasks.

Sadly these tasks are much harder.

Yep.

Memory is not fully utilized either since IO through sync()/fsync()
happens everywhere

non-pseudo tasks?

I'll try to check this case once more.

-Mikko