the downside of parallelism
Robert P. J. Day
refactoring existing (legacy) code base into more bite-sized
bitbake recipes to speed up build by taking advantage of 6-core
(12-thread) dell laptop ... end result is that i get so much
parallelism that laptop shuts down from overheating.
le *sigh* ...
toggle quoted messageShow quoted text
This is hardly Yocto's fault: putting a 6 core CPU into a laptop should be validated by saturating all cores for several hours and ensuring the cooling and frequency throttling can handle it. Laptop OEM is trying to be cheap I'd say.
On Sun, 20 Jun 2021 at 11:55, Robert P. J. Day <rpjday@...> wrote:
Robert P. J. Day
On Sun, 20 Jun 2021, Alexander Kanavin wrote:
This is hardly Yocto's fault: putting a 6 core CPU into a laptopi wasn't blaming yocto ... that was an attempt at wry humour. or
On 20/06/2021 12:55, Robert P. J. Day wrote:
refactoring existing (legacy) code base into more bite-sizedSorry for my stupid question;)
If you have 12 threads, it's 12 threads and not more.
I guess there needs to be some kind of dependency between your new recipes.
make -j can use as many cores as you like even with a single recipe.
How would it speed up builds if you break up a recipe into multiple recipes?
What is your refactoring approach?
le *sigh* ...Regards,
Robert P. J. Day
On Sun, 20 Jun 2021, Robert Berger@yocto.user wrote:
Hi,the (legacy) code base i was handed involves several sizable
makefiles that were not designed to take advantage of parallelism --
apparently, they come from an environment (QNX?) where "make" does not
do parallelism. so rather than design makefiles with parallelism in
mind, many of the makefiles correspond to some large subsystem where
the top-level target has a rule that simply descends into the numerous
sub-component directories one at a time:
$(MAKE) -C fubar1
$(MAKE) -C fubar2
$(MAKE) -C fubar42
and last i heard, the commands in any rule are definitely processed
serially. so when the local folks started transmogrifying the code
base to use YP, they just used SRC_URI to point at the existing
makefiles (which *must* stay where they are for the purpose of legacy
compatibility). so as a start, there is a recipe "fubar.bb" which just
runs that top-level Makefile, but given its structure, once it starts,
all you see is "fubar compiling" as it serially processes all of its
now, most of the major components have sub-directories with, say,
the shared libs, the test suite, and so on, and a lot of other
components that allegedly DEPENDS on "fubar" obviously don't need
*all* of fubar to finish compiling, just normally the library
sub-component, but as it is, they have to wait for all of it, so you
normally see "fubar compiling", and nothing else, for several minutes,
and as soon as that finishes, a bunch of deps that have been patiently
waiting finally kick off.
as i refactor the above into smaller recipes, then the dependencies
build much faster and as soon as the shared lib recipe finishes, all
those other recipes can start, so i'm definitely getting way more
parallelism but, as i mentioned, that comes with a price.
now, given my unlimited cleverness into refactoring recipes, most of
my 6 cores (12 threads) are busy, which is driving up the load average
over the course of the build (easily over an hour for the whole
thing), and sometimes the load average peaks at over 130, and the air
vent on this laptop is blowing air so hot, it can actually burn you if
you leave your hand there too long, and every so often, it just locks
up and shuts down.
what irony -- i redesign the recipes to boost parallelism, only to
fall victim to hardware limitations.