Re: [linux-yocto] how can be improved the yocto build speed using network computer resources ?


Randy MacLeod
 

Oops, I just noticed that this was posted to the linux-yocto not
the yocto list. I'm BCCing linux-yocto and redirecting the
discussion to the yocto list where it's more appropriate.

See:
https://www.yoctoproject.org/community/mailing-lists/
for the scope of each list.

Thanks,
../Randy

On 2022-01-09 16:10, Randy MacLeod wrote:
On 2022-01-07 07:44, Weingart, Mircea wrote:
Hello ,

How can be improved the yocto build  speed using network computer resources ?

Suppose that the virtual linux machine where the yocto build is run , is in a network with other computers that may share their resources

with the yocto build virtual machine , what solution may work to improve significantly the speed of the yocto build run.

Thanks,

Mircea
Hi Mircea,
Building images using bitbake (or any source-based build system), does take
significant computational resources. If you're using a VM, you'd best be sure
that it is well-tuned and doesn't have a significant overhead compared to a
bare metal distro. Containers tend to be somewhat better here but can also
introduce IO overhead.
Have you tried using icecream?
https://stackoverflow.com/questions/14472175/distributed-compile-with-bitbake https://elinux.org/images/c/cc/Sweeten_Your_Yocto_Build_Times_with_Icecream.pdf Are you sharing sstate-cache across the nodes on the network?
https://stackoverflow.com/questions/31748577/how-does-shared-state-cache-in-yocto-work    https://docs.yoctoproject.org/singleindex.html#shared-state-cache
   https://wiki.yoctoproject.org/wiki/Enable_sstate_cache
Are you caching and sharing the downloads directory?
You could also buy a system with > ~100 cores and lots of RAM, SSDs but that does cost
quite a bit. Depending on your environment it may be a good decision.
At work, I almost never build on my laptop but rather use a hefty server with 192 cores!
$ time bitbake core-image-minimal
...
NOTE: Tasks Summary: Attempted 3104 tasks of which 0 didn't need to be rerun and all succeeded.
real    20m8.536s
user    0m20.574s
sys     0m3.759s
--------------------------
You can see that even on a large machine the build takes ~20 minutes due ot Amhahl's law,
i.e. there are some serialization points that prevent bitbake from continuously using
all of the resources on a machine.
Shameless plug:
There are some vendors, such as Wind River, who provide a cloud-based build environment to
try to help people who face this problem.
Good luck,
--
# Randy MacLeod
# Wind River Linux

Join yocto@lists.yoctoproject.org to automatically receive all group messages.