Hi Emily,

On Wed, Mar 18, 2020 at 10:16:23AM -0500, Emily wrote:
> Hi Laurent -
>
> Unfortunately I don't have full control over the repo that's using
> boost_python-mt so I'm not sure I can switch it right now.
>

You can create a patch for it. You can use devtool modify <your-recipe>
and create the patch from there, you have access to the sources with
that. devtool build <your-recipe> to check it builds okay.

> I realize this is not a long-term solution, as I'll need to update that
> code (and my OS) to python3 soon, but for now I've just copied the boost
> recipe from this commit
> <https://github.com/openembedded/openembedded-core/commit/ef603f41b5df4772bb598ec9d389dd5f858592af#diff-9c24742c4bfe7eb2853f86cce86b91c6>
> to
> my own layer, and added a BBMASK to the openembedded-core's boost recipe.
> This seems to work, except I get a QA error from the commit I'm using for
> the boost recipe now:
>

I don't think there is a need for BBMASK, you should be able to set
PREFERRED_VERSION_boost = "1.63.0" in local.conf or your machine
configuration file.

> ERROR: boost-1.63.0-r1 do_package: QA Issue: boost: Files/directories were
> installed but not shipped in any package:
>   /usr/lib/libboost_numpy.so.1.63.0
> Please set FILES such that these items are packaged. Alternatively if they
> are unneeded, avoid installing them or delete them within do_install.
> boost: 1 installed and not shipped files. [installed-vs-shipped]
> ERROR: boost-1.63.0-r1 do_package: Fatal QA errors found, failing task.
> ERROR: boost-1.63.0-r1 do_package: Function failed: do_package
>
> I found a log
> <https://www.yoctoproject.org/irc/%23yocto.2017-03-09.log.html> that
> mentions this exact error, and also mentions a patch that fixes it - I've
> tried and thus far failed to find that patch. I'm not sure if this will
> actually work, but I thought I'd check and see if anyone had any ideas.
>

If all of this is really temporary you can install it in some package,
or even create a new package for it. (PACKAGES =+ "boost-numpy",
FILES_${PN}-numpy = "/usr/lib/libboost_numpy.so.1.63.0").

No more ideas tbh. I would try to patch "your" SW first and see if it
brings you somewhere, but that's what I would do, each their own way.

Quentin

4 hours seems extremely long to me for a 8 core system, but I have not tried this in a while. Were you removing all the sources and re-downloading them for every build?

 

From: <yocto@...> on behalf of Srini <rsrinivasan@...>
Date: Wednesday, March 18, 2020 at 10:13 AM
To: "mikko.rapeli@..." <mikko.rapeli@...>, "mike.looijmans@..." <mike.looijmans@...>
Cc: "yocto@..." <yocto@...>
Subject: Re: <EXT> Re: [yocto] What are the key factors for yocto build speed?

 

My own experience (pardon me if already discussed)

Fought the build times for several months - ending up eventually at 8 cores (but specifying 16 threads in poky builds). Best times for my build about 4 hours. Clearly impractical during engineering.

Generated an sdk and used it for app development. Each build is now a minute or 2.

Using a homegrown utility, updated the image file with applications in a jiffy - to produce burnable sdcard image.

Complete build required only for the final release -- or made major changes like python2 to python3!

YMMV.

srini

-----Original Message-----
From: yocto@... <yocto@...> On Behalf Of Mikko Rapeli via Lists.Yoctoproject.Org
Sent: Wednesday, March 18, 2020 11:52 AM
To: mike.looijmans@...
Cc: yocto@...
Subject: <EXT> Re: [yocto] What are the key factors for yocto build speed?

On Wed, Mar 18, 2020 at 04:09:39PM +0100, Mike Looijmans wrote:
> On 18-03-2020 15:49, Adrian Bunk via Lists.Yoctoproject.Org wrote:
> > On Wed, Mar 18, 2020 at 10:12:26AM -0400, Jean-Marie Lemetayer wrote:
> > > ...
> > > For example one of our build servers is using:
> > > - AMD Ryzen 9 3900X
> > > ...
> > > - 32Go DDR4 3200 MHZ CL14
> > > ...
> > > It is a really good price / build time ratio configuration.
> >
> > Depends on what you are building.
> >
> > Building non-trivial C++ code (e.g. webkitgtk) with 24 cores but
> > only 32 GB RAM will not work, for such code you need more than 2
> > GB/core.
>
> Seems a bit excessive to buy hardware just to handle a particular
> corner case. Most of OE/Yocto code is plain C, not even C++.
>
> My rig only has 8GB but doesn't run into memory issues during big GUI
> builds. The only thing that made it swap was the populate_sdk task
> that created a 1.1GB fiel and needed 20GB of RAM to compress that.
> Took a few minutes more due to swapping.
> I submitted a patch today to fix that in OE.
>
> Your mileage may vary. But RAM is easy to add.

Well, I can't build with under 2 gigs per core or I run out of physical memory and kernel oom-killer kicks in to kill the build. Also can't run with yocto default parallel settings which only take into account the number of cores and thus have a custom script which does caps the threads so that 2 gigs of RAM for each are available.

Though I'm sure plain C and plain poky projects have less requirements for RAM.

-Mikko

________________________________

CONFIDENTIALITY NOTICE: This email message and any attachments are confidential and may be privileged and are meant to be read by the intended recipient only. If you are not the intended recipient, please notify sender immediately and destroy all copies of this message and any attachments without reading or disclosing their contents. Thank you

Also as Quentin suggested it might be needed to remove the "-mt"
suffix, but I see it in the do_install() method of the "boost" recipe
for revision 1.64.

Kind regards, Laurent.

On Tue, Mar 17, 2020 at 11:17 PM Laurent Gauthier via
Lists.Yoctoproject.Org
<laurent.gauthier=soccasys.com@...> wrote:
>
> Hi again Emily,
>
> You have put your finger on the core issue there.
>
> The name of the boost python library which is linked in your
> "opc-ua-server-gfex" recipe is "boost_python-mt" where it seems that
> it should be "boost_python3-mt".
>
> I guess you should have a look at the CMakeLists.txt of the
> "opc-ua-server-gfex" software package to see how to switch to linking
> with "boost_python3-mt" instead as a quick review of the boost recipe
> seems to reveal no python2 support.
>
> Kind Regards, Laurent.
>
>
> On Tue, Mar 17, 2020 at 7:19 PM Emily S <easmith5555@...> wrote:
> >
> > Hi Laurent -
> >
> > Thanks for the suggestion. It looks like there are a few packages that provide libboost_python3:
> >
> > find tmp/work/*/*/*/packages-split -name libboost_python\*
> > tmp/work/aarch64-poky-linux/boost/1.64.0-r0/packages-split/boost-dev/usr/lib/libboost_python3.so
> > tmp/work/aarch64-poky-linux/boost/1.64.0-r0/packages-split/boost-python/usr/lib/libboost_python3.so.1.64.0
> > tmp/work/aarch64-poky-linux/boost/1.64.0-r0/packages-split/boost-staticdev/usr/lib/libboost_python3.a
> > tmp/work/aarch64-poky-linux/boost/1.64.0-r0/packages-split/boost-dbg/usr/lib/.debug/libboost_python3.so.1.64.0
> >
> > Adding boost-python to RDEPENDS doesn't help, I have tried it before also. This sort of made sense to me as I thought RDEPENDS was for run time dependencies, but perhaps I have misunderstood.
> >
> > I am working with the rocko branch and python2, so perhaps that's the problem?
> >
> > Thanks,
> > Emily
> >
> >
> > On Tue, Mar 17, 2020 at 1:02 PM Laurent Gauthier <laurent.gauthier@...> wrote:
> >>
> >> Hi Emily,
> >>
> >> To find the solution to your issue in a rational and deterministic way
> >> we need to start from the error message.
> >>
> >> What I understand is that while build recipe "opc-ua-server-gfex" you
> >> get an error message that says in short "ld: cannot find
> >> -lboost_python-mt".
> >>
> >> Therefore you need to determine which recipe (and which package in
> >> that recipe) provides the "boost_python-mt" library.
> >>
> >> One way to determine this is to run something like this:
> >>
> >>     find /local/d6/easmith5/rocko_bitbake/poky/build/tmp/work/*/*/*/packages-split
> >> -name libboost_python\*
> >>
> >> This should show you which recipe, and which package produced by that
> >> recipe has the library you are looking for.
> >>
> >> The names of the first-level directories inside "packages-split" are
> >> packages names.
> >>
> >> Based on things you have mentioned in your previous e-mail my guess
> >> would be that adding an RDEPENDS = "boost-python" should help, but I
> >> might be wrong.
> >>
> >> I hope this will help you move in the right direction.
> >>
> >> Kind regards, Laurent.
> >>
> >>
> >> On Tue, Mar 17, 2020 at 6:07 PM Emily <easmith5555@...> wrote:
> >> >
> >> > Hi Quentin -
> >> >
> >> > That's what I tried originally! DEPENDS on boost and not boost-python gives me the following error when trying to build my original recipe:
> >> >
> >> > /local/d6/easmith5/rocko_bitbake/poky/build/tmp/work/aarch64-poky-linux/opc-ua-server-gfex/1.0+gitAUTOINC+921c563309-r0/recipe-sysroot-native/usr/bin/aarch64-poky-linux/../../libexec/aarch64-poky-linux/gcc/aarch64-poky-linux/7.3.0/ld:
> >> > cannot find -lboost_python-mt
> >> >
> >> > So I tried a bunch of other this which also didn't seem to work.
> >> >
> >> > Thanks for the response!
> >> > Emily
> >> >
> >> >
> >> > On Tue, Mar 17, 2020 at 11:04 AM Quentin Schulz <quentin.schulz@...> wrote:
> >> >>
> >> >> Hi Emily,
> >> >>
> >> >> On Tue, Mar 17, 2020 at 10:44:10AM -0500, Emily wrote:
> >> >> > Hi all -
> >> >> >
> >> >> > I'm trying to build an opca recipe (
> >> >> > https://github.com/kratsg/meta-l1calo/blob/add/opcServer/recipes-core/opc-ua/opc-ua-server-gfex_git.bb)
> >> >> > and it's giving me a build error like:
> >> >> >
> >> >> > |
> >> >> > /local/d6/easmith5/rocko_bitbake/poky/build/tmp/work/aarch64-poky-linux/opc-ua-server-gfex/1.0+gitAUTOINC+921c563309-r0/recipe-sysroot-native/usr/bin/aarch64-poky-linux/../../libexec/aarch64-poky-linux/gcc/aarch64-poky-linux/7.3.0/ld:
> >> >> > cannot find -lboost_python-mt
> >> >> >
> >> >> > Which seems to indicated I need to add boost-python to my list of DEPENDS.
> >> >> > When I do that I get the error:
> >> >> >
> >> >> > ERROR: Nothing PROVIDES 'boost-python' (but
> >> >> > /local/d6/easmith5/rocko_bitbake/meta-l1calo/recipes-core/opc-ua/
> >> >> > opc-ua-server-gfex_git.bb DEPENDS on or otherwise requires it). Close
> >> >> > matches:
> >> >> >   boost RPROVIDES boost-python
> >> >> >
> >> >> > I've tried adding PACKAGECONFIG_pn-boost="python" to both my local.conf and
> >> >> > to the image definition as I saw this online, but neither seems to work. It
> >> >> > seems like boost-python is being built (I can see it in the list when I run
> >> >> > oe-pkgdata-utils list-pkgs -p boost) but it doesn't seem to be available at
> >> >> > build for this recipe.
> >> >> >
> >> >> > If I remove boost from my list of DEPENDS I get an error about that, so
> >> >> > obviously boost itself is available at build for the original recipe. I've
> >> >> > also tried adding boost-native to DEPENDS, also did not work.
> >> >> >
> >> >> > Is there something obvious I'm missing, or some trick to making
> >> >> > boost-python available at build for this other recipe?
> >> >> >
> >> >>
> >> >> Ok so two things.
> >> >>
> >> >> Yes, I'd say you need python in PACKAGECONFIG for boost. However, it
> >> >> seems it's already part of the default value of PACKAGECONFIG, c.f.
> >> >> http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-support/boost/boost.inc?h=master#n45
> >> >> so if no bbappend is overriding it, I'd say you're safe. Better check
> >> >> with the version in your layer that it's there.
> >> >> If you don't want to check manually, try `bitbake -e boost | grep -e
> >> >> "^PACKAGECONFIG="`
> >> >>
> >> >> DEPENDS contains only recipes (well, PROVIDES but PROVIDES has the name
> >> >> of the recipe in it at all times) while RDEPENDS_* (usually ${PN}*) contains
> >> >> only packages.
> >> >>
> >> >> So you need to DEPENDS on boost, not boost-python. That should be enough
> >> >> hopefully!
> >> >>
> >> >> Quentin
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Laurent Gauthier
> >> Phone: +33 630 483 429
> >> http://soccasys.com
>
>
>
> --
> Laurent Gauthier
> Phone: +33 630 483 429
> http://soccasys.com
>



--
Laurent Gauthier
Phone: +33 630 483 429
http://soccasys.com

Hi,

In my company we have tested some "big Ryzen" configurations to speed up Yocto builds.

The conclusion of these tests is that the build time is almost only related to the CPU score: https://www.cpubenchmark.net/high_end_cpus.html

The speed (overclock) and size of the RAM does not influence the build time, neither the the use of a SATA SSD or a NVME.

For example one of our build servers is using:
- AMD Ryzen 9 3900X
- ASUS PRIME X570-P
- 32Go DDR4 3200 MHZ CL14
- 500Go SSD

It is a really good price / build time ratio configuration.

Best regards,
Jean-Marie

On Mar 18, 2020, at 2:29 PM, Paul Barker pbarker@... wrote:
> On Wed, 18 Mar 2020 at 13:01, Mikko Rapeli <mikko.rapeli@...> wrote:
>>
>> On Wed, Mar 18, 2020 at 05:52:37AM -0700, Oliver Westermann wrote:
>> > Hey,
>> >
>> > We're currently using a VM on Windows and it's a lot slower than the native
>> > linux build (which is expected).
>> > We're looking into getting a dedicated build server for our team (basically a
>> > self-build tower PC). Any suggestions what to put in that build to get the most
>> > out of it?
>> >
>> > Currently we're looking at a big Ryzen, 64G of RAM and one or multiple SSDs on a
>> > "consumer grade" board like the X570.
>>
>> Drop all virtualization and go for Linux on bare metal. Then make sure there
>> is enough(tm) physical RAM for each CPU thread. For a "big Ryzen" the 64G of RAM
>> sounds
>> too little. I'd go higher there, but it all depends what kind of project
>> is being compiled.
>>
>> I would also setup CPU, RAM, disk IO and network IO monitoring on the build
>> machines
>> and review and monitor the build times and results when the project evolves.
>> There are
>> times when most CPUs will be idling and there will be times when IO to disk is
>> happening even when RAM is available. Linux kernel can be tuned to avoid disk IO
>> if
>> RAM is still available for example.
>
> What Mikko said is excellent advice.
>
> I'd recommend NVMe drives if you can. My build machine has two large
> M.2 NVMe drives, one is used for the working directory and the other
> for sstate, downloads and source checkouts to make the best use of the
> available bandwidth. I find that XFS is a better fit than ext4 when
> you've got fast drives and highly parallel I/O workloads.
>
> Make sure you spread your RAM across the different memory channels on
> your motherboard as this increases memory bandwidth - this usually
> means either filling your RAM slots or half-filling them with RAM in
> every 2nd slot but you'll need to check the motherboard manual to
> confirm the channel layout.
>
> Let me know if you need a detailed review of your proposed setup.
>
> Thanks,
> Paul
>
>