Hello,

Leading up to our 0.9 release, our autobuilder has been building an increasing number of targets for our nightly-release buildset. We've now reached the point that the nightly build takes more than 24 hours to run (> 26 hours, in fact) - which is clearly a problem on a build that we'd like to generate on a daily basis.

The following is a list of everything which is built within nightly-release:

The following targets are built for qemux86, qemux86-64, qemuarm, qemumips, and qemuppc:

* poky-image-minimal
* poky-image-sato
* poky-image-lsb
* poky-image-sdk
* meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)

For emenlow and atom-pc, we build:

* poky-image-minimal-live
* poky-image-sato-live
* poky-image-sdk-live
* meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)

Finally, we also build the Eclipse plugin, and copy the shared state prebuilds and RPM output at the end of the build.

I was going to post build times for some of these targets for reference, but it would be misleading as we build the targets in succession (e.g, we start with poky-image-sdk which takes the bulk of the time, and then the other targets can largely rely on the shared state builds).

Ideally I think our nightly build should take much less than 24 hours to build. The question is what we can move out of the nightly build and do on perhaps a weekly basis instead?

Our buildserver hardware is a dual quad-core Xeon server with 12 GB of RAM. Throwing hardware at the problem is another solution, but not an inexpensive one (we'd be looking at a 4-socket machine filled with quad-cores and 32 GB of RAM).

I'm open to ideas on how to address this issue. QA will be driving a lot of the requirements and I'm especially interested to hear your thoughts.

Scott

On Mon, 2010-11-01 at 02:38 -0700, Scott Garman wrote:

Leading up to our 0.9 release, our autobuilder has been building an
increasing number of targets for our nightly-release buildset. We've now
reached the point that the nightly build takes more than 24 hours to run
(> 26 hours, in fact) - which is clearly a problem on a build that we'd
like to generate on a daily basis.

The following is a list of everything which is built within nightly-release:

The following targets are built for qemux86, qemux86-64, qemuarm,
qemumips, and qemuppc:

* poky-image-minimal
* poky-image-sato
* poky-image-lsb
* poky-image-sdk
* meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)

For emenlow and atom-pc, we build:

* poky-image-minimal-live
* poky-image-sato-live
* poky-image-sdk-live
* meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)

Finally, we also build the Eclipse plugin, and copy the shared state
prebuilds and RPM output at the end of the build.

I was going to post build times for some of these targets for reference,
but it would be misleading as we build the targets in succession (e.g,
we start with poky-image-sdk which takes the bulk of the time, and then
the other targets can largely rely on the shared state builds).

Ideally I think our nightly build should take much less than 24 hours to
build. The question is what we can move out of the nightly build and do
on perhaps a weekly basis instead?

Our buildserver hardware is a dual quad-core Xeon server with 12 GB of
RAM. Throwing hardware at the problem is another solution, but not an
inexpensive one (we'd be looking at a 4-socket machine filled with
quad-cores and 32 GB of RAM).

There doesn't just have to be one build machine, we are going to end up
needing multiple machines and we can split the load between them quite
easily. I think there is going to be a second machine needed alongside
the existing one regardless of what other optimisations we make.

The changes in development at the moment will have a mixed effect on the
autobuilder's load. On the plus side we know there are performance
regressions with 0.9 which we're about to investigate. I can think of
one change I have in mind which on its own should get the builds back
under the 24 hour time.

On the down side there are going to be changes that need increased
horsepower from the build machines such as the runtime testing we're
close to enabling or enabling builds of world.

Cheers,

Richard

From: yocto-bounces@... [mailto:yocto-
bounces@...] On Behalf Of Richard Purdie
Sent: Monday, November 01, 2010 4:03 AM

On Mon, 2010-11-01 at 02:38 -0700, Scott Garman wrote:

Our buildserver hardware is a dual quad-core Xeon server with 12 GB of
RAM. Throwing hardware at the problem is another solution, but not an
inexpensive one (we'd be looking at a 4-socket machine filled with
quad-cores and 32 GB of RAM).

There doesn't just have to be one build machine, we are going to end up
needing multiple machines and we can split the load between them quite
easily. I think there is going to be a second machine needed alongside
the existing one regardless of what other optimisations we make.

I can buy another server and contribute it to the build effort. I had intended to buy one this quarter to begin hosting yoctoproject.org and a source mirror, but we could offload part of the builds to it as well.

As you know, I want us to get to the root of the problem, which is efficiency of the build process. Since RP has already committed to addressing this, I'm fine with a short-term fix if it will help. :-)

Scott - please put in an order, will tell you the max amount today when we're f2f.

Dave

Hello,

Leading up to our 0.9 release, our autobuilder has been building an
increasing number of targets for our nightly-release buildset. We've
now reached the point that the nightly build takes more than 24 hours
to run (> 26 hours, in fact)
- which is clearly a problem on a build that we'd like to generate on
a daily basis.

The following is a list of everything which is built within nightly-release:

The following targets are built for qemux86, qemux86-64, qemuarm,
qemumips, and qemuppc:

* poky-image-minimal
* poky-image-sato
* poky-image-lsb
* poky-image-sdk
* meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)

For emenlow and atom-pc, we build:

* poky-image-minimal-live
* poky-image-sato-live
* poky-image-sdk-live
* meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)

Finally, we also build the Eclipse plugin, and copy the shared state
prebuilds and RPM output at the end of the build.

I was going to post build times for some of these targets for
reference, but it would be misleading as we build the targets in
succession (e.g, we start with poky-image-sdk which takes the bulk of
the time, and then the other targets can largely rely on the shared state builds).

Ideally I think our nightly build should take much less than 24 hours
to build. The question is what we can move out of the nightly build
and do on perhaps a weekly basis instead?

How about customizing the nightly build to build part of targets for every day? For example, x86/x86_64 for Monday, atom-pc/emenlow for Tuesday..... etc.
And we can schedule a new build task for weekly, which is only triggered on Wednesday(or maybe Tuesday night) for QA weekly testing and with all targets built out.

Our buildserver hardware is a dual quad-core Xeon server with 12 GB of RAM.
Throwing hardware at the problem is another solution, but not an
inexpensive one (we'd be looking at a 4-socket machine filled with
quad-cores and 32 GB of RAM).

I'm open to ideas on how to address this issue. QA will be driving a
lot of the requirements and I'm especially interested to hear your thoughts.

Scott


_______________________________________________
yocto mailing list
yocto@...
https://lists.pokylinux.org/listinfo/yocto

Best Regards,
Jiajun

From: Richard Purdie
Sent: Monday, November 01, 2010 7:03 PM

On Mon, 2010-11-01 at 02:38 -0700, Scott Garman wrote:

Leading up to our 0.9 release, our autobuilder has been building an
increasing number of targets for our nightly-release buildset. We've now
reached the point that the nightly build takes more than 24 hours to run
(> 26 hours, in fact) - which is clearly a problem on a build that we'd
like to generate on a daily basis.

The following is a list of everything which is built within nightly-release:

The following targets are built for qemux86, qemux86-64, qemuarm,
qemumips, and qemuppc:

* poky-image-minimal
* poky-image-sato
* poky-image-lsb
* poky-image-sdk
* meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)

For emenlow and atom-pc, we build:

* poky-image-minimal-live
* poky-image-sato-live
* poky-image-sdk-live
* meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)

Finally, we also build the Eclipse plugin, and copy the shared state
prebuilds and RPM output at the end of the build.

I was going to post build times for some of these targets for reference,
but it would be misleading as we build the targets in succession (e.g,
we start with poky-image-sdk which takes the bulk of the time, and then
the other targets can largely rely on the shared state builds).

Ideally I think our nightly build should take much less than 24 hours to
build. The question is what we can move out of the nightly build and do
on perhaps a weekly basis instead?

Our buildserver hardware is a dual quad-core Xeon server with 12 GB of
RAM. Throwing hardware at the problem is another solution, but not an
inexpensive one (we'd be looking at a 4-socket machine filled with
quad-cores and 32 GB of RAM).

There doesn't just have to be one build machine, we are going to end up
needing multiple machines and we can split the load between them quite
easily. I think there is going to be a second machine needed alongside
the existing one regardless of what other optimisations we make.

The changes in development at the moment will have a mixed effect on the
autobuilder's load. On the plus side we know there are performance
regressions with 0.9 which we're about to investigate. I can think of
one change I have in mind which on its own should get the builds back
under the 24 hour time.

In case you didn't note Qing's investigation last week, attach his mail for
your reference.

Thanks
Kevin

On the down side there are going to be changes that need increased
horsepower from the build machines such as the runtime testing we're
close to enabling or enabling builds of world.

Cheers,

Richard

_______________________________________________
yocto mailing list
yocto@...
https://lists.pokylinux.org/listinfo/yocto

On 11/01/2010 06:35 AM, Stewart, David C wrote:

I can buy another server and contribute it to the build effort. I had
intended to buy one this quarter to begin hosting yoctoproject.org
and a source mirror, but we could offload part of the builds to it as
well.

Great, thanks Dave! :)

I have spec'ed out and initiated an order for our newest server. In case anyone likes to geek out on hardware specs, here they are:

Dell PowerEdge R710
2x Xeon X5680 @ 3.33Ghz
24 GB RAM
6x 1.5 TB hard drives for use in a RAID5 array

While it's not a 4-processor system, the additional resources should be sufficient for us to tackle our resource problems my distributing the workload more evenly.

The system will likely arrive and be set up around the end of the month.

Scott