Re: [yocto-autobuilder2][PATCH] add prioritizeBuilders


Richard Purdie
 

On Thu, 2021-09-09 at 09:10 -0400, Trevor Gamblin wrote:
This prioritizeBuilders function sorts builders by the length of their
associated worker lists, so that builders that can only be assigned to a
small number of workers are assigned to those workers before other
builds that don't have specific needs when resources are limited. An
example might be when a slot is available on an Ubuntu-based worker, and
"oe-selftest-ubuntu" and "genericx86-64" build requests exist in the
queue. Since oe-selftest-ubuntu requires an Ubuntu-based worker and
genericx86-64 does not, genericx86-64 will be assigned a higher value
(lower priority) so that oe-selftest-ubuntu is assigned to that worker
first.

Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
---
builders.py | 13 +++++++++++++
master.cfg | 6 ++++++
2 files changed, 19 insertions(+)
We did merge and start using this. Unfortunately whilst I thought it would work,
it has some unfortunate side effects and we'll probably have to disable it.

If you have two build requests for say "a-full" builds, it will allocate oe-
selftest-XXX and reproducible-XXX to the XXX workers, starving generic targets.
You therefore end up with two builds running, both with half their builder
targets and it will hurt completion.

I did talk with upstream buildbot a little on irc about this challenge.

The default prioritizeBuilders function (_defaultSorter in
buildrequestdistributor.py) sorts by build request time.

They were suggesting some things which are fairly invasive like adding a
estimated completion time to the defaultSorter so we could weight slow builds.
To be honest it's beyond my comfort level with twisted defereds trying to query
and obtain the right data.

I have started wondering if we can in fact cheat and tweak the order we trigger
the builds in the a-full and a-quick targets. I think we could influence the
order of builderNames in the Triggerable and that would then change the
buildrequest timestamps so the default sorting method would work better for us
again.

I think that translates to adding your sorting method in this patch to
builderNamesFromConfigQuick and builderNamesFromConfigFull. Probably worth a
try?

Cheers,

Richard

Join yocto@lists.yoctoproject.org to automatically receive all group messages.