Re: Master stability and the autobuilder.

Tian, Kevin <kevin.tian@...>

From: Elizabeth Flanagan
Sent: Thursday, December 23, 2010 4:57 AM


In preparation for the M3 milestone, I've been taking notes on what our
build and release process is and where the pain points are over the
course of the M2 milestone. A few things I've noted that are concerning:
Beth, thanks for sharing your observations and suggestions which are great.

1. We have a really long build cycle. It takes a good 40 hours before
contamination of master is recognized and a further 40 hours *at the
very minimum* before it can be verified as corrected.

2. We are very resource constrained. Because the build loop is so long,
our two build slaves are essentially in use 24/7. This will be
alleviated soon with a new build slave, but not until after the new year.

3. There was an issue in M2 (which I'm not totally filled in on yet, I'm
going to be pinging people for details) which caused some kernel issues
around the M2 branch for ppc and arm. I haven't seen (and I've been
swamped with email recently, so I may have missed it) any resolution on
this in master.

4. People are relying on the autobuilder to do their builds or to verify
their code.
I think this is not true. Current guideline for distro team is to make incremental
builds for all archs and make a full build for one arch before sending out the
pull request, which however can't promise catching all potential issues. Then
Saul's distro/{master,stage} is another trampoline stage to further filter out
issues before slipping into the master.

If this guideline is broken somehow, then we should reemphasize it.

My main goal in M3 and on, is the stability of master. If we were less
resource constrained, I could allow the autobuilder to be a global
community resource where people could just queue up their various
builds. That, unfortunately, is not the case for now. What this means is
that I have to ask people for a few things.

1. I'll be cleaning up the poky autobuilder config within the git repo
and making it very developer specific so that you should be able to run
a script and have the autobuilder set up to run incremental and fulls
(and an optional swabber target) on your own machine and have it work
right out of the box. Once my pull for the new autobuilder comes
through, please, set up your own local autobuilder instance. This should
be by the EOD today. I'll be somewhat available through the holidays to
assist people with local machine setups, but it should be fairly
that's really cool. Could you also provide some rough data how long it may
take for the various build targets you'll suggest to run in local machine? That
data would be useful to check out which ones are feasible to ensure both
quality and also development efficiency...

2. Please, make certain that your code builds on your local buildbot
before submitting a pull request. As the project grows, maintaining the
stability of master will become more and more important and the limited
resources of our maintainers and our infrastructure will become more and
more stretched.
Agree, we'll look into that after you send out new autobuilder script and then
we can discuss which targets are mandatory for the pull request.

3. I've dropped this recently, due to limited time, but I'm going to
start sending out more and more broken build emails. Please, do not
ignore these. Act on them. If you're think you may be responsible for
something that is broken, let people know that you recognize the
breakage and what you're planning on doing to fix it.

4. Until we know that master is stable (right now I know of 3 or 4
issues, email forthcoming), the autobuilder will be tasked with a very
limited number of buildsets. I'll be generating a milestone type build
off of master to get some quicker response (20 hours verses 38), and
incremental versions of milestone and sanity testing. We will still be
running nightly (although you can't see it on the autobuilder right now)
and distro-testing but until master is stable, those have got to be the
priorities on the autobuilder.

Join to automatically receive all group messages.