suggestions for version controlling multi-layer reproducible builds?


Robert P. J. Day
 

(if there's already a doc section for this somewhere, a pointer to
it would be just ducky.)

if one is building an image for a releasable, commercial product,
and that image involves pulling in numerous layers, then dumping all
sorts of proprietary apps on top of it, what are the possibilities for
how to version number the released images such that, if a client has
an issue, they can identify precisely the state of components that
went into the system they're working with?

in addition to all of the layers involved in the build, one has to
take into account that, when critical bugs are identified, an updated
RPM might be sent out and applied, whereupon that system's version
number is no longer perfectly accurate. in the end, the state of
someone's running system will be determined by a possibly huge
combination of selected packages, preferred package versions, build
config options, additional user space apps, hot fixes that have been
applied and so on.

what sort of meaningful "version number" can be applied to something
like that? i'm sure at least a few other people have to be doing
something like that, so i'm open to suggestions.

rday

--

========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================


Christopher Larson <clarson@...>
 


On Mon, Dec 12, 2016 at 8:20 AM, Robert P. J. Day <rpjday@...> wrote:
  if one is building an image for a releasable, commercial product,
and that image involves pulling in numerous layers, then dumping all
sorts of proprietary apps on top of it, what are the possibilities for
how to version number the released images such that, if a client has
an issue, they can identify precisely the state of components that
went into the system they're working with?

  in addition to all of the layers involved in the build, one has to
take into account that, when critical bugs are identified, an updated
RPM might be sent out and applied, whereupon that system's version
number is no longer perfectly accurate. in the end, the state of
someone's running system will be determined by a possibly huge
combination of selected packages, preferred package versions, build
config options, additional user space apps, hot fixes that have been
applied and so on.

  what sort of meaningful "version number" can be applied to something
like that? i'm sure at least a few other people have to be doing
something like that, so i'm open to suggestions.

I don’t think it’s possible for a version number to capture this at all. Better to use a script to capture the state of the system as needed. I.e. write build info to a file on target, and then combine that with a query of the package manager if you use packages for update distribution (though I wouldn’t advise use of packages for update distribution at all, that’s a different discussion :).
--
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics


Bryan Evenson
 

Robert,

-----Original Message-----
From: yocto-bounces@... [mailto:yocto-
bounces@...] On Behalf Of Robert P. J. Day
Sent: Monday, December 12, 2016 10:20 AM
To: Yocto discussion list <yocto@...>
Subject: [yocto] suggestions for version controlling multi-layer reproducible
builds?


(if there's already a doc section for this somewhere, a pointer to it would be
just ducky.)

if one is building an image for a releasable, commercial product, and that
image involves pulling in numerous layers, then dumping all sorts of
proprietary apps on top of it, what are the possibilities for how to version
number the released images such that, if a client has an issue, they can
identify precisely the state of components that went into the system they're
working with?
For my setup I have a separate package that is the distribution version number. Whenever we release an update, the distribution version number is updated. We then tag all our layers with the distribution version number for proper record keeping. The Angstrom distribution was (or maybe still is?) doing something similar and was the inspiration for our setup.

in addition to all of the layers involved in the build, one has to take into
account that, when critical bugs are identified, an updated RPM might be
sent out and applied, whereupon that system's version number is no longer
perfectly accurate. in the end, the state of someone's running system will be
determined by a possibly huge combination of selected packages, preferred
package versions, build config options, additional user space apps, hot fixes
that have been applied and so on.

what sort of meaningful "version number" can be applied to something like
that? i'm sure at least a few other people have to be doing something like
that, so i'm open to suggestions.

rday

Any time we release an update, no matter how minor, we update the distribution version package. Otherwise, as you state, you could have an update that you can't reproduce. We also archive the package repository and generated image files for each release so we can flash a board with a previous release and test from there. It can be a pain to get the process down the first time, but after that a simple Bash script can take care of all the hard work for you and ensure you don't skip a step.

Bryan


--

==========================================================
==============
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
==========================================================
==============


--
_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto


Richard Leitner
 

Hi,
On 12/12/2016 05:03 PM, Bryan Evenson wrote:

For my setup I have a separate package that is the distribution
version number. Whenever we release an update, the distribution
version number is updated. We then tag all our layers with the
distribution version number for proper record keeping. The Angstrom
distribution was (or maybe still is?) doing something similar and was
the inspiration for our setup.

...


Any time we release an update, no matter how minor, we update the
distribution version package. Otherwise, as you state, you could
have an update that you can't reproduce. We also archive the package
repository and generated image files for each release so we can flash
a board with a previous release and test from there. It can be a
pain to get the process down the first time, but after that a simple
Bash script can take care of all the hard work for you and ensure you
don't skip a step.
I'm using a similar approach, but instead of using a separate "version
package" I'm using the "BUILDNAME" and "DISTRO_VERSION" variables in my
distro.conf.

Then depending on how "large" the release is and which changes it
includes the Major, Minor or Patchlevel of the version is increased.

Furthermore (like Bryan already mentioned) for every release a tag in
every layer containing the DISTRO_VERSION is created.

The layer setup/checkout and build then is automated by a shellscript.
This shellscript takes the version as an argument and produces the
images, source archives, update packages and everything else needed for
a (nearly) reproducible builds.

Hope that helps.

regards,
Richard L


Mike Looijmans
 

On 12-12-16 16:20, Robert P. J. Day wrote:

(if there's already a doc section for this somewhere, a pointer to
it would be just ducky.)

if one is building an image for a releasable, commercial product,
and that image involves pulling in numerous layers, then dumping all
sorts of proprietary apps on top of it, what are the possibilities for
how to version number the released images such that, if a client has
an issue, they can identify precisely the state of components that
went into the system they're working with?
I always make a top-level git repository for the project. It contains all the "meta" layers as submodules (sometimes nested).

That way, the version of the top layer is the version of the whole product and can be reproduced any time.

in addition to all of the layers involved in the build, one has to
take into account that, when critical bugs are identified, an updated
RPM might be sent out and applied, whereupon that system's version
number is no longer perfectly accurate. in the end, the state of
someone's running system will be determined by a possibly huge
combination of selected packages, preferred package versions, build
config options, additional user space apps, hot fixes that have been
applied and so on.
The only way to version such a system is to actually dump the whole package version table (e.g. "opkg list-installed"). You could compare the table to the initially installed one and only send the difference, as an optimization.

what sort of meaningful "version number" can be applied to something
like that? i'm sure at least a few other people have to be doing
something like that, so i'm open to suggestions.
Version "numbers" are for marketing purposes only and have no useful meaning in version management. The git hash is the "technical" version number. Create a table somewhere to map the commercial version number to a git hash. The simplest implementation is to "tag" the version numbers in the top-level repository.



Kind regards,

Mike Looijmans
System Expert

TOPIC Products
Materiaalweg 4, NL-5681 RJ Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
E-mail: mike.looijmans@...
Website: www.topicproducts.com

Please consider the environment before printing this e-mail