Framework to implement mirroring in Yocto


Gutierrez, Hernan Ildefonso (Boise R&D, FW) <hernan_gutierrez@...>
 

Hi,

 

We are planning to implement a mirror for both source code downloaded and sscache in our work environment.

 

We are planning to use Nexus and Nuget to allow storage and versioning control. We don’t know if these are the right tools.

 

Since we are about to embark in this project, before starting I wanted to know if you have some pointers on how can be implemented a mirroring framework for yocto.

 

Not sure if there are some common tools (other tools) with which yocto integrates nicely.

 

Any pointers will be appreciated,

 

--Hernan

 

PS> We are in morty branch currently.


Andre McCurdy <armccurdy@...>
 

On Fri, Oct 6, 2017 at 3:08 PM, Gutierrez, Hernan Ildefonso (Boise
R&D, FW) <hernan_gutierrez@hp.com> wrote:
Hi,

We are planning to implement a mirror for both source code downloaded and
sscache in our work environment.

We are planning to use Nexus and Nuget to allow storage and versioning
control. We don’t know if these are the right tools.
There are two main aspects of a mirror, fetching from the mirror
(happens during the build process) and populating the mirror
(generally happens outside the main build process).

For fetching from the mirror during the build process, the bitbake
fetchers are used, so your server needs to be accessible via a
protocol which the fetchers support, e.g http or https would be
recommended. Support for sstate mirrors via https with a password (ie
if you want private mirrors which can be accessed over the internet
without needing a VPN) was first added in pyro, so you'll need to
backport a couple of patches to make that work with morty. If the
standard bitbake fetchers don't meet your needs then it's not a huge
task to write a custom fetcher - see the Amazon S3 fetcher recently
added to bitbake. It's less than 100 lines of code.

For populating the mirror, you will likely need to implement something
custom, ie outside bitbake and the build process, since bitbake only
really supports writing sstate and downloads to local directories. It
could be something as simple as running rsync after every build to
upload from the build server's local sstate and downloads directories
to the mirror server(s). It could be something more complex - it's up
to you.

Note that for both sstate and downloads the files on the mirror would
be expected to be unique (e.g. there would never be two different
versions of "gcc-6.3.0.tar.bz2", etc), so there's no obvious reason to
be able to snapshot or version the state of the mirrors. Just let the
files accumulate over time - the old versions won't be lost or
over-written.

Since we are about to embark in this project, before starting I wanted to
know if you have some pointers on how can be implemented a mirroring
framework for yocto.

Not sure if there are some common tools (other tools) with which yocto
integrates nicely.

Any pointers will be appreciated,


Mark Hatle <mark.hatle@...>
 

On 10/6/17 5:08 PM, Gutierrez, Hernan Ildefonso (Boise R&D, FW) wrote:
Hi,

 

We are planning to implement a mirror for both source code downloaded and
sscache in our work environment.
We mirror a ton of code for our customers. We have two types or mirrors that we
deliver. Raw 'git servers' and tarball (or equivalent) mirrors.

The way that we populate these mirrors is to use bitbake with a fetchall
'universe'. That way everything that could be downloaded (with a few
exceptions) is downloaded.

git archives, we identify and create git mirrors in an external server. This
server regularly updates the git servers from the upstream location. This
allows us to have git mirrors that are 'live' git servers, and not just tarballs
that could get out of sync.

For this work, you can easily setup a 'premirror' in your projects of the format:

PREMIRRORS_append = " \
git://.*/.* git://mirror.hostname.here/git/MIRRORNAME;protocol=git \n \"
"

You can adjust the protocol to be file, git, or http (as necessary for your
environment). (In our case, it's usually file.)

For the files, it's even easier. Just have a shared fileserver that contains
all of the downloads:

PREMIRRORS_append = " \
.*://.*/.* file://<path_to_mirror>/downloads/ \n \
"


We are planning to use Nexus and Nuget to allow storage and versioning control.
We don’t know if these are the right tools.
If you want to formalize this further, there are alternative ways to do this.
(We use an approach where we have 'download layers'.) Between the setup program
(https://github.com/Wind-River/wr-lx-setup) and these special download layers
our users can download (to their local system) both the mirrors and tarball
downloads.

Our typical download layer looks like:

conf/layer.conf:

BBFILE_COLLECTIONS += "core-dl"
BBFILE_PATTERN_core-dl = ""
BBFILE_PATTERN_IGNORE_EMPTY_core-dl = "1"

# This should only be incremented on significant changes that will
# cause compatibility issues with other layers
LAYERVERSION_core-dl = "2.4"

# We have a pre-populated downloads directory, add to PREMIRRORS
PREMIRRORS_append = " \
.*://.*/.* file://${LAYERDIR}/downloads/ \n \
git://.*/.* git://${LAYERDIR}/git/MIRRORNAME;protocol=file \n \
"

Then insides of the 'downloads' directory we have a copy of all of the tarballs
used by the 'meta' layer (oe-core). [note this can grow fairly large, so we
generally have a 2.4, then a 2.5, etc versions to keep the size managable.]

The setup program uses a local layer index to control everything. We have a
single local patch to our oe-core that adds a 'recommendation' on this new
download layer.

LAYERRECOMMENDS_core = "core-dl (= 2.4)"


The layer index will bring this is as an 'optional' dependency which the setup
program can then use. So if you do this for all of the layers you use -- then
a user calling setup (with our private layer index) will automatically get the
downloads that match only the layers they choose to bring into their project.

(The setup program also has a --mirror option, will will allow a local user to
mirror down the layer index contents, as well as the layers that they are using.
This is intended both for our commercial product delivery, but also for
customers to be able to snapshot a given installation so they have all of the
code necessary to reproduce the environment in the future.)


Since we are about to embark in this project, before starting I wanted to know
if you have some pointers on how can be implemented a mirroring framework for yocto.
You can start simple, and just setup a local fileserver and git server with the
premirror setup.

You can extend this to include custom delivery and installation to customers if
this is appropriate for your environment.

(Note, the setup program is current external to OE / YP. We are working to
integrate this into bitbake. While there is no specific timeline promised, we
started this work in the 2.4 timeframe, and I expect much of what I explained
above will be implemented in the 2.5 timeframe. But until then, you can use the
wr-lx-setup repository I mentioned.)

--Mark


Not sure if there are some common tools (other tools) with which yocto
integrates nicely.

 

Any pointers will be appreciated,

 

--Hernan

 

PS> We are in morty branch currently.