For my autobuilder, I am setting up a local mirror to hold sources for packages that have been unreliable for download. For my day-to-day work, I have setup the PREMIRRORS in my local.conf as suggested in the latest Yocto manual and it has been working fine. I am trying to get my autobuilder to use the PREMIRRORS and I am having issues since the autobuilder does not modify local.conf. I have one package that at this time is only available at our local mirror, so testing has at least been easy.

I have tried adding the PREMIRRORS to the BuildSettings section of autobuilder.conf as follows:
PREMIRRORS = 'git://.*/.* http://our/local/mirror/path/mirror/sources/ \n \
ftp://.*/.* http://our/local/mirror/path /mirror/sources/ \n \
http://.*/.* http://our/local/mirror/path/mirror/sources/ \n \
https://.*/.* http://our/local/mirror/path/mirror/sources/ \n'

My settings showed up on the initial list of environment variables but the system but the build did not fetch source from our local mirror. I then tried adding a PREMIRRORS string in the CreateAutoConf step for a build as follows:

'PREMIRRORS' : ['git://.*/.* http://our/local/mirror/path/mirror/sources/ \n',
'ftp://.*/.* http://our/local/mirror/path /mirror/sources/ \n',
'http://.*/.* http://our/local/mirror/path/mirror/sources/ \n',
'https://.*/.* http://our/local/mirror/path/mirror/sources/ \n']

But PREMIRRORS remained empty in the generated auto.conf. I have not pulled in updates for autobuilder for a few months, but I don't see any changes in the logs that look like are related to the PREMIRRORS variable.

Is there a way to add PREMIRRORS to a build's configuration? If so, how do you do it?

Thanks,
Bryan

All,

After looking through the autobuilder code, I don't see anywhere in which PREMIRRORS can be set and used. I see in CreateAutoConf.py that PREMIRRORS is always set to "". I don't think it'd be that difficult to add PREMIRRORS as a CreateAutoConf parameter for the buildsets. I can work on the change and submit the patch. But before I start I have a few questions:

1. I plan on implementing the PREMIRRORS parameter as an array, similar to 'layerdirs' for CreateBBLayersConf. For example:

{'CreateAutoConf' : {'PREMIRRORS' : ['git://.*/.* http://our/local/mirror/path/mirror/sources/ ',
'ftp://.*/.* http://our/local/mirror/path /mirror/sources/ ',
'http://.*/.* http://our/local/mirror/path/mirror/sources/ ',
'https://.*/.* http://our/local/mirror/path/mirror/sources/ ']}

would add the following to auto.conf:

PREMIRRORS = 'git://.*/.* http://our/local/mirror/path/mirror/sources/ \
ftp://.*/.* http://our/local/mirror/path /mirror/sources/ \
http://.*/.* http://our/local/mirror/path/mirror/sources/ \
https://.*/.* http://our/local/mirror/path/mirror/sources/ '

Does this sound reasonable, or would a different parameter format be preferred?

2. Should the final text in auto.conf be 'PREMIRRORS =' or 'PREMIRRORS +='? Since there is no other way to add to PREMIRRORS that I can see, I don't know if it matters or not.

3. I see there is now an html document for autobuilder. Should any additions still be documented in README-NEW-AUTOBUILDER or is that file being abandoned?

Thanks,
Bryan

All,

I'm getting close to implementing this feature, but I'm having some issues getting auto.conf formatted correctly. See below.

2014-11-19 15:12 GMT+00:00 Bryan Evenson <bevenson@...>:

All,

I'm getting close to implementing this feature, but I'm having some issues getting auto.conf formatted correctly. See below.

-----Original Message-----
From: Bryan Evenson
Sent: Monday, November 17, 2014 9:08 AM
To: Bryan Evenson; yocto@...; elizabeth.flanagan@...
Subject: RE: autobuilder: How to set PREMIRRORS?

All,

After looking through the autobuilder code, I don't see anywhere in which
PREMIRRORS can be set and used. I see in CreateAutoConf.py that
PREMIRRORS is always set to "". I don't think it'd be that difficult to add
PREMIRRORS as a CreateAutoConf parameter for the buildsets. I can work
on the change and submit the patch. But before I start I have a few
questions:

1. I plan on implementing the PREMIRRORS parameter as an array, similar to
'layerdirs' for CreateBBLayersConf. For example:

{'CreateAutoConf' : {'PREMIRRORS' : ['git://.*/.*
http://our/local/mirror/path/mirror/sources/ ',
'ftp://.*/.* http://our/local/mirror/path /mirror/sources/ ',
'http://.*/.* http://our/local/mirror/path/mirror/sources/ ',
'https://.*/.* http://our/local/mirror/path/mirror/sources/ ']}

would add the following to auto.conf:

PREMIRRORS = 'git://.*/.* http://our/local/mirror/path/mirror/sources/ \
ftp://.*/.* http://our/local/mirror/path /mirror/sources/ \
http://.*/.* http://our/local/mirror/path/mirror/sources/ \
https://.*/.* http://our/local/mirror/path/mirror/sources/ '

Does this sound reasonable, or would a different parameter format be
preferred?

I would actually do this similar to how we have DEVKERNEL_MUT_REPO.

PREMIRRORS = ['git://.*/.*
http://our/local/mirror/path/mirror/sources/', 'ftp://.*/.*
http://our/local/mirror/path /mirror/sources/', 'http://.*/.*
http://our/local/mirror/path/mirror/sources/', 'https://.*/.*
http://our/local/mirror/path/mirror/sources/']

I am trying to add a single PREMIRROR using the syntax I specified above. At this time it is getting inserted in auto.conf. However, the system is going into an infinite loop building the PREMIRROR path. The added content to auto.conf at this time looks like:

PREMIRRORS = " \
http://.*.* http://server.repo.local/mirror/sources/ \n \
"

And from looking at log.do_fetch for the package that attempts to use the PREMIRROR, the log is filled with "server.repo.localserver.repo.localserver.repo.local" repeating. Any tips on how PREMIRRORS should be built?

Here's what I have so far in CreateAutoConf.py:

self.PREMIRRORS=""
fout = fout + 'PREMIRRORS = " \ \n'
if list(self.PREMIRRORS):
for premirror in self.PREMIRRORS:
fout = fout + premirror + ' \\\\n \ \n'
fout = fout + '" \n'

Any tips on how to change things so PREMIRRORS is correctly parsed?

#make sure you add
#
#export PREMIRRORS=""
# to yocto-autobuilder-setup

if os.environ.get('PREMIRRORS') is not None:
premirrors=ast.liternal_eval(os.environ.get('PREMIRRORS').encode('utf-8')
fout =+ 'PREMIRRORS = "\ \n'
for mirror in premirrors:.....

I think this should do what you're looking for.

Thanks,
Bryan

--
Elizabeth Flanagan
Yocto Project
Build and Release

On Tuesday 16 December 2014 20:14:23 Flanagan, Elizabeth wrote:

2014-11-19 15:12 GMT+00:00 Bryan Evenson <bevenson@...>:

All,

I'm getting close to implementing this feature, but I'm having some issues
getting auto.conf formatted correctly. See below.>

-----Original Message-----
From: Bryan Evenson
Sent: Monday, November 17, 2014 9:08 AM
To: Bryan Evenson; yocto@...; elizabeth.flanagan@...
Subject: RE: autobuilder: How to set PREMIRRORS?

All,

After looking through the autobuilder code, I don't see anywhere in which
PREMIRRORS can be set and used. I see in CreateAutoConf.py that
PREMIRRORS is always set to "". I don't think it'd be that difficult to
add PREMIRRORS as a CreateAutoConf parameter for the buildsets. I can
work on the change and submit the patch. But before I start I have a
few questions:

1. I plan on implementing the PREMIRRORS parameter as an array, similar
to
'layerdirs' for CreateBBLayersConf. For example:

{'CreateAutoConf' : {'PREMIRRORS' : ['git://.*/.*
http://our/local/mirror/path/mirror/sources/ ',

'ftp://.*/.* http://our/local/mirror/path /mirror/sources/ ',
'http://.*/.* http://our/local/mirror/path/mirror/sources/ ',
'https://.*/.* http://our/local/mirror/path/mirror/sources/ ']}

would add the following to auto.conf:
PREMIRRORS = 'git://.*/.* http://our/local/mirror/path/mirror/sources/
\

ftp://.*/.* http://our/local/mirror/path /mirror/sources/ \
http://.*/.* http://our/local/mirror/path/mirror/sources/ \
https://.*/.* http://our/local/mirror/path/mirror/sources/ '

Does this sound reasonable, or would a different parameter format be
preferred?

I would actually do this similar to how we have DEVKERNEL_MUT_REPO.

PREMIRRORS = ['git://.*/.*
http://our/local/mirror/path/mirror/sources/', 'ftp://.*/.*
http://our/local/mirror/path /mirror/sources/', 'http://.*/.*
http://our/local/mirror/path/mirror/sources/', 'https://.*/.*
http://our/local/mirror/path/mirror/sources/']

I don't think we support setting a variable to a list like this - does that
actually work?

Cheers,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre

On Thursday 18 December 2014 13:16:31 Paul Eggleton wrote:

On Tuesday 16 December 2014 20:14:23 Flanagan, Elizabeth wrote:

2014-11-19 15:12 GMT+00:00 Bryan Evenson <bevenson@...>:

All,

I'm getting close to implementing this feature, but I'm having some
issues
getting auto.conf formatted correctly. See below.>

-----Original Message-----
From: Bryan Evenson
Sent: Monday, November 17, 2014 9:08 AM
To: Bryan Evenson; yocto@...; elizabeth.flanagan@...
Subject: RE: autobuilder: How to set PREMIRRORS?

All,

After looking through the autobuilder code, I don't see anywhere in
which
PREMIRRORS can be set and used. I see in CreateAutoConf.py that
PREMIRRORS is always set to "". I don't think it'd be that difficult
to
add PREMIRRORS as a CreateAutoConf parameter for the buildsets. I can
work on the change and submit the patch. But before I start I have a
few questions:

1. I plan on implementing the PREMIRRORS parameter as an array, similar
to
'layerdirs' for CreateBBLayersConf. For example:

{'CreateAutoConf' : {'PREMIRRORS' : ['git://.*/.*
http://our/local/mirror/path/mirror/sources/ ',

'ftp://.*/.* http://our/local/mirror/path /mirror/sources/ ',
'http://.*/.* http://our/local/mirror/path/mirror/sources/ ',
'https://.*/.* http://our/local/mirror/path/mirror/sources/ ']}

would add the following to auto.conf:
PREMIRRORS = 'git://.*/.* http://our/local/mirror/path/mirror/sources/
\

ftp://.*/.* http://our/local/mirror/path /mirror/sources/ \
http://.*/.* http://our/local/mirror/path/mirror/sources/ \
https://.*/.* http://our/local/mirror/path/mirror/sources/ '

Does this sound reasonable, or would a different parameter format be
preferred?

I would actually do this similar to how we have DEVKERNEL_MUT_REPO.

PREMIRRORS = ['git://.*/.*
http://our/local/mirror/path/mirror/sources/', 'ftp://.*/.*
http://our/local/mirror/path /mirror/sources/', 'http://.*/.*
http://our/local/mirror/path/mirror/sources/', 'https://.*/.*
http://our/local/mirror/path/mirror/sources/']

I don't think we support setting a variable to a list like this - does that
actually work?

Oops, I missed the context was the autobuilder python code - ignore me ;)

Cheers,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre

Elizabeth,

Thanks for the tips. I think I have something close to working, just need to finish testing. I should have a patch for you the first week of January.

Thanks,
Bryan E.