inconsistent pages out there for setting up your yocto dev host


Robert P. J. Day
 

i've noticed there are various web pages purporting to explain how
to set up a proper OE/yocto development host, but they give what is
pretty clearly contradictory information.

as one example, there's this page on getting started with OE:

http://www.openembedded.org/wiki/Getting_started

that claims to steal from a number of sources including the yocto QS
guide, but look at the packages one is instructed to install on that
page, particularly under fedora: python, perl, git, and so on.

as i read it, the sanity.bbclass and ASSUME_PROVIDED will dictate
what needs to be there and what will be used if it's installed
natively, no? it certainly seems that that wiki page is insructing
the developer to install a lot of software that OE will handle
automatically, no?

rday

--

========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================


Paul Eggleton
 

On Wednesday 21 November 2012 17:23:56 Robert P. J. Day wrote:
i've noticed there are various web pages purporting to explain how
to set up a proper OE/yocto development host, but they give what is
pretty clearly contradictory information.

as one example, there's this page on getting started with OE:

http://www.openembedded.org/wiki/Getting_started
I produced this page recently; FWIW I also came up with the pared-down package
lists that went into the Quick Start Guide which this page borrows. How is
this contradictory if it uses the same information?

that claims to steal from a number of sources including the yocto QS
guide, but look at the packages one is instructed to install on that
page, particularly under fedora: python, perl, git, and so on.
So I don't recall exactly how perl got on that list; but python and git are
absolutely required on the host. That's why they're in ASSUME_PROVIDED.

as i read it, the sanity.bbclass and ASSUME_PROVIDED will dictate
what needs to be there and what will be used if it's installed
natively, no? it certainly seems that that wiki page is insructing
the developer to install a lot of software that OE will handle
automatically, no?
Er, no. Well, if by "handle automatically" you mean "error out when they are
not present" then that's not very helpful - it's much easier if people just
get a list of what they need to install up front.

Cheers,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre


Rifenbark, Scott M <scott.m.rifenbark@...>
 

-----Original Message-----
From: yocto-bounces@... [mailto:yocto-
bounces@...] On Behalf Of Paul Eggleton
Sent: Wednesday, November 21, 2012 2:51 PM
To: Robert P. J. Day
Cc: yocto@...
Subject: Re: [yocto] inconsistent pages out there for setting up your
yocto dev host

On Wednesday 21 November 2012 17:23:56 Robert P. J. Day wrote:
i've noticed there are various web pages purporting to explain how
to set up a proper OE/yocto development host, but they give what is
pretty clearly contradictory information.

as one example, there's this page on getting started with OE:

http://www.openembedded.org/wiki/Getting_started
I produced this page recently; FWIW I also came up with the pared-down
package
lists that went into the Quick Start Guide which this page borrows. How
is
this contradictory if it uses the same information?

that claims to steal from a number of sources including the yocto QS
guide, but look at the packages one is instructed to install on that
page, particularly under fedora: python, perl, git, and so on.
So I don't recall exactly how perl got on that list; but python and git
are
absolutely required on the host. That's why they're in ASSUME_PROVIDED.
Paul - Should I remove perl from the essentials list for fedora and centos?


as i read it, the sanity.bbclass and ASSUME_PROVIDED will dictate
what needs to be there and what will be used if it's installed
natively, no? it certainly seems that that wiki page is insructing
the developer to install a lot of software that OE will handle
automatically, no?
Er, no. Well, if by "handle automatically" you mean "error out when they
are
not present" then that's not very helpful - it's much easier if people
just
get a list of what they need to install up front.

Cheers,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre
_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto


Paul Eggleton
 

On Wednesday 21 November 2012 23:01:22 Rifenbark, Scott M wrote:
-----Original Message-----
From: yocto-bounces@... [mailto:yocto-
bounces@...] On Behalf Of Paul Eggleton
Sent: Wednesday, November 21, 2012 2:51 PM
To: Robert P. J. Day
Cc: yocto@...
Subject: Re: [yocto] inconsistent pages out there for setting up your
yocto dev host

On Wednesday 21 November 2012 17:23:56 Robert P. J. Day wrote:
i've noticed there are various web pages purporting to explain how

to set up a proper OE/yocto development host, but they give what is
pretty clearly contradictory information.

as one example, there's this page on getting started with OE:
http://www.openembedded.org/wiki/Getting_started
I produced this page recently; FWIW I also came up with the pared-down
package
lists that went into the Quick Start Guide which this page borrows. How
is
this contradictory if it uses the same information?

that claims to steal from a number of sources including the yocto QS
guide, but look at the packages one is instructed to install on that
page, particularly under fedora: python, perl, git, and so on.
So I don't recall exactly how perl got on that list; but python and git
are
absolutely required on the host. That's why they're in ASSUME_PROVIDED.
Paul - Should I remove perl from the essentials list for fedora and centos?
Let's leave it there for the moment - you can't have git installed on either
distro without perl anyway so it's not going to make too much difference.

Cheers,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre


Robert P. J. Day
 

On Wed, 21 Nov 2012, Paul Eggleton wrote:

On Wednesday 21 November 2012 17:23:56 Robert P. J. Day wrote:
i've noticed there are various web pages purporting to explain how
to set up a proper OE/yocto development host, but they give what is
pretty clearly contradictory information.

as one example, there's this page on getting started with OE:

http://www.openembedded.org/wiki/Getting_started
I produced this page recently; FWIW I also came up with the
pared-down package lists that went into the Quick Start Guide which
this page borrows. How is this contradictory if it uses the same
information?

that claims to steal from a number of sources including the yocto
QS guide, but look at the packages one is instructed to install on
that page, particularly under fedora: python, perl, git, and so
on.
So I don't recall exactly how perl got on that list; but python and
git are absolutely required on the host. That's why they're in
ASSUME_PROVIDED.

as i read it, the sanity.bbclass and ASSUME_PROVIDED will
dictate what needs to be there and what will be used if it's
installed natively, no? it certainly seems that that wiki page is
insructing the developer to install a lot of software that OE will
handle automatically, no?
Er, no. Well, if by "handle automatically" you mean "error out when
they are not present" then that's not very helpful - it's much
easier if people just get a list of what they need to install up
front.
at the risk of asking a dumb question, let me make sure i understand
the different categories of software.

first, there's what is absolutely *required* to be installed on the
development system already before doing any oe/yocto work. those
would the packages/utilities that are listed on the wiki page as "you
must install this", and that's what's represented in the
sanity.bbclass. is that about right?

on the other hand, there's what's listed in ASSUME_PROVIDED, which
represents software that, if it *is* natively installed, will be used;
otherwise, oe/yocto will download and build it as part of the build
process. correct? or have i misunderstood the distinction here?

rday

--

========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================


Paul Eggleton
 

On Wednesday 21 November 2012 20:19:05 Robert P. J. Day wrote:
On Wed, 21 Nov 2012, Paul Eggleton wrote:
On Wednesday 21 November 2012 17:23:56 Robert P. J. Day wrote:
as i read it, the sanity.bbclass and ASSUME_PROVIDED will
dictate what needs to be there and what will be used if it's
installed natively, no? it certainly seems that that wiki page is
insructing the developer to install a lot of software that OE will
handle automatically, no?
Er, no. Well, if by "handle automatically" you mean "error out when
they are not present" then that's not very helpful - it's much
easier if people just get a list of what they need to install up
front.
at the risk of asking a dumb question, let me make sure i understand
the different categories of software.

first, there's what is absolutely *required* to be installed on the
development system already before doing any oe/yocto work. those
would the packages/utilities that are listed on the wiki page as "you
must install this", and that's what's represented in the
sanity.bbclass. is that about right?

on the other hand, there's what's listed in ASSUME_PROVIDED, which
represents software that, if it *is* natively installed, will be used;
otherwise, oe/yocto will download and build it as part of the build
process. correct? or have i misunderstood the distinction here?
I think you may have. All ASSUME_PROVIDED does is tell bitbake that
dependencies on the things within it should be considered to be satisfied -
i.e. "assume these things are provided". If you add something to
ASSUME_PROVIDED that is genuinely needed by the build process and it isn't
actually provided by the host system, then you will get build failures or at
the very least undesirable changes in build behaviour.

Cheers,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre


Robert P. J. Day
 

On Thu, 22 Nov 2012, Paul Eggleton wrote:

On Wednesday 21 November 2012 20:19:05 Robert P. J. Day wrote:
On Wed, 21 Nov 2012, Paul Eggleton wrote:
On Wednesday 21 November 2012 17:23:56 Robert P. J. Day wrote:
as i read it, the sanity.bbclass and ASSUME_PROVIDED will
dictate what needs to be there and what will be used if it's
installed natively, no? it certainly seems that that wiki
page is insructing the developer to install a lot of software
that OE will handle automatically, no?
Er, no. Well, if by "handle automatically" you mean "error out
when they are not present" then that's not very helpful - it's
much easier if people just get a list of what they need to
install up front.
at the risk of asking a dumb question, let me make sure i
understand the different categories of software.

first, there's what is absolutely *required* to be installed on
the development system already before doing any oe/yocto work.
those would the packages/utilities that are listed on the wiki
page as "you must install this", and that's what's represented in
the sanity.bbclass. is that about right?

on the other hand, there's what's listed in ASSUME_PROVIDED,
which represents software that, if it *is* natively installed,
will be used; otherwise, oe/yocto will download and build it as
part of the build process. correct? or have i misunderstood the
distinction here?
I think you may have. All ASSUME_PROVIDED does is tell bitbake that
dependencies on the things within it should be considered to be
satisfied - i.e. "assume these things are provided". If you add
something to ASSUME_PROVIDED that is genuinely needed by the build
process and it isn't actually provided by the host system, then you
will get build failures or at the very least undesirable changes in
build behaviour.
ok, after a good night's sleep, i re-read what i wrote above and i
sound like an idiot so let me try again.

first, there's sanity.bbclass, which contains:

SANITY_REQUIRED_UTILITIES ?= "patch diffstat makeinfo git bzip2 tar gzip gawk chrpath wget cpio"

which represents the utilities that *must* already exist before
bitbake will offer to do anything. yes, it's obvious, but i just
confirmed that by removing "diffstat", whereupon i got:

ERROR: OE-core's config sanity checker detected a potential misconfiguration.
Either fix the cause of this error or at your own risk disable the checker (see sanity.conf).
Following is the list of potential problems / advisories:

Please install the following missing utilities: diffstat

so that's good, it's what i expected. it also suggests that, in
terms of minimal software, it's valid to just point people at that
variable in sanity.bbclass -- technically, those utilities are the
ones developers should be ordered to install beforehand. so far, so
good.

now, regarding ASSUME_PROVIDED in bitbake.conf, as you say, that
represents any dependencies assumed to be satisfied for further
processing, and it's not surprising that those two lists are closely
related -- if, for example, the sanity checker requires bzip2 (as it
does), it's only natural that ASSUME_PROVIDED would contain
"bzip2-native." and so on, and so on. and now, a few questions that
probably have obvious answers but i'll ask them, anyway.

first, is ASSUME_PROVIDED technically completely superfluous? it's
clearly meant to speed up processing, but could one (if one wanted)
unset it and have the processing still work (albeit more slowly, of
course)?

next, what about this in bitbake.conf, right after the setting of
ASSUME_PROVIDED?

# gzip-native should be listed above?

should it? seems to make sense -- why did someone feel the need to
leave a comment asking that question?

next, while ASSUME_PROVIDED contains "grep-native", the sanity check
doesn't list grep. should it? it just seems natural that anything
you assume to be provided should be listed as an installation
requirement first.

finally (and because i don't read python all that well, i can't
check the code as quickly as i would like), how would one add
subversion to one or both of the above? after all, the utility name
is "svn" but the package name (used by recipes) is "subversion".
(sure seems like subversion would be a good requirement to avoid
building it.)

ok, i'm done.

rday

--

========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================


Paul Eggleton
 

On Thursday 22 November 2012 06:54:33 Robert P. J. Day wrote:
On Thu, 22 Nov 2012, Paul Eggleton wrote:
On Wednesday 21 November 2012 20:19:05 Robert P. J. Day wrote:
On Wed, 21 Nov 2012, Paul Eggleton wrote:
On Wednesday 21 November 2012 17:23:56 Robert P. J. Day wrote:
as i read it, the sanity.bbclass and ASSUME_PROVIDED will
dictate what needs to be there and what will be used if it's
installed natively, no? it certainly seems that that wiki
page is insructing the developer to install a lot of software
that OE will handle automatically, no?
Er, no. Well, if by "handle automatically" you mean "error out
when they are not present" then that's not very helpful - it's
much easier if people just get a list of what they need to
install up front.
at the risk of asking a dumb question, let me make sure i

understand the different categories of software.

first, there's what is absolutely *required* to be installed on

the development system already before doing any oe/yocto work.
those would the packages/utilities that are listed on the wiki
page as "you must install this", and that's what's represented in
the sanity.bbclass. is that about right?

on the other hand, there's what's listed in ASSUME_PROVIDED,

which represents software that, if it *is* natively installed,
will be used; otherwise, oe/yocto will download and build it as
part of the build process. correct? or have i misunderstood the
distinction here?
I think you may have. All ASSUME_PROVIDED does is tell bitbake that
dependencies on the things within it should be considered to be
satisfied - i.e. "assume these things are provided". If you add
something to ASSUME_PROVIDED that is genuinely needed by the build
process and it isn't actually provided by the host system, then you
will get build failures or at the very least undesirable changes in
build behaviour.
ok, after a good night's sleep, i re-read what i wrote above and i
sound like an idiot so let me try again.

first, there's sanity.bbclass, which contains:

SANITY_REQUIRED_UTILITIES ?= "patch diffstat makeinfo git bzip2 tar gzip
gawk chrpath wget cpio"

which represents the utilities that *must* already exist before
bitbake will offer to do anything. yes, it's obvious, but i just
confirmed that by removing "diffstat", whereupon i got:

ERROR: OE-core's config sanity checker detected a potential
misconfiguration. Either fix the cause of this error or at your own risk
disable the checker (see sanity.conf). Following is the list of potential
problems / advisories:

Please install the following missing utilities: diffstat

so that's good, it's what i expected. it also suggests that, in
terms of minimal software, it's valid to just point people at that
variable in sanity.bbclass -- technically, those utilities are the
ones developers should be ordered to install beforehand. so far, so
good.

now, regarding ASSUME_PROVIDED in bitbake.conf, as you say, that
represents any dependencies assumed to be satisfied for further
processing, and it's not surprising that those two lists are closely
related -- if, for example, the sanity checker requires bzip2 (as it
does), it's only natural that ASSUME_PROVIDED would contain
"bzip2-native." and so on, and so on. and now, a few questions that
probably have obvious answers but i'll ask them, anyway.

first, is ASSUME_PROVIDED technically completely superfluous? it's
clearly meant to speed up processing, but could one (if one wanted)
unset it and have the processing still work (albeit more slowly, of
course)?
If you like building things that you don't have to, sure, but I wouldn't
equate that to it being superfluous. Note that if you do change this variable
you are deviating from what has been tested, so you do so at your own risk.

next, what about this in bitbake.conf, right after the setting of
ASSUME_PROVIDED?

# gzip-native should be listed above?

should it? seems to make sense -- why did someone feel the need to
leave a comment asking that question?
Not sure. We do now build pigz-native instead of using the system gzip though,
that might have something to do with it not being in the list.

next, while ASSUME_PROVIDED contains "grep-native", the sanity check
doesn't list grep. should it? it just seems natural that anything
you assume to be provided should be listed as an installation
requirement first.
Probably yes.

finally (and because i don't read python all that well, i can't
check the code as quickly as i would like), how would one add
subversion to one or both of the above? after all, the utility name
is "svn" but the package name (used by recipes) is "subversion".
(sure seems like subversion would be a good requirement to avoid
building it.)
The important thing to remember is SANITY_REQUIRED_UTILITIES is a list of
commands, and ASSUME_PROVIDED is a list of targets that the build system
understands; items in each must be specified accordingly. For subversion the
command is "svn" and the target is "subversion-native". Specifically in the
case of subversion, if I recall corectly we used to list it there but a few
versions ago we found that some people's installed versions of subversion were
too old to handle the new repository format that upstream projects have
upgraded to, so we had little choice but to build it ourselves.

Cheers,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre


Robert P. J. Day
 

On Thu, 22 Nov 2012, Paul Eggleton wrote:

On Thursday 22 November 2012 06:54:33 Robert P. J. Day wrote:
first, is ASSUME_PROVIDED technically completely superfluous?
it's clearly meant to speed up processing, but could one (if one
wanted) unset it and have the processing still work (albeit more
slowly, of course)?
If you like building things that you don't have to, sure, but I
wouldn't equate that to it being superfluous. Note that if you do
change this variable you are deviating from what has been tested, so
you do so at your own risk.
wait, this point interests me so indulge me for a minute.
obviously, there is a minimum collection of software one must install
before starting to use OE/yocto -- at the very least, say, the
fetching software -- so there will always be a page instructing a
developer to install that minimum.

afterwards, you can use ASSUME_PROVIDED to take advantage of that
software, correct? and that will certainly speed things up. but in
what way does that represent a "tested" configuration?

if you're on an allegedly supported distro, and you do an upgrade,
it's entirely possible that one of those bits of software gets
upgraded in a way that breaks OE/yocto, and by using ASSUME_PROVIDED,
you'll automatically start using that newer utility. or is the list
of supported distros specifically only those installed out of the box
and never updated? if that's the case, then, yes, i agree with your
position.

i'm not trying to be maddeningly pedantic, i am merely succeeding.

rday


--

========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================


Paul Eggleton
 

On Friday 23 November 2012 09:31:39 Robert P. J. Day wrote:
On Thu, 22 Nov 2012, Paul Eggleton wrote:
On Thursday 22 November 2012 06:54:33 Robert P. J. Day wrote:
first, is ASSUME_PROVIDED technically completely superfluous?

it's clearly meant to speed up processing, but could one (if one
wanted) unset it and have the processing still work (albeit more
slowly, of course)?
If you like building things that you don't have to, sure, but I
wouldn't equate that to it being superfluous. Note that if you do
change this variable you are deviating from what has been tested, so
you do so at your own risk.
wait, this point interests me so indulge me for a minute.
obviously, there is a minimum collection of software one must install
before starting to use OE/yocto -- at the very least, say, the
fetching software -- so there will always be a page instructing a
developer to install that minimum.

afterwards, you can use ASSUME_PROVIDED to take advantage of that
software, correct? and that will certainly speed things up. but in
what way does that represent a "tested" configuration?
Well, we don't test with any other value of ASSUME_PROVIDED. If you add tools
to ASSUME_PROVIDED that we would have otherwise built so that the host tools
are used, we haven't tested the build with those host tools. Equally if you
remove items from the default value of ASSUME_PROVIDED, we don't necessarily
fully test the system with native tools that we don't normally build (although
that's less likely to be an issue than the other case).

I'm not saying you can't change the value of ASSUME_PROVIDED - just that if
you do and the build breaks, you get to keep both pieces :)

if you're on an allegedly supported distro, and you do an upgrade,
it's entirely possible that one of those bits of software gets
upgraded in a way that breaks OE/yocto, and by using ASSUME_PROVIDED,
you'll automatically start using that newer utility. or is the list
of supported distros specifically only those installed out of the box
and never updated? if that's the case, then, yes, i agree with your
position.
That's why we test specific versions of distributions, and with the Poky distro
config we warn the user if the distro/distro version is untested. It's rare
that you get that kind of breakage between major versions; if there were then
other applications are likely to be affected so it would be considered a
regression in that distro.

Cheers,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre


Robert P. J. Day
 

On Fri, 23 Nov 2012, Paul Eggleton wrote:

Well, we don't test with any other value of ASSUME_PROVIDED. If you
add tools to ASSUME_PROVIDED that we would have otherwise built so
that the host tools are used, we haven't tested the build with those
host tools. Equally if you remove items from the default value of
ASSUME_PROVIDED, we don't necessarily fully test the system with
native tools that we don't normally build (although that's less
likely to be an issue than the other case).

I'm not saying you can't change the value of ASSUME_PROVIDED - just
that if you do and the build breaks, you get to keep both pieces :)
completely understood. for the sake of classroom work, i'm going to
experiment with adding extra entries to speed things up, and watch if
anything breaks. my first choice would be to add subversion to
ASSUME_PROVIDED -- that seems fairly safe, any moderately current
version of subversion should work.

thanks.

rday

--

========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================


Wolfgang Denk <wd@...>
 

Dear Paul,

In message <3038126.PEqOPPdATB@helios> you wrote:

if you're on an allegedly supported distro, and you do an upgrade,
it's entirely possible that one of those bits of software gets
upgraded in a way that breaks OE/yocto, and by using ASSUME_PROVIDED,
you'll automatically start using that newer utility. or is the list
of supported distros specifically only those installed out of the box
and never updated? if that's the case, then, yes, i agree with your
position.
That's why we test specific versions of distributions, and with the Poky distro
config we warn the user if the distro/distro version is untested. It's rare
that you get that kind of breakage between major versions; if there were then
other applications are likely to be affected so it would be considered a
regression in that distro.
But of course there have always been, and will always be, such
regressions. Just look at Fedora 17 (one of the supported versions of
distributions), if one of the updates pulls in git version 1.7.11.7
and suddenly all git downloads using HTTP protocol will fail for you
(due to bug 865692, see https://bugzilla.redhat.com/show_bug.cgi?id=865692)

Actually it may make a lot of sense for certain configurations to
bootstrap more tools from a known version of the sources.

Best regards,

Wolfgang Denk

--
DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@...
Beware of the Turing Tar-pit in which everything is possible but
nothing of interest is easy.