Hi,

Recently, Paul, Ross, Richard and I had a video conference meeting where we had some initial discussion on how to satisfy YOCTO #2808 (https://bugzilla.yoctoproject.org/show_bug.cgi?id=2808), which calls for an illustration showing source and destination directories during a build using YP. This bug originated from a discussion Dave Stewart and I had a while back around an idea of a more detailed "flow diagram" that would go into greater detail than the now famous and ubiquitous "The Yocto Project Development Environment" illustration, which appears in the YP Quick Start (http://www.yoctoproject.org/docs/1.4/yocto-project-qs/yocto-project-qs.html) and has been used in many other places and in many other forms (some quite elaborate).

We can't really create a completely accurate, all-encompassing illustration that breaks apart the entire build process. It is not a static thing and it is quite complicated inside the BitBake process. We can, however, show where metadata comes from, how it is provided, what it defines, where and how source files are found, what processes occur, what output is produced, and where it ends up. We can also provide some sort of idea of how key BitBake and environment variables affect things along the way. The idea here is to dig deeper into this conceptual figure and root out the details to a level that would be useful to the user but not impossible to maintain or develop. This type of information can be communicated through a mix of several illustrations with supporting text.

This first meeting started with some detailed discussion of the configuration inputs for a typical YP build but soon migrated to discussing the bigger picture and possible ways to provide more information. It became obvious we were not going to dig the solution and all the needed information out in one short meeting. Consequently, I am sending out this email to help open up some discussion on the issue and to also solicit information for some basic blocks of information.

Two things are needed at this point: 1) a presentation solution for this new and more detailed information, and 2) a starting point on some of the information itself.

PRESENTATION SOLUTION

Here are some thoughts on how to present this information. There are disadvantages and advantages to each of these methods of which I will not list. I would like to see what people think about them:

* Manual - Create a section in the "Technical Details" Chapter of the YP Reference Manual that holds this information. The section would be pretty much self-contained and would consist of several illustrations that would stem from an overall figure that is similar to the figure we now have in the YP Quick Start. However, we would use a drill-down strategy that would progressively reveal more detail through subsequent figures. This method is similar to how hardware devices used to be documented where functional blocks would be connected and described in one area and then subsequent areas would elaborate more deeply on a particular block. Linking within the manual could connect up the various functional blocks (inputs, outputs, and tasks) that comprise the overall flow.

* Manual / Website Mix - Devise a mix between the YP Reference Manual and some pages in the YP Website that provide the information. Create a section in the "Technical details" Chapter of the YP Reference Manual that covers this information at a high level. For example, the overall flow of the system with its "big-block" inputs and outputs and tasks could be discussed at some length. Links in the text could go to the YP website where more detail would be revealed. This strategy effectively splits the content into "overview" and "details" between the manual and website, respectively.

* Website - With this strategy, everything is pretty much in the YP website. This area would exist in a stand-alone fashion. However, you could link to the website from within the existing YP documentation set from existing areas that deal with the build process. Several exit points from within the manual set already exist. We would obviously add a primary one as well that would likely originate from the YP Reference Manual's "Technical Details" Chapter.

SOLICITATION FOR INFORMATION

We can get started now on this by starting to define details for some obvious points or large areas of the flow. What would be nice to get would be some graphical breakdowns of these areas of concern:

* User-defined layers
* YP provided layers
* User configuration
* Machine Configuration
* Policy Configuration
* Patches
* YP provided recipes
* User provided source code
* YP provided source code
* SCMs
* Generated images
* Generated SDKs/toolchains
* Package Output
* Source fetching process
* Patch application process
* Configuration process (fragments, etc.)
* Key variable use and effects
* User-initiated commands along the way

Much of this list is directly from our existing "The Yocto Project Development Environment" illustration.

Thanks,
Scott R.



Scott Rifenbark
Intel Corporation
Yocto Project Documentation
503.712.2702
503.341.0418 (cell)

On 13-06-20 04:40 AM, Rifenbark, Scott M wrote:

Hi,

Recently, Paul, Ross, Richard and I had a video conference meeting where we had some initial discussion on how to satisfy YOCTO #2808 (https://bugzilla.yoctoproject.org/show_bug.cgi?id=2808), which calls for an illustration showing source and destination directories during a build using YP. This bug originated from a discussion Dave Stewart and I had a while back around an idea of a more detailed "flow diagram" that would go into greater detail than the now famous and ubiquitous "The Yocto Project Development Environment" illustration, which appears in the YP Quick Start (http://www.yoctoproject.org/docs/1.4/yocto-project-qs/yocto-project-qs.html) and has been used in many other places and in many other forms (some quite elaborate).

We can't really create a completely accurate, all-encompassing illustration that breaks apart the entire build process. It is not a static thing and it is quite complicated inside the BitBake process. We can, however, show where metadata comes from, how it is provided, what it defines, where and how source files are found, what processes occur, what output is produced, and where it ends up. We can also provide some sort of idea of how key BitBake and environment variables affect things along the way. The idea here is to dig deeper into this conceptual figure and root out the details to a level that would be useful to the user but not impossible to maintain or develop. This type of information can be communicated through a mix of several illustrations with supporting text.

This first meeting started with some detailed discussion of the configuration inputs for a typical YP build but soon migrated to discussing the bigger picture and possible ways to provide more information. It became obvious we were not going to dig the solution and all the needed information out in one short meeting. Consequently, I am sending out this email to help open up some discussion on the issue and to also solicit information for some basic blocks of information.

Two things are needed at this point: 1) a presentation solution for this new and more detailed information, and 2) a starting point on some of the information itself.

PRESENTATION SOLUTION

Here are some thoughts on how to present this information. There are disadvantages and advantages to each of these methods of which I will not list. I would like to see what people think about them:

* Manual - Create a section in the "Technical Details" Chapter of the YP Reference Manual that holds this information. The section would be pretty much self-contained and would consist of several illustrations that would stem from an overall figure that is similar to the figure we now have in the YP Quick Start. However, we would use a drill-down strategy that would progressively reveal more detail through subsequent figures. This method is similar to how hardware devices used to be documented where functional blocks would be connected and described in one area and then subsequent areas would elaborate more deeply on a particular block. Linking within the manual could connect up the various functional blocks (inputs, outputs, and tasks) that comprise the overall flow.

* Manual / Website Mix - Devise a mix between the YP Reference Manual and some pages in the YP Website that provide the information. Create a section in the "Technical details" Chapter of the YP Reference Manual that covers this information at a high level. For example, the overall flow of the system with its "big-block" inputs and outputs and tasks could be discussed at some length. Links in the text could go to the YP website where more detail would be revealed. This strategy effectively splits the content into "overview" and "details" between the manual and website, respectively.

* Website - With this strategy, everything is pretty much in the YP website. This area would exist in a stand-alone fashion. However, you could link to the website from within the existing YP documentation set from existing areas that deal with the build process. Several exit points from within the manual set already exist. We would obviously add a primary one as well that would likely originate from the YP Reference Manual's "Technical Details" Chapter.

SOLICITATION FOR INFORMATION

We can get started now on this by starting to define details for some obvious points or large areas of the flow. What would be nice to get would be some graphical breakdowns of these areas of concern:

* User-defined layers
* YP provided layers
* User configuration
* Machine Configuration
* Policy Configuration
* Patches
* YP provided recipes
* User provided source code
* YP provided source code
* SCMs
* Generated images
* Generated SDKs/toolchains
* Package Output
* Source fetching process
* Patch application process
* Configuration process (fragments, etc.)

Hi Scott,

Just so I'm clear .. are you looking for graphical breakdowns to be created and sent,
or information offered so graphical breakdowns can be created ?

The reason I ask, is that if there's any interest in the linux-yocto
(for lack of a better term) flow being described graphically, I'm
happy to offer up information, but my graphical skills are limited
to what you find in a typical hackers bag of ticks :)

Cheers,

Bruce

* Key variable use and effects
* User-initiated commands along the way

Much of this list is directly from our existing "The Yocto Project Development Environment" illustration.

Thanks,
Scott R.



Scott Rifenbark
Intel Corporation
Yocto Project Documentation
503.712.2702
503.341.0418 (cell)

_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto

On Thu, 2013-06-20 at 09:49 -0400, Bruce Ashfield wrote:

On 13-06-20 04:40 AM, Rifenbark, Scott M wrote:

Hi,

Recently, Paul, Ross, Richard and I had a video conference meeting where we had some initial discussion on how to satisfy YOCTO #2808 (https://bugzilla.yoctoproject.org/show_bug.cgi?id=2808), which calls for an illustration showing source and destination directories during a build using YP. This bug originated from a discussion Dave Stewart and I had a while back around an idea of a more detailed "flow diagram" that would go into greater detail than the now famous and ubiquitous "The Yocto Project Development Environment" illustration, which appears in the YP Quick Start (http://www.yoctoproject.org/docs/1.4/yocto-project-qs/yocto-project-qs.html) and has been used in many other places and in many other forms (some quite elaborate).

We can't really create a completely accurate, all-encompassing illustration that breaks apart the entire build process. It is not a static thing and it is quite complicated inside the BitBake process. We can, however, show where metadata comes from, how it is provided, what it defines, where and how source files are found, what processes occur, what output is produced, and where it ends up. We can also provide some sort of idea of how key BitBake and environment variables affect things along the way. The idea here is to dig deeper into this conceptual figure and root out the details to a level that would be useful to the user but not impossible to maintain or develop. This type of information can be communicated through a mix of several illustrations with supporting text.

This first meeting started with some detailed discussion of the configuration inputs for a typical YP build but soon migrated to discussing the bigger picture and possible ways to provide more information. It became obvious we were not going to dig the solution and all the needed information out in one short meeting. Consequently, I am sending out this email to help open up some discussion on the issue and to also solicit information for some basic blocks of information.

Two things are needed at this point: 1) a presentation solution for this new and more detailed information, and 2) a starting point on some of the information itself.

PRESENTATION SOLUTION

Here are some thoughts on how to present this information. There

are disadvantages and advantages to each of these methods of which I
will not list. I would like to see what people think about them:

* Manual - Create a section in the "Technical Details" Chapter of

the YP Reference Manual that holds this information. The section
would be pretty much self-contained and would consist of several
illustrations that would stem from an overall figure that is similar
to the figure we now have in the YP Quick Start. However, we would
use a drill-down strategy that would progressively reveal more detail
through subsequent figures. This method is similar to how hardware
devices used to be documented where functional blocks would be
connected and described in one area and then subsequent areas would
elaborate more deeply on a particular block. Linking within the
manual could connect up the various functional blocks (inputs,
outputs, and tasks) that comprise the overall flow.

* Manual / Website Mix - Devise a mix between the YP Reference

Manual and some pages in the YP Website that provide the information.
Create a section in the "Technical details" Chapter of the YP
Reference Manual that covers this information at a high level. For
example, the overall flow of the system with its "big-block" inputs
and outputs and tasks could be discussed at some length. Links in the
text could go to the YP website where more detail would be revealed.
This strategy effectively splits the content into "overview" and
"details" between the manual and website, respectively.

* Website - With this strategy, everything is pretty much in the YP

website. This area would exist in a stand-alone fashion. However,
you could link to the website from within the existing YP
documentation set from existing areas that deal with the build
process. Several exit points from within the manual set already
exist. We would obviously add a primary one as well that would likely
originate from the YP Reference Manual's "Technical Details" Chapter.

I'm wondering if a hybrid is possible here. Can we for example embed
image maps into the docbook so that if you hover over areas of the
image, you can then have links to the different sections?

SOLICITATION FOR INFORMATION

We can get started now on this by starting to define details for

some obvious points or large areas of the flow. What would be nice to
get would be some graphical breakdowns of these areas of concern:

* User-defined layers
* YP provided layers
* User configuration
* Machine Configuration
* Policy Configuration
* Patches
* YP provided recipes
* User provided source code
* YP provided source code
* SCMs
* Generated images
* Generated SDKs/toolchains
* Package Output
* Source fetching process
* Patch application process
* Configuration process (fragments, etc.)

Just so I'm clear .. are you looking for graphical breakdowns to be
created and sent,
or information offered so graphical breakdowns can be created ?

The reason I ask, is that if there's any interest in the linux-yocto
(for lack of a better term) flow being described graphically, I'm
happy to offer up information, but my graphical skills are limited
to what you find in a typical hackers bag of ticks :)

I suggested that Scott try and accumulate information for each topic
like the following:

Fetcher:

Code Areas: Bitbake Fetch Module (bitbake/lib/bb/fetch2, called from
classes/base.bbclass)
Tasks Covered: do_fetch/do_unpack
Key Variables: SRC_URI, checksums etc

Generated images:

Code Areas: classes/image*.bbclass classes/rootfs*.bbclass
Tasks Covered: do_rootfs
Key Variables: PACKAGE_INSTALL

along with a description about what the function

Cheers,

Bruce

* Key variable use and effects
* User-initiated commands along the way

Much of this list is directly from our existing "The Yocto Project

Development Environment" illustration.

Thanks,
Scott R.



Scott Rifenbark
Intel Corporation
Yocto Project Documentation
503.712.2702
503.341.0418 (cell)

_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto

al block does and when its used. The above would then link into the

class reference, the variable glossary and so on. So its less about
graphics and more about giving Scott the information to create a kind of
details index of some of these parts of the system like an expanded
table of contents.

I've suggested a good start would be picking a few areas (like the
above) and trying to create the info and maybe see what kind of diagram
would present itself from that. If successful, it could then be expanded
to each area. I'd certainly hope that linux-yocto would be one area
covered.

Cheers,

Richard

On 13-06-20 10:25 AM, Richard Purdie wrote:

On Thu, 2013-06-20 at 09:49 -0400, Bruce Ashfield wrote:

On 13-06-20 04:40 AM, Rifenbark, Scott M wrote:

Hi,

Recently, Paul, Ross, Richard and I had a video conference meeting where we had some initial discussion on how to satisfy YOCTO #2808 (https://bugzilla.yoctoproject.org/show_bug.cgi?id=2808), which calls for an illustration showing source and destination directories during a build using YP. This bug originated from a discussion Dave Stewart and I had a while back around an idea of a more detailed "flow diagram" that would go into greater detail than the now famous and ubiquitous "The Yocto Project Development Environment" illustration, which appears in the YP Quick Start (http://www.yoctoproject.org/docs/1.4/yocto-project-qs/yocto-project-qs.html) and has been used in many other places and in many other forms (some quite elaborate).

We can't really create a completely accurate, all-encompassing illustration that breaks apart the entire build process. It is not a static thing and it is quite complicated inside the BitBake process. We can, however, show where metadata comes from, how it is provided, what it defines, where and how source files are found, what processes occur, what output is produced, and where it ends up. We can also provide some sort of idea of how key BitBake and environment variables affect things along the way. The idea here is to dig deeper into this conceptual figure and root out the details to a level that would be useful to the user but not impossible to maintain or develop. This type of information can be communicated through a mix of several illustrations with supporting text.

This first meeting started with some detailed discussion of the configuration inputs for a typical YP build but soon migrated to discussing the bigger picture and possible ways to provide more information. It became obvious we were not going to dig the solution and all the needed information out in one short meeting. Consequently, I am sending out this email to help open up some discussion on the issue and to also solicit information for some basic blocks of information.

Two things are needed at this point: 1) a presentation solution for this new and more detailed information, and 2) a starting point on some of the information itself.

PRESENTATION SOLUTION

Here are some thoughts on how to present this information. There

are disadvantages and advantages to each of these methods of which I
will not list. I would like to see what people think about them:

* Manual - Create a section in the "Technical Details" Chapter of

the YP Reference Manual that holds this information. The section
would be pretty much self-contained and would consist of several
illustrations that would stem from an overall figure that is similar
to the figure we now have in the YP Quick Start. However, we would
use a drill-down strategy that would progressively reveal more detail
through subsequent figures. This method is similar to how hardware
devices used to be documented where functional blocks would be
connected and described in one area and then subsequent areas would
elaborate more deeply on a particular block. Linking within the
manual could connect up the various functional blocks (inputs,
outputs, and tasks) that comprise the overall flow.

* Manual / Website Mix - Devise a mix between the YP Reference

Manual and some pages in the YP Website that provide the information.
Create a section in the "Technical details" Chapter of the YP
Reference Manual that covers this information at a high level. For
example, the overall flow of the system with its "big-block" inputs
and outputs and tasks could be discussed at some length. Links in the
text could go to the YP website where more detail would be revealed.
This strategy effectively splits the content into "overview" and
"details" between the manual and website, respectively.

* Website - With this strategy, everything is pretty much in the YP

website. This area would exist in a stand-alone fashion. However,
you could link to the website from within the existing YP
documentation set from existing areas that deal with the build
process. Several exit points from within the manual set already
exist. We would obviously add a primary one as well that would likely
originate from the YP Reference Manual's "Technical Details" Chapter.

I'm wondering if a hybrid is possible here. Can we for example embed
image maps into the docbook so that if you hover over areas of the
image, you can then have links to the different sections?

SOLICITATION FOR INFORMATION

We can get started now on this by starting to define details for

some obvious points or large areas of the flow. What would be nice to
get would be some graphical breakdowns of these areas of concern:

* User-defined layers
* YP provided layers
* User configuration
* Machine Configuration
* Policy Configuration
* Patches
* YP provided recipes
* User provided source code
* YP provided source code
* SCMs
* Generated images
* Generated SDKs/toolchains
* Package Output
* Source fetching process
* Patch application process
* Configuration process (fragments, etc.)

Just so I'm clear .. are you looking for graphical breakdowns to be
created and sent,
or information offered so graphical breakdowns can be created ?

The reason I ask, is that if there's any interest in the linux-yocto
(for lack of a better term) flow being described graphically, I'm
happy to offer up information, but my graphical skills are limited
to what you find in a typical hackers bag of ticks :)

I suggested that Scott try and accumulate information for each topic
like the following:

Fetcher:

Code Areas: Bitbake Fetch Module (bitbake/lib/bb/fetch2, called from
classes/base.bbclass)
Tasks Covered: do_fetch/do_unpack
Key Variables: SRC_URI, checksums etc

Generated images:

Code Areas: classes/image*.bbclass classes/rootfs*.bbclass
Tasks Covered: do_rootfs
Key Variables: PACKAGE_INSTALL

along with a description about what the function

Aha. Great! Keeping things up to date will of course be the trick. I'll
be on standby if anything is needed from my front.

Cheers,

Bruce

Cheers,

Bruce

* Key variable use and effects
* User-initiated commands along the way

Much of this list is directly from our existing "The Yocto Project

Development Environment" illustration.

Thanks,
Scott R.



Scott Rifenbark
Intel Corporation
Yocto Project Documentation
503.712.2702
503.341.0418 (cell)

_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto

al block does and when its used. The above would then link into the

class reference, the variable glossary and so on. So its less about
graphics and more about giving Scott the information to create a kind of
details index of some of these parts of the system like an expanded
table of contents.

I've suggested a good start would be picking a few areas (like the
above) and trying to create the info and maybe see what kind of diagram
would present itself from that. If successful, it could then be expanded
to each area. I'd certainly hope that linux-yocto would be one area
covered.

Cheers,

Richard

On Thu, Jun 20, 2013 at 10:25 AM, Richard Purdie
<richard.purdie@...> wrote:

On Thu, 2013-06-20 at 09:49 -0400, Bruce Ashfield wrote:

On 13-06-20 04:40 AM, Rifenbark, Scott M wrote:

Hi,

Recently, Paul, Ross, Richard and I had a video conference meeting where we had some initial discussion on how to satisfy YOCTO #2808 (https://bugzilla.yoctoproject.org/show_bug.cgi?id=2808), which calls for an illustration showing source and destination directories during a build using YP. This bug originated from a discussion Dave Stewart and I had a while back around an idea of a more detailed "flow diagram" that would go into greater detail than the now famous and ubiquitous "The Yocto Project Development Environment" illustration, which appears in the YP Quick Start (http://www.yoctoproject.org/docs/1.4/yocto-project-qs/yocto-project-qs.html) and has been used in many other places and in many other forms (some quite elaborate).

We can't really create a completely accurate, all-encompassing illustration that breaks apart the entire build process. It is not a static thing and it is quite complicated inside the BitBake process. We can, however, show where metadata comes from, how it is provided, what it defines, where and how source files are found, what processes occur, what output is produced, and where it ends up. We can also provide some sort of idea of how key BitBake and environment variables affect things along the way. The idea here is to dig deeper into this conceptual figure and root out the details to a level that would be useful to the user but not impossible to maintain or develop. This type of information can be communicated through a mix of several illustrations with supporting text.

This first meeting started with some detailed discussion of the configuration inputs for a typical YP build but soon migrated to discussing the bigger picture and possible ways to provide more information. It became obvious we were not going to dig the solution and all the needed information out in one short meeting. Consequently, I am sending out this email to help open up some discussion on the issue and to also solicit information for some basic blocks of information.

Two things are needed at this point: 1) a presentation solution for this new and more detailed information, and 2) a starting point on some of the information itself.

PRESENTATION SOLUTION

Here are some thoughts on how to present this information. There

are disadvantages and advantages to each of these methods of which I
will not list. I would like to see what people think about them:

* Manual - Create a section in the "Technical Details" Chapter of

the YP Reference Manual that holds this information. The section
would be pretty much self-contained and would consist of several
illustrations that would stem from an overall figure that is similar
to the figure we now have in the YP Quick Start. However, we would
use a drill-down strategy that would progressively reveal more detail
through subsequent figures. This method is similar to how hardware
devices used to be documented where functional blocks would be
connected and described in one area and then subsequent areas would
elaborate more deeply on a particular block. Linking within the
manual could connect up the various functional blocks (inputs,
outputs, and tasks) that comprise the overall flow.

* Manual / Website Mix - Devise a mix between the YP Reference

Manual and some pages in the YP Website that provide the information.
Create a section in the "Technical details" Chapter of the YP
Reference Manual that covers this information at a high level. For
example, the overall flow of the system with its "big-block" inputs
and outputs and tasks could be discussed at some length. Links in the
text could go to the YP website where more detail would be revealed.
This strategy effectively splits the content into "overview" and
"details" between the manual and website, respectively.

* Website - With this strategy, everything is pretty much in the YP

website. This area would exist in a stand-alone fashion. However,
you could link to the website from within the existing YP
documentation set from existing areas that deal with the build
process. Several exit points from within the manual set already
exist. We would obviously add a primary one as well that would likely
originate from the YP Reference Manual's "Technical Details" Chapter.

I'm wondering if a hybrid is possible here. Can we for example embed
image maps into the docbook so that if you hover over areas of the
image, you can then have links to the different sections?

FWIW, docbook supports image maps via the mediaobject tag but they are
somewhat limited in their usefulness.

http://docbook.org/tdg5/en/html/mediaobject.html
http://www.sagehill.net/docbookxsl/Imagemaps.html

Scott, you may run into trouble when converting the docs between doc
types, in particular with scaling of images.

SOLICITATION FOR INFORMATION

We can get started now on this by starting to define details for

some obvious points or large areas of the flow. What would be nice to
get would be some graphical breakdowns of these areas of concern:

* User-defined layers
* YP provided layers
* User configuration
* Machine Configuration
* Policy Configuration
* Patches
* YP provided recipes
* User provided source code
* YP provided source code
* SCMs
* Generated images
* Generated SDKs/toolchains
* Package Output
* Source fetching process
* Patch application process
* Configuration process (fragments, etc.)

Just so I'm clear .. are you looking for graphical breakdowns to be
created and sent,
or information offered so graphical breakdowns can be created ?

The reason I ask, is that if there's any interest in the linux-yocto
(for lack of a better term) flow being described graphically, I'm
happy to offer up information, but my graphical skills are limited
to what you find in a typical hackers bag of ticks :)

I suggested that Scott try and accumulate information for each topic
like the following:

Fetcher:

Code Areas: Bitbake Fetch Module (bitbake/lib/bb/fetch2, called from
classes/base.bbclass)
Tasks Covered: do_fetch/do_unpack
Key Variables: SRC_URI, checksums etc

Generated images:

Code Areas: classes/image*.bbclass classes/rootfs*.bbclass
Tasks Covered: do_rootfs
Key Variables: PACKAGE_INSTALL

along with a description about what the function

Cheers,

Bruce

* Key variable use and effects
* User-initiated commands along the way

Much of this list is directly from our existing "The Yocto Project

Development Environment" illustration.

Thanks,
Scott R.



Scott Rifenbark
Intel Corporation
Yocto Project Documentation
503.712.2702
503.341.0418 (cell)

_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto

al block does and when its used. The above would then link into the

class reference, the variable glossary and so on. So its less about
graphics and more about giving Scott the information to create a kind of
details index of some of these parts of the system like an expanded
table of contents.

I've suggested a good start would be picking a few areas (like the
above) and trying to create the info and maybe see what kind of diagram
would present itself from that. If successful, it could then be expanded
to each area. I'd certainly hope that linux-yocto would be one area
covered.

Cheers,

Richard








_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto

On 6/20/13 8:23 AM, "Rifenbark, Scott M" <scott.m.rifenbark@...>
wrote:

-----Original Message-----
From: yocto-bounces@... [mailto:yocto-
bounces@...] On Behalf Of Bill Traynor
Sent: Thursday, June 20, 2013 8:12 AM
To: Richard Purdie
Cc: yocto@...; Paul Eggleton; Osier-mixon, Jeffrey
Subject: Re: [yocto] Documenting YP Development Environment in more
detail

On Thu, Jun 20, 2013 at 10:25 AM, Richard Purdie
<richard.purdie@...> wrote:

On Thu, 2013-06-20 at 09:49 -0400, Bruce Ashfield wrote:

On 13-06-20 04:40 AM, Rifenbark, Scott M wrote:

Hi,

Recently, Paul, Ross, Richard and I had a video conference meeting

where we had some initial discussion on how to satisfy YOCTO #2808
(https://bugzilla.yoctoproject.org/show_bug.cgi?id=2808), which calls
for an illustration showing source and destination directories during a
build using YP. This bug originated from a discussion Dave Stewart and
I had a while back around an idea of a more detailed "flow diagram" that
would go into greater detail than the now famous and ubiquitous "The
Yocto Project Development Environment" illustration, which appears in
the YP Quick Start (http://www.yoctoproject.org/docs/1.4/yocto-project-
qs/yocto-project-qs.html) and has been used in many other places and in
many other forms (some quite elaborate).

We can't really create a completely accurate, all-encompassing

illustration that breaks apart the entire build process. It is not a
static thing and it is quite complicated inside the BitBake process. We
can, however, show where metadata comes from, how it is provided, what
it defines, where and how source files are found, what processes occur,
what output is produced, and where it ends up. We can also provide some
sort of idea of how key BitBake and environment variables affect things
along the way. The idea here is to dig deeper into this conceptual
figure and root out the details to a level that would be useful to the
user but not impossible to maintain or develop. This type of information
can be communicated through a mix of several illustrations with
supporting text.

This first meeting started with some detailed discussion of the

configuration inputs for a typical YP build but soon migrated to
discussing the bigger picture and possible ways to provide more
information. It became obvious we were not going to dig the solution
and all the needed information out in one short meeting. Consequently, I
am sending out this email to help open up some discussion on the issue
and to also solicit information for some basic blocks of information.

I would recommend that we don't over-engineer this thing if we can avoid
it.
Remember the goal: People have told me that they are confused about which
directories
are "active" in the build process, i.e. Where does stuff come from and
where does it go?
My advice would be to think of bitbake as a black box with inputs and
outputs.
I know that any directory could be *potentially* an input or output (why
else would it be there?)
But I'm hoping we can up level a little.

Two things are needed at this point: 1) a presentation solution

for this new and more detailed information, and 2) a starting point on
some of the information itself.

PRESENTATION SOLUTION

Here are some thoughts on how to present this information. There

are disadvantages and advantages to each of these methods of which I
will not list. I would like to see what people think about them:

* Manual - Create a section in the "Technical Details" Chapter of

the YP Reference Manual that holds this information. The section
would be pretty much self-contained and would consist of several
illustrations that would stem from an overall figure that is similar
to the figure we now have in the YP Quick Start. However, we would
use a drill-down strategy that would progressively reveal more detail
through subsequent figures. This method is similar to how hardware
devices used to be documented where functional blocks would be
connected and described in one area and then subsequent areas would
elaborate more deeply on a particular block. Linking within the
manual could connect up the various functional blocks (inputs,
outputs, and tasks) that comprise the overall flow.

* Manual / Website Mix - Devise a mix between the YP Reference

Manual and some pages in the YP Website that provide the information.
Create a section in the "Technical details" Chapter of the YP
Reference Manual that covers this information at a high level. For
example, the overall flow of the system with its "big-block" inputs
and outputs and tasks could be discussed at some length. Links in

the

text could go to the YP website where more detail would be revealed.
This strategy effectively splits the content into "overview" and
"details" between the manual and website, respectively.

* Website - With this strategy, everything is pretty much in the YP

website. This area would exist in a stand-alone fashion. However,
you could link to the website from within the existing YP
documentation set from existing areas that deal with the build
process. Several exit points from within the manual set already
exist. We would obviously add a primary one as well that would

likely

originate from the YP Reference Manual's "Technical Details" Chapter.

I'm wondering if a hybrid is possible here. Can we for example embed
image maps into the docbook so that if you hover over areas of the
image, you can then have links to the different sections?

FWIW, docbook supports image maps via the mediaobject tag but they are
somewhat limited in their usefulness.

http://docbook.org/tdg5/en/html/mediaobject.html
http://www.sagehill.net/docbookxsl/Imagemaps.html

Scott, you may run into trouble when converting the docs between doc
types, in particular with scaling of images.

I will do some playing around with image maps and see what comes up.
It would be nice to see if we could make some "hot-spots" within
Illustrations.

SOLICITATION FOR INFORMATION

We can get started now on this by starting to define details for

some obvious points or large areas of the flow. What would be nice

to

get would be some graphical breakdowns of these areas of concern:

* User-defined layers
* YP provided layers
* User configuration
* Machine Configuration
* Policy Configuration
* Patches
* YP provided recipes
* User provided source code
* YP provided source code
* SCMs
* Generated images
* Generated SDKs/toolchains
* Package Output
* Source fetching process
* Patch application process
* Configuration process (fragments, etc.)

Just so I'm clear .. are you looking for graphical breakdowns to be
created and sent,
or information offered so graphical breakdowns can be created ?

Bruce - any kind of information in any form is useful. If you need to
hack up a drawing to get a point across... do so. Or, if you can send
textual information that is good too.

The reason I ask, is that if there's any interest in the linux-yocto
(for lack of a better term) flow being described graphically, I'm
happy to offer up information, but my graphical skills are limited
to what you find in a typical hackers bag of ticks :)

I suggested that Scott try and accumulate information for each topic
like the following:

Fetcher:

Code Areas: Bitbake Fetch Module (bitbake/lib/bb/fetch2, called from
classes/base.bbclass)
Tasks Covered: do_fetch/do_unpack
Key Variables: SRC_URI, checksums etc

Generated images:

Code Areas: classes/image*.bbclass classes/rootfs*.bbclass
Tasks Covered: do_rootfs
Key Variables: PACKAGE_INSTALL

along with a description about what the function

Cheers,

Bruce

* Key variable use and effects
* User-initiated commands along the way

Much of this list is directly from our existing "The Yocto Project

Development Environment" illustration.

Thanks,
Scott R.



Scott Rifenbark
Intel Corporation
Yocto Project Documentation
503.712.2702
503.341.0418 (cell)

_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto

al block does and when its used. The above would then link into the

class reference, the variable glossary and so on. So its less about
graphics and more about giving Scott the information to create a kind

of

details index of some of these parts of the system like an expanded
table of contents.

I've suggested a good start would be picking a few areas (like the
above) and trying to create the info and maybe see what kind of

diagram

would present itself from that. If successful, it could then be

expanded

to each area. I'd certainly hope that linux-yocto would be one area
covered.

Cheers,

Richard








_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto

_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto

_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto

I think this is an excellent project and am excited to see it coming together.

Are the meetings open for participation? I can't guarantee that I'd be
available when you run them, but I wouldn't mind listening in if I am.

If I were writing a book about Yocto/OE, in the first chapter I would
have the readers build their own filesystem/kernel from scratch (I
can't decide if I would also have them build their own cross-compiler
from scratch or if I'd cheat and let them use crosstool-NG). I think a
lot of the variables and tweaks involved in using Yocto/OE, and
certainly the steps Yocto/OE follows while doing its work, would be
more obvious to someone who has done what Yocto/OE is doing at least
once by hand.

I'm not suggesting any documentation effort should go down to the "do
it yourself from first principles at least once" level; but on the
other hand I think many people would understand Yocto/OE better if
they had that knowledge.

Coincidentally, Nicolas and I were recently involved in providing a
whirlwind tour of Yocto/OE to some developers last week.

On 20 June 2013 04:40, Rifenbark, Scott M <scott.m.rifenbark@...> wrote:

We can get started now on this by starting to define details for some obvious points or large areas of the flow. What would be nice to get would be some graphical breakdowns of these areas of concern:

* User-defined layers
* YP provided layers
* User configuration
* Machine Configuration
* Policy Configuration
* Patches
* YP provided recipes
* User provided source code
* YP provided source code
* SCMs
* Generated images
* Generated SDKs/toolchains
* Package Output
* Source fetching process
* Patch application process
* Configuration process (fragments, etc.)
* Key variable use and effects
* User-initiated commands along the way

In addition to the above, excellent segments, I observed the following
based on my recent experience:

I think it would be fair to assume that most people reading this new
documentation didn't just install Linux for the first time a day or
two ago and are now wondering about Yocto. I think many people have
some sort of build experience. So I think it might be worth
considering having a "how do I ...?" section which attempts to cover
as many common questions/scenarios as possible. Perhaps it might be
interesting to directly compare Yocto/OE to other build systems (not,
of course, to try to get anyone's defenses up, but rather to be able
to answer questions such as "in build system X I do this to get that
result, how do I do the same thing in Yocto?"). Maybe these questions
and answers could be organized based on assuming a user is coming for
a specific other build system? Or perhaps have sections explaining why
certain procedures which are required in other build systems aren't
required in Yocto.

For example: some build systems require you to restart your entire
build process if there is a failure anywhere during this process. So
one of the questions we got was "is bitbake smart enough to restart
the build at the point of failure if there is an error, or does the
build have to be started from scratch every time?".

To some people it might seem bizarre that Yocto generates packages
(e.g. rpms or debs) when (by default) the core-image-minimal target
didn't have any package management tools. It took us a while to
clarify that Yocto is building these (e.g.) rpms by default as part of
its own internal process. The fact package management tools can be
installed with a very simple configuration file change and then these
rpms or debs can be used to update a running image is just an extra
bonus; but you're getting these (e.g.) rpms whether you want them or
not :-) Stranger still: choosing rpms instead of debs doesn't mean
your final system is going to look a lot more like Fedora than Debian.

Tied closely to the above, lots of clarity was required to explain
that Yocto builds each recipe individually, installs it (somewhere),
uses the files in "somewhere" to generate a set of packages, and then
uses the selected packages (i.e. the -doc, -dev, -dbg, ...) to create
the target's root filesystem image. I think the multiple "install"
stages was not intuitive. I think most people by default would simply
install the build output from each component directly into a sysroot
without the indirection of creating a set of packages which are then
selected from to make up the final image. Obviously what Yocto is
doing is brilliant and correct, it's just not obvious as to why it
would do things that way (I think).

As I mentioned in my other comments, sometimes the best way to explain
why Yocto is doing things the way it is can be best understood once a
person has actually tried to do these things by hand. Polluting the
sysroot during the build would be quite obviously wrong if/when a new
version of a component were required, or you wanted to remove a given
thing from the output image.

Has anyone else heard of vimcasts.org?

Short, concise, detailed screen-casts of specific Yocto tasks coupled
with "show notes", I think, would be an awesome other way of
delivering this information.

yoctocasts.org anyone?

On 23 June 2013 00:01, Trevor Woerner <trevor.woerner@...> wrote:

Has anyone else heard of vimcasts.org?

http://vimcasts.org/episodes/archive

is a good example of what I'm thinking about.

On Mon, Jun 24, 2013 at 1:41 AM, Rudolf Streif
<rstreif@...> wrote:

Hi Trever et al:

If I were writing a book about Yocto/OE,

This is a project I am currently working on, a book about the Yocto Project.
The goal is to enable readers to do practical projects with YP. As the
subject matter for the project described in the book I have chosen a home
automation project. Reason being, it interests me personally and it uses
different devices such as a UI-less controller, remotes with touch screens
etc.

I have gotten a lot of feedback from the YP training class I have developed
and been teaching for the Linux Foundation. I am putting this out here to
solicit more feedback from the community on what you think a book on YP
should include. For instance as an advanced topic I included a chapter on
how to run YP on AWS EC2 and I will be adding Autobuilder to it too.

in the first chapter I would
have the readers build their own filesystem/kernel from scratch (I
can't decide if I would also have them build their own cross-compiler
from scratch or if I'd cheat and let them use crosstool-NG).

I thought about that too but I found it distracting. It's like "let me show
you the hard way with crosstool-ng and buildroot and then I show you a
better way with YP." What I am doing though is an intro into Bitbake: how to
use just Bitbake to build something. That proved valuable during the
training classes. It's like a HelloWorld (and I published that before on
this mailing list) introducing the concepts of Bitbake with it's layers,
recipes, syntax etc.

FWIW, I'm incorporating a HelloWorld chapter into the new BitBake
manual. It builds on your post of the same that you made to the
mailing list.

Cheers,
Rudi

_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto

Hi Rudi,

On 24 June 2013 01:41, Rudolf Streif <rstreif@...> wrote:

If I were writing a book about Yocto/OE,

This is a project I am currently working on, a book about the Yocto Project.

Awesome! That's great news! I was hoping someone would "beat me to it"
:-) I look forward to reading it once it's done. Do you have a
publisher or an expected release date?

An appendix on Android's "repo" tool might make for an excellent
addition; it seems some Yocto projects are showing lots of interest in
using it.

in the first chapter I would
have the readers build their own filesystem/kernel from scratch

I thought about that too but I found it distracting.

Fair enough. Maybe I'll write some blog-ish post about it for those
rare few who would like to see a bottom-up perspective :-)

On Fri, Jun 28, 2013 at 10:02 PM, Khem Raj <raj.khem@...> wrote:

We use android repo internally to manage the layers and I think there are
projects out there like here

http://www.jann.cc/2012/08/08/building_freescale_s_community_yocto_bsp_for_the_olinuxino.html

which are also using android repo to manage layers.

indeed. gumstix too: https://github.com/gumstix/Gumstix-YoctoProject-Repo

and at Linaro we are moving all our OE jobs to use repo tool as well.