Date   

Re: Basic failures building 1st Yocto Project and Yocto Raspberry Pi

Jack Mitchell <ml@...>
 

On 01/05/13 01:06, Alex J Lennon wrote:
It should just be a matter of changing a few git revisions, and tweaking
the wording a bit to involve the new meta-yocto, meta-yocto-bsp split.
If anyone wishes to step up to the plate the source for this blog post
is available at [1] and I will happily accept patches.
Jack, I've taken up your invitation, forked the repo and committed a
change to my fork which updates the document to current meta-raspberrypi
commit and a recent (circa two weeks) Yocto commit.

I've just built this from scratch here and aside from a couple of GitHub
feed hiccups it works. It runs up on the RPi.

I've sent a pull request over to your base repo (or at least I think I
have...)
Hi Alex,

Pulled, Merged and should be live on the site now.

Thanks for your contributions!

Regards,
Jack.


Cheers,

Alex

Cheers,
Jack.

[1] https://github.com/CommunistCode/PimpMyPi-Community-Blog-Posts


Starting point ... Help?

Ian Graham <ian@...>
 

Hello,

Apologies in advance if this is not the appropriate resource for this request. Someone recommended this email list as a potential resource.

Where is the best place to go to get help on Yocto project if you are brand new to this type of embedded development? ... I have basic hardware, embedded, computer and software skills but no specific knowledge of linux, yocto or build environment?

What I am looking to do is to setup a very simple development environment to learn by doing with Raspberry Pi as the BSP and yocto as the SDK. Then do some fun little projects to get familiar with the build environment and start doing more advanced projects.

I am totally new to yocto, linux and this type of embedded development. However, I have been playing with Yocto over the past few weeks trying to get a development environment setup up, reading the Yoctoproject.org site, googling various resources and trying to learn as much as I can. Setup VMware player and build appliance as per the https://www.yoctoproject.org/documentation/build-appliance-manual but dead ended because of some networking issues. Next tried mysysGit but, encountered some errors (attached) when setting up the project. 

Any assistance greatly appreciated.

Ian Graham

--

Making Start-ups Happen!


bitbake problem or a meta-ti problem.

Edward Vidal <vidal.develone@...>
 



---------- Forwarded message ----------
From: Edward Vidal <vidal.develone@...>
Date: Sat, Jan 5, 2013 at 7:20 AM
Subject: bitbake problem or a meta-ti problem.
To: Discussion of the angstrom distribution development <angstrom-distro-devel@...>


Hello,
Any and all help will be appreciated.

With Angstrom the linux-omap_3.1.0-r1a is built.  This uses meta-ti/receipes-kernel/linux/linux-omap4_3.1.0.bb which has
SRC_URI = "git://dev.omapzoom.org/pub/scm/integration/kernel-ubuntu.git;protocol=git;branch=ti-ubuntu-3.1-1282 \
Angstrom is using bitbake 1.15.2
OE Build Configuration:
BB_VERSION        = "1.15.2"
TARGET_ARCH       = "arm"
TARGET_OS         = "linux-gnueabi"
MACHINE           = "pandaboard"
DISTRO            = "angstrom"
DISTRO_VERSION    = "v2012.05"
TUNE_FEATURES     = "armv7a vfp neon cortexa9"
TARGET_FPU        = "vfp-neon"
meta-angstrom     = "angstrom-v2012.05-yocto1.2:6a2febed13f179c042c2371b2e86286c
80c0df5e"
meta-oe          
toolchain-layer  
meta-efl         
meta-gpe         
meta-gnome       
meta-xfce        
meta-initramfs    = "denzil:a6f56d7eabe50dc55198aa8b8b349dbbb40757b1"
meta-opie         = "master:efa3892b20a4ef80274e56e5633ebd62c16f9731"
meta-java         = "master:3386ea6c96096f107f43f282f654e5afa456109e"
meta-browser      = "master:c47f59df2e723495679c751cbdf6a8c6adec4b6a"
meta-mono         = "master:83f8233b0498aadb18bf7605c3ba6c71d9e13a3a"
meta-kde          = "master:5b0882d951cfd71886d423c190faaa7c7f932333"
meta-ti           = "angstrom-staging-yocto1.2:edfd9649cbd16b2cdb5385954c0441c7b

With meta-ti and yocto the linux-omap_3.1.0-r0 would hang up do a do_fetch.  This uses meta-ti/receipes-kernel/linux/linux-omap4_3.1.0.bb which has
SRC_URI = "git://dev.omapzoom.org/pub/scm/integration/kernel-ubuntu.git;protocol=git;branch=ti-ubuntu-3.1-1282 \

Yocto is using bitbake 1.17.0
Build Configuration:
BB_VERSION = "1.17.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "Unknown"
TARGET_SYS = "arm-poky-linux-gnueabi"
MACHINE = "pandaboard"
DISTRO = "poky"
DISTRO_VERSION = "1.3+snapshot-20121230"
TUNE_FEATURES = "armv7a vfp neon"
TARGET_FPU = "vfp-neon"
meta
meta-yocto = "master:4a36a32567ecfbc7ce7b967803e6e23314953ef5"
meta-ti = "master:d3ff5630a74640f46ca2ceaef1aa1baa86b9b795"

I found the following at http://www.mail-archive.com/yocto@.../msg09315.html

and https://lists.yoctoproject.org/pipermail/yocto/2012-October/012023.html

Both of these sites have deferences to changing protocol=git to protocol=http

I just changed my recipe meta-ti/receipes-kernel/linux/linux-omap4_3.1.0.bb which has
SRC_URI = "git://dev.omapzoom.org/pub/scm/integration/kernel-ubuntu.git;protocol=git;branch=ti-ubuntu-3.1-1282 \ to meta-ti/receipes-kernel/linux/linux-omap4_3.1.0.bb which has
SRC_URI = "git://dev.omapzoom.org/pub/scm/integration/kernel-ubuntu.git;protocol=http;branch=ti-ubuntu-3.1-1282 \ and the fetch worked.
Is this a bitbake problem or a meta-ti problem.
Thanks Ed vidal
On Fri, Jan 4, 2013 at 11:09 AM, Edward Vidal <vidal.develone@...> wrote:

Hello,
I am still trying to get the pandaboard to boot with meta-ti and yocto.
I built the kernel on a Fedora17 pandaboard,  This took appoximately 9 hrs.
The following are the steps that I have taken.  What needs to set to have the kernel display the process as it is booting?
This is far as get with Angstrom and meta-ti with Yocto.
Basic steps
fetch the kernel
checkout ti-ubuntu-3.1.0-1282.11
configure the kernel
make
change zImage to uImage with mkimage
copy files to SD card

 As always any and all help will be appreciated.
regards Ed Vidal

On Thu, Jan 3, 2013 at 1:25 PM, Edward Vidal <vidal.develone@...> wrote:
Hello,
the Angstrom build just finished.
I get the same as meta-ti and yocto

Texas Instruments OMAP4460 ES1.1
OMAP SD/MMC: 0
reading    u-boot.img
reading    u-boot.img


U-Boot 2011.12-dirty (Jan 03 2013 - 12:29:15)


CPU  : OMAP4460 ES1.1
Board: OMAP4 Panda
I2C:   ready
DRAM:  1 GiB
MMC:   OMAP SD/MMC: 0
Using default environment

In:    serial
Out:   serial
Err:   serial
Hit any key to stop autoboot:  0
reading boot.scr

342 bytes read

Running bootscript from mmc0 ...
## Executing script at 82000000
Booting from mmc0
Device:    OMAP SD/MMC
Manufacturer ID: 2
OEM: 544d
Name: SA16G
Tran Speed: 25000000
Rd Block Len: 512
SD version 2.0
High Capacity: Yes
Capacity: 14.6 GiB
Bus Width: 4-bit
reading uImage

4164360 bytes read
## Booting kernel from Legacy Image at 80300000 ...
   Image Name:   Angstrom/3.1.0/pandaboard

   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:      4164296 Bytes = 4 MiB

   Load Address: 80008000
   Entry Point:     80008000
   Verifying Checksum ... OK
   Loading Kernel Image ... OK
OK

Starting kernel ...

Uncompressing Linux... done, booting the kernel.
As always any and all help will be appreciated.
regards Ed Vidal

On Thu, Jan 3, 2013 at 1:09 PM, Edward Vidal <vidal.develone@...> wrote:
Hello All from Ed Vidal
I have not changed any recipes.

 I have been trying with a current version of angstrom which downloads linux-omap_3.1.0-r1a. cd ~/angstrom/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/deploy/images/pandaboard/  This creates the following files
 ls
MLO
MLO-pandaboard
MLO-pandaboard-2011.12
modules-3.1.0-r1a-pandaboard.tgz
README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt
u-boot.img
u-boot-pandaboard-2011.12-r8.img
u-boot-pandaboard.img
uImage
uImage-3.1.0-r1a-pandaboard-20130103162548.bin
uImage-pandaboard.bin
I also have tried using meta-ti with yocto which creates the following files

u-boot-pandaboard-2011.12-r8.img
u-boot-pandaboard.img
uImage
uImage-3.1.0-r1a-pandaboard-20130103162548.bin
uImage-pandaboard.bin
[vidal@ws009 pandaboard]$ cd ~/POKY/build122912_panda/poky/build/tmp/deploy/images/
[vidal@ws009 images]$ ls
core-image-sato-sdk-pandaboard-20121230164443.rootfs.tar.bz2
core-image-sato-sdk-pandaboard-20130101172231.rootfs.tar.bz2
core-image-sato-sdk-pandaboard.tar.bz2
MLO
MLO-pandaboard
MLO-pandaboard-2011.12
modules--3.1.0-r0-pandaboard-20121230164443.tgz
modules--3.1.0-r0-pandaboard-20130101172231.tgz
README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt
u-boot.img
u-boot-pandaboard-2011.12-r8.img
u-boot-pandaboard.img
uImage
uImage-3.1.0-r0-pandaboard-20121230164443.bin
uImage-3.1.0-r0-pandaboard-20130101172231.bin
The difference between uImage-3.1.0-r0-pandaboard-20121230164443.bin and uImage-3.1.0-r0-pandaboard-20130101172231.bin is I added to local.conf
CMDLINE_DEBUG ?= "debug" to see if I could be some debug out of the boot process. All
I get is Starting kernel ...


Uncompressing Linux... done, booting the kernel.
My angstrom build should be done shortly. May this will give a tar.vz2 file so I can test the kernel that I get above in  ~/angstrom/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/deploy/images/pandaboard/
 As always any and all help will be appreciated.
regards Ed Vidal

On Thu, Jan 3, 2013 at 11:50 AM, Gary Thomas <gary@...> wrote:
On 2013-01-03 11:40, Alan DuBoff wrote:
On Tue, 25 Dec 2012, Gary Thomas wrote:

Please try to answer the question that was asked: what did you change in the *recipe*??

I *DID* answer, please go back and read my messages, here it is again for your convenience:

setup-scripts/sources/meta-ti/recipes-kernel/linux/linux-ti33x-psp_3.2.bb

If you change the kernel version on that recipe it will build the image with that kernel, AFAICT.

Exactly what do you mean?  Did you actually change the recipe (i.e. edit the file
mentioned above)?  Or did you try to select that kernel via some other means?  If
so, how?

If you'll be more explicit about how you are trying to get this alternate
kernel to be used, we can help figure out why it's not working for you.



The kernel and the modules [package] have to match or else you'll have problems.

They do. I've included the kernel-modules in my recipes.

query: why isn't this discussion on the meta-ti list where the supporters
of this recipe (and layer) hang out?


--
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

_______________________________________________
Angstrom-distro-devel mailing list
Angstrom-distro-devel@linuxtogo.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/angstrom-distro-devel






Re: Where does the 'PN' is set.

Biao <huanmateme@...>
 


At 2013-01-05 14:24:07,Biao <huanmateme@...> wrote:

At 2013-01-04 19:35:58,"Burton, Ross" <ross.burton@...> wrote: >On 4 January 2013 11:34, Bill Traynor <wmat@...> wrote: >>> I would like to know where does the so-called 'gather' exactly happens? >>> For example, does it mean the bitbake core get the 'PN = u-boot' by >>> cutting of the name of 'u-boot_2011.03.bb'? > >If you want to know exactly where it happens, see bitbake.conf: > >PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE'),d)[0] or >'defaultpkgname'}" >PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE'),d)[1] or '1.0'}" >PR = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE'),d)[2] or 'r0'}" Now i began to realize that my confusion seems due to a misunderstanding of the parsing procedure.
So the bitbake is NOT a ONE-TIME parsing file, it will be parsed again before parsing any bb file of the whole BBFILES set. 
Error recovery: So the bitbake.conf is NOT a ONE-TIME parsing file ...
step_1: parsing bitbake.conf, parsing pakage1.bb
step_2: parsing bitbake.conf, parsing pakage2.bb
step_n: parsing bitbake.conf, parsing pakagen.bb
Is that correct?
>
>Ross


Re: Where does the 'PN' is set.

Biao <huanmateme@...>
 


At 2013-01-04 19:35:58,"Burton, Ross" <ross.burton@...> wrote: >On 4 January 2013 11:34, Bill Traynor <wmat@...> wrote: >>> I would like to know where does the so-called 'gather' exactly happens? >>> For example, does it mean the bitbake core get the 'PN = u-boot' by >>> cutting of the name of 'u-boot_2011.03.bb'? > >If you want to know exactly where it happens, see bitbake.conf: > >PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE'),d)[0] or >'defaultpkgname'}" >PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE'),d)[1] or '1.0'}" >PR = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE'),d)[2] or 'r0'}" Now i began to realize that my confusion seems due to a misunderstanding of the parsing procedure.
So the bitbake is NOT a ONE-TIME parsing file, it will be parsed again before parsing any bb file of the whole BBFILES set. 
step_1: parsing bitbake.conf, parsing pakage1.bb
step_2: parsing bitbake.conf, parsing pakage2.bb
step_n: parsing bitbake.conf, parsing pakagen.bb
Is that correct?
>
>Ross


Re: Host authenticity failures

Khem Raj
 

On 01/04/2013 06:50 PM, Stewart, David C wrote:
Is this still a problem?

From: <Thornburg>, Christopher A <christopher.a.thornburg@...<mailto:christopher.a.thornburg@...>>
Date: Wednesday, January 2, 2013 2:48 PM
To: "yocto@...<mailto:yocto@...>" <yocto@...<mailto:yocto@...>>
Subject: [yocto] Host authenticity failures

I’ve started getting the following error from bitbake when it’s trying to access my git server:

The authenticity of host '[<githost>]:<port>([<IP>]:<port>)' can't be established.
DSA key fingerprint is <blah>

I understand this error means that the server’s key has probably changed, and I understand in general how to deal with this via SSH config. When I manually ssh’d to the server the first time after receiving the error, I got the same message, as expected. I entered “yes” and tried again to verify that the error went away. It did. I think ran bitbake again and got the same error. I changed my ~/.ssh/ssh_config file to disable StrictHostKeyChecking and still get the error.

Is bitbake somehow using different known_hosts and ssh_config files from the ones I use for interactive logins? Other ideas on how to resolve this?
bitbake is not hosted on yp dot org but on oe dot org


Re: Host authenticity failures

David Stewart
 

Is this still a problem?

From: <Thornburg>, Christopher A <christopher.a.thornburg@...<mailto:christopher.a.thornburg@...>>
Date: Wednesday, January 2, 2013 2:48 PM
To: "yocto@...<mailto:yocto@...>" <yocto@...<mailto:yocto@...>>
Subject: [yocto] Host authenticity failures

I’ve started getting the following error from bitbake when it’s trying to access my git server:

The authenticity of host '[<githost>]:<port>([<IP>]:<port>)' can't be established.
DSA key fingerprint is <blah>

I understand this error means that the server’s key has probably changed, and I understand in general how to deal with this via SSH config. When I manually ssh’d to the server the first time after receiving the error, I got the same message, as expected. I entered “yes” and tried again to verify that the error went away. It did. I think ran bitbake again and got the same error. I changed my ~/.ssh/ssh_config file to disable StrictHostKeyChecking and still get the error.

Is bitbake somehow using different known_hosts and ssh_config files from the ones I use for interactive logins? Other ideas on how to resolve this?


Re: Where does the 'PN' is set.

Biao <huanmateme@...>
 


在 2013-01-04 19:34:10,"Bill Traynor" <wmat@...> 写道:
On Fri, Jan 4, 2013 at 1:53 AM, Biao <huanmateme@...> wrote:
Hi,

In meta/conf/documentation.conf
PN[doc] = "PN holds the name of the package (Package Name). It is gathered from the bitbake-file filename"

I would like to know where does the so-called 'gather' exactly happens?
For example, does it mean the bitbake core get the 'PN = u-boot' by cutting of the name of 'u-boot_2011.03.bb'?

Yes, within the context of the build process, PN refers to the Package Name that is extracted from the recipe file name.  So your example is correct.

For reference, see the PN variable description in the Yocto Project Reference manual:  http://www.yoctoproject.org/docs/current/poky-ref-manual/poky-ref-manual.html#var-P

The "gathering" occurs during the Bitbake parsing step.  Roughly, parsing occurs in the following order:  configuration files, classes, bbfiles.

Thanks for you kindly help. 



Thanks,
Biao 

_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto



Re: Basic failures building 1st Yocto Project and Yocto Raspberry Pi

Alex J Lennon <ajlennon@...>
 

It should just be a matter of changing a few git revisions, and tweaking
the wording a bit to involve the new meta-yocto, meta-yocto-bsp split.
If anyone wishes to step up to the plate the source for this blog post
is available at [1] and I will happily accept patches.
Jack, I've taken up your invitation, forked the repo and committed a
change to my fork which updates the document to current meta-raspberrypi
commit and a recent (circa two weeks) Yocto commit.

I've just built this from scratch here and aside from a couple of GitHub
feed hiccups it works. It runs up on the RPi.

I've sent a pull request over to your base repo (or at least I think I
have...)

Cheers,

Alex

Cheers,
Jack.

[1] https://github.com/CommunistCode/PimpMyPi-Community-Blog-Posts


Re: Bitbake and task offloading onto multiple cloud-based servers

Flanagan, Elizabeth <elizabeth.flanagan@...>
 

On Fri, Jan 4, 2013 at 2:47 PM, Alex J Lennon
<ajlennon@...> wrote:
On 04/01/2013 22:43, Mark Hatle wrote:
On 1/4/13 2:56 PM, Alex J Lennon wrote:

Can anybody advise on whether bitbake currently supports offloading of
build tasks onto multiple systems? Perhaps cloud based?
I see Chris Larson has already answer the main question here.. but I've got some
additional insight I can add.

I'm thinking that it would be more efficient for me if I could bring up
a number of Amazon EC2 servers (or similar) then have bitbake
parallelise the build onto those servers to significantly reduce my
build times?
In the past I've had people ask me questions about remote builds, parallel
building packages, etc. In the past when I've attempted to figure out if it
would be useful, the cost of the data transfer time always comes into play. How
long does it take to simply perform the task locally vs go to the remote
machine, perform the task, and get the data back. For a lot of the tasks that
OE does, the remote tasks will end up being slower on a reasonably modern i7 (or
similar Xeon) machine. I'm not sure using something like a cloud server would
end up helping due to this.
Thanks for your thoughts Mark.

Another idea that occurred was that it might be interesting if there
were some metrics available somewhere on recipe build times (are timing
metrics currently generated in any detail?).
Yes, they're generated by buildstats.bbclass and should be available
within TMPDIR/buildstats. You can also use scripts/pybootchartgui.py
to get a visual representation of the timeline of a build.

-b


Using that as a starting point it might then be possible to determine
how best to parallelise tasks?

From experience I'd say that if anything you'd likely want to modify the
bitbake scheduler with some type of locality information. I.e. if you extract
the sources on a node, you most likely want to patch, configure, compile,
install, QA and package on that node as well.... but that also means a way to
establish node specific tmp/work space is needed for performance as well.

I see bitbake supports a level of task parallelisation on a single box.

Can parallelisation of build onto multiple systems be achieved?
All of the above are possible, but I suspect there is a lot of design and
refactoring required to come up with a system that is both efficient and fast.

Is it something that should even be a goal?
It's not currently a goal. But it does get asked about every now and then. It
would be nice to have something more then anecdotal information about if it
would be a good idea or not. This is the type of thing that I think would make
an excellent research project. (Hey and if it makes things able to be built for
more people, faster, and cheaper great! If not, we'd have a more informed
response to questions like this.)

--Mark

Cheers,

Alex

_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto
_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto
_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto


--
Elizabeth Flanagan
Yocto Project
Build and Release


Re: Raspberry Pi do_fetch failure

Andrei Gherzan
 

On Fri, Jan 4, 2013 at 10:22 PM, Alex J Lennon <ajlennon@...> wrote:
On 01/01/2013 17:34, ed nelson wrote:
> I am getting a do_fetch failure on raspberrypi kernel.  Is anyone else
> having this problem?
>
> Build Configuration:
> BB_VERSION        = "1.16.0"
> TARGET_ARCH       = "arm"
> TARGET_OS         = "linux-gnueabi"
> MACHINE           = "raspberrypi"
> DISTRO            = "poky"
> DISTRO_VERSION    = "1.3"
> TUNE_FEATURES     = "armv6 vfp"
> TARGET_FPU        = "vfp"
> meta
> meta-yocto
> meta-yocto-bsp    = "danny:bf909b267498dbab4d7695c26b0dce903ac8b6b0"
> meta-raspberrypi  = "master:305c5259e24eaa9fb285ca983dc4f9454743fa0c"
>
> NOTE: Resolving any missing task queue dependencies
> NOTE: Preparing runqueue
> NOTE: Executing SetScene Tasks
> NOTE: Executing RunQueue Tasks
> WARNING: Failed to fetch URL
> git://github.com/raspberrypi/linux.git;protocol=git;branch=rpi-3.2.27,
> attempting MIRRORS if available
> ERROR: Fetcher failure: Fetch command failed with exit code 128, output:
> Cloning into bare repository
> '/home/ed/oe/poky-danny/rasperryPiBuild/downloads/git2/github.com.raspberrypi.linux.git'...
>
> fatal: read error: Connection reset by peer
> fatal: early EOF
> fatal: index-pack failed
>
> ERROR: Function failed: Fetcher failure for URL:
> 'git://github.com/raspberrypi/linux.git;protocol=git;branch=rpi-3.2.27'.
> Unable to fetch URL from any source.
>
> Thanks

I'm rebuilding an RPi image from scratch here now for some testing and I
see the same issue, which looks to be a GitHub hiccup.

I do have a cached download of the RPi linux GitHub repository from a
previous build, which you're welcome to use if it'll help you move
forward whilst GitHub has this issue.

I've tarred it up as github.com.raspberrypi.linux.git.tgz. You can
extract this into the Yocto downloads directory and I believe it'll be
picked up by the build tool, e.g.

cd /path/to/yoctoProject/raspberryPiBuild/downloads
tar xzf  /path/to/github.com.raspberrypi.linux.git.tgz

You should see it go into a "git2" folder in downloads.

It's currently uploading to a DropBox public link at

https://dl.dropbox.com/u/12130167/rpi/github.com.raspberrypi.linux.git.tgz

(It's around 620MB so it'll be about an hour before it's up. I'll
probably leave it up there over the weekend but it may disappear next week!)

Cheers,

Alex

 
Thank you Alex. Hope it will help Nelson.
ag 


Re: Bitbake and task offloading onto multiple cloud-based servers

Alex J Lennon <ajlennon@...>
 

On 04/01/2013 22:43, Mark Hatle wrote:
On 1/4/13 2:56 PM, Alex J Lennon wrote:

Can anybody advise on whether bitbake currently supports offloading of
build tasks onto multiple systems? Perhaps cloud based?
I see Chris Larson has already answer the main question here.. but I've got some
additional insight I can add.

I'm thinking that it would be more efficient for me if I could bring up
a number of Amazon EC2 servers (or similar) then have bitbake
parallelise the build onto those servers to significantly reduce my
build times?
In the past I've had people ask me questions about remote builds, parallel
building packages, etc. In the past when I've attempted to figure out if it
would be useful, the cost of the data transfer time always comes into play. How
long does it take to simply perform the task locally vs go to the remote
machine, perform the task, and get the data back. For a lot of the tasks that
OE does, the remote tasks will end up being slower on a reasonably modern i7 (or
similar Xeon) machine. I'm not sure using something like a cloud server would
end up helping due to this.
Thanks for your thoughts Mark.

Another idea that occurred was that it might be interesting if there
were some metrics available somewhere on recipe build times (are timing
metrics currently generated in any detail?).

Using that as a starting point it might then be possible to determine
how best to parallelise tasks?

From experience I'd say that if anything you'd likely want to modify the
bitbake scheduler with some type of locality information. I.e. if you extract
the sources on a node, you most likely want to patch, configure, compile,
install, QA and package on that node as well.... but that also means a way to
establish node specific tmp/work space is needed for performance as well.

I see bitbake supports a level of task parallelisation on a single box.

Can parallelisation of build onto multiple systems be achieved?
All of the above are possible, but I suspect there is a lot of design and
refactoring required to come up with a system that is both efficient and fast.

Is it something that should even be a goal?
It's not currently a goal. But it does get asked about every now and then. It
would be nice to have something more then anecdotal information about if it
would be a good idea or not. This is the type of thing that I think would make
an excellent research project. (Hey and if it makes things able to be built for
more people, faster, and cheaper great! If not, we'd have a more informed
response to questions like this.)

--Mark

Cheers,

Alex

_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto
_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto


Re: Bitbake and task offloading onto multiple cloud-based servers

Mark Hatle <mark.hatle@...>
 

On 1/4/13 2:56 PM, Alex J Lennon wrote:

Can anybody advise on whether bitbake currently supports offloading of
build tasks onto multiple systems? Perhaps cloud based?
I see Chris Larson has already answer the main question here.. but I've got some additional insight I can add.

I'm thinking that it would be more efficient for me if I could bring up
a number of Amazon EC2 servers (or similar) then have bitbake
parallelise the build onto those servers to significantly reduce my
build times?
In the past I've had people ask me questions about remote builds, parallel building packages, etc. In the past when I've attempted to figure out if it would be useful, the cost of the data transfer time always comes into play. How long does it take to simply perform the task locally vs go to the remote machine, perform the task, and get the data back. For a lot of the tasks that OE does, the remote tasks will end up being slower on a reasonably modern i7 (or similar Xeon) machine. I'm not sure using something like a cloud server would end up helping due to this.

From experience I'd say that if anything you'd likely want to modify the bitbake scheduler with some type of locality information. I.e. if you extract the sources on a node, you most likely want to patch, configure, compile, install, QA and package on that node as well.... but that also means a way to establish node specific tmp/work space is needed for performance as well.

I see bitbake supports a level of task parallelisation on a single box.

Can parallelisation of build onto multiple systems be achieved?
All of the above are possible, but I suspect there is a lot of design and refactoring required to come up with a system that is both efficient and fast.

Is it something that should even be a goal?
It's not currently a goal. But it does get asked about every now and then. It would be nice to have something more then anecdotal information about if it would be a good idea or not. This is the type of thing that I think would make an excellent research project. (Hey and if it makes things able to be built for more people, faster, and cheaper great! If not, we'd have a more informed response to questions like this.)

--Mark

Cheers,

Alex

_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto


Re: Bitbake and task offloading onto multiple cloud-based servers

Chris Larson <clarson@...>
 



On Fri, Jan 4, 2013 at 2:17 PM, Alex J Lennon <ajlennon@...> wrote:
Where would one start to look in the bitbake code to add this kind of
support in?

See the subclasses of RunQueueExecute in bb.runqueue, this is what executes the tasks. For an idea of how to implement opting in to a different class, see how BB_SCHEDULER is implemented.
--
Christopher Larson


Re: Bitbake and task offloading onto multiple cloud-based servers

Alex J Lennon <ajlennon@...>
 

On 04/01/2013 21:08, Chris Larson wrote:


On Fri, Jan 4, 2013 at 1:56 PM, Alex J Lennon
<ajlennon@... <mailto:ajlennon@...>>
wrote:

Can anybody advise on whether bitbake currently supports offloading of
build tasks onto multiple systems? Perhaps cloud based?

I'm thinking that it would be more efficient for me if I could bring up
a number of Amazon EC2 servers (or similar) then have bitbake
parallelise the build onto those servers to significantly reduce my
build times?

I see bitbake supports a level of task parallelisation on a single box.

Can parallelisation of build onto multiple systems be achieved?

Is it something that should even be a goal?


It's not supported today. It could be implemented, but nobody has made
it a priority and done so.
Do you have any feeling for the level of difficulty of such an
implementation / what would have to change / how invasive it would be to
the codebase ?

I'm wondering if it could be along the lines of creating a "remote task"
class and then, say, having that class ssh into one of a pool of servers
(running a standard image with all tools preinstalled maybe) then
bitbaking the recipes for the particular and waiting on completion
before pulling back the output rpm/deb/ipkg ?

Things are usually more complex than expected when you get into the
nitty gritty though. What would the challenges be do you think?

Where would one start to look in the bitbake code to add this kind of
support in?

Thanks,

Alex


--
Christopher Larson


Re: Bitbake and task offloading onto multiple cloud-based servers

Chris Larson <clarson@...>
 



On Fri, Jan 4, 2013 at 1:56 PM, Alex J Lennon <ajlennon@...> wrote:
Can anybody advise on whether bitbake currently supports offloading of
build tasks onto multiple systems? Perhaps cloud based?

I'm thinking that it would be more efficient for me if I could bring up
a number of Amazon EC2 servers (or similar) then have bitbake
parallelise the build onto those servers to significantly reduce my
build times?

I see bitbake supports a level of task parallelisation on a single box.

Can parallelisation of build onto multiple systems be achieved?

Is it something that should even be a goal?

It's not supported today. It could be implemented, but nobody has made it a priority and done so.
--
Christopher Larson


Bitbake and task offloading onto multiple cloud-based servers

Alex J Lennon <ajlennon@...>
 

Can anybody advise on whether bitbake currently supports offloading of
build tasks onto multiple systems? Perhaps cloud based?

I'm thinking that it would be more efficient for me if I could bring up
a number of Amazon EC2 servers (or similar) then have bitbake
parallelise the build onto those servers to significantly reduce my
build times?

I see bitbake supports a level of task parallelisation on a single box.

Can parallelisation of build onto multiple systems be achieved?

Is it something that should even be a goal?

Cheers,

Alex


Re: Raspberry Pi do_fetch failure

Alex J Lennon <ajlennon@...>
 

On 01/01/2013 17:34, ed nelson wrote:
I am getting a do_fetch failure on raspberrypi kernel. Is anyone else
having this problem?

Build Configuration:
BB_VERSION = "1.16.0"
TARGET_ARCH = "arm"
TARGET_OS = "linux-gnueabi"
MACHINE = "raspberrypi"
DISTRO = "poky"
DISTRO_VERSION = "1.3"
TUNE_FEATURES = "armv6 vfp"
TARGET_FPU = "vfp"
meta
meta-yocto
meta-yocto-bsp = "danny:bf909b267498dbab4d7695c26b0dce903ac8b6b0"
meta-raspberrypi = "master:305c5259e24eaa9fb285ca983dc4f9454743fa0c"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
WARNING: Failed to fetch URL
git://github.com/raspberrypi/linux.git;protocol=git;branch=rpi-3.2.27,
attempting MIRRORS if available
ERROR: Fetcher failure: Fetch command failed with exit code 128, output:
Cloning into bare repository
'/home/ed/oe/poky-danny/rasperryPiBuild/downloads/git2/github.com.raspberrypi.linux.git'...

fatal: read error: Connection reset by peer
fatal: early EOF
fatal: index-pack failed

ERROR: Function failed: Fetcher failure for URL:
'git://github.com/raspberrypi/linux.git;protocol=git;branch=rpi-3.2.27'.
Unable to fetch URL from any source.

Thanks
I'm rebuilding an RPi image from scratch here now for some testing and I
see the same issue, which looks to be a GitHub hiccup.

I do have a cached download of the RPi linux GitHub repository from a
previous build, which you're welcome to use if it'll help you move
forward whilst GitHub has this issue.

I've tarred it up as github.com.raspberrypi.linux.git.tgz. You can
extract this into the Yocto downloads directory and I believe it'll be
picked up by the build tool, e.g.

cd /path/to/yoctoProject/raspberryPiBuild/downloads
tar xzf /path/to/github.com.raspberrypi.linux.git.tgz

You should see it go into a "git2" folder in downloads.

It's currently uploading to a DropBox public link at

https://dl.dropbox.com/u/12130167/rpi/github.com.raspberrypi.linux.git.tgz

(It's around 620MB so it'll be about an hour before it's up. I'll
probably leave it up there over the weekend but it may disappear next week!)

Cheers,

Alex




_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto


Re: cairo failed on openSUSE

Martin Jansa
 

On Thu, Jan 03, 2013 at 01:48:15PM +0100, Ronan Le Martret wrote:
- add libpng version 15 to configure.ac of cairo. (libpng12 confict.)
Please follow:
http://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded

And this belongs to oe-core ML.

Cheers,

--
Martin 'JaMa' Jansa jabber: Martin.Jansa@...


Re: Fails to launch the hob

Barros Pena, Belen <belen.barros.pena@...>
 

We have updated the Hob Manual to include the required packages and
screenshots of the 1.3 Hob

https://www.yoctoproject.org/documentation/hob-manual

Belen

On 03/01/2013 23:11, "Zhang, Jessica" <jessica.zhang@...> wrote:

Hi Belen,

I'll modify the error message per suggestion. By looking at the existing
code, seems we only check for GTK+ and PyGtk version, so probably I'll
drop PyGobject in the error message. So you probably can update the hob
info on the web-site accordingly.

Thanks,
Jessica

-----Original Message-----
From: yocto-bounces@...
[mailto:yocto-bounces@...] On Behalf Of Barros Pena, Belen
Sent: Thursday, January 03, 2013 8:48 AM
To: Biao; Mihai Lindner
Cc: yocto@...
Subject: Re: [yocto] Fails to launch the hob

Any chance we could change the content of the error message to provide
better indication of what the problem is? The idea would be adding
version numbers and printing only what's causing trouble.

So, from:

$ hob
FATAL: Gtk+, PyGtk and PyGobject are required to use Hob, You have Gtk+
2.20.1 and PyGtk 2.17.0.

To something like:

$ hob
FATAL: Hob requires Gtk+ 2.20.0 or higher, PyGtk 2.21.0 or higher and
PyGobject [insert minimum version here] or higher You have PyGtk 2.17.0.

Would this be possible?

We should probably also add a paragraph listing essential packages to the
Hob manual page on the website
(https://www.yoctoproject.org/documentation/hob-manual). If someone tells
me what's the minimum version of PyGobject required I could probably edit
the page myself.

Thanks!

Belen



From: Biao <huanmateme@...<mailto:huanmateme@...>>
Date: Tuesday, 25 December 2012 06:16
To: Mihai Lindner
<mihaix.lindner@...<mailto:mihaix.lindner@...>>
Cc: "yocto@...<mailto:yocto@...>"
<yocto@...<mailto:yocto@...>>
Subject: Re: [yocto] Fails to launch the hob


At 2012-12-24 20:37:26,"Mihai Lindner"
<mihaix.lindner@...<mailto:mihaix.lindner@...>>
wrote:
On Mon, 2012-12-24 at 14:23 +0200, Mihai Lindner wrote:
On Mon, 2012-12-24 at 15:37 +0800, Biao wrote:
Hi all,


I can not launch hob on my ubun-10.04, it seems saying "no
PyGobject" whereas python-object has already been installed. Can anyone
kindly give a help?
Oh, forgot to mention here, that you should probably be using Denzil,
instead of Danny or 'master'.
Thanks for the kindly reminder.

$ hob
FATAL: Gtk+, PyGtk and PyGobject are required to use Hob, You have
Gtk+ 2.20.1 and PyGtk 2.17.0.


$ sudo apt-get install python-gobject Reading package lists... Done
Building dependency tree Reading state information... Done
python-gobject is already the newest version.
Python-gobject seems fine, Gtk seems fine (you have 2.20.1, Hob wants
higher than 2.20.0), which leaves python-gtk2 version 2.17.0 on your
system. Hob wants a version higher than 2.21.0.
Thanks for this information, it seems the prompt message on the
environment checking is not so clear. Where are the hob users supposed
to get such kind of information from?

I did not found this information from the hob manual.


--Mihai

$ uname -a
Linux zb-d 2.6.32-33-generic #72-Ubuntu SMP Fri Jul 29 21:07:13 UTC
2011 x86_64 GNU/Linux


Thanks,
Biao
_______________________________________________ yocto mailing list
yocto@...<mailto:yocto@...>
https://lists.yoctoproject.org/listinfo/yocto

_______________________________________________
yocto mailing list
yocto@...<mailto:yocto@...>
https://lists.yoctoproject.org/listinfo/yocto

_______________________________________________
yocto mailing list
yocto@...<mailto:yocto@...>
https://lists.yoctoproject.org/listinfo/yocto
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for the
sole use of the intended recipient(s). Any review or distribution by
others is strictly prohibited. If you are not the intended recipient,
please contact the sender and delete all copies.

_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.