On 10/22/2010 04:38 PM, Richard Purdie wrote:

On Fri, 2010-10-22 at 11:22 -0700, Darren Hart wrote:

Here is an example of the do_fetch/do_unpack failure that I believe is
representative of what Dirk saw and what I have been seeing periodically.

I saw the following while building libproxy in poky-image-sdk-live on a
Fedora 13 box behind the firewall. I saw identical failures with opkg on
a different machine - also behind the firewall. Both SRC_URIs point to
googlecode.

log.do_fetch:
-------------
NOTE: fetch http://libproxy.googlecode.com/files/libproxy-0.4.3.tar.gz
NOTE: libproxy-0.4.3:
http://libproxy.googlecode.com/files/libproxy-0.4.3.tar.gz has no entry
in conf/checksums.ini, not checking URI

log.do_unpack:
--------------
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Exiting with failure status due to previous errors
NOTE: Unpacking
/vol/1/dvhart/poky.git/build/downloads/libproxy-0.4.3.tar.gz to
/vol/1/dvhart/poky.git/build/tmp/work/i586-poky-linux/libproxy-0.4.3-r2/
ERROR: Task failed:

With opkg, I tried the following which resulted in the same error.

$ bitbake -c clean;
$ rm downloads/*opkg*
$ bitbake opkg

In both cases, using wget to manually download the file and place it in
the downloads directory allowed the build to continue.

We've talked about this a lot over jabber. We don't have a reproducer.
What we do see is its always the wget fetcher that hits this (for mirror
urls in the case of git repos) and its failing creating a zero length
file.

If we make the assumption that wget is creating the empty file and
exiting with an error code and look at what bitbake would do, it would
give the behaviour described in these bugs.

Why? If the original fetch fails, it falls back to the mirror code. That
looks, sees a file on the disk and says "nothing to do".

Solution is therefore to make sure if the wget fetcher fails we wipe any
file that may be present.

Is is possible the wget fetcher is creating a zero length file and no
setting an error code but I really hope its not that broken.

So I've pushed this patch as a bandaid for the problem:

http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=c5fab99a6f979a4a0ce246c6395b35a3082aec0d

I've applied and am running another build. This looks sane to me. If I still run into this, I think the next step is to do a brute force zero-length test, and throw the FetchError and take this modified path from this patch in that case too.

wget will generated a 0 length file when using -O (which we confirmed we do not) and there are old bugs against it for creating 0 length files when a valid host fails to connect (without the -O option) but I wasn't able to reproduce that with a current wget.

Here's to hoping this fixes it.

--
Darren Hart
Embedded Linux Kernel

On 10/22/2010 04:44 PM, Darren Hart wrote:

On 10/22/2010 04:38 PM, Richard Purdie wrote:

On Fri, 2010-10-22 at 11:22 -0700, Darren Hart wrote:

Here is an example of the do_fetch/do_unpack failure that I believe is
representative of what Dirk saw and what I have been seeing
periodically.

I saw the following while building libproxy in poky-image-sdk-live on a
Fedora 13 box behind the firewall. I saw identical failures with opkg on
a different machine - also behind the firewall. Both SRC_URIs point to
googlecode.

log.do_fetch:
-------------
NOTE: fetch http://libproxy.googlecode.com/files/libproxy-0.4.3.tar.gz
NOTE: libproxy-0.4.3:
http://libproxy.googlecode.com/files/libproxy-0.4.3.tar.gz has no entry
in conf/checksums.ini, not checking URI

log.do_unpack:
--------------
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Exiting with failure status due to previous errors
NOTE: Unpacking
/vol/1/dvhart/poky.git/build/downloads/libproxy-0.4.3.tar.gz to
/vol/1/dvhart/poky.git/build/tmp/work/i586-poky-linux/libproxy-0.4.3-r2/
ERROR: Task failed:

With opkg, I tried the following which resulted in the same error.

$ bitbake -c clean;
$ rm downloads/*opkg*
$ bitbake opkg

In both cases, using wget to manually download the file and place it in
the downloads directory allowed the build to continue.

We've talked about this a lot over jabber. We don't have a reproducer.
What we do see is its always the wget fetcher that hits this (for mirror
urls in the case of git repos) and its failing creating a zero length
file.

If we make the assumption that wget is creating the empty file and
exiting with an error code and look at what bitbake would do, it would
give the behaviour described in these bugs.

Why? If the original fetch fails, it falls back to the mirror code. That
looks, sees a file on the disk and says "nothing to do".

Solution is therefore to make sure if the wget fetcher fails we wipe any
file that may be present.

Is is possible the wget fetcher is creating a zero length file and no
setting an error code but I really hope its not that broken.

So I've pushed this patch as a bandaid for the problem:

http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=c5fab99a6f979a4a0ce246c6395b35a3082aec0d

I've applied and am running another build.

One poky-image-sdk-live build completed from a clean slate from inside the firewall. I have another outside the firewall that will take a bit longer, but is well on its way without any errors. Looking good.

--
Darren Hart
Embedded Linux Kernel

On 10/22/2010 03:48 PM, Stewart, David C wrote:

We have an RC3 which we feel confident in based on our no-nogo launch decision.

However, there are a handful of issues which are not critical but would be desirable to address them - for example, having to do with errors being "fatal" which are not, etc.

So I have asked Saul to create an RC4a with these patched in. Would like Sanity and BAT tests run on RC4a to see if we should ship it, or with stick with RC3.

RP - will complete the patch tonight.
Saul - will complete the RC4a build by 5PM Sunday (Pacific)

The build has been running overnight and should be complete, the images are in the standard location on the internal and external autobuilder dated 20101024.

http://autobuilder.pokylinux.org/milestone/20101024/

Kevin and Jiajun and whole team - test this build as much as you have time for.

Please pass this request to the Windriver team also as I am not sure if they are on the list

Late Monday, we will decide whether to go with the RC3 build or RC4a build.

Once we (Jessica and I) get through Heathrow and I get a phone card (I hope), I will call Richard. If it's approved, Scott G. can move the bits to the yoctoproject.org machine.

Sau!

Dave
_______________________________________________
yocto mailing list
yocto@...
https://lists.pokylinux.org/listinfo/yocto

From: Stewart, David C
Sent: Saturday, October 23, 2010 6:48 AM

We have an RC3 which we feel confident in based on our no-nogo launch decision.

However, there are a handful of issues which are not critical but would be desirable to
address them - for example, having to do with errors being "fatal" which are not, etc.

So I have asked Saul to create an RC4a with these patched in. Would like Sanity and BAT
tests run on RC4a to see if we should ship it, or with stick with RC3.

RP - will complete the patch tonight.
Saul - will complete the RC4a build by 5PM Sunday (Pacific)
Kevin and Jiajun and whole team - test this build as much as you have time for.

OK, we'll try latest RC4a and report any new issues if seen.

Thanks
Kevin

Late Monday, we will decide whether to go with the RC3 build or RC4a build.

Dave
_______________________________________________
yocto mailing list
yocto@...
https://lists.pokylinux.org/listinfo/yocto

From: Stewart, David C
Sent: Saturday, October 23, 2010 6:48 AM

We have an RC3 which we feel confident in based on our no-nogo launch
decision.

However, there are a handful of issues which are not critical but would
be desirable to address them - for example, having to do with errors
being "fatal" which are not, etc.

So I have asked Saul to create an RC4a with these patched in. Would
like Sanity and BAT tests run on RC4a to see if we should ship it, or
with stick with RC3.

RP - will complete the patch tonight.
Saul - will complete the RC4a build by 5PM Sunday (Pacific) Kevin and
Jiajun and whole team - test this build as much as you have time for.

OK, we'll try latest RC4a and report any new issues if seen.

QA will try as more as possible and send out result today.

Thanks
Kevin

Late Monday, we will decide whether to go with the RC3 build or RC4a build.

Dave
_______________________________________________
yocto mailing list
yocto@...
https://lists.pokylinux.org/listinfo/yocto

_______________________________________________
yocto mailing list
yocto@...
https://lists.pokylinux.org/listinfo/yocto

Best Regards,
Jiajun

Hi all,
This is the Poky weekly bug trend charts for last week (WW43). The open bug number increased to 128. 22 bugs are marked as fixed in last week.
The 1st chart indicates the total open bug number. The 2nd chart indicates the new bug submission and bug fixing count every week.

Best Regards,
Jiajun

Hi jiajun,

This is the Sanity Test Report for M4 build 20101024. Changhui and I  are responsible for qemumips/arm/ppc testing.

For zypper test cases with qemuarm, since create repo fails in my host, the related cases were not tested.

Test Summary

---------------------------------------

Target	Total TCs	Not Run	Passed	Failed	Not testable (Blocked)	Completed Rate [%]	Pass rate of total [%]
Qemumips Sato	19	0	16	3	0	100	84.21
Qemumips SDK	26	2	21	3	0	92.31	80.77
Qemuppc Sato	17	5	11	1	5	70.59	64.71
Qemuppc SDK	24	7	16	1	5	70.83	66.67
Qemuarm Sato	19	3	16	0	3	84.21	84.21
Qemuarm SDK	26	5	21	0	3	80.77	80.77
Total	131	22	101	8	16	83.21	77.10

Images

---------------------------------------

http://autobuilder.pokylinux.org/milestone/20101024/

Best Regards, xianchao

meta/classes/package.bbclass | 1 +
1 file changed, 1 insertion(+)

commit 2cd6d6d7957cf46114c8b25ed13e6f8030cd9c06
Author: Kevin Tian <kevin.tian@...>
Date: Tue Oct 26 15:54:43 2010 +0800

package.bbclass: make sure 'sysroots' created before lockfile

package sstate requires a lock under sysroots/, which however may not be
created when sstate_setscene functions are executed and then causes failures.
here we make sure 'sysroots' created before do_package_setscene is executed.

Signed-off-by: Kevin Tian <kevin.tian@...>
Pull URL: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=tk/master

Thanks
Kevin

As we know, many of us have experienced slow builds of recent poky,
and it also takes larger disk space. This affects user exprience thus
is one of our directions in 1.0.

To find the problems leading to performance issues, I tried some
profiling on poky builds, below is a very brief summary of the data.
I profiled poky-image-minimal of both the current master branch and
green release, with similar parameters (4 CPUs) on NHM. Note that
both rpm and ipk packages are built for current master branch, while
oonly ipk packages are built for green release


I. some stats
1. recipes (including -native)
green release:
recipes built: 76
tasks run: 998
master:
recipes built: 133
tasks run: 1600

2. time
green release:
real: 28m7s
user: 57m45s
sys: 9m41s
user+sys: 67m26s
master:
real: 66m39s
user: 152m17s
sys: 27m37s
user+sys: 179m54s

3. space (haven't analyze though)
green release: 7.8G
master: 26.6G


II. profiling
I tried a brief profile by collecting the time used for every task, so
we can scrutinize the result from a microscopic point of view. I'm still
looking into the full result, but there's something of immediate
attention.

In master, hardly any task consume less than 1.3s, this is quite
surprising, since many tasks like do_patch virtually do nothing, while
in the green release, these tasks may simply consume 0.1~0.2s. A deeper
investigation shows that this large overhead goes to bitbake-runtask,
the bitbake config and cache mechanism is executed for every task,
considerably increased the time. The overhead introduced solely by
this is around 1600*1.3=2080s, approximately 35 minutes (user+sys).

Also, we should count the additional rpm packaging system,
do_package_write_rpm costs around 1400s, (excluding 1.3s per task,
btw, this is about 50% slower than do_package_write_ipk, in average),
that's around 23 minutes.

Roughly considering that the build time is proportional to recipes
count, we can try to estimate master build time from green release:
67 * (76 / 133) + 35 + 23 = 175
very close to the real time consuming (although somewhat too closed...),
so possibly the above two are the most significant time consumers
in the slowness of current poky

Thanks,
Qing

From: Qing He
Sent: Wednesday, October 27, 2010 5:23 PM

As we know, many of us have experienced slow builds of recent poky,
and it also takes larger disk space. This affects user exprience thus
is one of our directions in 1.0.

thanks Qing, that's a great start.

To find the problems leading to performance issues, I tried some
profiling on poky builds, below is a very brief summary of the data.
I profiled poky-image-minimal of both the current master branch and
green release, with similar parameters (4 CPUs) on NHM. Note that
both rpm and ipk packages are built for current master branch, while
oonly ipk packages are built for green release


I. some stats
1. recipes (including -native)
green release:
recipes built: 76
tasks run: 998
master:
recipes built: 133
tasks run: 1600

could you get a compare list which recipes have been newly added? I thought that
minimal image was seldom changed...

2. time
green release:
real: 28m7s
user: 57m45s
sys: 9m41s
user+sys: 67m26s
master:
real: 66m39s
user: 152m17s
sys: 27m37s
user+sys: 179m54s

3. space (haven't analyze though)
green release: 7.8G
master: 26.6G

I recall that RP mentioned debug symbol is enabled in current master, but I'm not sure
whether green has done the same. You may double-confirm that part first.

II. profiling
I tried a brief profile by collecting the time used for every task, so
we can scrutinize the result from a microscopic point of view. I'm still
looking into the full result, but there's something of immediate
attention.

In master, hardly any task consume less than 1.3s, this is quite
surprising, since many tasks like do_patch virtually do nothing, while
in the green release, these tasks may simply consume 0.1~0.2s. A deeper
investigation shows that this large overhead goes to bitbake-runtask,
the bitbake config and cache mechanism is executed for every task,
considerably increased the time. The overhead introduced solely by
this is around 1600*1.3=2080s, approximately 35 minutes (user+sys).

It's said this change comes from pseudo integration. Now it's time for us to
revisit this implementation then. RP can have more insights here.

Also, we should count the additional rpm packaging system,
do_package_write_rpm costs around 1400s, (excluding 1.3s per task,
btw, this is about 50% slower than do_package_write_ipk, in average),
that's around 23 minutes.

Roughly considering that the build time is proportional to recipes
count, we can try to estimate master build time from green release:
67 * (76 / 133) + 35 + 23 = 175
very close to the real time consuming (although somewhat too closed...),
so possibly the above two are the most significant time consumers
in the slowness of current poky

above are all good findings!

Thanks
Kevin

Hello everyone!
 
 I have a question. I have just read about the yocto/poky projects and here I am with a hope. :)
 
 I have a nokia n800 (an internet tablet abandoned by Nokia just about 2 years ago).
 
 With poky/yocto project tools, is it possible to install a linux toolchain with X and the touchscreen/wireless components working, for an example, in a SD Card and boot my device into it?
 
 I'm anxious for that.
 
 Thanks for your support.
 
 Regard,
 

--
***
TIAGOPRN
Desenvolvedor Web, Linux e Windows
(Django, Python, Delphi, Lazarus, MySQL, SQLite)
E-mail: tiagoprn@...
Blog: http://tgplima.net84.net
LinkedIn (profile público): http://br.linkedin.com/in/tiagoparanhoslima
twitter: https://twitter.com/tiagoprn

On 10-10-29 07:50 AM, tiagoprn wrote:

Hello everyone!

I have a question. I have just read about the yocto/poky projects and here I am with a hope. :)

I have a nokia n800 (an internet tablet abandoned by Nokia just about 2 years ago).

With poky/yocto project tools, is it possible to install a linux toolchain with X and the touchscreen/wireless components working, for an example, in a SD Card and boot my device into it?

I can comment on some of this from the kernel point of view,
I'll leave others to comment on platform/project/userspace
issues.

It is definitely possible to enable (basic or more capabilities)
in the N800 (in particular since most of the required support
is mainline) in the yocto kernel (or other). We could extend
the common set of features and sanity to the n800 and then
enable peripherals or boot methods .. it all depends on the
capabilities and interest.

maintable/doable in the kernel and userspace has the required
capabilities (from my point of view), so this is something
that could come back (others can comment more on the previous
support), with a little bit of assistance.

Cheers,

Bruce

I'm anxious for that.

Thanks for your support.

Regard,

Folks,

Please find attached a list of packages that are in the world build but not currently contained in any tasks or images. This is an early warning that we are putting some of these up for consideration for relocation.

As we work through this determination process, we will document it to use as in the future for package relocation. There maybe recipes in this list that will move into some specific tasks or be moved to an external layer.

Please review this spreadsheet and provide any input for these packages.

Thanks
Sau!

Saul Wold
Yocto Component Wrangler @ Intel
Yocto Project / Poky Build System

We need to review the existing Bugzilla and update the Products and Categories to reflect the projects correctly. Please review this email and make comments, suggestions for moving forward with a better Bugzilla categorization.

Currently we have "Core OS" with the following Components:
General
Graphics Driver
Kernel
Tool Chain

Along with "Poky" which contains:
General
SDK Tools

There are also product categories for "Runtime Distribution", "Sato" and "SDK Plugins". Along with other infrastructure items.

I would propose that we clearly define the some new products and move bugs as appropriate:

Poky Build System - for Poky class and configuration issues
User Space - for user space, patching and runtime failures
Tool Chain - break it down to compiler, tools, libraries, and general
Kernel - Break it down to Arch / Config components
SDK - For all SDK related issues, have components for plugin, tools, ...
Sato - as it exists today
Runtime Distribution- Delete this, we are not a distro (no bugs now)

Additionally, there is other discussion about Poky Test components for the standards tests such as LSB, LTP, Posix.

We will need to add Product Categories for other Yocto Projects that do not have bugzilla yet.

Finally we need to update the Bugzilla Interface to be Yocto Project, changing naming as appropriate.

Please take a few minutes to review this and give some feedback.


Thanks

Sau!

Saul Wold
Yocto Component Wrangler @ Intel
Yocto Project / Poky Build System

Folks,

I just completed a world build and the situation is not that bad currently, the summary of the failures are listed below.


libsndfile1_1.0.17
- Failed in compilation of flac.c
loudmouth_1.4.0
- Failed in compilation of asyncns.c
- asyncns.c:159:14: error: static declaration of 'strndup' follows non-static declaration
plugin-evolution2_0.36
- Failed in do_configure
CMake Error at cmake/modules/FindEPackage.cmake:166 (MESSAGE):
Evolution Data Server not found.
Call Stack (most recent call first):
CMakeLists.txt:9 (FIND_PACKAGE)
clutter-gtk-0.10_git
- Failed during linking
ohm_git
- Failed during compilation of ohm-plugin-dpms.c
- missing X11/extensions/dpmsstr.h
git_1.7.2.1
- Failed during do_configure
- Need a configure update for checking of "fopen'ed directory"
openswan_2.4.7
- Failed during compilation of optionsfrom.c
- optionsfrom.c:34:14: error: conflicting types for 'getline'
clutter-gtk-1.0_git
- Failed during linking
gnome-terminal_2.26.3
- Failed during linking
matchbox-themes-extra_svn
- Failed during install with an overwrite error
- will not overwrite just-created `/intel/poky/demo/build/tmp/work/core2-poky-linux/matchbox-themes-extra-0.3+svnr1524-r0/image/usr/share/themes/mbcrystal/matchbox/theme.desktop' with `theme.desktop'
libapplewm_1.0.0
- Failed during compilation of applewm.c
- missing X11/extensions/applewmstr.h
telepathy-idle_0.1.2
- Failed during compilation of idle-connection.c


Bugs will be files soon on these items.

Sau!

Saul Wold
Yocto Component Wrangler @ Intel
Yocto Project / Poky Build System

On Fri, 2010-10-29 at 04:50 -0700, tiagoprn wrote:

Hello everyone!

I have a question. I have just read about the yocto/poky projects and
here I am with a hope. :)

I have a nokia n800 (an internet tablet abandoned by Nokia just about
2 years ago).

With poky/yocto project tools, is it possible to install a linux
toolchain with X and the touchscreen/wireless components working, for
an example, in a SD Card and boot my device into it?

I'm anxious for that.

Hi,

It looks like yocto already has support for the n800 - there's this
entry in local.conf:

# Other supported machines
.
.






#MACHINE ?= "nokia800"


and see this section in README.hardware:

'Nokia 770/N800/N810 Internet Tablets (nokia770 and nokia800)'

which says:

"The nokia800 images and kernel will run on both the N800 and N810"

Sounds like what you're looking for...

Tom