Hi Richard,

This afternoon I performed a build from scratch using master, which included your git fetcher change to use mirror tarballs (1077021f7010388e889d85630b13c4c7901d7718).

When I performed my build from scratch (using a pre-populated DL_DIR from previous builds), linux-wrs failed on do_fetch, with the following in log.do_fetch:

From git://git.pokylinux.org/linux-2.6-windriver
* branch common_pc-standard -> FETCH_HEAD
From git://git.pokylinux.org/linux-2.6-windriver
* [new tag] v2.6.36-rc5 -> v2.6.36-rc5
* [new tag] v2.6.36-rc6 -> v2.6.36-rc6
NOTE: Creating tarball of git repository
ERROR: Task failed: Unknown fetch Error: coercing to Unicode: need string or buffer, NoneType found

I performed a -c clean and then -c fetch, but ran into the same problem.

I then went and deleted the linux-wrs-related sources from DL_DIR, and ran -c clean and -c fetch again, and it looks to me like the fetcher is actually downloading tarballs from the source mirrors first, and then trying git directly:

NOTE: Running task 130 of 130 (ID: 0, /raid0/sources/poky/meta/recipes-kernel/linux/linux-wrs_git.bb, do_fetch)
NOTE: package linux-wrs-2.6.34+git0+b67e060194a38c6331da1532bd06446087a42b3b_0+0431115c9d720fee5bb105f6a7411efb4f851d26-r12: task do_fetch: Started
NOTE: fetch http://pokylinux.org/sources/git_git.pokylinux.org.linux-2.6-windriver.git.tar.gz
NOTE: fetch http://autobuilder.pokylinux.org/sources/git_git.pokylinux.org.linux-2.6-windriver.git.tar.gz

I confirmed this by seeing wget running in my process list, and then git eventually shows up.

Finally, I get the bitbake message:

NOTE: Creating tarball of git repository

...and I see gzip running, but the task ends up failing with the same error as the log.do_fetch example I mentioned above.

Scott

On 10/18/2010 05:54 PM, Scott Garman wrote:

Hi Richard,

This afternoon I performed a build from scratch using master, which
included your git fetcher change to use mirror tarballs
(1077021f7010388e889d85630b13c4c7901d7718).

As a follow-up, when I revert this commit, -c clean and then -c fetch succeeds for linux-wrs.

Scott

On Mon, 2010-10-18 at 18:00 -0700, Scott Garman wrote:

On 10/18/2010 05:54 PM, Scott Garman wrote:

Hi Richard,

This afternoon I performed a build from scratch using master, which
included your git fetcher change to use mirror tarballs
(1077021f7010388e889d85630b13c4c7901d7718).

Thanks for testing and reporting this quickly, its appreciated.

As a follow-up, when I revert this commit, -c clean and then -c fetch
succeeds for linux-wrs.

but fails for libzypp :/.

I started debugging the fetcher code and the whole thing basically fell
apart as I was finding all kinds of issues, some of which we're just not
hitting by pure luck. I had hoped that code would last us into the 1.0
development window when I've some significant fetcher changes in mind.
Short summary is that its not going to do that.

I've pushed some more invasive changes to the fetcher in the Poky master
branch. I'd appreciate the teams help in finding out if these work or
not, with both existing builds and with totally fresh from scratch
builds.

If they prove not to have major issues we're likely to merge these into
the release branch as what is there is going to bite people and look
rather bad.

Cheers,

Richard

On Tue, 2010-10-19 at 13:55 +0100, Richard Purdie wrote:

I've pushed some more invasive changes to the fetcher in the Poky master
branch. I'd appreciate the teams help in finding out if these work or
not, with both existing builds and with totally fresh from scratch
builds.

If they prove not to have major issues we're likely to merge these into
the release branch as what is there is going to bite people and look
rather bad.

There were some issues that Tom and Joshua were seeing which I debugged
and found that we weren't adding in new branches in the fetching code.

I was able to reproduce and then have added a patch to the git fetcher
which ensures these things are cloned correctly and any new branches are
created. I talked with Bruce to confirm my diagnosis and the solution.

This fix makes an awful lot of sense and explains various issues we've
been chasing for a while with the fetcher bugs so I'm very pleased to
have this fix checked in.

Could people continue to test the code and report any problems...

Thanks,

Richard

From: Richard Purdie
Sent: Wednesday, October 20, 2010 12:29 AM

On Tue, 2010-10-19 at 13:55 +0100, Richard Purdie wrote:

I've pushed some more invasive changes to the fetcher in the Poky master
branch. I'd appreciate the teams help in finding out if these work or
not, with both existing builds and with totally fresh from scratch
builds.

If they prove not to have major issues we're likely to merge these into
the release branch as what is there is going to bite people and look
rather bad.

There were some issues that Tom and Joshua were seeing which I debugged
and found that we weren't adding in new branches in the fetching code.

I was able to reproduce and then have added a patch to the git fetcher
which ensures these things are cloned correctly and any new branches are
created. I talked with Bruce to confirm my diagnosis and the solution.

This fix makes an awful lot of sense and explains various issues we've
been chasing for a while with the fetcher bugs so I'm very pleased to
have this fix checked in.

Could people continue to test the code and report any problems...

I tried "bitbake poky-image-sdk -c fetchall" both with an existing source directory
and with a fresh new download. Everything works fine.

Thanks
Kevin

On Wed, 2010-10-20 at 16:20 +0800, Tian, Kevin wrote:

I tried "bitbake poky-image-sdk -c fetchall" both with an existing source directory
and with a fresh new download. Everything works fine.

Thanks for testing and the feedback!

Cheers,

Richard