I have used both git and the tarball methods of bitbaking projects, all of them derivatives of the examples in the Yocto documentation.  I was having issues using the local clone of the Yocto kernel git repository this weekend.  I had successfully done that before, but I was rebuilding the PC workstation, and getting everything setup and tested some of the meta-intel BSPs to make sure I had everything right.  Cloning the linux-yocto-3.0 repository was successful, but the bakes against it failed. I made sure I had poky-extras setup right, but I still had problems.  To isolate the problem, I changed to building with the tarballs and everything worked fine.

On 2012-01-22 13:19, James Abernathy wrote:

I have used both git and the tarball methods of bitbaking projects, all of them derivatives of the examples in the Yocto documentation. I was having issues using the local clone of
the Yocto kernel git repository this weekend. I had successfully done that before, but I was rebuilding the PC workstation, and getting everything setup and tested some of the
meta-intel BSPs to make sure I had everything right. Cloning the linux-yocto-3.0 repository was successful, but the bakes against it failed. I made sure I had poky-extras setup
right, but I still had problems. To isolate the problem, I changed to building with the tarballs and everything worked fine.

So that got me thinking what are the differences between the 2 methods:

* I assume that if I use the tarball method, bitbake, using the recipes, pulls down files from the online repositories and puts those files into the centralized local download
directory ($DL_DIR), allowing reuse instead of re-downloading each time. The content downloaded for linux-yocto-3.0 is exactly what would be pulled from the local repository if
I used a local clone of the git repository for linux-yocto-3.0.
* If my assumption above is correct, if I'm not modifying the source code of the kernel (only changing config parameters), then once you've run at least one build with the
tarball method, the $DL_DIR directory contains all the files you'll need to build any image with linux-yocto-3.0. So there is no need to have a local clone of the kernel
repository for speeding up development. Am I right?
* If I have a successful creation of a bare clone of linux-yocto-3.0.git, how could builds of Edison packages be failing? That makes me concerned about using git and successfully
repeating builds of stable branches like Edison.

If you set BB_GENERATE_MIRROR_TARBALLS = "1" (e.g. in local.conf)
then you'll get tarballs which hold the git repositories after
download. You can then reuse these (by sharing the DL_DIR or
using a local mirror). Does that help with the issue you're seeing?

--
------------------------------------------------------------
Gary Thomas | Consulting for the
MLB Associates | Embedded world
------------------------------------------------------------

On 01/22/2012 08:12 PM, Gary Thomas wrote:

On 2012-01-22 13:19, James Abernathy wrote:

I have used both git and the tarball methods of bitbaking projects, all of them derivatives of the examples in the Yocto documentation. I was having issues using the local clone of
the Yocto kernel git repository this weekend. I had successfully done that before, but I was rebuilding the PC workstation, and getting everything setup and tested some of the
meta-intel BSPs to make sure I had everything right. Cloning the linux-yocto-3.0 repository was successful, but the bakes against it failed. I made sure I had poky-extras setup
right, but I still had problems. To isolate the problem, I changed to building with the tarballs and everything worked fine.

So that got me thinking what are the differences between the 2 methods:

* I assume that if I use the tarball method, bitbake, using the recipes, pulls down files from the online repositories and puts those files into the centralized local download
directory ($DL_DIR), allowing reuse instead of re-downloading each time. The content downloaded for linux-yocto-3.0 is exactly what would be pulled from the local repository if
I used a local clone of the git repository for linux-yocto-3.0.
* If my assumption above is correct, if I'm not modifying the source code of the kernel (only changing config parameters), then once you've run at least one build with the
tarball method, the $DL_DIR directory contains all the files you'll need to build any image with linux-yocto-3.0. So there is no need to have a local clone of the kernel
repository for speeding up development. Am I right?
* If I have a successful creation of a bare clone of linux-yocto-3.0.git, how could builds of Edison packages be failing? That makes me concerned about using git and successfully
repeating builds of stable branches like Edison.

If you set BB_GENERATE_MIRROR_TARBALLS = "1" (e.g. in local.conf)
then you'll get tarballs which hold the git repositories after
download. You can then reuse these (by sharing the DL_DIR or
using a local mirror). Does that help with the issue you're seeing?

I'm not sure it does. I don't want poky to do more work. I have my download directory, $DL_DIR, outside my build directory so I can keep it run to run, as mentioned in the comments in local.conf. I was trying to understand 2 basic questions:

1. what could be causing build failures using a freshly created bare clone yesterday vs. using the poky-edison-6.0 tarball. It would be scary if you could clone the linux-yocto-3.0 successfully one day and have it be used in a build successfully, but clone it another day and it not work. I figured that bitbake/poky pulled the same information into DL_DIR regardless of whether you pulled from the bare clone locally or straight from the on-line repository.
2. I was trying to look at items to speed up build runs. I thought that if the downloads in DL_DIR were reused if they existed, it would speed up a run. It seems to. So the question is, after the first build, the files I need for linux-yocto-3.0 are in DL_DIR regardless of whether they came from the online repository or the local bare clone. right?

Jim A

On 2012-01-23 05:51, jfabernathy wrote:

On 01/22/2012 08:12 PM, Gary Thomas wrote:

On 2012-01-22 13:19, James Abernathy wrote:

I have used both git and the tarball methods of bitbaking projects, all of them derivatives of the examples in the Yocto documentation. I was having issues using the local clone of
the Yocto kernel git repository this weekend. I had successfully done that before, but I was rebuilding the PC workstation, and getting everything setup and tested some of the
meta-intel BSPs to make sure I had everything right. Cloning the linux-yocto-3.0 repository was successful, but the bakes against it failed. I made sure I had poky-extras setup
right, but I still had problems. To isolate the problem, I changed to building with the tarballs and everything worked fine.

So that got me thinking what are the differences between the 2 methods:

* I assume that if I use the tarball method, bitbake, using the recipes, pulls down files from the online repositories and puts those files into the centralized local download
directory ($DL_DIR), allowing reuse instead of re-downloading each time. The content downloaded for linux-yocto-3.0 is exactly what would be pulled from the local repository if
I used a local clone of the git repository for linux-yocto-3.0.
* If my assumption above is correct, if I'm not modifying the source code of the kernel (only changing config parameters), then once you've run at least one build with the
tarball method, the $DL_DIR directory contains all the files you'll need to build any image with linux-yocto-3.0. So there is no need to have a local clone of the kernel
repository for speeding up development. Am I right?
* If I have a successful creation of a bare clone of linux-yocto-3.0.git, how could builds of Edison packages be failing? That makes me concerned about using git and successfully
repeating builds of stable branches like Edison.

If you set BB_GENERATE_MIRROR_TARBALLS = "1" (e.g. in local.conf)
then you'll get tarballs which hold the git repositories after
download. You can then reuse these (by sharing the DL_DIR or
using a local mirror). Does that help with the issue you're seeing?

I'm not sure it does. I don't want poky to do more work. I have my download directory, $DL_DIR, outside my build directory so I can keep it run to run, as mentioned in the comments
in local.conf. I was trying to understand 2 basic questions:

1. what could be causing build failures using a freshly created bare clone yesterday vs. using the poky-edison-6.0 tarball. It would be scary if you could clone the linux-yocto-3.0
successfully one day and have it be used in a build successfully, but clone it another day and it not work. I figured that bitbake/poky pulled the same information into DL_DIR
regardless of whether you pulled from the bare clone locally or straight from the on-line repository.

What kind of errors? Some details might help understand what the problem is.

2. I was trying to look at items to speed up build runs. I thought that if the downloads in DL_DIR were reused if they existed, it would speed up a run. It seems to. So the
question is, after the first build, the files I need for linux-yocto-3.0 are in DL_DIR regardless of whether they came from the online repository or the local bare clone. right?

I think so, but I'm not 100% sure.

The way I have my system set up, I never download anything more
than once :-) I have a shared source repository which I point to
using MIRRORS and then all my builds are relative to that. If there
is new code, it gets downloaded and saved (as a tarball) in my sources
repository to be used by subsequent builds. To do this, I just have
these lines in <distro>.conf
# Provide pre-staged sources
SOURCE_MIRROR_URL ?= "file://${COREBASE}/sources/"
INHERIT += "own-mirrors"
BB_GENERATE_MIRROR_TARBALLS = "1"
I don't define DL_DIR in local.conf so files only come from my mirror.

This process is so successful that I almost always run with BB_NO_NETWORK="1"
which causes the fetcher to die if it can't find the file locally. If I find
that I do need to download something new, I disable this and let the fetcher
download it. I'll then have a saved tarball (either downloaded directly or
synthesized) in <build_dir>/downloads that I can push to my mirror.

--
------------------------------------------------------------
Gary Thomas | Consulting for the
MLB Associates | Embedded world
------------------------------------------------------------

On 12-01-23 08:10 AM, Gary Thomas wrote:

On 2012-01-23 05:51, jfabernathy wrote:

On 01/22/2012 08:12 PM, Gary Thomas wrote:

On 2012-01-22 13:19, James Abernathy wrote:

I have used both git and the tarball methods of bitbaking projects,
all of them derivatives of the examples in the Yocto documentation.
I was having issues using the local clone of
the Yocto kernel git repository this weekend. I had successfully
done that before, but I was rebuilding the PC workstation, and
getting everything setup and tested some of the
meta-intel BSPs to make sure I had everything right. Cloning the
linux-yocto-3.0 repository was successful, but the bakes against it
failed. I made sure I had poky-extras setup
right, but I still had problems. To isolate the problem, I changed
to building with the tarballs and everything worked fine.

So that got me thinking what are the differences between the 2 methods:

* I assume that if I use the tarball method, bitbake, using the
recipes, pulls down files from the online repositories and puts
those files into the centralized local download
directory ($DL_DIR), allowing reuse instead of re-downloading each
time. The content downloaded for linux-yocto-3.0 is exactly what
would be pulled from the local repository if
I used a local clone of the git repository for linux-yocto-3.0.
* If my assumption above is correct, if I'm not modifying the source
code of the kernel (only changing config parameters), then once
you've run at least one build with the
tarball method, the $DL_DIR directory contains all the files you'll
need to build any image with linux-yocto-3.0. So there is no need to
have a local clone of the kernel
repository for speeding up development. Am I right?
* If I have a successful creation of a bare clone of
linux-yocto-3.0.git, how could builds of Edison packages be failing?
That makes me concerned about using git and successfully
repeating builds of stable branches like Edison.

If you set BB_GENERATE_MIRROR_TARBALLS = "1" (e.g. in local.conf)
then you'll get tarballs which hold the git repositories after
download. You can then reuse these (by sharing the DL_DIR or
using a local mirror). Does that help with the issue you're seeing?

I'm not sure it does. I don't want poky to do more work. I have my
download directory, $DL_DIR, outside my build directory so I can keep
it run to run, as mentioned in the comments
in local.conf. I was trying to understand 2 basic questions:

1. what could be causing build failures using a freshly created bare
clone yesterday vs. using the poky-edison-6.0 tarball. It would be
scary if you could clone the linux-yocto-3.0
successfully one day and have it be used in a build successfully, but
clone it another day and it not work. I figured that bitbake/poky
pulled the same information into DL_DIR
regardless of whether you pulled from the bare clone locally or
straight from the on-line repository.

What kind of errors? Some details might help understand what the problem
is.

2. I was trying to look at items to speed up build runs. I thought
that if the downloads in DL_DIR were reused if they existed, it would
speed up a run. It seems to. So the
question is, after the first build, the files I need for
linux-yocto-3.0 are in DL_DIR regardless of whether they came from the
online repository or the local bare clone. right?

I think so, but I'm not 100% sure.

From my experience, this is true. Regardless of the SRCI_URI, the
downloads directory contains everything that poky/bitbake need to
do builds after the initial fetch has been performed. And that's
where subsequent updates are pullled.

Cheers,

Bruce

The way I have my system set up, I never download anything more
than once :-) I have a shared source repository which I point to
using MIRRORS and then all my builds are relative to that. If there
is new code, it gets downloaded and saved (as a tarball) in my sources
repository to be used by subsequent builds. To do this, I just have
these lines in <distro>.conf
# Provide pre-staged sources
SOURCE_MIRROR_URL ?= "file://${COREBASE}/sources/"
INHERIT += "own-mirrors"
BB_GENERATE_MIRROR_TARBALLS = "1"
I don't define DL_DIR in local.conf so files only come from my mirror.

This process is so successful that I almost always run with
BB_NO_NETWORK="1"
which causes the fetcher to die if it can't find the file locally. If I
find
that I do need to download something new, I disable this and let the
fetcher
download it. I'll then have a saved tarball (either downloaded directly or
synthesized) in <build_dir>/downloads that I can push to my mirror.

On Jan 23, 2012, at 9:01 AM, Bruce Ashfield wrote:

On 12-01-23 08:10 AM, Gary Thomas wrote:

On 2012-01-23 05:51, jfabernathy wrote:

On 01/22/2012 08:12 PM, Gary Thomas wrote:

On 2012-01-22 13:19, James Abernathy wrote:

I have used both git and the tarball methods of bitbaking projects,
all of them derivatives of the examples in the Yocto documentation.
I was having issues using the local clone of
the Yocto kernel git repository this weekend. I had successfully
done that before, but I was rebuilding the PC workstation, and
getting everything setup and tested some of the
meta-intel BSPs to make sure I had everything right. Cloning the
linux-yocto-3.0 repository was successful, but the bakes against it
failed. I made sure I had poky-extras setup
right, but I still had problems. To isolate the problem, I changed
to building with the tarballs and everything worked fine.

So that got me thinking what are the differences between the 2 methods:

* I assume that if I use the tarball method, bitbake, using the
recipes, pulls down files from the online repositories and puts
those files into the centralized local download
directory ($DL_DIR), allowing reuse instead of re-downloading each
time. The content downloaded for linux-yocto-3.0 is exactly what
would be pulled from the local repository if
I used a local clone of the git repository for linux-yocto-3.0.
* If my assumption above is correct, if I'm not modifying the source
code of the kernel (only changing config parameters), then once
you've run at least one build with the
tarball method, the $DL_DIR directory contains all the files you'll
need to build any image with linux-yocto-3.0. So there is no need to
have a local clone of the kernel
repository for speeding up development. Am I right?
* If I have a successful creation of a bare clone of
linux-yocto-3.0.git, how could builds of Edison packages be failing?
That makes me concerned about using git and successfully
repeating builds of stable branches like Edison.

If you set BB_GENERATE_MIRROR_TARBALLS = "1" (e.g. in local.conf)
then you'll get tarballs which hold the git repositories after
download. You can then reuse these (by sharing the DL_DIR or
using a local mirror). Does that help with the issue you're seeing?

I'm not sure it does. I don't want poky to do more work. I have my
download directory, $DL_DIR, outside my build directory so I can keep
it run to run, as mentioned in the comments
in local.conf. I was trying to understand 2 basic questions:

1. what could be causing build failures using a freshly created bare
clone yesterday vs. using the poky-edison-6.0 tarball. It would be
scary if you could clone the linux-yocto-3.0
successfully one day and have it be used in a build successfully, but
clone it another day and it not work. I figured that bitbake/poky
pulled the same information into DL_DIR
regardless of whether you pulled from the bare clone locally or
straight from the on-line repository.

What kind of errors? Some details might help understand what the problem
is.

2. I was trying to look at items to speed up build runs. I thought
that if the downloads in DL_DIR were reused if they existed, it would
speed up a run. It seems to. So the
question is, after the first build, the files I need for
linux-yocto-3.0 are in DL_DIR regardless of whether they came from the
online repository or the local bare clone. right?

I think so, but I'm not 100% sure.

From my experience, this is true. Regardless of the SRCI_URI, the
downloads directory contains everything that poky/bitbake need to
do builds after the initial fetch has been performed. And that's
where subsequent updates are pullled.

Cheers,

Bruce

I'm going to retest all this and try some of these tricks when I have some time. If I get any more errors with the local repository, I'll post them.

Thanks, Jim A

The way I have my system set up, I never download anything more
than once :-) I have a shared source repository which I point to
using MIRRORS and then all my builds are relative to that. If there
is new code, it gets downloaded and saved (as a tarball) in my sources
repository to be used by subsequent builds. To do this, I just have
these lines in <distro>.conf
# Provide pre-staged sources
SOURCE_MIRROR_URL ?= "file://${COREBASE}/sources/"
INHERIT += "own-mirrors"
BB_GENERATE_MIRROR_TARBALLS = "1"
I don't define DL_DIR in local.conf so files only come from my mirror.

This process is so successful that I almost always run with
BB_NO_NETWORK="1"
which causes the fetcher to die if it can't find the file locally. If I
find
that I do need to download something new, I disable this and let the
fetcher
download it. I'll then have a saved tarball (either downloaded directly or
synthesized) in <build_dir>/downloads that I can push to my mirror.