Reproducible builds and RPM packages


Anders Montonen
 

Hi,

When going from Zeus to Dunfell, I noticed that all files on the rootfs had timestamps long in the past, which I assume is from reproducible builds now being on by default. While that is a good thing, running “rpm -V” on any installed package now reports that the mtime differs. Is this the intentional behavior?

Regards,
Anders Montonen


Randy MacLeod
 

On 2020-11-03 6:16 a.m., Anders Montonen wrote:
Hi,

When going from Zeus to Dunfell, I noticed that all files on the rootfs had timestamps long in the past, which I assume is from reproducible builds now being on by default. While that is a good thing, running “rpm -V” on any installed package now reports that the mtime differs. Is this the intentional behavior?

Hi Anders,

I haven't played with that for a while but I'm pretty sure the answer is yes, it's intentional.

You can read about reproducible builds here:
   https://wiki.yoctoproject.org/wiki/Reproducible_Builds
and compare to the source if needed:
   https://git.openembedded.org/openembedded-core/tree/meta/classes/reproducible_build.bbclass?id=189630ca6cdf7ceb6cf9b8f9d86c58997f505efc&h=dunfell

I don't see it in the documentation yet:
   http://docs.yoctoproject.org/search.html?q=reproducible&check_keywords=yes&area=default

but I didn't do more than skim those search results.

If you also can't find it in the docs, would you be able to make a first draft of that and contribute to the project?

If so please read and ask for guidance on: docs@...

Thanks,

../Randy


Regards,
Anders Montonen




-- 
# Randy MacLeod
# Wind River Linux


Anders Montonen
 



On 4 Nov 2020, at 23:23, Randy MacLeod <randy.macleod@...> wrote:

On 2020-11-03 6:16 a.m., Anders Montonen wrote:
Hi,

When going from Zeus to Dunfell, I noticed that all files on the rootfs had timestamps long in the past, which I assume is from reproducible builds now being on by default. While that is a good thing, running “rpm -V” on any installed package now reports that the mtime differs. Is this the intentional behavior?

Hi Anders,

I haven't played with that for a while but I'm pretty sure the answer is yes, it's intentional.

You can read about reproducible builds here:
   https://wiki.yoctoproject.org/wiki/Reproducible_Builds
and compare to the source if needed:
   https://git.openembedded.org/openembedded-core/tree/meta/classes/reproducible_build.bbclass?id=189630ca6cdf7ceb6cf9b8f9d86c58997f505efc&h=dunfell



Just to be clear, I’m referring to the RPM metadata. If you build a package, and then run "rpm -q --qf '[%{FILENAMES} %{FILEMTIMES}\n]' -p package” on the output, the listed timestamps will not match either SOURCE_DATE_EPOCH, or REPRODUCIBLE_TIMESTAMP_ROOTFS. When you build an image, the timestamps in the filesystem will not be consistent with the timestamps in the RPM database, leading to errors if you try to verify an installed package. I would consider this a bug or an oversight.

Regards,
Anders Montonen


Jate Sujjavanich
 

I have sussed out several behaviors of the image build having to do with reproducible builds. It seems like bitbake creates the rpm with the correct modification times per the reproducible_builds bbclass. When do_rootfs installs the packages, the timestamps seem correct.

The function reproducible_final_image_task sets all files in rootfs to be REPRODUCIBLE_TIMESTAMP_ROOTFS which is the last commit time in poky by default. This occurs after package installation and before image creation.

When the image creation code runs, it seems like each utility has different behavior on respecting SOURCE_DATE_EPOCH in the environment. The ext3 utility allowed mtime's beyond the default value of 0. The squashfs utility enforced the restriction of no mtime's being beyond 0 seconds since the epoch. Once I set this to the build timestamp (DATETIME) in the image recipe, the squashfs image had correct times.

So to get this working, you'd need to disable reproducible_final_image_task, and make sure the image creation utility is not wiping out the modification times of files created by rpm.

I'd argue that we'd want reproducible_final_image_task to not override all the mtime's of files. We probably have to check the other file system type to see their behavior with SOURCE_DATE_EPOCH as well.

- Jate


Richard Purdie
 

On Thu, 2020-11-12 at 15:43 -0800, Jate Sujjavanich wrote:
I have sussed out several behaviors of the image build having to do
with reproducible builds. It seems like bitbake creates the rpm with
the correct modification times per the reproducible_builds bbclass.
When do_rootfs installs the packages, the timestamps seem correct.

The function reproducible_final_image_task sets all files in rootfs
to be REPRODUCIBLE_TIMESTAMP_ROOTFS which is the last commit time in
poky by default. This occurs after package installation and before
image creation.

When the image creation code runs, it seems like each utility has
different behavior on respecting SOURCE_DATE_EPOCH in the
environment. The ext3 utility allowed mtime's beyond the default
value of 0. The squashfs utility enforced the restriction of no
mtime's being beyond 0 seconds since the epoch. Once I set this to
the build timestamp (DATETIME) in the image recipe, the squashfs
image had correct times.

So to get this working, you'd need to disable
reproducible_final_image_task, and make sure the image creation
utility is not wiping out the modification times of files created by
rpm.

I'd argue that we'd want reproducible_final_image_task to not
override all the mtime's of files. We probably have to check the
other file system type to see their behavior with SOURCE_DATE_EPOCH
as well.
Thanks for mentioning that, it does sound like that is doing something
we no longer should be doing now source_date_epoch is working in most
packages properly.

There are some files such as the rpm database and any other files not
under control of package management which would need this handling to
have the images be 100% reproducible so its not entirely
straightforward though.

Cheers,

Richard