<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 2018-09-17 10:27, Burton, Ross
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAJTo0LbjCN31UUNUP9WX_NxMhoAr0xCt45WrQaWDvqovDeNhYw@mail.gmail.com">
<pre wrap="">On Mon, 17 Sep 2018 at 08:13, Peter Bergin <a class="moz-txt-link-rfc2396E" href="mailto:peter@berginkonsult.se"><peter@berginkonsult.se></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">I'm pretty sure I have narrowed down the root cause to the restriction
of virtual memory and that liblzma base its memory calculations on
physical RAM.
To prove this I added a printout in rpm-native/rpmio/rpmio.c and the
function lzopen_internal.
uint64_t memory_usage = lzma_stream_encoder_mt_memusage(&mt_options);
rpmlog(RPMLOG_NOTICE, "DBG: memory_usage %lu\n", memory_usage);
The value of memory_usage is the same regardless of which 'ulimit -v'
value I set. On the host with 256GB of physical RAM and 32GB of virtual
memory, memory_usage is ~5.1GB. On another host with 16GB of physical
RAM I get memory_usage of ~660MB.
I guess you have not seen this kind of failure if you not have
restricted virutal memory on your host. If you want to try to reproduce
this set 'ulimit -v 8388608' (8GB) in your shell and then 'bitbake
glibc-locale -c package_write_rpm -f'.
</pre>
</blockquote>
<pre wrap="">Wouldn't a solution be to change lzma to look at free memory, not
total physical memory?
Ross
</pre>
</blockquote>
I have been in contact with the maintainer of liblzma. There is
currently no way to restrict the memory usage in liblzma during
multi threaded compression. He recommended to adjust the number of
threads used during compression. This kind of check is done for
32-bits but not when running 64-bits (in rpmio/rpmio.c
lzopen_internal). To rewrite liblzma is another option but I took an
easier way out.<br>
<br>
I have come up with a patch
(<a class="moz-txt-link-freetext" href="https://patchwork.openembedded.org/patch/155017/">https://patchwork.openembedded.org/patch/155017/</a>) that solves my
problem and do a restriction of memory usage when the 'ulimit -v' is
set. The calculation is based on the assumption that lzopen_internal
is run in parallel with as many instances as cpu threads as '#pragma
omp parallel' is used in build/pack.c. <br>
<br>
When running test on my machine with 4 cores 16GB of physical RAM
and 'ulimit -v 2097152' (2GB). It works good and the log shows:<br>
<br>
XZ: virtual memory restricted to 2147483648 and per CPU thread
536870912<br>
XZ: Adjusted the number of threads from 4 to 3 to not exceed the
memory usage limit of 2147483648 bytes<br>
<br>
Didn't get a clear answer if this is something Yocto/OE should
support but I hope my patch solves the issue and it will not affect
the normal environments where 'ulimit -v' is not set.<br>
<br>
/Peter<br>
</body>
</html>