Date   

centos7-ty-4 dmesg - OOM issues

Richard Purdie
 

Hi All,

I've included the dmesg output from centos7-ty-4 below. This happened
while https://autobuilder.yoctoproject.org/typhoon/#/builders/108/builds/1644
was running. The build shows 4 worker tasks exiting unexpectedly with SIGKILL (-9).
The cause was the OOM killer. The data on what was running is interesting/useful. 
Appears to be qemu-arm and qemu-aarch64 using much of the memory. I haven't worked
out what those tasks were doing yet but seems odd in a qemux86 build, there
must have been a different build running in parallel?

Wanted to save this data before it gets lost with the maint reboot.

Cheers,

Richard



[ 17.739308] IPv6: ADDRCONF(NETDEV_CHANGE): enp4s0f0: link becomes ready
[ 28.972675] FS-Cache: Loaded
[ 29.018320] FS-Cache: Netfs 'nfs' registered for caching
[ 29.023237] Key type dns_resolver registered
[ 29.053043] NFS: Registering the id_resolver key type
[ 29.053050] Key type id_resolver registered
[ 29.053051] Key type id_legacy registered
[ 1748.231528] perf: interrupt took too long (2535 > 2500), lowering kernel.perf_event_max_sample_rate to 78000
[ 1984.255013] perf: interrupt took too long (3170 > 3168), lowering kernel.perf_event_max_sample_rate to 63000
[ 2738.306387] perf: interrupt took too long (3970 > 3962), lowering kernel.perf_event_max_sample_rate to 50000
[ 4273.200988] L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details.
[ 4273.204776] IPv6: ADDRCONF(NETDEV_CHANGE): tap0: link becomes ready
[ 4273.226163] IPv6: ADDRCONF(NETDEV_CHANGE): tap1: link becomes ready
[ 4277.195775] IPv6: ADDRCONF(NETDEV_CHANGE): tap2: link becomes ready
[11100.221690] IPv6: ADDRCONF(NETDEV_CHANGE): tap3: link becomes ready
[11351.574432] IPv6: ADDRCONF(NETDEV_CHANGE): tap4: link becomes ready
[11497.451559] perf: interrupt took too long (4973 > 4962), lowering kernel.perf_event_max_sample_rate to 40000
[12875.688229] IPv6: ADDRCONF(NETDEV_CHANGE): tap5: link becomes ready
[12876.451160] IPv6: ADDRCONF(NETDEV_CHANGE): tap6: link becomes ready
[12876.583216] IPv6: ADDRCONF(NETDEV_CHANGE): tap7: link becomes ready
[12877.476481] IPv6: ADDRCONF(NETDEV_CHANGE): tap8: link becomes ready
[12878.526927] IPv6: ADDRCONF(NETDEV_CHANGE): tap9: link becomes ready
[15610.692395] perf: interrupt took too long (6218 > 6216), lowering kernel.perf_event_max_sample_rate to 32000
[17638.201714] kthreadd invoked oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0
[17638.201723] kthreadd cpuset=/ mems_allowed=0-1
[17638.201729] CPU: 14 PID: 2 Comm: kthreadd Kdump: loaded Not tainted 3.10.0-1160.25.1.el7.x86_64 #1
[17638.201732] Hardware name: Silicon Mechanics Rackform R346.v5/X10DRI-T, BIOS 1.1 04/14/2015
[17638.201735] Call Trace:
[17638.201752] [<ffffffff95d8311a>] dump_stack+0x19/0x1b
[17638.201758] [<ffffffff95d7da3a>] dump_header+0x90/0x229
[17638.201767] [<ffffffff9590ec6b>] ? cred_has_capability+0x6b/0x120
[17638.201776] [<ffffffff957c232d>] oom_kill_process+0x2cd/0x490
[17638.201781] [<ffffffff9590ed3c>] ? selinux_capable+0x1c/0x40
[17638.201787] [<ffffffff957c2a1a>] out_of_memory+0x31a/0x500
[17638.201793] [<ffffffff95d7e557>] __alloc_pages_slowpath+0x5db/0x729
[17638.201799] [<ffffffff957c8f96>] __alloc_pages_nodemask+0x436/0x450
[17638.201808] [<ffffffff956c5cd0>] ? insert_kthread_work+0x40/0x40
[17638.201816] [<ffffffff9569876d>] copy_process+0x1dd/0x1a80
[17638.201823] [<ffffffff956e61d1>] ? put_prev_entity+0x31/0x400
[17638.201828] [<ffffffff9569a1c1>] do_fork+0x91/0x330
[17638.201835] [<ffffffff9569a486>] kernel_thread+0x26/0x30
[17638.201840] [<ffffffff956c6781>] kthreadd+0x2e1/0x320
[17638.201847] [<ffffffff956c64a0>] ? kthread_create_on_cpu+0x60/0x60
[17638.201855] [<ffffffff95d95df7>] ret_from_fork_nospec_begin+0x21/0x21
[17638.201860] [<ffffffff956c64a0>] ? kthread_create_on_cpu+0x60/0x60
[17638.201863] Mem-Info:
[17638.201881] active_anon:30291948 inactive_anon:1358082 isolated_anon:0
active_file:452 inactive_file:163 isolated_file:0
unevictable:0 dirty:2 writeback:0 unstable:0
slab_reclaimable:85921 slab_unreclaimable:57676
mapped:1690 shmem:8852 pagetables:67061 bounce:0
free:91998 free_pcp:1021 free_cma:0
[17638.201891] Node 0 DMA free:14160kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[17638.201904] lowmem_reserve[]: 0 1695 64154 64154
[17638.201911] Node 0 DMA32 free:251284kB min:1184kB low:1480kB high:1776kB active_anon:1106512kB inactive_anon:368856kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1973304kB managed:1735704kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:128kB slab_unreclaimable:196kB kernel_stack:64kB pagetables:1772kB unstable:0kB bounce:0kB free_pcp:248kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:23 all_unreclaimable? yes
[17638.201923] lowmem_reserve[]: 0 0 62459 62459
[17638.201929] Node 0 Normal free:48508kB min:43744kB low:54680kB high:65616kB active_anon:58997720kB inactive_anon:2519156kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63958384kB mlocked:0kB dirty:0kB writeback:0kB mapped:144kB shmem:120kB slab_reclaimable:169656kB slab_unreclaimable:116132kB kernel_stack:6560kB pagetables:135728kB unstable:0kB bounce:0kB free_pcp:3364kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:148 all_unreclaimable? yes
[17638.201940] lowmem_reserve[]: 0 0 0 0
[17638.201947] Node 1 Normal free:54040kB min:45164kB low:56452kB high:67744kB active_anon:61063560kB inactive_anon:2544316kB active_file:2140kB inactive_file:1104kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66039288kB mlocked:0kB dirty:8kB writeback:0kB mapped:6632kB shmem:35288kB slab_reclaimable:173900kB slab_unreclaimable:114376kB kernel_stack:5984kB pagetables:130744kB unstable:0kB bounce:0kB free_pcp:576kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:8077 all_unreclaimable? yes
[17638.201957] lowmem_reserve[]: 0 0 0 0
[17638.201963] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 0*32kB 1*64kB (U) 0*128kB 1*256kB (U) 1*512kB (U) 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 14160kB
[17638.201983] Node 0 DMA32: 36*4kB (U) 43*8kB (UM) 35*16kB (UM) 30*32kB (UEM) 13*64kB (UEM) 5*128kB (UM) 2*256kB (EM) 3*512kB (UEM) 2*1024kB (EM) 1*2048kB (E) 59*4096kB (UM) = 251288kB
[17638.202024] Node 0 Normal: 12322*4kB (UEM) 5*8kB (EM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 49328kB
[17638.202040] Node 1 Normal: 13118*4kB (UEM) 400*8kB (UEM) 44*16kB (UEM) 1*32kB (E) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 56408kB
[17638.202059] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[17638.202063] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[17638.202066] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[17638.202069] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[17638.202072] 19903 total pagecache pages
[17638.202076] 9847 pages in swap cache
[17638.202080] Swap cache stats: add 1051677, delete 1041830, find 313/424
[17638.202082] Free swap = 0kB
[17638.202084] Total swap = 4194300kB
[17638.202087] 33527464 pages RAM
[17638.202089] 0 pages HighMem/MovableOnly
[17638.202091] 590147 pages reserved
[17638.202093] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[17638.202119] [ 962] 0 962 9764 1527 25 0 0 systemd-journal
[17638.202124] [ 997] 0 997 50277 833 31 0 0 lvmetad
[17638.202129] [ 1009] 0 1009 12390 790 26 0 -1000 systemd-udevd
[17638.202136] [ 1361] 0 1361 13883 101 26 0 -1000 auditd
[17638.202141] [ 1392] 0 1392 22652 231 47 0 0 rngd
[17638.202145] [ 1394] 0 1394 5479 146 15 0 0 irqbalance
[17638.202150] [ 1396] 0 1396 6596 73 19 0 0 systemd-logind
[17638.202155] [ 1397] 81 1397 16621 194 34 0 -900 dbus-daemon
[17638.202160] [ 1430] 0 1430 119577 992 85 0 0 NetworkManager
[17638.202165] [ 1439] 997 1439 153083 2380 62 0 0 polkitd
[17638.202169] [ 1444] 996 1444 5634 70 17 0 0 chronyd
[17638.202174] [ 1464] 0 1464 28853 65 11 0 0 ksmtuned
[17638.202179] [ 1473] 0 1473 50358 124 38 0 0 gssproxy
[17638.202184] [ 1512] 0 1512 6477 51 18 0 0 atd
[17638.202188] [ 1513] 0 1513 31598 158 19 0 0 crond
[17638.202193] [ 1615] 0 1615 28235 257 58 0 -1000 sshd
[17638.202197] [ 1618] 0 1618 143572 3366 98 0 0 tuned
[17638.202202] [ 1629] 802 1629 1518412 30179 321 534 0 agent
[17638.202207] [ 1633] 802 1633 734699 7723 111 3 0 trace-agent
[17638.202212] [ 1811] 0 1811 5840 103 14 0 0 ipmievd
[17638.202216] [ 1863] 0 1863 27552 33 10 0 0 agetty
[17638.202221] [ 2678] 0 2678 22437 270 43 0 0 master
[17638.202226] [ 2740] 89 2740 22507 273 45 0 0 qmgr
[17638.202231] [ 2905] 6000 2905 77948 3608 103 0 0 Xvnc
[17638.202235] [ 4069] 6000 4069 43378 1052 40 0 0 xterm
[17638.202240] [ 4071] 6000 4071 28920 123 13 0 0 bash
[17638.202244] [ 4088] 0 4088 25766 557 52 0 0 dhclient
[17638.202249] [ 4185] 0 4185 54100 1460 41 0 0 rsyslogd
[17638.202254] [ 6542] 6000 6542 72442 1438 28 0 0 python3
[17638.202259] [ 7397] 6000 7397 79864 5111 74 0 0 buildbot-worker
[17638.202266] [51239] 89 51239 22463 271 44 0 0 pickup
[17638.202270] [ 8599] 6000 8599 35612 1505 25 0 0 python3
[17638.202275] [ 8600] 6000 8600 28321 56 12 0 0 sh
[17638.202280] [ 8634] 6000 8634 327050 107698 242 0 0 python3.9.real
[17638.202286] [ 9006] 6000 9006 91257 8787 36 0 0 KnottyUI
[17638.202292] [10057] 6000 10057 3631 1236 11 0 0 unfsd
[17638.202297] [36936] 6000 36936 28208 2136 15 0 0 python3.9.real
[17638.202301] [36937] 6000 36937 72824 8782 35 0 0 KnottyUI
[17638.202306] [36954] 6000 36954 342765 78355 243 0 0 qemu-system-i38
[17638.202311] [38209] 6000 38209 50766 4886 24 0 0 KnottyUI
[17638.202316] [38234] 6000 38234 85672 37452 91 1 0 Cooker
[17638.202320] [39697] 6000 39697 64177 18404 53 0 0 Worker
[17638.202325] [ 8572] 6000 8572 68967 23148 61 0 0 glibc-testsuite
[17638.202329] [ 8978] 6000 8978 28888 82 13 0 0 run.do_check.85
[17638.202334] [12731] 6000 12731 24798 92 10 0 0 make
[17638.202338] [12733] 6000 12733 25567 572 11 0 0 make
[17638.202344] [25457] 6000 25457 35612 1505 27 0 0 python3
[17638.202349] [25465] 6000 25465 28321 56 12 0 0 sh
[17638.202354] [25500] 6000 25500 318492 107973 243 0 0 python3.9.real
[17638.202360] [26299] 6000 26299 72363 8060 33 0 0 python3.9.real
[17638.202365] [26643] 6000 26643 50767 4881 24 0 0 KnottyUI
[17638.202369] [26710] 6000 26710 84928 36453 89 3 0 Cooker
[17638.202374] [29569] 6000 29569 61504 15741 47 0 0 Worker
[17638.202378] [45981] 6000 45981 66782 20960 55 0 0 glibc-testsuite
[17638.202383] [46470] 6000 46470 28888 83 13 0 0 run.do_check.45
[17638.202388] [51014] 6000 51014 24799 92 10 0 0 make
[17638.202392] [51021] 6000 51021 25514 614 12 0 0 make
[17638.202399] [27307] 6000 27307 29485 4643 20 0 0 make
[17638.202403] [33817] 6000 33817 28325 73 12 0 0 bash
[17638.202408] [33818] 6000 33818 26944 1041 13 0 0 python3.9.real
[17638.202413] [33821] 6000 33821 1104747 1068 21 0 0 qemu-arm
[17638.202418] [33823] 6000 33823 33589813 31118859 63883 1045422 0 qemu-arm
[17638.202423] [ 8706] 6000 8706 26197 1445 12 0 0 make
[17638.202427] [13423] 0 13423 27014 18 10 0 0 sleep
[17638.202432] [15307] 6000 15307 1265 52 6 0 0 i686-poky-linux
[17638.202436] [15311] 6000 15311 14078 2393 29 0 0 cc1
[17638.202441] [15312] 6000 15312 25736 102 9 0 0 as
[17638.202445] Out of memory: Kill process 33823 (qemu-arm) score 948 or sacrifice child
[17638.202524] Killed process 33823 (qemu-arm), UID 6000, total-vm:134359252kB, anon-rss:124475284kB, file-rss:152kB, shmem-rss:0kB
[139934.388947] prelink-rtld[21423]: segfault at 1392c70 ip 000000000040997e sp 00007ffea1a26db0 error 4 in prelink-rtld[402000+c000]
[191913.493838] qemu-arm invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[191913.493844] qemu-arm cpuset=/ mems_allowed=0-1
[191913.493849] CPU: 2 PID: 56603 Comm: qemu-arm Kdump: loaded Not tainted 3.10.0-1160.25.1.el7.x86_64 #1
[191913.493851] Hardware name: Silicon Mechanics Rackform R346.v5/X10DRI-T, BIOS 1.1 04/14/2015
[191913.493853] Call Trace:
[191913.493869] [<ffffffff95d8311a>] dump_stack+0x19/0x1b
[191913.493873] [<ffffffff95d7da3a>] dump_header+0x90/0x229
[191913.493881] [<ffffffff9590ec6b>] ? cred_has_capability+0x6b/0x120
[191913.493889] [<ffffffff957c232d>] oom_kill_process+0x2cd/0x490
[191913.493901] [<ffffffff9590ed4e>] ? selinux_capable+0x2e/0x40
[191913.493904] [<ffffffff957c2a1a>] out_of_memory+0x31a/0x500
[191913.493907] [<ffffffff95d7e557>] __alloc_pages_slowpath+0x5db/0x729
[191913.493911] [<ffffffff957c8f96>] __alloc_pages_nodemask+0x436/0x450
[191913.493919] [<ffffffff95818c88>] alloc_pages_current+0x98/0x110
[191913.493922] [<ffffffff957bdde7>] __page_cache_alloc+0x97/0xb0
[191913.493925] [<ffffffff957c0d80>] filemap_fault+0x270/0x420
[191913.493952] [<ffffffffc0cf7756>] ext4_filemap_fault+0x36/0x50 [ext4]
[191913.493958] [<ffffffff957ee08a>] __do_fault.isra.61+0x8a/0x100
[191913.493961] [<ffffffff957ee63c>] do_read_fault.isra.63+0x4c/0x1b0
[191913.493965] [<ffffffff957f5e80>] handle_mm_fault+0xa20/0xfb0
[191913.493971] [<ffffffff95d90653>] __do_page_fault+0x213/0x500
[191913.493973] [<ffffffff95d90975>] do_page_fault+0x35/0x90
[191913.493976] [<ffffffff95d8c778>] page_fault+0x28/0x30
[191913.493978] Mem-Info:
[191913.493991] active_anon:30170820 inactive_anon:1337842 isolated_anon:0
active_file:44 inactive_file:0 isolated_file:0
unevictable:0 dirty:0 writeback:0 unstable:0
slab_reclaimable:219795 slab_unreclaimable:66530
mapped:2905 shmem:3175 pagetables:66390 bounce:0
free:88518 free_pcp:5469 free_cma:0
[191913.493998] Node 0 DMA free:14160kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[191913.494006] lowmem_reserve[]: 0 1695 64154 64154
[191913.494010] Node 0 DMA32 free:251016kB min:1184kB low:1480kB high:1776kB active_anon:1106092kB inactive_anon:369004kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1973304kB managed:1735704kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:4kB slab_reclaimable:224kB slab_unreclaimable:120kB kernel_stack:0kB pagetables:2260kB unstable:0kB bounce:0kB free_pcp:340kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:2 all_unreclaimable? yes
[191913.494017] lowmem_reserve[]: 0 0 62459 62459
[191913.494020] Node 0 Normal free:43748kB min:43744kB low:54680kB high:65616kB active_anon:58794116kB inactive_anon:2449792kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63958384kB mlocked:0kB dirty:0kB writeback:0kB mapped:2652kB shmem:3488kB slab_reclaimable:434384kB slab_unreclaimable:131328kB kernel_stack:6656kB pagetables:126956kB unstable:0kB bounce:0kB free_pcp:10984kB local_pcp:784kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[191913.494025] lowmem_reserve[]: 0 0 0 0
[191913.494029] Node 1 Normal free:45148kB min:45164kB low:56452kB high:67744kB active_anon:60783072kB inactive_anon:2532572kB active_file:504kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66039288kB mlocked:0kB dirty:0kB writeback:0kB mapped:8964kB shmem:9208kB slab_reclaimable:444572kB slab_unreclaimable:134672kB kernel_stack:5648kB pagetables:136344kB unstable:0kB bounce:0kB free_pcp:10552kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:72672 all_unreclaimable? yes
[191913.494034] lowmem_reserve[]: 0 0 0 0
[191913.494036] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 0*32kB 1*64kB (U) 0*128kB 1*256kB (U) 1*512kB (U) 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 14160kB
[191913.494047] Node 0 DMA32: 14*4kB (U) 36*8kB (UE) 37*16kB (UEM) 25*32kB (UE) 17*64kB (UE) 7*128kB (UEM) 2*256kB (U) 0*512kB 1*1024kB (E) 2*2048kB (EM) 59*4096kB (UM) = 251016kB
[191913.494057] Node 0 Normal: 10537*4kB (UEM) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 42148kB
[191913.494065] Node 1 Normal: 10706*4kB (UEM) 269*8kB (UE) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44976kB
[191913.494074] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[191913.494076] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[191913.494078] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[191913.494079] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[191913.494081] 57521 total pagecache pages
[191913.494084] 53928 pages in swap cache
[191913.494086] Swap cache stats: add 3155236, delete 3101308, find 1344/2052
[191913.494087] Free swap = 0kB
[191913.494088] Total swap = 4194300kB
[191913.494090] 33527464 pages RAM
[191913.494091] 0 pages HighMem/MovableOnly
[191913.494092] 590147 pages reserved
[191913.494093] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[191913.494110] [ 962] 0 962 13959 3789 33 2 0 systemd-journal
[191913.494113] [ 997] 0 997 50277 320 31 513 0 lvmetad
[191913.494116] [ 1009] 0 1009 12390 788 26 2 -1000 systemd-udevd
[191913.494120] [ 1361] 0 1361 13883 109 26 0 -1000 auditd
[191913.494122] [ 1392] 0 1392 22652 231 47 0 0 rngd
[191913.494125] [ 1394] 0 1394 5479 147 15 0 0 irqbalance
[191913.494128] [ 1396] 0 1396 6596 73 19 0 0 systemd-logind
[191913.494131] [ 1397] 81 1397 16621 194 34 0 -900 dbus-daemon
[191913.494134] [ 1430] 0 1430 119577 1008 85 0 0 NetworkManager
[191913.494136] [ 1439] 997 1439 153083 1384 62 1024 0 polkitd
[191913.494139] [ 1444] 996 1444 5634 73 17 0 0 chronyd
[191913.494141] [ 1464] 0 1464 28885 86 11 0 0 ksmtuned
[191913.494144] [ 1473] 0 1473 50358 134 38 0 0 gssproxy
[191913.494146] [ 1512] 0 1512 6477 51 18 0 0 atd
[191913.494149] [ 1513] 0 1513 31598 158 19 0 0 crond
[191913.494151] [ 1615] 0 1615 28235 257 58 0 -1000 sshd
[191913.494154] [ 1618] 0 1618 143572 3367 98 0 0 tuned
[191913.494156] [ 1629] 802 1629 1700941 27816 339 3387 0 agent
[191913.494159] [ 1633] 802 1633 1249670 11610 172 534 0 trace-agent
[191913.494161] [ 1811] 0 1811 5840 105 14 0 0 ipmievd
[191913.494163] [ 1863] 0 1863 27552 33 10 0 0 agetty
[191913.494166] [ 2678] 0 2678 22437 268 43 1 0 master
[191913.494169] [ 2740] 89 2740 22507 271 45 1 0 qmgr
[191913.494171] [ 2905] 6000 2905 77998 3617 103 33 0 Xvnc
[191913.494174] [ 4069] 6000 4069 43378 1048 40 4 0 xterm
[191913.494176] [ 4071] 6000 4071 28920 83 13 40 0 bash
[191913.494179] [ 4088] 0 4088 25766 546 52 11 0 dhclient
[191913.494182] [ 4185] 0 4185 58189 2478 47 126 0 rsyslogd
[191913.494184] [ 6542] 6000 6542 72442 1433 28 5 0 python3
[191913.494189] [ 7397] 6000 7397 79928 3253 75 1951 0 buildbot-worker
[191913.494198] [20606] 89 20606 22463 271 47 0 0 pickup
[191913.494201] [18459] 6000 18459 35612 1505 27 0 0 python3
[191913.494203] [18460] 6000 18460 28321 56 12 0 0 sh
[191913.494206] [18495] 6000 18495 304122 52405 224 41986 0 python3.9.real
[191913.494210] [18866] 6000 18866 72364 8060 33 2 0 python3.9.real
[191913.494213] [19193] 6000 19193 50767 4889 25 0 0 KnottyUI
[191913.494215] [19240] 6000 19240 101290 36585 91 46 0 Cooker
[191913.494218] [21839] 6000 21839 61513 15738 49 2 0 Worker
[191913.494221] [38081] 6000 38081 66772 20999 57 2 0 glibc-testsuite
[191913.494223] [38319] 6000 38319 28888 82 13 0 0 run.do_check.38
[191913.494226] [41924] 6000 41924 24799 93 11 0 0 make
[191913.494228] [41931] 6000 41931 25514 610 12 0 0 make
[191913.494233] [46537] 6000 46537 29485 4744 20 0 0 make
[191913.494236] [56597] 6000 56597 28325 72 11 0 0 bash
[191913.494238] [56598] 6000 56598 26944 1042 13 0 0 python3.9.real
[191913.494240] [56601] 6000 56601 1105257 1067 21 0 0 qemu-arm
[191913.494243] [56603] 6000 56603 33669619 31262088 64069 997477 0 qemu-arm
[191913.494246] [27093] 0 27093 27014 18 10 0 0 sleep
[191913.494248] Out of memory: Kill process 56603 (qemu-arm) score 951 or sacrifice child
[191913.494297] Killed process 56603 (qemu-arm), UID 6000, total-vm:134678476kB, anon-rss:125048252kB, file-rss:100kB, shmem-rss:0kB
[321248.631139] qemu-aarch64 invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
[321248.631144] qemu-aarch64 cpuset=/ mems_allowed=0-1
[321248.631147] CPU: 2 PID: 41162 Comm: qemu-aarch64 Kdump: loaded Not tainted 3.10.0-1160.25.1.el7.x86_64 #1
[321248.631149] Hardware name: Silicon Mechanics Rackform R346.v5/X10DRI-T, BIOS 1.1 04/14/2015
[321248.631150] Call Trace:
[321248.631162] [<ffffffff95d8311a>] dump_stack+0x19/0x1b
[321248.631164] [<ffffffff95d7da3a>] dump_header+0x90/0x229
[321248.631170] [<ffffffff9590ec6b>] ? cred_has_capability+0x6b/0x120
[321248.631176] [<ffffffff957c232d>] oom_kill_process+0x2cd/0x490
[321248.631178] [<ffffffff9590ed4e>] ? selinux_capable+0x2e/0x40
[321248.631180] [<ffffffff957c2a1a>] out_of_memory+0x31a/0x500
[321248.631183] [<ffffffff95d7e557>] __alloc_pages_slowpath+0x5db/0x729
[321248.631186] [<ffffffff957c8f96>] __alloc_pages_nodemask+0x436/0x450
[321248.631191] [<ffffffff9581c519>] alloc_pages_vma+0xa9/0x200
[321248.631196] [<ffffffff957f6117>] handle_mm_fault+0xcb7/0xfb0
[321248.631201] [<ffffffff95d90653>] __do_page_fault+0x213/0x500
[321248.631203] [<ffffffff95d90975>] do_page_fault+0x35/0x90
[321248.631205] [<ffffffff95d8cb6b>] ? error_exit+0xb/0x60
[321248.631207] [<ffffffff95d8c778>] page_fault+0x28/0x30
[321248.631208] Mem-Info:
[321248.631217] active_anon:30159628 inactive_anon:1337147 isolated_anon:0
active_file:636 inactive_file:713 isolated_file:0
unevictable:0 dirty:2 writeback:78 unstable:0
slab_reclaimable:230492 slab_unreclaimable:69608
mapped:4392 shmem:4436 pagetables:65398 bounce:0
free:88460 free_pcp:1479 free_cma:0
[321248.631221] Node 0 DMA free:14160kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[321248.631227] lowmem_reserve[]: 0 1695 64154 64154
[321248.631230] Node 0 DMA32 free:250992kB min:1184kB low:1480kB high:1776kB active_anon:1105668kB inactive_anon:368676kB active_file:44kB inactive_file:64kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1973304kB managed:1735704kB mlocked:0kB dirty:0kB writeback:8kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:936kB kernel_stack:128kB pagetables:2240kB unstable:0kB bounce:0kB free_pcp:368kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:24898 all_unreclaimable? yes
[321248.631236] lowmem_reserve[]: 0 0 62459 62459
[321248.631238] Node 0 Normal free:43548kB min:43744kB low:54680kB high:65616kB active_anon:58752740kB inactive_anon:2448088kB active_file:1440kB inactive_file:1048kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63958384kB mlocked:0kB dirty:0kB writeback:0kB mapped:4084kB shmem:4164kB slab_reclaimable:470668kB slab_unreclaimable:144892kB kernel_stack:8032kB pagetables:120796kB unstable:0kB bounce:0kB free_pcp:4272kB local_pcp:212kB free_cma:0kB writeback_tmp:0kB pages_scanned:55686 all_unreclaimable? yes
[321248.631243] lowmem_reserve[]: 0 0 0 0
[321248.631245] Node 1 Normal free:45140kB min:45164kB low:56452kB high:67744kB active_anon:60780104kB inactive_anon:2531824kB active_file:1060kB inactive_file:1740kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66039288kB mlocked:0kB dirty:8kB writeback:304kB mapped:13484kB shmem:13580kB slab_reclaimable:451300kB slab_unreclaimable:132604kB kernel_stack:5920kB pagetables:138556kB unstable:0kB bounce:0kB free_pcp:1276kB local_pcp:232kB free_cma:0kB writeback_tmp:0kB pages_scanned:132037 all_unreclaimable? yes
[321248.631249] lowmem_reserve[]: 0 0 0 0
[321248.631251] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 0*32kB 1*64kB (U) 0*128kB 1*256kB (U) 1*512kB (U) 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 14160kB
[321248.631260] Node 0 DMA32: 41*4kB (UM) 31*8kB (U) 44*16kB (UM) 21*32kB (U) 14*64kB (UM) 2*128kB (UM) 1*256kB (U) 2*512kB (UM) 3*1024kB (U) 3*2048kB (E) 58*4096kB (EM) = 251004kB
[321248.631269] Node 0 Normal: 10940*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 43760kB
[321248.631275] Node 1 Normal: 12170*4kB (UE) 109*8kB (UE) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 49552kB
[321248.631282] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[321248.631284] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[321248.631285] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[321248.631286] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[321248.631287] 20048 total pagecache pages
[321248.631289] 14137 pages in swap cache
[321248.631290] Swap cache stats: add 4208173, delete 4192510, find 95447/98666
[321248.631291] Free swap = 0kB
[321248.631292] Total swap = 4194300kB
[321248.631293] 33527464 pages RAM
[321248.631294] 0 pages HighMem/MovableOnly
[321248.631295] 590147 pages reserved
[321248.631296] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[321248.631306] [ 962] 0 962 16007 5494 37 0 0 systemd-journal
[321248.631308] [ 997] 0 997 50277 320 31 513 0 lvmetad
[321248.631310] [ 1009] 0 1009 12390 788 26 2 -1000 systemd-udevd
[321248.631313] [ 1361] 0 1361 13883 109 26 0 -1000 auditd
[321248.631315] [ 1392] 0 1392 22652 231 47 0 0 rngd
[321248.631317] [ 1394] 0 1394 5479 147 15 0 0 irqbalance
[321248.631319] [ 1396] 0 1396 6596 74 19 0 0 systemd-logind
[321248.631321] [ 1397] 81 1397 16621 194 34 0 -900 dbus-daemon
[321248.631323] [ 1430] 0 1430 119577 1019 85 0 0 NetworkManager
[321248.631325] [ 1439] 997 1439 153083 1395 62 1025 0 polkitd
[321248.631327] [ 1444] 996 1444 5634 73 17 0 0 chronyd
[321248.631329] [ 1464] 0 1464 28885 100 11 0 0 ksmtuned
[321248.631331] [ 1473] 0 1473 50358 134 38 0 0 gssproxy
[321248.631333] [ 1512] 0 1512 6477 51 18 0 0 atd
[321248.631335] [ 1513] 0 1513 31598 158 19 0 0 crond
[321248.631337] [ 1615] 0 1615 28235 257 58 0 -1000 sshd
[321248.631339] [ 1618] 0 1618 143572 3367 98 0 0 tuned
[321248.631340] [ 1629] 802 1629 1750157 27549 341 2110 0 agent
[321248.631343] [ 1633] 802 1633 1298886 11683 175 511 0 trace-agent
[321248.631344] [ 1811] 0 1811 5840 105 14 0 0 ipmievd
[321248.631346] [ 1863] 0 1863 27552 33 10 0 0 agetty
[321248.631349] [ 2678] 0 2678 22437 269 43 1 0 master
[321248.631351] [ 2740] 89 2740 22507 271 45 1 0 qmgr
[321248.631353] [ 2905] 6000 2905 77883 3584 103 0 0 Xvnc
[321248.631355] [ 4069] 6000 4069 43378 1048 40 4 0 xterm
[321248.631357] [ 4071] 6000 4071 28920 83 13 40 0 bash
[321248.631359] [ 4088] 0 4088 25766 548 52 10 0 dhclient
[321248.631361] [ 4185] 0 4185 60237 3442 52 123 0 rsyslogd
[321248.631363] [ 6542] 6000 6542 72442 1433 28 5 0 python3
[321248.631365] [ 7397] 6000 7397 79928 4701 75 505 0 buildbot-worker
[321248.631371] [48289] 6000 48289 35612 1506 25 0 0 python3
[321248.631373] [48302] 6000 48302 28321 57 11 0 0 sh
[321248.631375] [48349] 6000 48349 349660 119098 314 4612 0 python3.9.real
[321248.631378] [51503] 6000 51503 72356 8089 34 4 0 python3.9.real
[321248.631380] [51985] 6000 51985 50768 4880 23 0 0 KnottyUI
[321248.631382] [52117] 6000 52117 87192 38560 95 442 0 Cooker
[321248.631384] [ 1174] 6000 1174 63962 18194 54 0 0 Worker
[321248.631386] [28103] 6000 28103 68016 20528 60 1688 0 glibc-testsuite
[321248.631388] [28695] 6000 28695 28888 82 13 0 0 run.do_check.28
[321248.631390] [32628] 6000 32628 24798 91 12 0 0 make
[321248.631392] [32636] 6000 32636 25462 599 12 0 0 make
[321248.631395] [28364] 89 28364 22463 272 44 0 0 pickup
[321248.631397] [55643] 6000 55643 17064380 16398680 33217 561270 0 qemu-aarch64
[321248.631399] [15058] 6000 15058 27078 112 9 0 0 rm
[321248.631401] [32089] 6000 32089 25717 974 13 0 0 make
[321248.631403] [33037] 6000 33037 28324 71 11 0 0 bash
[321248.631404] [33041] 6000 33041 26942 1039 13 0 0 python3.9.real
[321248.631407] [33049] 6000 33049 28324 70 12 0 0 bash
[321248.631408] [33051] 6000 33051 26941 1039 14 0 0 python3.9.real
[321248.631410] [33081] 6000 33081 42968 601 19 0 0 qemu-aarch64
[321248.631412] [33092] 6000 33092 42968 600 20 0 0 qemu-aarch64
[321248.631414] [33122] 6000 33122 28324 71 11 0 0 bash
[321248.631416] [33123] 6000 33123 42968 616 16 0 0 qemu-aarch64
[321248.631418] [33125] 6000 33125 26942 1038 13 0 0 python3.9.real
[321248.631420] [33127] 6000 33127 1929171 1885934 3703 2 0 qemu-aarch64
[321248.631422] [33171] 6000 33171 43016 649 18 0 0 qemu-aarch64
[321248.631424] [33176] 6000 33176 42969 610 18 0 0 qemu-aarch64
[321248.631426] [33179] 6000 33179 42969 615 18 0 0 qemu-aarch64
[321248.631428] [33183] 6000 33183 42969 616 18 0 0 qemu-aarch64
[321248.631430] [33185] 6000 33185 42969 616 18 0 0 qemu-aarch64
[321248.631432] [33188] 6000 33188 42969 616 18 0 0 qemu-aarch64
[321248.631434] [33190] 6000 33190 42969 616 18 0 0 qemu-aarch64
[321248.631436] [33278] 6000 33278 42967 601 19 0 0 qemu-aarch64
[321248.631438] [33382] 6000 33382 1791439 1743369 3431 18 0 qemu-aarch64
[321248.631439] [34578] 6000 34578 28324 70 12 0 0 bash
[321248.631441] [34582] 6000 34582 26941 1039 12 0 0 python3.9.real
[321248.631443] [34687] 6000 34687 28324 70 12 0 0 bash
[321248.631445] [34693] 6000 34693 26942 1040 13 0 0 python3.9.real
[321248.631447] [34740] 6000 34740 59364 943 21 0 0 qemu-aarch64
[321248.631449] [34805] 6000 34805 59364 946 18 0 0 qemu-aarch64
[321248.631451] [34814] 6000 34814 6375919 6198686 12383 61 0 qemu-aarch64
[321248.631453] [34854] 6000 34854 59364 943 21 0 0 qemu-aarch64
[321248.631455] [34934] 6000 34934 59480 964 19 0 0 qemu-aarch64
[321248.631456] [34944] 6000 34944 59365 946 19 0 0 qemu-aarch64
[321248.631458] [34949] 6000 34949 59365 948 19 0 0 qemu-aarch64
[321248.631460] [34954] 6000 34954 59365 949 19 0 0 qemu-aarch64
[321248.631462] [34957] 6000 34957 59365 949 19 0 0 qemu-aarch64
[321248.631464] [34962] 6000 34962 59365 949 19 0 0 qemu-aarch64
[321248.631466] [34964] 6000 34964 59365 949 19 0 0 qemu-aarch64
[321248.631468] [35244] 6000 35244 28324 70 12 0 0 bash
[321248.631470] [35254] 6000 35254 26941 1040 12 0 0 python3.9.real
[321248.631472] [35454] 6000 35454 59363 608 21 0 0 qemu-aarch64
[321248.631473] [35584] 6000 35584 1891479 1794260 3534 0 0 qemu-aarch64
[321248.631475] [40916] 6000 40916 28324 71 11 0 0 bash
[321248.631478] [40925] 6000 40925 26943 1039 12 0 0 python3.9.real
[321248.631480] [41095] 6000 41095 59287 589 22 0 0 qemu-aarch64
[321248.631481] [41162] 6000 41162 4253942 3183717 6238 47 0 qemu-aarch64
[321248.631483] [29994] 0 29994 27014 18 10 0 0 sleep
[321248.631485] Out of memory: Kill process 55643 (qemu-aarch64) score 500 or sacrifice child
[321248.631522] Killed process 55643 (qemu-aarch64), UID 6000, total-vm:68257520kB, anon-rss:65594720kB, file-rss:0kB, shmem-rss:0kB
[326470.414378] prelink-rtld[49701]: segfault at 1fe2144 ip 000000000040a436 sp 00007fff987d94d0 error 4 in prelink-rtld[402000+c000]
[497998.895864] qemu-aarch64 invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
[497998.895869] qemu-aarch64 cpuset=/ mems_allowed=0-1
[497998.895872] CPU: 41 PID: 55898 Comm: qemu-aarch64 Kdump: loaded Not tainted 3.10.0-1160.25.1.el7.x86_64 #1
[497998.895873] Hardware name: Silicon Mechanics Rackform R346.v5/X10DRI-T, BIOS 1.1 04/14/2015
[497998.895875] Call Trace:
[497998.895886] [<ffffffff95d8311a>] dump_stack+0x19/0x1b
[497998.895888] [<ffffffff95d7da3a>] dump_header+0x90/0x229
[497998.895894] [<ffffffff9590ec6b>] ? cred_has_capability+0x6b/0x120
[497998.895899] [<ffffffff957c232d>] oom_kill_process+0x2cd/0x490
[497998.895901] [<ffffffff9590ed4e>] ? selinux_capable+0x2e/0x40
[497998.895903] [<ffffffff957c2a1a>] out_of_memory+0x31a/0x500
[497998.895906] [<ffffffff95d7e557>] __alloc_pages_slowpath+0x5db/0x729
[497998.895909] [<ffffffff957c8f96>] __alloc_pages_nodemask+0x436/0x450
[497998.895914] [<ffffffff9581c519>] alloc_pages_vma+0xa9/0x200
[497998.895918] [<ffffffff957f6117>] handle_mm_fault+0xcb7/0xfb0
[497998.895923] [<ffffffff95d90653>] __do_page_fault+0x213/0x500
[497998.895925] [<ffffffff95d90975>] do_page_fault+0x35/0x90
[497998.895926] [<ffffffff95d8c778>] page_fault+0x28/0x30
[497998.895928] Mem-Info:
[497998.895936] active_anon:30446141 inactive_anon:1034986 isolated_anon:0
active_file:392 inactive_file:80 isolated_file:0
unevictable:0 dirty:18 writeback:0 unstable:0
slab_reclaimable:244615 slab_unreclaimable:70739
mapped:2690 shmem:10676 pagetables:64792 bounce:0
free:88465 free_pcp:1379 free_cma:0
[497998.895940] Node 0 DMA free:14160kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[497998.895946] lowmem_reserve[]: 0 1695 64154 64154
[497998.895949] Node 0 DMA32 free:251008kB min:1184kB low:1480kB high:1776kB active_anon:1106040kB inactive_anon:368724kB active_file:0kB inactive_file:24kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1973304kB managed:1735704kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:312kB kernel_stack:144kB pagetables:1996kB unstable:0kB bounce:0kB free_pcp:892kB local_pcp:56kB free_cma:0kB writeback_tmp:0kB pages_scanned:185 all_unreclaimable? yes
[497998.895954] lowmem_reserve[]: 0 0 62459 62459
[497998.895956] Node 0 Normal free:43636kB min:43744kB low:54680kB high:65616kB active_anon:59920748kB inactive_anon:1239804kB active_file:416kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63958384kB mlocked:0kB dirty:16kB writeback:0kB mapped:5964kB shmem:13784kB slab_reclaimable:497128kB slab_unreclaimable:152504kB kernel_stack:8672kB pagetables:128380kB unstable:0kB bounce:0kB free_pcp:2168kB local_pcp:240kB free_cma:0kB writeback_tmp:0kB pages_scanned:6966 all_unreclaimable? yes
[497998.895961] lowmem_reserve[]: 0 0 0 0
[497998.895963] Node 1 Normal free:45056kB min:45164kB low:56452kB high:67744kB active_anon:60757776kB inactive_anon:2531416kB active_file:1152kB inactive_file:824kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66039288kB mlocked:0kB dirty:56kB writeback:0kB mapped:4796kB shmem:28920kB slab_reclaimable:481332kB slab_unreclaimable:130140kB kernel_stack:6240kB pagetables:128792kB unstable:0kB bounce:0kB free_pcp:2456kB local_pcp:172kB free_cma:0kB writeback_tmp:0kB pages_scanned:29928 all_unreclaimable? yes
[497998.895968] lowmem_reserve[]: 0 0 0 0
[497998.895970] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 0*32kB 1*64kB (U) 0*128kB 1*256kB (U) 1*512kB (U) 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 14160kB
[497998.895978] Node 0 DMA32: 21*4kB (UM) 45*8kB (UM) 102*16kB (UM) 54*32kB (UM) 23*64kB (UM) 6*128kB (UM) 5*256kB (UM) 2*512kB (UM) 1*1024kB (U) 0*2048kB 59*4096kB (M) = 251036kB
[497998.895987] Node 0 Normal: 11210*4kB (UM) 9*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44912kB
[497998.895993] Node 1 Normal: 11399*4kB (UEM) 162*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46892kB
[497998.896000] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[497998.896002] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[497998.896003] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[497998.896004] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[497998.896005] 18289 total pagecache pages
[497998.896008] 6717 pages in swap cache
[497998.896009] Swap cache stats: add 4786749, delete 4778506, find 186128/192469
[497998.896010] Free swap = 0kB
[497998.896011] Total swap = 4194300kB
[497998.896012] 33527464 pages RAM
[497998.896013] 0 pages HighMem/MovableOnly
[497998.896014] 590147 pages reserved
[497998.896014] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[497998.896024] [ 962] 0 962 21142 3488 47 17 0 systemd-journal
[497998.896026] [ 997] 0 997 50277 9 31 824 0 lvmetad
[497998.896029] [ 1009] 0 1009 12390 140 26 650 -1000 systemd-udevd
[497998.896031] [ 1361] 0 1361 13883 99 26 10 -1000 auditd
[497998.896033] [ 1392] 0 1392 22652 8 47 223 0 rngd
[497998.896035] [ 1394] 0 1394 5479 91 15 56 0 irqbalance
[497998.896037] [ 1396] 0 1396 6596 34 19 40 0 systemd-logind
[497998.896040] [ 1397] 81 1397 16621 94 34 100 -900 dbus-daemon
[497998.896042] [ 1430] 0 1430 119577 728 85 292 0 NetworkManager
[497998.896043] [ 1439] 997 1439 153083 235 62 2185 0 polkitd
[497998.896045] [ 1444] 996 1444 5634 73 17 0 0 chronyd
[497998.896047] [ 1464] 0 1464 28917 105 11 10 0 ksmtuned
[497998.896049] [ 1473] 0 1473 50358 78 38 56 0 gssproxy
[497998.896051] [ 1512] 0 1512 6477 0 18 52 0 atd
[497998.896053] [ 1513] 0 1513 31598 158 19 0 0 crond
[497998.896055] [ 1615] 0 1615 28235 257 58 0 -1000 sshd
[497998.896057] [ 1618] 0 1618 143572 3252 98 115 0 tuned
[497998.896059] [ 1629] 802 1629 1897741 26043 349 6392 0 agent
[497998.896061] [ 1633] 802 1633 1298886 11586 175 641 0 trace-agent
[497998.896063] [ 1811] 0 1811 5840 9 14 96 0 ipmievd
[497998.896065] [ 1863] 0 1863 27552 33 10 0 0 agetty
[497998.896067] [ 2678] 0 2678 22437 25 43 245 0 master
[497998.896069] [ 2740] 89 2740 22507 1 45 271 0 qmgr
[497998.896071] [ 2905] 6000 2905 77967 1586 103 2066 0 Xvnc
[497998.896091] [ 4069] 6000 4069 43378 1042 40 10 0 xterm
[497998.896094] [ 4071] 6000 4071 28920 83 13 40 0 bash
[497998.896096] [ 4088] 0 4088 25766 548 52 10 0 dhclient
[497998.896098] [ 4185] 0 4185 64293 2737 57 146 0 rsyslogd
[497998.896100] [ 6542] 6000 6542 72442 1431 28 7 0 python3
[497998.896102] [ 7397] 6000 7397 79928 4603 75 605 0 buildbot-worker
[497998.896106] [50574] 6000 50574 629298 160 23 795 0 qemu-mips.real
[497998.896108] [ 6415] 6000 6415 35612 1218 26 282 0 python3
[497998.896110] [ 8252] 6000 8252 28321 53 11 3 0 sh
[497998.896111] [ 8737] 6000 8737 50769 4867 24 11 0 KnottyUI
[497998.896113] [14322] 6000 14322 95223 45406 120 3650 0 Cooker
[497998.896115] [22193] 6000 22193 62190 5330 50 11479 0 Worker
[497998.896117] [23465] 6000 23465 63345 9035 56 11964 0 core-image-sato
[497998.896119] [23811] 6000 23811 34806 1335 30 827 0 python3
[497998.896121] [23815] 6000 23815 62756 2762 49 14197 0 core-image-sato
[497998.896123] [25668] 6000 25668 1539474 277620 790 1483 0 qemu-system-x86
[497998.896125] [39665] 6000 39665 44713 241 42 0 0 ssh
[497998.896129] [19143] 89 19143 22463 271 46 0 0 pickup
[497998.896131] [43552] 6000 43552 35612 1506 24 0 0 python3
[497998.896133] [43553] 6000 43553 28321 57 12 0 0 sh
[497998.896135] [43587] 6000 43587 349658 123659 312 0 0 python3.9.real
[497998.896138] [43962] 6000 43962 72358 8096 33 0 0 python3.9.real
[497998.896140] [44277] 6000 44277 50766 4870 24 0 0 KnottyUI
[497998.896142] [44353] 6000 44353 85314 37194 91 0 0 Cooker
[497998.896144] [47056] 6000 47056 63962 18194 55 0 0 Worker
[497998.896146] [43771] 6000 43771 68026 22144 59 0 0 glibc-testsuite
[497998.896148] [44494] 6000 44494 28888 81 12 0 0 run.do_check.43
[497998.896150] [51940] 6000 51940 24798 92 10 0 0 make
[497998.896152] [51942] 6000 51942 25462 678 11 0 0 make
[497998.896155] [25270] 6000 25270 16534250 16428908 32180 683 0 qemu-aarch64
[497998.896157] [47022] 6000 47022 25725 974 12 0 0 make
[497998.896159] [47982] 6000 47982 28324 71 12 0 0 bash
[497998.896160] [47992] 6000 47992 28324 70 13 0 0 bash
[497998.896162] [47994] 6000 47994 26942 1037 11 0 0 python3.9.real
[497998.896164] [48006] 6000 48006 26941 1039 13 0 0 python3.9.real
[497998.896166] [48030] 6000 48030 28324 71 11 0 0 bash
[497998.896168] [48034] 6000 48034 26942 1038 12 0 0 python3.9.real
[497998.896170] [48060] 6000 48060 43105 729 19 0 0 qemu-aarch64
[497998.896172] [48066] 6000 48066 43105 730 18 0 0 qemu-aarch64
[497998.896174] [48089] 6000 48089 43104 730 20 0 0 qemu-aarch64
[497998.896176] [48093] 6000 48093 43307 795 16 0 0 qemu-aarch64
[497998.896178] [48099] 6000 48099 43106 739 16 0 0 qemu-aarch64
[497998.896180] [48101] 6000 48101 43106 744 16 0 0 qemu-aarch64
[497998.896182] [48104] 6000 48104 43106 745 16 0 0 qemu-aarch64
[497998.896184] [48107] 6000 48107 43106 745 16 0 0 qemu-aarch64
[497998.896186] [48109] 6000 48109 43106 745 16 0 0 qemu-aarch64
[497998.896188] [48110] 6000 48110 43106 745 16 0 0 qemu-aarch64
[497998.896190] [48114] 6000 48114 43105 744 17 0 0 qemu-aarch64
[497998.896191] [48123] 6000 48123 2265008 2221724 4359 0 0 qemu-aarch64
[497998.896193] [48203] 6000 48203 2050955 2002835 3939 0 0 qemu-aarch64
[497998.896195] [49694] 6000 49694 28324 70 12 0 0 bash
[497998.896197] [49703] 6000 49703 26941 1038 12 0 0 python3.9.real
[497998.896199] [49729] 6000 49729 28324 70 11 0 0 bash
[497998.896201] [49738] 6000 49738 26942 1040 13 0 0 python3.9.real
[497998.896203] [49831] 6000 49831 59514 1074 22 0 0 qemu-aarch64
[497998.896205] [49873] 6000 49873 59514 1074 21 0 0 qemu-aarch64
[497998.896207] [49899] 6000 49899 59514 1077 20 0 0 qemu-aarch64
[497998.896208] [49910] 6000 49910 4922987 4744960 9430 0 0 qemu-aarch64
[497998.896210] [49996] 6000 49996 59548 1086 20 0 0 qemu-aarch64
[497998.896212] [50005] 6000 50005 59515 1077 20 0 0 qemu-aarch64
[497998.896214] [50011] 6000 50011 59515 1079 20 0 0 qemu-aarch64
[497998.896216] [50015] 6000 50015 59515 1080 20 0 0 qemu-aarch64
[497998.896218] [50020] 6000 50020 59515 1080 20 0 0 qemu-aarch64
[497998.896220] [50022] 6000 50022 59515 1080 20 0 0 qemu-aarch64
[497998.896222] [50024] 6000 50024 59515 1080 20 0 0 qemu-aarch64
[497998.896224] [50247] 6000 50247 28324 70 12 0 0 bash
[497998.896226] [50266] 6000 50266 26941 1038 13 0 0 python3.9.real
[497998.896228] [50446] 6000 50446 59513 1075 21 0 0 qemu-aarch64
[497998.896230] [50557] 6000 50557 2146645 2049198 4031 0 0 qemu-aarch64
[497998.896231] [55638] 6000 55638 28324 71 11 0 0 bash
[497998.896233] [55647] 6000 55647 26943 1040 13 0 0 python3.9.real
[497998.896235] [55806] 6000 55806 59436 1062 21 0 0 qemu-aarch64
[497998.896237] [55898] 6000 55898 4254087 3410611 6679 0 0 qemu-aarch64
[497998.896238] [10393] 0 10393 27014 19 9 0 0 sleep
[497998.896240] Out of memory: Kill process 25270 (qemu-aarch64) score 484 or sacrifice child
[497998.896277] Killed process 25270 (qemu-aarch64), UID 6000, total-vm:66137000kB, anon-rss:65715632kB, file-rss:0kB, shmem-rss:0kB
[513555.330153] git.real[18750]: segfault at 10 ip 00005566f9e397b9 sp 00007fffcd0f2d50 error 4 in git.real[5566f9dd4000+1f3000]
[559103.152819] qemu-arm invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[559103.152828] qemu-arm cpuset=/ mems_allowed=0-1
[559103.152834] CPU: 6 PID: 48850 Comm: qemu-arm Kdump: loaded Not tainted 3.10.0-1160.25.1.el7.x86_64 #1
[559103.152837] Hardware name: Silicon Mechanics Rackform R346.v5/X10DRI-T, BIOS 1.1 04/14/2015
[559103.152840] Call Trace:
[559103.152859] [<ffffffff95d8311a>] dump_stack+0x19/0x1b
[559103.152865] [<ffffffff95d7da3a>] dump_header+0x90/0x229
[559103.152875] [<ffffffff9590ec6b>] ? cred_has_capability+0x6b/0x120
[559103.152885] [<ffffffff957c232d>] oom_kill_process+0x2cd/0x490
[559103.152890] [<ffffffff9590ed3c>] ? selinux_capable+0x1c/0x40
[559103.152895] [<ffffffff957c2a1a>] out_of_memory+0x31a/0x500
[559103.152900] [<ffffffff95d7e557>] __alloc_pages_slowpath+0x5db/0x729
[559103.152906] [<ffffffff957c8f96>] __alloc_pages_nodemask+0x436/0x450
[559103.152916] [<ffffffff95818c88>] alloc_pages_current+0x98/0x110
[559103.152922] [<ffffffff957bdde7>] __page_cache_alloc+0x97/0xb0
[559103.152926] [<ffffffff957c0d80>] filemap_fault+0x270/0x420
[559103.152956] [<ffffffffc0cf7756>] ext4_filemap_fault+0x36/0x50 [ext4]
[559103.152964] [<ffffffff957ee08a>] __do_fault.isra.61+0x8a/0x100
[559103.152969] [<ffffffff957ee63c>] do_read_fault.isra.63+0x4c/0x1b0
[559103.152975] [<ffffffff957f5e80>] handle_mm_fault+0xa20/0xfb0
[559103.152983] [<ffffffff9584a9aa>] ? __check_object_size+0x1ca/0x250
[559103.152991] [<ffffffff95d90653>] __do_page_fault+0x213/0x500
[559103.152995] [<ffffffff95d90975>] do_page_fault+0x35/0x90
[559103.152999] [<ffffffff95d8c778>] page_fault+0x28/0x30
[559103.153003] Mem-Info:
[559103.153020] active_anon:30121084 inactive_anon:1346570 isolated_anon:0
active_file:60 inactive_file:0 isolated_file:0
unevictable:0 dirty:4 writeback:0 unstable:0
slab_reclaimable:252392 slab_unreclaimable:68275
mapped:4351 shmem:4772 pagetables:66162 bounce:0
free:88453 free_pcp:11121 free_cma:0
[559103.153029] Node 0 DMA free:14160kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[559103.153042] lowmem_reserve[]: 0 1695 64154 64154
[559103.153048] Node 0 DMA32 free:250936kB min:1184kB low:1480kB high:1776kB active_anon:1097636kB inactive_anon:366004kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1973304kB managed:1735704kB mlocked:0kB dirty:0kB writeback:0kB mapped:8kB shmem:0kB slab_reclaimable:2304kB slab_unreclaimable:2028kB kernel_stack:96kB pagetables:2772kB unstable:0kB bounce:0kB free_pcp:7172kB local_pcp:256kB free_cma:0kB writeback_tmp:0kB pages_scanned:211 all_unreclaimable? yes
[559103.153058] lowmem_reserve[]: 0 0 62459 62459
[559103.153064] Node 0 Normal free:43728kB min:43744kB low:54680kB high:65616kB active_anon:58652772kB inactive_anon:2489608kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63958384kB mlocked:0kB dirty:12kB writeback:0kB mapped:7916kB shmem:8400kB slab_reclaimable:514868kB slab_unreclaimable:136060kB kernel_stack:7232kB pagetables:126652kB unstable:0kB bounce:0kB free_pcp:22128kB local_pcp:608kB free_cma:0kB writeback_tmp:0kB pages_scanned:320 all_unreclaimable? yes
[559103.153074] lowmem_reserve[]: 0 0 0 0
[559103.153079] Node 1 Normal free:44988kB min:45164kB low:56452kB high:67744kB active_anon:60733928kB inactive_anon:2530668kB active_file:620kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66039288kB mlocked:0kB dirty:4kB writeback:0kB mapped:9480kB shmem:10688kB slab_reclaimable:492396kB slab_unreclaimable:135012kB kernel_stack:5696kB pagetables:135224kB unstable:0kB bounce:0kB free_pcp:15184kB local_pcp:164kB free_cma:0kB writeback_tmp:0kB pages_scanned:2579 all_unreclaimable? yes
[559103.153089] lowmem_reserve[]: 0 0 0 0
[559103.153093] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 0*32kB 1*64kB (U) 0*128kB 1*256kB (U) 1*512kB (U) 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 14160kB
[559103.153111] Node 0 DMA32: 191*4kB (U) 304*8kB (U) 220*16kB (U) 118*32kB (UE) 27*64kB (UEM) 5*128kB (U) 6*256kB (UE) 6*512kB (UM) 2*1024kB (UE) 1*2048kB (M) 56*4096kB (M) = 250940kB
[559103.153130] Node 0 Normal: 10425*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41700kB
[559103.153143] Node 1 Normal: 10949*4kB (UE) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 43796kB
[559103.153158] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[559103.153161] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[559103.153164] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[559103.153167] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[559103.153169] 13708 total pagecache pages
[559103.153173] 7613 pages in swap cache
[559103.153176] Swap cache stats: add 5375963, delete 5366824, find 287337/304675
[559103.153178] Free swap = 0kB
[559103.153180] Total swap = 4194300kB
[559103.153182] 33527464 pages RAM
[559103.153184] 0 pages HighMem/MovableOnly
[559103.153186] 590147 pages reserved
[559103.153188] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[559103.153209] [ 962] 0 962 23190 6017 51 1 0 systemd-journal
[559103.153214] [ 997] 0 997 50277 9 31 824 0 lvmetad
[559103.153219] [ 1009] 0 1009 12390 152 26 638 -1000 systemd-udevd
[559103.153225] [ 1361] 0 1361 13883 99 26 10 -1000 auditd
[559103.153229] [ 1392] 0 1392 22652 8 47 223 0 rngd
[559103.153233] [ 1394] 0 1394 5479 97 15 50 0 irqbalance
[559103.153237] [ 1396] 0 1396 6596 42 19 31 0 systemd-logind
[559103.153242] [ 1397] 81 1397 16621 98 34 96 -900 dbus-daemon
[559103.153246] [ 1430] 0 1430 119577 742 85 278 0 NetworkManager
[559103.153251] [ 1439] 997 1439 153083 263 62 2157 0 polkitd
[559103.153255] [ 1444] 996 1444 5634 73 17 0 0 chronyd
[559103.153272] [ 1464] 0 1464 28917 110 11 10 0 ksmtuned
[559103.153277] [ 1473] 0 1473 50358 78 38 56 0 gssproxy
[559103.153281] [ 1512] 0 1512 6477 5 18 48 0 atd
[559103.153285] [ 1513] 0 1513 31598 158 19 0 0 crond
[559103.153289] [ 1615] 0 1615 28235 258 58 0 -1000 sshd
[559103.153293] [ 1618] 0 1618 143572 3252 98 115 0 tuned
[559103.153297] [ 1629] 802 1629 1897749 23436 350 6374 0 agent
[559103.153301] [ 1633] 802 1633 1298886 11722 175 491 0 trace-agent
[559103.153305] [ 1811] 0 1811 5840 9 14 96 0 ipmievd
[559103.153324] [ 1863] 0 1863 27552 33 10 0 0 agetty
[559103.153329] [ 2678] 0 2678 22437 26 43 243 0 master
[559103.153332] [ 2740] 89 2740 22507 12 45 260 0 qmgr
[559103.153336] [ 2905] 6000 2905 77809 1891 103 1636 0 Xvnc
[559103.153340] [ 4069] 6000 4069 43378 1046 40 6 0 xterm
[559103.153344] [ 4071] 6000 4071 28920 83 13 40 0 bash
[559103.153348] [ 4088] 0 4088 25766 547 52 10 0 dhclient
[559103.153352] [ 4185] 0 4185 66340 3310 61 133 0 rsyslogd
[559103.153356] [ 6542] 6000 6542 72442 1432 28 6 0 python3
[559103.153360] [ 7397] 6000 7397 79928 4616 75 592 0 buildbot-worker
[559103.153367] [50574] 6000 50574 629298 160 23 795 0 qemu-mips.real
[559103.153376] [34431] 6000 34431 35612 1499 24 0 0 python3
[559103.153379] [34434] 6000 34434 28321 55 12 0 0 sh
[559103.153383] [34476] 6000 34476 51898 5868 27 0 0 KnottyUI
[559103.153387] [34561] 6000 34561 226921 162461 336 6 0 Cooker
[559103.153391] [ 9298] 6000 9298 107726 61941 137 1 0 Worker
[559103.153395] [45389] 6000 45389 1351 38 6 0 0 pseudo
[559103.153399] [45440] 6000 45440 109044 63144 142 3 0 Worker (Fakeroo
[559103.153404] [15508] 6000 15508 34014 32276 70 2 0 pseudo
[559103.153408] [45145] 6000 45145 35612 1505 26 0 0 python3
[559103.153412] [45147] 6000 45147 28321 54 12 0 0 sh
[559103.153416] [45193] 6000 45193 304848 92624 223 0 0 python3.9.real
[559103.153421] [47880] 6000 47880 72364 8063 32 0 0 python3.9.real
[559103.153425] [48805] 6000 48805 50767 4874 25 0 0 KnottyUI
[559103.153429] [49111] 6000 49111 84581 36415 90 0 0 Cooker
[559103.153433] [54112] 6000 54112 63876 18106 53 0 0 Worker
[559103.153437] [32080] 6000 32080 66769 20986 58 0 0 glibc-testsuite
[559103.153441] [32423] 6000 32423 28888 83 13 0 0 run.do_check.32
[559103.153445] [36150] 6000 36150 24799 92 10 0 0 make
[559103.153451] [36160] 6000 36160 25514 560 11 0 0 make
[559103.153457] [ 7870] 6000 7870 27041 74 10 0 0 rm
[559103.153461] [38241] 6000 38241 29485 4766 20 0 0 make
[559103.153464] [47836] 6000 47836 28325 72 12 0 0 bash
[559103.153468] [47837] 6000 47837 26944 1041 12 0 0 python3.9.real
[559103.153472] [47841] 6000 47841 1104893 1179 21 11 0 qemu-arm
[559103.153476] [47844] 6000 47844 32709572 30734711 62165 556065 0 qemu-arm
[559103.153481] [36569] 6000 36569 108621 62816 143 3 0 lib32-piglit:pa
[559103.153484] [37472] 6000 37472 28999 103 13 0 0 run.BUILDSPEC.3
[559103.153488] [37481] 6000 37481 1117322 111089 395 1 0 rpmbuild.real
[559103.153493] [20411] 89 20411 22463 270 44 0 0 pickup
[559103.153497] [27599] 6000 27599 109023 63286 141 3 0 ltp:package_qa
[559103.153500] [27610] 6000 27610 8682 6944 21 0 0 pseudo
[559103.153504] [47733] 0 47733 27014 18 10 0 0 sleep
[559103.153508] [48851] 6000 48851 109023 63234 141 3 0 ltp:package_qa
[559103.153512] Out of memory: Kill process 47844 (qemu-arm) score 922 or sacrifice child
[559103.153577] Killed process 47844 (qemu-arm), UID 6000, total-vm:130838288kB, anon-rss:122938644kB, file-rss:200kB, shmem-rss:0kB
[559103.154704] agent invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[559103.154709] agent cpuset=/ mems_allowed=0-1
[559103.154713] CPU: 13 PID: 3688 Comm: agent Kdump: loaded Not tainted 3.10.0-1160.25.1.el7.x86_64 #1
[559103.154716] Hardware name: Silicon Mechanics Rackform R346.v5/X10DRI-T, BIOS 1.1 04/14/2015
[559103.154717] Call Trace:
[559103.154728] [<ffffffff95d8311a>] dump_stack+0x19/0x1b
[559103.154732] [<ffffffff95d7da3a>] dump_header+0x90/0x229
[559103.154738] [<ffffffff9590ec6b>] ? cred_has_capability+0x6b/0x120
[559103.154744] [<ffffffff957c232d>] oom_kill_process+0x2cd/0x490
[559103.154748] [<ffffffff9590ed3c>] ? selinux_capable+0x1c/0x40
[559103.154752] [<ffffffff957c2a1a>] out_of_memory+0x31a/0x500
[559103.154756] [<ffffffff95d7e557>] __alloc_pages_slowpath+0x5db/0x729
[559103.154761] [<ffffffff957c8f96>] __alloc_pages_nodemask+0x436/0x450
[559103.154767] [<ffffffff95818c88>] alloc_pages_current+0x98/0x110
[559103.154771] [<ffffffff957bdde7>] __page_cache_alloc+0x97/0xb0
[559103.154775] [<ffffffff957c0d80>] filemap_fault+0x270/0x420
[559103.154834] [<ffffffffc06a391e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
[559103.154856] [<ffffffffc06a3b1c>] xfs_filemap_fault+0x2c/0x30 [xfs]
[559103.154860] [<ffffffff957ee08a>] __do_fault.isra.61+0x8a/0x100
[559103.154864] [<ffffffff957ee63c>] do_read_fault.isra.63+0x4c/0x1b0
[559103.154868] [<ffffffff957f5e80>] handle_mm_fault+0xa20/0xfb0
[559103.154872] [<ffffffff95d90653>] __do_page_fault+0x213/0x500
[559103.154875] [<ffffffff95d90975>] do_page_fault+0x35/0x90
[559103.154878] [<ffffffff95d8c778>] page_fault+0x28/0x30
[559103.154880] Mem-Info:
[559103.154892] active_anon:30121623 inactive_anon:1346058 isolated_anon:0
active_file:55 inactive_file:0 isolated_file:0
unevictable:0 dirty:4 writeback:0 unstable:0
slab_reclaimable:252392 slab_unreclaimable:68275
mapped:4278 shmem:4772 pagetables:66162 bounce:0
free:88489 free_pcp:11181 free_cma:0
[559103.154898] Node 0 DMA free:14160kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[559103.154908] lowmem_reserve[]: 0 1695 64154 64154
[559103.154912] Node 0 DMA32 free:250936kB min:1184kB low:1480kB high:1776kB active_anon:1097636kB inactive_anon:366004kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1973304kB managed:1735704kB mlocked:0kB dirty:0kB writeback:0kB mapped:8kB shmem:0kB slab_reclaimable:2304kB slab_unreclaimable:2028kB kernel_stack:96kB pagetables:2772kB unstable:0kB bounce:0kB free_pcp:7172kB local_pcp:716kB free_cma:0kB writeback_tmp:0kB pages_scanned:267 all_unreclaimable? yes
[559103.154920] lowmem_reserve[]: 0 0 62459 62459
[559103.154925] Node 0 Normal free:43872kB min:43744kB low:54680kB high:65616kB active_anon:58654928kB inactive_anon:2487048kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63958384kB mlocked:0kB dirty:12kB writeback:0kB mapped:7624kB shmem:8400kB slab_reclaimable:514868kB slab_unreclaimable:136060kB kernel_stack:7232kB pagetables:126652kB unstable:0kB bounce:0kB free_pcp:22360kB local_pcp:668kB free_cma:0kB writeback_tmp:0kB pages_scanned:174 all_unreclaimable? yes
[559103.154933] lowmem_reserve[]: 0 0 0 0
[559103.154937] Node 1 Normal free:44988kB min:45164kB low:56452kB high:67744kB active_anon:60733928kB inactive_anon:2530668kB active_file:620kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66039288kB mlocked:0kB dirty:4kB writeback:0kB mapped:9480kB shmem:10688kB slab_reclaimable:492396kB slab_unreclaimable:135012kB kernel_stack:5696kB pagetables:135224kB unstable:0kB bounce:0kB free_pcp:15192kB local_pcp:132kB free_cma:0kB writeback_tmp:0kB pages_scanned:3283 all_unreclaimable? yes
[559103.154945] lowmem_reserve[]: 0 0 0 0
[559103.154948] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 0*32kB 1*64kB (U) 0*128kB 1*256kB (U) 1*512kB (U) 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 14160kB
[559103.154962] Node 0 DMA32: 191*4kB (U) 304*8kB (U) 220*16kB (U) 118*32kB (UE) 27*64kB (UEM) 5*128kB (U) 6*256kB (UE) 6*512kB (UM) 2*1024kB (UE) 1*2048kB (M) 56*4096kB (M) = 250940kB
[559103.154978] Node 0 Normal: 10487*4kB (UEM) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41948kB
[559103.154988] Node 1 Normal: 10949*4kB (UE) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 43796kB
[559103.155000] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[559103.155002] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[559103.155004] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[559103.155006] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[559103.155008] 13635 total pagecache pages
[559103.155011] 7573 pages in swap cache
[559103.155013] Swap cache stats: add 5375963, delete 5366864, find 287337/304675
[559103.155014] Free swap = 0kB
[559103.155016] Total swap = 4194300kB
[559103.155018] 33527464 pages RAM
[559103.155019] 0 pages HighMem/MovableOnly
[559103.155021] 590147 pages reserved
[559103.155022] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[559103.155041] [ 962] 0 962 23190 6017 51 1 0 systemd-journal
[559103.155044] [ 997] 0 997 50277 9 31 824 0 lvmetad
[559103.155048] [ 1009] 0 1009 12390 152 26 638 -1000 systemd-udevd
[559103.155053] [ 1361] 0 1361 13883 99 26 10 -1000 auditd
[559103.155056] [ 1392] 0 1392 22652 8 47 223 0 rngd
[559103.155059] [ 1394] 0 1394 5479 97 15 50 0 irqbalance
[559103.155063] [ 1396] 0 1396 6596 42 19 31 0 systemd-logind
[559103.155066] [ 1397] 81 1397 16621 98 34 96 -900 dbus-daemon
[559103.155070] [ 1430] 0 1430 119577 742 85 278 0 NetworkManager
[559103.155073] [ 1439] 997 1439 153083 263 62 2157 0 polkitd
[559103.155076] [ 1444] 996 1444 5634 73 17 0 0 chronyd
[559103.155080] [ 1464] 0 1464 28917 110 11 10 0 ksmtuned
[559103.155083] [ 1473] 0 1473 50358 78 38 56 0 gssproxy
[559103.155086] [ 1512] 0 1512 6477 5 18 48 0 atd
[559103.155089] [ 1513] 0 1513 31598 158 19 0 0 crond
[559103.155093] [ 1615] 0 1615 28235 258 58 0 -1000 sshd
[559103.155096] [ 1618] 0 1618 143572 3252 98 115 0 tuned
[559103.155099] [ 1629] 802 1629 1897749 23436 350 6374 0 agent
[559103.155103] [ 1633] 802 1633 1298886 11722 175 491 0 trace-agent
[559103.155106] [ 1811] 0 1811 5840 9 14 96 0 ipmievd
[559103.155109] [ 1863] 0 1863 27552 33 10 0 0 agetty
[559103.155113] [ 2678] 0 2678 22437 26 43 243 0 master
[559103.155116] [ 2740] 89 2740 22507 12 45 260 0 qmgr
[559103.155120] [ 2905] 6000 2905 77809 1891 103 1636 0 Xvnc
[559103.155123] [ 4069] 6000 4069 43378 1046 40 6 0 xterm
[559103.155126] [ 4071] 6000 4071 28920 83 13 40 0 bash
[559103.155129] [ 4088] 0 4088 25766 547 52 10 0 dhclient
[559103.155133] [ 4185] 0 4185 66340 3310 61 133 0 rsyslogd
[559103.155137] [ 6542] 6000 6542 72442 1432 28 6 0 python3
[559103.155140] [ 7397] 6000 7397 79928 4616 75 592 0 buildbot-worker
[559103.155146] [50574] 6000 50574 629298 160 23 795 0 qemu-mips.real
[559103.155154] [34431] 6000 34431 35612 1499 24 0 0 python3
[559103.155157] [34434] 6000 34434 28321 55 12 0 0 sh
[559103.155161] [34476] 6000 34476 51898 5867 27 0 0 KnottyUI
[559103.155164] [34561] 6000 34561 226921 162459 336 6 0 Cooker
[559103.155167] [ 9298] 6000 9298 107726 61941 137 1 0 Worker
[559103.155170] [45389] 6000 45389 1351 38 6 0 0 pseudo
[559103.155174] [45440] 6000 45440 109044 63144 142 3 0 Worker (Fakeroo
[559103.155178] [15508] 6000 15508 34014 32276 70 2 0 pseudo
[559103.155182] [45145] 6000 45145 35612 1505 26 0 0 python3
[559103.155185] [45147] 6000 45147 28321 54 12 0 0 sh
[559103.155188] [45193] 6000 45193 304848 92624 223 0 0 python3.9.real
[559103.155192] [47880] 6000 47880 72364 8063 32 0 0 python3.9.real
[559103.155196] [48805] 6000 48805 50767 4874 25 0 0 KnottyUI
[559103.155199] [49111] 6000 49111 84581 36415 90 0 0 Cooker
[559103.155202] [54112] 6000 54112 63876 18106 53 0 0 Worker
[559103.155206] [32080] 6000 32080 66769 20986 58 0 0 glibc-testsuite
[559103.155209] [32423] 6000 32423 28888 83 13 0 0 run.do_check.32
[559103.155212] [36150] 6000 36150 24799 92 10 0 0 make
[559103.155215] [36160] 6000 36160 25514 560 11 0 0 make
[559103.155220] [ 7870] 6000 7870 27041 74 10 0 0 rm
[559103.155223] [38241] 6000 38241 29485 4766 20 0 0 make
[559103.155226] [47836] 6000 47836 28325 72 12 0 0 bash
[559103.155230] [47837] 6000 47837 26944 1041 12 0 0 python3.9.real
[559103.155233] [47841] 6000 47841 1104893 1179 21 11 0 qemu-arm
[559103.155238] [36569] 6000 36569 108621 62817 143 3 0 lib32-piglit:pa
[559103.155241] [37472] 6000 37472 28999 103 13 0 0 run.BUILDSPEC.3
[559103.155244] [37481] 6000 37481 1117322 111087 395 1 0 rpmbuild.real
[559103.155248] [20411] 89 20411 22463 270 44 0 0 pickup
[559103.155251] [27599] 6000 27599 109023 63284 141 3 0 ltp:package_qa
[559103.155254] [27610] 6000 27610 8682 6944 21 0 0 pseudo
[559103.155268] [47733] 0 47733 27014 18 10 0 0 sleep
[559103.155272] [48851] 6000 48851 109023 63231 141 3 0 ltp:package_qa
[559103.155275] Out of memory: Kill process 34561 (Cooker) score 4 or sacrifice child
[559103.155326] Killed process 9298 (Worker), UID 6000, total-vm:430904kB, anon-rss:247764kB, file-rss:0kB, shmem-rss:0kB


SWAT Rotation

Alexandre Belloni
 

Hello Jon,

SWAT duty will rotate from Chee Yang to you at the end of this Friday.

Please reply to let me know whether you will be able to work on this
task next week.

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


Debug data for ltp qemu hang on ubuntu1804-arm-1

Richard Purdie
 

Hi All,

We had a hanging qemu process:


https://autobuilder.yoctoproject.org/typhoon/#/builders/96/builds/1748

This is on ubuntu1804-arm-1.yocto.io.

I was able to ssh into the image on the worker with ssh root@192.168.7.2
and it looks to be hung during fs ltp tests. I've dumped some 
output below, I'm not too familiar with ltp so struggling to understand
where it is at.

The processes are still around so we may be able to save data from
them but the servers will be rebooted tomorrow during maint at which
point we'd lose this unless we ask Michael to defer.

It looks like the proc01 -m 128 is hanging.

Cheers,

Richard

root@qemuarm64:~# ps ax
PID TTY STAT TIME COMMAND
1 ? Ss 1:45 init [5]
2 ? S 0:00 [kthreadd]
3 ? I< 0:00 [rcu_gp]
4 ? I< 0:00 [rcu_par_gp]
6 ? I< 0:00 [kworker/0:0H-events_highpri]
8 ? I< 0:00 [mm_percpu_wq]
9 ? S 0:00 [rcu_tasks_kthre]
10 ? S 0:00 [rcu_tasks_rude_]
11 ? S 0:00 [rcu_tasks_trace]
12 ? S 0:42 [ksoftirqd/0]
13 ? I 0:08 [rcu_preempt]
14 ? S 0:00 [migration/0]
15 ? S 0:00 [cpuhp/0]
16 ? S 0:00 [kdevtmpfs]
17 ? I< 0:00 [netns]
19 ? S 0:00 [oom_reaper]
20 ? I< 0:00 [writeback]
21 ? S 0:04 [kcompactd0]
54 ? I< 0:00 [kblockd]
55 ? I< 0:00 [blkcg_punt_bio]
56 ? I< 0:00 [md]
57 ? I< 0:06 [kworker/0:1H-kblockd]
58 ? I< 0:00 [rpciod]
59 ? I< 0:00 [kworker/u3:0]
60 ? I< 0:00 [xprtiod]
61 ? S 0:01 [kswapd0]
62 ? I< 0:00 [nfsiod]
63 ? I< 0:00 [cifsiod]
64 ? I< 0:00 [smb3decryptd]
65 ? I< 0:00 [cifsfileinfoput]
66 ? I< 0:00 [cifsoplockd]
68 ? S 0:05 [hwrng]
69 ? I< 0:00 [ttm_swap]
71 ? S 0:00 [khvcd]
73 ? I< 0:00 [raid5wq]
74 ? I< 0:00 [dm_bufio_cache]
75 ? I< 0:00 [ipv6_addrconf]
80 ? S 0:24 [jbd2/vda-8]
81 ? I< 0:00 [ext4-rsv-conver]
116 ? Ss 0:00 /sbin/udevd -d
445 ? Ss 0:00 /usr/bin/dbus-daemon --system
453 ? Ss 3:47 /usr/sbin/connmand
460 ? S 0:00 xinit /etc/X11/Xsession -- /usr/bin/Xorg :0 -br -pn
463 ? S 0:58 /usr/sbin/wpa_supplicant -u
467 ? S<sl 1:07 /usr/bin/Xorg :0 -br -pn
471 ? Ss 0:00 /usr/sbin/dropbear -r /etc/dropbear/dropbear_rsa_host_key -p 22 -B
476 ? Ss 0:25 /usr/sbin/rpcbind
491 ? S 0:00 matchbox-window-manager -theme Sato -use_cursor yes
496 ? S 0:00 dbus-launch --sh-syntax --exit-with-session
497 ? Ss 0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 5 --print-address 7 --session
514 ? Sl 0:00 /usr/libexec/at-spi-bus-launcher --launch-immediately
518 ? S 0:00 connman-applet
534 ? S 0:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
553 ? S 0:00 matchbox-desktop
554 ? S 10:50 matchbox-panel --start-applets showdesktop,windowselector --end-applets clock,,systray,startup-notify,notify
557 ? Sl 0:00 /usr/sbin/console-kit-daemon --no-daemon
621 ? S 1:41 /usr/libexec/gconfd-2
623 ? S 0:00 /usr/bin/settings-daemon
638 ? S 0:00 avahi-daemon: running [qemuarm64.local]
639 ? S 0:00 avahi-daemon: chroot helper
643 ? Ss 0:00 /usr/sbin/ofonod
648 ? Sl 0:00 /usr/libexec/at-spi2-registryd --use-gnome-session
650 ? Ss 1:22 /usr/sbin/crond
660 ? Ss 0:00 /bin/sh /bin/start_getty 115200 ttyAMA0 vt102
661 ? Ss 0:00 /bin/sh /bin/start_getty 115200 hvc0 vt102
662 tty1 Ss+ 0:00 /sbin/getty 38400 tty1
665 hvc0 Ss+ 0:00 /sbin/getty -L 115200 hvc0 vt102
666 ttyAMA0 Ss+ 0:00 -sh
7112 ? S 0:02 /usr/sbin/dropbear -r /etc/dropbear/dropbear_rsa_host_key -p 22 -B
7113 ? S 0:00 /bin/sh /opt/ltp/runltp -f fs -p -q -r /opt/ltp -l /opt/ltp/results/fs -I 1 -d /opt/ltp
7155 ? S 0:00 /opt/ltp/bin/ltp-pan -q -e -S -a 7113 -n 7113 -p -f /opt/ltp/ltp-HZxaOUahq7/alltests -l /opt/ltp/results/fs -C /opt/ltp/output/LTP_RUN_ON-fs.failed -T /opt/ltp/output/LTP_RUN_ON-fs.tconf
9361 ? S 0:00 /sbin/syslogd -n -O /var/log/messages
9364 ? S 0:01 /sbin/klogd -n
11412 ? I 0:00 [kworker/u2:0-ext4-rsv-conversion]
11413 ? I 0:00 [kworker/u2:3-events_unbound]
11578 ? S 0:00 proc01 -m 128
11580 ? S 0:00 /usr/sbin/dropbear -r /etc/dropbear/dropbear_rsa_host_key -p 22 -B
11581 pts/0 Ss 0:00 -sh
11585 pts/0 R+ 0:00 ps ax
27489 ? I 0:00 [kworker/0:1-cgroup_pidlist_destroy]
27490 ? I 0:55 [kworker/0:4-events]

root@qemuarm64:~# tail /opt/ltp/results/fs
ftest02 PASS 0
ftest03 PASS 0
ftest04 PASS 0
ftest05 PASS 0
ftest06 PASS 0
ftest07 PASS 0
ftest08 PASS 0
lftest01 PASS 0
writetest01 PASS 0
fs_di PASS 0
root@qemuarm64:~# tail /opt/ltp/output/LTP_RUN_ON-fs.failed

Lots in dmesg but the last bit shows handy timestamp:

[10052.451191] cgroup: fork rejected by pids controller in /ltp_pids_9/child2
[10052.500603] new mount options do not match the existing superblock, will be ignored
[10052.543519] cgroup: fork rejected by pids controller in /ltp_pids_9/child2
[10052.619108] new mount options do not match the existing superblock, will be ignored
[10052.733366] cgroup: fork rejected by pids controller in /ltp_pids_9/child2
[10052.942680] new mount options do not match the existing superblock, will be ignored
[10053.146175] cgroup: fork rejected by pids controller in /ltp_pids_9/child2
[10092.243267] input: virtual-device-ltp as /devices/virtual/input/input2
[10092.327062] input: virtual-device-ltp as /devices/virtual/input/input3
[10092.422283] input: virtual-device-ltp as /devices/virtual/input/input4
[10092.436818] input: virtual-device-ltp as /devices/virtual/input/input5
[10092.517318] input: virtual-device-ltp as /devices/virtual/input/input6
[10092.596125] input: virtual-device-ltp as /devices/virtual/input/input7
[11205.107302] ICMPv6: process `proc01' is using deprecated sysctl (syscall) net.ipv6.neigh.default.base_reachable_time - use net.ipv6.neigh.default.base_reachable_time_ms instead


Debug data for ltp qemu hang on debian10-ty-2

Richard Purdie
 

Hi All,

We had a hanging qemu process:

https://autobuilder.yoctoproject.org/typhoon/#/builders/95/builds/1898

This is on debian10-ty-2.yocto.io.

I was able to ssh into the image on the worker with ssh root@192.168.7.2
and it looks to be hung during controller ltp tests. I've dumped some 
output below, I'm not too familiar with ltp so struggling to understand
where it is at.

The processes are still around so we may be able to save data from
them but the servers will be rebooted tomorrow during maint at which
point we'd lose this unless we ask Michael to defer.

Cheers,

Richard


root@qemux86-64:/opt/ltp/results# ps ax
PID TTY STAT TIME COMMAND
1 ? Ss 0:03 init [5]
2 ? S 0:00 [kthreadd]
3 ? I< 0:00 [rcu_gp]
4 ? I< 0:00 [rcu_par_gp]
6 ? I< 0:00 [kworker/0:0H-events_highpri]
8 ? I< 0:00 [mm_percpu_wq]
9 ? S 0:00 [rcu_tasks_kthre]
10 ? S 0:00 [rcu_tasks_rude_]
11 ? S 0:00 [rcu_tasks_trace]
12 ? S 0:00 [ksoftirqd/0]
13 ? I 0:06 [rcu_preempt]
14 ? S 0:00 [migration/0]
15 ? S 0:00 [cpuhp/0]
16 ? S 0:00 [cpuhp/1]
17 ? S 0:00 [migration/1]
18 ? S 0:00 [ksoftirqd/1]
20 ? I< 0:00 [kworker/1:0H-events_highpri]
21 ? S 0:00 [cpuhp/2]
22 ? S 0:00 [migration/2]
23 ? S 0:00 [ksoftirqd/2]
25 ? I< 0:00 [kworker/2:0H-kblockd]
26 ? S 0:00 [cpuhp/3]
27 ? S 0:00 [migration/3]
28 ? S 0:00 [ksoftirqd/3]
30 ? I< 0:00 [kworker/3:0H-kblockd]
31 ? S 0:00 [kdevtmpfs]
32 ? I< 0:00 [netns]
34 ? S 0:00 [oom_reaper]
35 ? I< 0:00 [writeback]
36 ? S 0:07 [kcompactd0]
73 ? I< 0:00 [kblockd]
74 ? I< 0:00 [blkcg_punt_bio]
75 ? I< 0:00 [ata_sff]
76 ? I< 0:00 [md]
78 ? I< 0:00 [kworker/0:1H-kblockd]
79 ? I< 0:00 [rpciod]
80 ? I< 0:00 [kworker/u9:0]
81 ? I< 0:00 [xprtiod]
82 ? S 0:00 [kswapd0]
83 ? I< 0:00 [nfsiod]
84 ? I< 0:00 [cifsiod]
85 ? I< 0:00 [smb3decryptd]
86 ? I< 0:00 [cifsfileinfoput]
87 ? I< 0:00 [cifsoplockd]
89 ? I< 0:00 [acpi_thermal_pm]
91 ? S 0:00 [hwrng]
92 ? I< 0:00 [ttm_swap]
93 ? I< 0:00 [nvme-wq]
94 ? I< 0:00 [nvme-reset-wq]
95 ? I< 0:00 [nvme-delete-wq]
96 ? S 0:00 [scsi_eh_0]
97 ? I< 0:00 [scsi_tmf_0]
98 ? S 0:00 [scsi_eh_1]
99 ? I< 0:00 [scsi_tmf_1]
100 ? S 0:00 [scsi_eh_2]
101 ? I< 0:00 [scsi_tmf_2]
102 ? S 0:00 [scsi_eh_3]
103 ? I< 0:00 [scsi_tmf_3]
104 ? S 0:00 [scsi_eh_4]
105 ? I< 0:00 [scsi_tmf_4]
106 ? S 0:00 [scsi_eh_5]
107 ? I< 0:00 [scsi_tmf_5]
114 ? I< 0:00 [raid5wq]
115 ? I< 0:00 [dm_bufio_cache]
116 ? I< 0:00 [ipv6_addrconf]
124 ? S 0:03 [jbd2/vda-8]
125 ? I< 0:00 [ext4-rsv-conver]
160 ? Ss 0:00 /sbin/udevd -d
358 ? I< 0:00 [kworker/2:1H-kblockd]
525 ? Ss 0:00 /usr/bin/dbus-daemon --system
533 ? Ss 0:05 /usr/sbin/connmand
539 ? S 0:00 xinit /etc/X11/Xsession -- /usr/bin/Xorg :0 -br -pn
545 ? S 0:01 /usr/sbin/wpa_supplicant -u
548 ? S<sl 0:12 /usr/bin/Xorg :0 -br -pn
551 ? S 0:00 matchbox-window-manager -theme Sato -use_cursor no
556 ? S 0:00 dbus-launch --sh-syntax --exit-with-sessionsh-keygen -f "/home/pokybuild/.ssh/known_hosts" -R "192.168.7.2"
557 ? Ss 0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 5 --print-address 7 --session
576 ? Sl 0:00 /usr/libexec/at-spi-bus-launcher --launch-immediately
580 ? Sl 0:00 connman-applet
602 ? S 0:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
612 ? Sl 0:00 matchbox-desktop
613 ? Sl 0:06 matchbox-panel --start-applets showdesktop,windowselector --end-applets clock,,systray,startup-notify,notify
616 ? S 0:00 /usr/libexec/gconfd-2
620 ? Ss 0:00 /usr/sbin/dropbear -r /etc/dropbear/dropbear_rsa_host_key -p 22 -B
621 ? S 0:00 /usr/bin/settings-daemon
628 ? Sl 0:00 /usr/sbin/console-kit-daemon --no-daemon
629 ? Ss 0:00 /usr/sbin/rpcbind
700 ? S 0:00 /usr/libexec/bluetooth/bluetoothd
712 ? Sl 0:00 /usr/libexec/at-spi2-registryd --use-gnome-session
732 ? S 0:00 avahi-daemon: running [qemux86-64.local]
733 ? S 0:00 avahi-daemon: chroot helper
737 ? Ss 0:00 /usr/sbin/ofonod
741 ? Ss 0:00 /usr/sbin/crond
750 ? Ss 0:00 /bin/sh /bin/start_getty 115200 ttyS0 vt102
751 ? Ss 0:00 /bin/sh /bin/start_getty 115200 ttyS1 vt102
752 tty1 Ss+ 0:00 /sbin/getty 38400 tty1
755 ttyS0 Ss+ 0:00 /sbin/getty -L 115200 ttyS0 vt102
756 ttyS1 Ss+ 0:00 -sh
760 ? I< 0:00 [kworker/1:1H-kblockd]
1087 ? I< 0:00 [kworker/3:1H-kblockd]
3886 ? D 0:00 cgroup_xattr
3887 ? R 0:00 /usr/sbin/dropbear -r /etc/dropbear/dropbear_rsa_host_key -p 22 -B
3888 pts/0 Ss 0:00 -sh
3892 ? I 0:00 [kworker/u8:2-events_unbound]
3906 ? I 0:00 [kworker/u8:0-events_unbound]
3911 ? S 0:00 /usr/sbin/dropbear -r /etc/dropbear/dropbear_rsa_host_key -p 22 -B
3912 pts/1 Ss+ 0:00 -sh
3918 pts/0 R+ 0:00 ps ax
4451 ? I 0:00 [kworker/2:1-mm_percpu_wq]
4454 ? I 0:00 [kworker/1:0-rcu_gp]
4569 ? I 0:00 [kworker/2:4-rcu_gp]
4620 ? I 0:00 [kworker/0:2-rcu_gp]
5157 ? S 0:00 /usr/sbin/dropbear -r /etc/dropbear/dropbear_rsa_host_key -p 22 -B
5158 ? S 0:00 /bin/sh /opt/ltp/runltp -f controllers -p -q -r /opt/ltp -l /opt/ltp/results/controllers -I 1 -d /opt/ltp
5200 ? S 0:00 /opt/ltp/bin/ltp-pan -q -e -S -a 5158 -n 5158 -p -f /opt/ltp/ltp-qYESo05pbN/alltests -l /opt/ltp/results/controllers -C /opt/ltp/output/LTP_RUN_ON-controllers.failed -T /opt/ltp/output/LTP_RUN_ON-controllers.tconf
12503 ? I 0:00 [kworker/u8:1-events_unbound]
12921 ? I 0:00 [kworker/1:2-mm_percpu_wq]
12961 ? I 0:00 [kworker/0:0-events]
13462 ? I 0:00 [kworker/3:3-rcu_gp]
14966 ? S 0:00 /sbin/syslogd -n -O /var/log/messages
14969 ? S 0:01 /sbin/klogd -n
21420 ? I 0:00 [kworker/3:0-mm_percpu_wq]

root@qemux86-64:/opt/ltp/results# ls -la
total 172
drwxr-xr-x 2 root root 4096 May 11 10:55 .
drwxr-xr-x 13 root root 4096 May 11 10:55 ..
-rw-r--r-- 1 root root 6207 May 11 10:55 containers
-rw-r--r-- 1 root root 20494 May 11 10:58 controllers
-rw-r--r-- 1 root root 2466 May 11 10:46 dio
-rw-r--r-- 1 root root 561 May 11 10:46 io
-rw-r--r-- 1 root root 969 May 11 10:53 ipc
-rw-r--r-- 1 root root 1106 May 11 10:22 math
-rw-r--r-- 1 root root 5255 May 11 10:53 mm
-rw-r--r-- 1 root root 493 May 11 10:55 nptl
-rw-r--r-- 1 root root 901 May 11 10:55 pty
-rw-r--r-- 1 root root 1106 May 11 10:55 sched
-rw-r--r-- 1 root root 91210 May 11 10:40 syscalls

root@qemux86-64:/opt/ltp/results# tail controllers
cpuset_exclusive CONF 32
cpuset_hierarchy CONF 32
cpuset_syscall CONF 32
cpuset_sched_domains CONF 32
cpuset_load_balance CONF 32
cpuset_hotplug CONF 32
cpuset_memory CONF 32
cpuset_memory_pressure CONF 32
cpuset_memory_spread CONF 32
cpuset_regression_test FAIL 6


root@qemux86-64:/opt/ltp/results# tail ../output/LTP_RUN_ON-controllers.failed
cgroup_fj_stress_net_prio_10_3_each cgroup_fj_stress.sh net_prio 10 3 each
cgroup_fj_stress_net_prio_1_200_each cgroup_fj_stress.sh net_prio 1 200 each
cgroup_fj_stress_net_prio_200_1_each cgroup_fj_stress.sh net_prio 200 1 each
cpuacct_1_1 cpuacct.sh 1 1
cpuacct_1_10 cpuacct.sh 1 10
cpuacct_10_10 cpuacct.sh 10 10
cpuacct_1_100 cpuacct.sh 1 100
cpuacct_100_1 cpuacct.sh 100 1
cpuacct_100_100 cpuacct.sh 100 100
cpuset_regression_test cpuset_regression_test.sh


Re: SWAT Rotation

Lee Chee Yang
 

Noted. Let me know when swatbot account is ready.

BTW, is there a timetable for the SWAT rotation?
So we can check the schedule and inform ahead if there is any needs for reschedule


Chee Yang

-----Original Message-----
From: swat@lists.yoctoproject.org <swat@lists.yoctoproject.org> On Behalf
Of Alexandre Belloni
Sent: Friday, 7 May, 2021 6:50 AM
To: Lee, Chee Yang <chee.yang.lee@intel.com>
Cc: swat@lists.yoctoproject.org; Stephen Jolley <sjolley.yp.pm@gmail.com>
Subject: [swat] SWAT Rotation

Hello Lee Chee,

SWAT duty will rotate from me to you at the end of this Friday.

Please reply to let me know whether you will be able to work on this task next
week.

I'll be available to walk you through the swatbot process on Monday, don't
hesitate to contact me by email or on IRC.
I'll create your swatbot account soon.

--
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel
engineering https://bootlin.com




SWAT Rotation

Alexandre Belloni
 

Hello Lee Chee,

SWAT duty will rotate from me to you at the end of this Friday.

Please reply to let me know whether you will be able to work on this
task next week.

I'll be available to walk you through the swatbot process on Monday,
don't hesitate to contact me by email or on IRC.
I'll create your swatbot account soon.

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


Re: SWAT Rotation

Alexandre Belloni
 

Hello,

On 03/05/2021 19:34:55+0000, Larson, Chris wrote:
Apolzogies, I'm swamped this week and am dealing with an injured kid, I'll need to shift my position to a future week again.
That's OK, I didn't do a great job last week and I'll handle SWAT this
week.


--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


Re: SWAT Rotation

Larson, Chris
 

Apologies, I'm swamped this week and am dealing with an injured kid, I'll need to shift my position to a future week again.

-Chris

-----Original Message-----
From: swat@lists.yoctoproject.org <swat@lists.yoctoproject.org> On Behalf Of Alexandre Belloni
Sent: Thursday, April 29, 2021 3:09 PM
To: Christopher Larson <kergoth@gmail.com>
Cc: swat@lists.yoctoproject.org; Stephen Jolley <sjolley.yp.pm@gmail.com>
Subject: [swat] SWAT Rotation

Hello Christopher,

SWAT duty will rotate from me to you at the end of this Friday.

Please reply to let me know whether you will be able to work on this task next week.

I'll be available to walk you through the swatbot process on Monday, don't hesitate to contact me by email or on IRC.
I'll create your swatbot account soon.

--
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com


SWAT Rotation

Alexandre Belloni
 

Hello Christopher,

SWAT duty will rotate from me to you at the end of this Friday.

Please reply to let me know whether you will be able to work on this
task next week.

I'll be available to walk you through the swatbot process on Monday,
don't hesitate to contact me by email or on IRC.
I'll create your swatbot account soon.

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


SWAT statistics for week 16

Alexandre Belloni
 

Hello,

Here are the statistics for last week. Naveen was on SWAT duty.

168 failures were triaged:

* 168 by Naveen
- 73 for an issue in opkg-utils-native
- 22 for check-layer-nightly issues Richard was working on
- 13 for go-runtime source fetching issues
- 8 for an issue in epiphany
- 7 for issues already solved at the time of triage
- 7 for individual issues reported on the mailing list
- 6 for space issues on the tmpfs
- 5 for kernel config warnings
- 4 for a build started for ebugging purposes
- 4 occurrences of new bug 14353
- 1 occurrence of new bug 14352
- 1 occurrence of new bug 14359
- 1 occurrence of new bug 14360
- 8 new occurrences of bug 14128
- 3 new occurrences of bug 14251
- 2 new occurrences of bug 14177
- 1 new occurrence of bug 14002
- 1 new occurrence of both bug 14163 and 14244
- 1 new occurrence of bug 14353 and 14354

Regards,

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


Re: SWAT Rotation

Alexandre Belloni
 

To be clear that was the end of Friday 23rd. I see you got the email on
Saturday ;)

On 26/04/2021 01:02:44+0000, Saini, Naveen Kumar wrote:
Noted.

Thanks,
Naveen

-----Original Message-----
From: Alexandre Belloni <alexandre.belloni@bootlin.com>
Sent: Saturday, April 24, 2021 12:10 AM
To: Saini, Naveen Kumar <naveen.kumar.saini@intel.com>
Cc: swat@lists.yoctoproject.org; Stephen Jolley <sjolley.yp.pm@gmail.com>
Subject: SWAT Rotation

Hello Naveen,

SWAT duty will rotate from you to me at the end of this Friday, thanks for
your work!

--
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel
engineering https://bootlin.com
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


Re: SWAT Rotation

Naveen Saini
 

Noted.

Thanks,
Naveen

-----Original Message-----
From: Alexandre Belloni <alexandre.belloni@bootlin.com>
Sent: Saturday, April 24, 2021 12:10 AM
To: Saini, Naveen Kumar <naveen.kumar.saini@intel.com>
Cc: swat@lists.yoctoproject.org; Stephen Jolley <sjolley.yp.pm@gmail.com>
Subject: SWAT Rotation

Hello Naveen,

SWAT duty will rotate from you to me at the end of this Friday, thanks for
your work!

--
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel
engineering https://bootlin.com


SWAT Rotation

Alexandre Belloni
 

Hello Naveen,

SWAT duty will rotate from you to me at the end of this Friday, thanks
for your work!

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


Space issues on the autobuilder

Richard Purdie
 

Hi All,

I wanted to give a headsup that we're seeing out of space issues on the autobuilder.
The cause is the tmpfs change for qemu images we added recently. It appears 'someone'
forgot to remove the files from the tmpfs after testing completed. This is the cause
of the space errors on debian8 in particular. Was a bit puzzled for a while as it
had plenty of space!

I'll work out a patch to clean up the files after runqemu completes! Just wanted
to ensure people knew the reason and that it should get fixed soon. It has 
implications for backporting those patches to other releases.

Cheers,

Richard


SWAT statistics for week 15

Alexandre Belloni
 

Hello,

Here are the statistics for last week. Alejandro was still on SWAT duty.

415 failures were triaged:

* 221 by Alejandro
- 184 for an issue with a gnutls change that was already worked on
- 16 for the gcc upgrade to 10.3.0 that was then fixed by Richard
- 7 for the go upgrade patch, the fix is still being worked on
- 4 for cancelled builds without any failure
- 2 occurrences of new bug 14345
- 1 occurrence of new bug 14346
- 2 new occurrences of bug 13802
- 2 new occurrences of bug 14029
- 2 new occurrences of bug 14251
- 1 new occurrence of bug 14230

* 194 by Richard
- 180 for a patch he fixed
- 8 for an issue in a patch for prserv he reported on the list
- 6 for an issue with ab-helper changes he reported on the list

Regards,

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


Re: SWAT Rotation

Naveen Saini
 

Hello Alexandre,
I have received this email and will take over tomorrow morning.

Thanks,
Naveen

-----Original Message-----
From: swat@lists.yoctoproject.org <swat@lists.yoctoproject.org> On Behalf
Of Alexandre Belloni
Sent: Friday, April 16, 2021 5:54 AM
To: Saini, Naveen Kumar <naveen.kumar.saini@intel.com>; Alejandro
Hernandez Samaniego <alhe@linux.microsoft.com>
Cc: swat@lists.yoctoproject.org; Stephen Jolley <sjolley.yp.pm@gmail.com>
Subject: [swat] SWAT Rotation

Hello Naveen,

SWAT duty will rotate from Alejandro to you at the end of this Friday.

Please reply to let me know whether you will be able to work on this task.

I'll be available to walk you through the process on Monday, don't hesitate to
contact me by email or on IRC.

Thanks!

--
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel
engineering https://bootlin.com




SWAT Rotation

Alexandre Belloni
 

Hello Naveen,

SWAT duty will rotate from Alejandro to you at the end of this Friday.

Please reply to let me know whether you will be able to work on this
task.

I'll be available to walk you through the process on Monday, don't
hesitate to contact me by email or on IRC.

Thanks!

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


SWAT statistics for week 14

Alexandre Belloni
 

Hello,

Here are the statistics for last week. Alejandro was on SWAT duty.

10 failures were triaged:

* 6 by Alejandro
- 1 occurrence of new bug 14341
- 2 new occurrences of bug 14251
- 1 new occurrence of bug 14201
- 1 new occurrence of bug 14230
- 1 new occurrence of bug 14295

* 4 by Richard
- 4 for an experimental build

There was an issue with the autobuilder/swatbot reporter code that
prevented more failures to be logged, hence the low number.

Regards,

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


Issues with buildbot to swatbot connection

Richard Purdie
 

Hi All,

We've had some issues with the connection between swatbot and the autobuilder 
meaning that swatbot wasn't getting build entries or updates for some periods
over the last few weeks.

With some help from Michael we figured out the issue and I've updated the 
autobuilder notifier code so it that it reconnects properly if logins expire.

Unfortunately we don't yet have a way to replay lost builds data back to swatbot
to bring it's database back in sync.

Over the weekend I did test:

gcc 11 - issues were reported to Khem on the mailing list
gcc 10.3 - there were several failures from this and we're now testing revised patches

Cheers,

Richard


SWAT statistics for week 13

Alexandre Belloni
 

Hello,

Here are the statistics for last week. Paul was on SWAT duty.

21 failures were triaged:

* 18 by Paul
- 1 occurrence of new bug 14331
- 4 new occurrences of bug 14329
- 3 new occurrences of bug 14029
- 2 new occurrences of bug 13802
- 2 new occurrences of bug 14294
- 1 new occurrence of bug 14163
- 1 new occurrence of bug 14200
- 1 new occurrence of bug 14274
- 1 for a new occurence of 14177, 14294 and 14296
- 1 for the valgrind update he replied to on the list
- 1 failure because of a previous build failure

* 1 by Richard
- 1 for a build that was not for SWAT

* 2 by Michael
- 2 builds to test the infra

Regards,

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

101 - 120 of 220