Date   

Re: Creating a QEmu (x86-64) Image, which can execute binary applications from other x86-64 linux OSes

Christian Lohr <christian.lohr.ext@...>
 

Hello Khem,

Thanks for that advice, it seems to be the best way to handle this.

Best regards,

Christian Lohr

-----Ursprüngliche Nachricht-----
Von: yocto@... <yocto@...> Im Auftrag von Khem Raj
Gesendet: Mittwoch, 12. Februar 2020 19:12
An: Lohr, Christian [ext] <christian.lohr.ext@...>; Alexander Kanavin <alex.kanavin@...>
Cc: yocto@...
Betreff: Re: [yocto] Creating a QEmu (x86-64) Image, which can execute binary applications from other x86-64 linux OSes

On 2/12/20 3:19 AM, Christian Lohr wrote:
Ok, I created a symbolic link with "ln -s /lib /lib64" and it seemed
to work. Thanks a lot.
right, if you built multilib image then it will be using /lib64 automatically.

https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.yoctoproject.org%2Fdocs%2Fcurrent%2Fdev-manual%2Fdev-manual.html%23combining-multiple-versions-library-files-into-one-image&;data=02%7C01%7C%7Cafab24d3bd894869269008d7afe71947%7C28042244bb514cd680347776fa3703e8%7C1%7C0%7C637171279406982418&amp;sdata=k7AMcA2eXvLgqBbtaUQFNSFIFBVitJ8v9oQK5%2FYacRU%3D&amp;reserved=0


*Von:* yocto@... <yocto@...> *Im
Auftrag von *Alexander Kanavin
*Gesendet:* Mittwoch, 12. Februar 2020 12:00
*An:* Lohr, Christian [ext] <christian.lohr.ext@...>
*Cc:* yocto@...
*Betreff:* Re: [yocto] Creating a QEmu (x86-64) Image, which can
execute binary applications from other x86-64 linux OSes

But this is exactly what happens: the kernel reads the dynamic
loader/interpreter path from the binary (which is different than the
list of dynamically linked libraries printed by ldd), isn't able to
find it and stops right there.

Try like this:

/lib/ld-linux-x86-64.so.2 /usr/share/dotnet/dotnet

Alex

On Wed, 12 Feb 2020 at 11:45, Lohr, Christian [ext]
<christian.lohr.ext@... <mailto:christian.lohr.ext@...>> wrote:

I used the x86_64 variant from the layer (it only downloads the
binaries and copies them).

And I checked that with ldd, it seemed ok so far:

---------------------------------------------------------

root@qemux86-64:/usr/share/dotnet# ldd dotnet

            linux-vdso.so.1 (0x00007fffea543000)

            libpthread.so.0 => /lib/libpthread.so.0
(0x00007f9fde06c000)

            libdl.so.2 => /lib/libdl.so.2 (0x00007f9fde067000)

libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f9fddee5000)

libm.so.6 => /lib/libm.so.6 (0x00007f9fddda4000)

            libgcc_s.so.1 => /lib/libgcc_s.so.1
(0x00007f9fddd8a000)

            libc.so.6 => /lib/libc.so.6 (0x00007f9fddbd0000)

            /lib64/ld-linux-x86-64.so.2 =>
/lib/ld-linux-x86-64.so.2 (0x00007f9fde097000)

Strace didn't help either:

-------------------------------

root@qemux86-64:/# strace /usr/share/dotnet/dotnet

execve("/usr/share/dotnet/dotnet", ["/usr/share/dotnet/dotnet"],
0x7ffe22f0ab70 /* 22 vars */) = -1 ENOENT (No such file or
directory)

strace: exec: No such file or directory

+++ exited with 1 +++

It's strange that it denies that the binaries are there. Normally I
would have expected something like "wrong elf architecture" or
something about missing libraries. The only thing I think I could do
now, is to turn this "-enable-default-pie" off, but I'm not sure if
this helps, and I don't know where to turn it off. And what I'm also
trying is to go back to Yocto Rocko Release (for this experiment I
used Warrior Release)

*Von:* yocto@...
<mailto:yocto@...> <yocto@...
<mailto:yocto@...>> *Im Auftrag von *Alexander
Kanavin
*Gesendet:* Mittwoch, 12. Februar 2020 10:26
*An:* Lohr, Christian [ext] <christian.lohr.ext@...
<mailto:christian.lohr.ext@...>>
*Cc:* yocto@... <mailto:yocto@...>
*Betreff:* Re: [yocto] Creating a QEmu (x86-64) Image, which can
execute binary applications from other x86-64 linux OSes

That layer does have the x86_64 variant as well, no? Is it not working?

https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRDunkley%2Fmeta-dotnet-core%2Fblob%2Fmaster%2Frecipes-runtime%2Fdotnet-core%2Fdotnet-core_3.1.1.inc&;data=02%7C01%7C%7Cafab24d3bd894869269008d7afe71947%7C28042244bb514cd680347776fa3703e8%7C1%7C0%7C637171279406982418&amp;sdata=S0WxuzuKiJcCdXLuVKJjgCxcl0kup6TUNT1fDPyQNUc%3D&amp;reserved=0

<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
hub.com%2FRDunkley%2Fmeta-dotnet-core%2Fblob%2Fmaster%2Frecipes-runtim
e%2Fdotnet-core%2Fdotnet-core_3.1.1.inc&amp;data=02%7C01%7C%7Cafab24d3
bd894869269008d7afe71947%7C28042244bb514cd680347776fa3703e8%7C1%7C0%7C
637171279406982418&amp;sdata=S0WxuzuKiJcCdXLuVKJjgCxcl0kup6TUNT1fDPyQN
Uc%3D&amp;reserved=0>

The error you're seeing is almost certainly due to Yocto using
/lib/ld-so... for dynamic loader, and the binary hardcoding /lib64/....

Alex

On Wed, 12 Feb 2020 at 10:15, Lohr, Christian [ext]
<christian.lohr.ext@... <mailto:christian.lohr.ext@...>>
wrote:

Hello Alex,

Thanks for replying. Yes, I know that this isn't the way it
works on Yocto (and I told the managers it is a crappy idea to
do that more than once). But they need .NET Core in that company
I work for, and Mono doesn't work (that's what they told me).
Compiling .NET Core through the Yocto process is ugly, because
Microsoft used a mixture of shell scripts to compile it for some
platforms, it won't work this way on Yocto. Actually one already
tried it, but only until .NET Core 2.2:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FTragetaschen%2Fmeta-aspnet&;data=02%7C01%7C%7Cafab24d3bd894869269008d7afe71947%7C28042244bb514cd680347776fa3703e8%7C1%7C0%7C637171279406982418&amp;sdata=NxojWXOJbRlLijXNQ74h4%2FbPWgvmkvX47w%2FhyqjJUL0%3D&amp;reserved=0

<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
hub.com%2FTragetaschen%2Fmeta-aspnet&amp;data=02%7C01%7C%7Cafab24d3bd8
94869269008d7afe71947%7C28042244bb514cd680347776fa3703e8%7C1%7C0%7C637
171279406982418&amp;sdata=NxojWXOJbRlLijXNQ74h4%2FbPWgvmkvX47w%2FhyqjJ
UL0%3D&amp;reserved=0>

And despite this, I already managed to get the dotnet binaries
for ARM32 and ARM64 already working on a i.MX6 and i.MX8

There's a layer which just deploys the binaries:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRDunkley%2Fmeta-dotnet-core&;data=02%7C01%7C%7Cafab24d3bd894869269008d7afe71947%7C28042244bb514cd680347776fa3703e8%7C1%7C0%7C637171279406982418&amp;sdata=T00TXDmEFPBo4T66Quvzy4P%2Bkqki5BWaeWqehh%2B2gXY%3D&amp;reserved=0

<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
hub.com%2FRDunkley%2Fmeta-dotnet-core&amp;data=02%7C01%7C%7Cafab24d3bd
894869269008d7afe71947%7C28042244bb514cd680347776fa3703e8%7C1%7C0%7C63
7171279406982418&amp;sdata=T00TXDmEFPBo4T66Quvzy4P%2Bkqki5BWaeWqehh%2B
2gXY%3D&amp;reserved=0>

This is currently the last step. I thought if it worked on i.MX6
and i.MX8 it shouldn't be a problem to get it running on
Virtualbox with x86-64. It should only make the things easy for
the developers. It isn't even our target platform.

Best regards,

Christian Lohr

*Von:* yocto@...
<mailto:yocto@...>
<yocto@...
<mailto:yocto@...>> *Im Auftrag von
*Alexander Kanavin
*Gesendet:* Mittwoch, 12. Februar 2020 09:51
*An:* Lohr, Christian [ext] <christian.lohr.ext@...
<mailto:christian.lohr.ext@...>>
*Cc:* yocto@...
<mailto:yocto@...>
*Betreff:* Re: [yocto] Creating a QEmu (x86-64) Image, which can
execute binary applications from other x86-64 linux OSes

Yocto generally does not support this use case. The binary was
compiled in a different environment and expects things in
different places, and probably being different versions too. I
could point out the specific problem why the executable doesn't
even start, but it's really the wrong way to approach it. Is the
source code for it available?

Microsoft specifically lists which distributions are supported,
and there is nothing Yocto-based in it:

https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fdotnet%2Fcore%2Finstall%2Fdependencies%3Ftabs%3Dnetcore31%26pivots%3Dos-linux&;data=02%7C01%7C%7Cafab24d3bd894869269008d7afe71947%7C28042244bb514cd680347776fa3703e8%7C1%7C0%7C637171279406982418&amp;sdata=RAHH6UvIQpXPUyxI5OhlYv1UZyuInRwBQgptqc8g%2Bss%3D&amp;reserved=0

<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoc
s.microsoft.com%2Fen-us%2Fdotnet%2Fcore%2Finstall%2Fdependencies%3Ftab
s%3Dnetcore31%26pivots%3Dos-linux&amp;data=02%7C01%7C%7Cafab24d3bd8948
69269008d7afe71947%7C28042244bb514cd680347776fa3703e8%7C1%7C0%7C637171
279406982418&amp;sdata=RAHH6UvIQpXPUyxI5OhlYv1UZyuInRwBQgptqc8g%2Bss%3
D&amp;reserved=0>

For mono things you can use meta-mono layer, but I am not sure
if it provides exactly the item you're after.

Alex

On Wed, 12 Feb 2020 at 09:36, Christian Lohr
<christian.lohr.ext@...
<mailto:christian.lohr.ext@...>> wrote:

Hello,

I'm trying to create a normal QEmu (x86-64) Image, which I
can let run in Virtualbox. As a addition I deployed .NET
Core, which I got from this side:

https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdotnet.microsoft.com%2Fdownload%2Fdotnet-core%2Fthank-you%2Fruntime-aspnetcore-3.1.1-linux-x64-binaries&;data=02%7C01%7C%7Cafab24d3bd894869269008d7afe71947%7C28042244bb514cd680347776fa3703e8%7C1%7C0%7C637171279406992365&amp;sdata=hDN%2BmtyxOpjAzwD9oNPzqVcSJrYgnoZYV6e31yLtlsI%3D&amp;reserved=0

<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdot
net.microsoft.com%2Fdownload%2Fdotnet-core%2Fthank-you%2Fruntime-aspne
tcore-3.1.1-linux-x64-binaries&amp;data=02%7C01%7C%7Cafab24d3bd8948692
69008d7afe71947%7C28042244bb514cd680347776fa3703e8%7C1%7C0%7C637171279
406992365&amp;sdata=hDN%2BmtyxOpjAzwD9oNPzqVcSJrYgnoZYV6e31yLtlsI%3D&a
mp;reserved=0>

But I can't execute it:

----------------------------

root@qemux86-64:/usr/share/dotnet# ./dotnet

-sh: ./dotnet: No such file or directory

But it is there:

------------------

root@qemux86-64:/usr/share/dotnet# ls -lh

total 116K

-rw-r--r-- 1 root root 1.1K Feb 10 02:33 LICENSE.txt

-rw-r--r-- 1 root root  31K Feb 10 02:33
ThirdPartyNotices.txt

-rwxr-xr-x 1 root root  72K Feb 10 02:33 dotnet

drwxr-xr-x 3 root root 4.0K Feb 10 02:36 host

drwxr-xr-x 4 root root 4.0K Feb 10 02:36 shared

It tried to get more information about the
dotnet-executable


----------------------------------------------------------------------
--------

root@qemux86-64:/usr/share/dotnet# readelf -h dotnet

ELF Header:

  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00
00

  Class:                             ELF64

  Data:                              2's complement, little
endian

  Version:                           1 (current)

  OS/ABI:                            UNIX - System V

  ABI Version:                       0

  Type:                              EXEC (Executable
file)

  Machine:                           Advanced Micro Devices
X86-64

  Version:                           0x1

  Entry point address:               0x402f17

  Start of program headers:          64 (bytes into file)

  Start of section headers:          71032 (bytes into
file)

  Flags:                             0x0

  Size of this header:               64 (bytes)

  Size of program headers:           56 (bytes)

  Number of program headers:         10

  Size of section headers:           64 (bytes)

  Number of section headers:         31

  Section header string table index: 30

root@qemux86-64:/usr/share/dotnet#  file dotnet

dotnet: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2,
for GNU/Linux 2.6.32,
BuildID[sha1]=28c244c1953bcbee994709a4b849086ee7cf0c99,
stripped

I compared those values with that from Python, which does
run on this system


----------------------------------------------------------------------
---------------------------------

root@qemux86-64:/opt/jre-8/bin# readelf -h
/usr/bin/python3.7

ELF Header:

  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00
00

  Class:                             ELF64

  Data:                              2's complement, little
endian

  Version:                           1 (current)

  OS/ABI:                            UNIX - System V

  ABI Version:                       0

  Type:                              DYN (Shared object
file)

  Machine:                           Advanced Micro Devices
X86-64

  Version:                           0x1

  Entry point address:               0x1060

  Start of program headers:          64 (bytes into file)

  Start of section headers:          12568 (bytes into
file)

  Flags:                             0x0

  Size of this header:               64 (bytes)

  Size of program headers:           56 (bytes)

  Number of program headers:         11

  Size of section headers:           64 (bytes)

  Number of section headers:         27

  Section header string table index: 26

root@qemux86-64:/usr/share/dotnet# file /usr/bin/python3.7

/usr/bin/python3.7: ELF 64-bit LSB pie executable, x86-64,
version 1 (SYSV), dynamically linked, interpreter
/lib/ld-linux-x86-64.so.2,
BuildID[sha1]=a455873f278466378405802b0e0171337e52a81c, for
GNU/Linux 3.2.0, stripped


======================================================================
==========

The only difference I found, is that Python is a "ELF 64-bit
LSB pie executable" whereas dotnet is a "ELF 64-bit LSB
executable". I tried to turn that PIE (seemed to be a gcc
option: --enable-default-pie) feature of, but that didn't
work well, and I couldn't find a way to remove it.

----

Best regards,

Christian Lohr

Im Auftrag von:

Carl Zeiss Meditec AG
Göschwitzer Strasse 51-52

07745 Jena, Deutschland

christian.lohr.ext@...
<mailto:christian.lohr.ext@...>

Tel: +493641220206




Re: should all "IMAGE_INSTALL +=" become "IMAGE_INSTALL_append"?

Jeremy Puhlman
 

You are referring to section 5.2.1 if I am not correct, not 3.2.1?

The context is important here. 5.2.1 is discussing using the variable as part of your local.conf. Which is
a different animal then adding it to an image recipe.  In local.conf you really need _append to for it to
even take effect, where its more correct and natural in my opinion to use += in bb since it provides
more flexibility to alter the recipe with out using too much magic.


On 2/13/2020 9:35 AM, rpjday@... wrote:
  again, some nitpickiness, but in the current YP dev manual, section
3.2.1, the manual is quite explicit about using "_append" with
IMAGE_INSTALL:

"Furthermore, you must use _append instead of the += operator if you
want to avoid ordering issues."

  the way that's worded, that's not just a recommendation, it's a
pretty clear directive to the reader, which flies in the face of an
admittedly small number of examples of that in current OE:

$ grep -r "IMAGE_INSTALL +=" *
meta/recipes-sato/images/core-image-sato-ptest-fast.bb:IMAGE_INSTALL += "${PTESTS_FAST}"
meta/recipes-sato/images/core-image-sato-sdk-ptest.bb:IMAGE_INSTALL += "${PTESTS_FAST} ${PTESTS_SLOW}"
meta/recipes-sato/images/core-image-sato-sdk.bb:IMAGE_INSTALL += "kernel-devsrc"
meta/recipes-extended/images/core-image-testmaster.bb:IMAGE_INSTALL += "\
meta/recipes-rt/images/core-image-rt-sdk.bb:IMAGE_INSTALL += "rt-tests hwlatdetect kernel-dev"
meta/recipes-rt/images/core-image-rt.bb:IMAGE_INSTALL += "rt-tests hwlatdetect"
meta/recipes-core/images/core-image-minimal-mtdutils.bb:IMAGE_INSTALL += "mtd-utils"
meta-selftest/recipes-test/container-image/container-test-image.bb:IMAGE_INSTALL += "container-image-testpkg"
meta-skeleton/recipes-multilib/images/core-image-multilib-example.bb:IMAGE_INSTALL += "lib32-bash"
$

  i'm fine leaving things as is, but those few examples clearly fly in
the face of the instruction given in the dev manual.

  thoughts?

rday




    

-- 
Jeremy A. Puhlman
jpuhlman@...


should all "IMAGE_INSTALL +=" become "IMAGE_INSTALL_append"?

Robert P. J. Day
 

again, some nitpickiness, but in the current YP dev manual, section
3.2.1, the manual is quite explicit about using "_append" with
IMAGE_INSTALL:

"Furthermore, you must use _append instead of the += operator if you
want to avoid ordering issues."

the way that's worded, that's not just a recommendation, it's a
pretty clear directive to the reader, which flies in the face of an
admittedly small number of examples of that in current OE:

$ grep -r "IMAGE_INSTALL +=" *
meta/recipes-sato/images/core-image-sato-ptest-fast.bb:IMAGE_INSTALL += "${PTESTS_FAST}"
meta/recipes-sato/images/core-image-sato-sdk-ptest.bb:IMAGE_INSTALL += "${PTESTS_FAST} ${PTESTS_SLOW}"
meta/recipes-sato/images/core-image-sato-sdk.bb:IMAGE_INSTALL += "kernel-devsrc"
meta/recipes-extended/images/core-image-testmaster.bb:IMAGE_INSTALL += "\
meta/recipes-rt/images/core-image-rt-sdk.bb:IMAGE_INSTALL += "rt-tests hwlatdetect kernel-dev"
meta/recipes-rt/images/core-image-rt.bb:IMAGE_INSTALL += "rt-tests hwlatdetect"
meta/recipes-core/images/core-image-minimal-mtdutils.bb:IMAGE_INSTALL += "mtd-utils"
meta-selftest/recipes-test/container-image/container-test-image.bb:IMAGE_INSTALL += "container-image-testpkg"
meta-skeleton/recipes-multilib/images/core-image-multilib-example.bb:IMAGE_INSTALL += "lib32-bash"
$

i'm fine leaving things as is, but those few examples clearly fly in
the face of the instruction given in the dev manual.

thoughts?

rday


[PATCH] dev-manual: layer priority takes precedence over PV

Robert P. J. Day
 

Clarify that if there are multiple recipes with the same name, the
recipe with the highest PV from the highest priority layer will be the
one selected.

Signed-off-by: Robert P. J. Day <rpjday@...>

---

i'm using docbook's <emphasis> markup here, which renders in HTML as
both bold and italic.

diff --git a/documentation/dev-manual/dev-manual-common-tasks.xml b/documentation/dev-manual/dev-manual-common-tasks.xml
index 1e6addde4..951263291 100644
--- a/documentation/dev-manual/dev-manual-common-tasks.xml
+++ b/documentation/dev-manual/dev-manual-common-tasks.xml
@@ -825,9 +825,11 @@
</para>

<note>
- <para>It is possible for a recipe with a lower version number
+ <para>If the same recipe exists in several layers, the recipe
+ with the highest value of
<ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>
- in a layer that has a higher priority to take precedence.</para>
+ <emphasis>from the layer with the
+ highest priority</emphasis> will be selected.</para>
<para>Also, the layer priority does not currently affect the
precedence order of <filename>.conf</filename>
or <filename>.bbclass</filename> files.

--

========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================


Re: Incrementally building constantly changing application source in a workflow-friendly way with #yocto

Alexander Kanavin
 

You can specify a directory relative to the one the recipe is in in SRC_URI. Bitbake will checksum all files in it (so will trigger a re-build if they change), and will copy the directory into the $WORKDIR. I have never actually tried this, but I'd start there, maybe with a hello world type project for proof of concept.

Alex

On Thu, 13 Feb 2020 at 17:29, Pieter Smith <smipi01@...> wrote:
I am defining a meta-layer with our business logic (applications), one or two recipes and an image definition. I would like to maintain the C/C++ sources with this meta-layer in a single repository, because they are tightly coupled from a lifecycle perspective.

Is there a way to always build the current state of the source-code from a path relative to the recipe.bb file in a workflow friendly way? After scouring the web and mailing lists for days, I have no satisfactory answer. With OpenWRT, Buildroot 2 and ptxdist this was extremely simple to achieve.

My requirements are:
  1. Treat the meta-layer, it's recipes and images as well as package source-code as a monolithic whole with a single lifecycle within a single git repository,
  2. Have the current checked out state of git repository built whenever I bake an image,
  3. Not have things fall apart when I check-out in another location.
I do not mind having to do a commit before each build so that the state can be tracked, but having to push and edit the recipe with the new commit ID so every change can be built deployed and tested is driving me nuts, especially since some changes are cross application, requiring an atomic update of both applications.


Incrementally building constantly changing application source in a workflow-friendly way with #yocto

Pieter Smith
 

I am defining a meta-layer with our business logic (applications), one or two recipes and an image definition. I would like to maintain the C/C++ sources with this meta-layer in a single repository, because they are tightly coupled from a lifecycle perspective.

Is there a way to always build the current state of the source-code from a path relative to the recipe.bb file in a workflow friendly way? After scouring the web and mailing lists for days, I have no satisfactory answer. With OpenWRT, Buildroot 2 and ptxdist this was extremely simple to achieve.

My requirements are:
  1. Treat the meta-layer, it's recipes and images as well as package source-code as a monolithic whole with a single lifecycle within a single git repository,
  2. Have the current checked out state of git repository built whenever I bake an image,
  3. Not have things fall apart when I check-out in another location.
I do not mind having to do a commit before each build so that the state can be tracked, but having to push and edit the recipe with the new commit ID so every change can be built deployed and tested is driving me nuts, especially since some changes are cross application, requiring an atomic update of both applications.


Re: dev-manual: differing versions of same recipe in different priority layers?

Robert P. J. Day
 

On Thu, 13 Feb 2020, Leon Woestenberg wrote:

Hello Robert,

On Thu, Feb 13, 2020 at 3:14 PM rpjday@...
<rpjday@...> wrote:
in the YP dev manual, section 3.1.6, there is a maddeningly vague
note:

"It is possible for a recipe with a lower version number PV in a layer
that has a higher priority to take precedence."

so does it take precedence or not? just saying it's "possible"
doesn't clarify it for the reader.
Yes, this should be re-worded. This is actually a FAQ.
(I remember more sentences like this exist, that especially for
non-English speakers might be difficult to interpret.)
i'm betting i'm going to find all of them eventually. :-)

My proposal would be:

If a recipe exists in multiple layers, the recipe with highest PV
*from the layer with the highest priority* is selected. Be aware
that recipes with the same PN in lower priority layers are thus not
selected, *even if their PV is higher*.
right ... higher layer priority takes precedence over higher version
number. i'll submit something for that.

rday


Re: dev-manual: differing versions of same recipe in different priority layers?

Leon Woestenberg
 

Hello Robert,

On Thu, Feb 13, 2020 at 3:14 PM rpjday@...
<rpjday@...> wrote:
in the YP dev manual, section 3.1.6, there is a maddeningly vague
note:

"It is possible for a recipe with a lower version number PV in a layer
that has a higher priority to take precedence."

so does it take precedence or not? just saying it's "possible"
doesn't clarify it for the reader.
Yes, this should be re-worded. This is actually a FAQ.
(I remember more sentences like this exist, that especially for
non-English speakers might be difficult to interpret.)

My proposal would be:

If a recipe exists in multiple layers, the recipe with highest PV
*from the layer with the highest priority* is selected.
Be aware that recipes with the same PN in lower priority layers are
thus not selected, *even if their PV is higher*.

Regards,

Leon.


Re: [PATCH] overview-manual: minor editing, rendering, wording changes

Armin Kuster
 

Robert,


On 2/13/20 1:04 AM, rpjday@... wrote:
Signed-off-by: Robert P. J. Day <rpjday@...>

Please send doc changes to the docs@...

- armin

---

diff --git a/documentation/overview-manual/overview-manual-concepts.xml b/documentation/overview-manual/overview-manual-concepts.xml
index f085dd710..6ab783ce7 100644
--- a/documentation/overview-manual/overview-manual-concepts.xml
+++ b/documentation/overview-manual/overview-manual-concepts.xml
@@ -144,11 +144,11 @@

             <para>
                 Files that have the <filename>.bb</filename> suffix are
-                "recipes" files.
+                "recipe" files.
                 In general, a recipe contains information about a single piece
-                of software.
-                This information includes the location from which to download
-                the unaltered source, any source patches to be applied to that
+                of software,
+                including the location from which to download
+                the pristine source, any (local) source patches to be applied to that
                 source (if needed), which special configuration options to
                 apply, how to compile the source files, and how to package the
                 compiled output.
@@ -189,9 +189,9 @@
                 various configuration variables that govern the OpenEmbedded
                 build process.
                 These files fall into several areas that define machine
-                configuration options, distribution configuration options,
-                compiler tuning options, general common configuration options,
-                and user configuration options in
+                configuration, distribution configuration,
+                compiler tuning, general common configuration,
+                and user configuration in
                 <filename>conf/local.conf</filename>, which is found in the
                 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
             </para>
@@ -336,7 +336,7 @@
                 complete a build.
                 These files are <filename>*.conf</filename> files.
                 The minimally necessary ones reside as example files in the
-                <filename>build/conf</filename> directory of the
+                <filename>meta/conf</filename> directory of the
                 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>.
                 For simplicity, this section refers to the Source Directory as
                 the "Poky Directory."
@@ -555,7 +555,7 @@
                 reads the configuration files in a specific order:
                 <filename>site.conf</filename>, <filename>auto.conf</filename>,
                 and <filename>local.conf</filename>.
-                And, the build system applies the normal assignment statement
+                In addition, the build system applies the normal assignment statement
                 rules as described in the
                 "<ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual-metadata'>Syntax and Operators</ulink>"
                 chapter of the BitBake User Manual.
@@ -584,7 +584,7 @@
             </para>

             <para>
-                In general, three types of layer input exists.
+                In general, three types of layer input exist.
                 You can see them below the "User Configuration" box in the
                 <link linkend='general-workflow-figure'>general workflow figure</link>:
                 <itemizedlist>
@@ -613,6 +613,9 @@
                         is the
                         <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-yocto-bsp'><filename>meta-yocto-bsp</filename></ulink>
                         layer.
+                        Individual hardware manufacturers, such as Intel,
+                        Xilinx, Altera, NXP and so on will typically provide
+                        BSP layers for their own products as well.
                         </para></listitem>
                     <listitem><para>
                         <emphasis>Policy Configuration:</emphasis>
@@ -620,14 +623,14 @@
                         following figure) providing top-level or general
                         policies for the images or SDKs being built for a
                         particular distribution.
-                        For example, in the Poky Reference Distribution the
+                        For example, in the Poky Reference Distribution, the
                         distro layer is the
                         <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky'><filename>meta-poky</filename></ulink>
                         layer.
                         Within the distro layer is a
                         <filename>conf/distro</filename> directory that
                         contains distro configuration files (e.g.
-                        <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky/conf/distro/poky.conf'><filename>poky.conf</filename></ulink>
+                        <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky/conf/distro/poky.conf'><filename>poky.conf</filename></ulink>)
                         that contain many policy configurations for the
                         Poky distribution.
                         </para></listitem>
@@ -748,7 +751,7 @@
                 <title>BSP Layer</title>

                 <para>
-                    The BSP Layer provides machine configurations that
+                    The BSP layer provides machine configurations that
                     target specific hardware.
                     Everything in this layer is specific to the machine for
                     which you are building the image or the SDK.
@@ -763,8 +766,8 @@
                 </para>

                 <para>
-                    The BSP Layer's configuration directory contains
-                    configuration files for the machine
+                    A BSP layer's configuration directory contains
+                    configuration files for the corresponding machines
                     (<filename>conf/machine/<replaceable>machine</replaceable>.conf</filename>)
                     and, of course, the layer
                     (<filename>conf/layer.conf</filename>).
@@ -798,8 +801,8 @@
                 </para>

                 <para>
-                    This layer contains any recipes, append files, and
-                    patches, that your project needs.
+                    This layer contains any recipes, append files and
+                    patches that your project needs.
                 </para>
             </section>
         </section>
@@ -913,7 +916,7 @@
                     project is to use the
                     <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink>
                     class to include that local project.
-                    You use either the <filename>local.conf</filename> or a
+                    You use either the <filename>local.conf</filename> file or a
                     recipe's append file to override or set the
                     recipe to point to the local directory on your disk to pull
                     in the whole source tree.
@@ -1739,7 +1742,7 @@
                 <para>
                     The <filename>do_populate_sdk_ext</filename> task helps
                     create the extensible SDK and handles host and target parts
-                    differently than its counter part does for the standard SDK.
+                    differently than its counterpart does for the standard SDK.
                     For the extensible SDK, the task encapsulates the build
                     system, which includes everything needed (host and target)
                     for the SDK.
@@ -2449,7 +2452,7 @@
             parts are built fresh and no possibility of stale data exists that
             can cause problems.
             When developers hit problems, they typically default back to
-            building from scratch so they have a know state from the
+            building from scratch so they have a known state from the
             start.
         </para>

@@ -2665,9 +2668,9 @@
                 Thus far, this section has limited discussion to the direct
                 inputs into a task.
                 Information based on direct inputs is referred to as the
-                "basehash" in the code.
+                <firstterm>basehash</firstterm> in the code.
                 However, the question of a task's indirect inputs still
-                exits - items already built and present in the
+                exists - items already built and present in the
                 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
                 The checksum (or signature) for a particular task needs to add
                 the hashes of all the tasks on which the particular task

rday



    


dev-manual: differing versions of same recipe in different priority layers?

Robert P. J. Day
 

in the YP dev manual, section 3.1.6, there is a maddeningly vague
note:

"It is possible for a recipe with a lower version number PV in a layer
that has a higher priority to take precedence."

so does it take precedence or not? just saying it's "possible"
doesn't clarify it for the reader.

rday


[PATCH] dev-manual: chs 1,2; rewording, cleanup

Robert P. J. Day
 

Also removal of way too much unnecessary capitalization.

Signed-off-by: Robert P. J. Day <rpjday@...>

---

diff --git a/documentation/dev-manual/dev-manual-intro.xml b/documentation/dev-manual/dev-manual-intro.xml
index 3a34094b8..5e231c6ac 100644
--- a/documentation/dev-manual/dev-manual-intro.xml
+++ b/documentation/dev-manual/dev-manual-intro.xml
@@ -22,22 +22,22 @@
This manual provides the following:
<itemizedlist>
<listitem><para>
- Procedures that help you get going with the Yocto Project.
- For example, procedures that show you how to set up
+ Procedures that help you get going with the Yocto Project;
+ for example, procedures that show you how to set up
a build host and work with the Yocto Project
source repositories.
</para></listitem>
<listitem><para>
Procedures that show you how to submit changes to the
- Yocto Project.
- Changes can be improvements, new features, or bug
+ Yocto Project;
+ such changes can be improvements, new features, or bug
fixes.
</para></listitem>
<listitem><para>
Procedures related to "everyday" tasks you perform while
developing images and applications using the Yocto
- Project.
- For example, procedures to create a layer, customize an
+ Project;
+ for example, procedures to create a layer, customize an
image, write a new recipe, and so forth.
</para></listitem>
</itemizedlist>
@@ -47,26 +47,26 @@
This manual does not provide the following:
<itemizedlist>
<listitem><para>
- Redundant Step-by-step Instructions:
- For example, the
+ Redundant step-by-step instructions:
+ The
<ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink>
- manual contains detailed instructions on how to install an
+ manual already contains detailed instructions on how to install an
SDK, which is used to develop applications for target
hardware.
</para></listitem>
<listitem><para>
- Reference or Conceptual Material:
+ Reference or conceptual material:
This type of material resides in an appropriate reference
manual.
For example, system variables are documented in the
<ulink url='&YOCTO_DOCS_REF_URL;'>Yocto Project Reference Manual</ulink>.
</para></listitem>
<listitem><para>
- Detailed Public Information Not Specific to the
+ Detailed public information not specific to the
Yocto Project:
- For example, exhaustive information on how to use the
+ As an example, exhaustive information on how to use the
Source Control Manager Git is better covered with Internet
- searches and official Git Documentation than through the
+ searches and official Git documentation than through the
Yocto Project documentation.
</para></listitem>
</itemizedlist>
diff --git a/documentation/dev-manual/dev-manual-start.xml b/documentation/dev-manual/dev-manual-start.xml
index 59ffa49bb..340cefd37 100644
--- a/documentation/dev-manual/dev-manual-start.xml
+++ b/documentation/dev-manual/dev-manual-start.xml
@@ -24,8 +24,8 @@
Project in a team development environment, or how to scale it for a
large team of developers.
You can adapt the Yocto Project to many different use cases and
- scenarios.
- However, this flexibility could cause difficulties if you are trying
+ scenarios;
+ however, this flexibility could cause difficulties if you are trying
to create a working setup that scales across a large team.
</para>

@@ -117,7 +117,7 @@
Of the SCMs BitBake supports, the Yocto Project team strongly
recommends using
<ulink url='&YOCTO_DOCS_OM_URL;#git'>Git</ulink>.
- Git is a distributed system that is easy to backup,
+ Git is a distributed version control system that is easy to backup,
allows you to work remotely, and then connects back to the
infrastructure.
<note>
@@ -168,8 +168,8 @@
<itemizedlist>
<listitem><para>
Use a pre-built toolchain that contains the software
- stack itself.
- Then, develop the application code on top of the
+ stack itself,
+ and develop the application code on top of the
stack.
This method works well for small numbers of relatively
isolated applications.
@@ -302,7 +302,7 @@
<para>As with any development environment, it is important
to document the policy used as well as any main project
guidelines so they are understood by everyone.
- It is also a good idea to have well structured
+ It is also a good idea to have well-structured
commit messages, which are usually a part of a project's
guidelines.
Good commit messages are essential when looking back in time and

--

========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================


[psplash][PATCH 1/1] psplash-fb: Avoid racing issues on reading fb0

Andrei Gherzan
 

When starting psplash as early as possible in the boot process, the fb
device node might not be ready. This patch adds a loop on reading the
fb0 device with a timeout of 5 seconds.

Signed-off-by: Andrei Gherzan <andrei@...>
---
psplash-fb.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/psplash-fb.c b/psplash-fb.c
index 6603572..6700a3b 100644
--- a/psplash-fb.c
+++ b/psplash-fb.c
@@ -137,6 +137,7 @@ psplash_fb_new (int angle, int fbdev_id)
struct fb_fix_screeninfo fb_fix;
int off;
char fbdev[9] = "/dev/fb0";
+ int retries = 0;

PSplashFB *fb = NULL;

@@ -156,7 +157,9 @@ psplash_fb_new (int angle, int fbdev_id)

fb->fd = -1;

- if ((fb->fd = open (fbdev, O_RDWR)) < 0)
+ while ((fb->fd = open(fbdev, O_RDWR)) < 0 && retries++ <= 100)
+ usleep(50000);
+ if (fb->fd < 0)
{
fprintf(stderr,
"Error opening %s\n",
--
2.17.1


[PATCH] overview-manual: minor editing, rendering, wording changes

Robert P. J. Day
 

Signed-off-by: Robert P. J. Day <rpjday@...>

---

diff --git a/documentation/overview-manual/overview-manual-concepts.xml b/documentation/overview-manual/overview-manual-concepts.xml
index f085dd710..6ab783ce7 100644
--- a/documentation/overview-manual/overview-manual-concepts.xml
+++ b/documentation/overview-manual/overview-manual-concepts.xml
@@ -144,11 +144,11 @@

<para>
Files that have the <filename>.bb</filename> suffix are
- "recipes" files.
+ "recipe" files.
In general, a recipe contains information about a single piece
- of software.
- This information includes the location from which to download
- the unaltered source, any source patches to be applied to that
+ of software,
+ including the location from which to download
+ the pristine source, any (local) source patches to be applied to that
source (if needed), which special configuration options to
apply, how to compile the source files, and how to package the
compiled output.
@@ -189,9 +189,9 @@
various configuration variables that govern the OpenEmbedded
build process.
These files fall into several areas that define machine
- configuration options, distribution configuration options,
- compiler tuning options, general common configuration options,
- and user configuration options in
+ configuration, distribution configuration,
+ compiler tuning, general common configuration,
+ and user configuration in
<filename>conf/local.conf</filename>, which is found in the
<ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
</para>
@@ -336,7 +336,7 @@
complete a build.
These files are <filename>*.conf</filename> files.
The minimally necessary ones reside as example files in the
- <filename>build/conf</filename> directory of the
+ <filename>meta/conf</filename> directory of the
<ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>.
For simplicity, this section refers to the Source Directory as
the "Poky Directory."
@@ -555,7 +555,7 @@
reads the configuration files in a specific order:
<filename>site.conf</filename>, <filename>auto.conf</filename>,
and <filename>local.conf</filename>.
- And, the build system applies the normal assignment statement
+ In addition, the build system applies the normal assignment statement
rules as described in the
"<ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual-metadata'>Syntax and Operators</ulink>"
chapter of the BitBake User Manual.
@@ -584,7 +584,7 @@
</para>

<para>
- In general, three types of layer input exists.
+ In general, three types of layer input exist.
You can see them below the "User Configuration" box in the
<link linkend='general-workflow-figure'>general workflow figure</link>:
<itemizedlist>
@@ -613,6 +613,9 @@
is the
<ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-yocto-bsp'><filename>meta-yocto-bsp</filename></ulink>
layer.
+ Individual hardware manufacturers, such as Intel,
+ Xilinx, Altera, NXP and so on will typically provide
+ BSP layers for their own products as well.
</para></listitem>
<listitem><para>
<emphasis>Policy Configuration:</emphasis>
@@ -620,14 +623,14 @@
following figure) providing top-level or general
policies for the images or SDKs being built for a
particular distribution.
- For example, in the Poky Reference Distribution the
+ For example, in the Poky Reference Distribution, the
distro layer is the
<ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky'><filename>meta-poky</filename></ulink>
layer.
Within the distro layer is a
<filename>conf/distro</filename> directory that
contains distro configuration files (e.g.
- <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky/conf/distro/poky.conf'><filename>poky.conf</filename></ulink>
+ <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky/conf/distro/poky.conf'><filename>poky.conf</filename></ulink>)
that contain many policy configurations for the
Poky distribution.
</para></listitem>
@@ -748,7 +751,7 @@
<title>BSP Layer</title>

<para>
- The BSP Layer provides machine configurations that
+ The BSP layer provides machine configurations that
target specific hardware.
Everything in this layer is specific to the machine for
which you are building the image or the SDK.
@@ -763,8 +766,8 @@
</para>

<para>
- The BSP Layer's configuration directory contains
- configuration files for the machine
+ A BSP layer's configuration directory contains
+ configuration files for the corresponding machines
(<filename>conf/machine/<replaceable>machine</replaceable>.conf</filename>)
and, of course, the layer
(<filename>conf/layer.conf</filename>).
@@ -798,8 +801,8 @@
</para>

<para>
- This layer contains any recipes, append files, and
- patches, that your project needs.
+ This layer contains any recipes, append files and
+ patches that your project needs.
</para>
</section>
</section>
@@ -913,7 +916,7 @@
project is to use the
<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink>
class to include that local project.
- You use either the <filename>local.conf</filename> or a
+ You use either the <filename>local.conf</filename> file or a
recipe's append file to override or set the
recipe to point to the local directory on your disk to pull
in the whole source tree.
@@ -1739,7 +1742,7 @@
<para>
The <filename>do_populate_sdk_ext</filename> task helps
create the extensible SDK and handles host and target parts
- differently than its counter part does for the standard SDK.
+ differently than its counterpart does for the standard SDK.
For the extensible SDK, the task encapsulates the build
system, which includes everything needed (host and target)
for the SDK.
@@ -2449,7 +2452,7 @@
parts are built fresh and no possibility of stale data exists that
can cause problems.
When developers hit problems, they typically default back to
- building from scratch so they have a know state from the
+ building from scratch so they have a known state from the
start.
</para>

@@ -2665,9 +2668,9 @@
Thus far, this section has limited discussion to the direct
inputs into a task.
Information based on direct inputs is referred to as the
- "basehash" in the code.
+ <firstterm>basehash</firstterm> in the code.
However, the question of a task's indirect inputs still
- exits - items already built and present in the
+ exists - items already built and present in the
<ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
The checksum (or signature) for a particular task needs to add
the hashes of all the tasks on which the particular task

rday

--

========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================


[meta-security][PATCH] python3-fail2ban: add 2-3 conversion changes

Armin Kuster
 

Had to use the fail2ban-2.3 program to create py3 code
Add it as a patch

Signed-off-by: Armin Kuster <akuster808@...>
---
...0001-python3-fail2ban-2-3-conversion.patch | 2527 +++++++++++++++++
.../fail2ban/files/fail2ban_setup.py | 1 -
.../fail2ban/python3-fail2ban_0.10.4.0.bb | 4 +-
3 files changed, 2530 insertions(+), 2 deletions(-)
create mode 100644 recipes-security/fail2ban/files/0001-python3-fail2ban-2-3-conversion.patch

diff --git a/recipes-security/fail2ban/files/0001-python3-fail2ban-2-3-conversion.patch b/recipes-security/fail2ban/files/0001-python3-fail2ban-2-3-conversion.patch
new file mode 100644
index 0000000..ee872ec
--- /dev/null
+++ b/recipes-security/fail2ban/files/0001-python3-fail2ban-2-3-conversion.patch
@@ -0,0 +1,2527 @@
+From abaa20435bac7decffa69e6f965aac9ce29aff6a Mon Sep 17 00:00:00 2001
+From: Armin Kuster <akuster808@...>
+Date: Wed, 12 Feb 2020 17:19:15 +0000
+Subject: [PATCH] python3-fail2ban: 2-3 conversion
+
+Upstream-Status: OE specific.
+
+fail2ban handles py3 via a 2-3 conversion utility.
+
+Signed-off-by: Armin Kuster <akuster808@...>
+---
+ fail2ban/client/actionreader.py | 4 +-
+ fail2ban/client/configparserinc.py | 10 +-
+ fail2ban/client/configreader.py | 4 +-
+ fail2ban/client/csocket.py | 4 +-
+ fail2ban/client/fail2banclient.py | 4 +-
+ fail2ban/client/fail2banregex.py | 20 +-
+ fail2ban/client/filterreader.py | 2 +-
+ fail2ban/client/jailreader.py | 4 +-
+ fail2ban/helpers.py | 15 +-
+ fail2ban/server/action.py | 19 +-
+ fail2ban/server/actions.py | 24 +-
+ fail2ban/server/asyncserver.py | 4 +-
+ fail2ban/server/banmanager.py | 18 +-
+ fail2ban/server/database.py | 6 +-
+ fail2ban/server/failmanager.py | 8 +-
+ fail2ban/server/failregex.py | 9 +-
+ fail2ban/server/filter.py | 12 +-
+ fail2ban/server/filterpoll.py | 2 +-
+ fail2ban/server/filterpyinotify.py | 6 +-
+ fail2ban/server/ipdns.py | 16 +-
+ fail2ban/server/jail.py | 14 +-
+ fail2ban/server/mytime.py | 2 +-
+ fail2ban/server/server.py | 18 +-
+ fail2ban/server/strptime.py | 6 +-
+ fail2ban/server/ticket.py | 14 +-
+ fail2ban/server/transmitter.py | 2 +-
+ fail2ban/server/utils.py | 6 +-
+ fail2ban/tests/action_d/test_badips.py | 2 +-
+ fail2ban/tests/actiontestcase.py | 4 +-
+ fail2ban/tests/clientreadertestcase.py | 4 +-
+ fail2ban/tests/databasetestcase.py | 16 +-
+ fail2ban/tests/datedetectortestcase.py | 6 +-
+ fail2ban/tests/fail2banclienttestcase.py | 8 +-
+ fail2ban/tests/failmanagertestcase.py | 10 +-
+ .../tests/files/config/apache-auth/digest.py | 20 +-
+ fail2ban/tests/filtertestcase.py | 92 ++---
+ fail2ban/tests/misctestcase.py | 22 +-
+ fail2ban/tests/observertestcase.py | 34 +-
+ fail2ban/tests/samplestestcase.py | 8 +-
+ fail2ban/tests/servertestcase.py | 28 +-
+ fail2ban/tests/sockettestcase.py | 2 +-
+ fail2ban/tests/utils.py | 22 +-
+ setup.py | 326 ------------------
+ 43 files changed, 264 insertions(+), 593 deletions(-)
+ delete mode 100755 setup.py
+
+diff --git a/fail2ban/client/actionreader.py b/fail2ban/client/actionreader.py
+index 80617a50..ecf323c5 100644
+--- a/fail2ban/client/actionreader.py
++++ b/fail2ban/client/actionreader.py
+@@ -90,11 +90,11 @@ class ActionReader(DefinitionInitConfigReader):
+ stream = list()
+ stream.append(head + ["addaction", self._name])
+ multi = []
+- for opt, optval in opts.iteritems():
++ for opt, optval in opts.items():
+ if opt in self._configOpts and not opt.startswith('known/'):
+ multi.append([opt, optval])
+ if self._initOpts:
+- for opt, optval in self._initOpts.iteritems():
++ for opt, optval in self._initOpts.items():
+ if opt not in self._configOpts and not opt.startswith('known/'):
+ multi.append([opt, optval])
+ if len(multi) > 1:
+diff --git a/fail2ban/client/configparserinc.py b/fail2ban/client/configparserinc.py
+index e0f39579..45c77437 100644
+--- a/fail2ban/client/configparserinc.py
++++ b/fail2ban/client/configparserinc.py
+@@ -62,7 +62,7 @@ if sys.version_info >= (3,2):
+ parser, option, accum, rest, section, map, *args, **kwargs)
+
+ else: # pragma: no cover
+- from ConfigParser import SafeConfigParser, \
++ from configparser import SafeConfigParser, \
+ InterpolationMissingOptionError, NoOptionError, NoSectionError
+
+ # Interpolate missing known/option as option from default section
+@@ -327,7 +327,7 @@ after = 1.conf
+ # mix it with defaults:
+ return set(opts.keys()) | set(self._defaults)
+ # only own option names:
+- return opts.keys()
++ return list(opts.keys())
+
+ def read(self, filenames, get_includes=True):
+ if not isinstance(filenames, list):
+@@ -356,7 +356,7 @@ after = 1.conf
+ ret += i
+ # merge defaults and all sections to self:
+ alld.update(cfg.get_defaults())
+- for n, s in cfg.get_sections().iteritems():
++ for n, s in cfg.get_sections().items():
+ # conditional sections
+ cond = SafeConfigParserWithIncludes.CONDITIONAL_RE.match(n)
+ if cond:
+@@ -366,7 +366,7 @@ after = 1.conf
+ del(s['__name__'])
+ except KeyError:
+ pass
+- for k in s.keys():
++ for k in list(s.keys()):
+ v = s.pop(k)
+ s[k + cond] = v
+ s2 = alls.get(n)
+@@ -399,7 +399,7 @@ after = 1.conf
+ sec.update(options)
+ return
+ sk = {}
+- for k, v in options.iteritems():
++ for k, v in options.items():
+ if not k.startswith(pref) and k != '__name__':
+ sk[pref+k] = v
+ sec.update(sk)
+diff --git a/fail2ban/client/configreader.py b/fail2ban/client/configreader.py
+index 20709b72..b5167409 100644
+--- a/fail2ban/client/configreader.py
++++ b/fail2ban/client/configreader.py
+@@ -26,7 +26,7 @@ __license__ = "GPL"
+
+ import glob
+ import os
+-from ConfigParser import NoOptionError, NoSectionError
++from configparser import NoOptionError, NoSectionError
+
+ from .configparserinc import sys, SafeConfigParserWithIncludes, logLevel
+ from ..helpers import getLogger, _as_bool, _merge_dicts, substituteRecursiveTags
+@@ -197,7 +197,7 @@ class ConfigReaderUnshared(SafeConfigParserWithIncludes):
+ config_files += sorted(glob.glob('%s/*.local' % config_dir))
+
+ # choose only existing ones
+- config_files = filter(os.path.exists, config_files)
++ config_files = list(filter(os.path.exists, config_files))
+
+ if len(config_files):
+ # at least one config exists and accessible
+diff --git a/fail2ban/client/csocket.py b/fail2ban/client/csocket.py
+index ab3e294b..9417cde9 100644
+--- a/fail2ban/client/csocket.py
++++ b/fail2ban/client/csocket.py
+@@ -47,7 +47,7 @@ class CSocket:
+
+ def send(self, msg, nonblocking=False, timeout=None):
+ # Convert every list member to string
+- obj = dumps(map(CSocket.convert, msg), HIGHEST_PROTOCOL)
++ obj = dumps(list(map(CSocket.convert, msg)), HIGHEST_PROTOCOL)
+ self.__csock.send(obj + CSPROTO.END)
+ return self.receive(self.__csock, nonblocking, timeout)
+
+@@ -71,7 +71,7 @@ class CSocket:
+ @staticmethod
+ def convert(m):
+ """Convert every "unexpected" member of message to string"""
+- if isinstance(m, (basestring, bool, int, float, list, dict, set)):
++ if isinstance(m, (str, bool, int, float, list, dict, set)):
+ return m
+ else: # pragma: no cover
+ return str(m)
+diff --git a/fail2ban/client/fail2banclient.py b/fail2ban/client/fail2banclient.py
+index 7c90ca40..7eb11684 100755
+--- a/fail2ban/client/fail2banclient.py
++++ b/fail2ban/client/fail2banclient.py
+@@ -45,7 +45,7 @@ def _thread_name():
+ return threading.current_thread().__class__.__name__
+
+ def input_command(): # pragma: no cover
+- return raw_input(PROMPT)
++ return input(PROMPT)
+
+ ##
+ #
+@@ -444,7 +444,7 @@ class Fail2banClient(Fail2banCmdLine, Thread):
+ return False
+ finally:
+ self._alive = False
+- for s, sh in _prev_signals.iteritems():
++ for s, sh in _prev_signals.items():
+ signal.signal(s, sh)
+
+
+diff --git a/fail2ban/client/fail2banregex.py b/fail2ban/client/fail2banregex.py
+index 513b765d..4a71b3c0 100644
+--- a/fail2ban/client/fail2banregex.py
++++ b/fail2ban/client/fail2banregex.py
+@@ -41,10 +41,10 @@ import shlex
+ import sys
+ import time
+ import time
+-import urllib
++import urllib.request, urllib.parse, urllib.error
+ from optparse import OptionParser, Option
+
+-from ConfigParser import NoOptionError, NoSectionError, MissingSectionHeaderError
++from configparser import NoOptionError, NoSectionError, MissingSectionHeaderError
+
+ try: # pragma: no cover
+ from ..server.filtersystemd import FilterSystemd
+@@ -68,7 +68,7 @@ def debuggexURL(sample, regex, multiline=False, useDns="yes"):
+ 'flavor': 'python'
+ }
+ if multiline: args['flags'] = 'm'
+- return 'https://www.debuggex.com/?' + urllib.urlencode(args)
++ return 'https://www.debuggex.com/?' + urllib.parse.urlencode(args)
+
+ def output(args): # pragma: no cover (overriden in test-cases)
+ print(args)
+@@ -244,7 +244,7 @@ class Fail2banRegex(object):
+
+ def __init__(self, opts):
+ # set local protected members from given options:
+- self.__dict__.update(dict(('_'+o,v) for o,v in opts.__dict__.iteritems()))
++ self.__dict__.update(dict(('_'+o,v) for o,v in opts.__dict__.items()))
+ self._opts = opts
+ self._maxlines_set = False # so we allow to override maxlines in cmdline
+ self._datepattern_set = False
+@@ -304,7 +304,7 @@ class Fail2banRegex(object):
+ realopts = {}
+ combopts = reader.getCombined()
+ # output all options that are specified in filter-argument as well as some special (mostly interested):
+- for k in ['logtype', 'datepattern'] + fltOpt.keys():
++ for k in ['logtype', 'datepattern'] + list(fltOpt.keys()):
+ # combined options win, but they contain only a sub-set in filter expected keys,
+ # so get the rest from definition section:
+ try:
+@@ -424,7 +424,7 @@ class Fail2banRegex(object):
+ self.output( "Use %11s line : %s" % (regex, shortstr(value)) )
+ regex_values = {regextype: [RegexStat(value)]}
+
+- for regextype, regex_values in regex_values.iteritems():
++ for regextype, regex_values in regex_values.items():
+ regex = regextype + 'regex'
+ setattr(self, "_" + regex, regex_values)
+ for regex in regex_values:
+@@ -523,10 +523,10 @@ class Fail2banRegex(object):
+ output(ret[1])
+ elif self._opts.out == 'msg':
+ for ret in ret:
+- output('\n'.join(map(lambda v:''.join(v for v in v), ret[3].get('matches'))))
++ output('\n'.join([''.join(v for v in v) for v in ret[3].get('matches')]))
+ elif self._opts.out == 'row':
+ for ret in ret:
+- output('[%r,\t%r,\t%r],' % (ret[1],ret[2],dict((k,v) for k, v in ret[3].iteritems() if k != 'matches')))
++ output('[%r,\t%r,\t%r],' % (ret[1],ret[2],dict((k,v) for k, v in ret[3].items() if k != 'matches')))
+ else:
+ for ret in ret:
+ output(ret[3].get(self._opts.out))
+@@ -565,9 +565,9 @@ class Fail2banRegex(object):
+ ans = [[]]
+ for arg in [l, regexlist]:
+ ans = [ x + [y] for x in ans for y in arg ]
+- b = map(lambda a: a[0] + ' | ' + a[1].getFailRegex() + ' | ' +
++ b = [a[0] + ' | ' + a[1].getFailRegex() + ' | ' +
+ debuggexURL(self.encode_line(a[0]), a[1].getFailRegex(),
+- multiline, self._opts.usedns), ans)
++ multiline, self._opts.usedns) for a in ans]
+ pprint_list([x.rstrip() for x in b], header)
+ else:
+ output( "%s too many to print. Use --print-all-%s " \
+diff --git a/fail2ban/client/filterreader.py b/fail2ban/client/filterreader.py
+index 413f125e..4f0cc4cf 100644
+--- a/fail2ban/client/filterreader.py
++++ b/fail2ban/client/filterreader.py
+@@ -71,7 +71,7 @@ class FilterReader(DefinitionInitConfigReader):
+ @staticmethod
+ def _fillStream(stream, opts, jailName):
+ prio0idx = 0
+- for opt, value in opts.iteritems():
++ for opt, value in opts.items():
+ if opt in ("failregex", "ignoreregex"):
+ if value is None: continue
+ multi = []
+diff --git a/fail2ban/client/jailreader.py b/fail2ban/client/jailreader.py
+index 50c1d047..969d0bc0 100644
+--- a/fail2ban/client/jailreader.py
++++ b/fail2ban/client/jailreader.py
+@@ -117,7 +117,7 @@ class JailReader(ConfigReader):
+ }
+ _configOpts.update(FilterReader._configOpts)
+
+- _ignoreOpts = set(['action', 'filter', 'enabled'] + FilterReader._configOpts.keys())
++ _ignoreOpts = set(['action', 'filter', 'enabled'] + list(FilterReader._configOpts.keys()))
+
+ def getOptions(self):
+
+@@ -236,7 +236,7 @@ class JailReader(ConfigReader):
+ stream.extend(self.__filter.convert())
+ # and using options from jail:
+ FilterReader._fillStream(stream, self.__opts, self.__name)
+- for opt, value in self.__opts.iteritems():
++ for opt, value in self.__opts.items():
+ if opt == "logpath":
+ if self.__opts.get('backend', '').startswith("systemd"): continue
+ found_files = 0
+diff --git a/fail2ban/helpers.py b/fail2ban/helpers.py
+index 6f2bcdd7..7e563696 100644
+--- a/fail2ban/helpers.py
++++ b/fail2ban/helpers.py
+@@ -31,6 +31,7 @@ import traceback
+ from threading import Lock
+
+ from .server.mytime import MyTime
++import importlib
+
+ try:
+ import ctypes
+@@ -63,7 +64,7 @@ if sys.version_info < (3,): # pragma: 3.x no cover
+ from imp import load_dynamic as __ldm
+ _sys = __ldm('_sys', 'sys')
+ except ImportError: # pragma: no cover - only if load_dynamic fails
+- reload(sys)
++ importlib.reload(sys)
+ _sys = sys
+ if hasattr(_sys, "setdefaultencoding"):
+ _sys.setdefaultencoding(encoding)
+@@ -101,7 +102,7 @@ if sys.version_info >= (3,): # pragma: 2.x no cover
+ else: # pragma: 3.x no cover
+ def uni_decode(x, enc=PREFER_ENC, errors='strict'):
+ try:
+- if isinstance(x, unicode):
++ if isinstance(x, str):
+ return x.encode(enc, errors)
+ return x
+ except (UnicodeDecodeError, UnicodeEncodeError): # pragma: no cover - unsure if reachable
+@@ -110,7 +111,7 @@ else: # pragma: 3.x no cover
+ return x.encode(enc, 'replace')
+ if sys.getdefaultencoding().upper() != 'UTF-8': # pragma: no cover - utf-8 is default encoding now
+ def uni_string(x):
+- if not isinstance(x, unicode):
++ if not isinstance(x, str):
+ return str(x)
+ return x.encode(PREFER_ENC, 'replace')
+ else:
+@@ -118,7 +119,7 @@ else: # pragma: 3.x no cover
+
+
+ def _as_bool(val):
+- return bool(val) if not isinstance(val, basestring) \
++ return bool(val) if not isinstance(val, str) \
+ else val.lower() in ('1', 'on', 'true', 'yes')
+
+
+@@ -326,7 +327,7 @@ def splitwords(s):
+ """
+ if not s:
+ return []
+- return filter(bool, map(lambda v: v.strip(), re.split('[ ,\n]+', s)))
++ return list(filter(bool, [v.strip() for v in re.split('[ ,\n]+', s)]))
+
+ if sys.version_info >= (3,5):
+ eval(compile(r'''if 1:
+@@ -436,7 +437,7 @@ def substituteRecursiveTags(inptags, conditional='',
+ while True:
+ repFlag = False
+ # substitute each value:
+- for tag in tags.iterkeys():
++ for tag in tags.keys():
+ # ignore escaped or already done (or in ignore list):
+ if tag in ignore or tag in done: continue
+ # ignore replacing callable items from calling map - should be converted on demand only (by get):
+@@ -476,7 +477,7 @@ def substituteRecursiveTags(inptags, conditional='',
+ m = tre_search(value, m.end())
+ continue
+ # if calling map - be sure we've string:
+- if not isinstance(repl, basestring): repl = uni_string(repl)
++ if not isinstance(repl, str): repl = uni_string(repl)
+ value = value.replace('<%s>' % rtag, repl)
+ #logSys.log(5, 'value now: %s' % value)
+ # increment reference count:
+diff --git a/fail2ban/server/action.py b/fail2ban/server/action.py
+index 5c817fc0..81d50689 100644
+--- a/fail2ban/server/action.py
++++ b/fail2ban/server/action.py
+@@ -111,9 +111,9 @@ class CallingMap(MutableMapping, object):
+ def _asdict(self, calculated=False, checker=None):
+ d = dict(self.data, **self.storage)
+ if not calculated:
+- return dict((n,v) for n,v in d.iteritems() \
++ return dict((n,v) for n,v in d.items() \
+ if not callable(v) or n in self.CM_REPR_ITEMS)
+- for n,v in d.items():
++ for n,v in list(d.items()):
+ if callable(v):
+ try:
+ # calculate:
+@@ -179,7 +179,7 @@ class CallingMap(MutableMapping, object):
+ return self.__class__(_merge_copy_dicts(self.data, self.storage))
+
+
+-class ActionBase(object):
++class ActionBase(object, metaclass=ABCMeta):
+ """An abstract base class for actions in Fail2Ban.
+
+ Action Base is a base definition of what methods need to be in
+@@ -209,7 +209,6 @@ class ActionBase(object):
+ Any additional arguments specified in `jail.conf` or passed
+ via `fail2ban-client` will be passed as keyword arguments.
+ """
+- __metaclass__ = ABCMeta
+
+ @classmethod
+ def __subclasshook__(cls, C):
+@@ -420,7 +419,7 @@ class CommandAction(ActionBase):
+ if not callable(family): # pragma: no cover
+ return self.__substCache.get(key, {}).get(family)
+ # family as expression - use it to filter values:
+- return [v for f, v in self.__substCache.get(key, {}).iteritems() if family(f)]
++ return [v for f, v in self.__substCache.get(key, {}).items() if family(f)]
+ cmd = args[0]
+ if cmd: # set:
+ try:
+@@ -432,7 +431,7 @@ class CommandAction(ActionBase):
+ try:
+ famd = self.__substCache[key]
+ cmd = famd.pop(family)
+- for family, v in famd.items():
++ for family, v in list(famd.items()):
+ if v == cmd:
+ del famd[family]
+ except KeyError: # pragma: no cover
+@@ -448,7 +447,7 @@ class CommandAction(ActionBase):
+ res = True
+ err = 'Script error'
+ if not family: # all started:
+- family = [famoper for (famoper,v) in self.__started.iteritems() if v]
++ family = [famoper for (famoper,v) in self.__started.items() if v]
+ for famoper in family:
+ try:
+ cmd = self._getOperation(tag, famoper)
+@@ -617,7 +616,7 @@ class CommandAction(ActionBase):
+ and executes the resulting command.
+ """
+ # collect started families, may be started on demand (conditional):
+- family = [f for (f,v) in self.__started.iteritems() if v & 3 == 3]; # started and contains items
++ family = [f for (f,v) in self.__started.items() if v & 3 == 3]; # started and contains items
+ # if nothing contains items:
+ if not family: return True
+ # flush:
+@@ -642,7 +641,7 @@ class CommandAction(ActionBase):
+ """
+ # collect started families, if started on demand (conditional):
+ if family is None:
+- family = [f for (f,v) in self.__started.iteritems() if v]
++ family = [f for (f,v) in self.__started.items() if v]
+ # if no started (on demand) actions:
+ if not family: return True
+ self.__started = {}
+@@ -676,7 +675,7 @@ class CommandAction(ActionBase):
+ ret = True
+ # for each started family:
+ if self.actioncheck:
+- for (family, started) in self.__started.items():
++ for (family, started) in list(self.__started.items()):
+ if started and not self._invariantCheck(family, beforeRepair):
+ # reset started flag and command of executed operation:
+ self.__started[family] = 0
+diff --git a/fail2ban/server/actions.py b/fail2ban/server/actions.py
+index 24fea838..94b9c3ed 100644
+--- a/fail2ban/server/actions.py
++++ b/fail2ban/server/actions.py
+@@ -156,11 +156,11 @@ class Actions(JailThread, Mapping):
+ else:
+ if hasattr(self, '_reload_actions'):
+ # reload actions after all parameters set via stream:
+- for name, initOpts in self._reload_actions.iteritems():
++ for name, initOpts in self._reload_actions.items():
+ if name in self._actions:
+ self._actions[name].reload(**(initOpts if initOpts else {}))
+ # remove obsolete actions (untouched by reload process):
+- delacts = OrderedDict((name, action) for name, action in self._actions.iteritems()
++ delacts = OrderedDict((name, action) for name, action in self._actions.items()
+ if name not in self._reload_actions)
+ if len(delacts):
+ # unban all tickets using removed actions only:
+@@ -289,7 +289,7 @@ class Actions(JailThread, Mapping):
+ """
+ if actions is None:
+ actions = self._actions
+- revactions = actions.items()
++ revactions = list(actions.items())
+ revactions.reverse()
+ for name, action in revactions:
+ try:
+@@ -314,7 +314,7 @@ class Actions(JailThread, Mapping):
+ True when the thread exits nicely.
+ """
+ cnt = 0
+- for name, action in self._actions.iteritems():
++ for name, action in self._actions.items():
+ try:
+ action.start()
+ except Exception as e:
+@@ -474,7 +474,7 @@ class Actions(JailThread, Mapping):
+ Observers.Main.add('banFound', bTicket, self._jail, btime)
+ logSys.notice("[%s] %sBan %s", self._jail.name, ('' if not bTicket.restored else 'Restore '), ip)
+ # do actions :
+- for name, action in self._actions.iteritems():
++ for name, action in self._actions.items():
+ try:
+ if ticket.restored and getattr(action, 'norestored', False):
+ continue
+@@ -511,13 +511,13 @@ class Actions(JailThread, Mapping):
+ if bTicket.banEpoch == self.banEpoch and diftm > 3:
+ # avoid too often checks:
+ if not rebanacts and MyTime.time() > self.__lastConsistencyCheckTM + 3:
+- for action in self._actions.itervalues():
++ for action in self._actions.values():
+ action.consistencyCheck()
+ self.__lastConsistencyCheckTM = MyTime.time()
+ # check epoch in order to reban it:
+ if bTicket.banEpoch < self.banEpoch:
+ if not rebanacts: rebanacts = dict(
+- (name, action) for name, action in self._actions.iteritems()
++ (name, action) for name, action in self._actions.items()
+ if action.banEpoch > bTicket.banEpoch)
+ cnt += self.__reBan(bTicket, actions=rebanacts)
+ else: # pragma: no cover - unexpected: ticket is not banned for some reasons - reban using all actions:
+@@ -542,8 +542,8 @@ class Actions(JailThread, Mapping):
+ ip = ticket.getIP()
+ aInfo = self.__getActionInfo(ticket)
+ if log:
+- logSys.notice("[%s] Reban %s%s", self._jail.name, aInfo["ip"], (', action %r' % actions.keys()[0] if len(actions) == 1 else ''))
+- for name, action in actions.iteritems():
++ logSys.notice("[%s] Reban %s%s", self._jail.name, aInfo["ip"], (', action %r' % list(actions.keys())[0] if len(actions) == 1 else ''))
++ for name, action in actions.items():
+ try:
+ logSys.debug("[%s] action %r: reban %s", self._jail.name, name, ip)
+ if not aInfo.immutable: aInfo.reset()
+@@ -567,7 +567,7 @@ class Actions(JailThread, Mapping):
+ if not self.__banManager._inBanList(ticket): return
+ # do actions :
+ aInfo = None
+- for name, action in self._actions.iteritems():
++ for name, action in self._actions.items():
+ try:
+ if ticket.restored and getattr(action, 'norestored', False):
+ continue
+@@ -616,7 +616,7 @@ class Actions(JailThread, Mapping):
+ cnt = 0
+ # first we'll execute flush for actions supporting this operation:
+ unbactions = {}
+- for name, action in (actions if actions is not None else self._actions).iteritems():
++ for name, action in (actions if actions is not None else self._actions).items():
+ try:
+ if hasattr(action, 'flush') and (not isinstance(action, CommandAction) or action.actionflush):
+ logSys.notice("[%s] Flush ticket(s) with %s", self._jail.name, name)
+@@ -671,7 +671,7 @@ class Actions(JailThread, Mapping):
+ aInfo = self.__getActionInfo(ticket)
+ if log:
+ logSys.notice("[%s] Unban %s", self._jail.name, aInfo["ip"])
+- for name, action in unbactions.iteritems():
++ for name, action in unbactions.items():
+ try:
+ logSys.debug("[%s] action %r: unban %s", self._jail.name, name, ip)
+ if not aInfo.immutable: aInfo.reset()
+diff --git a/fail2ban/server/asyncserver.py b/fail2ban/server/asyncserver.py
+index e3400737..f5f9740b 100644
+--- a/fail2ban/server/asyncserver.py
++++ b/fail2ban/server/asyncserver.py
+@@ -178,7 +178,7 @@ def loop(active, timeout=None, use_poll=False, err_count=None):
+ elif err_count['listen'] > 100: # pragma: no cover - normally unreachable
+ if (
+ e.args[0] == errno.EMFILE # [Errno 24] Too many open files
+- or sum(err_count.itervalues()) > 1000
++ or sum(err_count.values()) > 1000
+ ):
+ logSys.critical("Too many errors - critical count reached %r", err_count)
+ break
+@@ -220,7 +220,7 @@ class AsyncServer(asyncore.dispatcher):
+ elif self.__errCount['accept'] > 100:
+ if (
+ (isinstance(e, socket.error) and e.args[0] == errno.EMFILE) # [Errno 24] Too many open files
+- or sum(self.__errCount.itervalues()) > 1000
++ or sum(self.__errCount.values()) > 1000
+ ):
+ logSys.critical("Too many errors - critical count reached %r", self.__errCount)
+ self.stop()
+diff --git a/fail2ban/server/banmanager.py b/fail2ban/server/banmanager.py
+index 5770bfd7..9bb44971 100644
+--- a/fail2ban/server/banmanager.py
++++ b/fail2ban/server/banmanager.py
+@@ -105,9 +105,9 @@ class BanManager:
+ def getBanList(self, ordered=False, withTime=False):
+ with self.__lock:
+ if not ordered:
+- return self.__banList.keys()
++ return list(self.__banList.keys())
+ lst = []
+- for ticket in self.__banList.itervalues():
++ for ticket in self.__banList.values():
+ eob = ticket.getEndOfBanTime(self.__banTime)
+ lst.append((ticket,eob))
+ lst.sort(key=lambda t: t[1])
+@@ -126,7 +126,7 @@ class BanManager:
+
+ def __iter__(self):
+ with self.__lock:
+- return self.__banList.itervalues()
++ return iter(self.__banList.values())
+
+ ##
+ # Returns normalized value
+@@ -165,7 +165,7 @@ class BanManager:
+ return return_dict
+ # get ips in lock:
+ with self.__lock:
+- banIPs = [banData.getIP() for banData in self.__banList.values()]
++ banIPs = [banData.getIP() for banData in list(self.__banList.values())]
+ # get cymru info:
+ try:
+ for ip in banIPs:
+@@ -341,7 +341,7 @@ class BanManager:
+ # Gets the list of ticket to remove (thereby correct next unban time).
+ unBanList = {}
+ nextUnbanTime = BanTicket.MAX_TIME
+- for fid,ticket in self.__banList.iteritems():
++ for fid,ticket in self.__banList.items():
+ # current time greater as end of ban - timed out:
+ eob = ticket.getEndOfBanTime(self.__banTime)
+ if time > eob:
+@@ -357,15 +357,15 @@ class BanManager:
+ if len(unBanList):
+ if len(unBanList) / 2.0 <= len(self.__banList) / 3.0:
+ # few as 2/3 should be removed - remove particular items:
+- for fid in unBanList.iterkeys():
++ for fid in unBanList.keys():
+ del self.__banList[fid]
+ else:
+ # create new dictionary without items to be deleted:
+- self.__banList = dict((fid,ticket) for fid,ticket in self.__banList.iteritems() \
++ self.__banList = dict((fid,ticket) for fid,ticket in self.__banList.items() \
+ if fid not in unBanList)
+
+ # return list of tickets:
+- return unBanList.values()
++ return list(unBanList.values())
+
+ ##
+ # Flush the ban list.
+@@ -375,7 +375,7 @@ class BanManager:
+
+ def flushBanList(self):
+ with self.__lock:
+- uBList = self.__banList.values()
++ uBList = list(self.__banList.values())
+ self.__banList = dict()
+ return uBList
+
+diff --git a/fail2ban/server/database.py b/fail2ban/server/database.py
+index ed736a7a..0e8c9aec 100644
+--- a/fail2ban/server/database.py
++++ b/fail2ban/server/database.py
+@@ -67,13 +67,13 @@ if sys.version_info >= (3,): # pragma: 2.x no cover
+ else: # pragma: 3.x no cover
+ def _normalize(x):
+ if isinstance(x, dict):
+- return dict((_normalize(k), _normalize(v)) for k, v in x.iteritems())
++ return dict((_normalize(k), _normalize(v)) for k, v in x.items())
+ elif isinstance(x, (list, set)):
+ return [_normalize(element) for element in x]
+- elif isinstance(x, unicode):
++ elif isinstance(x, str):
+ # in 2.x default text_factory is unicode - so return proper unicode here:
+ return x.encode(PREFER_ENC, 'replace').decode(PREFER_ENC)
+- elif isinstance(x, basestring):
++ elif isinstance(x, str):
+ return x.decode(PREFER_ENC, 'replace')
+ return x
+
+diff --git a/fail2ban/server/failmanager.py b/fail2ban/server/failmanager.py
+index 93c028fb..a9c6b5f6 100644
+--- a/fail2ban/server/failmanager.py
++++ b/fail2ban/server/failmanager.py
+@@ -57,7 +57,7 @@ class FailManager:
+ def getFailCount(self):
+ # may be slow on large list of failures, should be used for test purposes only...
+ with self.__lock:
+- return len(self.__failList), sum([f.getRetry() for f in self.__failList.values()])
++ return len(self.__failList), sum([f.getRetry() for f in list(self.__failList.values())])
+
+ def getFailTotal(self):
+ with self.__lock:
+@@ -125,7 +125,7 @@ class FailManager:
+ # in case of having many active failures, it should be ran only
+ # if debug level is "low" enough
+ failures_summary = ', '.join(['%s:%d' % (k, v.getRetry())
+- for k,v in self.__failList.iteritems()])
++ for k,v in self.__failList.items()])
+ logSys.log(logLevel, "Total # of detected failures: %d. Current failures from %d IPs (IP:count): %s"
+ % (self.__failTotal, len(self.__failList), failures_summary))
+
+@@ -138,7 +138,7 @@ class FailManager:
+
+ def cleanup(self, time):
+ with self.__lock:
+- todelete = [fid for fid,item in self.__failList.iteritems() \
++ todelete = [fid for fid,item in self.__failList.items() \
+ if item.getLastTime() + self.__maxTime <= time]
+ if len(todelete) == len(self.__failList):
+ # remove all:
+@@ -152,7 +152,7 @@ class FailManager:
+ del self.__failList[fid]
+ else:
+ # create new dictionary without items to be deleted:
+- self.__failList = dict((fid,item) for fid,item in self.__failList.iteritems() \
++ self.__failList = dict((fid,item) for fid,item in self.__failList.items() \
+ if item.getLastTime() + self.__maxTime > time)
+ self.__bgSvc.service()
+
+diff --git a/fail2ban/server/failregex.py b/fail2ban/server/failregex.py
+index f7dafbef..fb75187d 100644
+--- a/fail2ban/server/failregex.py
++++ b/fail2ban/server/failregex.py
+@@ -128,10 +128,7 @@ class Regex:
+ self._regexObj = re.compile(regex, re.MULTILINE if multiline else 0)
+ self._regex = regex
+ self._altValues = {}
+- for k in filter(
+- lambda k: len(k) > len(ALTNAME_PRE) and k.startswith(ALTNAME_PRE),
+- self._regexObj.groupindex
+- ):
++ for k in [k for k in self._regexObj.groupindex if len(k) > len(ALTNAME_PRE) and k.startswith(ALTNAME_PRE)]:
+ n = ALTNAME_CRE.match(k).group(1)
+ self._altValues[k] = n
+ self._altValues = list(self._altValues.items()) if len(self._altValues) else None
+@@ -211,7 +208,7 @@ class Regex:
+ #
+ @staticmethod
+ def _tupleLinesBuf(tupleLines):
+- return "\n".join(map(lambda v: "".join(v[::2]), tupleLines)) + "\n"
++ return "\n".join(["".join(v[::2]) for v in tupleLines]) + "\n"
+
+ ##
+ # Searches the regular expression.
+@@ -223,7 +220,7 @@ class Regex:
+
+ def search(self, tupleLines, orgLines=None):
+ buf = tupleLines
+- if not isinstance(tupleLines, basestring):
++ if not isinstance(tupleLines, str):
+ buf = Regex._tupleLinesBuf(tupleLines)
+ self._matchCache = self._regexObj.search(buf)
+ if self._matchCache:
+diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py
+index 998fe298..d181fd38 100644
+--- a/fail2ban/server/filter.py
++++ b/fail2ban/server/filter.py
+@@ -292,7 +292,7 @@ class Filter(JailThread):
+ dd = DateDetector()
+ dd.default_tz = self.__logtimezone
+ if not isinstance(pattern, (list, tuple)):
+- pattern = filter(bool, map(str.strip, re.split('\n+', pattern)))
++ pattern = list(filter(bool, list(map(str.strip, re.split('\n+', pattern)))))
+ for pattern in pattern:
+ dd.appendTemplate(pattern)
+ self.dateDetector = dd
+@@ -987,7 +987,7 @@ class FileFilter(Filter):
+ # @return log paths
+
+ def getLogPaths(self):
+- return self.__logs.keys()
++ return list(self.__logs.keys())
+
+ ##
+ # Get the log containers
+@@ -995,7 +995,7 @@ class FileFilter(Filter):
+ # @return log containers
+
+ def getLogs(self):
+- return self.__logs.values()
++ return list(self.__logs.values())
+
+ ##
+ # Get the count of log containers
+@@ -1021,7 +1021,7 @@ class FileFilter(Filter):
+
+ def setLogEncoding(self, encoding):
+ encoding = super(FileFilter, self).setLogEncoding(encoding)
+- for log in self.__logs.itervalues():
++ for log in self.__logs.values():
+ log.setEncoding(encoding)
+
+ def getLog(self, path):
+@@ -1183,7 +1183,7 @@ class FileFilter(Filter):
+ """Status of Filter plus files being monitored.
+ """
+ ret = super(FileFilter, self).status(flavor=flavor)
+- path = self.__logs.keys()
++ path = list(self.__logs.keys())
+ ret.append(("File list", path))
+ return ret
+
+@@ -1191,7 +1191,7 @@ class FileFilter(Filter):
+ """Stop monitoring of log-file(s)
+ """
+ # stop files monitoring:
+- for path in self.__logs.keys():
++ for path in list(self.__logs.keys()):
+ self.delLogPath(path)
+ # stop thread:
+ super(Filter, self).stop()
+diff --git a/fail2ban/server/filterpoll.py b/fail2ban/server/filterpoll.py
+index 228a2c8b..d49315cc 100644
+--- a/fail2ban/server/filterpoll.py
++++ b/fail2ban/server/filterpoll.py
+@@ -176,4 +176,4 @@ class FilterPoll(FileFilter):
+ return False
+
+ def getPendingPaths(self):
+- return self.__file404Cnt.keys()
++ return list(self.__file404Cnt.keys())
+diff --git a/fail2ban/server/filterpyinotify.py b/fail2ban/server/filterpyinotify.py
+index ca6b253f..b683b860 100644
+--- a/fail2ban/server/filterpyinotify.py
++++ b/fail2ban/server/filterpyinotify.py
+@@ -158,7 +158,7 @@ class FilterPyinotify(FileFilter):
+ except KeyError: pass
+
+ def getPendingPaths(self):
+- return self.__pending.keys()
++ return list(self.__pending.keys())
+
+ def _checkPending(self):
+ if not self.__pending:
+@@ -168,7 +168,7 @@ class FilterPyinotify(FileFilter):
+ return
+ found = {}
+ minTime = 60
+- for path, (retardTM, isDir) in self.__pending.iteritems():
++ for path, (retardTM, isDir) in self.__pending.items():
+ if ntm - self.__pendingChkTime < retardTM:
+ if minTime > retardTM: minTime = retardTM
+ continue
+@@ -184,7 +184,7 @@ class FilterPyinotify(FileFilter):
+ self.__pendingChkTime = time.time()
+ self.__pendingMinTime = minTime
+ # process now because we've missed it in monitoring:
+- for path, isDir in found.iteritems():
++ for path, isDir in found.items():
+ self._delPending(path)
+ # refresh monitoring of this:
+ self._refreshWatcher(path, isDir=isDir)
+diff --git a/fail2ban/server/ipdns.py b/fail2ban/server/ipdns.py
+index 6648dac6..fe8f8db8 100644
+--- a/fail2ban/server/ipdns.py
++++ b/fail2ban/server/ipdns.py
+@@ -275,7 +275,7 @@ class IPAddr(object):
+ raise ValueError("invalid ipstr %r, too many plen representation" % (ipstr,))
+ if "." in s[1] or ":" in s[1]: # 255.255.255.0 resp. ffff:: style mask
+ s[1] = IPAddr.masktoplen(s[1])
+- s[1] = long(s[1])
++ s[1] = int(s[1])
+ return s
+
+ def __init(self, ipstr, cidr=CIDR_UNSPEC):
+@@ -309,7 +309,7 @@ class IPAddr(object):
+
+ # mask out host portion if prefix length is supplied
+ if cidr is not None and cidr >= 0:
+- mask = ~(0xFFFFFFFFL >> cidr)
++ mask = ~(0xFFFFFFFF >> cidr)
+ self._addr &= mask
+ self._plen = cidr
+
+@@ -321,13 +321,13 @@ class IPAddr(object):
+
+ # mask out host portion if prefix length is supplied
+ if cidr is not None and cidr >= 0:
+- mask = ~(0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFL >> cidr)
++ mask = ~(0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF >> cidr)
+ self._addr &= mask
+ self._plen = cidr
+
+ # if IPv6 address is a IPv4-compatible, make instance a IPv4
+ elif self.isInNet(IPAddr.IP6_4COMPAT):
+- self._addr = lo & 0xFFFFFFFFL
++ self._addr = lo & 0xFFFFFFFF
+ self._family = socket.AF_INET
+ self._plen = 32
+ else:
+@@ -445,7 +445,7 @@ class IPAddr(object):
+ elif self.isIPv6:
+ # convert network to host byte order
+ hi = self._addr >> 64
+- lo = self._addr & 0xFFFFFFFFFFFFFFFFL
++ lo = self._addr & 0xFFFFFFFFFFFFFFFF
+ binary = struct.pack("!QQ", hi, lo)
+ if self._plen and self._plen < 128:
+ add = "/%d" % self._plen
+@@ -503,9 +503,9 @@ class IPAddr(object):
+ if self.family != net.family:
+ return False
+ if self.isIPv4:
+- mask = ~(0xFFFFFFFFL >> net.plen)
++ mask = ~(0xFFFFFFFF >> net.plen)
+ elif self.isIPv6:
+- mask = ~(0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFL >> net.plen)
++ mask = ~(0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF >> net.plen)
+ else:
+ return False
+
+@@ -517,7 +517,7 @@ class IPAddr(object):
+ m4 = (1 << 32)-1
+ mmap = {m6: 128, m4: 32, 0: 0}
+ m = 0
+- for i in xrange(0, 128):
++ for i in range(0, 128):
+ m |= 1 << i
+ if i < 32:
+ mmap[m ^ m4] = 32-1-i
+diff --git a/fail2ban/server/jail.py b/fail2ban/server/jail.py
+index ce9968a8..5fa5ef10 100644
+--- a/fail2ban/server/jail.py
++++ b/fail2ban/server/jail.py
+@@ -26,7 +26,7 @@ __license__ = "GPL"
+ import logging
+ import math
+ import random
+-import Queue
++import queue
+
+ from .actions import Actions
+ from ..helpers import getLogger, _as_bool, extractOptions, MyTime
+@@ -76,7 +76,7 @@ class Jail(object):
+ "might not function correctly. Please shorten"
+ % name)
+ self.__name = name
+- self.__queue = Queue.Queue()
++ self.__queue = queue.Queue()
+ self.__filter = None
+ # Extra parameters for increase ban time
+ self._banExtra = {};
+@@ -127,25 +127,25 @@ class Jail(object):
+ "Failed to initialize any backend for Jail %r" % self.name)
+
+ def _initPolling(self, **kwargs):
+- from filterpoll import FilterPoll
++ from .filterpoll import FilterPoll
+ logSys.info("Jail '%s' uses poller %r" % (self.name, kwargs))
+ self.__filter = FilterPoll(self, **kwargs)
+
+ def _initGamin(self, **kwargs):
+ # Try to import gamin
+- from filtergamin import FilterGamin
++ from .filtergamin import FilterGamin
+ logSys.info("Jail '%s' uses Gamin %r" % (self.name, kwargs))
+ self.__filter = FilterGamin(self, **kwargs)
+
+ def _initPyinotify(self, **kwargs):
+ # Try to import pyinotify
+- from filterpyinotify import FilterPyinotify
++ from .filterpyinotify import FilterPyinotify
+ logSys.info("Jail '%s' uses pyinotify %r" % (self.name, kwargs))
+ self.__filter = FilterPyinotify(self, **kwargs)
+
+ def _initSystemd(self, **kwargs): # pragma: systemd no cover
+ # Try to import systemd
+- from filtersystemd import FilterSystemd
++ from .filtersystemd import FilterSystemd
+ logSys.info("Jail '%s' uses systemd %r" % (self.name, kwargs))
+ self.__filter = FilterSystemd(self, **kwargs)
+
+@@ -213,7 +213,7 @@ class Jail(object):
+ try:
+ ticket = self.__queue.get(False)
+ return ticket
+- except Queue.Empty:
++ except queue.Empty:
+ return False
+
+ def setBanTimeExtra(self, opt, value):
+diff --git a/fail2ban/server/mytime.py b/fail2ban/server/mytime.py
+index 98b69bd4..24bba5cf 100644
+--- a/fail2ban/server/mytime.py
++++ b/fail2ban/server/mytime.py
+@@ -162,7 +162,7 @@ class MyTime:
+
+ @returns number (calculated seconds from expression "val")
+ """
+- if isinstance(val, (int, long, float, complex)):
++ if isinstance(val, (int, float, complex)):
+ return val
+ # replace together standing abbreviations, example '1d12h' -> '1d 12h':
+ val = MyTime._str2sec_prep.sub(r" \1", val)
+diff --git a/fail2ban/server/server.py b/fail2ban/server/server.py
+index 159f6506..fc948e8c 100644
+--- a/fail2ban/server/server.py
++++ b/fail2ban/server/server.py
+@@ -97,7 +97,7 @@ class Server:
+
+ def start(self, sock, pidfile, force=False, observer=True, conf={}):
+ # First set the mask to only allow access to owner
+- os.umask(0077)
++ os.umask(0o077)
+ # Second daemonize before logging etc, because it will close all handles:
+ if self.__daemon: # pragma: no cover
+ logSys.info("Starting in daemon mode")
+@@ -190,7 +190,7 @@ class Server:
+
+ # Restore default signal handlers:
+ if _thread_name() == '_MainThread':
+- for s, sh in self.__prev_signals.iteritems():
++ for s, sh in self.__prev_signals.items():
+ signal.signal(s, sh)
+
+ # Give observer a small chance to complete its work before exit
+@@ -268,10 +268,10 @@ class Server:
+ logSys.info("Stopping all jails")
+ with self.__lock:
+ # 1st stop all jails (signal and stop actions/filter thread):
+- for name in self.__jails.keys():
++ for name in list(self.__jails.keys()):
+ self.delJail(name, stop=True, join=False)
+ # 2nd wait for end and delete jails:
+- for name in self.__jails.keys():
++ for name in list(self.__jails.keys()):
+ self.delJail(name, stop=False, join=True)
+
+ def reloadJails(self, name, opts, begin):
+@@ -302,7 +302,7 @@ class Server:
+ if "--restart" in opts:
+ self.stopAllJail()
+ # first set all affected jail(s) to idle and reset filter regex and other lists/dicts:
+- for jn, jail in self.__jails.iteritems():
++ for jn, jail in self.__jails.items():
+ if name == '--all' or jn == name:
+ jail.idle = True
+ self.__reload_state[jn] = jail
+@@ -313,7 +313,7 @@ class Server:
+ # end reload, all affected (or new) jails have already all new parameters (via stream) and (re)started:
+ with self.__lock:
+ deljails = []
+- for jn, jail in self.__jails.iteritems():
++ for jn, jail in self.__jails.items():
+ # still in reload state:
+ if jn in self.__reload_state:
+ # remove jails that are not reloaded (untouched, so not in new configuration)
+@@ -513,7 +513,7 @@ class Server:
+ jails = [self.__jails[name]]
+ else:
+ # in all jails:
+- jails = self.__jails.values()
++ jails = list(self.__jails.values())
+ # unban given or all (if value is None):
+ cnt = 0
+ ifexists |= (name is None)
+@@ -551,7 +551,7 @@ class Server:
+ def isAlive(self, jailnum=None):
+ if jailnum is not None and len(self.__jails) != jailnum:
+ return 0
+- for jail in self.__jails.values():
++ for jail in list(self.__jails.values()):
+ if not jail.isAlive():
+ return 0
+ return 1
+@@ -759,7 +759,7 @@ class Server:
+ return "flushed"
+
+ def setThreadOptions(self, value):
+- for o, v in value.iteritems():
++ for o, v in value.items():
+ if o == 'stacksize':
+ threading.stack_size(int(v)*1024)
+ else: # pragma: no cover
+diff --git a/fail2ban/server/strptime.py b/fail2ban/server/strptime.py
+index 498d284b..a5579fdc 100644
+--- a/fail2ban/server/strptime.py
++++ b/fail2ban/server/strptime.py
+@@ -79,7 +79,7 @@ timeRE['ExY'] = r"(?P<Y>%s\d)" % _getYearCentRE(cent=(0,3), distance=3)
+ timeRE['Exy'] = r"(?P<y>%s\d)" % _getYearCentRE(cent=(2,3), distance=3)
+
+ def getTimePatternRE():
+- keys = timeRE.keys()
++ keys = list(timeRE.keys())
+ patt = (r"%%(%%|%s|[%s])" % (
+ "|".join([k for k in keys if len(k) > 1]),
+ "".join([k for k in keys if len(k) == 1]),
+@@ -134,7 +134,7 @@ def zone2offset(tz, dt):
+ """
+ if isinstance(tz, int):
+ return tz
+- if isinstance(tz, basestring):
++ if isinstance(tz, str):
+ return validateTimeZone(tz)
+ tz, tzo = tz
+ if tzo is None or tzo == '': # without offset
+@@ -171,7 +171,7 @@ def reGroupDictStrptime(found_dict, msec=False, default_tz=None):
+ year = month = day = hour = minute = tzoffset = \
+ weekday = julian = week_of_year = None
+ second = fraction = 0
+- for key, val in found_dict.iteritems():
++ for key, val in found_dict.items():
+ if val is None: continue
+ # Directives not explicitly handled below:
+ # c, x, X
+diff --git a/fail2ban/server/ticket.py b/fail2ban/server/ticket.py
+index f67e0d23..f0b727c2 100644
+--- a/fail2ban/server/ticket.py
++++ b/fail2ban/server/ticket.py
+@@ -55,7 +55,7 @@ class Ticket(object):
+ self._time = time if time is not None else MyTime.time()
+ self._data = {'matches': matches or [], 'failures': 0}
+ if data is not None:
+- for k,v in data.iteritems():
++ for k,v in data.items():
+ if v is not None:
+ self._data[k] = v
+ if ticket:
+@@ -89,7 +89,7 @@ class Ticket(object):
+
+ def setIP(self, value):
+ # guarantee using IPAddr instead of unicode, str for the IP
+- if isinstance(value, basestring):
++ if isinstance(value, str):
+ value = IPAddr(value)
+ self._ip = value
+
+@@ -181,7 +181,7 @@ class Ticket(object):
+ if len(args) == 1:
+ # todo: if support >= 2.7 only:
+ # self._data = {k:v for k,v in args[0].iteritems() if v is not None}
+- self._data = dict([(k,v) for k,v in args[0].iteritems() if v is not None])
++ self._data = dict([(k,v) for k,v in args[0].items() if v is not None])
+ # add k,v list or dict (merge):
+ elif len(args) == 2:
+ self._data.update((args,))
+@@ -192,7 +192,7 @@ class Ticket(object):
+ # filter (delete) None values:
+ # todo: if support >= 2.7 only:
+ # self._data = {k:v for k,v in self._data.iteritems() if v is not None}
+- self._data = dict([(k,v) for k,v in self._data.iteritems() if v is not None])
++ self._data = dict([(k,v) for k,v in self._data.items() if v is not None])
+
+ def getData(self, key=None, default=None):
+ # return whole data dict:
+@@ -201,17 +201,17 @@ class Ticket(object):
+ # return default if not exists:
+ if not self._data:
+ return default
+- if not isinstance(key,(str,unicode,type(None),int,float,bool,complex)):
++ if not isinstance(key,(str,type(None),int,float,bool,complex)):
+ # return filtered by lambda/function:
+ if callable(key):
+ # todo: if support >= 2.7 only:
+ # return {k:v for k,v in self._data.iteritems() if key(k)}
+- return dict([(k,v) for k,v in self._data.iteritems() if key(k)])
++ return dict([(k,v) for k,v in self._data.items() if key(k)])
+ # return filtered by keys:
+ if hasattr(key, '__iter__'):
+ # todo: if support >= 2.7 only:
+ # return {k:v for k,v in self._data.iteritems() if k in key}
+- return dict([(k,v) for k,v in self._data.iteritems() if k in key])
++ return dict([(k,v) for k,v in self._data.items() if k in key])
+ # return single value of data:
+ return self._data.get(key, default)
+
+diff --git a/fail2ban/server/transmitter.py b/fail2ban/server/transmitter.py
+index f83e9d5f..80726cb4 100644
+--- a/fail2ban/server/transmitter.py
++++ b/fail2ban/server/transmitter.py
+@@ -475,7 +475,7 @@ class Transmitter:
+ opt = command[1][len("bantime."):]
+ return self.__server.getBanTimeExtra(name, opt)
+ elif command[1] == "actions":
+- return self.__server.getActions(name).keys()
++ return list(self.__server.getActions(name).keys())
+ elif command[1] == "action":
+ actionname = command[2]
+ actionvalue = command[3]
+diff --git a/fail2ban/server/utils.py b/fail2ban/server/utils.py
+index d4461a7d..13c24e76 100644
+--- a/fail2ban/server/utils.py
++++ b/fail2ban/server/utils.py
+@@ -57,7 +57,7 @@ _RETCODE_HINTS = {
+
+ # Dictionary to lookup signal name from number
+ signame = dict((num, name)
+- for name, num in signal.__dict__.iteritems() if name.startswith("SIG"))
++ for name, num in signal.__dict__.items() if name.startswith("SIG"))
+
+ class Utils():
+ """Utilities provide diverse static methods like executes OS shell commands, etc.
+@@ -109,7 +109,7 @@ class Utils():
+ break
+ else: # pragma: 3.x no cover (dict is in 2.6 only)
+ remlst = []
+- for (ck, cv) in cache.iteritems():
++ for (ck, cv) in cache.items():
+ # if expired:
+ if cv[1] <= t:
+ remlst.append(ck)
+@@ -152,7 +152,7 @@ class Utils():
+ if not isinstance(realCmd, list):
+ realCmd = [realCmd]
+ i = len(realCmd)-1
+- for k, v in varsDict.iteritems():
++ for k, v in varsDict.items():
+ varsStat += "%s=$%s " % (k, i)
+ realCmd.append(v)
+ i += 1
+diff --git a/fail2ban/tests/action_d/test_badips.py b/fail2ban/tests/action_d/test_badips.py
+index 013c0fdb..3c35e4d7 100644
+--- a/fail2ban/tests/action_d/test_badips.py
++++ b/fail2ban/tests/action_d/test_badips.py
+@@ -32,7 +32,7 @@ from ..utils import LogCaptureTestCase, CONFIG_DIR
+ if sys.version_info >= (3, ): # pragma: 2.x no cover
+ from urllib.error import HTTPError, URLError
+ else: # pragma: 3.x no cover
+- from urllib2 import HTTPError, URLError
++ from urllib.error import HTTPError, URLError
+
+ def skip_if_not_available(f):
+ """Helper to decorate tests to skip in case of timeout/http-errors like "502 bad gateway".
+diff --git a/fail2ban/tests/actiontestcase.py b/fail2ban/tests/actiontestcase.py
+index 1a00c040..ecd09246 100644
+--- a/fail2ban/tests/actiontestcase.py
++++ b/fail2ban/tests/actiontestcase.py
+@@ -244,14 +244,14 @@ class CommandActionTest(LogCaptureTestCase):
+ setattr(self.__action, 'ab', "<ac>")
+ setattr(self.__action, 'x?family=inet6', "")
+ # produce self-referencing properties except:
+- self.assertRaisesRegexp(ValueError, r"properties contain self referencing definitions",
++ self.assertRaisesRegex(ValueError, r"properties contain self referencing definitions",
+ lambda: self.__action.replaceTag("<a><b>",
+ self.__action._properties, conditional="family=inet4")
+ )
+ # remore self-referencing in props:
+ delattr(self.__action, 'ac')
+ # produce self-referencing query except:
+- self.assertRaisesRegexp(ValueError, r"possible self referencing definitions in query",
++ self.assertRaisesRegex(ValueError, r"possible self referencing definitions in query",
+ lambda: self.__action.replaceTag("<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x<x>>>>>>>>>>>>>>>>>>>>>",
+ self.__action._properties, conditional="family=inet6")
+ )
+diff --git a/fail2ban/tests/clientreadertestcase.py b/fail2ban/tests/clientreadertestcase.py
+index 2c1d0a0e..aa7908c4 100644
+--- a/fail2ban/tests/clientreadertestcase.py
++++ b/fail2ban/tests/clientreadertestcase.py
+@@ -390,7 +390,7 @@ class JailReaderTest(LogCaptureTestCase):
+ # And multiple groups (`][` instead of `,`)
+ result = extractOptions(option.replace(',', ']['))
+ expected2 = (expected[0],
+- dict((k, v.replace(',', '][')) for k, v in expected[1].iteritems())
++ dict((k, v.replace(',', '][')) for k, v in expected[1].items())
+ )
+ self.assertEqual(expected2, result)
+
+@@ -975,7 +975,7 @@ filter = testfilter1
+ self.assertEqual(add_actions[-1][-1], "{}")
+
+ def testLogPathFileFilterBackend(self):
+- self.assertRaisesRegexp(ValueError, r"Have not found any log file for .* jail",
++ self.assertRaisesRegex(ValueError, r"Have not found any log file for .* jail",
+ self._testLogPath, backend='polling')
+
+ def testLogPathSystemdBackend(self):
+diff --git a/fail2ban/tests/databasetestcase.py b/fail2ban/tests/databasetestcase.py
+index 9a5e9fa1..562461a6 100644
+--- a/fail2ban/tests/databasetestcase.py
++++ b/fail2ban/tests/databasetestcase.py
+@@ -67,7 +67,7 @@ class DatabaseTest(LogCaptureTestCase):
+
+ @property
+ def db(self):
+- if isinstance(self._db, basestring) and self._db == ':auto-create-in-memory:':
++ if isinstance(self._db, str) and self._db == ':auto-create-in-memory:':
+ self._db = getFail2BanDb(self.dbFilename)
+ return self._db
+ @db.setter
+@@ -159,7 +159,7 @@ class DatabaseTest(LogCaptureTestCase):
+ self.db = Fail2BanDb(self.dbFilename)
+ self.assertEqual(self.db.getJailNames(), set(['DummyJail #29162448 with 0 tickets']))
+ self.assertEqual(self.db.getLogPaths(), set(['/tmp/Fail2BanDb_pUlZJh.log']))
+- ticket = FailTicket("127.0.0.1", 1388009242.26, [u"abc\n"])
++ ticket = FailTicket("127.0.0.1", 1388009242.26, ["abc\n"])
+ self.assertEqual(self.db.getBans()[0], ticket)
+
+ self.assertEqual(self.db.updateDb(Fail2BanDb.__version__), Fail2BanDb.__version__)
+@@ -185,9 +185,9 @@ class DatabaseTest(LogCaptureTestCase):
+ self.assertEqual(len(bans), 2)
+ # compare first ticket completely:
+ ticket = FailTicket("1.2.3.7", 1417595494, [
+- u'Dec 3 09:31:08 f2btest test:auth[27658]: pam_unix(test:auth): authentication failure; logname= uid=0 euid=0 tty=test ruser= rhost=1.2.3.7',
+- u'Dec 3 09:31:32 f2btest test:auth[27671]: pam_unix(test:auth): authentication failure; logname= uid=0 euid=0 tty=test ruser= rhost=1.2.3.7',
+- u'Dec 3 09:31:34 f2btest test:auth[27673]: pam_unix(test:auth): authentication failure; logname= uid=0 euid=0 tty=test ruser= rhost=1.2.3.7'
++ 'Dec 3 09:31:08 f2btest test:auth[27658]: pam_unix(test:auth): authentication failure; logname= uid=0 euid=0 tty=test ruser= rhost=1.2.3.7',
++ 'Dec 3 09:31:32 f2btest test:auth[27671]: pam_unix(test:auth): authentication failure; logname= uid=0 euid=0 tty=test ruser= rhost=1.2.3.7',
++ 'Dec 3 09:31:34 f2btest test:auth[27673]: pam_unix(test:auth): authentication failure; logname= uid=0 euid=0 tty=test ruser= rhost=1.2.3.7'
+ ])
+ ticket.setAttempt(3)
+ self.assertEqual(bans[0], ticket)
+@@ -286,11 +286,11 @@ class DatabaseTest(LogCaptureTestCase):
+ # invalid + valid, invalid + valid unicode, invalid + valid dual converted (like in filter:readline by fallback) ...
+ tickets = [
+ FailTicket("127.0.0.1", 0, ['user "test"', 'user "\xd1\xe2\xe5\xf2\xe0"', 'user "\xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f"']),
+- FailTicket("127.0.0.2", 0, ['user "test"', u'user "\xd1\xe2\xe5\xf2\xe0"', u'user "\xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f"']),
++ FailTicket("127.0.0.2", 0, ['user "test"', 'user "\xd1\xe2\xe5\xf2\xe0"', 'user "\xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f"']),
+ FailTicket("127.0.0.3", 0, ['user "test"', b'user "\xd1\xe2\xe5\xf2\xe0"', b'user "\xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f"']),
+- FailTicket("127.0.0.4", 0, ['user "test"', 'user "\xd1\xe2\xe5\xf2\xe0"', u'user "\xe4\xf6\xfc\xdf"']),
++ FailTicket("127.0.0.4", 0, ['user "test"', 'user "\xd1\xe2\xe5\xf2\xe0"', 'user "\xe4\xf6\xfc\xdf"']),
+ FailTicket("127.0.0.5", 0, ['user "test"', 'unterminated \xcf']),
+- FailTicket("127.0.0.6", 0, ['user "test"', u'unterminated \xcf']),
++ FailTicket("127.0.0.6", 0, ['user "test"', 'unterminated \xcf']),
+ FailTicket("127.0.0.7", 0, ['user "test"', b'unterminated \xcf'])
+ ]
+ for ticket in tickets:
+diff --git a/fail2ban/tests/datedetectortestcase.py b/fail2ban/tests/datedetectortestcase.py
+index 458f76ef..49ada60d 100644
+--- a/fail2ban/tests/datedetectortestcase.py
++++ b/fail2ban/tests/datedetectortestcase.py
+@@ -279,7 +279,7 @@ class DateDetectorTest(LogCaptureTestCase):
+ self.assertEqual(logTime, mu)
+ self.assertEqual(logMatch.group(1), '2012/10/11 02:37:17')
+ # confuse it with year being at the end
+- for i in xrange(10):
++ for i in range(10):
+ ( logTime, logMatch ) = self.datedetector.getTime('11/10/2012 02:37:17 [error] 18434#0')
+ self.assertEqual(logTime, mu)
+ self.assertEqual(logMatch.group(1), '11/10/2012 02:37:17')
+@@ -505,7 +505,7 @@ class CustomDateFormatsTest(unittest.TestCase):
+ date = dd.getTime(line)
+ if matched:
+ self.assertTrue(date)
+- if isinstance(matched, basestring):
++ if isinstance(matched, str):
+ self.assertEqual(matched, date[1].group(1))
+ else:
+ self.assertEqual(matched, date[0])
+@@ -537,7 +537,7 @@ class CustomDateFormatsTest(unittest.TestCase):
+ date = dd.getTime(line)
+ if matched:
+ self.assertTrue(date)
+- if isinstance(matched, basestring): # pragma: no cover
++ if isinstance(matched, str): # pragma: no cover
+ self.assertEqual(matched, date[1].group(1))
+ else:
+ self.assertEqual(matched, date[0])
+diff --git a/fail2ban/tests/fail2banclienttestcase.py b/fail2ban/tests/fail2banclienttestcase.py
+index 95f73ed3..bba354fa 100644
+--- a/fail2ban/tests/fail2banclienttestcase.py
++++ b/fail2ban/tests/fail2banclienttestcase.py
+@@ -367,10 +367,10 @@ def with_foreground_server_thread(startextra={}):
+ # several commands to server in body of decorated function:
+ return f(self, tmp, startparams, *args, **kwargs)
+ except Exception as e: # pragma: no cover
+- print('=== Catch an exception: %s' % e)
++ print(('=== Catch an exception: %s' % e))
+ log = self.getLog()
+ if log:
+- print('=== Error of server, log: ===\n%s===' % log)
++ print(('=== Error of server, log: ===\n%s===' % log))
+ self.pruneLog()
+ raise
+ finally:
+@@ -440,7 +440,7 @@ class Fail2banClientServerBase(LogCaptureTestCase):
+ )
+ except: # pragma: no cover
+ if _inherited_log(startparams):
+- print('=== Error by wait fot server, log: ===\n%s===' % self.getLog())
++ print(('=== Error by wait fot server, log: ===\n%s===' % self.getLog()))
+ self.pruneLog()
+ log = pjoin(tmp, "f2b.log")
+ if isfile(log):
+@@ -1610,6 +1610,6 @@ class Fail2banServerTest(Fail2banClientServerBase):
+ self.stopAndWaitForServerEnd(SUCCESS)
+
+ def testServerStartStop(self):
+- for i in xrange(2000):
++ for i in range(2000):
+ self._testServerStartStop()
+
+diff --git a/fail2ban/tests/failmanagertestcase.py b/fail2ban/tests/failmanagertestcase.py
+index a5425286..2a94cc82 100644
+--- a/fail2ban/tests/failmanagertestcase.py
++++ b/fail2ban/tests/failmanagertestcase.py
+@@ -45,11 +45,11 @@ class AddFailure(unittest.TestCase):
+ super(AddFailure, self).tearDown()
+
+ def _addDefItems(self):
+- self.__items = [[u'193.168.0.128', 1167605999.0],
+- [u'193.168.0.128', 1167605999.0],
+- [u'193.168.0.128', 1167605999.0],
+- [u'193.168.0.128', 1167605999.0],
+- [u'193.168.0.128', 1167605999.0],
++ self.__items = [['193.168.0.128', 1167605999.0],
++ ['193.168.0.128', 1167605999.0],
++ ['193.168.0.128', 1167605999.0],
++ ['193.168.0.128', 1167605999.0],
++ ['193.168.0.128', 1167605999.0],
+ ['87.142.124.10', 1167605999.0],
+ ['87.142.124.10', 1167605999.0],
+ ['87.142.124.10', 1167605999.0],
+diff --git a/fail2ban/tests/files/config/apache-auth/digest.py b/fail2ban/tests/files/config/apache-auth/digest.py
+index 03588594..e2297ab3 100755
+--- a/fail2ban/tests/files/config/apache-auth/digest.py
++++ b/fail2ban/tests/files/config/apache-auth/digest.py
+@@ -41,7 +41,7 @@ def auth(v):
+ response="%s"
+ """ % ( username, algorithm, realm, url, nonce, qop, response )
+ # opaque="%s",
+- print(p.method, p.url, p.headers)
++ print((p.method, p.url, p.headers))
+ s = requests.Session()
+ return s.send(p)
+
+@@ -76,18 +76,18 @@ r = auth(v)
+
+ # [Sun Jul 28 21:41:20 2013] [error] [client 127.0.0.1] Digest: unknown algorithm `super funky chicken' received: /digest/
+
+-print(r.status_code,r.headers, r.text)
++print((r.status_code,r.headers, r.text))
+ v['algorithm'] = algorithm
+
+
+ r = auth(v)
+-print(r.status_code,r.headers, r.text)
++print((r.status_code,r.headers, r.text))
+
+ nonce = v['nonce']
+ v['nonce']=v['nonce'][5:-5]
+
+ r = auth(v)
+-print(r.status_code,r.headers, r.text)
++print((r.status_code,r.headers, r.text))
+
+ # [Sun Jul 28 21:05:31.178340 2013] [auth_digest:error] [pid 24224:tid 139895539455744] [client 127.0.0.1:56906] AH01793: invalid qop `auth' received: /digest/qop_none/
+
+@@ -95,7 +95,7 @@ print(r.status_code,r.headers, r.text)
+ v['nonce']=nonce[0:11] + 'ZZZ' + nonce[14:]
+
+ r = auth(v)
+-print(r.status_code,r.headers, r.text)
++print((r.status_code,r.headers, r.text))
+
+ #[Sun Jul 28 21:18:11.769228 2013] [auth_digest:error] [pid 24752:tid 139895505884928] [client 127.0.0.1:56964] AH01776: invalid nonce b9YAiJDiBAZZZ1b1abe02d20063ea3b16b544ea1b0d981c1bafe received - hash is not d42d824dee7aaf50c3ba0a7c6290bd453e3dd35b
+
+@@ -107,7 +107,7 @@ import time
+ time.sleep(1)
+
+ r = auth(v)
+-print(r.status_code,r.headers, r.text)
++print((r.status_code,r.headers, r.text))
+
+ # Obtained by putting the following code in modules/aaa/mod_auth_digest.c
+ # in the function initialize_secret
+@@ -137,7 +137,7 @@ s = sha.sha(apachesecret)
+
+ v=preauth()
+
+-print(v['nonce'])
++print((v['nonce']))
+ realm = v['Digest realm'][1:-1]
+
+ (t,) = struct.unpack('l',base64.b64decode(v['nonce'][1:13]))
+@@ -156,13 +156,13 @@ print(v)
+
+ r = auth(v)
+ #[Mon Jul 29 02:12:55.539813 2013] [auth_digest:error] [pid 9647:tid 139895522670336] [client 127.0.0.1:58474] AH01777: invalid nonce 59QJppTiBAA=b08983fd166ade9840407df1b0f75b9e6e07d88d received - user attempted time travel
+-print(r.status_code,r.headers, r.text)
++print((r.status_code,r.headers, r.text))
+
+ url='/digest_onetime/'
+ v=preauth()
+
+ # Need opaque header handling in auth
+ r = auth(v)
+-print(r.status_code,r.headers, r.text)
++print((r.status_code,r.headers, r.text))
+ r = auth(v)
+-print(r.status_code,r.headers, r.text)
++print((r.status_code,r.headers, r.text))
+diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py
+index 35785a58..8eeb6902 100644
+--- a/fail2ban/tests/filtertestcase.py
++++ b/fail2ban/tests/filtertestcase.py
+@@ -22,7 +22,7 @@
+ __copyright__ = "Copyright (c) 2004 Cyril Jaquier; 2012 Yaroslav Halchenko"
+ __license__ = "GPL"
+
+-from __builtin__ import open as fopen
++from builtins import open as fopen
+ import unittest
+ import os
+ import re
+@@ -204,7 +204,7 @@ def _copy_lines_between_files(in_, fout, n=None, skip=0, mode='a', terminal_line
+ else:
+ fin = in_
+ # Skip
+- for i in xrange(skip):
++ for i in range(skip):
+ fin.readline()
+ # Read
+ i = 0
+@@ -244,7 +244,7 @@ def _copy_lines_to_journal(in_, fields={},n=None, skip=0, terminal_line=""): # p
+ # Required for filtering
+ fields.update(TEST_JOURNAL_FIELDS)
+ # Skip
+- for i in xrange(skip):
++ for i in range(skip):
+ fin.readline()
+ # Read/Write
+ i = 0
+@@ -306,18 +306,18 @@ class BasicFilter(unittest.TestCase):
+ def testTest_tm(self):
+ unittest.F2B.SkipIfFast()
+ ## test function "_tm" works correct (returns the same as slow strftime):
+- for i in xrange(1417512352, (1417512352 // 3600 + 3) * 3600):
++ for i in range(1417512352, (1417512352 // 3600 + 3) * 3600):
+ tm = MyTime.time2str(i)
+ if _tm(i) != tm: # pragma: no cover - never reachable
+ self.assertEqual((_tm(i), i), (tm, i))
+
+ def testWrongCharInTupleLine(self):
+ ## line tuple has different types (ascii after ascii / unicode):
+- for a1 in ('', u'', b''):
+- for a2 in ('2016-09-05T20:18:56', u'2016-09-05T20:18:56', b'2016-09-05T20:18:56'):
++ for a1 in ('', '', b''):
++ for a2 in ('2016-09-05T20:18:56', '2016-09-05T20:18:56', b'2016-09-05T20:18:56'):
+ for a3 in (
+ 'Fail for "g\xc3\xb6ran" from 192.0.2.1',
+- u'Fail for "g\xc3\xb6ran" from 192.0.2.1',
++ 'Fail for "g\xc3\xb6ran" from 192.0.2.1',
+ b'Fail for "g\xc3\xb6ran" from 192.0.2.1'
+ ):
+ # join should work if all arguments have the same type:
+@@ -435,7 +435,7 @@ class IgnoreIP(LogCaptureTestCase):
+
+ def testAddAttempt(self):
+ self.filter.setMaxRetry(3)
+- for i in xrange(1, 1+3):
++ for i in range(1, 1+3):
+ self.filter.addAttempt('192.0.2.1')
+ self.assertLogged('Attempt 192.0.2.1', '192.0.2.1:%d' % i, all=True, wait=True)
+ self.jail.actions._Actions__checkBan()
+@@ -472,7 +472,7 @@ class IgnoreIP(LogCaptureTestCase):
+ # like both test-cases above, just cached (so once per key)...
+ self.filter.ignoreCache = {"key":"<ip>"}
+ self.filter.ignoreCommand = 'if [ "<ip>" = "10.0.0.1" ]; then exit 0; fi; exit 1'
+- for i in xrange(5):
++ for i in range(5):
+ self.pruneLog()
+ self.assertTrue(self.filter.inIgnoreIPList("10.0.0.1"))
+ self.assertFalse(self.filter.inIgnoreIPList("10.0.0.0"))
+@@ -483,7 +483,7 @@ class IgnoreIP(LogCaptureTestCase):
+ # by host of IP:
+ self.filter.ignoreCache = {"key":"<ip-host>"}
+ self.filter.ignoreCommand = 'if [ "<ip-host>" = "test-host" ]; then exit 0; fi; exit 1'
+- for i in xrange(5):
++ for i in range(5):
+ self.pruneLog()
+ self.assertTrue(self.filter.inIgnoreIPList(FailTicket("2001:db8::1")))
+ self.assertFalse(self.filter.inIgnoreIPList(FailTicket("2001:db8::ffff")))
+@@ -495,7 +495,7 @@ class IgnoreIP(LogCaptureTestCase):
+ self.filter.ignoreCache = {"key":"<F-USER>", "max-count":"10", "max-time":"1h"}
+ self.assertEqual(self.filter.ignoreCache, ["<F-USER>", 10, 60*60])
+ self.filter.ignoreCommand = 'if [ "<F-USER>" = "tester" ]; then exit 0; fi; exit 1'
+- for i in xrange(5):
++ for i in range(5):
+ self.pruneLog()
+ self.assertTrue(self.filter.inIgnoreIPList(FailTicket("tester", data={'user': 'tester'})))
+ self.assertFalse(self.filter.inIgnoreIPList(FailTicket("root", data={'user': 'root'})))
+@@ -644,7 +644,7 @@ class LogFileFilterPoll(unittest.TestCase):
+ fc = FileContainer(fname, self.filter.getLogEncoding())
+ fc.open()
+ # no time - nothing should be found :
+- for i in xrange(10):
++ for i in range(10):
+ f.write("[sshd] error: PAM: failure len 1\n")
+ f.flush()
+ fc.setPos(0); self.filter.seekToTime(fc, time)
+@@ -718,14 +718,14 @@ class LogFileFilterPoll(unittest.TestCase):
+ # variable length of file (ca 45K or 450K before and hereafter):
+ # write lines with smaller as search time:
+ t = time - count - 1
+- for i in xrange(count):
++ for i in range(count):
+ f.write("%s [sshd] error: PAM: failure\n" % _tm(t))
+ t += 1
+ f.flush()
+ fc.setPos(0); self.filter.seekToTime(fc, time)
+ self.assertEqual(fc.getPos(), 47*count)
+ # write lines with exact search time:
+- for i in xrange(10):
++ for i in range(10):
+ f.write("%s [sshd] error: PAM: failure\n" % _tm(time))
+ f.flush()
+ fc.setPos(0); self.filter.seekToTime(fc, time)
+@@ -734,8 +734,8 @@ class LogFileFilterPoll(unittest.TestCase):
+ self.assertEqual(fc.getPos(), 47*count)
+ # write lines with greater as search time:
+ t = time+1
+- for i in xrange(count//500):
+- for j in xrange(500):
++ for i in range(count//500):
++ for j in range(500):
+ f.write("%s [sshd] error: PAM: failure\n" % _tm(t))
+ t += 1
+ f.flush()
+@@ -1488,10 +1488,10 @@ def get_monitor_failures_journal_testcase(Filter_): # pragma: systemd no cover
+ # Add direct utf, unicode, blob:
+ for l in (
+ "error: PAM: Authentication failure for \xe4\xf6\xfc\xdf from 192.0.2.1",
+- u"error: PAM: Authentication failure for \xe4\xf6\xfc\xdf from 192.0.2.1",
++ "error: PAM: Authentication failure for \xe4\xf6\xfc\xdf from 192.0.2.1",
+ b"error: PAM: Authentication failure for \xe4\xf6\xfc\xdf from 192.0.2.1".decode('utf-8', 'replace'),
+ "error: PAM: Authentication failure for \xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f from 192.0.2.2",
+- u"error: PAM: Authentication failure for \xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f from 192.0.2.2",
++ "error: PAM: Authentication failure for \xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f from 192.0.2.2",
+ b"error: PAM: Authentication failure for \xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f from 192.0.2.2".decode('utf-8', 'replace')
+ ):
+ fields = self.journal_fields
+@@ -1520,7 +1520,7 @@ class GetFailures(LogCaptureTestCase):
+
+ # so that they could be reused by other tests
+ FAILURES_01 = ('193.168.0.128', 3, 1124013599.0,
+- [u'Aug 14 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 193.168.0.128']*3)
++ ['Aug 14 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 193.168.0.128']*3)
+
+ def setUp(self):
+ """Call before every test case."""
+@@ -1595,8 +1595,8 @@ class GetFailures(LogCaptureTestCase):
+
+ def testGetFailures02(self):
+ output = ('141.3.81.106', 4, 1124013539.0,
+- [u'Aug 14 11:%d:59 i60p295 sshd[12365]: Failed publickey for roehl from ::ffff:141.3.81.106 port 51332 ssh2'
+- % m for m in 53, 54, 57, 58])
++ ['Aug 14 11:%d:59 i60p295 sshd[12365]: Failed publickey for roehl from ::ffff:141.3.81.106 port 51332 ssh2'
++ % m for m in (53, 54, 57, 58)])
+
+ self.filter.addLogPath(GetFailures.FILENAME_02, autoSeek=0)
+ self.filter.addFailRegex(r"Failed .* from <HOST>")
+@@ -1691,17 +1691,17 @@ class GetFailures(LogCaptureTestCase):
+ # We should still catch failures with usedns = no ;-)
+ output_yes = (
+ ('93.184.216.34', 2, 1124013539.0,
+- [u'Aug 14 11:54:59 i60p295 sshd[12365]: Failed publickey for roehl from example.com port 51332 ssh2',
+- u'Aug 14 11:58:59 i60p295 sshd[12365]: Failed publickey for roehl from ::ffff:93.184.216.34 port 51332 ssh2']
++ ['Aug 14 11:54:59 i60p295 sshd[12365]: Failed publickey for roehl from example.com port 51332 ssh2',
++ 'Aug 14 11:58:59 i60p295 sshd[12365]: Failed publickey for roehl from ::ffff:93.184.216.34 port 51332 ssh2']
+ ),
+ ('2606:2800:220:1:248:1893:25c8:1946', 1, 1124013299.0,
+- [u'Aug 14 11:54:59 i60p295 sshd[12365]: Failed publickey for roehl from example.com port 51332 ssh2']
++ ['Aug 14 11:54:59 i60p295 sshd[12365]: Failed publickey for roehl from example.com port 51332 ssh2']
+ ),
+ )
+
+ output_no = (
+ ('93.184.216.34', 1, 1124013539.0,
+- [u'Aug 14 11:58:59 i60p295 sshd[12365]: Failed publickey for roehl from ::ffff:93.184.216.34 port 51332 ssh2']
++ ['Aug 14 11:58:59 i60p295 sshd[12365]: Failed publickey for roehl from ::ffff:93.184.216.34 port 51332 ssh2']
+ )
+ )
+
+@@ -1807,9 +1807,9 @@ class DNSUtilsTests(unittest.TestCase):
+ self.assertTrue(c.get('a') is None)
+ self.assertEqual(c.get('a', 'test'), 'test')
+ # exact 5 elements :
+- for i in xrange(5):
++ for i in range(5):
+ c.set(i, i)
+- for i in xrange(5):
++ for i in range(5):
+ self.assertEqual(c.get(i), i)
+ # remove unavailable key:
+ c.unset('a'); c.unset('a')
+@@ -1817,30 +1817,30 @@ class DNSUtilsTests(unittest.TestCase):
+ def testCacheMaxSize(self):
+ c = Utils.Cache(maxCount=5, maxTime=60)
+ # exact 5 elements :
+- for i in xrange(5):
++ for i in range(5):
+ c.set(i, i)
+- self.assertEqual([c.get(i) for i in xrange(5)], [i for i in xrange(5)])
+- self.assertNotIn(-1, (c.get(i, -1) for i in xrange(5)))
++ self.assertEqual([c.get(i) for i in range(5)], [i for i in range(5)])
++ self.assertNotIn(-1, (c.get(i, -1) for i in range(5)))
+ # add one - too many:
+ c.set(10, i)
+ # one element should be removed :
+- self.assertIn(-1, (c.get(i, -1) for i in xrange(5)))
++ self.assertIn(-1, (c.get(i, -1) for i in range(5)))
+ # test max size (not expired):
+- for i in xrange(10):
++ for i in range(10):
+ c.set(i, 1)
+ self.assertEqual(len(c), 5)
+
+ def testCacheMaxTime(self):
+ # test max time (expired, timeout reached) :
+ c = Utils.Cache(maxCount=5, maxTime=0.0005)
+- for i in xrange(10):
++ for i in range(10):
+ c.set(i, 1)
+ st = time.time()
+ self.assertTrue(Utils.wait_for(lambda: time.time() >= st + 0.0005, 1))
+ # we have still 5 elements (or fewer if too slow test mashine):
+ self.assertTrue(len(c) <= 5)
+ # but all that are expiered also:
+- for i in xrange(10):
++ for i in range(10):
+ self.assertTrue(c.get(i) is None)
+ # here the whole cache should be empty:
+ self.assertEqual(len(c), 0)
+@@ -1861,7 +1861,7 @@ class DNSUtilsTests(unittest.TestCase):
+ c = count
+ while c:
+ c -= 1
+- s = xrange(0, 256, 1) if forw else xrange(255, -1, -1)
++ s = range(0, 256, 1) if forw else range(255, -1, -1)
+ if random: shuffle([i for i in s])
+ for i in s:
+ IPAddr('192.0.2.'+str(i), IPAddr.FAM_IPv4)
+@@ -1983,15 +1983,15 @@ class DNSUtilsNetworkTests(unittest.TestCase):
+
+ def testAddr2bin(self):
+ res = IPAddr('10.0.0.0')
+- self.assertEqual(res.addr, 167772160L)
++ self.assertEqual(res.addr, 167772160)
+ res = IPAddr('10.0.0.0', cidr=None)
+- self.assertEqual(res.addr, 167772160L)
+- res = IPAddr('10.0.0.0', cidr=32L)
+- self.assertEqual(res.addr, 167772160L)
+- res = IPAddr('10.0.0.1', cidr=32L)
+- self.assertEqual(res.addr, 167772161L)
+- res = IPAddr('10.0.0.1', cidr=31L)
+- self.assertEqual(res.addr, 167772160L)
++ self.assertEqual(res.addr, 167772160)
++ res = IPAddr('10.0.0.0', cidr=32)
++ self.assertEqual(res.addr, 167772160)
++ res = IPAddr('10.0.0.1', cidr=32)
++ self.assertEqual(res.addr, 167772161)
++ res = IPAddr('10.0.0.1', cidr=31)
++ self.assertEqual(res.addr, 167772160)
+
+ self.assertEqual(IPAddr('10.0.0.0').hexdump, '0a000000')
+ self.assertEqual(IPAddr('1::2').hexdump, '00010000000000000000000000000002')
+@@ -2067,9 +2067,9 @@ class DNSUtilsNetworkTests(unittest.TestCase):
+ '93.184.216.34': 'ip4-test',
+ '2606:2800:220:1:248:1893:25c8:1946': 'ip6-test'
+ }
+- d2 = dict([(IPAddr(k), v) for k, v in d.iteritems()])
+- self.assertTrue(isinstance(d.keys()[0], basestring))
+- self.assertTrue(isinstance(d2.keys()[0], IPAddr))
++ d2 = dict([(IPAddr(k), v) for k, v in d.items()])
++ self.assertTrue(isinstance(list(d.keys())[0], str))
++ self.assertTrue(isinstance(list(d2.keys())[0], IPAddr))
+ self.assertEqual(d.get(ip4[2], ''), 'ip4-test')
+ self.assertEqual(d.get(ip6[2], ''), 'ip6-test')
+ self.assertEqual(d2.get(str(ip4[2]), ''), 'ip4-test')
+diff --git a/fail2ban/tests/misctestcase.py b/fail2ban/tests/misctestcase.py
+index 9b986f53..94f7a8de 100644
+--- a/fail2ban/tests/misctestcase.py
++++ b/fail2ban/tests/misctestcase.py
+@@ -29,9 +29,9 @@ import tempfile
+ import shutil
+ import fnmatch
+ from glob import glob
+-from StringIO import StringIO
++from io import StringIO
+
+-from utils import LogCaptureTestCase, logSys as DefLogSys
++from .utils import LogCaptureTestCase, logSys as DefLogSys
+
+ from ..helpers import formatExceptionInfo, mbasename, TraceBack, FormatterWithTraceBack, getLogger, \
+ splitwords, uni_decode, uni_string
+@@ -67,7 +67,7 @@ class HelpersTest(unittest.TestCase):
+ self.assertEqual(splitwords(' 1\n 2'), ['1', '2'])
+ self.assertEqual(splitwords(' 1\n 2, 3'), ['1', '2', '3'])
+ # string as unicode:
+- self.assertEqual(splitwords(u' 1\n 2, 3'), ['1', '2', '3'])
++ self.assertEqual(splitwords(' 1\n 2, 3'), ['1', '2', '3'])
+
+
+ if sys.version_info >= (2,7):
+@@ -197,11 +197,11 @@ class TestsUtilsTest(LogCaptureTestCase):
+
+ def testUniConverters(self):
+ self.assertRaises(Exception, uni_decode,
+- (b'test' if sys.version_info >= (3,) else u'test'), 'f2b-test::non-existing-encoding')
+- uni_decode((b'test\xcf' if sys.version_info >= (3,) else u'test\xcf'))
++ (b'test' if sys.version_info >= (3,) else 'test'), 'f2b-test::non-existing-encoding')
++ uni_decode((b'test\xcf' if sys.version_info >= (3,) else 'test\xcf'))
+ uni_string(b'test\xcf')
+ uni_string('test\xcf')
+- uni_string(u'test\xcf')
++ uni_string('test\xcf')
+
+ def testSafeLogging(self):
+ # logging should be exception-safe, to avoid possible errors (concat, str. conversion, representation failures, etc)
+@@ -213,7 +213,7 @@ class TestsUtilsTest(LogCaptureTestCase):
+ if self.err:
+ raise Exception('no represenation for test!')
+ else:
+- return u'conv-error (\xf2\xf0\xe5\xf2\xe8\xe9), unterminated utf \xcf'
++ return 'conv-error (\xf2\xf0\xe5\xf2\xe8\xe9), unterminated utf \xcf'
+ test = Test()
+ logSys.log(logging.NOTICE, "test 1a: %r", test)
+ self.assertLogged("Traceback", "no represenation for test!")
+@@ -261,7 +261,7 @@ class TestsUtilsTest(LogCaptureTestCase):
+ func_raise()
+
+ try:
+- print deep_function(3)
++ print(deep_function(3))
+ except ValueError:
+ s = tb()
+
+@@ -278,7 +278,7 @@ class TestsUtilsTest(LogCaptureTestCase):
+ self.assertIn(':', s)
+
+ def _testAssertionErrorRE(self, regexp, fun, *args, **kwargs):
+- self.assertRaisesRegexp(AssertionError, regexp, fun, *args, **kwargs)
++ self.assertRaisesRegex(AssertionError, regexp, fun, *args, **kwargs)
+
+ def testExtendedAssertRaisesRE(self):
+ ## test _testAssertionErrorRE several fail cases:
+@@ -316,13 +316,13 @@ class TestsUtilsTest(LogCaptureTestCase):
+ self._testAssertionErrorRE(r"'a' unexpectedly found in 'cba'",
+ self.assertNotIn, 'a', 'cba')
+ self._testAssertionErrorRE(r"1 unexpectedly found in \[0, 1, 2\]",
+- self.assertNotIn, 1, xrange(3))
++ self.assertNotIn, 1, range(3))
+ self._testAssertionErrorRE(r"'A' unexpectedly found in \['C', 'A'\]",
+ self.assertNotIn, 'A', (c.upper() for c in 'cba' if c != 'b'))
+ self._testAssertionErrorRE(r"'a' was not found in 'xyz'",
+ self.assertIn, 'a', 'xyz')
+ self._testAssertionErrorRE(r"5 was not found in \[0, 1, 2\]",
+- self.assertIn, 5, xrange(3))
++ self.assertIn, 5, range(3))
+ self._testAssertionErrorRE(r"'A' was not found in \['C', 'B'\]",
+ self.assertIn, 'A', (c.upper() for c in 'cba' if c != 'a'))
+ ## assertLogged, assertNotLogged positive case:
+diff --git a/fail2ban/tests/observertestcase.py b/fail2ban/tests/observertestcase.py
+index 8e944454..ed520286 100644
+--- a/fail2ban/tests/observertestcase.py
++++ b/fail2ban/tests/observertestcase.py
+@@ -69,7 +69,7 @@ class BanTimeIncr(LogCaptureTestCase):
+ a.setBanTimeExtra('multipliers', multipliers)
+ # test algorithm and max time 24 hours :
+ self.assertEqual(
+- [a.calcBanTime(600, i) for i in xrange(1, 11)],
++ [a.calcBanTime(600, i) for i in range(1, 11)],
+ [1200, 2400, 4800, 9600, 19200, 38400, 76800, 86400, 86400, 86400]
+ )
+ # with extra large max time (30 days):
+@@ -81,38 +81,38 @@ class BanTimeIncr(LogCaptureTestCase):
+ if multcnt < 11:
+ arr = arr[0:multcnt-1] + ([arr[multcnt-2]] * (11-multcnt))
+ self.assertEqual(
+- [a.calcBanTime(600, i) for i in xrange(1, 11)],
++ [a.calcBanTime(600, i) for i in range(1, 11)],
+ arr
+ )
+ a.setBanTimeExtra('maxtime', '1d')
+ # change factor :
+ a.setBanTimeExtra('factor', '2');
+ self.assertEqual(
+- [a.calcBanTime(600, i) for i in xrange(1, 11)],
++ [a.calcBanTime(600, i) for i in range(1, 11)],
+ [2400, 4800, 9600, 19200, 38400, 76800, 86400, 86400, 86400, 86400]
+ )
+ # factor is float :
+ a.setBanTimeExtra('factor', '1.33');
+ self.assertEqual(
+- [int(a.calcBanTime(600, i)) for i in xrange(1, 11)],
++ [int(a.calcBanTime(600, i)) for i in range(1, 11)],
+ [1596, 3192, 6384, 12768, 25536, 51072, 86400, 86400, 86400, 86400]
+ )
+ a.setBanTimeExtra('factor', None);
+ # change max time :
+ a.setBanTimeExtra('maxtime', '12h')
+ self.assertEqual(
+- [a.calcBanTime(600, i) for i in xrange(1, 11)],
++ [a.calcBanTime(600, i) for i in range(1, 11)],
+ [1200, 2400, 4800, 9600, 19200, 38400, 43200, 43200, 43200, 43200]
+ )
+ a.setBanTimeExtra('maxtime', '24h')
+ ## test randomization - not possibe all 10 times we have random = 0:
+ a.setBanTimeExtra('rndtime', '5m')
+ self.assertTrue(
+- False in [1200 in [a.calcBanTime(600, 1) for i in xrange(10)] for c in xrange(10)]
++ False in [1200 in [a.calcBanTime(600, 1) for i in range(10)] for c in range(10)]
+ )
+ a.setBanTimeExtra('rndtime', None)
+ self.assertFalse(
+- False in [1200 in [a.calcBanTime(600, 1) for i in xrange(10)] for c in xrange(10)]
++ False in [1200 in [a.calcBanTime(600, 1) for i in range(10)] for c in range(10)]
+ )
+ # restore default:
+ a.setBanTimeExtra('multipliers', None)
+@@ -124,7 +124,7 @@ class BanTimeIncr(LogCaptureTestCase):
+ # this multipliers has the same values as default formula, we test stop growing after count 9:
+ self.testDefault('1 2 4 8 16 32 64 128 256')
+ # this multipliers has exactly the same values as default formula, test endless growing (stops by count 31 only):
+- self.testDefault(' '.join([str(1<<i) for i in xrange(31)]))
++ self.testDefault(' '.join([str(1<<i) for i in range(31)]))
+
+ def testFormula(self):
+ a = self.__jail;
+@@ -136,38 +136,38 @@ class BanTimeIncr(LogCaptureTestCase):
+ a.setBanTimeExtra('multipliers', None)
+ # test algorithm and max time 24 hours :
+ self.assertEqual(
+- [int(a.calcBanTime(600, i)) for i in xrange(1, 11)],
++ [int(a.calcBanTime(600, i)) for i in range(1, 11)],
+ [1200, 2400, 4800, 9600, 19200, 38400, 76800, 86400, 86400, 86400]
+ )
+ # with extra large max time (30 days):
+ a.setBanTimeExtra('maxtime', '30d')
+ self.assertEqual(
+- [int(a.calcBanTime(600, i)) for i in xrange(1, 11)],
++ [int(a.calcBanTime(600, i)) for i in range(1, 11)],
+ [1200, 2400, 4800, 9600, 19200, 38400, 76800, 153601, 307203, 614407]
+ )
+ a.setBanTimeExtra('maxtime', '24h')
+ # change factor :
+ a.setBanTimeExtra('factor', '1');
+ self.assertEqual(
+- [int(a.calcBanTime(600, i)) for i in xrange(1, 11)],
++ [int(a.calcBanTime(600, i)) for i in range(1, 11)],
+ [1630, 4433, 12051, 32758, 86400, 86400, 86400, 86400, 86400, 86400]
+ )
+ a.setBanTimeExtra('factor', '2.0 / 2.885385')
+ # change max time :
+ a.setBanTimeExtra('maxtime', '12h')
+ self.assertEqual(
+- [int(a.calcBanTime(600, i)) for i in xrange(1, 11)],
++ [int(a.calcBanTime(600, i)) for i in range(1, 11)],
+ [1200, 2400, 4800, 9600, 19200, 38400, 43200, 43200, 43200, 43200]
+ )
+ a.setBanTimeExtra('maxtime', '24h')
+ ## test randomization - not possibe all 10 times we have random = 0:
+ a.setBanTimeExtra('rndtime', '5m')
+ self.assertTrue(
+- False in [1200 in [int(a.calcBanTime(600, 1)) for i in xrange(10)] for c in xrange(10)]
++ False in [1200 in [int(a.calcBanTime(600, 1)) for i in range(10)] for c in range(10)]
+ )
+ a.setBanTimeExtra('rndtime', None)
+ self.assertFalse(
+- False in [1200 in [int(a.calcBanTime(600, 1)) for i in xrange(10)] for c in xrange(10)]
++ False in [1200 in [int(a.calcBanTime(600, 1)) for i in range(10)] for c in range(10)]
+ )
+ # restore default:
+ a.setBanTimeExtra('factor', None);
+@@ -230,7 +230,7 @@ class BanTimeIncrDB(LogCaptureTestCase):
+ ticket = FailTicket(ip, stime, [])
+ # test ticket not yet found
+ self.assertEqual(
+- [self.incrBanTime(ticket, 10) for i in xrange(3)],
++ [self.incrBanTime(ticket, 10) for i in range(3)],
+ [10, 10, 10]
+ )
+ # add a ticket banned
+@@ -285,7 +285,7 @@ class BanTimeIncrDB(LogCaptureTestCase):
+ )
+ # increase ban multiple times:
+ lastBanTime = 20
+- for i in xrange(10):
++ for i in range(10):
+ ticket.setTime(stime + lastBanTime + 5)
+ banTime = self.incrBanTime(ticket, 10)
+ self.assertEqual(banTime, lastBanTime * 2)
+@@ -481,7 +481,7 @@ class BanTimeIncrDB(LogCaptureTestCase):
+ ticket = FailTicket(ip, stime-120, [])
+ failManager = FailManager()
+ failManager.setMaxRetry(3)
+- for i in xrange(3):
++ for i in range(3):
+ failManager.addFailure(ticket)
+ obs.add('failureFound', failManager, jail, ticket)
+ obs.wait_empty(5)
+diff --git a/fail2ban/tests/samplestestcase.py b/fail2ban/tests/samplestestcase.py
+index 0bbd05f5..479b564a 100644
+--- a/fail2ban/tests/samplestestcase.py
++++ b/fail2ban/tests/samplestestcase.py
+@@ -138,7 +138,7 @@ class FilterSamplesRegex(unittest.TestCase):
+
+ @staticmethod
+ def _filterOptions(opts):
+- return dict((k, v) for k, v in opts.iteritems() if not k.startswith('test.'))
++ return dict((k, v) for k, v in opts.items() if not k.startswith('test.'))
+
+ def testSampleRegexsFactory(name, basedir):
+ def testFilter(self):
+@@ -249,10 +249,10 @@ def testSampleRegexsFactory(name, basedir):
+ self.assertTrue(faildata.get('match', False),
+ "Line matched when shouldn't have")
+ self.assertEqual(len(ret), 1,
+- "Multiple regexs matched %r" % (map(lambda x: x[0], ret)))
++ "Multiple regexs matched %r" % ([x[0] for x in ret]))
+
+ # Verify match captures (at least fid/host) and timestamp as expected
+- for k, v in faildata.iteritems():
++ for k, v in faildata.items():
+ if k not in ("time", "match", "desc", "filter"):
+ fv = fail.get(k, None)
+ if fv is None:
+@@ -294,7 +294,7 @@ def testSampleRegexsFactory(name, basedir):
+ '\n'.join(pprint.pformat(fail).splitlines())))
+
+ # check missing samples for regex using each filter-options combination:
+- for fltName, flt in self._filters.iteritems():
++ for fltName, flt in self._filters.items():
+ flt, regexsUsedIdx = flt
+ regexList = flt.getFailRegex()
+ for failRegexIndex, failRegex in enumerate(regexList):
+diff --git a/fail2ban/tests/servertestcase.py b/fail2ban/tests/servertestcase.py
+index 55e72455..7925ab1e 100644
+--- a/fail2ban/tests/servertestcase.py
++++ b/fail2ban/tests/servertestcase.py
+@@ -124,14 +124,14 @@ class TransmitterBase(LogCaptureTestCase):
+ self.transm.proceed(["get", jail, cmd]), (0, []))
+ for n, value in enumerate(values):
+ ret = self.transm.proceed(["set", jail, cmdAdd, value])
+- self.assertSortedEqual((ret[0], map(str, ret[1])), (0, map(str, values[:n+1])), level=2)
++ self.assertSortedEqual((ret[0], list(map(str, ret[1]))), (0, list(map(str, values[:n+1]))), level=2)
+ ret = self.transm.proceed(["get", jail, cmd])
+- self.assertSortedEqual((ret[0], map(str, ret[1])), (0, map(str, values[:n+1])), level=2)
++ self.assertSortedEqual((ret[0], list(map(str, ret[1]))), (0, list(map(str, values[:n+1]))), level=2)
+ for n, value in enumerate(values):
+ ret = self.transm.proceed(["set", jail, cmdDel, value])
+- self.assertSortedEqual((ret[0], map(str, ret[1])), (0, map(str, values[n+1:])), level=2)
++ self.assertSortedEqual((ret[0], list(map(str, ret[1]))), (0, list(map(str, values[n+1:]))), level=2)
+ ret = self.transm.proceed(["get", jail, cmd])
+- self.assertSortedEqual((ret[0], map(str, ret[1])), (0, map(str, values[n+1:])), level=2)
++ self.assertSortedEqual((ret[0], list(map(str, ret[1]))), (0, list(map(str, values[n+1:]))), level=2)
+
+ def jailAddDelRegexTest(self, cmd, inValues, outValues, jail):
+ cmdAdd = "add" + cmd
+@@ -930,7 +930,7 @@ class TransmitterLogging(TransmitterBase):
+
+ def testLogTarget(self):
+ logTargets = []
+- for _ in xrange(3):
++ for _ in range(3):
+ tmpFile = tempfile.mkstemp("fail2ban", "transmitter")
+ logTargets.append(tmpFile[1])
+ os.close(tmpFile[0])
+@@ -1003,26 +1003,26 @@ class TransmitterLogging(TransmitterBase):
+ self.assertEqual(self.transm.proceed(["flushlogs"]), (0, "rolled over"))
+ l.warning("After flushlogs")
+ with open(fn2,'r') as f:
+- line1 = f.next()
++ line1 = next(f)
+ if line1.find('Changed logging target to') >= 0:
+- line1 = f.next()
++ line1 = next(f)
+ self.assertTrue(line1.endswith("Before file moved\n"))
+- line2 = f.next()
++ line2 = next(f)
+ self.assertTrue(line2.endswith("After file moved\n"))
+ try:
+- n = f.next()
++ n = next(f)
+ if n.find("Command: ['flushlogs']") >=0:
+- self.assertRaises(StopIteration, f.next)
++ self.assertRaises(StopIteration, f.__next__)
+ else:
+ self.fail("Exception StopIteration or Command: ['flushlogs'] expected. Got: %s" % n)
+ except StopIteration:
+ pass # on higher debugging levels this is expected
+ with open(fn,'r') as f:
+- line1 = f.next()
++ line1 = next(f)
+ if line1.find('rollover performed on') >= 0:
+- line1 = f.next()
++ line1 = next(f)
+ self.assertTrue(line1.endswith("After flushlogs\n"))
+- self.assertRaises(StopIteration, f.next)
++ self.assertRaises(StopIteration, f.__next__)
+ f.close()
+ finally:
+ os.remove(fn2)
+@@ -1185,7 +1185,7 @@ class LoggingTests(LogCaptureTestCase):
+ os.remove(f)
+
+
+-from clientreadertestcase import ActionReader, JailsReader, CONFIG_DIR
++from .clientreadertestcase import ActionReader, JailsReader, CONFIG_DIR
+
+ class ServerConfigReaderTests(LogCaptureTestCase):
+
+diff --git a/fail2ban/tests/sockettestcase.py b/fail2ban/tests/sockettestcase.py
+index 69bf8d8b..60f49e57 100644
+--- a/fail2ban/tests/sockettestcase.py
++++ b/fail2ban/tests/sockettestcase.py
+@@ -153,7 +153,7 @@ class Socket(LogCaptureTestCase):
+ org_handler = RequestHandler.found_terminator
+ try:
+ RequestHandler.found_terminator = lambda self: self.close()
+- self.assertRaisesRegexp(RuntimeError, r"socket connection broken",
++ self.assertRaisesRegex(RuntimeError, r"socket connection broken",
+ lambda: client.send(testMessage, timeout=unittest.F2B.maxWaitTime(10)))
+ finally:
+ RequestHandler.found_terminator = org_handler
+diff --git a/fail2ban/tests/utils.py b/fail2ban/tests/utils.py
+index fcfddba7..cb234e0d 100644
+--- a/fail2ban/tests/utils.py
++++ b/fail2ban/tests/utils.py
+@@ -35,7 +35,7 @@ import time
+ import threading
+ import unittest
+
+-from cStringIO import StringIO
++from io import StringIO
+ from functools import wraps
+
+ from ..helpers import getLogger, str2LogLevel, getVerbosityFormat, uni_decode
+@@ -174,8 +174,8 @@ def initProcess(opts):
+
+ # Let know the version
+ if opts.verbosity != 0:
+- print("Fail2ban %s test suite. Python %s. Please wait..." \
+- % (version, str(sys.version).replace('\n', '')))
++ print(("Fail2ban %s test suite. Python %s. Please wait..." \
++ % (version, str(sys.version).replace('\n', ''))))
+
+ return opts;
+
+@@ -322,7 +322,7 @@ def initTests(opts):
+ c = DNSUtils.CACHE_ipToName
+ # increase max count and max time (too many entries, long time testing):
+ c.setOptions(maxCount=10000, maxTime=5*60)
+- for i in xrange(256):
++ for i in range(256):
+ c.set('192.0.2.%s' % i, None)
+ c.set('198.51.100.%s' % i, None)
+ c.set('203.0.113.%s' % i, None)
+@@ -541,8 +541,8 @@ def gatherTests(regexps=None, opts=None):
+ import difflib, pprint
+ if not hasattr(unittest.TestCase, 'assertDictEqual'):
+ def assertDictEqual(self, d1, d2, msg=None):
+- self.assert_(isinstance(d1, dict), 'First argument is not a dictionary')
+- self.assert_(isinstance(d2, dict), 'Second argument is not a dictionary')
++ self.assertTrue(isinstance(d1, dict), 'First argument is not a dictionary')
++ self.assertTrue(isinstance(d2, dict), 'Second argument is not a dictionary')
+ if d1 != d2:
+ standardMsg = '%r != %r' % (d1, d2)
+ diff = ('\n' + '\n'.join(difflib.ndiff(
+@@ -560,7 +560,7 @@ def assertSortedEqual(self, a, b, level=1, nestedOnly=True, key=repr, msg=None):
+ # used to recognize having element as nested dict, list or tuple:
+ def _is_nested(v):
+ if isinstance(v, dict):
+- return any(isinstance(v, (dict, list, tuple)) for v in v.itervalues())
++ return any(isinstance(v, (dict, list, tuple)) for v in v.values())
+ return any(isinstance(v, (dict, list, tuple)) for v in v)
+ # level comparison routine:
+ def _assertSortedEqual(a, b, level, nestedOnly, key):
+@@ -573,7 +573,7 @@ def assertSortedEqual(self, a, b, level=1, nestedOnly=True, key=repr, msg=None):
+ return
+ raise ValueError('%r != %r' % (a, b))
+ if isinstance(a, dict) and isinstance(b, dict): # compare dict's:
+- for k, v1 in a.iteritems():
++ for k, v1 in a.items():
+ v2 = b[k]
+ if isinstance(v1, (dict, list, tuple)) and isinstance(v2, (dict, list, tuple)):
+ _assertSortedEqual(v1, v2, level-1 if level != 0 else 0, nestedOnly, key)
+@@ -608,14 +608,14 @@ if not hasattr(unittest.TestCase, 'assertRaisesRegexp'):
+ self.fail('\"%s\" does not match \"%s\"' % (regexp, e))
+ else:
+ self.fail('%s not raised' % getattr(exccls, '__name__'))
+- unittest.TestCase.assertRaisesRegexp = assertRaisesRegexp
++ unittest.TestCase.assertRaisesRegex = assertRaisesRegexp
+
+ # always custom following methods, because we use atm better version of both (support generators)
+ if True: ## if not hasattr(unittest.TestCase, 'assertIn'):
+ def assertIn(self, a, b, msg=None):
+ bb = b
+ wrap = False
+- if msg is None and hasattr(b, '__iter__') and not isinstance(b, basestring):
++ if msg is None and hasattr(b, '__iter__') and not isinstance(b, str):
+ b, bb = itertools.tee(b)
+ wrap = True
+ if a not in b:
+@@ -626,7 +626,7 @@ if True: ## if not hasattr(unittest.TestCase, 'assertIn'):
+ def assertNotIn(self, a, b, msg=None):
+ bb = b
+ wrap = False
+- if msg is None and hasattr(b, '__iter__') and not isinstance(b, basestring):
++ if msg is None and hasattr(b, '__iter__') and not isinstance(b, str):
+ b, bb = itertools.tee(b)
+ wrap = True
+ if a in b:
+diff --git a/setup.py b/setup.py
+deleted file mode 100755
+index ce1eedf6..00000000
+--- a/setup.py
++++ /dev/null
+@@ -1,326 +0,0 @@
+-#!/usr/bin/env python
+-# emacs: -*- mode: python; py-indent-offset: 4; indent-tabs-mode: t -*-
+-# vi: set ft=python sts=4 ts=4 sw=4 noet :
+-
+-# This file is part of Fail2Ban.
+-#
+-# Fail2Ban is free software; you can redistribute it and/or modify
+-# it under the terms of the GNU General Public License as published by
+-# the Free Software Foundation; either version 2 of the License, or
+-# (at your option) any later version.
+-#
+-# Fail2Ban is distributed in the hope that it will be useful,
+-# but WITHOUT ANY WARRANTY; without even the implied warranty of
+-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+-# GNU General Public License for more details.
+-#
+-# You should have received a copy of the GNU General Public License
+-# along with Fail2Ban; if not, write to the Free Software
+-# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+-
+-__author__ = "Cyril Jaquier, Steven Hiscocks, Yaroslav Halchenko"
+-__copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2008-2016 Fail2Ban Contributors"
+-__license__ = "GPL"
+-
+-import platform
+-
+-try:
+- import setuptools
+- from setuptools import setup
+- from setuptools.command.install import install
+- from setuptools.command.install_scripts import install_scripts
+-except ImportError:
+- setuptools = None
+- from distutils.core import setup
+-
+-# all versions
+-from distutils.command.build_py import build_py
+-from distutils.command.build_scripts import build_scripts
+-if setuptools is None:
+- from distutils.command.install import install
+- from distutils.command.install_scripts import install_scripts
+-try:
+- # python 3.x
+- from distutils.command.build_py import build_py_2to3
+- from distutils.command.build_scripts import build_scripts_2to3
+- _2to3 = True
+-except ImportError:
+- # python 2.x
+- _2to3 = False
+-
+-import os
+-from os.path import isfile, join, isdir, realpath
+-import re
+-import sys
+-import warnings
+-from glob import glob
+-
+-from fail2ban.setup import updatePyExec
+-
+-
+-source_dir = os.path.realpath(os.path.dirname(
+- # __file__ seems to be overwritten sometimes on some python versions (e.g. bug of 2.6 by running under cProfile, etc.):
+- sys.argv[0] if os.path.basename(sys.argv[0]) == 'setup.py' else __file__
+-))
+-
+-# Wrapper to install python binding (to current python version):
+-class install_scripts_f2b(install_scripts):
+-
+- def get_outputs(self):
+- outputs = install_scripts.get_outputs(self)
+- # setup.py --dry-run install:
+- dry_run = not outputs
+- self.update_scripts(dry_run)
+- if dry_run:
+- #bindir = self.install_dir
+- bindir = self.build_dir
+- print('creating fail2ban-python binding -> %s (dry-run, real path can be different)' % (bindir,))
+- print('Copying content of %s to %s' % (self.build_dir, self.install_dir));
+- return outputs
+- fn = None
+- for fn in outputs:
+- if os.path.basename(fn) == 'fail2ban-server':
+- break
+- bindir = os.path.dirname(fn)
+- print('creating fail2ban-python binding -> %s' % (bindir,))
+- updatePyExec(bindir)
+- return outputs
+-
+- def update_scripts(self, dry_run=False):
+- buildroot = os.path.dirname(self.build_dir)
+- install_dir = self.install_dir
+- try:
+- # remove root-base from install scripts path:
+- root = self.distribution.command_options['install']['root'][1]
+- if install_dir.startswith(root):
+- install_dir = install_dir[len(root):]
+- except: # pragma: no cover
+- print('WARNING: Cannot find root-base option, check the bin-path to fail2ban-scripts in "fail2ban.service".')
+- print('Creating %s/fail2ban.service (from fail2ban.service.in): @BINDIR@ -> %s' % (buildroot, install_dir))
+- with open(os.path.join(source_dir, 'files/fail2ban.service.in'), 'r') as fn:
+- lines = fn.readlines()
+- fn = None
+- if not dry_run:
+- fn = open(os.path.join(buildroot, 'fail2ban.service'), 'w')
+- try:
+- for ln in lines:
+- ln = re.sub(r'@BINDIR@', lambda v: install_dir, ln)
+- if dry_run:
+- sys.stdout.write(' | ' + ln)
+- continue
+- fn.write(ln)
+- finally:
+- if fn: fn.close()
+- if dry_run:
+- print(' `')
+-
+-
+-# Wrapper to specify fail2ban own options:
+-class install_command_f2b(install):
+- user_options = install.user_options + [
+- ('disable-2to3', None, 'Specify to deactivate 2to3, e.g. if the install runs from fail2ban test-cases.'),
+- ('without-tests', None, 'without tests files installation'),
+- ]
+- def initialize_options(self):
+- self.disable_2to3 = None
+- self.without_tests = None
+- install.initialize_options(self)
+- def finalize_options(self):
+- global _2to3
+- ## in the test cases 2to3 should be already done (fail2ban-2to3):
+- if self.disable_2to3:
+- _2to3 = False
+- if _2to3:
+- cmdclass = self.distribution.cmdclass
+- cmdclass['build_py'] = build_py_2to3
+- cmdclass['build_scripts'] = build_scripts_2to3
+- if self.without_tests:
+- self.distribution.scripts.remove('bin/fail2ban-testcases')
+-
+- self.distribution.packages.remove('fail2ban.tests')
+- self.distribution.packages.remove('fail2ban.tests.action_d')
+-
+- del self.distribution.package_data['fail2ban.tests']
+- install.finalize_options(self)
+- def run(self):
+- install.run(self)
+-
+-
+-# Update fail2ban-python env to current python version (where f2b-modules located/installed)
+-updatePyExec(os.path.join(source_dir, 'bin'))
+-
+-if setuptools and "test" in sys.argv:
+- import logging
+- logSys = logging.getLogger("fail2ban")
+- hdlr = logging.StreamHandler(sys.stdout)
+- fmt = logging.Formatter("%(asctime)-15s %(message)s")
+- hdlr.setFormatter(fmt)
+- logSys.addHandler(hdlr)
+- if set(["-q", "--quiet"]) & set(sys.argv):
+- logSys.setLevel(logging.CRITICAL)
+- warnings.simplefilter("ignore")
+- sys.warnoptions.append("ignore")
+- elif set(["-v", "--verbose"]) & set(sys.argv):
+- logSys.setLevel(logging.DEBUG)
+- else:
+- logSys.setLevel(logging.INFO)
+-elif "test" in sys.argv:
+- print("python distribute required to execute fail2ban tests")
+- print("")
+-
+-longdesc = '''
+-Fail2Ban scans log files like /var/log/pwdfail or
+-/var/log/apache/error_log and bans IP that makes
+-too many password failures. It updates firewall rules
+-to reject the IP address or executes user defined
+-commands.'''
+-
+-if setuptools:
+- setup_extra = {
+- 'test_suite': "fail2ban.tests.utils.gatherTests",
+- 'use_2to3': True,
+- }
+-else:
+- setup_extra = {}
+-
+-data_files_extra = []
+-if os.path.exists('/var/run'):
+- # if we are on the system with /var/run -- we are to use it for having fail2ban/
+- # directory there for socket file etc.
+- # realpath is used to possibly resolve /var/run -> /run symlink
+- data_files_extra += [(realpath('/var/run/fail2ban'), '')]
+-
+-# Installing documentation files only under Linux or other GNU/ systems
+-# (e.g. GNU/kFreeBSD), since others might have protective mechanisms forbidding
+-# installation there (see e.g. #1233)
+-platform_system = platform.system().lower()
+-doc_files = ['README.md', 'DEVELOP', 'FILTERS', 'doc/run-rootless.txt']
+-if platform_system in ('solaris', 'sunos'):
+- doc_files.append('README.Solaris')
+-if platform_system in ('linux', 'solaris', 'sunos') or platform_system.startswith('gnu'):
+- data_files_extra.append(
+- ('/usr/share/doc/fail2ban', doc_files)
+- )
+-
+-# Get version number, avoiding importing fail2ban.
+-# This is due to tests not functioning for python3 as 2to3 takes place later
+-exec(open(join("fail2ban", "version.py")).read())
+-
+-setup(
+- name = "fail2ban",
+- version = version,
+- description = "Ban IPs that make too many password failures",
+- long_description = longdesc,
+- author = "Cyril Jaquier & Fail2Ban Contributors",
+- author_email = "cyril.jaquier@...",
+- url = "http://www.fail2ban.org",
+- license = "GPL",
+- platforms = "Posix",
+- cmdclass = {
+- 'build_py': build_py, 'build_scripts': build_scripts,
+- 'install_scripts': install_scripts_f2b, 'install': install_command_f2b
+- },
+- scripts = [
+- 'bin/fail2ban-client',
+- 'bin/fail2ban-server',
+- 'bin/fail2ban-regex',
+- 'bin/fail2ban-testcases',
+- # 'bin/fail2ban-python', -- link (binary), will be installed via install_scripts_f2b wrapper
+- ],
+- packages = [
+- 'fail2ban',
+- 'fail2ban.client',
+- 'fail2ban.server',
+- 'fail2ban.tests',
+- 'fail2ban.tests.action_d',
+- ],
+- package_data = {
+- 'fail2ban.tests':
+- [ join(w[0], f).replace("fail2ban/tests/", "", 1)
+- for w in os.walk('fail2ban/tests/files')
+- for f in w[2]] +
+- [ join(w[0], f).replace("fail2ban/tests/", "", 1)
+- for w in os.walk('fail2ban/tests/config')
+- for f in w[2]] +
+- [ join(w[0], f).replace("fail2ban/tests/", "", 1)
+- for w in os.walk('fail2ban/tests/action_d')
+- for f in w[2]]
+- },
+- data_files = [
+- ('/etc/fail2ban',
+- glob("config/*.conf")
+- ),
+- ('/etc/fail2ban/filter.d',
+- glob("config/filter.d/*.conf")
+- ),
+- ('/etc/fail2ban/filter.d/ignorecommands',
+- [p for p in glob("config/filter.d/ignorecommands/*") if isfile(p)]
+- ),
+- ('/etc/fail2ban/action.d',
+- glob("config/action.d/*.conf") +
+- glob("config/action.d/*.py")
+- ),
+- ('/etc/fail2ban/fail2ban.d',
+- ''
+- ),
+- ('/etc/fail2ban/jail.d',
+- ''
+- ),
+- ('/var/lib/fail2ban',
+- ''
+- ),
+- ] + data_files_extra,
+- **setup_extra
+-)
+-
+-# Do some checks after installation
+-# Search for obsolete files.
+-obsoleteFiles = []
+-elements = {
+- "/etc/":
+- [
+- "fail2ban.conf"
+- ],
+- "/usr/bin/":
+- [
+- "fail2ban.py"
+- ],
+- "/usr/lib/fail2ban/":
+- [
+- "version.py",
+- "protocol.py"
+- ]
+-}
+-
+-for directory in elements:
+- for f in elements[directory]:
+- path = join(directory, f)
+- if isfile(path):
+- obsoleteFiles.append(path)
+-
+-if obsoleteFiles:
+- print("")
+- print("Obsolete files from previous Fail2Ban versions were found on "
+- "your system.")
+- print("Please delete them:")
+- print("")
+- for f in obsoleteFiles:
+- print("\t" + f)
+- print("")
+-
+-if isdir("/usr/lib/fail2ban"):
+- print("")
+- print("Fail2ban is not installed under /usr/lib anymore. The new "
+- "location is under /usr/share. Please remove the directory "
+- "/usr/lib/fail2ban and everything under this directory.")
+- print("")
+-
+-# Update config file
+-if sys.argv[1] == "install":
+- print("")
+- print("Please do not forget to update your configuration files.")
+- print("They are in \"/etc/fail2ban/\".")
+- print("")
+- print("You can also install systemd service-unit file from \"build/fail2ban.service\"")
+- print("resp. corresponding init script from \"files/*-initd\".")
+- print("")
+--
+2.17.1
+
diff --git a/recipes-security/fail2ban/files/fail2ban_setup.py b/recipes-security/fail2ban/files/fail2ban_setup.py
index a5d4ed6..e231949 100755
--- a/recipes-security/fail2ban/files/fail2ban_setup.py
+++ b/recipes-security/fail2ban/files/fail2ban_setup.py
@@ -1,4 +1,3 @@
-#!/usr/bin/env python
# emacs: -*- mode: python; py-indent-offset: 4; indent-tabs-mode: t -*-
# vi: set ft=python sts=4 ts=4 sw=4 noet :

diff --git a/recipes-security/fail2ban/python3-fail2ban_0.10.4.0.bb b/recipes-security/fail2ban/python3-fail2ban_0.10.4.0.bb
index 79bd2b0..8673c10 100644
--- a/recipes-security/fail2ban/python3-fail2ban_0.10.4.0.bb
+++ b/recipes-security/fail2ban/python3-fail2ban_0.10.4.0.bb
@@ -14,6 +14,7 @@ SRC_URI = " git://github.com/fail2ban/fail2ban.git;branch=0.11 \
file://initd \
file://fail2ban_setup.py \
file://run-ptest \
+ file://0001-python3-fail2ban-2-3-conversion.patch \
"

inherit update-rc.d ptest setuptools3
@@ -45,5 +46,6 @@ INITSCRIPT_PARAMS = "defaults 25"

INSANE_SKIP_${PN}_append = "already-stripped"

-RDEPENDS_${PN} = "sysklogd iptables sqlite3 ${PYTHON_PN} ${PYTHON_PN}-pyinotify"
+RDEPENDS_${PN} = "sysklogd iptables sqlite3 python3-core python3-pyinotify python3-logging python3-fcntl"
+RDEPENDS_${PN} += " python3-json "
RDEPENDS_${PN}-ptest = "python3-core python3-io python3-modules python3-fail2ban"
--
2.17.1


Re: QA notification for completed autobuilder build (yocto-3.0.2.rc2)

Armin Kuster
 

On 2/12/20 7:06 PM, Jain, Sangeeta wrote:


Hello All,

Intel and WR YP QA is planning for QA execution for YP build yocto-3.0.2.rc2 starting from WW8.1
We are planning to execute following tests for this cycle:

OEQA-manual tests for following module:
1. OE-Core
2. BSP-hw

Runtime auto test for following platforms:
1. MinnowTurbot 32-bit
2. Coffee Lake
3. NUC 7
4. NUC 6
5. Edgerouter
6. MPC8315e-rdb


ETA for completion is next Wednesday, February 19.
That sounds great.

I appreciate both QA teams and their effort towards ensuring the stabled
branch quality.

regards,
Armin

Thanks,
Sangeeta


-----Original Message-----
From: yocto@... <yocto@...> On Behalf
Of pokybuild@...
Sent: Wednesday, 12 February, 2020 3:56 PM
To: yocto@...
Cc: otavio@...; yi.zhao@...; Sangal, Apoorv
<apoorv.sangal@...>; Yeoh, Ee Peng <ee.peng.yeoh@...>; Chan,
Aaron Chun Yew <aaron.chun.yew.chan@...>;
richard.purdie@...; akuster808@...;
sjolley.yp.pm@...; Jain, Sangeeta <sangeeta.jain@...>
Subject: [yocto] QA notification for completed autobuilder build (yocto-
3.0.2.rc2)


A build flagged for QA (yocto-3.0.2.rc2) was completed on the autobuilder and is
available at:


https://autobuilder.yocto.io/pub/releases/yocto-3.0.2.rc2


Build hash information:

bitbake: 95687be83e716220eb3893b67428f97fd59fc2c5
meta-gplv2: 0f4eecc000f66d114ad258fa31aed66afa292166
meta-intel: b04e1edb9300a57e200a187a3255f67b50519202
meta-mingw: 756963cc28ebc163df7d7f4b4ee004c18d3d3260
oecore: 799b3cd1016bd765f4452a5e81ea5613c9089bce
poky: fe857e4179355bcfb79303c16baf3ad87fca59a4



This is an automated message from the Yocto Project Autobuilder
Git: git://git.yoctoproject.org/yocto-autobuilder2
Email: richard.purdie@...



Re: QA notification for completed autobuilder build (yocto-3.0.2.rc2)

Sangeeta Jain
 

Hello All,

Intel and WR YP QA is planning for QA execution for YP build yocto-3.0.2.rc2 starting from WW8.1
We are planning to execute following tests for this cycle:

OEQA-manual tests for following module:
1. OE-Core
2. BSP-hw

Runtime auto test for following platforms:
1. MinnowTurbot 32-bit
2. Coffee Lake
3. NUC 7
4. NUC 6
5. Edgerouter
6. MPC8315e-rdb


ETA for completion is next Wednesday, February 19.

Thanks,
Sangeeta

-----Original Message-----
From: yocto@... <yocto@...> On Behalf
Of pokybuild@...
Sent: Wednesday, 12 February, 2020 3:56 PM
To: yocto@...
Cc: otavio@...; yi.zhao@...; Sangal, Apoorv
<apoorv.sangal@...>; Yeoh, Ee Peng <ee.peng.yeoh@...>; Chan,
Aaron Chun Yew <aaron.chun.yew.chan@...>;
richard.purdie@...; akuster808@...;
sjolley.yp.pm@...; Jain, Sangeeta <sangeeta.jain@...>
Subject: [yocto] QA notification for completed autobuilder build (yocto-
3.0.2.rc2)


A build flagged for QA (yocto-3.0.2.rc2) was completed on the autobuilder and is
available at:


https://autobuilder.yocto.io/pub/releases/yocto-3.0.2.rc2


Build hash information:

bitbake: 95687be83e716220eb3893b67428f97fd59fc2c5
meta-gplv2: 0f4eecc000f66d114ad258fa31aed66afa292166
meta-intel: b04e1edb9300a57e200a187a3255f67b50519202
meta-mingw: 756963cc28ebc163df7d7f4b4ee004c18d3d3260
oecore: 799b3cd1016bd765f4452a5e81ea5613c9089bce
poky: fe857e4179355bcfb79303c16baf3ad87fca59a4



This is an automated message from the Yocto Project Autobuilder
Git: git://git.yoctoproject.org/yocto-autobuilder2
Email: richard.purdie@...



Re: #yocto update the kernel with a rauc bundle #yocto

Enrico Jörns
 

Hi Hans,

Am 12.02.20 um 18:54 schrieb Hans-Ulrich Schlieben:
Hi,

i have a working configuration with images and rauc bundles. My problem
is that kernel and devicetree are located outside the rootfs and so a
bundle update never includes these.
a RAUC bundle can contain images for multiple partitions / slots.
You just have to specify them in your system.conf to make them updatable.

My image.wks looks this:
part BAREBOX --source rawcopy
--sourceparams="file=barebox.bin,skip=1024" --ondisk mmc --no-table
--align 1
part   --source bootimg-partition --fstype=vfat --label boot --ondisk
mmc --active --align 4096 --size 20
part / --source rootfs            --fstype=ext4 --label rootA --ondisk
mmc --align 1024 --use-uuid --fixed-size 700
part / --source rootfs            --fstype=ext4 --label rootB --ondisk
mmc --align 1024 --use-uuid --fixed-size 700

How do i move kernel and devicetree into /boot? 
An

IMAGE_INSTALL_append = " kernel-image kernel-devicetree"

, for example in your image recipe, should do the trick.

When using barebox you have full bootspec support thus you probably als
want to place a bootspec entry in /loder/entries to automate loading the
right kernel and dtb. A helper class is here:

https://github.com/pengutronix/meta-ptx/blob/master/classes/bootspec.bbclass

Any help would be greatly appreciated. Thank you 
Best regards, Enrico


--
Pengutronix e.K. | Enrico Jörns |
Industrial Linux Solutions | http://www.pengutronix.de/ |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-5080 |
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |


#raspberrypi #yocto Adding delay for initiation / startup of usb connected device #raspberrypi #yocto

markus4dev@...
 

Hi!

 

I am having problems with my raspberry pi3 a+ yocto thud build when connecting a  LTE E3372 modem. Sometimes it works but most of the times the startup of the LTE device interfere with services that are started ending up with blocking the raspberry pi.

 

I want to test to delay the startup of the LTE device with kernel-parameters similar to the following.

 

usb-storage.delay_use=90

 

Is it possible to use the RPI_EXTRA_CONFIG variable to add a few line to config.txt or are there any other ways to delay the initiation of the lte modem? If possible, please attach an example.

 

Thanks,

Markus


Re: [yocto-builds] /sbin/telinit conflicts between systemd and sysvinit in yocto 2.6.3

Randy MacLeod
 

Hi Vijay,

Note that the yocto-builds list is for automated tests and reports.
The usual list is:
   yocto@...

See below for my reply once I address the email list confusion.

Michael,
Can you help here or should I contact someone else?
Since this isn't the first time someone thinks that their
build problem emails should be sent to the yocto-builds email
list, could you add a WARNING, say:
   Automated feed for autobuilder results.
   WARNING, ONLY FOR: Automated feed for autobuilder results.
on:


Vijay, would have helped you find the right list?
Maybe we need to change:

  • Yocto Project Discussions (yocto): Discussion with Yocto Project developers – start here with general questions.
  • Yocto Project Discussions (yocto): Discussion with Yocto Project developers and users – start here with general questions.

  • On 2/9/20 7:28 AM, ansurivijay.r@... wrote:
    Hi,

    I have added below lines in local.conf of yocto 2.6.3
    VIRTUAL-RUNTIME_init_manager = " systemd"
    DISTRO_FEATURES_append = " systemd"
    DISTRO_FEATURES_BACKFILL_CONSIDERED += "sysvinit"
     
    When I building sdk using "bitbake core-image-sato-sdk -c populate_sdk"
    You likely need all of:

    # Select the default init manager
    # use systemd as the default init manager
    DISTRO_FEATURES_append = " systemd"
    DISTRO_FEATURES_BACKFILL_CONSIDERED += "sysvinit"
    VIRTUAL-RUNTIME_init_manager = "systemd"
    VIRTUAL-RUNTIME_initscripts = "systemd-compat-units"
    KERNEL_FEATURES_append = " cfg/systemd.scc"

    As shown in the WR Linux sample local.conf:

    https://github.com/WindRiver-Labs/wrlinux-x/blob/WRLINUX_10_19_BASE/data/samples/local.conf.sample#L302

    and documented for Yocto distros in geneal here:

    https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#selecting-an-initialization-manager

    ../Randy

     
    I'm facing " file /sbin/telinit conflicts between attempted installs of systemd-1.239-r0.armv7vet2hf_neon and sysvinit-2.88dsf-r14.armv7vet2hf_neon
     
    Can anyone help me fix this?
     
    Thanks &Regards, 
    Vijay

    
        


    -- 
    # Randy MacLeod
    # Wind River Linux
    


    Re: Creating a QEmu (x86-64) Image, which can execute binary applications from other x86-64 linux OSes

    Khem Raj
     

    On 2/12/20 3:19 AM, Christian Lohr wrote:
    Ok, I created a symbolic link with “ln -s /lib /lib64” and it seemed to work. Thanks a lot.
    right, if you built multilib image then it will be using /lib64 automatically.

    https://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#combining-multiple-versions-library-files-into-one-image

    *Von:* yocto@... <yocto@...> *Im Auftrag von *Alexander Kanavin
    *Gesendet:* Mittwoch, 12. Februar 2020 12:00
    *An:* Lohr, Christian [ext] <christian.lohr.ext@...>
    *Cc:* yocto@...
    *Betreff:* Re: [yocto] Creating a QEmu (x86-64) Image, which can execute binary applications from other x86-64 linux OSes
    But this is exactly what happens: the kernel reads the dynamic loader/interpreter path from the binary (which is different than the list of dynamically linked libraries printed by ldd), isn't able to find it and stops right there.
    Try like this:
    /lib/ld-linux-x86-64.so.2 /usr/share/dotnet/dotnet
    Alex
    On Wed, 12 Feb 2020 at 11:45, Lohr, Christian [ext] <christian.lohr.ext@... <mailto:christian.lohr.ext@...>> wrote:
    I used the x86_64 variant from the layer (it only downloads the
    binaries and copies them).
    And I checked that with ldd, it seemed ok so far:
    ---------------------------------------------------------
    root@qemux86-64:/usr/share/dotnet# ldd dotnet
                linux-vdso.so.1 (0x00007fffea543000)
                libpthread.so.0 => /lib/libpthread.so.0
    (0x00007f9fde06c000)
                libdl.so.2 => /lib/libdl.so.2 (0x00007f9fde067000)
    libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f9fddee5000)
    libm.so.6 => /lib/libm.so.6 (0x00007f9fddda4000)
                libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f9fddd8a000)
                libc.so.6 => /lib/libc.so.6 (0x00007f9fddbd0000)
                /lib64/ld-linux-x86-64.so.2 =>
    /lib/ld-linux-x86-64.so.2 (0x00007f9fde097000)
    Strace didn’t help either:
    -------------------------------
    root@qemux86-64:/# strace /usr/share/dotnet/dotnet
    execve("/usr/share/dotnet/dotnet", ["/usr/share/dotnet/dotnet"],
    0x7ffe22f0ab70 /* 22 vars */) = -1 ENOENT (No such file or directory)
    strace: exec: No such file or directory
    +++ exited with 1 +++
    It’s strange that it denies that the binaries are there. Normally I
    would have expected something like “wrong elf architecture” or
    something about missing libraries. The only thing I think I could do
    now, is to turn this “—enable-default-pie” off, but I’m not sure if
    this helps, and I don’t know where to turn it off. And what I’m also
    trying is to go back to Yocto Rocko Release (for this experiment I
    used Warrior Release)
    *Von:* yocto@...
    <mailto:yocto@...> <yocto@...
    <mailto:yocto@...>> *Im Auftrag von *Alexander
    Kanavin
    *Gesendet:* Mittwoch, 12. Februar 2020 10:26
    *An:* Lohr, Christian [ext] <christian.lohr.ext@...
    <mailto:christian.lohr.ext@...>>
    *Cc:* yocto@... <mailto:yocto@...>
    *Betreff:* Re: [yocto] Creating a QEmu (x86-64) Image, which can
    execute binary applications from other x86-64 linux OSes
    That layer does have the x86_64 variant as well, no? Is it not working?
    https://github.com/RDunkley/meta-dotnet-core/blob/master/recipes-runtime/dotnet-core/dotnet-core_3.1.1.inc
    <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRDunkley%2Fmeta-dotnet-core%2Fblob%2Fmaster%2Frecipes-runtime%2Fdotnet-core%2Fdotnet-core_3.1.1.inc&data=02%7C01%7C%7Cdb5e5b579ece4304296a08d7afaabc5d%7C28042244bb514cd680347776fa3703e8%7C1%7C0%7C637171020148948337&sdata=Vsh9c35%2B8gIqU9Hh%2BXz07z6NkZ6bF9LekpSOXdMZ5wU%3D&reserved=0>
    The error you're seeing is almost certainly due to Yocto using
    /lib/ld-so... for dynamic loader, and the binary hardcoding /lib64/....
    Alex
    On Wed, 12 Feb 2020 at 10:15, Lohr, Christian [ext]
    <christian.lohr.ext@... <mailto:christian.lohr.ext@...>>
    wrote:
    Hello Alex,
    Thanks for replying. Yes, I know that this isn’t the way it
    works on Yocto (and I told the managers it is a crappy idea to
    do that more than once). But they need .NET Core in that company
    I work for, and Mono doesn’t work (that’s what they told me).
    Compiling .NET Core through the Yocto process is ugly, because
    Microsoft used a mixture of shell scripts to compile it for some
    platforms, it won’t work this way on Yocto. Actually one already
    tried it, but only until .NET Core 2.2:
    https://github.com/Tragetaschen/meta-aspnet
    <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FTragetaschen%2Fmeta-aspnet&data=02%7C01%7C%7Cdb5e5b579ece4304296a08d7afaabc5d%7C28042244bb514cd680347776fa3703e8%7C1%7C0%7C637171020148948337&sdata=lQ2kX9cti11SXfWsL9sQM5n1PCYRLd1g0BYXOenXXJw%3D&reserved=0>
    And despite this, I already managed to get the dotnet binaries
    for ARM32 and ARM64 already working on a i.MX6 and i.MX8
    There’s a layer which just deploys the binaries:
    https://github.com/RDunkley/meta-dotnet-core
    <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRDunkley%2Fmeta-dotnet-core&data=02%7C01%7C%7Cdb5e5b579ece4304296a08d7afaabc5d%7C28042244bb514cd680347776fa3703e8%7C1%7C0%7C637171020148958296&sdata=VZrIi9EKPKTzlGZ9el0qMCZLKXrLL119L18Ds%2FhzQg4%3D&reserved=0>
    This is currently the last step. I thought if it worked on i.MX6
    and i.MX8 it shouldn’t be a problem to get it running on
    Virtualbox with x86-64. It should only make the things easy for
    the developers. It isn’t even our target platform.
    Best regards,
    Christian Lohr
    *Von:* yocto@...
    <mailto:yocto@...>
    <yocto@...
    <mailto:yocto@...>> *Im Auftrag von
    *Alexander Kanavin
    *Gesendet:* Mittwoch, 12. Februar 2020 09:51
    *An:* Lohr, Christian [ext] <christian.lohr.ext@...
    <mailto:christian.lohr.ext@...>>
    *Cc:* yocto@...
    <mailto:yocto@...>
    *Betreff:* Re: [yocto] Creating a QEmu (x86-64) Image, which can
    execute binary applications from other x86-64 linux OSes
    Yocto generally does not support this use case. The binary was
    compiled in a different environment and expects things in
    different places, and probably being different versions too. I
    could point out the specific problem why the executable doesn't
    even start, but it's really the wrong way to approach it. Is the
    source code for it available?
    Microsoft specifically lists which distributions are supported,
    and there is nothing Yocto-based in it:
    https://docs.microsoft.com/en-us/dotnet/core/install/dependencies?tabs=netcore31&pivots=os-linux
    <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fdotnet%2Fcore%2Finstall%2Fdependencies%3Ftabs%3Dnetcore31%26pivots%3Dos-linux&data=02%7C01%7C%7Cdb5e5b579ece4304296a08d7afaabc5d%7C28042244bb514cd680347776fa3703e8%7C1%7C0%7C637171020148958296&sdata=eh%2BrdniGyyrkD8B9%2BihYKM0GSun3W%2FTrftIpptywNnY%3D&reserved=0>
    For mono things you can use meta-mono layer, but I am not sure
    if it provides exactly the item you're after.
    Alex
    On Wed, 12 Feb 2020 at 09:36, Christian Lohr
    <christian.lohr.ext@...
    <mailto:christian.lohr.ext@...>> wrote:
    Hello,
    I’m trying to create a normal QEmu (x86-64) Image, which I
    can let run in Virtualbox. As a addition I deployed .NET
    Core, which I got from this side:
    https://dotnet.microsoft.com/download/dotnet-core/thank-you/runtime-aspnetcore-3.1.1-linux-x64-binaries
    <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdotnet.microsoft.com%2Fdownload%2Fdotnet-core%2Fthank-you%2Fruntime-aspnetcore-3.1.1-linux-x64-binaries&data=02%7C01%7C%7Cdb5e5b579ece4304296a08d7afaabc5d%7C28042244bb514cd680347776fa3703e8%7C1%7C0%7C637171020148968250&sdata=qENira3bdwQEE8B2QfPoY6yymGoHtVBrYRIFq4NBUWg%3D&reserved=0>
    But I can’t execute it:
    ----------------------------
    root@qemux86-64:/usr/share/dotnet# ./dotnet
    -sh: ./dotnet: No such file or directory
    But it is there:
    ------------------
    root@qemux86-64:/usr/share/dotnet# ls -lh
    total 116K
    -rw-r--r-- 1 root root 1.1K Feb 10 02:33 LICENSE.txt
    -rw-r--r-- 1 root root  31K Feb 10 02:33 ThirdPartyNotices.txt
    -rwxr-xr-x 1 root root  72K Feb 10 02:33 dotnet
    drwxr-xr-x 3 root root 4.0K Feb 10 02:36 host
    drwxr-xr-x 4 root root 4.0K Feb 10 02:36 shared
    It tried to get more information about the dotnet-executable
    ------------------------------------------------------------------------------
    root@qemux86-64:/usr/share/dotnet# readelf -h dotnet
    ELF Header:
      Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
      Class:                             ELF64
      Data:                              2's complement, little
    endian
      Version:                           1 (current)
      OS/ABI:                            UNIX - System V
      ABI Version:                       0
      Type:                              EXEC (Executable file)
      Machine:                           Advanced Micro Devices
    X86-64
      Version:                           0x1
      Entry point address:               0x402f17
      Start of program headers:          64 (bytes into file)
      Start of section headers:          71032 (bytes into file)
      Flags:                             0x0
      Size of this header:               64 (bytes)
      Size of program headers:           56 (bytes)
      Number of program headers:         10
      Size of section headers:           64 (bytes)
      Number of section headers:         31
      Section header string table index: 30
    root@qemux86-64:/usr/share/dotnet#  file dotnet
    dotnet: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
    dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2,
    for GNU/Linux 2.6.32,
    BuildID[sha1]=28c244c1953bcbee994709a4b849086ee7cf0c99, stripped
    I compared those values with that from Python, which does
    run on this system
    -------------------------------------------------------------------------------------------------------
    root@qemux86-64:/opt/jre-8/bin# readelf -h /usr/bin/python3.7
    ELF Header:
      Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
      Class:                             ELF64
      Data:                              2's complement, little
    endian
      Version:                           1 (current)
      OS/ABI:                            UNIX - System V
      ABI Version:                       0
      Type:                              DYN (Shared object file)
      Machine:                           Advanced Micro Devices
    X86-64
      Version:                           0x1
      Entry point address:               0x1060
      Start of program headers:          64 (bytes into file)
      Start of section headers:          12568 (bytes into file)
      Flags:                             0x0
      Size of this header:               64 (bytes)
      Size of program headers:           56 (bytes)
      Number of program headers:         11
      Size of section headers:           64 (bytes)
      Number of section headers:         27
      Section header string table index: 26
    root@qemux86-64:/usr/share/dotnet# file /usr/bin/python3.7
    /usr/bin/python3.7: ELF 64-bit LSB pie executable, x86-64,
    version 1 (SYSV), dynamically linked, interpreter
    /lib/ld-linux-x86-64.so.2,
    BuildID[sha1]=a455873f278466378405802b0e0171337e52a81c, for
    GNU/Linux 3.2.0, stripped
    ================================================================================
    The only difference I found, is that Python is a “ELF 64-bit
    LSB pie executable” whereas dotnet is a “ELF 64-bit LSB
    executable”. I tried to turn that PIE (seemed to be a gcc
    option: --enable-default-pie) feature of, but that didn’t
    work well, and I couldn’t find a way to remove it.
    ----
    Best regards,
    Christian Lohr
    Im Auftrag von:
    Carl Zeiss Meditec AG
    Göschwitzer Strasse 51-52
    07745 Jena, Deutschland
    christian.lohr.ext@...
    <mailto:christian.lohr.ext@...>
    Tel: +493641220206