[PATCH] kunit: tool: Assert version requirement

[SeongJae Park]

From: SeongJae Park <sjp...@amazon.de>

Commit 87c9c1631788 ("kunit: tool: add support for QEMU") on the 'next'
tree adds 'from __future__ import annotations' in 'kunit_kernel.py'.
Because it is supported on only >=3.7 Python, people using older Python
will get below error:

Traceback (most recent call last):
File "./tools/testing/kunit/kunit.py", line 20, in <module>
import kunit_kernel
File "/home/sjpark/linux/tools/testing/kunit/kunit_kernel.py", line 9
from __future__ import annotations
^
SyntaxError: future feature annotations is not defined

This commit adds a version assertion in 'kunit.py', so that people get
more explicit error message like below:

Traceback (most recent call last):
File "./tools/testing/kunit/kunit.py", line 15, in <module>
assert sys.version_info >= (3, 7)
AssertionError

Signed-off-by: SeongJae Park <sjp...@amazon.de>
---
tools/testing/kunit/kunit.py | 2 ++
1 file changed, 2 insertions(+)

diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index be8d8d4a4e08..748d88178506 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -12,6 +12,8 @@ import sys
import os
import time

+assert sys.version_info >= (3, 7)
+
from collections import namedtuple
from enum import Enum, auto

--
2.17.1

[Daniel Latypov]

On Wed, Jun 16, 2021 at 2:40 AM SeongJae Park <sj38...@gmail.com> wrote:
>
> From: SeongJae Park <sjp...@amazon.de>
>
> Commit 87c9c1631788 ("kunit: tool: add support for QEMU") on the 'next'
> tree adds 'from __future__ import annotations' in 'kunit_kernel.py'.
> Because it is supported on only >=3.7 Python, people using older Python
> will get below error:

Ah, we had been fine with just using 3.6 before this (and could have
dropped down to 3.5 with a few lines changed).

But 3.7 came out initially in 2018*, so I assume we're probably fine
to rely on that in kunit tool.
*https://www.python.org/downloads/release/python-370/

>
> Traceback (most recent call last):
> File "./tools/testing/kunit/kunit.py", line 20, in <module>
> import kunit_kernel
> File "/home/sjpark/linux/tools/testing/kunit/kunit_kernel.py", line 9
> from __future__ import annotations
> ^
> SyntaxError: future feature annotations is not defined
>
> This commit adds a version assertion in 'kunit.py', so that people get
> more explicit error message like below:
>
> Traceback (most recent call last):
> File "./tools/testing/kunit/kunit.py", line 15, in <module>
> assert sys.version_info >= (3, 7)
> AssertionError
>
> Signed-off-by: SeongJae Park <sjp...@amazon.de>

Acked-by: Daniel Latypov <dlat...@google.com>

> ---
> tools/testing/kunit/kunit.py | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
> index be8d8d4a4e08..748d88178506 100755
> --- a/tools/testing/kunit/kunit.py
> +++ b/tools/testing/kunit/kunit.py
> @@ -12,6 +12,8 @@ import sys
> import os
> import time
>
> +assert sys.version_info >= (3, 7)

Do we perhaps want
assert sys.version_info >= (3, 7), "Python version is too old"

Then the error message would be

Traceback (most recent call last):
File "./tools/testing/kunit/kunit.py", line 15, in <module>

assert sys.version_info >= (3, 7), "Python version is too old"
AssertionError: Python version is too old

I assume kernel devs know some Python, but not necessarily that
sys.version_info == "my python version"

> +
> from collections import namedtuple
> from enum import Enum, auto
>
> --
> 2.17.1
>

> --
> You received this message because you are subscribed to the Google Groups "KUnit Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/20210616094033.18246-1-sj38.park%40gmail.com.

[SeongJae Park]

From: SeongJae Park <sjp...@amazon.de>

Hello Daniel,

On Wed, 16 Jun 2021 14:14:40 -0700 Daniel Latypov <dlat...@google.com> wrote:

> On Wed, Jun 16, 2021 at 2:40 AM SeongJae Park <sj38...@gmail.com> wrote:
> >
> > From: SeongJae Park <sjp...@amazon.de>
> >
> > Commit 87c9c1631788 ("kunit: tool: add support for QEMU") on the 'next'
> > tree adds 'from __future__ import annotations' in 'kunit_kernel.py'.
> > Because it is supported on only >=3.7 Python, people using older Python
> > will get below error:
>
> Ah, we had been fine with just using 3.6 before this (and could have
> dropped down to 3.5 with a few lines changed).
>
> But 3.7 came out initially in 2018*, so I assume we're probably fine
> to rely on that in kunit tool.
> *https://www.python.org/downloads/release/python-370/

Agreed.

>
> >
> > Traceback (most recent call last):
> > File "./tools/testing/kunit/kunit.py", line 20, in <module>
> > import kunit_kernel
> > File "/home/sjpark/linux/tools/testing/kunit/kunit_kernel.py", line 9
> > from __future__ import annotations
> > ^
> > SyntaxError: future feature annotations is not defined
> >
> > This commit adds a version assertion in 'kunit.py', so that people get
> > more explicit error message like below:
> >
> > Traceback (most recent call last):
> > File "./tools/testing/kunit/kunit.py", line 15, in <module>
> > assert sys.version_info >= (3, 7)
> > AssertionError
> >
> > Signed-off-by: SeongJae Park <sjp...@amazon.de>
>
> Acked-by: Daniel Latypov <dlat...@google.com>

Thank you! :)

>
> > ---
> > tools/testing/kunit/kunit.py | 2 ++
> > 1 file changed, 2 insertions(+)
> >
> > diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
> > index be8d8d4a4e08..748d88178506 100755
> > --- a/tools/testing/kunit/kunit.py
> > +++ b/tools/testing/kunit/kunit.py
> > @@ -12,6 +12,8 @@ import sys
> > import os
> > import time
> >
> > +assert sys.version_info >= (3, 7)
>
> Do we perhaps want
> assert sys.version_info >= (3, 7), "Python version is too old"
>
> Then the error message would be
> Traceback (most recent call last):
> File "./tools/testing/kunit/kunit.py", line 15, in <module>
> assert sys.version_info >= (3, 7), "Python version is too old"
> AssertionError: Python version is too old

That looks easier to understand. I will post v2 in reply to this.


Thanks,
SeongJae Park

[SeongJae Park]

Commit 87c9c1631788 ("kunit: tool: add support for QEMU") on the 'next'
tree adds 'from __future__ import annotations' in 'kunit_kernel.py'.
Because it is supported on only >=3.7 Python, people using older Python
will get below error:

Traceback (most recent call last):
File "./tools/testing/kunit/kunit.py", line 20, in <module>
import kunit_kernel
File "/home/sjpark/linux/tools/testing/kunit/kunit_kernel.py", line 9
from __future__ import annotations
^
SyntaxError: future feature annotations is not defined

This commit adds a version assertion in 'kunit.py', so that people get
more explicit error message like below:

Traceback (most recent call last):
File "./tools/testing/kunit/kunit.py", line 15, in <module>

assert sys.version_info >= (3, 7), "Python version is too old"
AssertionError: Python version is too old

Signed-off-by: SeongJae Park <sjp...@amazon.de>
Acked-by: Daniel Latypov <dlat...@google.com>

---

Changes from v1
- Add assertion failure message (Daniel Latypov)
- Add Acked-by: Daniel Latypov <dlat...@google.com>

tools/testing/kunit/kunit.py | 2 ++
1 file changed, 2 insertions(+)

diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py

index be8d8d4a4e08..6276ce0c0196 100755

--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -12,6 +12,8 @@ import sys
import os
import time

+assert sys.version_info >= (3, 7), "Python version is too old"

[Daniel Latypov]

On Thu, Jun 17, 2021 at 12:46 AM SeongJae Park <sj38...@gmail.com> wrote:
>
> Commit 87c9c1631788 ("kunit: tool: add support for QEMU") on the 'next'
> tree adds 'from __future__ import annotations' in 'kunit_kernel.py'.
> Because it is supported on only >=3.7 Python, people using older Python
> will get below error:
>
> Traceback (most recent call last):
> File "./tools/testing/kunit/kunit.py", line 20, in <module>
> import kunit_kernel
> File "/home/sjpark/linux/tools/testing/kunit/kunit_kernel.py", line 9
> from __future__ import annotations

Chatted offline with David about this.
He was thinking if we could instead drop the minimal version back to 3.6.

I think we can do so, see below.
Perhaps we should drop the import and then chain this patch on top of
that, specifying a minimum version of 3.6?

Checking out https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/?h=kunit-fixes

The offending "annotations" import is related to type annotations.
Specifically https://www.python.org/dev/peps/pep-0563/

So let's see how the two most popular typecheckers fare.

pytype is happy with or without import.
mypy has the same issues with or without the import.

$ mypy tools/testing/kunit/*.py
tools/testing/kunit/kunit_kernel.py:227: error: Item "_Loader" of
"Optional[_Loader]" has no attribute "exec_module"
tools/testing/kunit/kunit_kernel.py:227: error: Item "None" of
"Optional[_Loader]" has no attribute "exec_module"
tools/testing/kunit/kunit_kernel.py:228: error: Module has no
attribute "QEMU_ARCH"
tools/testing/kunit/kunit_kernel.py:229: error: Module has no
attribute "QEMU_ARCH"

So clearly it's not doing anything for them.

Taking a look over 87c9c1631788 ("kunit: tool: add support for QEMU")
next then...
I don't see anything that would warrant the import, so we should
probably drop it.

In that case, the minimum supported version should drop back down to 3.6.
We use enum.auto, which is from 3.6
https://docs.python.org/3/library/enum.html#enum.auto

We could consider stopping using that, and I think we might be then
3.5-compatible.
Maybe we have a chain of 3 patches then, drop the import, drop auto,
and then add in a >=3.5 version check?

[Daniel Latypov]

On Tue, Jun 22, 2021 at 4:28 PM Daniel Latypov <dlat...@google.com> wrote:
>
> On Thu, Jun 17, 2021 at 12:46 AM SeongJae Park <sj38...@gmail.com> wrote:
> >
> > Commit 87c9c1631788 ("kunit: tool: add support for QEMU") on the 'next'
> > tree adds 'from __future__ import annotations' in 'kunit_kernel.py'.
> > Because it is supported on only >=3.7 Python, people using older Python
> > will get below error:
> >
> > Traceback (most recent call last):
> > File "./tools/testing/kunit/kunit.py", line 20, in <module>
> > import kunit_kernel
> > File "/home/sjpark/linux/tools/testing/kunit/kunit_kernel.py", line 9
> > from __future__ import annotations
>
> Chatted offline with David about this.
> He was thinking if we could instead drop the minimal version back to 3.6.
>
> I think we can do so, see below.
> Perhaps we should drop the import and then chain this patch on top of
> that, specifying a minimum version of 3.6?

Actually, now I've gotten python3.6 installed on my machine, I see we
have another issue.

We pass text=true to subprocess.
That didn't exist back in 3.6, see
https://docs.python.org/3.6/library/subprocess.html

We can workaround that, but there's more chance of subtle bugs that
I'd rather we don't touch it.

>
> Checking out https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/?h=kunit-fixes
>
> The offending "annotations" import is related to type annotations.
> Specifically https://www.python.org/dev/peps/pep-0563/
>
> So let's see how the two most popular typecheckers fare.
>
> pytype is happy with or without import.
> mypy has the same issues with or without the import.
>
> $ mypy tools/testing/kunit/*.py
> tools/testing/kunit/kunit_kernel.py:227: error: Item "_Loader" of
> "Optional[_Loader]" has no attribute "exec_module"
> tools/testing/kunit/kunit_kernel.py:227: error: Item "None" of
> "Optional[_Loader]" has no attribute "exec_module"
> tools/testing/kunit/kunit_kernel.py:228: error: Module has no
> attribute "QEMU_ARCH"
> tools/testing/kunit/kunit_kernel.py:229: error: Module has no
> attribute "QEMU_ARCH"
>
> So clearly it's not doing anything for them.
>
> Taking a look over 87c9c1631788 ("kunit: tool: add support for QEMU")
> next then...
> I don't see anything that would warrant the import, so we should
> probably drop it.

Also, using 3.6 now I have it installed, I found what it was added for.
But it doesn't need to be there.

This patch drops it and makes things work, afaict:
diff --git a/tools/testing/kunit/kunit_kernel.py
b/tools/testing/kunit/kunit_kernel.py
index e1951fa60027..5987d5b1b874 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -6,15 +6,13 @@
# Author: Felix Guo <felixgu...@gmail.com>
# Author: Brendan Higgins <brendan...@google.com>

-from __future__ import annotations
import importlib.util
import logging
import subprocess
import os
import shutil
import signal
-from typing import Iterator
-from typing import Optional
+from typing import Iterator, Optional, Tuple

from contextlib import ExitStack

@@ -208,7 +206,7 @@ def get_source_tree_ops(arch: str, cross_compile:
Optional[str]) -> LinuxSourceT
raise ConfigError(arch + ' is not a valid arch')

def get_source_tree_ops_from_qemu_config(config_path: str,
- cross_compile: Optional[str]) -> tuple[
+ cross_compile: Optional[str]) -> Tuple[
str,
LinuxSourceTreeOperations]:
# The module name/path has very little to do with where the actual file
# exists (I learned this through experimentation and could not find it

>
> In that case, the minimum supported version should drop back down to 3.6.
> We use enum.auto, which is from 3.6
> https://docs.python.org/3/library/enum.html#enum.auto
>
> We could consider stopping using that, and I think we might be then
> 3.5-compatible.
> Maybe we have a chain of 3 patches then, drop the import, drop auto,
> and then add in a >=3.5 version check?
>
> > ^
> > SyntaxError: future feature annotations is not defined
> >
> > This commit adds a version assertion in 'kunit.py', so that people get
> > more explicit error message like below:
> >
> > Traceback (most recent call last):
> > File "./tools/testing/kunit/kunit.py", line 15, in <module>
> > assert sys.version_info >= (3, 7), "Python version is too old"
> > AssertionError: Python version is too old
> >
> > Signed-off-by: SeongJae Park <sjp...@amazon.de>
> > Acked-by: Daniel Latypov <dlat...@google.com>

Reviewed-by: Daniel Latypov <dlat...@google.com>

As mentioned above, we do actually need 3.7, and not just for the extra import.
Now I know that, I feel more strongly that this patch should go in, as-is.

[SeongJae Park]

Hello Brendan, could I ask you opinion for this patch?


Thanks,
SeongJae Park

[...]

[Brendan Higgins]

On Thu, Jun 17, 2021 at 12:46 AM SeongJae Park <sj38...@gmail.com> wrote:
>
> Commit 87c9c1631788 ("kunit: tool: add support for QEMU") on the 'next'
> tree adds 'from __future__ import annotations' in 'kunit_kernel.py'.
> Because it is supported on only >=3.7 Python, people using older Python
> will get below error:
>
> Traceback (most recent call last):
> File "./tools/testing/kunit/kunit.py", line 20, in <module>
> import kunit_kernel
> File "/home/sjpark/linux/tools/testing/kunit/kunit_kernel.py", line 9
> from __future__ import annotations
> ^
> SyntaxError: future feature annotations is not defined
>
> This commit adds a version assertion in 'kunit.py', so that people get
> more explicit error message like below:
>
> Traceback (most recent call last):
> File "./tools/testing/kunit/kunit.py", line 15, in <module>
> assert sys.version_info >= (3, 7), "Python version is too old"
> AssertionError: Python version is too old
>
> Signed-off-by: SeongJae Park <sjp...@amazon.de>
> Acked-by: Daniel Latypov <dlat...@google.com>

Reviewed-by: Brendan Higgins <brendan...@google.com>

[Brendan Higgins]

On Mon, Jun 28, 2021 at 6:38 AM SeongJae Park <sj38...@gmail.com> wrote:
>
> Hello Brendan, could I ask you opinion for this patch?

Yep, sorry, I just wanted to make sure everything else (there were a
lot of things in the queue) was in order for the 5.14 merge window
first.

Looks good!

Thanks

[Shuah Khan]

Your from and Signed-off-by email addresses don't match.

Please resend the patch with the correction.

thanks,
-- Shuah

[SeongJae Park]

From: SeongJae Park <sjp...@amazon.de>

Sorry, my fault. Will send fixed one as a reply to this.


Thanks,
SeongJae Park

>
> thanks,
> -- Shuah

[SeongJae Park]

From: SeongJae Park <sjp...@amazon.de>

Commit 87c9c1631788 ("kunit: tool: add support for QEMU") on the 'next'
tree adds 'from __future__ import annotations' in 'kunit_kernel.py'.
Because it is supported on only >=3.7 Python, people using older Python
will get below error:

Traceback (most recent call last):
File "./tools/testing/kunit/kunit.py", line 20, in <module>
import kunit_kernel
File "/home/sjpark/linux/tools/testing/kunit/kunit_kernel.py", line 9
from __future__ import annotations
^
SyntaxError: future feature annotations is not defined

This commit adds a version assertion in 'kunit.py', so that people get
more explicit error message like below:

Traceback (most recent call last):
File "./tools/testing/kunit/kunit.py", line 15, in <module>
assert sys.version_info >= (3, 7), "Python version is too old"
AssertionError: Python version is too old

Signed-off-by: SeongJae Park <sjp...@amazon.de>

Acked-by: Daniel Latypov <dlat...@google.com>
---

Changes from v2
- Fix mismatch of Signed-off and from