Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify AIX platform_tag so it provides PEP425 needs #190

Open
wants to merge 27 commits into
base: master
from

Conversation

@aixtools
Copy link

@aixtools aixtools commented Aug 25, 2019

Provide an adequate platform tag for AIX. The generic tag provided by distutils.util.get_platform() is insufficient - besides missing "bitness" it does not provide any information regarding Technology Level, nor build date.

@@ -310,6 +310,42 @@ def _have_compatible_glibc(required_major, minimum_minor):
return _check_glibc_version(version_str, required_major, minimum_minor)


def _aix_tag():
'''

This comment has been minimized.

@brettcannon

brettcannon Aug 26, 2019
Member

Suggested change
'''
"""
The program lslpp is used to gather these values.
The pep425 platform tag for AIX becomes:
AIX_V_R_M_YYWW_bitness
'''

This comment has been minimized.

@brettcannon

brettcannon Aug 26, 2019
Member

Suggested change
'''
"""
The pep425 platform tag for AIX becomes:
AIX_V_R_M_YYWW_bitness
'''

This comment has been minimized.

@brettcannon

brettcannon Aug 26, 2019
Member

Suggested change
AIX_V_R_M_YYWW_bitness
'''

from subprocess import Popen, PIPE, DEVNULL

This comment has been minimized.

@brettcannon

brettcannon Aug 26, 2019
Member

Why the inline import?

This comment has been minimized.

@aixtools

aixtools Sep 1, 2019
Author

I tried to respond to this, days ago, via email. I still do not see my reply - so I'll try again.

Basically, I was thinking - if not _AIX, then why import something additional that noone else is using. Simple enough to move to global imports.

p.s. I will be trying to get an _aix_support (similar to _osx_support) added into CPython, that would make this unneeded.

This comment has been minimized.

@brettcannon

brettcannon Sep 9, 2019
Member

Imports are generally cheap, so there's no need to hide it unless you're running some extremely performance-sensitive code.

_lslppLqc = p.stdout.read().strip().split(":")
p.wait()
(lpp, vrmf, bd) = list(_lslppLqc[index] for index in [0, 2, -1])
assert lpp == "bos.mp64", "%s != %s" % (lpp, "bos.mp64")

This comment has been minimized.

@brettcannon

brettcannon Aug 26, 2019
Member

str.format() is preferred.

This comment has been minimized.

@aixtools

aixtools Sep 1, 2019
Author

Thanks for the comment - I'll do this later in the week.

@pradyunsg
Copy link
Member

@pradyunsg pradyunsg commented Aug 27, 2019

Related to #188. That issue also contains a link to a broader discussion happening over on pip's issue tracker.

@aixtools aixtools force-pushed the aixtools:AIX-platform-tag branch from 1ebd62e to 66d7b40 Nov 6, 2019
@aixtools
Copy link
Author

@aixtools aixtools commented Nov 6, 2019

I'll run tox seperately, to try and figure out why it is failing ci, but better for me would be a pointer to the documentation I am sure is there - on how to test the code.

tox -e lint passes locally, failed just now because of some HTTP server error.

The other tests fail showing this block:



Name                         Stmts   Miss Branch BrPart  Cover   Missing
------------------------------------------------------------------------
packaging/_AIX_platform.py      20     20      2      0     0%   3-50
packaging/__about__.py          10      0      0      0   100%
packaging/__init__.py            3      0      0      0   100%
packaging/_structures.py        41      0      0      0   100%
packaging/_typing.py             3      0      0      0   100%
packaging/markers.py           133      0     45      0   100%
packaging/requirements.py       79      0     24      0   100%
packaging/specifiers.py        275      0    122      0   100%
packaging/tags.py              259      6    123      1    98%   491-499, 490->491
packaging/utils.py              25      0     14      0   100%

Thanks for pointers!

@brettcannon
Copy link
Member

@brettcannon brettcannon commented Nov 6, 2019

You can either read the tox.ini file to see how tox does it or the files in https://github.com/pypa/packaging/tree/master/.github/workflows as another example of how to run the tests.

@aixtools aixtools force-pushed the aixtools:AIX-platform-tag branch from 66d7b40 to e74cdc6 Nov 6, 2019
@aixtools
Copy link
Author

@aixtools aixtools commented Nov 6, 2019

Well, I am getting - read have become - so frustrated trying to figure out how mypi 'works' that I'll just give up on the whole project. A wasted day.

@brettcannon
Copy link
Member

@brettcannon brettcannon commented Nov 6, 2019

@aixtools if you're giving up then I'm going to close this PR. If you choose to try again then feel free to open the PR once more.

@brettcannon brettcannon closed this Nov 6, 2019
@aixtools
Copy link
Author

@aixtools aixtools commented Nov 7, 2019

Reopen? How? Only button I see is delete branch.

Sigh - refrain from other comments re: frustration - as they are too likely to be misunderstood.

p.s. I was hoping for some assistance with dealing with mypi.

@xavfernandez xavfernandez reopened this Nov 7, 2019
@xavfernandez
Copy link
Member

@xavfernandez xavfernandez commented Nov 7, 2019

@aixtools from my understanding mypy considers that sysconfig.get_config_var is an Optional[str] meaning it can return a None so you'll need to do something like:

from sysconfig import get_config_var


def test() -> str:
    value = get_config_var("test")
    if value is None:
        # or raise ValueError ?
        return ""
    return value
@aixtools
Copy link
Author

@aixtools aixtools commented Nov 8, 2019

@xavfernandez - Many Thanks - that helped a lot.

Python still makes my head spin when it comes to how the "absolute" typing of things constantly changes.

I have modified this little bit many times - as I get into issues with what is available "when".

So, while lint (read mypy) gets further, I am still stuck in what appears to be a circle for me. Or is the only solution to write another helper function to nudge typing.

Currently - stuck with:

   +51      # p = run(["/usr/bin/lslpp", "-Lqc", "bos.mp64"],
   +52      #       capture_output=True, text=True)
   +53      output = check_output(["/usr/bin/lslpp", "-Lqc", "bos.mp64"])
   +54      result = output.decode("utf-8").strip().split(":") # type: str

I have also tried # type: ignore

mypy.....................................................................Passed
mypy for Python 2........................................................Failed
hookid: mypy

packaging/_AIX_platform.py:54: error: Incompatible types in assignment (expression has type "List[unicode]", variable has type "str")

Also, a hint to see what black does not like.

black....................................................................Failed
hookid: black

Files were modified by this hook. Additional output:

reformatted /data/prj/python/git/packaging/packaging/_AIX_platform.py
All done! \u2728 \U0001f370 \u2728
1 file reformatted, 24 files left unchanged.

Merci!

@aixtools
Copy link
Author

@aixtools aixtools commented Nov 10, 2019

Some help - with mypi - is much appreciated. I just go in circles. If I understand correctly, there is some 'science' that can be done to satisfy mypi - but it is too much 'magic' for me at this point. I just go around in circles - same error, or new error. Very frustrating.

@pradyunsg
Copy link
Member

@pradyunsg pradyunsg commented Nov 10, 2019

@aixtools This should work:

diff --git a/packaging/_AIX_platform.py b/packaging/_AIX_platform.py
index 6d77aa0..64bd31e 100644
--- a/packaging/_AIX_platform.py
+++ b/packaging/_AIX_platform.py
@@ -17,12 +17,11 @@ def aix_tag(v, r, tl, bd):
 
 def aix_buildtag():
     # type: () -> str
-    bd = get_var("AIX_BUILDDATE")  # type: str
-    # because I do not understand howto tell mypi that
-    # get_var() returns a string _ I getting it to a temp variable
-    # and using that as argument to split()
-    # v, r, tl, f = get_var("BUILD_GNU_TYPE").split(".")
-    gt = get_var("BUILD_GNU_TYPE")  # type: str
+    bd = get_var("AIX_BUILDDATE")
+    assert bd
+
+    gt = get_var("BUILD_GNU_TYPE")
+    assert gt
     v, r, tl, f = gt.split(".")
     return aix_tag(v[-1], r, tl, bd)
 
@@ -47,7 +46,7 @@ def aix_pep425():
     # p = run(["/usr/bin/lslpp", "-Lqc", "bos.mp64"],
     #       capture_output=True, text=True)
     result = check_output(["/usr/bin/lslpp", "-Lqc", "bos.mp64"])
-    result = result.decode("utf-8").strip().split(":")  # type : ignore
+    result = result.decode("utf-8").strip().split(":")  # type: ignore
 
     (lpp, vrmf, bd) = list(result[index] for index in [0, 2, -1])
     assert lpp == "bos.mp64", "%s != %s" % (lpp, "bos.mp64")

I don't have much bandwidth to comment further or do a review, but checking locally, this seems to fix the mypy warnings.

@aixtools
Copy link
Author

@aixtools aixtools commented Nov 11, 2019

@pradyunsg Thx. Shall give it a try. As mypy was still failing I had not bothered to post my latest changes. So. I'll stash and try again.

@aixtools
Copy link
Author

@aixtools aixtools commented Nov 11, 2019

@pradyunsg - success for mypy, but tox -e py37 fails, I guess on coverage. Again, a result I know nothing about.

So, packaging/_AIX_platform.py has 0% coverage, and packaging/tags.py drops to 98%.

Again, hints welcome (I'll push these changes, even though failure is expected).


py37 run-test: commands[1] | python -m coverage report -m --fail-under 100
Name                         Stmts   Miss Branch BrPart  Cover   Missing
------------------------------------------------------------------------
packaging/_AIX_platform.py      24     24      4      0     0%   3-62
packaging/__about__.py          10      0      0      0   100%
packaging/__init__.py            3      0      0      0   100%
packaging/_structures.py        41      0      0      0   100%
packaging/_typing.py             3      0      0      0   100%
packaging/markers.py           133      0     49      0   100%
packaging/requirements.py       79      0     24      0   100%
packaging/specifiers.py        275      0    124      0   100%
packaging/tags.py              259      6    125      1    98%   491-498, 490->491
packaging/utils.py              25      0     14      0   100%
packaging/version.py           189      0     88      0   100%
------------------------------------------------------------------------
TOTAL                         1041     30    428      1    97%
ERROR: InvocationError for command /data/prj/python/git/packaging/.tox/py37/bin/python -m coverage report -m --fail-under 100 (exited with code 2)
@xavfernandez
Copy link
Member

@xavfernandez xavfernandez commented Nov 11, 2019

@aixtools the packaging project has a policy to have 100% of its code tested so you'll need to write tests in https://github.com/pypa/packaging/blob/master/tests/test_tags.py exercising the code you wrote.

@aixtools
Copy link
Author

@aixtools aixtools commented Nov 11, 2019

I guessed as much, and have been playing.

But in all honesty, it has taken me 400% more time to try to understand, let alone accomplish, the additional tasks to get it past CI.

My approach has always been to insert as little as possible to the existing code - merely a trigger, such as platform.system == "AIX" and have all my contribution external to the existing project.

I suppose if your main interest in life is Python, you have either had all of these subjects in school, training at work, or just because you have done other projects. It feels (think something very negative) and also becomes extremely distracting. I have actually forgotten some of my ideas re: step by step progression. And now I fear something I had hoped might be accomplished in a few weeks may not be finished in a year - just because I run out of time between (work) trips - to get this past your requirements.

Well, time to get ready of tomorrow. I'll see when I can get back to this. Please do not close just because I lack time and/or skills to work with all these additional requirements. I shall not be updating the PR to test - I test locally as best I can and then push.

update: just checking (read hoping) for suggestions.

@aixtools
Copy link
Author

@aixtools aixtools commented Nov 20, 2019

I spent hours trying different things to get the last bit of code in packaging/tags.py to be covered, but I cannot find anything to convince the tool that the code is executing.

I can force errors and see that it gets reached.

After hours, stuck with this:

py37 run-test: commands[1] | python -m coverage report -m --fail-under 100
Name                         Stmts   Miss Branch BrPart  Cover   Missing
------------------------------------------------------------------------
packaging/_AIX_platform.py      25      0      2      0   100%
packaging/__about__.py          10      0      0      0   100%
packaging/__init__.py            3      0      0      0   100%
packaging/_structures.py        41      0      0      0   100%
packaging/_typing.py             3      0      0      0   100%
packaging/markers.py           133      0     49      0   100%
packaging/requirements.py       79      0     24      0   100%
packaging/specifiers.py        275      0    124      0   100%
packaging/tags.py              260      1    125      1    99%   459, 454->459
packaging/utils.py              25      0     14      0   100%
packaging/version.py           189      0     88      0   100%
------------------------------------------------------------------------
TOTAL                         1043      1    426      1    99%
@aixtools
Copy link
Author

@aixtools aixtools commented Nov 20, 2019

OMG!! They all passed! I guess I should celebrate - like when my daughter said - hey grandpa, it's a boy!

@aixtools aixtools changed the title Add AIX platform tag support to Tag class Modify AIX platform_tag so it provides PEP425 needs Nov 28, 2019
@aixtools
Copy link
Author

@aixtools aixtools commented Nov 28, 2019

Hmm - says I left a review. If so, by accident.

Rebase, etc. etc. is making a big mess. Not a slight change.... Bah conflicts!

@chrahunt
Copy link
Member

@chrahunt chrahunt commented Dec 11, 2019

The required test has passed, and I would gladly do the extra work to get it to pass on Windows - but I do not have a Windows system to emulate the test process, and pushing guesses is just messy.

For what it's worth, Microsoft offers free evaluation VM images with Windows here. That's what I use personally for any Windows-related open source work.

@aixtools
Copy link
Author

@aixtools aixtools commented Dec 12, 2019

Thanks. I'll have to figure out how to set all of that up though. Time...

atm, though, it seems master does not pass it's own tests. That is, at least my conclusion based on what I see re: #244 - where I seemed to have hit it off all wrong.

Happy Holidays all.

@aixtools
Copy link
Author

@aixtools aixtools commented Dec 16, 2019

Please note: CI failures are not related to the changes in this PR.

@aixtools
Copy link
Author

@aixtools aixtools commented Dec 17, 2019

CI failures are not related to the changes in this PR.

@aixtools
Copy link
Author

@aixtools aixtools commented Dec 26, 2019

9 days further. Any comments? I cannot resolve the CI issues - as they have nothing to do with my proposal.

@aixtools
Copy link
Author

@aixtools aixtools commented Feb 2, 2020

Question: What happens if I press the button "Update Branch"?

@aixtools
Copy link
Author

@aixtools aixtools commented Feb 3, 2020

@di Thanks for the help (magic!)

aixtools and others added 8 commits Mar 17, 2020
Remove all references to old name _AIX_platform.py
Rename module used for monkeypatch
* Make our mypy type-casting magic more robust

Newer versions of mypy are better at following code flow, and assign
Any types to our reimplemented cast function. This commit significantly
restructures our typing.cast re-definition, to make it more robust.

* Rename MYPY_CHECK_RUNNING -> TYPE_CHECKING

This allows for some significant cleanups in our typing helpers, and
makes it much easier to document our... workarounds.

* Bump to newer mypy

* Improve `canonicalize_name` safety with type hints
@aixtools aixtools force-pushed the aixtools:AIX-platform-tag branch from add56af to 68e9c80 Apr 20, 2020
aixtools added 2 commits Apr 20, 2020
@aixtools
Copy link
Author

@aixtools aixtools commented Apr 20, 2020

Well, changes to codebase - seems what was working no longer works. Will take this up again later when I have more time to investigagte what, exactly, I need to update on my system so I can run these tests locally.

aixtools added 3 commits Nov 1, 2020
Additional conflicts in coding expected due to changes in `master`.
@aixtools
Copy link
Author

@aixtools aixtools commented Nov 3, 2020

working through typos - slowly.

tox no arguments - passes

nox and nox -s lint both fail as:

nox > Running session lint
nox > Session lint skipped: Python interpreter 3.8 not found.

As I am working - as before - with py36 - what do I need to do extra to use nox et al, locally.

Thx.

@aixtools
Copy link
Author

@aixtools aixtools commented Nov 3, 2020

OK. Next step:

One of my new tests looks like:

def test_aix_bgt(monkeypatch):
    monkeypatch.setattr(_aix_support, "_have_subprocess", False)
    result = _aix_support._aix_bgt()
    assert result == [6, 1, 7]
    monkeypatch.setattr(_aix_support, "_have_subprocess", True)
    monkeypatch.setattr(sysconfig, "get_config_var", lambda k: "powerpc-ibm-aix5.3.7.0")
    result = _aix_support._aix_bgt()
    assert result == [5, 3, 7]

Something seems to have changed - because a year ago - the sysconfig.get_config_var('GNU_BUILD_TYPE') would return
"powerpc-ibm-aix5.3.7.0" and the strip command would work.

    # type: () -> List[int]
    if _have_subprocess:
        bgt = get_config_var("BUILD_GNU_TYPE")
    else:
        bgt = "powerpc-ibm-aix6.1.7.0"
    return _aix_vrtl(vrmf=str(bgt))

The test now fails - because the monkeypatch value is not getting pickup.

What do I need to change in:

monkeypatch.setattr(sysconfig, "get_config_var", lambda k: "powerpc-ibm-aix5.3.7.0")

so that the correct value is returned.

FYI: the current python returns (when _have_subprocess is TRUE)

Python 3.6.12 (default, Sep 23 2020, 08:27:01) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_var("BUILD_GNU_TYPE")
'powerpc-ibm-aix5.3.12.0'
>>>

Not the value shown in the CI test:


tests/test_tags.py:1297: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
packaging/_aix_support.py:86: in _aix_bgt
    return _aix_vrtl(vrmf=str(bgt))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

vrmf = 'x86_64-pc-linux-gnu'

    def _aix_vrtl(vrmf):
        # type: (str) -> List[int]
>       v, r, tl = vrmf.split(".")[:3]
E       ValueError: not enough values to unpack (expected 3, got 1)

Many thanks for your assistence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants
You can’t perform that action at this time.