Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Plotting Int64 columns with nulled integers (NAType) fails #32073

Open
Khris777 opened this issue Feb 18, 2020 · 15 comments
Open

Plotting Int64 columns with nulled integers (NAType) fails #32073

Khris777 opened this issue Feb 18, 2020 · 15 comments

Comments

@Khris777
Copy link

@Khris777 Khris777 commented Feb 18, 2020

Code Sample, a copy-pastable example if possible

import pandas as pd
import numpy as np

df = pd.DataFrame({'A': [1, 2, 3, 4, 5], 'B': [7, 5, np.nan, 3, 2]})
df.plot(x='A', y='B')
df = df.astype('Int64')
df.plot(x='A', y='B')

Problem description

The first plotting command works, the second throws the error message

TypeError: float() argument must be a string or a number, not 'NAType'

Expected Output

NAType should be treated the same way as numpy nan in plotting. Maybe transformed on the fly?

(I'm unsure if this is a pandas, a numpy, or a matplotlib issue, I'm starting here)

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.6.final.0
python-bits : 64
OS : Windows
OS-release : 10
machine : AMD64
processor : Intel64 Family 6 Model 79 Stepping 1, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en
LOCALE : None.None

pandas : 1.0.1
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 45.2.0.post20200209
Cython : 0.29.15
pytest : None
hypothesis : None
sphinx : 2.4.1
blosc : None
feather : None
xlsxwriter : 1.2.7
lxml.etree : 4.5.0
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.1
IPython : 7.12.0
pandas_datareader: None
bs4 : 4.8.2
bottleneck : None
fastparquet : 0.3.3
gcsfs : None
lxml.etree : 4.5.0
matplotlib : 3.1.3
numexpr : None
odfpy : None
openpyxl : 3.0.3
pandas_gbq : None
pyarrow : 0.16.0
pytables : None
pytest : None
pyxlsb : None
s3fs : None
scipy : 1.4.1
sqlalchemy : 1.3.13
tables : None
tabulate : None
xarray : None
xlrd : 1.2.0
xlwt : None
xlsxwriter : 1.2.7
numba : 0.48.0

@TomAugspurger
Copy link
Contributor

@TomAugspurger TomAugspurger commented Feb 18, 2020

Can you post the full traceback?

@Khris777
Copy link
Author

@Khris777 Khris777 commented Feb 19, 2020

Gladly:

Traceback (most recent call last):

  File "C:\Users\My.Name\Documents\Python_Projects\GIT\miscellaneous\temp.py", line 14, in <module>
    df.plot(x='A', y='B')

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\pandas\plotting\_core.py", line 847, in __call__
    return plot_backend.plot(data, kind=kind, **kwargs)

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\pandas\plotting\_matplotlib\__init__.py", line 61, in plot
    plot_obj.generate()

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\pandas\plotting\_matplotlib\core.py", line 263, in generate
    self._make_plot()

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\pandas\plotting\_matplotlib\core.py", line 1085, in _make_plot
    **kwds,

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\pandas\plotting\_matplotlib\core.py", line 1104, in _plot
    lines = MPLPlot._plot(ax, x, y_values, style=style, **kwds)

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\pandas\plotting\_matplotlib\converter.py", line 66, in wrapper
    return func(*args, **kwargs)

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\pandas\plotting\_matplotlib\core.py", line 656, in _plot
    return ax.plot(*args, **kwds)

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\matplotlib\axes\_axes.py", line 1667, in plot
    self.add_line(line)

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\matplotlib\axes\_base.py", line 1902, in add_line
    self._update_line_limits(line)

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\matplotlib\axes\_base.py", line 1924, in _update_line_limits
    path = line.get_path()

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\matplotlib\lines.py", line 1027, in get_path
    self.recache()

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\matplotlib\lines.py", line 675, in recache
    y = _to_unmasked_float_array(yconv).ravel()

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\matplotlib\cbook\__init__.py", line 1388, in _to_unmasked_float_array
    return np.ma.asarray(x, float).filled(np.nan)

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\numpy\ma\core.py", line 7849, in asarray
    subok=False, order=order)

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\numpy\ma\core.py", line 2795, in __new__
    order=order, subok=True, ndmin=ndmin)

TypeError: float() argument must be a string or a number, not 'NAType'
@jorisvandenbossche
Copy link
Member

@jorisvandenbossche jorisvandenbossche commented Feb 19, 2020

We need to convert to floats with NaNs before passing the data to matplotlib, I suppose.

@jeandersonbc
Copy link
Contributor

@jeandersonbc jeandersonbc commented Feb 19, 2020

Hi! Can I make an attempt on this one? I'm looking for issues that can be useful to the community and help me to understand better how pandas works in depth :)

@jorisvandenbossche
Copy link
Member

@jorisvandenbossche jorisvandenbossche commented Feb 19, 2020

@jeandersonbc yes, that's welcome!
If you have any questions, don't hesitate to ask

@AnnaDaglis
Copy link
Contributor

@AnnaDaglis AnnaDaglis commented Feb 20, 2020

take

@jorisvandenbossche
Copy link
Member

@jorisvandenbossche jorisvandenbossche commented Feb 20, 2020

@AnnaDaglis someone else (@jeandersonbc) already just commented he would like to look at it, so I would first give him some time before taking this issue

@jeandersonbc
Copy link
Contributor

@jeandersonbc jeandersonbc commented Feb 21, 2020

Thanks @jorisvandenbossche, I didn't manage time to work on the issue since last time that I posted so that's why I didn't reply early.
As far as I can see, it is my understanding from your suggestion that there should be verification for NA values before calling plot_backend.plot(data, kind=kind, **kwargs), right? I'll be submitting a PR later today with some tests to see how it goes.

@jorisvandenbossche
Copy link
Member

@jorisvandenbossche jorisvandenbossche commented Feb 21, 2020

there should be verification for NA values before calling plot_backend.plot(data, kind=kind, **kwargs)

Yes, I think so. But the check might need to be done lower in the stack, as when calling plot_backend.plot(), data is still a dataframe. I think we can focus here on the matplotlib backend included in pandas, and thus we need to ensure converting to float dtype (with to_numpy(float, na_value=np.nan) if it is nullable integer dtype) before passing it to matplotlib's plot().

@AnnaDaglis AnnaDaglis removed their assignment Feb 21, 2020
@jeandersonbc
Copy link
Contributor

@jeandersonbc jeandersonbc commented Feb 21, 2020

take

jeandersonbc added a commit to jeandersonbc/pandas that referenced this issue Feb 28, 2020
jeandersonbc added a commit to jeandersonbc/pandas that referenced this issue Mar 1, 2020
jeandersonbc added a commit to jeandersonbc/pandas that referenced this issue Mar 1, 2020
jeandersonbc added a commit to jeandersonbc/pandas that referenced this issue Mar 1, 2020
@jeandersonbc
Copy link
Contributor

@jeandersonbc jeandersonbc commented Mar 2, 2020

So, I approached the problem by checking the numerical data in _compute_plot_data from matplot's backend. all tests pass, but I wonder if more tests should be added (e.g., at "pandas/tests/plotting/test_frame.py"). Any thoughts? Thanks already!

jeandersonbc added a commit to jeandersonbc/pandas that referenced this issue Mar 3, 2020
@MarcoGorelli
Copy link
Member

@MarcoGorelli MarcoGorelli commented May 22, 2020

Hi @AnnaDaglis - looks like the linked PR went stale (unfortunately), are you still interested in working on this?

@cvanweelden
Copy link
Contributor

@cvanweelden cvanweelden commented Jun 20, 2020

take

@cvanweelden
Copy link
Contributor

@cvanweelden cvanweelden commented Jul 7, 2020

I'm no longer working on this, as my minimal fix wasn't accepted and this might need a more structural solution for 3rd party lib operations on nullable types.

@vetedde
Copy link

@vetedde vetedde commented Jul 13, 2020

take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.