Skip to content

sigma_clip() does not work properly with Masked (MaskedNDArray / MaskedQuantity) #13200

Description

@orionlee

Description

sigma_clip() does not work properly with Masked (MaskedNDArray / MaskedQuantity) .

  • primary: the mask is ignored.
  • secondary: when bottleneck is installed and the data is MaskedQuantity, it fails with ValueError in some cases.

Primary: the mask is ignored.

Steps to Reproduce:

In [11]: from astropy.stats.sigma_clipping import sigma_clip
    ...: from astropy.utils.masked import Masked
    ...: from astropy import units as u
    ...: 
    ...: # MaskedNDArray: the mask is ignored by sigma_clip()
    ...: flux = Masked([1., 2., 300.], mask=[False, True, True])

In [12]: sigma_clip(flux)
Out[12]: MaskedNDArray([  1.,   2., 300.])

# np.ma.MaskedArray is handled properly
In [13]: sigma_clip(np.ma.MaskedArray(flux))
Out[13]: 
masked_array(data=[1.0, --, --],
             mask=[False,  True,  True],
       fill_value=1e+20)

Secondary: bottleneck + MaskedQuantity

When bottleneck is installed, sigma_clip() fails with ValueError, when MaskedQuantity is supplied in some cases.

The precise condition for the failure is not yet determined. In the isolated cases tried, they fail consistently.
Yet it also works in some real life example (e.g., the flux data for TESS lightcurve fits files, which are read as MaskedQuantity as well).

Steps to Reproduce: See the following snippets.
I've also tried some variations, they all fail in the same way.

  • change whether there are actual masked values in flux instances.
  • change the unit of the MaskedQuantity.
from astropy.stats.sigma_clipping import sigma_clip
from astropy.utils.masked import Masked
from astropy import units as u

# MaskedNDArray: it works
flux = Masked([1., 2., 3.], mask=[False, False, False])
clipped = sigma_clip(data=flux, sigma=5)

# MaskedQuantity: it fails with ValueError
flux_q = flux * u.percent
clipped_q = sigma_clip(data=flux_q, sigma=5)

The error:

ValueError                                Traceback (most recent call last)
Input In [626], in <cell line: 12>()
     10 # MaskedQuantity: it fails with ValueError
     11 flux_q = flux * u.percent
---> 12 clipped_q = sigma_clip(data=flux_q, sigma=5)
     13 print(clipped_q)

File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\stats\sigma_clipping.py:835, in sigma_clip(data, sigma, sigma_lower, sigma_upper, maxiters, cenfunc, stdfunc, axis, masked, return_bounds, copy, grow)
    650 """
    651 Perform sigma-clipping on the provided data.
    652 
   (...)
    829 standard deviation is higher.
    830 """
    831 sigclip = SigmaClip(sigma=sigma, sigma_lower=sigma_lower,
    832                     sigma_upper=sigma_upper, maxiters=maxiters,
    833                     cenfunc=cenfunc, stdfunc=stdfunc, grow=grow)
--> 835 return sigclip(data, axis=axis, masked=masked,
    836                return_bounds=return_bounds, copy=copy)

File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\stats\sigma_clipping.py:638, in SigmaClip.__call__(self, data, axis, masked, return_bounds, copy)
    633 # These two cases are treated separately because when
    634 # ``axis=None`` we can simply remove clipped values from the
    635 # array. This is not possible when ``axis`` or ``grow`` is
    636 # specified.
    637 if axis is None and not self.grow:
--> 638     return self._sigmaclip_noaxis(data, masked=masked,
    639                                   return_bounds=return_bounds,
    640                                   copy=copy)
    641 else:
    642     return self._sigmaclip_withaxis(data, axis=axis, masked=masked,
    643                                     return_bounds=return_bounds,
    644                                     copy=copy)

File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\stats\sigma_clipping.py:420, in SigmaClip._sigmaclip_noaxis(self, data, masked, return_bounds, copy)
    418 iteration += 1
    419 size = filtered_data.size
--> 420 self._compute_bounds(filtered_data, axis=None)
    421 filtered_data = filtered_data[
    422     (filtered_data >= self._min_value)
    423     & (filtered_data <= self._max_value)]
    424 nchanged = size - filtered_data.size

File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\stats\sigma_clipping.py:302, in SigmaClip._compute_bounds(self, data, axis)
    300 with warnings.catch_warnings():
    301     warnings.simplefilter("ignore", category=RuntimeWarning)
--> 302     self._max_value = self._cenfunc_parsed(data, axis=axis)
    303     std = self._stdfunc_parsed(data, axis=axis)
    304     self._min_value = self._max_value - (std * self.sigma_lower)

File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\stats\sigma_clipping.py:69, in _nanmedian(array, axis)
     66     axis = 0
     68 if isinstance(array, Quantity):
---> 69     return array.__array_wrap__(bottleneck.nanmedian(array, axis=axis))
     70 else:
     71     return bottleneck.nanmedian(array, axis=axis)

File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\units\quantity.py:562, in Quantity.__array_wrap__(self, obj, context)
    557 def __array_wrap__(self, obj, context=None):
    559     if context is None:
    560         # Methods like .squeeze() created a new `ndarray` and then call
    561         # __array_wrap__ to turn the array into self's subclass.
--> 562         return self._new_view(obj)
    564     raise NotImplementedError('__array_wrap__ should not be used '
    565                               'with a context any more since all use '
    566                               'should go through array_function. '
    567                               'Please raise an issue on '
    568                               'https://github.com/astropy/astropy')

File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\units\quantity.py:745, in Quantity._new_view(self, obj, unit)
    743 view = obj.view(quantity_subclass)
    744 view._set_unit(unit)
--> 745 view.__array_finalize__(self)
    746 return view

File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\units\quantity.py:539, in Quantity.__array_finalize__(self, obj)
    537 super_array_finalize = super().__array_finalize__
    538 if super_array_finalize is not None:
--> 539     super_array_finalize(obj)
    541 # If we're a new object or viewing an ndarray, nothing has to be done.
    542 if obj is None or obj.__class__ is np.ndarray:

File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\utils\masked\core.py:572, in MaskedNDArray.__array_finalize__(self, obj)
    567     super_array_finalize(obj)
    569 if self._mask is None:
    570     # Got here after, e.g., a view of another masked class.
    571     # Get its mask, or initialize ours.
--> 572     self._set_mask(getattr(obj, '_mask', False))
    574 if 'info' in obj.__dict__:
    575     self.info = obj.info

File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\utils\masked\core.py:228, in Masked._set_mask(self, mask, copy)
    225 if ma.shape != self.shape:
    226     # This will fail (correctly) if not broadcastable.
    227     self._mask = np.empty(self.shape, dtype=mask_dtype)
--> 228     self._mask[...] = ma
    229 elif ma is mask:
    230     # Even if not copying use a view so that shape setting
    231     # does not propagate.
    232     self._mask = mask.copy() if copy else mask.view()

ValueError: could not broadcast input array from shape (3,) into shape ()

System Details

Also tested on current main (astropy 5.1.dev786+gf54df7643). The error is still there.

Windows-10-10.0.22000-SP0
Python 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:21:54) [MSC v.1929 64 bit (AMD64)]
Numpy 1.22.3
pyerfa 2.0.0.1
astropy 5.0.4
Scipy 1.8.0
Matplotlib 3.5.1

Bottleneck:

bottleneck 1.3.4

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions