Description
sigma_clip() does not work properly with Masked (MaskedNDArray / MaskedQuantity) .
- primary: the mask is ignored.
- secondary: when
bottleneck is installed and the data is MaskedQuantity, it fails with ValueError in some cases.
Primary: the mask is ignored.
Steps to Reproduce:
In [11]: from astropy.stats.sigma_clipping import sigma_clip
...: from astropy.utils.masked import Masked
...: from astropy import units as u
...:
...: # MaskedNDArray: the mask is ignored by sigma_clip()
...: flux = Masked([1., 2., 300.], mask=[False, True, True])
In [12]: sigma_clip(flux)
Out[12]: MaskedNDArray([ 1., 2., 300.])
# np.ma.MaskedArray is handled properly
In [13]: sigma_clip(np.ma.MaskedArray(flux))
Out[13]:
masked_array(data=[1.0, --, --],
mask=[False, True, True],
fill_value=1e+20)
Secondary: bottleneck + MaskedQuantity
When bottleneck is installed, sigma_clip() fails with ValueError, when MaskedQuantity is supplied in some cases.
The precise condition for the failure is not yet determined. In the isolated cases tried, they fail consistently.
Yet it also works in some real life example (e.g., the flux data for TESS lightcurve fits files, which are read as MaskedQuantity as well).
Steps to Reproduce: See the following snippets.
I've also tried some variations, they all fail in the same way.
- change whether there are actual masked values in
flux instances.
- change the unit of the MaskedQuantity.
from astropy.stats.sigma_clipping import sigma_clip
from astropy.utils.masked import Masked
from astropy import units as u
# MaskedNDArray: it works
flux = Masked([1., 2., 3.], mask=[False, False, False])
clipped = sigma_clip(data=flux, sigma=5)
# MaskedQuantity: it fails with ValueError
flux_q = flux * u.percent
clipped_q = sigma_clip(data=flux_q, sigma=5)
The error:
ValueError Traceback (most recent call last)
Input In [626], in <cell line: 12>()
10 # MaskedQuantity: it fails with ValueError
11 flux_q = flux * u.percent
---> 12 clipped_q = sigma_clip(data=flux_q, sigma=5)
13 print(clipped_q)
File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\stats\sigma_clipping.py:835, in sigma_clip(data, sigma, sigma_lower, sigma_upper, maxiters, cenfunc, stdfunc, axis, masked, return_bounds, copy, grow)
650 """
651 Perform sigma-clipping on the provided data.
652
(...)
829 standard deviation is higher.
830 """
831 sigclip = SigmaClip(sigma=sigma, sigma_lower=sigma_lower,
832 sigma_upper=sigma_upper, maxiters=maxiters,
833 cenfunc=cenfunc, stdfunc=stdfunc, grow=grow)
--> 835 return sigclip(data, axis=axis, masked=masked,
836 return_bounds=return_bounds, copy=copy)
File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\stats\sigma_clipping.py:638, in SigmaClip.__call__(self, data, axis, masked, return_bounds, copy)
633 # These two cases are treated separately because when
634 # ``axis=None`` we can simply remove clipped values from the
635 # array. This is not possible when ``axis`` or ``grow`` is
636 # specified.
637 if axis is None and not self.grow:
--> 638 return self._sigmaclip_noaxis(data, masked=masked,
639 return_bounds=return_bounds,
640 copy=copy)
641 else:
642 return self._sigmaclip_withaxis(data, axis=axis, masked=masked,
643 return_bounds=return_bounds,
644 copy=copy)
File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\stats\sigma_clipping.py:420, in SigmaClip._sigmaclip_noaxis(self, data, masked, return_bounds, copy)
418 iteration += 1
419 size = filtered_data.size
--> 420 self._compute_bounds(filtered_data, axis=None)
421 filtered_data = filtered_data[
422 (filtered_data >= self._min_value)
423 & (filtered_data <= self._max_value)]
424 nchanged = size - filtered_data.size
File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\stats\sigma_clipping.py:302, in SigmaClip._compute_bounds(self, data, axis)
300 with warnings.catch_warnings():
301 warnings.simplefilter("ignore", category=RuntimeWarning)
--> 302 self._max_value = self._cenfunc_parsed(data, axis=axis)
303 std = self._stdfunc_parsed(data, axis=axis)
304 self._min_value = self._max_value - (std * self.sigma_lower)
File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\stats\sigma_clipping.py:69, in _nanmedian(array, axis)
66 axis = 0
68 if isinstance(array, Quantity):
---> 69 return array.__array_wrap__(bottleneck.nanmedian(array, axis=axis))
70 else:
71 return bottleneck.nanmedian(array, axis=axis)
File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\units\quantity.py:562, in Quantity.__array_wrap__(self, obj, context)
557 def __array_wrap__(self, obj, context=None):
559 if context is None:
560 # Methods like .squeeze() created a new `ndarray` and then call
561 # __array_wrap__ to turn the array into self's subclass.
--> 562 return self._new_view(obj)
564 raise NotImplementedError('__array_wrap__ should not be used '
565 'with a context any more since all use '
566 'should go through array_function. '
567 'Please raise an issue on '
568 'https://github.com/astropy/astropy')
File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\units\quantity.py:745, in Quantity._new_view(self, obj, unit)
743 view = obj.view(quantity_subclass)
744 view._set_unit(unit)
--> 745 view.__array_finalize__(self)
746 return view
File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\units\quantity.py:539, in Quantity.__array_finalize__(self, obj)
537 super_array_finalize = super().__array_finalize__
538 if super_array_finalize is not None:
--> 539 super_array_finalize(obj)
541 # If we're a new object or viewing an ndarray, nothing has to be done.
542 if obj is None or obj.__class__ is np.ndarray:
File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\utils\masked\core.py:572, in MaskedNDArray.__array_finalize__(self, obj)
567 super_array_finalize(obj)
569 if self._mask is None:
570 # Got here after, e.g., a view of another masked class.
571 # Get its mask, or initialize ours.
--> 572 self._set_mask(getattr(obj, '_mask', False))
574 if 'info' in obj.__dict__:
575 self.info = obj.info
File C:\pkg\_winNonPortables\Anaconda3\envs\lkv2_1_dev\lib\site-packages\astropy\utils\masked\core.py:228, in Masked._set_mask(self, mask, copy)
225 if ma.shape != self.shape:
226 # This will fail (correctly) if not broadcastable.
227 self._mask = np.empty(self.shape, dtype=mask_dtype)
--> 228 self._mask[...] = ma
229 elif ma is mask:
230 # Even if not copying use a view so that shape setting
231 # does not propagate.
232 self._mask = mask.copy() if copy else mask.view()
ValueError: could not broadcast input array from shape (3,) into shape ()
System Details
Also tested on current main (astropy 5.1.dev786+gf54df7643). The error is still there.
Windows-10-10.0.22000-SP0
Python 3.9.10 | packaged by conda-forge | (main, Feb 1 2022, 21:21:54) [MSC v.1929 64 bit (AMD64)]
Numpy 1.22.3
pyerfa 2.0.0.1
astropy 5.0.4
Scipy 1.8.0
Matplotlib 3.5.1
Bottleneck:
Description
sigma_clip() does not work properly with
Masked(MaskedNDArray/MaskedQuantity) .bottleneckis installed and the data isMaskedQuantity, it fails withValueErrorin some cases.Primary: the mask is ignored.
Steps to Reproduce:
Secondary:
bottleneck+MaskedQuantityWhen
bottleneckis installed, sigma_clip() fails withValueError, whenMaskedQuantityis supplied in some cases.The precise condition for the failure is not yet determined. In the isolated cases tried, they fail consistently.
Yet it also works in some real life example (e.g., the flux data for TESS lightcurve fits files, which are read as
MaskedQuantityas well).Steps to Reproduce: See the following snippets.
I've also tried some variations, they all fail in the same way.
fluxinstances.The error:
System Details
Also tested on current
main(astropy 5.1.dev786+gf54df7643). The error is still there.Bottleneck: