Skip to content

BUG: Fix from_float_positional errors for huge pads #28149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
seberg merged 9 commits into from
Jan 21, 2025
Merged

BUG: Fix from_float_positional errors for huge pads #28149

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
32b5ccb
DOC: added limits to pad_left, pad_right
yakovdan Jan 11, 2025
1e4fe5c
TST: Added tests for correct handling of overflow
yakovdan Jan 11, 2025
b530a08
BUG: fixed pad_left and pad_right causing overflow if too large
yakovdan Jan 11, 2025
e5e13e3
DOC: modified doc for simpler exception
yakovdan Jan 15, 2025
82d1481
TST: added overflow test and fixed formatting
yakovdan Jan 15, 2025
3d85e35
BUG: fixed overflow checks and simplified error handling
yakovdan Jan 15, 2025
b02ef67
BUG: rewritten excpetion message and fixed overflow check
yakovdan Jan 18, 2025
fa0273d
TST: split test into smaller tests, added large input value
yakovdan Jan 18, 2025
6e6882b
Apply suggestions from code review
seberg Jan 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions numpy/_core/arrayprint.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1232,10 +1232,14 @@ def format_float_positional(x, precision=None, unique=True,
Whether to show the sign for positive values.
pad_left : non-negative integer, optional
Pad the left side of the string with whitespace until at least that
many characters are to the left of the decimal point.
many characters are to the left of the decimal point. Total length of
string representation of the floating point value cannot exceed 16384,
including pad_right, if any.

@seberg seberg Jan 13, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel this is a lot of detail, nobody should ever see. If we want it, maybe we should put it into the Exceptions section.

@yakovdan yakovdan Jan 15, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've simplified this part and moved to a separate section below. Is this what you've meant?

pad_right : non-negative integer, optional
Pad the right side of the string with whitespace until at least that
many characters are to the right of the decimal point.
many characters are to the right of the decimal point. Total length of
string representation of the floating point value cannot exceed 16384,
including pad_left, if any.
min_digits : non-negative integer or None, optional
Minimum number of digits to print. Only has an effect if `unique=True`
in which case additional digits past those necessary to uniquely
Expand Down
86 changes: 65 additions & 21 deletions numpy/_core/src/multiarray/dragon4.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1614,8 +1614,15 @@ typedef struct Dragon4_Options {
* hasUnequalMargins - is the high margin twice as large as the low margin
*
* See Dragon4_Options for description of remaining arguments.
*
* Required left or right padding may exceed the size of the string buffer.
* Error codes for those conditions are defined below
Comment thread
seberg marked this conversation as resolved.
Outdated
*/
static npy_uint32

#define RIGHT_PADDING_OVERFLOW -1
#define LEFT_PADDING_OVERFLOW -2

static npy_int32
FormatPositional(char *buffer, npy_uint32 bufferSize, BigInt *mantissa,
npy_int32 exponent, char signbit, npy_uint32 mantissaBit,
npy_bool hasUnequalMargins, DigitMode digit_mode,
Expand Down Expand Up @@ -1646,7 +1653,7 @@ FormatPositional(char *buffer, npy_uint32 bufferSize, BigInt *mantissa,
buffer[pos++] = '-';
has_sign = 1;
}

numDigits = Dragon4(mantissa, exponent, mantissaBit, hasUnequalMargins,
digit_mode, cutoff_mode, precision, min_digits,
buffer + has_sign, maxPrintLen - has_sign,
Expand All @@ -1658,7 +1665,7 @@ FormatPositional(char *buffer, npy_uint32 bufferSize, BigInt *mantissa,
/* if output has a whole number */
if (printExponent >= 0) {
/* leave the whole number at the start of the buffer */
numWholeDigits = printExponent+1;
numWholeDigits = printExponent+1;
if (numDigits <= numWholeDigits) {
npy_int32 count = numWholeDigits - numDigits;
pos += numDigits;
Expand Down Expand Up @@ -1803,38 +1810,57 @@ FormatPositional(char *buffer, npy_uint32 bufferSize, BigInt *mantissa,

/* add any whitespace padding to right side */
if (digits_right >= numFractionDigits) {
npy_bool right_overflow = NPY_FALSE;
npy_int32 count = digits_right - numFractionDigits;

/* in trim_mode DptZeros, if right padding, add a space for the . */
if (trim_mode == TrimMode_DptZeros && numFractionDigits == 0
&& pos < maxPrintLen) {
buffer[pos++] = ' ';
}


/* provided digits_rights is too large, can't create the string required*/
if (pos + count > maxPrintLen) {

@seberg seberg Jan 13, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect this (and likely others) can still overflow for ridiculous inputs like 2**31-1.

Suggested change
if (pos + count > maxPrintLen) {
if (count > maxPrintLen - pos) {

re-arranging it should keep us safe?

@yakovdan yakovdan Jan 15, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. I've fixed this where it's relevant.

right_overflow = NPY_TRUE;

@seberg seberg Jan 13, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why continue and not set the error and return here right away, we shouldn't use the result, so it doesn't matter if the buffer is null terminated.

@yakovdan yakovdan Jan 15, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed according to your suggestion

count = maxPrintLen - pos;
}

for ( ; count > 0; count--) {
buffer[pos++] = ' ';
}

/* handle overflow condition safely*/
if (NPY_TRUE == right_overflow) {
buffer[pos] = '\0';
return RIGHT_PADDING_OVERFLOW;

@seberg seberg Jan 13, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there is any reason to have multiple error returns here. You can just set the Python error here and return -1 as the only error value.

@yakovdan yakovdan Jan 15, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, fixed

}
}
/* add any whitespace padding to left side */
if (digits_left > numWholeDigits + has_sign) {
npy_bool left_overflow = NPY_FALSE;
npy_int32 shift = digits_left - (numWholeDigits + has_sign);
npy_int32 count = pos;


/* provided digits_left is too large, can't create the string required*/

@seberg seberg Jan 16, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think these comments are needed here because with the Python error text, the code reason is just as clear already, IMO. So I would slightly prefer if you remove them again.

@yakovdan yakovdan Jan 18, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, removed comment

if (count + shift > maxPrintLen) {

@seberg seberg Jan 16, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is similar to above shift is unbounded in size, so count + shift might overflow. So rephrase as count > maPrintLen - shift.

count = maxPrintLen - shift;
shift = maxPrintLen - count;
left_overflow = NPY_TRUE;
}

if (count > 0) {
memmove(buffer + shift, buffer, count);
}

pos = shift + count;
for ( ; shift > 0; shift--) {
buffer[shift - 1] = ' ';
}

/* handle overflow condition safely*/
if (NPY_TRUE == left_overflow) {
buffer[pos] = '\0';
return LEFT_PADDING_OVERFLOW;
}
}

/* terminate the buffer */
Expand All @@ -1860,7 +1886,7 @@ FormatPositional(char *buffer, npy_uint32 bufferSize, BigInt *mantissa,
*
* See Dragon4_Options for description of remaining arguments.
*/
static npy_uint32
static npy_int32
Comment thread
seberg marked this conversation as resolved.
FormatScientific (char *buffer, npy_uint32 bufferSize, BigInt *mantissa,
npy_int32 exponent, char signbit, npy_uint32 mantissaBit,
npy_bool hasUnequalMargins, DigitMode digit_mode,
Expand Down Expand Up @@ -2158,7 +2184,7 @@ PrintInfNan(char *buffer, npy_uint32 bufferSize, npy_uint64 mantissa,
* Helper function that takes Dragon4 parameters and options and
* calls Dragon4.
*/
static npy_uint32
static npy_int32
Format_floatbits(char *buffer, npy_uint32 bufferSize, BigInt *mantissa,
npy_int32 exponent, char signbit, npy_uint32 mantissaBit,
npy_bool hasUnequalMargins, Dragon4_Options *opt)
Expand Down Expand Up @@ -2187,7 +2213,7 @@ Format_floatbits(char *buffer, npy_uint32 bufferSize, BigInt *mantissa,
* exponent: 5 bits
* mantissa: 10 bits
*/
static npy_uint32
static npy_int32
Dragon4_PrintFloat_IEEE_binary16(
npy_half *value, Dragon4_Options *opt)
{
Expand Down Expand Up @@ -2274,7 +2300,7 @@ Dragon4_PrintFloat_IEEE_binary16(
* exponent: 8 bits
* mantissa: 23 bits
*/
static npy_uint32
static npy_int32
Dragon4_PrintFloat_IEEE_binary32(
npy_float32 *value,
Dragon4_Options *opt)
Expand Down Expand Up @@ -2367,7 +2393,7 @@ Dragon4_PrintFloat_IEEE_binary32(
* exponent: 11 bits
* mantissa: 52 bits
*/
static npy_uint32
static npy_int32
Dragon4_PrintFloat_IEEE_binary64(
npy_float64 *value, Dragon4_Options *opt)
{
Expand Down Expand Up @@ -2482,7 +2508,7 @@ typedef struct FloatVal128 {
* intbit 1 bit, first u64
* mantissa: 63 bits, first u64
*/
static npy_uint32
static npy_int32
Dragon4_PrintFloat_Intel_extended(
FloatVal128 value, Dragon4_Options *opt)
{
Expand Down Expand Up @@ -2580,7 +2606,7 @@ Dragon4_PrintFloat_Intel_extended(
* system. But numpy defines NPY_FLOAT80, so if we come across it, assume it is
* an Intel extended format.
*/
static npy_uint32
static npy_int32
Dragon4_PrintFloat_Intel_extended80(
npy_float80 *value, Dragon4_Options *opt)
{
Expand All @@ -2604,7 +2630,7 @@ Dragon4_PrintFloat_Intel_extended80(

#ifdef HAVE_LDOUBLE_INTEL_EXTENDED_12_BYTES_LE
/* Intel's 80-bit IEEE extended precision format, 96-bit storage */
static npy_uint32
static npy_int32
Dragon4_PrintFloat_Intel_extended96(
npy_float96 *value, Dragon4_Options *opt)
{
Expand All @@ -2628,7 +2654,7 @@ Dragon4_PrintFloat_Intel_extended96(

#ifdef HAVE_LDOUBLE_MOTOROLA_EXTENDED_12_BYTES_BE
/* Motorola Big-endian equivalent of the Intel-extended 96 fp format */
static npy_uint32
static npy_int32
Dragon4_PrintFloat_Motorola_extended96(
npy_float96 *value, Dragon4_Options *opt)
{
Expand Down Expand Up @@ -2665,7 +2691,7 @@ typedef union FloatUnion128

#ifdef HAVE_LDOUBLE_INTEL_EXTENDED_16_BYTES_LE
/* Intel's 80-bit IEEE extended precision format, 128-bit storage */
static npy_uint32
static npy_int32
Dragon4_PrintFloat_Intel_extended128(
npy_float128 *value, Dragon4_Options *opt)
{
Expand Down Expand Up @@ -2694,7 +2720,7 @@ Dragon4_PrintFloat_Intel_extended128(
* I am not sure if the arch also supports uint128, and C does not seem to
* support int128 literals. So we use uint64 to do manipulation.
*/
static npy_uint32
static npy_int32
Dragon4_PrintFloat_IEEE_binary128(
FloatVal128 val128, Dragon4_Options *opt)
{
Expand Down Expand Up @@ -2779,7 +2805,7 @@ Dragon4_PrintFloat_IEEE_binary128(
}

#if defined(HAVE_LDOUBLE_IEEE_QUAD_LE)
static npy_uint32
static npy_int32
Dragon4_PrintFloat_IEEE_binary128_le(
npy_float128 *value, Dragon4_Options *opt)
{
Expand All @@ -2799,7 +2825,7 @@ Dragon4_PrintFloat_IEEE_binary128_le(
* This function is untested, very few, if any, architectures implement
* big endian IEEE binary128 floating point.
*/
static npy_uint32
static npy_int32
Dragon4_PrintFloat_IEEE_binary128_be(
npy_float128 *value, Dragon4_Options *opt)
{
Expand Down Expand Up @@ -2854,7 +2880,7 @@ Dragon4_PrintFloat_IEEE_binary128_be(
* https://gcc.gnu.org/wiki/Ieee128PowerPCA
* https://www.ibm.com/support/knowledgecenter/en/ssw_aix_71/com.ibm.aix.genprogc/128bit_long_double_floating-point_datatype.htm
*/
static npy_uint32
static npy_int32
Dragon4_PrintFloat_IBM_double_double(
npy_float128 *value, Dragon4_Options *opt)
{
Expand Down Expand Up @@ -3041,13 +3067,31 @@ Dragon4_PrintFloat_IBM_double_double(
* which goes up to about 10^4932. The Dragon4_scratch struct provides a string
* buffer of this size.
*/

#define make_dragon4_typefuncs_inner(Type, npy_type, format) \
\
PyObject *\
Dragon4_Positional_##Type##_opt(npy_type *val, Dragon4_Options *opt)\
{\
PyObject *ret;\
if (Dragon4_PrintFloat_##format(val, opt) < 0) {\
npy_int32 status = Dragon4_PrintFloat_##format(val, opt);\
if (status < 0) { \
const char *error_msg; \
PyObject *error_type = PyExc_ValueError;\
\
switch (status) { \
case RIGHT_PADDING_OVERFLOW:\
error_msg = "Right padding exceeds buffer size of 16384";\
break;\
case LEFT_PADDING_OVERFLOW:\
error_msg = "Left padding exceeds buffer size of 16384";\
break;\
default:\
error_type = PyExc_RuntimeError;\
error_msg = "An unexpected error occurred";\
break;\
}\
PyErr_SetString(error_type, error_msg);\

@seberg seberg Jan 13, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we set the error above, this code can stay unchanged. Also Dragon4_PrintFloat_##format is used below a second time and both must stay aligned.

We should make sure there is a test for both calls.

@yakovdan yakovdan Jan 15, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reverted the code for this function and and switched to raising errors in FormatPositional

return NULL;\
}\
ret = PyUnicode_FromString(_bigint_static.repr);\
Expand Down
8 changes: 6 additions & 2 deletions numpy/_core/tests/test_scalarprint.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@

from tempfile import TemporaryFile
import numpy as np
from numpy.testing import assert_, assert_equal, assert_raises, IS_MUSL
from numpy.testing import assert_, assert_equal, assert_raises
from numpy.testing import assert_raises_regex, IS_MUSL
Comment thread
seberg marked this conversation as resolved.
Outdated

class TestRealScalars:
def test_str(self):
Expand Down Expand Up @@ -273,7 +274,7 @@ def test_dragon4_interface(self):
assert_equal(fpos(tp('-1.0'), pad_left=4, pad_right=4), " -1. ")
assert_equal(fpos(tp('-10.2'),
pad_left=4, pad_right=4), " -10.2 ")

# test exp_digits
assert_equal(fsci(tp('1.23e1'), exp_digits=5), "1.23e+00001")

Expand Down Expand Up @@ -305,6 +306,9 @@ def test_dragon4_interface(self):
"1.2" if tp != np.float16 else "1.2002")
assert_equal(fpos(tp('1.'), trim='-'), "1")
assert_equal(fpos(tp('1.001'), precision=1, trim='-'), "1")

assert_raises_regex(ValueError, "Left padding exceeds buffer size of 16384", fpos, tp('1.047'), precision=2, pad_left=int(1e5))
assert_raises_regex(ValueError, "Right padding exceeds buffer size of 16384", fpos, tp('1.047'), precision=2, pad_right=int(1e5))

@seberg seberg Jan 13, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@charris what is our preferred line length? It seems a bit that the new linter setup ignores line-length?

For here: Would be nice to break it. That is easiest if you use the nicer pattern of:

with pytest.raises(..., match="..."):

(there is a lot of code not using it, but that is mostly because it predates pytest.)

@yakovdan yakovdan Jan 15, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've split it into several lines to match the expected line-length

@charris charris Jan 16, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seberg Preferred line length is now <= 88 for Python code. Clang-format is still at 80, but we should probably increase that. Linux is, IIRC, now <= 90 suggested, but not inforced.

@seberg seberg Jan 16, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the thing is that the linter job should have complained: I guess it just means adding a specific line length somewhere.


@pytest.mark.skipif(not platform.machine().startswith("ppc64"),
reason="only applies to ppc float128 values")
Expand Down