Skip to content

BUG: Fix all kinds of problems when itemsize == 0 - #8977

Merged
charris merged 9 commits into
numpy:masterfrom
eric-wieser:fix-elsize-0
Oct 2, 2017
Merged

charris merged 9 commits into
numpy:masterfrom
eric-wieser:fix-elsize-0

Conversation

@eric-wieser

Copy link
Copy Markdown
Member

Split off all of the bugfixes from #8970.

Comment thread numpy/core/src/multiarray/ctors.c Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the else clause and the new_data variable can be removed

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by removing the variable?

Fixing that feels kinda out of scope for this patch

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC, realloc will free the memory if size is zero, which we don't want to do. Worth checking, normally we keep a minimum of 1 byte to avoid problems.

@eric-wieser eric-wieser Apr 23, 2017 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sometimes, yes - but we don't actually care. All that matters is if realloc returns NULL, (which realloc(..., 0) does on some platforms, in which case we fail with a memory error.

Either way, we don't need to worry about this yet - this function has a check for elsize == 0, which we shouldn't remove unless #8970 is merged.

So yes, technically, this file didn't need a patch at all - but I'd argue that npy_mul_with_overflow_intp does a better job of communicating the intention (which is why this change is in a MAINT commit)

Comment thread numpy/core/src/multiarray/shape.c Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks scetchy as the == 0 check is removed.
It might not go in here as then newbytes and oldnbytes are 0 but it's still dangerous

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea here is that if the number of bytes are different, the element must be non empty. I guess rather than doing the division, we could just calculate this directly though

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm its probably good as is.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Patch incoming, I think it makes it cleaner

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also uncovered another unrelated bug

Comment thread numpy/core/src/multiarray/getset.c Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

our usual style is having the operator on the line before but indenting the next line

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That doesn't match with what I remember seeing, eg in mapping.c. Operators should be at the start of the line, so that they're not missed when reading through.

I can wrap after NPY_VOID if you'd like, but I'm not sure what level to indent to as a result

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The usual style (Python recommended) is trailing operators and two indents on the the next line too separate the boolean expression from the body.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there something like CPEP 8 for cpython style guidelines?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean PEP7?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also doc/C_STYLE_GUIDE.rst.txt. Here, second line just two indents from the if.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already fixed here, I think

@eric-wieser

Copy link
Copy Markdown
Member Author

Rebased to solve conflicts.

Comment thread numpy/core/src/multiarray/descriptor.c Outdated

@charris charris Oct 1, 2017 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can nbytes be -1? The inputs to npy_mul_with_overflow_int must be non-negative.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, that's pretty weird. This couldn't happen before either, could it?

Comment thread numpy/core/src/multiarray/getset.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the same reason that np.dtype(np.void) is np.dtype(np.void) is desirable. Without this special case, v0_arr.view(np.void).dtype is not np.void.

Comment thread numpy/core/src/multiarray/methods.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't size need to be >= 0 in order to use npy_mul_with_overflow_intp? Seems it should be checked first to avoid a function call.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could do something like

invalid = size < 0 || npy_mul_with_overflow_intp(...)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need, there's already a explicit a != 0 case in that function.

There's no point avoiding the function call, because optimizing the performance of empty arrays is pretty pointless

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see, the issue is size == -1. Maybe I'll just duplicate the below if.

Comment thread numpy/core/src/multiarray/methods.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems sensible, but why was it done differently before?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid recalculating num, most likely

Comment thread numpy/core/src/multiarray/shape.c Outdated

@charris charris Oct 1, 2017 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could do

sd = newnbytes ? newnbytes : elsize;

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would prefer to keep the == 0, because that's handling the special meaning of 0 which is "allow resizing".

Could flatten the if though

Comment thread numpy/core/src/multiarray/shape.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newnbytes - oldnbytes >= 0?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, guaranteed by the enclosing if

Comment thread numpy/core/tests/test_multiarray.py Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an advantage to making this a staticmethod?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To document that it doesn't use self?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, it does that, don't know if it is worth it as the normal impulse would be to use it only if needed. Then again, my Python experience started when Python 21 was the newest and greatest, so programming conventions may have changed. In fact, it is the latter possibility that interests me. Do you know if it is the case that this sort of programming style is being taught or encouraged?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TestCase objects (or the inferred use of them by pytest) are pretty weird anyway, because an new one is instantiated for each method call by the test runner. I can change this if you feel strongly.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't feel strongly about it. On occasion I find your coding style a bit unusual, but not wrong. Just trying to figure out if it is a personal style, or the future of python coding ;)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could easily be personal style. I guess it probably comes from my java background, but even there it might just have been personal style.

@charris

charris commented Oct 1, 2017

Copy link
Copy Markdown
Member

Looks pretty good, just a couple of questions.

@charris

charris commented Oct 1, 2017

Copy link
Copy Markdown
Member

A release note under "Improvements" would be helpful. Someone might want to know just when the new functionality became available.

@eric-wieser

Copy link
Copy Markdown
Member Author

Caught a memory leak on the error path here too

Previously `newdescr` was often leaked
@charris
charris merged commit 04c43f1 into numpy:master Oct 2, 2017
@charris

charris commented Oct 2, 2017

Copy link
Copy Markdown
Member

Thanks Eric.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants