Skip to content

DOC, MAINT: Enforce np.ndarray arg for np.put and np.place - #7000

Merged
shoyer merged 1 commit into
numpy:masterfrom
gfyoung:ndarray_arg_enforce
Jan 15, 2016
Merged

shoyer merged 1 commit into
numpy:masterfrom
gfyoung:ndarray_arg_enforce

Conversation

@gfyoung

@gfyoung gfyoung commented Jan 12, 2016

Copy link
Copy Markdown
Contributor

np.put and np.place do something only when the first argument is an instance of np.ndarray. These changes will cause a TypeError to be thrown in either function should that requirement not be satisfied, which would be more useful to see when debugging.

@gfyoung

gfyoung commented Jan 12, 2016

Copy link
Copy Markdown
Contributor Author

Could someone explain to me what the purpose of the failing test (test_mem_insert) is? I can't seem to find the relevant ticket for this test. It seems like what I did is backwards-incompatible, but I just don't know how. I triednp.place(1, 1, 1) on the most recently released version of numpy, but I don't know what it is doing.

Comment thread numpy/core/fromnumeric.py Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My preference for ducktyping would suggest using a try/except? I.e.,

try:
    return a.put(ind, v, mode)
except AttributeError:
    if not isinstance(a, np.ndarray):
        raise TypeError(...)
    raise

This should also be a little faster for the most common case, where a is in fact an ndarray.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm...I'm a little on the fence on this one. While I do agree it should be faster in the most common case (though how much faster I'm not really sure), coupling the type checking with the function call makes us dependent on an AttributeError when calling the C API should a not be an ndarray. Should the API change, say to something like what is done with np.place, that try...except would become moot.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest using the _wrapit functionality here, as used elsewhere in this file. _wrapit uses duck typing, similarly to @mhvk's suggestion.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shoyer : IMHO using _wrapit does not help here because I want to enforce that an ndarray must be passed in. Because np.put modifies the ndarray in-place, passing in a non-ndarray object will do nothing (the array_like object itself is not changed because it has to be cast to an ndarray object first) and will not raise any errors / warnings, which is what I am trying to avoid. This was the issue that I saw with np.place.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good point. It's still not entirely clear to me why the current duck typing solution dies not suffice, though.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I would suggest:

try:
    put = a.put
except AttributeError:
    raise TypeError
return put(...)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shoyer : Ah, fair enough. I do see lots of similar wrappings in this file, so I'll follow suit then. By the way, might you happen to know what that test_mem_insert is all about? Even with your suggestion, that test will fail, but I'm not sure what it is or what the purpose of that test was. I'm tempted to remove it, but it was located in the test_regression.py file, so I'm hesitant to do so until I have more information.

@mhvk : Thanks for your input as well!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like @shoyer's solution best too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

@mhvk

mhvk commented Jan 13, 2016

Copy link
Copy Markdown
Contributor

@Gyoung - the relevant PR is #1170 -- there apparently was a memory leak, although to be honest I don't see how this test would actually be useful to determine that the leak had returned.

@gfyoung

gfyoung commented Jan 13, 2016

Copy link
Copy Markdown
Contributor Author

@mhvk : I think I'll take the test out. I'm doing a complete revamp of np.place in another PR that is very similar to np.putmask, which as far as I can tell has had no memory leak issues.

@gfyoung

gfyoung commented Jan 13, 2016

Copy link
Copy Markdown
Contributor Author

Changes are in, and Travis + Appveyor are both happy. Should be good to merge if there is nothing else.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we use the same duck-typing solution as above, with the try/except clause?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It feels little more awkward in this case because _insert is not an ndarray method. I suppose you could do something like the following:

try:
   put = arr.put
except AttributeError:
   raise TypeError(...)
return _insert(arr, mask, vals)

However, it seems like more contrived compared to the others since we're relying on a method that has nothing to do with what is being called.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What error message do you get when place is not valid? What sort of duck typing does it currently support, if any?

It might be worth updating _insert instead.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shoyer : See #7003, which does a complete revamp of arr_insert (that is tied to _insert) and does the type checking. In the case of np.place, this PR should serve as a stopgap measure until #7003 can be merged.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shoyer : Sorry, I didn't answer the remainder of your question; I only had seen the "It might be worth" part on my phone when I was reading it. place doesn't actually do any duck-typing if you pass in a non-ndarray to it. Rather, it does nothing, which is why I added that isinstance check to the code. However, #7003 will embed that check into the C code, which is preferable. However, I wouldn't mind merging the isinstance check into the codebase FTTB.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, agreed!

@shoyer

shoyer commented Jan 13, 2016

Copy link
Copy Markdown
Member

It looks like we still need a unit test for the put fix?

@gfyoung

gfyoung commented Jan 13, 2016

Copy link
Copy Markdown
Contributor Author

@shoyer : Oh, yes, of course! Let me do that quickly.

@gfyoung

gfyoung commented Jan 13, 2016

Copy link
Copy Markdown
Contributor Author

@shoyer : Actually, quick question : where should that test go? It seems that there are no tests for the fromnumeric.py file IINM.

@shoyer

shoyer commented Jan 13, 2016

Copy link
Copy Markdown
Member

Looks like test_place is in test_function_base.py.

On Wed, Jan 13, 2016 at 11:45 AM, gfyoung [email protected] wrote:

@shoyer https://github.com/shoyer : Actually, quick question : where
should that test go? It seems that there are no tests for the
fromnumeric.py file IINM.

—
Reply to this email directly or view it on GitHub
#7000 (comment).

@gfyoung

gfyoung commented Jan 13, 2016

Copy link
Copy Markdown
Contributor Author

@shoyer : I'm referring to np.put, not np.place. That was why I had said I couldn't find any file that tested the fromnumeric.py module.

@shoyer

shoyer commented Jan 13, 2016

Copy link
Copy Markdown
Member

Looks like that is in test_multiarray.py.

On Wed, Jan 13, 2016 at 11:52 AM, gfyoung [email protected] wrote:

@shoyer https://github.com/shoyer : I'm referring to np.put, not
np.place.

—
Reply to this email directly or view it on GitHub
#7000 (comment).

@gfyoung

gfyoung commented Jan 13, 2016

Copy link
Copy Markdown
Contributor Author

It's for the array method itself, not np.put.

@shoyer

shoyer commented Jan 13, 2016

Copy link
Copy Markdown
Member

Hmm. My search for np.put in test_* turns up nothing. So no wonder this was
broken! I guess you'll have to make a judgment call here on where these
belong... (I'm also pretty new to the numpy codebase so I don't have
immediate guidance)

On Wed, Jan 13, 2016 at 11:54 AM, gfyoung [email protected] wrote:

It's for the array method itself, not np.put.

—
Reply to this email directly or view it on GitHub
#7000 (comment).

@gfyoung

gfyoung commented Jan 13, 2016

Copy link
Copy Markdown
Contributor Author

Oh, okay! I'm relatively new as well (I started contributing last month), but I think I'm going to go for creating a new file called test_fromnumeric.py for organizational reasons.

@gfyoung

gfyoung commented Jan 13, 2016

Copy link
Copy Markdown
Contributor Author

@shoyer : I'm going to create a new file test_fromnumeric.py, and it will only contain a test for checking that TypeError. However, as a follow-up, I will develop this testing module to include tests for the other methods, where I can just port tests for the relevant array methods.

np.put and np.place do something only when the first argument
is an instance of np.ndarray. These changes will cause a TypeError
to be thrown in either function should that requirement not be
satisfied.
@gfyoung

gfyoung commented Jan 14, 2016

Copy link
Copy Markdown
Contributor Author

Test added and passing on both Travis and Appveyor. Should be good to merge if there is nothing else.

@shoyer

shoyer commented Jan 15, 2016

Copy link
Copy Markdown
Member

OK, I'm pretty happy with this. Going to merge unless someone else raises objections....

@shoyer

shoyer commented Jan 15, 2016

Copy link
Copy Markdown
Member

I'm not 100% sure that test_fromnumeric.py is a good idea but it's easy enough to remove later if someone cares.

shoyer added a commit that referenced this pull request Jan 15, 2016
DOC, MAINT: Enforce np.ndarray arg for np.put and np.place
@shoyer
shoyer merged commit 54224f4 into numpy:master Jan 15, 2016
@shoyer

shoyer commented Jan 15, 2016

Copy link
Copy Markdown
Member

Thanks @gfyoung!

@gfyoung
gfyoung deleted the ndarray_arg_enforce branch January 15, 2016 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants