Skip to content

Renamed function argument - deprecate decorator - #5214

Merged
pllim merged 4 commits into
astropy:masterfrom
MSeifert04:deprecated-argument
Oct 28, 2016
Merged

pllim merged 4 commits into
astropy:masterfrom
MSeifert04:deprecated-argument

Conversation

@MSeifert04

@MSeifert04 MSeifert04 commented Aug 2, 2016 •

Copy link
Copy Markdown
Contributor

This is a follow up on #5208 which was conceived in #5171.

I hope I did not mess up the commit history when trying to add @evertrol as author. (thanks @bsipocz and @pllim)

@pllim pllim added the utils label Aug 3, 2016
@pllim pllim added this to the v1.3.0 milestone Aug 3, 2016
@pllim

pllim commented Aug 3, 2016

Copy link
Copy Markdown
Member

👍 Looks good to me! But @evertrol should test to make sure that this works for his other FITS PR.

@MSeifert04

Copy link
Copy Markdown
Contributor Author

Not sure but I think the test failure is unrelated. Could someone take a look and restart it, if possible?

@pllim

pllim commented Aug 4, 2016

Copy link
Copy Markdown
Member

Restarted.

@MSeifert04

Copy link
Copy Markdown
Contributor Author

@evertrol I don't think there would be issues but did you check if the decorator works for your PR #5171?

Anyone interested in doing a code review?

@evertrol

Copy link
Copy Markdown
Contributor

Apologies; got distracted, and I have only now tested this.
This was indeed largely a matter of simply renaming the decorator to keep things working. No problems there.

There is one case where the decorator will cause a failure. This is only for Python 2, when there is already a decorator that alters the function signature (in this case @deprecated, such as for fits.table.tdump). If @deprecated_renamed_argument is placed on top of that, the "args_in_kwargs" TypeError is raised, because the new_name will not be found in arguments.
Placing @deprecated_renamed_argument below avoids this problem. I assume this is because utils.compat._funcsigs.signature is not a precise copy of the Python 3 inspect.signature function (and perhaps it simply can't be).
Arguably, there may not be a use for deprecated functions to have a keyword parameter renamed, but I just did that for consistency.
I'm noting the failure here in case someone else stumbles on a similar problem in the future.

(Pedantry: should we care about the parameter versus argument naming? As I had called my decorator replaced_parameter.)

@MSeifert04

MSeifert04 commented Aug 23, 2016 •

Copy link
Copy Markdown
Contributor Author

@evertrol Thank you. I think the TypeError is actually expected because functools.wraps keeps the signature only for python 3.4 and later. It's inconvenient i.e. because that makes it impossible to deprecate multiple argument-names.

If wanted - I can replace "argument" with "parameter". I generally prefer "argument" but technically it's incorrect here (https://en.wikipedia.org/wiki/Parameter_%28computer_science%29#Parameters_and_arguments).

@astrofrog

Copy link
Copy Markdown
Member

It's inconvenient i.e. because that makes it impossible to deprecate multiple argument-names.

Would it make sense to change the arguments for the decorator to allow any number of arguments to be deprecated?

@pllim

pllim commented Aug 23, 2016

Copy link
Copy Markdown
Member

to allow any number of arguments to be deprecated?

In practice, is it even a good idea to deprecate so many arguments at once? But I guess it does not hurt to have that feature anyway.

@saimn

saimn commented Aug 23, 2016

Copy link
Copy Markdown
Contributor

The signature problem remind me this recent article: https://hynek.me/articles/decorators/
Maybe it would be good to use one of the suggested solutions (i.e. bundling decorator.py) to fix the signatures on all Python versions, and make sure this kind of decorator can be applied in any order.
This would also fix the second point (changing several arguments) by allowing to use the decorator several times.

@astrofrog

Copy link
Copy Markdown
Member

In practice, is it even a good idea to deprecate so many arguments at once? But I guess it does not hurt to have that feature anyway.

Well, if you have two arguments x_fwhm and y_fwhm you want to deprecate, we need a way to do that (that specific use case happened in the past)

@MSeifert04

MSeifert04 commented Aug 23, 2016 •

Copy link
Copy Markdown
Contributor Author

@astrofrog I've added a commit which allows to provide multiple deprecated arguments.

But I didn't have time to update the documentation (yet).

@saimn Interesting reading, thank you for sharing it! Maybe it's worth discussing that topic on astropy-dev?

@MSeifert04

Copy link
Copy Markdown
Contributor Author

rebased because of merge-conflict.

If anyone wants to review this I would be very happy. Currently two other PRs depend on it (or are paused because of it).

Comment thread CHANGES.rst
- New decorator: ``deprecated_renamed_argument``. This can be used when
renaming a function argument, but still want to allow for use of
the older parameter name. [#5214]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could I suggest: "This can be used to rename a function argument, while it still allows for the use of the older argument name."

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. 👍

@evertrol

Copy link
Copy Markdown
Contributor

@MSeifert04 I've had a look through it, and it looks all fine to me (passing tests & pylint seems to agree). Though I'm not sure if that counts as a review.

@MSeifert04

Copy link
Copy Markdown
Contributor Author

@evertrol Thank you for looking over this. 👍

@MSeifert04

Copy link
Copy Markdown
Contributor Author

Rebased again.

Does someone else want to review this?

cc: @eteq because you are listed as the maintainer of utils.

@MSeifert04

Copy link
Copy Markdown
Contributor Author

bump

@pllim pllim left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since @saimn needs this and said this is okay for his PR. I will 👍 this with some minor comments. Would be nice for extra pair of eyes, but if not, we should merge this soon.

Comment thread astropy/utils/tests/test_decorators.py Outdated
assert method(clobber=1) == 1

assert len(w) == 1
assert '1.3' in str(w[0].message)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't these 2 lines that deal with w be within the with block?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've seen both styles and this specific example is copied from another test. I can indent it if you want it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least to me, it is easier to read if those lines are indented, since they actually depend on w generated by the with.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will put them in the with ... context.

Comment thread astropy/utils/tests/test_decorators.py Outdated
assert test(clobber=1) == 1

assert len(w) == 1
assert '1.3' in str(w[0].message)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as above about lines with w should be in with block.

Comment thread astropy/utils/tests/test_decorators.py Outdated
assert test(clobber=1) == 1

assert len(w) == 1
assert '1.3' in str(w[0].message)

@pllim pllim Oct 6, 2016 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above. Also applies to all subsequent occurences.

return LooseVersion(str(x[2]))

sorted_zipped = sorted(zip(old_name, new_name, since), key=item2)
for k, vals in itertools.groupby(sorted_zipped, key=item2):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for the ease of future maintenance by someone else, perhaps explain why there is a need to sort first and then do a grouping, both using the same keys.

Comment thread CHANGES.rst Outdated

- ``astropy.utils``

- Added a new decorator: ``deprecated_renamed_argument``. This can be used to

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be under "new features" for 1.3, not "bug fixes"?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're right!

@pllim

pllim commented Oct 6, 2016

Copy link
Copy Markdown
Member

I mistakenly mentioned @saimn in my review comment above (but GitHub won't let me edit it). Should have been @evertrol . I apologize for the confusion.

@MSeifert04

MSeifert04 commented Oct 6, 2016 •

Copy link
Copy Markdown
Contributor Author

@pllim Thank you for the comments!

I think I've adressed and pushed all of them.

@MSeifert04

Copy link
Copy Markdown
Contributor Author

The sphinx build failed because it couldn't resolve numpy classes/functions.

Maybe their intersphinx is currently down (but there was no warning when fetching) or corrupted. Could someone restart that test in a few hours?

@saimn

saimn commented Oct 6, 2016

Copy link
Copy Markdown
Contributor

@pllim - No problem :)

@saimn saimn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a few comments. I would prefer if it was possible to use the decorator multiple times instead of having lists of parameters, but it seems more difficult as functools.wraps does not preserve the signature before Python 3.5. So the currently implementation is fine.

Comment thread astropy/utils/decorators.py Outdated

Parameters
----------
old_name : str

@saimn saimn Oct 6, 2016 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be str or list of str, and the same below for new_name and since.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

totally forgot to update these. 👍

# Assume that new_name and since are correct (tuple/list with the
# appropriate length) in the spirit of the "consenting adults". But the
# optional parameters may not be set, so if these are not iterables
# wrap them.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe better to check that new_name and since are also list with the correct size. Otherwise, as str are iterable, it may not fail with something like @deprecated_renamed_argument(['a', 'b'], 'alpha', '1.2') (not tested, just guessing).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like these explicit checks that the user entered the right types and right amount when there is no use-case for it. I think such isinstance and len(x) = len(y) checks are defensive programming and/or code smell.

@MSeifert04 MSeifert04 Oct 6, 2016 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Especially because it's likely to fail nevertheless:

  • It's unlikely that single letters of the old_name are part of the signature.
  • It's likely that the "add sphinx directive" operation fails if parts of the "since" are used.

So it would just be a "more accurate error message" if I added these checks. OTOH what if new_name is a str but old_name and since are lists. That case would also fail later (because it doesn't enter the if branch here). It would require a serious amount of additional checks here to make it totally safe (and slower).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you prefer, I agree it is difficult to check everything (and it is not something I would recommend in general) and that's why it would be much cleaner without having to support multiple arguments. But yes, it is possible to do get some unexpected behaviour: https://gist.github.com/saimn/f9f30bab65a19fc1fa5dc2492b148559 (the first one is something that can really happen I think, with variants depending on the number of arguments, the second is just for the pleasure ;)).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, the LooseVersion is really liberal when comparing versions. 😄

I've included a Warning box in the parameter section but I'd prefer the current option over the "test it all" one. After all it's not really a user-facing function.

@MSeifert04

Copy link
Copy Markdown
Contributor Author

@saimn Do you know if any library or package (decorator or wrapt?) would make it possible to apply the decorator several times?

I have a similar problem with the support_nddata decorator (which also needs to access the signature) and I think bundling such a package (or make it a dependency) would be great if it fixes these annoying Python introspection issues.

@saimn

saimn commented Oct 15, 2016

Copy link
Copy Markdown
Contributor

@MSeifert04 - 👍 to bundle one of these packages as it could be useful for other decorators. I didn't use any of the packages listed in https://hynek.me/articles/decorators/ (boltons.funcutils (1), decorator (2), wrapt (3)), but this post gives some hint on the differences between these packages: 1 and 2 don't support class methods, 2 and 3 use a closure-less style to write the decorator, 3 seems more difficult to bundle. So, I would go for 1 or 2.

@MSeifert04

Copy link
Copy Markdown
Contributor Author

The suggestion from the mailing list: https://groups.google.com/forum/#!topic/astropy-dev/VITGjHcpSa4 to use astropy.utils.decorator.wraps instead of functools.wraps doesn't work in this case (unfortunatly).

The problem is that it keeps the signature (I thought that would be good but it's also a limitation) and therefore doesn't permit the deprecated name to be used - because that name is not part of the signature anymore.

I think the current version is the best way. 😄

@MSeifert04

MSeifert04 commented Oct 25, 2016 •

Copy link
Copy Markdown
Contributor Author

bump

There are still two PRs depending on this PR. Any outstanding issues or anyone else wanting to review?

@saimn

saimn commented Oct 25, 2016

Copy link
Copy Markdown
Contributor

The problem is that it keeps the signature (I thought that would be good but it's also a limitation) and therefore doesn't permit the deprecated name to be used - because that name is not part of the signature anymore.

Annoying, but logical. Thanks for having tried this !

@pllim

pllim commented Oct 26, 2016

Copy link
Copy Markdown
Member

I don't see why this should not get merged after all the discussions and updates based on them. But maybe one last squash and rebase and let Travis run on the squashed version? In that case, if anyone objects too late, it won't be hard to revert the merge.

@MSeifert04

Copy link
Copy Markdown
Contributor Author

@pllim I didn't dare to squash because of the commit attributed to evertrol (because of the cooperative work on this). Any suggestions which commits to squash/keep?

@pllim

pllim commented Oct 26, 2016

Copy link
Copy Markdown
Member

Okay, how about squash commit number 3 to last? That would leave only 3 commits total.

@MSeifert04

Copy link
Copy Markdown
Contributor Author

ok, done 👍

@pllim

pllim commented Oct 26, 2016

Copy link
Copy Markdown
Member

Close enough. 😄 I'll merge when tests pass unless I hear objections before that.

@pllim

pllim commented Oct 26, 2016

Copy link
Copy Markdown
Member

@MSeifert04 , what are the two PRs depending on this one again? I only see one listed in this PR. I want to make sure that all of them are milestoned to 1.3 and not earlier.

@MSeifert04

MSeifert04 commented Oct 26, 2016 •

Copy link
Copy Markdown
Contributor Author

Oups, I forgot one 😞

The PRs are #5232, #5171 (both milestoned 1.3 already)

@MSeifert04

MSeifert04 commented Oct 26, 2016 •

Copy link
Copy Markdown
Contributor Author

Travis hasn't started until now, so I've cancelled the build, unfortunatly I can't restart it. It does say it's restarted but Travis is obviously fooling me.

Did someone break Travis or the hooks because the build worked on my fork.

@pllim

pllim commented Oct 26, 2016

Copy link
Copy Markdown
Member

Not sure why it took so long. I restarted the build.

@MSeifert04

Copy link
Copy Markdown
Contributor Author

The other PR hasn't started either (https://travis-ci.org/astropy/astropy/pull_requests). But just for fun I've restarted my fork build (https://travis-ci.org/MSeifert04/astropy/builds/170797977) and it started immediatly. Is something in the astropy-project (including affiliated packages) stalling the builds?

@pllim

pllim commented Oct 26, 2016

Copy link
Copy Markdown
Member

Astropy org itself has many projects. Maybe it's just due to the sheer amount of tests being run across the board? Just a guess here.

@MSeifert04

Copy link
Copy Markdown
Contributor Author

ok, let's wait until it starts. The fork test build is mostly done (and all green).

@MSeifert04

Copy link
Copy Markdown
Contributor Author

Maybe that #5214 (comment) is responsible? 😄

@MSeifert04

Copy link
Copy Markdown
Contributor Author

@pllim It passed.

@pllim

pllim commented Oct 28, 2016

Copy link
Copy Markdown
Member

Ok. Merging. Thanks for your continued patience, @MSeifert04 !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants