Skip to content

Issue 1943 - systematically testing pickling of astropy classes - #1947

Merged
embray merged 5 commits into
astropy:masterfrom
ktchrn:pickle-tests-issue-1943
Nov 4, 2014
Merged

embray merged 5 commits into
astropy:masterfrom
ktchrn:pickle-tests-issue-1943

Conversation

@ktchrn

@ktchrn ktchrn commented Jan 6, 2014

Copy link
Copy Markdown

Made sure existing pickling tests included multiple protocols.
Added basic tests for most base classes and some subclasses with additional attributes.
Most of these tests only check whether or not attributes survive pickling, not whether or not they have the same value before and after. Only the NDData test checks for changes to the various arrays it stores.

Some of the failures may be irrelevant (e.g. is constants not being pickleable actually a problem?).
Some of the failures are currently xfails by design (e.g. coordinates.DynamicTransform, which can have a function as an attribute).

Successfully reproduces the Issue 1942 problem.

@taldcroft

Copy link
Copy Markdown
Member

NICE!

Comment thread astropy/constants/tests/test_pickle.py Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an advantage to pickling through a physical file instead of just using dumps and loads? At the least you could use a StringIO object to avoid the caller being required to give a file name testfile.

@ktchrn

ktchrn commented Jan 6, 2014

Copy link
Copy Markdown
Author

@taldcroft Good points. I've implemented them in my latest commit. Do you think the number of objects included should be expanded?

@mdboom

mdboom commented Jan 6, 2014

Copy link
Copy Markdown
Contributor

This is really cool! Out of curiosity, why are we not testing the values of the pickled object? Does a lot break if we do?

@ktchrn

ktchrn commented Jan 6, 2014

Copy link
Copy Markdown
Author

@mdboom I had been assuming that if an attribute could be pickled, then it was probably pickled correctly. I just added a fairly generic comparison step, and it hasn't broken anything so far.

@taldcroft

Copy link
Copy Markdown
Member

@ktchrn - I was having trouble with table Column objects going through pickle and having incorrect attribute values (e.g. losing meta values), so it's definitely a good idea to test the values.

@ktchrn

ktchrn commented Jan 6, 2014

Copy link
Copy Markdown
Author

Before recursion, tests weren't catching objects with non-pickleable objects as attributes. Only big change so far is that coordinate systems break because they use Longitude.

@astrofrog

Copy link
Copy Markdown
Member

This is great!

@astrofrog

Copy link
Copy Markdown
Member

It looks like this should be combined with #1961, which includes a fix for Longitude

@eteq

eteq commented Jan 16, 2014

Copy link
Copy Markdown
Member

Agree this is awesome! Given that #1961 is already in the works, I agree with @astrofrog that it makes sense to rebase this against that once it's merged. But for the other failures, I wonder if maybe we should make new issues and then xfail them, so that this actually gets into the tests now?

@mdboom

mdboom commented Feb 12, 2014

Copy link
Copy Markdown
Contributor

@ktchrn: Now that #1961 is merged, I'd love to see this reinvigorated. It's a nice piece of work!

@ktchrn

ktchrn commented Feb 17, 2014

Copy link
Copy Markdown
Author

@mdboom Okay, I'll start working on it again. Are there any features that it particularly needs, apart from passing the Travis build with older numpy versions and no optional dependencies?

Comment thread astropy/tests/helper.py Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a docstring.

@mdboom

mdboom commented Feb 18, 2014

Copy link
Copy Markdown
Contributor

I think mainly we just need to get the tests passing. I did make one comment about a missing docstring as well.

@cdeil

cdeil commented Apr 28, 2014

Copy link
Copy Markdown
Member

For me the GitHub UI doesn't show a link to the travis-ci tests for the latest commit: b2405cc !?
Does anyone know where the build log is or how to start it for this PR?

Comment thread astropy/wcs/tests/test_pickle.py Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you accidentally added whitespace on this line ... please remove it.

@astrofrog

Copy link
Copy Markdown
Member

@ktchrn - could you rebase this and address @cdeil's comment? Thanks!

@astrofrog astrofrog added this to the v1.0.0 milestone May 9, 2014
@astrofrog

Copy link
Copy Markdown
Member

@ktchrn - now that 0.4 is out we can consider merging this - could you rebase this and add a changelog entry to the 1.0 entry?

@astrofrog

Copy link
Copy Markdown
Member

@ktchrn - sorry for the delay, could you rebase this so we can go ahead and merge it? Please include a changelog entry in the 1.0 section.

@embray

embray commented Oct 27, 2014

Copy link
Copy Markdown
Member

I don't think this really requires a changelog entry. It's just adding some new tests--users don't care about that.

@ktchrn
ktchrn force-pushed the pickle-tests-issue-1943 branch from afea69b to 6696736 Compare October 28, 2014 13:08
@embray

embray commented Oct 29, 2014

Copy link
Copy Markdown
Member

Oops--what happened here? Did you try to rebase?

@ktchrn

ktchrn commented Oct 30, 2014

Copy link
Copy Markdown
Author

Yeah. I guess that's all stuff that has happened since the last time I worked on this. If it's more convenient, I can just open a new pull request starting from the project's current state.

@astrofrog

Copy link
Copy Markdown
Member

@ktchrn - you should be able to clean this up by rebasing - what I suspect is that you may not have rebased against the latest upstream master. Assuming the astropy repo (not your fork) is upstream, you can do:

git fetch upstream
git rebase upstream/master

Let us know if there are any conflicts and you are not sure how to deal with them. Once the rebase is complete, you can force push to the same branch:

git push -f origin pickle-tests-issue-1943

@ktchrn
ktchrn force-pushed the pickle-tests-issue-1943 branch from 6696736 to 8b76b3d Compare October 30, 2014 12:38
@ktchrn

ktchrn commented Oct 30, 2014

Copy link
Copy Markdown
Author

Thanks! You're right, I was rebasing on my outdated master instead of upstream/master.

@embray

embray commented Oct 30, 2014

Copy link
Copy Markdown
Member

Great, that looks a lot better :)

@embray

embray commented Oct 30, 2014

Copy link
Copy Markdown
Member

Except of course for the build failing, but we'll get to the bottom of that. It's probably something minor.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ktchrn - the CartesianPoints class no longer exists, can you remove this line?

@embray

embray commented Nov 4, 2014

Copy link
Copy Markdown
Member

Looks great--thanks @ktchrn !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants