cloudpickle can produce incorrect pickles

>>> import multiprocessing as mp, cloudpickle, pickletools
>>> v = mp.Value('i')
>>> cloudpickle.dumps(v)
b'\x80\x02'
>>> import pickletools
>>> pickletools.dis(cloudpickle.dumps(v))
    0: \x80 PROTO      2
Traceback (most recent call last):
  File "<ipython-input-28-9e637ba398b4>", line 1, in <module>
    pickletools.dis(cloudpickle.dumps(v))
  File "/home/antoine/miniconda3/envs/dask36/lib/python3.6/pickletools.py", line 2398, in dis
    for opcode, arg, pos in genops(pickle):
  File "/home/antoine/miniconda3/envs/dask36/lib/python3.6/pickletools.py", line 2238, in _genops
    raise ValueError("pickle exhausted before seeing STOP")
ValueError: pickle exhausted before seeing STOP

@pitrou

I've run into this as well.

The problem is runtime exceptions can be swallowed here:
https://github.com/cloudpipe/cloudpickle/blob/master/cloudpickle/cloudpickle.py#L269
in my case, it turned out it was a changing dictionary while cloudpickle iterated through it; in @pitrou's example, it's RuntimeError: Synchronized objects should only be shared between processes through inheritance Should the except block end with a raise instead?

Thanks for diagnosing this. It should be pretty easy to fix.

Fix cloudpipe#129: do not silence RuntimeError in dump()

Fix cloudpipe#129: do not silence RuntimeError in dump() (cloudpipe#140)

Fix #129: do not silence RuntimeError in dump() (#140)

Navigation Menu

Repository navigation

cloudpickle can produce incorrect pickles #129

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

cloudpickle can produce incorrect pickles #129

Description

Activity

adamklein commented on Dec 13, 2017

pitrou commented on Dec 13, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions