Repository navigation
Deleting a storage object created via createLargeObject() won't delete the segments that make up the file #286
Description
Activity
A little bit of research follows...
Looking at the code for python-swiftclient (which I know does magically delete the segments for a large object) it appears to grab the 'X-Object-Manifest' header and delete the segments itself (as opposed to asking Swift to do it on the server-side):
For SLO objects (up until now we're talking about DLO objects) it does ask Swift to delete the underlying segments for it, using a special query string parameter (
"multipart-manifest=delete"). I don't necessarily think this library needs to support deleting SLO objects, since it only supports creating DLO ones, but that could be a nice feature at some point.Another interesting thing about the Python implementation, is that it uses a thread pool to delete the segments, which would probably be great for performance. If you have 500 segments and each request takes 500ms, it would take 250 seconds to do them serially, but only 50 seconds if done in parallel with 5 threads.
Based on the above research, I'd like to propose the following:
- A solution to access extra object headers be implemented in Can't easily get ALL headers for a storage object #285
- The object
delete()method get a new argument likedelete(delete_segments=FALSE)which, if set to true, will grab the 'X-Object-Manifest' header via whatever gets implemented in Can't easily get ALL headers for a storage object #285, and then serially deletes the segments before deleting the manifest (doing it in parallel with a thread pool could be done manually by the library user if they choose)
The reason I'm proposing that we don't delete the segments by default is that it could be very slow. Although, if you guys aren't worried about that, I think it'd be better for the usability of this library if it defaulted to deleting the segments too.
I'll try to work on some PRs later...
Hey the issue still exists after this long time.
I for example encounter it on my nextcloud, large files in the _segements bucket are just never deleted.
I would really appreciate if you could look into this again.
If you create an object via
createLargeObject()and then delete it via this library, it won't actually delete all the segments that make up the object, just the manifest.I believe there is a way to do the delete request such that OpenStack will delete the segments for you when deleting the object, but I believe that depends on your OpenStack instance having particular middleware. I'm not 100% on either of those points, unfortunately. :-/
In any case, it is possible to ask a manifest where it's segments are, by doing a HEAD on the object and looking at the 'X-Object-Manifest' header. Unfortunately, there isn't a super easy way to get that header - see #285
Currently, in my application, I have a wrapper around delete that first does a HEAD on the object, checks the 'X-Object-Manifest' header, then lists and deletes all the segments before deleting the manifest. However, it would be really, really great, if this library could handle that automatically! There is builtin support for uploading large objects (via
createLargeObject()) so IMO there should be support for deleting them as well.