Repository navigation
Eliminate experimental modes #2228
Description
Activity
Experimental is a word, a more accurate word would be legacy. It's released, in the wild, and I'm pretty sure that a lot of it is used.
From what I can tell, there are 4 classes of support for modes:
- Full support - Anything that Pillow can do can be done in these modes.
RGB/RGBAwould be examples here. This mostly covers the1mode as well, but it's a little bit special. - Partial support - There's basic support, we can read, write and store (in memory) images in this format, but there are holes in the coverage for other operations. (e.g. cropping works, but not necessarily resampling). This is the case with the
I;16modes, and the partial widthRGB/BGRmodes - Pack/unpack support - These are the raw modes, and they're only useful in transit to one of the storage modes.
- Unsupported - Pixel formats that we just don't do, like multiband high bit depth and signed I;16.
I totally agree on putting together a list of the supported modes and monitoring the correctness of operations with them. Tests are always good.
I don't think that we should be dropping support for the special modes, as I know we've people using the
I;16modes in GIS type applications, and I'm pretty sure that theBGR;15type modes are used in texture mapping and other image compression for 3d graphics. I don't know if we should be making the various partial width modes just pack/unpack modes to the standardRGB, or if they should stay as storage modes. The 16 bit per channel modes seem to be something that's showing up in more places. There are outstanding requests forI;16Sand multi-band 16 bit images, and I think we've had a couple of people think about doing the work on the multi-band ones. That points to not dropping the support, but rather rationalizing the storage for wider channels.Lab is a good color space to have, it's the perceptual match to the RGB for phosphor and CMYK for ink. If you're doing resampling, it's probably the best/most accurate space to be in. A straight line in Lab space is perceptually smooth, where it's not in RGB or HSV. Internally, I think we're converting the a and b channels to an unsigned, then back on save. (or at least, I remember looking at samples and trying to rationalize what was happening in the image formats) HSV is useful for a few things. I didn't think that it was a signed format though.
PAis distinct from anRGBApalette. ARGBApalette image is limited to 256 color+transparency combinations, where aPAis a 256 color image with a 256 grey mask.- Full support - Anything that Pillow can do can be done in these modes.
Pack/unpack support
Oh, I missed this in the text, but all of this is not related to the packing/unpacking. Of course, we should support as many raw modes as we need for supported image formats.
I know we've people using the
I;16modes in GIS type applicationsI'm totally for 16 bit per channel modes support for
Land other modes, аnd I hope that we will do that but for now, this doesn't look like something useful. How they are using Pillow for this? Which exactly operations are working withI;16mode? On the other hand, it's s bit strange to removeI;16mode now and add complete support for 16-bit modes latter, so it is reasonable to keepI;16itself as is.Bit this is not applicable to
I;16B,I;16LandI;16Nas storage modes. We just need to choose one 16-bit format and use it. There are no advantages of using several endings for processing, they can be easily converted one to another. Its support just adds complexity.I'm pretty sure that the BGR;15 type modes are used in texture mapping
Yes, there are lots of storage formats in the world. Like BGRa in Cairo. But this doesn't mean that we should support processing in such modes. We can read and write them through
frombufferandtobytesand this is enough for manipulating.BGR;15,BGR;16andBGR;24have no advantages overRGBin terms of processing and the final result.Reacted by Yay295Internally, I think we're converting the a and b channels to an unsigned, then back on save.
This explains why
HSVresampling works as expected. So we need to fix comments inImagingNewPrologueSubtypeat least.I'm not sure what they're doing with the various modes, but I think that a huge part of the value is simply a load, offload to numpy/scipy and the reverse back to saving into a tiff.
The endian issues with the I;16 date back to some of the tiff processing, It's likely that most of those should just wind up as the pack/unpack raw modes.
One thing I've noticed with the various modes is that when converting between different widths of equivalent images, e.g. I;16 to L, the default conversion is to clip the range to 0-255. There's no simple way to upconvert an L image to the MSB of an I;16 or I;32, nor a way to downconvert that MSB to an L image. #2574 is an example of that.
Also need to investigate the ability to save in a specific format given a more general one, e.g. save an I;16S from an I;32 image.
Looking at C layer functions that dispatch or check on IMAGING_TYPE_*:
IMAGING_TYPE_SPECIAL limitations:
- Histogram is broken
- Point only works for I;16
- RankFilter raises ModeError
- Resample raises ModeError
IMAGING_TYPE_UINT8 only functions:
- ImageChops
- Alpha Composite
- GetBand
- ImagingBlend
- ImagingModeFilter
Appears to work with IMAGING_TYPE_SPECIAL:
- Geometry Filters
- Bounding Box
- GetPixel
- GetInk
Items that are not listed aren't dispatching on imaging type, so presumably they are supposed to work.
I'm not sure what they're doing with the various modes, but I think that a huge part of the value is simply a load, offload to numpy/scipy and the reverse back to saving into a tiff.
I think this is a good argument for keeping all of the modes. If someone is loading an
I;16Bimage and using something else to process it, they probably don't want Pillow to automatically convert it toI;16L.I do think a consistent internal format would be very helpful for processing images though. It would probably work to use RGB and native endianness internally, and have specific methods for exporting and importing using the experimental modes.
frombytes()andtobytes()could be those methods, and everything else would work with unpacked native-endian data.I think for the most part, native endianness is a good idea, legacy issues aside. On the other hand, I don't think that putting data into
RGBif it's anything other thanRGBor a lossless permutation is a good idea. There are too many other colorspaces that are used for that to be a good idea.There is definitely a broader discussion to be had on storage formats, especially when we're getting to multi-byte multi-planar images. There are definitely several ways this could go, depending on what we're really trying to support.
First -- It would be good to have an internal storage that's extendable to any raster image that we're likely to see. The most complex image I've personally dealt with would be a >10 channel Cloud Oriented GeoTiff, with 5 levels of pyramid previews. These are:
- Multi res. There are 5 image sizes in the tiff, from essentiall thumbnail to 500x500m across the globe.
- Multi-channel. There can be multiple planes of image data, each with a distinct meaning, all contained in the same file. (Not just RGB, things like wind velocity, mean wind energy, or individual month averages)
- Not just 8 bit channels. Bit, Byte, Int16, and Floats are all in play.
Now I'm not arguing that we're going to need to serve the interests of the Geo community, mainly because they've got tools that do this already that interface with python (gdal and friends). But if we're going to go through the effort of figuring out an internal format, I wouldn't want to do it twice.
Second -- We should be looking at something that's easily zero copy sharable with the data community. We've currently got numpy/pandas covered, but if we were using something like Arrow, we'd interop with a huge ecosystem, potentially even outside of the python world. (PyArrow is a common addition to pandas, and it allows for zero copy arrow->pandas dataframe. Also, polars is really nice) I don't know how well this would work with some of our procesing, but it may open up avenues for us to easily do SIMD accelerated operations on the data using external libraries (like polars).
Third -- I think we might want to think about composite storage for images -- either planar storage where we have n single channel data structures, one per band, or tiled storage where we have one per tile in the original image. This could allow us to partially load giant images (like GeoTiffs) and process them efficiently.
I think the only potential backwards compatibility issue would be with
Image.frombuffer()(andImage.fromarray(), which usesfrombuffer) because "changes to the original buffer object are reflected in this image". The buffer object would have to use native endianness instead of the mode's endianness.I have used these modes and really hoped that the PILLOW maintainers will add more of them and fix the naming issue, as stated in #8158, but instead they just removed modes completely.
So now the only option left to me is to reimplement these modes by myself either using numpy or by building my own image manipulation library in a low level language. Am I right?
Which modes were you using specifically? BGR;15, BGR;16 and BGR;24? Could you explain how you were using them? I would have thought that Pillow's prior implementation of those modes had extremely limited functionality.
The raw modes were useful in Pillow as an efficient way to interpret low-level pixel data that directly corresponds to well-known GPU formats, such as those used by OpenGL.
I primarily used the BGR;16 raw mode, which effectively corresponds to OpenGL formats such as RGB565, to load image stored in engine-specific format.
These raw modes provide a fast and correct path for decoding nonstandard image data and converting it into standard Pillow image modes. Before relying on Pillow’s raw decoders, I had an equivalent implementation in Python using Pillow’s ImageDraw, but it was orders of magnitude slower. Having this functionality in Pillow moved the byte-level unpacking into optimized native code and made such workflows practical.
It is valuable not for rich image manipulation, but for performance.
You are talking about raw modes and modes, so it sounds like you understand the distinction, but we didn't remove the BGR;16 raw mode, only the BGR;16 mode.
I imagine that something in your code stopped working in order to lead you here, but I'm still not sure what that would have been. If you have a code example, that would clear things up.
Every time when I open Pillow's C sources, I absolutely scared by all these experimental modes. Do I need to worry how not to break them? Do I need to implement features for all of them? How can I write tests for them?
In my opinion, an experiment must have the following characteristics: hypothesis; subject; deadline or exit condition; metric; outcomes. When someone adds another
else ifinImagingNewPrologueSubtype, this is not an experiment. This is just another unfinished feature which someone else needs to support.My suggestions are:
IMAGING_TYPE_SPECIALtype:I;16,I;16L,I;16B,I;16N,BGR;15,BGR;16,BGR;24,BGR;32RGBXmode as full equivalent ofRGB1,P,L,LA,RGBandRGBAmodesCMYKandYCbCrLaandRGBaFandIQuestionable modes:
LABandHSVPA. As I know palette supportsRGBAmodes, so it can handle "palette with alpha" in a different way