You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
would appreciate help with measuring and limiting memory spikiness
when dealing with images. memory consumption in steady state is
okay-ish, but during intense image loading/unloading it can grow
unbounded. For our applications it's generally good to have some sort
of memory cap that the engine may not exceed, and expensive
allocations should not run in parallel.
@dnfield I'm not sure we are on the same page regarding terminology. Do I understand correctly that your point is that Flutter applications (user Dart code) have to be aware of the image loading and decoding? If yes, then what are the APIs available to the application to control it?
Download status notifications. The app needs to know when a download has started, when it received Content-Size header from the server and is about to allocate a buffer for the compressed image, when it received certain amount of data, finished download successfully or not.
Decoding status notifications. The app needs to know when the image header is decoded, and an image buffer is about to be allocated.
Cancellation. If an image is no longer needed by the app, it needs to abort download and decoding processes.
Pause. The application needs to be able to put any download and decoding on hold until memory constraints are satisfied.
If the above mentioned APIs are available, then I agree, the app can implement efficient memory management itself.
What's your take on this?
PS. The customer's bug was closed with a temporary workaround. The cause is not fixed. Please reopen.
@alml - I suspect we should open some new bugs rather than reopening this one. It is fairly vague and was only meant to track the internal bug that was closed. I'm opening at least one new bug so far based on this.
To your specific points:
You can't reliably use the content-length header - it won't be supplied for chunked encoding, and it's not guaranteed to be supplied even when the server is sending over the whole image in the current frame. I don't think the framework should be managing this, but an application written network image provider could depend on it knowing what server(s) it will connect to and what types of encoding it should expect. I would be opposed to adding such logic to the framework, where in general use it is likely to be wrong or at least misleading.
I'm very skeptical about cancellation. Download cancellation is currently not supported on the VM (Add abort (or similar) to HttpClientRequest dart-lang/sdk#22265), and if we supported cancellation of decoding in the engine it's highly likely that the application will be too late - texture memory will already be allocated by the time it tries to cancel. I think it's better to just not make the request until you're reasonably sure you'll need it - e.g. what the new ScrollAwareImageProvider does (Defer image decoding when scrolling fast #49389). At any rate, cancellation would always be best effort - you'd never be guarnateed that your call to abort or cancel would actually save any work (and it actually might cost you more, if we made it to a point where almost all work was done except e.g. returning a handle back to the framework).
I'm also working on the fix for #49456, which should help with the internal customer use case of wanting to avoid resolving images multiple times that do not otherwise fit in the cache.
Thanks for opening the specific issues. It is definitely better than a vague bug. Please let me use this one to address your points.
(1) I agree, Content-Length is not always available. In case of chunked encoding or simply missing Content-Length with body interrupted by a closed connection we won't have this signal. We'll have to download those images in some fallback mode, but for the majority of static images I'd expect to have proper Content-Length and smart memory management.
(3) Cancellation doesn't have to have anything with the VM. HTTP download is an asynchronous operation. It should be possible to abort a HTTP download with some sort of cancellation token, cancellable context or a similar mechanism. Re decompression: I'm not talking about aborting an ongoing decompression (it's difficult indeed), but I do talk about cancelling no longer needed decompressions while they are still in the queue.
(4) I don't see why it couldn't be implemented. Image decoding pipeline consists of a few quite isolated steps: decoding the header, allocating a buffer, decompression itself. You agreed in (2) to provide some hooks to notify the application about the decoded header. Consider multiple big images are being downloaded in parallel. Each of them is relatively compact while compressed, but has a large resolution. The application receives a callback from the first image, sees 3000x3000, says "that's fine, go ahead", and then Flutter allocates the 36M buffer and decodes the image. Then it receives another notification "header is decoded for another 3000x3000". The app needs a way to say Flutter: "whoa! hold on!" until the first image is decompressed, resized and discarded, then it says "now continue with the second image". This way we can avoid allocating 2 large buffers at once.
For the first point about network images, I think we're basically in agreement - but I think that it's an application decision to make rather than something to put on Image.network, which doesn't have a good way to return anything to the user. For example, you might structure your application to acquire the network bytes in a way that's aware of content-length headers and has fallback modes, and then just uses Image.memory.
Cancellation does have to do with the VM - the VM doesn't support it, so Flutter can't without adding that feature to the VM. As far as cancelling unneeded decompressions that are still in the queue, I have doubts about whether an application would actually be able to cancel anything in time - and if you're pumping so many images into the queue that it's taking multiple frames to decode them, you're probably pumping too many images and should have throttled it earlier. But it's hard to determine that right now.
You can read the header of an image and get size information without ever decoding it fully. I don't think we should slow decoding by adding hooks to it - but we could instead provide some routines (or recommend packages, like package:image that do this) to determine an encoded image's size.
(1) I think most of the Flutter application developers would benefit from some generic solution should it be available. Altering Image.network or leaving 100% up to the application are the opposite poles. I'm sure there is some middle ground. If the memory management API was carved carefully, it should be possible to provide some default implementation with Flutter framework and give application developers an option to override or replace it.
Cancellation does have to do with the VM
I'm not a Dart expert, but I'm sure there are some means for cross-isolate communication. Essentially what Flutter needs is to send some message to the I/O isolate with updated instructions what to download. When the I/O isolate finishes its current operation it can "check mail" and update its queue accordingly. Do I misunderstand something?
You can read the header of an image and get size information without ever decoding it fully. I don't think we should slow decoding by adding hooks to it - but we could instead provide some routines (or recommend packages, like package:image that do this) to determine an encoded image's size.
Do I understand correctly that you propose the following flow:
Application requests image download without decompression.
Application gets a callback when then buffer is ready.
Application queries the image size without decoding.
Application performs a delay as much it needs.
Application requests image decoding by inserting the special widget "ImageFromBuffer" to the widget tree.
Re cancellation: It's okay if a request can't be interrupted if it's already running (due to the shortage of Dart HTTP API). Other requests that are waiting in the queue can be however cancelled. Right?
Re decoding from Image.memory: is there anything that needs to be done in Flutter before we could use it?
Cancellation: Right now, Image.network does no queueing. As soon as the state is inserted into the tree and resolve is called on the provider, it makes an HTTP request.
Decoding: No, this should be fine out of the box - but it means avoiding the use of Image.network and NetworkImageProvider in general.
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.
Activity
Hixie commentedon Jan 22, 2020
Status: We believe we have done what can be done and are waiting for feedback from the customer.
dnfield commentedon Jan 22, 2020
I'm working on a proposed change to an internal image class that should also improve things, and waiting for review to finish on #48536
kf6gpe commentedon Jan 22, 2020
Latest comment on the bug is
(edits for confidentiality)
6 remaining items
alml commentedon Feb 3, 2020
@dnfield I'm not sure we are on the same page regarding terminology. Do I understand correctly that your point is that Flutter applications (user Dart code) have to be aware of the image loading and decoding? If yes, then what are the APIs available to the application to control it?
If the above mentioned APIs are available, then I agree, the app can implement efficient memory management itself.
What's your take on this?
PS. The customer's bug was closed with a temporary workaround. The cause is not fixed. Please reopen.
dnfield commentedon Feb 3, 2020
@alml - I suspect we should open some new bugs rather than reopening this one. It is fairly vague and was only meant to track the internal bug that was closed. I'm opening at least one new bug so far based on this.
To your specific points:
dnfield commentedon Feb 3, 2020
I'm also working on the fix for #49456, which should help with the internal customer use case of wanting to avoid resolving images multiple times that do not otherwise fit in the cache.
alml commentedon Feb 4, 2020
Thanks for opening the specific issues. It is definitely better than a vague bug. Please let me use this one to address your points.
(1) I agree, Content-Length is not always available. In case of chunked encoding or simply missing Content-Length with body interrupted by a closed connection we won't have this signal. We'll have to download those images in some fallback mode, but for the majority of static images I'd expect to have proper Content-Length and smart memory management.
(3) Cancellation doesn't have to have anything with the VM. HTTP download is an asynchronous operation. It should be possible to abort a HTTP download with some sort of cancellation token, cancellable context or a similar mechanism. Re decompression: I'm not talking about aborting an ongoing decompression (it's difficult indeed), but I do talk about cancelling no longer needed decompressions while they are still in the queue.
(4) I don't see why it couldn't be implemented. Image decoding pipeline consists of a few quite isolated steps: decoding the header, allocating a buffer, decompression itself. You agreed in (2) to provide some hooks to notify the application about the decoded header. Consider multiple big images are being downloaded in parallel. Each of them is relatively compact while compressed, but has a large resolution. The application receives a callback from the first image, sees 3000x3000, says "that's fine, go ahead", and then Flutter allocates the 36M buffer and decodes the image. Then it receives another notification "header is decoded for another 3000x3000". The app needs a way to say Flutter: "whoa! hold on!" until the first image is decompressed, resized and discarded, then it says "now continue with the second image". This way we can avoid allocating 2 large buffers at once.
Does it make sense?
dnfield commentedon Feb 4, 2020
This does make sense, yes.
For the first point about network images, I think we're basically in agreement - but I think that it's an application decision to make rather than something to put on
Image.network, which doesn't have a good way to return anything to the user. For example, you might structure your application to acquire the network bytes in a way that's aware of content-length headers and has fallback modes, and then just usesImage.memory.Cancellation does have to do with the VM - the VM doesn't support it, so Flutter can't without adding that feature to the VM. As far as cancelling unneeded decompressions that are still in the queue, I have doubts about whether an application would actually be able to cancel anything in time - and if you're pumping so many images into the queue that it's taking multiple frames to decode them, you're probably pumping too many images and should have throttled it earlier. But it's hard to determine that right now.
You can read the header of an image and get size information without ever decoding it fully. I don't think we should slow decoding by adding hooks to it - but we could instead provide some routines (or recommend packages, like package:image that do this) to determine an encoded image's size.
alml commentedon Feb 5, 2020
(1) I think most of the Flutter application developers would benefit from some generic solution should it be available. Altering Image.network or leaving 100% up to the application are the opposite poles. I'm sure there is some middle ground. If the memory management API was carved carefully, it should be possible to provide some default implementation with Flutter framework and give application developers an option to override or replace it.
I'm not a Dart expert, but I'm sure there are some means for cross-isolate communication. Essentially what Flutter needs is to send some message to the I/O isolate with updated instructions what to download. When the I/O isolate finishes its current operation it can "check mail" and update its queue accordingly. Do I misunderstand something?
Do I understand correctly that you propose the following flow:
That might work. Is it what you meant?
dnfield commentedon Feb 5, 2020
I'm referring to the fact that the Dart http API has no cancel or abort. We don't otherwise queue or manage network requests.
Yes, that is more or less the flow I'm proposing.
alml commentedon Feb 5, 2020
Re cancellation: It's okay if a request can't be interrupted if it's already running (due to the shortage of Dart HTTP API). Other requests that are waiting in the queue can be however cancelled. Right?
Re decoding from Image.memory: is there anything that needs to be done in Flutter before we could use it?
dnfield commentedon Feb 5, 2020
Cancellation: Right now,
Image.networkdoes no queueing. As soon as the state is inserted into the tree and resolve is called on the provider, it makes an HTTP request.Decoding: No, this should be fine out of the box - but it means avoiding the use of
Image.networkandNetworkImageProviderin general.alml commentedon Feb 6, 2020
Is it also possible to decode image dimensions from the byte buffer?
dnfield commentedon Feb 6, 2020
It should be but we have no API for it today
alml commentedon Feb 7, 2020
Should we file an issue to implement this API?
dnfield commentedon Feb 7, 2020
That's #50022. Package:image might be able to just work for that too
github-actions commentedon Aug 23, 2021
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.