Skip to content

[Android][Impeller]we need to be notified before destroying the surface #152839

Closed as not planned

Description

@xxoo

Use case

I've read the migration guide and the migrated source of video_player.
Solution for the image loss problem in video_player:

In onSurfaceDestroyed, save and destroy the state of ExoPlayer, and then restore it in onSurfaceCreated. However, I believe this is a rather expensive operation, especially since the restoration process may cause noticeable pauses for the user. More importantly, it requires the video to stop playing, which may not be acceptable for some apps.
Therefore, I think a more efficient method should be used to solve this problem.

Proposal

Introduce an onSurfaceDestroying event.
In this event, we can remove the destroying surface from ExoPlayer to prevent errors. Then just call SetVideoSurface() in onSurfaceCreated (or even onSurfaceDestroyed). That should be able to avoid the steps of saving, destroying and restoring the state of ExoPlayer.

Activity

  1. changed the title [-][Android][Impeller]we need to be notified before the destroying surface[/-] [+][Android][Impeller]we need to be notified before destroying the surface[/+] on Aug 4, 2024
  2. jonahwilliams commented on Aug 4, 2024

    @jonahwilliams
    Contributor

    It's not really clear what you are proposing? What is the difference between onSurfaceDestroying and the existing lifecycle?

  3. added
    waiting for responseThe Flutter team cannot make further progress on this issue until the original reporter responds
    on Aug 4, 2024
  4. xxoo commented on Aug 4, 2024

    @xxoo
    Author

    @jonahwilliams Thanks for reply
    The difference is that in onSurfaceDestroying, you can do things that are not possible in onSurfaceDestroyed, such as saving the current frame. Additionally, since the surface has already been destroyed in onSurfaceDestroyed, ExoPlayer may encounter errors, meaning that the player instance also has to be destroyed. However, if you can receive a notification beforehand, you have the opportunity to avoid this issue.

  5. removed
    waiting for responseThe Flutter team cannot make further progress on this issue until the original reporter responds
    on Aug 4, 2024
  6. xxoo commented on Aug 4, 2024

    @xxoo
    Author

    @jonahwilliams I have updated the proposal to make it more clear. Please take another look thanks

  7. added
    in triagePresently being triaged by the triage team
    c: proposalA detailed proposal for a change to Flutter
    platform-androidAndroid applications specifically
    engineflutter/engine related. See also e: labels.
    e: impellerImpeller rendering backend issues and features requests
    team-androidOwned by Android platform team
    fyi-engineFor the attention of Engine team
    and removed
    in triagePresently being triaged by the triage team
    on Aug 5, 2024
  8. added
    P2Important issues not at the top of the work list
    on Aug 7, 2024
  9. 8 remaining items

  10. added
    P3Issues that are less important to the Flutter project
    and removed
    P2Important issues not at the top of the work list
    on Aug 13, 2024
  11. xxoo commented on Aug 14, 2024

    @xxoo
    Author

    @matanlurey Thanks for reply

    It would not be any more expensive than creating the surface and plugin for the first time (in practice it should be less).

    The cost indeed won’t be greater than the first time it was opened, but it still falls short of providing a seamless playback experience. This is because the media file may need to be reopened and buffered. Moreover, this issue only exists in Android.

    What errors? Do you have an example of a reproduction where an error occurs as a result of the migration?

    This is the error that occurs when I call clearVideoSurface in onSurfaceDestroyed. In fact, the error occurs whether or not clearVideoSurface is called. So, I believe that removing the Surface from ExoPlayer at this point might already be too late. To avoid the error, it might only be possible by saving, destroying, and restoring ExoPlayer’s state. I suspect this might be the reason why you did it this way.

    To migrate to SurfaceProducer, users should avoid caching Surface instances and instead use the getSurface() method each time. However, this may not always be practical in real-world scenarios. This is because a Surface instance might need to retain user-set states, or libraries like ExoPlayer may inherently require caching the Surface instance. To maintain the usability of Texture widget, I believe it is still necessary to introduce an event that ensures users have the opportunity to prevent the following errors from occurring.

    java.lang.IllegalStateException: releaseOutputBuffer() is valid only at Executing states; currently at Released state
    
        at android.media.MediaCodec.releaseOutputBuffer(Native Method)
        at android.media.MediaCodec.releaseOutputBufferInternal(MediaCodec.java:4173)
        at android.media.MediaCodec.releaseOutputBuffer(MediaCodec.java:4147)
        at androidx.media3.exoplayer.mediacodec.AsynchronousMediaCodecAdapter.releaseOutputBuffer(AsynchronousMediaCodecAdapter.java:206)
        at androidx.media3.exoplayer.video.MediaCodecVideoRenderer.renderOutputBufferV21(MediaCodecVideoRenderer.java:1716)
        at androidx.media3.exoplayer.video.MediaCodecVideoRenderer.maybeReleaseFrame(MediaCodecVideoRenderer.java:1448)
        at androidx.media3.exoplayer.video.MediaCodecVideoRenderer.processOutputBuffer(MediaCodecVideoRenderer.java:1419)
        at androidx.media3.exoplayer.mediacodec.MediaCodecRenderer.drainOutputBuffer(MediaCodecRenderer.java:2070)
        at androidx.media3.exoplayer.mediacodec.MediaCodecRenderer.render(MediaCodecRenderer.java:873)
        at androidx.media3.exoplayer.video.MediaCodecVideoRenderer.render(MediaCodecVideoRenderer.java:1018)
        at androidx.media3.exoplayer.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:1136)
        at androidx.media3.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:561)
        at android.os.Handler.dispatchMessage(Handler.java:103)
        at android.os.Looper.loopOnce(Looper.java:232)
        at android.os.Looper.loop(Looper.java:317)
        at android.os.HandlerThread.run(HandlerThread.java:85)
    2024-08-14 18:16:11.564  4369-4466  ExoPlayerImplInternal   dev.xx.av_media_player_example       E  Playback error
    androidx.media3.exoplayer.ExoPlaybackException: MediaCodecVideoRenderer error, index=0, format=Format(0, null, null, video/avc, avc1.64001F, -1, null, [848, 480, -1.0, ColorInfo(Unset color space, Unset color range, Unset color transfer, false, 8bit Luma, 8bit Chroma)], [-1, -1]), format_supported=YES
        at androidx.media3.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:640)
        at android.os.Handler.dispatchMessage(Handler.java:103)
        at android.os.Looper.loopOnce(Looper.java:232)
        at android.os.Looper.loop(Looper.java:317)
        at android.os.HandlerThread.run(HandlerThread.java:85)
    Caused by: androidx.media3.exoplayer.video.MediaCodecVideoDecoderException: Decoder failed: c2.goldfish.h264.decoder
        at androidx.media3.exoplayer.video.MediaCodecVideoRenderer.createDecoderException(MediaCodecVideoRenderer.java:1982)
        at androidx.media3.exoplayer.mediacodec.MediaCodecRenderer.render(MediaCodecRenderer.java:893)
        at androidx.media3.exoplayer.video.MediaCodecVideoRenderer.render(MediaCodecVideoRenderer.java:1018)
        at androidx.media3.exoplayer.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:1136)
        at androidx.media3.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:561)
        ... 4 more
    Caused by: java.lang.IllegalStateException: releaseOutputBuffer() is valid only at Executing states; currently at Released state
    
        at android.media.MediaCodec.releaseOutputBuffer(Native Method)
        at android.media.MediaCodec.releaseOutputBufferInternal(MediaCodec.java:4173)
        at android.media.MediaCodec.releaseOutputBuffer(MediaCodec.java:4147)
        at androidx.media3.exoplayer.mediacodec.AsynchronousMediaCodecAdapter.releaseOutputBuffer(AsynchronousMediaCodecAdapter.java:206)
        at androidx.media3.exoplayer.video.MediaCodecVideoRenderer.renderOutputBufferV21(MediaCodecVideoRenderer.java:1716)
        at androidx.media3.exoplayer.video.MediaCodecVideoRenderer.maybeReleaseFrame(MediaCodecVideoRenderer.java:1448)
        at androidx.media3.exoplayer.video.MediaCodecVideoRenderer.processOutputBuffer(MediaCodecVideoRenderer.java:1419)
        at androidx.media3.exoplayer.mediacodec.MediaCodecRenderer.drainOutputBuffer(MediaCodecRenderer.java:2070)
        at androidx.media3.exoplayer.mediacodec.MediaCodecRenderer.render(MediaCodecRenderer.java:873)
        ... 7 more
    2024-08-14 18:16:11.582  4369-4466  VideoFrameReleaseHelper dev.xx.av_media_player_example       E  Failed to call Surface.setFrameRate
    java.lang.IllegalStateException: Surface has already been released.
        at android.view.Surface.checkNotReleasedLocked(Surface.java:860)
        at android.view.Surface.setFrameRate(Surface.java:1068)
        at android.view.Surface.setFrameRate(Surface.java:1112)
        at androidx.media3.exoplayer.video.VideoFrameReleaseHelper$Api30.setSurfaceFrameRate(VideoFrameReleaseHelper.java:440)
        at androidx.media3.exoplayer.video.VideoFrameReleaseHelper.clearSurfaceFrameRate(VideoFrameReleaseHelper.java:385)
        at androidx.media3.exoplayer.video.VideoFrameReleaseHelper.onStopped(VideoFrameReleaseHelper.java:235)
        at androidx.media3.exoplayer.video.VideoFrameReleaseControl.onStopped(VideoFrameReleaseControl.java:231)
        at androidx.media3.exoplayer.video.MediaCodecVideoRenderer.onStopped(MediaCodecVideoRenderer.java:777)
        at androidx.media3.exoplayer.BaseRenderer.stop(BaseRenderer.java:209)
        at androidx.media3.exoplayer.ExoPlayerImplInternal.ensureStopped(ExoPlayerImplInternal.java:1836)
        at androidx.media3.exoplayer.ExoPlayerImplInternal.disableRenderer(ExoPlayerImplInternal.java:1845)
        at androidx.media3.exoplayer.ExoPlayerImplInternal.resetInternal(ExoPlayerImplInternal.java:1566)
        at androidx.media3.exoplayer.ExoPlayerImplInternal.stopInternal(ExoPlayerImplInternal.java:1523)
        at androidx.media3.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:686)
        at android.os.Handler.dispatchMessage(Handler.java:103)
        at android.os.Looper.loopOnce(Looper.java:232)
        at android.os.Looper.loop(Looper.java:317)
        at android.os.HandlerThread.run(HandlerThread.java:85)
    2024-08-14 18:16:11.582  4369-4490  MediaCodec              dev.xx.av_media_player_example       E  flush() is valid only at Executing states; currently at Released state
    2024-08-14 18:16:11.583  4369-4466  ExoPlayerImplInternal   dev.xx.av_media_player_example       E  Disable failed.
    java.lang.IllegalStateException: flush() is valid only at Executing states; currently at Released state
    
        at android.media.MediaCodec.native_flush(Native Method)
        at android.media.MediaCodec.flush(MediaCodec.java:2631)
        at androidx.media3.exoplayer.mediacodec.AsynchronousMediaCodecAdapter.flush(AsynchronousMediaCodecAdapter.java:247)
        at androidx.media3.exoplayer.mediacodec.MediaCodecRenderer.flushCodec(MediaCodecRenderer.java:960)
        at androidx.media3.exoplayer.mediacodec.MediaCodecRenderer.flushOrReleaseCodec(MediaCodecRenderer.java:953)
        at androidx.media3.exoplayer.mediacodec.MediaCodecRenderer.onDisabled(MediaCodecRenderer.java:780)
        at androidx.media3.exoplayer.video.MediaCodecVideoRenderer.onDisabled(MediaCodecVideoRenderer.java:794)
        at androidx.media3.exoplayer.BaseRenderer.disable(BaseRenderer.java:220)
        at androidx.media3.exoplayer.ExoPlayerImplInternal.disableRenderer(ExoPlayerImplInternal.java:1846)
        at androidx.media3.exoplayer.ExoPlayerImplInternal.resetInternal(ExoPlayerImplInternal.java:1566)
        at androidx.media3.exoplayer.ExoPlayerImplInternal.stopInternal(ExoPlayerImplInternal.java:1523)
        at androidx.media3.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:686)
        at android.os.Handler.dispatchMessage(Handler.java:103)
        at android.os.Looper.loopOnce(Looper.java:232)
        at android.os.Looper.loop(Looper.java:317)
        at android.os.HandlerThread.run(HandlerThread.java:85)
    
  12. removed
    waiting for responseThe Flutter team cannot make further progress on this issue until the original reporter responds
    on Aug 14, 2024
  13. matanlurey commented on Aug 14, 2024

    @matanlurey
    Contributor

    This is the error that occurs when I call clearVideoSurface in onSurfaceDestroyed. In fact, the error occurs whether or not clearVideoSurface is called.

    Why are you calling this? AFAICT this is not in migrated PR (which also doesn't crash).

  14. added
    waiting for responseThe Flutter team cannot make further progress on this issue until the original reporter responds
    on Aug 14, 2024
  15. matanlurey commented on Aug 14, 2024

    @matanlurey
    Contributor

    (At this point I need a specific code sample, i.e. "in this PR X, if I do Y, I observe a crash".)

  16. xxoo commented on Aug 15, 2024

    @xxoo
    Author

    @matanlurey

    Why are you calling this? AFAICT this is not in migrated PR (which also doesn't crash).

    Because I tried to avoid saving and restoring the state of ExoPlayer

    (At this point I need a specific code sample, i.e. "in this PR X, if I do Y, I observe a crash".)

    Please use this repo to reproduce the error
    And visit here to see what was changed
    I made the necessary changes on video_player_android 2.6.0

    steps:

    1. clone the repo
    2. setup and run the example on android
    3. play the video
    4. shutdown the screen
    5. check logcat you may see the errors
  17. removed
    waiting for responseThe Flutter team cannot make further progress on this issue until the original reporter responds
    on Aug 15, 2024
  18. matanlurey commented on Aug 15, 2024

    @matanlurey
    Contributor

    Because I tried to avoid saving and restoring the state of ExoPlayer

    That is not possible, and is the point of the PR.

    At this point I am going to close this bug as WAI - when the PR merges, if there are problems, we can address them.

  19. xxoo commented on Aug 15, 2024

    @xxoo
    Author

    That is not possible, and is the point of the PR.

    In the current api set, it is impossible indeed. And this is the reason I opened this proposal.

  20. github-actions commented on Aug 29, 2024

    @github-actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  21. locked as resolved and limited conversation to collaborators on Aug 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P3Issues that are less important to the Flutter projectc: proposalA detailed proposal for a change to Fluttere: impellerImpeller rendering backend issues and features requestsengineflutter/engine related. See also e: labels.platform-androidAndroid applications specificallyr: invalidIssue is closed as not validteam-engineOwned by Engine teamtriaged-engineTriaged by Engine team

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions