[Android] - Jitter/jank problem when panning content

@tvolkert

When panning and zooming content on Android tablets, we're experiencing a repeated "jitter" effect, which has rendered an important app unshippable.

The Problem

We have a tile-based PDF renderer, which renders textures for every tile. Our PDF tiles are pannable and zoomable.

We noticed that as the user panned around the PDF, there was a slight but noticeable jitter that would appear almost all the time. I call it a "jitter" instead of "jank" because typically jank is characterized by a single frame that's very slow, resulting in a single, noticeable jump in the UI. The behavior we're seeing is more like a repeated start-and-stop behavior. It's more like Flutter is rendering every other frame, or something like that.

Here's a video taken directly from an Android tablet, showing the jitter. This reproduction isn't using our PDF system. It's based on the standard Flutter widgets that I included below in the "Reproduction" section.

https://youtu.be/cwhRwdM_2es

Despite the visual indication that something is going wrong with the frame rate, the Flutter profiler shows very few red jank frames.

Why it Matters

This jitter might seem minor as compared to some other Flutter jank issues. However, this is a ship-blocking issue for my client.

My client is trying to release a premium PDF reading experience, built with Flutter. Every member of the team, and every early tester of this app, notices this problem, and complains about it. It feels cheap and buggy. This can't be what my client ships.

If this problem can't be solved, my client will likely have to throw out all of their work on this reading experience, and rebuild it without Flutter.

I've elaborated on these business implications with @tvolkert

Hypotheses We've Evaluated

We've worked quite a bit with @dnfield to try to root cause and solve this jitter. The primary issue with this jitter is that none of our performance signals indicate a problem. The Flutter profiler shows very few jank frames. The Android profiler doesn't show anything that my team is able to identify as an issue. The device reports a high frame rate. We have no mechanism through which to drill down, locate, and resolve the issue. Instead, we're forced to throw things at at the wall and hope something sticks.

Here are some of things we've thrown at the wall.

Maybe its a texture-specific problem: We reworked our tile system to use typical bitmap images instead of texture layers. The problem persists.

Maybe the images are too big: We tried decoding our bitmap images at smaller resolutions. We then tried shrinking the images on disk to make them only 400px wide. The problem persists.

Maybe the issues is that you're using TextureView on Android instead of SurfaceView: We switched back to SurfaceView and the problem persists.

Maybe the issue is the touch input system, rather than the rendering system: We created panning animations to completely bypass the touch input system. The problem persists.

Maybe it's a manufacturing issue with that one Redmi tablet: We've reproduced the issue on multiple Redmi tablets.

Maybe it's an issue with Redmi tablets: We've reproduced the issue (though less of an issue) on the Samsung Galaxy Tab S6 Lite. We've also reproduced it on the client's custom tablet.

Maybe it's an issue with the tile layout implementation: We threw away all of our code and reproduced the jitter with regular Flutter widgets, as described in the "Reproduction" section.

Maybe things would work better on Impeller: At a couple points in this process we tried Impeller. The problem persists.

Reproduction

The following Flutter code reproduces a jitter effect when running animated panning around small images displayed in an InteractiveViewer. A video of the reproduction is shown in "The Problem" section.

For a direct comparison, this code should be run on a Xiaomi Redmi tablet. That said, we've experienced the same problem, to a lesser degree, on a Samsung Galaxy Tab S6 Lite.

Code Sample

import 'package:flutter/material.dart';

void main() {
  runApp(
    const MaterialApp(
      home: _InteractiveViewerDemo(),
    ),
  );
}

class _InteractiveViewerDemo extends StatefulWidget {
  const _InteractiveViewerDemo({super.key});

  @override
  State<_InteractiveViewerDemo> createState() => _InteractiveViewerDemoState();
}

class _InteractiveViewerDemoState extends State<_InteractiveViewerDemo> with TickerProviderStateMixin {
  late final TransformationController _controller;

  int _currentImageIndex = 0;

  @override
  void initState() {
    super.initState();
    _controller = TransformationController();
  }

  final _verticalAnimationDistance = 800;
  AnimationController? _verticalPanningAnimation;
  double? _previousFrameVerticalOffset;
  void _toggleVerticalPanningAnimation() {
    if (_verticalPanningAnimation == null) {
      // Start the animation
      _previousFrameVerticalOffset = 0;
      _verticalPanningAnimation = AnimationController(
        vsync: this,
        duration: const Duration(seconds: 2),
      )
        ..addListener(() {
          final offsetAtTime =
              _verticalAnimationDistance * Curves.easeInOut.transform(_verticalPanningAnimation!.value);
          _controller.value.translate(0.0, _previousFrameVerticalOffset! - offsetAtTime);
          _controller.notifyListeners();
          _previousFrameVerticalOffset = offsetAtTime;
        })
        ..addStatusListener((status) {
          switch (status) {
            case AnimationStatus.dismissed:
              _verticalPanningAnimation!.forward();
              break;
            case AnimationStatus.completed:
              _verticalPanningAnimation!.reverse();
              break;
            case AnimationStatus.forward:
            case AnimationStatus.reverse:
              // TODO: Handle this case.
              break;
          }
        })
        ..forward();
    } else {
      // Stop the animation
      _verticalPanningAnimation!.dispose();
      _verticalPanningAnimation = null;
    }
  }

  final _horizontalAnimationDistance = 400;
  AnimationController? _horizontalPanningAnimation;
  double? _previousFrameHorizontalOffset;
  void _toggleHorizontalPanningAnimation() {
    if (_horizontalPanningAnimation == null) {
      // Start the animation
      _previousFrameHorizontalOffset = 0;
      _horizontalPanningAnimation = AnimationController(
        vsync: this,
        duration: const Duration(seconds: 2),
      )
        ..addListener(() {
          final offsetAtTime =
              _horizontalAnimationDistance * Curves.easeInOut.transform(_horizontalPanningAnimation!.value);
          _controller.value.translate(_previousFrameHorizontalOffset! - offsetAtTime, 0);
          _controller.notifyListeners();
          _previousFrameHorizontalOffset = offsetAtTime;
        })
        ..addStatusListener((status) {
          switch (status) {
            case AnimationStatus.dismissed:
              _horizontalPanningAnimation!.forward();
              break;
            case AnimationStatus.completed:
              _horizontalPanningAnimation!.reverse();
              break;
            case AnimationStatus.forward:
            case AnimationStatus.reverse:
              // TODO: Handle this case.
              break;
          }
        })
        ..forward();
    } else {
      // Stop the animation
      _horizontalPanningAnimation!.dispose();
      _horizontalPanningAnimation = null;
    }
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: InteractiveViewer(
        transformationController: _controller,
        constrained: false,
        alignment: Alignment.center,
        child: _buildImageGrid(),
      ),
      floatingActionButton: _buildAnimationButtons(),
    );
  }

  Widget _buildImageGrid() {
    _currentImageIndex = 0;

    return Column(
      children: [
        _buildImageRow(),
        _buildImageRow(),
        _buildImageRow(),
        _buildImageRow(),
      ],
    );
  }

  Widget _buildImageRow() {
    return Row(
      children: [
        _buildImage(),
        _buildImage(),
        _buildImage(),
        _buildImage(),
      ],
    );
  }

  Widget _buildImage() {
    final image = SizedBox(
      width: 400,
      height: 500,
      child: Image.asset(
        _photos[_currentImageIndex],
        fit: BoxFit.fill,
      ),
    );

    _currentImageIndex += 1;
    _currentImageIndex %= _photos.length;

    return image;
  }

  Widget _buildAnimationButtons() {
    return Column(
      mainAxisSize: MainAxisSize.min,
      children: [
        FloatingActionButton(
          onPressed: _toggleHorizontalPanningAnimation,
          child: const Icon(Icons.compare_arrows),
        ),
        const SizedBox(height: 12),
        FloatingActionButton(
          onPressed: _toggleVerticalPanningAnimation,
          child: const Icon(Icons.arrow_downward_sharp),
        ),
      ],
    );
  }
}

const _photos = [
  "assets/image-4_small.jpeg",
  "assets/image-5_small.jpeg",
  "assets/image-8_small.jpeg",
  "assets/image-9_small.jpeg",
  "assets/image-12_small.jpeg",
  "assets/image-13_small.jpeg",
];

Flutter version:

Flutter 3.10.5 • channel stable • https://github.com/flutter/flutter.git
Framework • revision 796c8ef792 (6 days ago) • 2023-06-13 15:51:02 -0700
Engine • revision 45f6e00911
Tools • Dart 3.0.5 • DevTools 2.23.1

Thanks for the report. Checking this on Realme 6, Android 11, I can observe janks while panning the content:

Demo video

PXL_20230620_090822142.TS.mp4

Adding:

Timeline tracing file: dart_devtools_2023-06-20_15_57_17.943.json.zip
Assets image using in sample code: images.zip (100x100 image)

flutter doctor -v (stable and master)

[✓] Flutter (Channel stable, 3.10.5, on macOS 13.0.1 22A400 darwin-x64, locale en-VN)
    • Flutter version 3.10.5 on channel stable at /Users/huynq/Documents/GitHub/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 796c8ef792 (29 hours ago), 2023-06-13 15:51:02 -0700
    • Engine revision 45f6e00911
    • Dart version 3.0.5
    • DevTools version 2.23.1

[✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0)
    • Android SDK at /Users/huynq/Library/Android/sdk
    • Platform android-33, build-tools 32.0.0
    • ANDROID_HOME = /Users/huynq/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14E222b
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2022.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)

[✓] VS Code (version 1.79.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.66.0

[✓] Connected device (5 available)
    • Pixel 3a (mobile)      • 964AY0WL20                               • android-arm64  • Android 12 (API 32)
    • iPhone (mobile)        • d9a94afe2b649fef56ba0bfeb052f0f2a7dae95e • ios            • iOS 15.7.2 19H218
    • iPhone 14 Pro (mobile) • 98082851-A0C4-4E39-A310-21C48B10A364     • ios            • com.apple.CoreSimulator.SimRuntime.iOS-16-4 (simulator)
    • macOS (desktop)        • macos                                    • darwin-x64     • macOS 13.0.1 22A400 darwin-x64
    • Chrome (web)           • chrome                                   • web-javascript • Google Chrome 114.0.5735.133

[✓] Network resources
    • All expected network resources are available.

• No issues found!

[!] Flutter (Channel master, 3.12.0-4.0.pre.121, on macOS 13.0.1 22A400 darwin-x64, locale en-VN)
    • Flutter version 3.12.0-4.0.pre.121 on channel master at /Users/huynq/Documents/GitHub/flutter_master
    ! Warning: `flutter` on your path resolves to /Users/huynq/Documents/GitHub/flutter/bin/flutter, which is not
      inside your current Flutter SDK checkout at /Users/huynq/Documents/GitHub/flutter_master. Consider adding
      /Users/huynq/Documents/GitHub/flutter_master/bin to the front of your path.
    ! Warning: `dart` on your path resolves to /Users/huynq/Documents/GitHub/flutter/bin/dart, which is not inside
      your current Flutter SDK checkout at /Users/huynq/Documents/GitHub/flutter_master. Consider adding
      /Users/huynq/Documents/GitHub/flutter_master/bin to the front of your path.
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 5182697c0d (2 hours ago), 2023-06-19 21:37:33 -0400
    • Engine revision a91bb3f566
    • Dart version 3.1.0 (build 3.1.0-230.0.dev)
    • DevTools version 2.24.0
    • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly
      to perform update checks and upgrades.

[✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0)
    • Android SDK at /Users/huynq/Library/Android/sdk
    • Platform android-33, build-tools 32.0.0
    • ANDROID_HOME = /Users/huynq/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14E222b
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2022.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)

[✓] VS Code (version 1.79.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.66.0

[✓] Connected device (2 available)
    • macOS (desktop) • macos  • darwin-x64     • macOS 13.0.1 22A400 darwin-x64
    • Chrome (web)    • chrome • web-javascript • Google Chrome 114.0.5735.133

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 1 category.

maybe pictures too big?

As we discussed offline, in a previous iteration of this problem the large images and usage of a texture view were demonstrably contributing to jank as seen in timeline traces and perfetto tracing showing long GPU times per frame.

Making the images smaller and using a surface view vastly cut down on GPU times (from ~9ms per frame to ~4ms per frame) on the Samsung tablet. I don't have a Redmi to test with, although the Redmi does have a different GPU (Mali vs Adreno in Samsung). I can try to take a look at the latest sample.

@dnfield

@dnfield I assure you that across every change you recommended, and which we directly tried, the jitter hasn't been eliminated. In fact, to my perception, I'm not sure it's gotten any better at all.

Your comment further makes my point about signals. As far as I can tell, we lack any dependable signal which correlates with this UX problem. I have no idea why that might be the case. This problem seems to originate in the depths of the engine in areas that I know nothing about. So all I can do, as we have done, is try whatever idea comes to mind, and report back.

The problem persists when using a SurfaceView instead of a TextureView. The problem persists when decoding images at a smaller resolution. The problem persists when reducing the actual image file resolutions down to 400x500 per image, which is what I used in the reproduction demo in this post. And the problem persists even when using Flutter's own InteractiveViewer with an incredibly simple widget tree, instead of using our complicated tile system. The problem persists.

This is why we're desperate. We feel like we're debugging the Flutter engine and we simply don't know what else we're supposed to do to build an experience that we can reasonably ship to customers. We remain open to trying anything that will allow us to ship the experience we've designed - an experience that we believe traditional Android apps have been shipping for a long time.

What are the assets for this repro?

https://github.com/Flutter-Bounty-Hunters/page_list_viewport/tree/main/example/assets

Is it frame pacing? Sure, each frame has both UI and Raster each under <11.111ms. And a sample of frame times may be <11.111ms. But are you really scheduling and drawing a complete 90 per second? (how is that green 90 being measured?) Stepping through the youtube video I can see some places where there is no change between captured frame. I have always suspected some frame pacing problem in flutter as the cause of some of my "mysterious/invisible" lag. Basically, when UI+Raster goes from taking 11ms to 11.2ms, you have to skip a frame when the pipeline goes from 1 frame latency to 2 frames latency. And from your devtools graph, it looks like you could be around that mark.

It would be interesting to have a way to opt into constant 2-frame latency to avoid any such pacing issues.

An Android System Trace could be useful to learn more, as it contains the real system vsync and buffer swap timings.

Add "Frame Request Pending" lag to collected metrics (#135279)

Add "Frame Request Pending" lag to collected metrics (flutter#135279)

Upload generated frame-request-pending stats (#135645)

Upload frame_request_pending_latency [attempt #2] (#135890)

Add "Frame Request Pending" lag to collected metrics (flutter#135279)

Upload generated frame-request-pending stats (flutter#135645)

Upload frame_request_pending_latency [attempt #2] (flutter#135890)

I wonder if this could be an accounting issue similar to #129894?

It think something like https://developer.android.com/games/sdk/frame-pacing/ could help solve the pacing/jank issues like these.

"The library handles multiple refresh rates if they are supported by the device, which gives a game more flexibility in presenting a frame. For example, for a device that supports a 60 Hz refresh rate as well as 90 Hz, a game that cannot produce 60 frames per second can drop to 45 FPS instead of 30 FPS to remain smooth. The library detects the expected game frame rate and auto-adjusts frame presentation times accordingly. The Frame Pacing library also improves battery life because it avoids unnecessary display updates. For example, if a game is rendering at 60 FPS but the display is updating at 120 Hz, the screen is updated twice for every frame. The Frame Pacing library avoids this by setting the refresh rate to the value supported by the device that's closest to the target frame rate. "

Navigation Menu

Repository navigation

[Android] - Jitter/jank problem when panning content #129150

The Problem

Why it Matters

Hypotheses We've Evaluated

Reproduction

26 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Android] - Jitter/jank problem when panning content #129150

Description

The Problem

Why it Matters

Hypotheses We've Evaluated

Reproduction

Activity

huycozy commented on Jun 20, 2023

pickalex commented on Jun 20, 2023

dnfield commented on Jun 20, 2023

matthew-carroll commented on Jun 20, 2023

dnfield commented on Jun 20, 2023

matthew-carroll commented on Jun 20, 2023

moffatman commented on Jun 20, 2023

26 remaining items

brunogm0 commented on Jan 10, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions