Repository navigation
Proposal: Framework needs to be aware of physical pixels #151065
Description
I raised this a while ago, but then gave up and closed the issue. It is a fundamental issue though so I'm reopening it.
Consider the following snippet:
runApp(
Align(
alignment: Alignment.topLeft,
child: Padding(
padding: const EdgeInsets.only(left: 20, top: 20),
child: Container(
decoration: BoxDecoration(
color: const Color.fromRGBO(255, 166, 0, 0.428),
border: Border.all(color: Colors.yellow, width: 1)),
width: 64,
height: 45,
child: Center(
child: Container(
decoration: BoxDecoration(
border: Border.all(color: Colors.yellow, width: 1)),
width: 42,
height: 22,
),
),
),
),
),
);When you run this on a device with 100% pixel scaling, you get the following result:
A commonly used 4k 27" monitor on windows will run with 150% pixel scaling factor. Running any Flutter application on such setup, you can pretty much guarantee that every single widget border will be blurry. It is near impossible to deliver a sharp looking Flutter application on Windows that works correctly with fractional scale factors (and as mentioned above sometimes even with a basic 100% scaling).
This is a problem that has been solved pretty much by every modern toolkit (WinUI, Compose, Browser) except for Flutter, with keeps stubbornly pretending that it is rendering into an infinite resolution surfaces, which is just not the case.
There are many places where this causes issues i.e. #14288, #90926, #25531, #31707 or #143420, last of which has been solved with a really ugly hack.
Another big issue stemming from this is that all layout in Flutter is performed in logical pixel space on floating point numbers. Floating point math is not associative, which causes many subtle and difficult to find issues, such as this:
Proposed solution
The solution for this issue is rather straightforward, unfortunately it's also a major breaking change which makes applying it difficult.
- Remove the device pixel ratio scaling transform from the app. Using transformation for this is simply too blunt and causes the issues above. This transform creates an illusion that we don't need to deal with the presence of physical pixels, but this quickly falls apart when running of anything with less than 200% device pixel ratio.
- All user specified unit (i.e. border size, padding, dimensions, font size) would be specified in device independent coordinates. This could even be an extension type to make things explicit, though I'm not sure what would happen with existing geometric primitives like
OffsetorRect. - All layout would be performed in physical pixel coordinates and integers.
- All painting would be performed in physical pixel coordinates.
The conversion from logical to physical pixel space is a simple scale + round(). i.e.
For example imagine that you have 1px border around widget and 1.5 device pixel ratio. In which case
physicalPixels = (1 * 1.5).round() = 2
This would offset the layout by two physical pixels, and would also paint as two physical pixels.
Constraints and geometry would probably need to be converted to integers and in physical pixel space. There is an issue where some of the Widget API is using constraints (i.e. LayoutBuilder, ConstrainedBox), we would need to have a convenient way to convert back and fort from physical to logical pixel space.
Changes to layout
All layout would be performed in physical pixel space on integral coordinates. This would ensure that under normal circumstances (no arbitrary transform in the app), all widgets would be laid out on exact physical pixel boundaries. This would require some changes in the layout algorithms, for example consider flex with 3 expanded widgets constrained to 100 physical pixels. The flex can only produce integral coordinates and must fill all the space, so the final dimensions of the widgets would be 33, 33 and 34 pixels.
Arbitrary scale transforms
Most Flutter applications likely don't use arbitrary scale transforms, apart from transient things like a zoom effect on hover. I'd wager that in majority of Flutter application there is exactly one persistent scale transform in the layer tree - the device pixel ratio scale. With the changes proposed here this would be removed, replaced by manual scale + roundToInteger in appropriate places in framework.
Of course nothing prevents user from adding other transforms that will not respect physical pixel boundaries, but that case aligning to physical pixels is likely not a concern so it is orthogonal to the proposal here.
Engine changes
This would not require any engine changes at all. This needs to be all handled at framework level, since it impacts layout. If done correctly, final transform in the layout tree will all be pixel snapped by the time they reach the engine.
Migration
This is the tricky part. If anyone has an idea of how we could gradually migrate Render Objects, Constraints, Geometries and painting to physical space and integral coordinates (for layout) I'd love to hear about it.
Activity
jonahwilliams commentedon Jun 30, 2024
What is the actual boundary where we'd be changing from dp to pixels? I'd assume:
But I'm not sure where that exact boundary lies. Breaking changes aside, It would be nice if we could keep dp/physical pixels separate. Because of the layered architecture of flutter I could imagine that developers might need to interact with both kinds of pixels and that seems like a recipie for confusion.
I mean this is part of the reason this idea isn't getting a lot of uptake, what you're proposing is a massively breaking change without a clear path on how to do it.
Assume I'm not against the change on principle (I'm not), and think up with a plan for how it could be done. I'm actually willing to bet that just changing the set of layout widgets/ROs in the framework to consider physical pixels is not that hard - its more about the long tail of custom layout et cetera. Suppose there was a way to do it in an opt-in or with a compile time switch; does a gradual roll out make this feasible?
jonahwilliams commentedon Jun 30, 2024
So maybe there are three changes here, if I'm correct:
EDIT: some thought
knopp commentedon Jun 30, 2024
Pretty much so. Though this would assume that we don't make distinction on type level between logical and physical pixels (and allow for layout to produce floating point physical pixel coordinates, which is something we want to avoid). I see some issues with this, but it's probably much more sensible for gradual migration.
knopp commentedon Jun 30, 2024
Maybe render object could also declare if it is aware of physical pixels, and if not the framework would convert constraints and geometry back and forth until the ecosystem gets migrated.
jonahwilliams commentedon Jun 30, 2024
For things like Rect, Size, et cetera we could create IRect/ ISize / IOffset that require integral values. But thinking about this more, I don't know that we could change the Canvas API. We'd still want people to be able to specify floating point coordinates for arbitrary draws. There are also all sorts of drawing operations that don't easily align to physical pixels, think arbitrary paths or strokes. So maybe we can't enforce integral values outside of layout?
knopp commentedon Jun 30, 2024
Mixing logical and physical coordinates is going to get quite hairy though. For example:
flutter/packages/flutter/lib/src/widgets/reorderable_list.dart
Lines 1114 to 1118 in 1c9c375
Here
SizedBoxis created with the item extent that was determined by reading render object geometry. So without type system distinction this would just compile, even though we're mixing logical (SizedBox) and physical (extent) values.Definitely. I would not suggest changing canvas API in any way.
Agreed. I think we only need to make sure that render objects land on physical pixel boundaries (layout), and common things like decorations (mostly border widths) and scaled and rounded before painting. But this would be handed at decoration painter level, not canvas level.
jonahwilliams commentedon Jun 30, 2024
Lets assume that an application will never mix migrated/unmigrated ROs, instead we have two states:
In this world, could we get away with converting all sizes/constraints from the widget level from dp -> physical pixels and adding rounding?
Example: https://github.com/flutter/flutter/compare/master...jonahwilliams:flutter:foobar_layout?expand=1
Having said that, I'm not exactly sure where I would round. Maybe constraints should be unrounded, and we only round for final size/posiiton?
knopp commentedon Jun 30, 2024
I think I'd need to round constraints, otherwise we might get edge cases where the geometry is not inside constraints, i.e. something like
maxWidth=14.999999and geometry width being 15. Or just relax some assertions :)If I have a third party package and migrate render objects to physical pixels, is it supposed to work in an application with
kPhysicalPixelLayoutset tofalse? If so, it might be difficult to migrate (every package having to support both modes), if not, not sure how the big switch would help with migrating the ecosystem.jonahwilliams commentedon Jun 30, 2024
Im making a few assumptions,.namely that the actual number of 3rd party layout ROs is quite small. And that the cost of supporting a mixed mode isn't worth it.
knopp commentedon Jul 1, 2024
There is another approach here - something I've implemented in pixel_snap. The approach there is that everything remain in logical coordinates, but layout and drawing are snapped in a way where when multiplied by the pixel aspect transform they land exactly on physical pixel boundaries.
I.e. this is the default pixel snapping function
The benefit of this would be that the change would be much less disruptive. We are not mixing physical and logical coordinates anywhere. Could be rolled out gradually, is quite easy to turn off (identity pixel snap function).
The downside is that we're still calculating layout with floating point numbers. But we can account for that.
jonahwilliams commentedon Jul 1, 2024
Is there really that much of a disadvantage using floating point numbers? As soon as the layout information turns into painting commands its going back to floating point.
knopp commentedon Jul 1, 2024
The problem is with the error accumulating. Imagine that you have 90 widgets, each of them takes 10 physical pixels and 150% device pixel ratio. You do the layout in logical pixels, so each of them is 10.0/1.5=6.66666 logical pixels. You sum them up, expecting 900/1.5 = 600 logical pixels, but it's actually 600.0000000001. This is not a new problem, but it's an annoying one and doing layout in physical pixels would resolve it nicely.
For example I was bitten by this in
SuperSliverList, where if thepaintExtentdidn't exactly matchremainingPaintExtent, viewport wrongly assumed that there is another sliver visible after this one.It's not a dealbreaker, but it would be nice to be able to prevent issues like this by having integral layout rather than rely on ad hoc fixes.
jonahwilliams commentedon Jul 1, 2024
If the cost is the occassional rendering bug we can work around, that doesn't seem too bad?
lukepighetti commentedon Jul 23, 2024
lukepighetti commentedon Jul 23, 2024
knopp commentedon Jul 23, 2024
I think you (and others downwoting it) misunderstood @jonahwilliams's comment. The "occasional rendering bug" refers to situation when doing pixel snapped layout in logical pixel coordinates for which some extra work would be needed to account for floating point math not being associative. It would still fix the fidelity issues mentioned in the issue description and overall I think it is a good compromise.