Support structured error messages

Currently Flutter error messages are at the core large blobs of text with limited structure which has been often slows users down fixing errors as it is hard to distinguish the signal from the noise and so users get lost in long non interactive stack traces and distinguishing what parts of the error is the core problem that occurred, background information, or hints on how to fix the problem.

We would like to add structure to error messages making the errors practical to format beautifully on the client. The goal is to have enough structure that users can quickly scan the message to see hints for how to solve the problem, the core error message, links to interactive ui to fix the ui, and interactive links to open objects related to the error in the Inspector. Similarly, errors will start with the minimum useful information to show expanded with ui affordances to expand out to see additional information such as extra stack frames and additional parent widgets.

Basic examples of what we could do with added structure:

Note in this example, the widget names are clickable navigating the user to the matching location in the widget tree. Lists of ancestors and stack frames stack out truncated and only expand when a user clicks. Portions of the error that are hints are shown with a different background color and additional padding. The core error message is in red so it is harder to miss.

Example for an error on the rendering layer. In this case the RenderObject is similarly clickable to navigate over to the inspector to view.

Example of a ui mock of a more advanced error message that really lets the user understand a layout issue without switching contexts. Note the links to interactive UI elements to apply fixes, graphical visualization of the overflow failure that occurred, and header titles calling out core elements of the error. All of these ui features are speculative and may or may not prove out with user studies but they are features are are promising enough that we would like to iterate on them without churning through refactoring code in package:flutter for every UI iteration.

Prototype CL showing what it would take to add structure to all flutter errors at the cost of
more structured syntax writing error messages than writing unstructured blob of text.
#27202
If nothing else this CL provides a good overview of what blocks of content exist in the existing error messages.
Some observations: there are common tasks like embedding a structured object for a RenderObject that are called 20 times across the code base. Currently the same boilerplate code to emit strings describing a widget and its immediate children is repeated for each of these cases. If you look at FlutterErrorBuilder, WidgetErrorBuilder, and RenderErrorBuilder you will see the full list of all commonly occurring patterns of structured data that is included in errors (StackTrace, RenderObject, Widget, ancestor chain of widget) and error parts (e.g. hint, error, constraint).

The core concern about this prototype CL is that the builder syntax used is not productive enough for users to use and will reduce the number or quality of errors. A counter point is that it took about 2 days to refactor all errors in package:flutter to use the format. For someone who wants to write an error message particularly in the simpler case an alternate approach is to add named arguments provided on the default FlutterError constructor but unfortunately once you exceed the capabilities of that constructor you will have to start using builder syntax. Builder syntax is needed to handle multiple messages of the same type such as multiple hints. Looking at the facts on the ground there are multiple flutter errors with multiple separate hints.

Benefits are it is hard for users to do the wrong thing and there are builder classes that clarify what helper methods should be used depending on the context the error is being thrown in (base, Render, or Widget).
The structure of the error messages is the same DiagnosticsNode format used by the WidgetInspector making it reasonable to include errors directly in the widget tree as properties of a widget indicating where the problem occurred.

Risks:

The builder syntax may be unwieldy and demotivate framework authors and users from writing good structured errors.

Alternate design:

Markdown lite for Flutter errors.
Add magic characters in the error templates that indicate error parts (e.g. hint, error, contract).

Risks:

Markdown like templates may be fragile and error prone as users will get extremely limited compile time checking that they are using the scheme in contrast to the other design where you will get an immediate compile time error if you attempt to pass the wrong type of object to a builder method.
Similarly, opening and closing blocks in this case will be done with some level of tags embedded as magic unicode characters so mismatched tags will cause the usual problems. This can be mitigated with careful test coverage although the current code base shows that there is limited coverage of errors.

Breaking change:

This change will modify the toString for all objects implementing Diagnosticable. As Diagnosticable objects have great toString functions, there tend to be a lot of tests that use them to easily write golden tests. Currently modifying the toString for Diagnosticable breaks 19 tests in package:flutter and would break third party tests as well. Modifying the equalsIgnoreHashCodes matcher to ignore the temporary object ids embedded in the toString as well would help somewhat but still leads to 16 tests failing. This is because not all tests using the equalsIgnoringHashCodes matcher as it wouldn't make sense for them to as the golden output did not contain hash codes and because adding the magic unicode character impacts line breaking for multiple diagnostics output so even though equalsIgnoreHashCodes is used. Fixing the line breaking helper used reduces the number of test failures in package:flutter to 13. The failures would be easy enough to fix by switching to a matcher that ignores the magic characters but users would need to know such a matcher exists and use it. Otherwise the tests will become mysteriously flaking as the object ids in the matchers would change unpredictably.

This breaks anyone who linebreaks debug output and doesn't know about the magic characters. Lines will be broken incorrectly resulting in shorter lines than otherwise. The common debugWrap helper can be updated to reflect the magic characters to mitigate this risk. With debugWrap modified to ignore magic characters the # of failing tests in package:flutter is reduced to
People with golden tests should expect to have to rebaseline them periodically so perhaps this isn't that big a deal.

Alternatives to minimize the breaking:

Make the magic unicode characters be off by default in tests. This will mean that debuggers depending on them will be broken for tests and tests testing debug output are testing the wrong thing but would minimize the breakage.

Example failing test:

    expect(description, <String>[
      'backgroundColor: Color(0xff123456)',
      'elevation: 8.0',
      'titleTextStyle: TextStyle(inherit: true, color: Color(0xffffffff))',
      'contentTextStyle: TextStyle(inherit: true, color: Color(0xff000000))',
    ]);

now the toString for each TextStyle contains at least one marker character.

Fixing debugWrap to account for the magic characters still leaves 13 failures.

Unsolved challenges:

Object lifecycles using this scheme are fragile as Dart string templating is quite limited so we have no efficient way of distinguishing a toString call part of a template that will be used for an error from a time someone is just calling toString arbitrarily on an object. This suggests the only feasible approach to avoid memory leaks is to have only a ring buffer of object references.
This can work alright as long as care is taken to quickly deserialize the object references out of the format when capturing them for display at a later point either in a logging view or in an interactive view on the device or in the inspector such as a FlutterError widget.

@goderbauer

@goderbauer @Hixie @pq @InMatrix @devoncarew

CL with strawman approach adding magic unicode markers in the string to indicate the styles instead of Dart code. master...Hixie:pretty_errors

To make it apples to apples, maybe we could look at a few side-by-side examples?

To start, how would this one look in a "Markdown lite" style?

throw FlutterError.from(FlutterErrorBuilder()
    ..addError('AnimationController.dispose() called more than once.')
    ..addHint('A given $runtimeType cannot be disposed more than once.\n')
    ..addProperty('The following $runtimeType object was disposed multiple times', this)
);

Here's a diff with side by side examples.
I think with the pretty_errors style it would be really important to enforce that the first line always has to be the red error problem line as adding it specifically definitely seems cumbersome.
81062a5

Builder style is on the left, magic unicode characters is on the right:
Here's an example of some complex error building cases:

Here are some simple cases:

Medium complexity case:

I find it easier to write (in general) If I have some structure to guide me. Speaking for myself, the API provided by the structured error messages is helpful, especially when combined with a corpus of existing error messages.

Regarding the API on the right, I think it would be difficult to make sure the characters are inserted correctly. This would presumably require more manually checking.

Regarding the API on the right, I think it would be difficult to make sure the characters are inserted correctly. This would presumably require more manually checking.

Or help from customized analysis. With either approach we could certainly add linting to nudge authors towards well-formed errors. Having said that, the explicit structure case would be far easier...

@Hixie

@Hixie, in response to Jacob's original PR you wrote:

I think we should attempt to find a solution that has a much lighter API. My fear is that engineers (that is to say, we) in general already find writing error messages to be highly aversive, and anything we do to raise the barrier to writing error messages would only make things worse. Ideally making objects in a string inspectable would require zero overhead from the perspective of framework developers. For example, one could imagine changing Object.toString or Diagnosticable.toString to encode information in the output (which normally gets stripped out but which IDEs and maybe the flutter tool can detect).

I don't know if the embedded tag idea is a straw-man proposal but I'm curious why you think something like it would be less of a barrier to authoring good errors (especially in the more complex cases). Like @jonahwilliams says above, to my eyes having to navigate an implicit structure is more burdensome (and has the added disadvantage of being much harder to accommodate with tooling). In contrast, a builder pattern is a pretty common approach in cases like this and I would expect it to be pretty familiar to authors generally, is less prone to simple errors and is easier to tool.

Could you maybe elaborate on your misgivings?

@InMatrix: any thoughts from the UX perspective?

I'm very excited about how we'll be able to consume this info from tools; it should improve both presentation and navigation, and give us more options going forward.

I'm less invested in how the errors are constructed; I'm sure the framework authors will have good insight here.

In terms of the UX of using a builder class (FlutterErrorBuilder) - if we feel that it adds a slight barrier to writing errors, I wonder if we can't identify a set of errors that could be formed w/ a slightly simpler API? Perhaps FlutterError could have a constructor with a set of named params. This may not work for creating a more sophisticated error - people could use the full FlutterErrorBuilder API there - but could work for simpler cases, and be slightly less work for the author when building the error.

Some edge cases with multiple hints and hints with multiple paragraphs.
These cases are useful to test how complex the markdown style syntax needs to be to avoid exposing api cliffs.
62d86ac
This case requires ending and closing tags. I'm not sure how discoverable this would be and likely most times users would forget the tags to open and close the block to ensure the hint goes over multiple paragraphs.

This case is fine because even though the hint is multiple paragraphs, we can automatically link them due to the mark down syntax implied by the indent and/or line ending in colon.

https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo#avoid-exposing-api-clif

@devoncarew

@devoncarew In the strawman CL the FlutterError class does have an API with named parameters for the common cases which could be used a lot more than it is with light wordsmithing of the flutter errors. Currently not all errors currently meet the expected order for error blocks but it would be reasonable to wordsmith the errors to match that consistent style.
If we went this route, we would want to add a lint if you call specific methods with named parameters in the wrong order as otherwise there would be significant bugs to error text not being in the error implied by viewing the definition code. In general I think such an api would be a fine alternative for common cases and the lint would be useful for other APIs where parameter order is important to a user understanding the API.

I find it easier to write (in general) If I have some structure to guide me. Speaking for myself, the API provided by the structured error messages is helpful, especially when combined with a corpus of existing error messages.

Regarding the API on the right, I think it would be difficult to make sure the characters are inserted correctly. This would presumably require more manually checking.

As graphical displays of the errors IDEs become more advanced and diverge more from the straight text output I think having the structure guide users in the right direction is a big win. With the the magic characters option I think we would want to provide a way to preview the renders of the structured output in the IDE to visually check that the error looks generally as expected when writing code. However this is a bit hard for complex cases like some of the ones shown where the errors are build up incrementally over multiple lines or where calling helper methods to perform common tasks like building a beautiful display of a render object where a single helper will result in 10+ lines of error output with significant structure.

@Hixie

Parts 3 and 4 are an explanation of what the error is, relevant background, etc.
I understand how to distinguish those four concepts. What I don't understand is how to distinguish parts 3 and 4.

Part 3 is explaining what's going on in the program, while Part 4 is talking about the programmer and their potential actions. In addition, two features make Part 4 distinctly different from Part 3. First, it's led by a goal statement: "To introduce a Material Widget..." Second, it talks from the second-person point of view, which is marked by the use of second-person pronoun: "you can either directly include one..."

@Hixie Please let me know if you still have concerns about the distinction. The users in our study took advantage of the visual separation between hint (titled "Potential Fix" in the study) and the rest of the error. I understand it's a separate concern that error authors might not see the distinction. It's Okay that some authors don't initially take advantage of the "hint" category. We or other contributors could easily call out the "hint" part later, as long as there is a way to do so via the API.

Looking at the data, the FlutterError class constructor is barely used outside of Package flutter so I actually think we could go with the breaking change of making the one FlutterError constructor take a list of DiagnosticsNode objects without causing much pain at all.
I think having FlutterError take a list of DiagnosticsNode objects leaves us with a core API that we will never have to change and can meet all our use cases going forward as DiagnosticsNode objects can support arbitrary structure.
I will push on #27673
to support nice inline display of objects without having to resort to the more cumbersome DiagnosticsProperty syntax unless a truly custom display of a property is really desired.

I'll start working on a branch summarizing just the changes we have consensus on to make sure we are on the same page.

I've filed #27695 to track the related issue of barely used FlutterErrorDetails subclasses that could be eliminated or at least deprecated as part of simplifying to a standard way to expose metadata.

Here's a branch summarizing the changes we have consensus on and changes i'd like to get consensus on next so I can start the real refactor avoiding doing throwaway work.
https://github.com/flutter/flutter/pull/27791/files

I've added comments to the CL calling out parts that are areas I'd like feedback on to reach consensus.

Are you ok with that breaking change or would it need to be a separate named constructor?

It's probably ok... Hard to tell how often people use it. Please make sure to go through the breaking change process: https://github.com/flutter/flutter/wiki/Tree-hygiene#handling-breaking-changes

(In particular, part of that process is asking the community for feedback on whether they think this will break them.)

how would you like to see that we avoid having users add a Widget to the error description when an Element is available?

We should just put that in the style guide and make sure reviewers are aware of it. Maybe we can add a lint or something.

Part 3 is explaining what's going on in the program, while Part 4 is talking about the programmer and their potential actions.

The following two statements seem identical to me, but one sounds like it is "explaining what's going on in the program" while the other seems to be "talking about the programmer and their potential actions":

The foo constructor requires a bar.
When calling the foo constructor, provide a bar.

These statements are in my mind identical in meaning. It seems very strange to me that depending on the mood of the person writing the error message, we might sometimes have something be described as a "detail" and sometimes as a "resolution".

I understand it's a separate concern that error authors might not see the distinction. It's Okay that some authors don't initially take advantage of the "hint" category. We or other contributors could easily call out the "hint" part later, as long as there is a way to do so via the API.

As the person who's probably written most of the error messages we're talking about, not to mention the person who's reviewed the PRs for most of the rest, I care very much that I should understand how to write error messages. I don't understand the distinction described here. As in, I wouldn't be able either write or review code that was required to distinguish between "description" and "hint".

@jacob314

The foo constructor requires a bar.
When calling the foo constructor, provide a bar.

These two statements are not really identical to me. While the second version tells me what I need to do to get out of the error state, it doesn't tell me the definitive cause of the error. If I were to complete the second version with a cause, I could come up with more than one plausible result:

When calling the foo constructor, provide a bar, because the foo constructor requires a bar.
When calling the foo constructor, provide a bar, because bar is referenced inside foo.build.

That said, I do agree that for simple error messages (e.g., the example you gave), a resolution could be implied by the error description, and there is no need to call out a "hint". It's optional, but it's useful when the error message is long and detailed guidance is given to the user to resolve the error. Here is an example from @jacob314's earlier prototype:

    throw FlutterError.from(FlutterErrorBuilder()
      ..addError('Scaffold.of() called with a context that does not contain a Scaffold.')
      ..addDescription(
        'No Scaffold ancestor could be found starting from the context that was passed to Scaffold.of(). '
      'This usually happens when the context provided is from the same StatefulWidget as that '
      'whose build function actually creates the Scaffold widget being sought.')
      ..addHint(
        'There are several ways to avoid this problem. The simplest is to use a Builder to get a '
        'context that is "under" the Scaffold. For an example of this, please see the '
        'documentation for Scaffold.of()',
        url: 'https://docs.flutter.io/flutter/material/Scaffold/of.html'
      )
      ..addHint('A more efficient solution is to split your build function into several widgets. This '
      'introduces a new context from which you can obtain the Scaffold. In this solution, '
      'you would have an outer widget that creates the Scaffold populated by instances of '
      'your new inner widgets, and then in these inner widgets you would use Scaffold.of().\n'
      'A less elegant but more expedient solution is assign a GlobalKey to the Scaffold, '
      'then use the key.currentState property to obtain the ScaffoldState rather than '
      'using the Scaffold.of() function.')
      ..addSeparator()
      ..addProperty('The context used was', context)

Isn't the distinction between detail and hint clearer in this example? Styling Hint differently can motivate the user to read through the message, instead of scaring them away by showing a large blob of text with unclear value at first sight.

I agree that in this case there are hints and that they are distinct from the error description.

How about this for a definition:

error message: A short (one line) description of the problem that was detected. Error messages from the same source location should have little variance, so that they can be recognized as related. For example, they shouldn't include hash codes.

description: An explanation of the problem and its cause, any information that may help track down the problem, background information, etc.

hint (optional): Specific, non-obvious advice that may be applicable.

Information that is always true (e.g. "this argument is required" or "you must include this argument") goes in the description. Information that may not be useful in all situations ("Consider using the Foo widget...") goes in a "hint" section.

That definition sounds good to me!

SGTM. That definition provides a nice crisp separation between what is a hint and what is a description. I'll go ahead and apply it to the existing error messages.

Landed the implementation of structured errors in package:flutter.

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

Navigation Menu

Repository navigation

Support structured error messages #27327

Basic examples of what we could do with added structure:

Risks:

Alternate design:

Risks:

Breaking change:

Alternatives to minimize the breaking:

Unsolved challenges:

29 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support structured error messages #27327

Description

Basic examples of what we could do with added structure:

Risks:

Alternate design:

Risks:

Breaking change:

Alternatives to minimize the breaking:

Unsolved challenges:

Activity

jacob314 commented on Jan 31, 2019

jacob314 commented on Jan 31, 2019

pq commented on Jan 31, 2019

jacob314 commented on Jan 31, 2019

jonahwilliams commented on Jan 31, 2019

pq commented on Jan 31, 2019

pq commented on Jan 31, 2019

devoncarew commented on Jan 31, 2019

jacob314 commented on Jan 31, 2019

jacob314 commented on Jan 31, 2019

jacob314 commented on Jan 31, 2019

29 remaining items

InMatrix commented on Feb 8, 2019

jacob314 commented on Feb 8, 2019

jacob314 commented on Feb 8, 2019

jacob314 commented on Feb 11, 2019

Hixie commented on Feb 11, 2019

InMatrix commented on Feb 12, 2019

Hixie commented on Feb 15, 2019

InMatrix commented on Feb 15, 2019

jacob314 commented on Feb 19, 2019

jacob314 commented on May 1, 2019

github-actions commented on Aug 14, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions