Provide Dart grammar in a machine-readable format

I'm working on a Visual Studio extension for Dart, and need to be able to parse Dart code for more reliably syntax highlighting (and ultimately statement completion, etc.).

The Dart spec contains a lot of EBNF definitions for the grammar, but it's embedded all over the spec, and difficult to extract (in an automated way).

Is it possible to provide/maintain some sort of machine-readable definition for the Dart spec? It doesn't really matter what the format is, as long as I I can parse it.

This would help tremendously with tooling support; which will ultimately help adoption of Dart!

@kaendfinger

This comment was originally written by @kaendfinger

Well, here is a BNF Grammar that IDEA uses, I'm sure somebody could clean it up and use it: https://github.com/JetBrains/intellij-plugins/blob/master/Dart/src/com/jetbrains/lang/dart/Dart.bnf

@gbracha

There is also a dart parser in this library - https://github.com/renggli/PetitParserDart

cc @gbracha.
Added Area-Language, Triaged labels.

@kaendfinger

@kaendfinger The IntelliJ one might be a good starting point, though seems to have anwful lot of "custom" tweaks to it. I think there's great value in an officially maintained clean one.

@kevmoo... I did find that one, but it's not very parseable. There are probably better sources if I need to do manual work (such as the spec, or one of two parsers in the Dart repo), but I was hoping to minimise that.

For performance, the parser really needs to be native for the environment I'm writing for (which means .net). There are lots of parsers out there that would probably just work out-of-the-box with a reliable grammar! :)

For what it's worth; this suggestion is of far less value to me now I've been pointed at the Analysis Server!

@gbracha

A usable API is indeed more valuable.

Set owner to @gbracha.
Added Accepted label.

Today we have Dart.g, which is a specification of the Dart grammar written for ANTLR v3. Closing.

@eernstg

@eernstg has that file moved? It's a 404 now (and FWIW, it's also linked to from https://github.com/dart-lang/sdk/wiki/The-Dart-specification-parser).

What's the current best thing to try and parse if I wanted to see if I can generate a TextMate syntax highlighting file? (I keep finding issues in VS Code like this so wonder if I could do something more reliable than hacking regexes every month! :-))

It's here now: https://github.com/dart-lang/sdk/blob/master/tools/spec_parser/Dart.g. I corrected several links in the wiki page, too.

What's the current best thing to try and parse

Dart.g and spec_parse.py should be the best choice for that. I've just updated it in https://dart-review.googlesource.com/c/sdk/+/101500 such that it covers a few features added recently (some of them not yet landed, like types of the form T?), so you might see some failures if you are parsing code that uses these very new features.

hacking regexes every month!

I can see why you'd want to avoid that! ;-)

Hey from 2021!

Looks like official EBNF schema for dart still doesn't exist in this world. Any update of this proposal?

The language specification contains grammar rules. In addition to the language specification itself, we have several specifications of new features (e.g., for null safety or inference).

This means, unfortunately, that it is necessary to take multiple documents into account in order to gather the grammar for a specific version of the language, and in particular in order to include the latest features.

So you may be justified in claiming that it doesn't exist, but you should be able to create it when needed. ;-)

The file https://github.com/dart-lang/sdk/blob/master/tools/spec_parser/Dart.g is still a good place to look, in order to get a complete and reasonably current grammar for Dart. It is being used to perform grammar design experiments for new features, and to verify that the language specification grammar rules are in sync with the actual language implementation (by keeping the spec and Dart.g in sync, and checking that Dart.g will parse the language test suite).

So we don't have a perfect solution (and in particular: no complete grammar for each version of the language), but I think it's sufficiently complete to useful in practice.

@eernstg

@eernstg Unfortunately, antlr != ebnf. First one is java-likely parser, second one is standartized ISO grammar format, which is supported by literally any devkits and sdks today.

So, honestly, Dart.g file makes no sense, unlike similar ebnf grammars for golang, rust, javascript, java, lua, python, ruby, etc. etc. etc. All of listed languages has officialy approved ebnf schema for side open source developers. Dart is not.

Thanks this, for example, i can't make dart linters, cause i want to inspect dart AST for linting, and guess what? I can't get ast of my dart file, cause i don't know how to parse it 🙃

I think the most practical approach would be to use existing Dart parsers, e.g., by building a tool that imports the dart analyzer package.

Admitted, this isn't always possible: If you wish to provide a Dart grammar specification to an existing non-Dart tool, it could require any format, including standard EBNF (I assume you mean this one: https://standards.iso.org/ittf/PubliclyAvailableStandards/s026153_ISO_IEC_14977_1996(E).zip).

However, if you wish to use a parser generator to parse Dart programs based on an EBNF grammar specification then it would have to specify an unambiguous grammar.

That isn't quite possible with Dart, because its grammar isn't "context free friendly" (it's probably actually not context free). In particular, it is necessary to maintain a separate stack of open string literal interpolations in order to be able to close the correct one. This gives rise to out-of-parser computations like enterBraceSingleQuote() in the following rules:

SINGLE_LINE_STRING_SQ_BEGIN_MID
    :    '\'' STRING_CONTENT_SQ* '${' { enterBraceSingleQuote(); }
    ;

SINGLE_LINE_STRING_SQ_MID_MID
    :    { currentBraceLevel(BRACE_SINGLE) }?
         { exitBrace(); } '}' STRING_CONTENT_SQ* '${'
         { enterBraceSingleQuote(); }
    ;

SINGLE_LINE_STRING_SQ_MID_END
    :    { currentBraceLevel(BRACE_SINGLE) }?
         { exitBrace(); } '}' STRING_CONTENT_SQ* '\''
    ;

The problem is that we need to keep track of arbitrary nested terms of the form ' ... ${e1} ... ${e2} ...' and similarly for the other quotes/triple-quotes, where ei are expressions (possibly containing other string interpolations), such that each single quote is matched up with a single quote, triple-single-quote with triple-single-quote, and so on. We'd presumably need a complete copy of the entire expression grammar for each of the possible stack states in order to know whether to accept a given " at some point or not (or we could have built-in support for parser states, which is just the same thing as what we do with exitBrace() and its brethren, and that's not context free).

So the nice parser that you'd generate from any given EBNF would probably need to be fixed up in ways that may not be supported by the given parser generator.

Another issue is that yield and await are treated specially in the body of an async/async*/sync* function, and yet another is that f(A < B, C > (D)) is parsed as a function invocation passing the result of calling the generic function A with type arguments B, C and value argument D, not as an invocation of f passing two arguments that compare A and B respectively C and (D).

In summary, I don't think it's going to work as smoothly as one might hope, even if we decide on a specific EBNF specification and announce that it is official.

@eernstg

@eernstg wow... That's a lot of issues.

Okay, first of all, thanks for the detailed explanation of these problems of issue, Now i understand real scale of the situation.

When we talk about grammar schema like ebnf, i don't mean that it exactly must generate ready-to-go parser, no, it's not. I don't remember any runnable ast generator from ebnf schema only. So, create super-universal grammar specification is too utopian.

Also, when we talk about code like f(A < B, C > (D)) is valid (what is too funny for me), i want ask you: if i have variables A,B,C and D, does this code will be parsed as class constructor, or as bool expressions? If it can be parsed in two variants with different situations, there is a huuuuuge problem with language. But i hope that i am wrong. I understand that the parsing option depends on what arguments f takes as input, but still believe that this can be described as context free and resolved in higher layer ast builders.

At the same time, it seems to me that this problem is solved (in my not the most thoughtful one opinion) in several ways, and here is one of them:

Let's take a look on await keyword: of course, we expected, that find await expression in text is valid only ins async function. But i think that code like this one:

Future notRealyAsync() {
    await doSomethingAnother()
}

contains not a syntax error, but a logic one. I mean, this code won't work in one type of error, or another, it's an error, code won't run with any error. Bur when we talk about parsing code this means a lot. so we can define something like:

expression ::= bitwiseExpression | blahblahblah | ... | awaitExpression;
awaitExpression ::= 'await' expression ';';

Is this context-free? If we parse syntax things like this, it would be! (But it's for my opinion, i could be wrong in real life). If we can define grammar in this way, what can we left? Of course, we must also add some logic rules like if non-async function body contains any awaitExpression, then parser must throw logic error..

I'll be honest with you, i really appreciate your and my time, so here is the thing: i've give you this idea, and really want your feedback, if you like it (or someone another from dart team) i'll try to make proposal of this idea with proof-of-concept tiny ebnf schema, set of simple rules, and simple parser written on golang (cuz i using go, lol). Then maybe we can discuss more about this stuff 😃

Btw, i don't think, that syntax grammar must define formatted strings. Yes, it's part of language specification, but this can be implemented as another grammar or even regex, which is way more simpler, and added into logic specification.

about .. f(A < B, C > (D))

It is parsed as an invocation of f with one argument, and it makes no difference whether A, B, C, D are variables or types (or even undefined): The parser does not rely on name resolution in order to make a parsing decision. The point is that f(A < B, C > (D)) is ambiguous, and the language makes the choice to disambiguate it in favor of the generic function call.

If you want to parse it as an invocation of f with two boolean arguments then you can remove the parentheses from D, or any of several other things that will prevent the parse as a generic function invocation.

With respect to

Future notRealyAsync() {
    await doSomethingAnother()
}

the issue is that await had been a regular identifier for years when the <awaitExpression> and all things async were added to Dart. So it had to get a special treatment in order to ensure that code like this continued to work:

void notAsync() {
  var await = true;
  ...
}

.. i don't think, that syntax grammar must define formatted strings .. another grammar or even regex

It won't work with regexes; regular expressions do not even have the expressive power needed to keep track of simple parenthesized structures (like (())() as opposed to wrong ones like ((), and we need the parser in order to parse e in "stuff ${e} more stuff". Having another parser is arguably what we already do (it's just a really, really simple one) by having the stack of currently open string literal delimiters and braces.

Navigation Menu

Repository navigation

Provide Dart grammar in a machine-readable format #19298

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Provide Dart grammar in a machine-readable format #19298

Description

Activity

DartBot commented on Jun 8, 2014

kevmoo commented on Jun 8, 2014

DanTup commented on Jun 8, 2014

DanTup commented on Jun 14, 2014

gbracha commented on Jun 30, 2014

eernstg commented on Jan 26, 2018

DanTup commented on May 8, 2019

eernstg commented on May 8, 2019

quenbyako commented on Jan 4, 2021

eernstg commented on Jan 5, 2021

quenbyako commented on Jan 14, 2021

eernstg commented on Jan 15, 2021

quenbyako commented on Jan 15, 2021

eernstg commented on Jan 15, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions