Repository navigation
Add API to (de)serialize parsed Node objects #26871
Description
Activity
Canonical S-expressions could be a candidate format.
- changed the title
[-]Add API to (de)derialize parsed Node objects[/-][+]Add API to (de)serialize parsed Node objects[/+]on Sep 4, 2018 Not sure if I misunderstand this issue, but is the watch mode not an option? I'm using that in my project https://github.com/AlCalzone/virtual-tsc (with a barebones language server host and an in-memory FS) where compilation times drop from a few seconds on the first compilation to a few milliseconds when only single files get changed.
Not sure if I misunderstand this issue, but is the watch mode not an option?
Not exactly. Watch mode by
tscis already fast (1-2 seconds delay is ok in watch mode for me) - this feature request is not about it.Also it can be added to tsc command as arg --use-cache (or something like that) to do it by default (but I believe in this case we also need to have new option like cacheDir or --force).
For example, you work on your feature branch and now you need to switch to master/dev branch. You stop
tsc --watch(to avoid conflicts with your vcs), switch to master/dev and then runtsc --watchagain and now it takes some time (for me it is up to 1 minute).Or, for example, even share caches between co-workers/builds in CI/etc to do not parse unchanged since some time files.
Is this an XY problem? i.e. is the request here to make compilation faster in general, or do you specifically need a serialized parse tree for some other reason?
I will say that 10-15s to parse 226k lines seems quite high and may indicate a real bug somewhere. Our self-host compilation of
compilerparses 83k lines in 0.71s; your parse time "should" only be 3-4s.Ryan Cavanaugh (@RyanCavanaugh)
Is this an XY problem? i.e. is the request here to make compilation faster in general
The thing which inspired me to create this issue is the compilation performance.
or do you specifically need a serialized parse tree for some other reason?
It would be nice to have such API, but if compilation/parse time will be fast, I hope that nobody needs to store parsed tree to file (because parse time will be similar to reading and restoring that tree from other file).
your parse time "should" only be 3-4s.
Wow, it looks good. But anyway I'm not sure will it be faster than restoring ast tree from file, but we need to check anyway.
10-15s to parse 226k lines seems quite high and may indicate a real bug somewhere
Hm. I'm open to help indicate that problem. What I can provide you to help find the bug if it is?
(I have no idea why --noEmit does not reduce "Total time" by "Emit time" - it is strange)
Also it would be nice to understand what's happened here and find the cause. Please let me know if something can help to understand the problem.
(I have no idea why --noEmit does not reduce "Total time" by "Emit time" - it is strange)
Are you using
declaration: true? In that case declaration emit will still happen under the hood to collect declaration diagnostics and therefore still needs quite some time.Are you using declaration: true?
No, at that moment I specially disable this flag to avoid extra checking/emitting. With enabled
declaration: trueflag total time is about 45-60s (but I haven't diagnostics for this runs - if you need this I can provide they tomorrow).Klaus Meinhardt (@ajafff) it seems that it was some lag on my side - I just run compilations again and
noEmitflag makes a sense on total time now. But parse/total time is still slow.noEmit: true:
$ ./node_modules/.bin/tsc --diagnostics --skipLibCheck --noEmit Files: 1767 Lines: 226078 Nodes: 1036945 Identifiers: 315470 Symbols: 380158 Types: 137886 Memory used: 495875K I/O read: 0.70s I/O write: 0.00s Parse time: 12.41s Bind time: 2.14s Check time: 13.04s Emit time: 0.00s Total time: 27.59s$ ./node_modules/.bin/tsc --diagnostics --skipLibCheck --noEmit --declaration true Files: 1767 Lines: 226078 Nodes: 1036945 Identifiers: 315470 Symbols: 380158 Types: 137886 Memory used: 491434K I/O read: 0.67s I/O write: 0.00s Parse time: 11.59s Bind time: 2.19s Check time: 13.24s Emit time: 0.00s Total time: 27.03snoEmit: false:
$ ./node_modules/.bin/tsc --diagnostics --skipLibCheck Files: 1767 Lines: 226078 Nodes: 1036945 Identifiers: 315470 Symbols: 380158 Types: 137886 Memory used: 498003K I/O read: 0.62s I/O write: 4.53s Parse time: 12.21s Bind time: 2.10s Check time: 12.84s Emit time: 14.89s Total time: 42.03s$ ./node_modules/.bin/tsc --diagnostics --skipLibCheck --declaration true Files: 1767 Lines: 226078 Nodes: 1036945 Identifiers: 315470 Symbols: 380158 Types: 137886 Memory used: 543786K I/O read: 0.65s I/O write: 8.64s Parse time: 11.57s Bind time: 2.05s Check time: 12.84s Emit time: 26.95s Total time: 53.40sOkay, I just ran with
--extendedDiagnosticsand now we haveParse timealong withProgram time:Parse time: 2.25s Program time: 10.83sIt seems that parse time is ok now.
Search Terms
Serialization, deserialization, dumping, dump source file, compiler cache
Related issues
#25658, TypeStrong/ts-loader#825
Suggestion
I'd like to suggest to add an API to be able to (de-)serialize any parsed Node or make parsed AST be easy (de-)serializable by, for example,
JSON.stringify().Use Cases
It can be used to make cache of parsed source files, which can be used between compilations without parsing unchanged source files again.
For example, for our project at the moment we have the following times for compilation:
(I have no idea why
--noEmitdoes not reduce "Total time" by "Emit time" - it is strange)I hope, if we'll have such API or even cache by
tsc, it can reduce compilation time, especiallyParse time.Also, in this case, the TypeScript package can be published with already parsed lib files, what can reduce compilation time even without enabling
--skipLibCheckoption.Examples
For example, we can override
getSourceFilemethod inCompilerHostand provide files from cache if file is not changed since last compilation. It can be done via provided methods likets.serializeNode(node: ts.Node): string | ByteArray/ts.deserializeNode(nodeData: string | ByteArray): ts.Node.Also it can be added to
tsccommand as arg--use-cache(or something like that) to do it by default (but I believe in this case we also need to have new option likecacheDiror--force).The similar cache can be found in bazel rules_typescript, but it is in-memory cache.
In our internal tool we have very similar in-memory cache too (it allows speedup multiple compilations the same files).
Checklist
My suggestion meets these guidelines: