Repository navigation
Support splitting transpilation into a worker in tscΒ #54461
Description
Activity
See also #54256
Reacted by Ryan Cavanaugh, Andrii Dieiev and Daniel RosenwasserReacted by Martin JohnsReacted by Daniel RosenwasserRelated: #54256
Reacted by Ryan Cavanaugh, Andrii Dieiev and Daniel RosenwasserReacted by Daniel Rosenwasserπ π | π |fatcerberus by 2 seconds!
Reacted by Martin Johns, Bruce Pascoe, Andrii Dieiev, Joshua Chen, Mateusz BurzyΕski and Ashley ClaymoreWorth noting that unlike parallelizing of parsing, splitting off the transpilation doesn't need the asynchronicity to penetrate nearly as deeply into the compiler. The transpilation worker can be kicked off and forgotten about until the very end, when the process waits for the result before logging diagnostics. Each resolve/parse/bind/check/emit step is still internally 100% synchronous, the asynchrononicity only gets introduced at the top-level orchestration.
Why is this dependent on
verbatimModuleSyntax?isolatedModulesis supposed to be sufficient to transpile single files without any whole-program info.As to why
verbatimModuleSyntaxis necessary:import { SomeInterface } from './foo'; export class Foo implements SomeInterface {};
Normal compilation:
export class Foo {};
Transpilation:
import { SomeInterface } from './foo'; // Invalid runtime code export class Foo {};
Edit: Above is due to a precise configuration of options I'm using for transpilation. Specifically,
preserveValueImportscombined with settinghasNoDefaultLib: trueon the source file object, which is what actually fully disables the type checker while still producing the correct output.More detailed example:
import { SomeInterface, SomeBaseClass } from './foo'; export class Bar extends SomeBaseClass implements SomeInterface {};
Expected result:
import { SomeBaseClass } from './foo'; export class Bar extends SomeBaseClass {};
The above works as-is with
transpileModule, but CPU profile shows a chunk of time spent in the type checker before emitting. Inspection of the TypeScript compiler source shows that the type checker can only be completely disabled by settinghasNoDefaultLib = trueon thets.SourceFileobject.
However, if we only set that flag and leave everything the same, we get:export class Bar extends SomeBaseClass {};
Note the missing import for
SomeBaseClass. It does remove the CPU time chunk for the type checker, however.To get that import back, we can add
preserveValueImports: trueon the compiler options, but then we get:import { SomeInterface, SomeBaseClass } from './foo'; export class Foo extends SomeBaseClass {};
This has an invalid runtime import of
SomeInterface.The final solution is to enable
verbatimModuleSyntax: true, at which point the compiler complains if we don't adjust the source to:import { type SomeInterface, SomeBaseClass } from './foo'; export class Bar extends SomeBaseClass implemenets SomeInterface {};
And this finally yields the expected result (while also not spending time in the type checker in the CPU profile):
import { SomeBaseClass } from './foo'; export class Bar extends SomeBaseClass {};
Edit again: TL;DR, reference counting of imports happens in the type checker, not the binder, so completely disabling the type checker disables reference counting of imports, so identifying used vs. unused imports goes with it.
Reacted by Rob PalmerFWIW
preserveValueImportsis de facto deprecated in favor ofverbatimModuleSyntax, see #51479The above works as-is with
transpileModule, but CPU profile shows a chunk of time spent in the type checker before emitting.Yeah, my understanding was that that function doesnβt actually disable type checking, it just suppresses the diagnostics.
FWIW
preserveValueImportsis de facto deprecated in favor ofverbatimModuleSyntax, see #51479I'm actually still on an older version and so using the
importsNotUsedAsValues: "error"field in tsconfig, so hadn't fully registered thatverbatimModuleSyntaxapplies the same effect aspreserveValueImports.My implementation in
@rushstack/heft-typescript-plugindoesn't actually use the publictranspileModuleAPI; instead it callscreateProgramand configures the same compiler host and compiler options that are internally used bytranspileModule, but with the full set of source files for the entire compilation. This avoids overhead associated with spinning up the program for each file that would be incurred if I had usedtranspileModule.
Suggestion
π Search Terms
isolatedModules, tsc, worker, parallel
β Viability Checklist
My suggestion meets these guidelines:
β Suggestion
For projects that have both
isolatedModules: trueandverbatimModuleSyntax: true, it is feasible to save end-to-end compile time by, after identifying changed source files, fork the transpilation part of the compilation into a worker (WebWorker on web, 'node:worker_threads' on NodeJs) so that it can be run in parallel with the type checker on the main thread (which getsdeclarationOnly: trueinjected into its config before emit).I've prototyped this in the RC version of Heft here: microsoft/rushstack#4120
However, it seems like a straightforward enough feature with enough of a performance benefit that it would be useful to have as part of core TypeScript.
π Motivating Example
With
isolatedModules: trueandverbatimModuleSyntax: true, the time calculations are thus:T_Transpile = Transform_js + Emit_jsT_Declaration = Check + Transform_dts + Emit_dtsT_Worker_Overhead = Parse + BindT_Startup = Resolve + Parse + BindT_Worker = T_Worker_Overhead + T_TranspileT_Original = T_Startup + T_Transpile + T_DeclarationT_WithWorker = T_Startup + max(T_Declaration, T_Worker)Testing on a modestly large local project (816 source files) has
T_Original = 19.7 sT_Declaration = 10.0 sT_Worker = 8.1 sT_Worker_Overhead = 1.9 sResulting in a net end-to-end savings of
6.2 sout of19.7 s=31%with no change to the output.π» Use Cases
Granted, custom build tools can do this, but a lot of developers prefer to use
tscdirectly, and by making this a core feature, it incentivizes improvements to the duplication that occurs in the implmentation (namely Parse + Bind).