11

The Intent

I want to use the TypeScript's Compiler API to experiment with operator overloading in TypeScript code. Specifically, I want to find all instances of x + y and turn them into op_add(x, y). However, I want the language services (eg. IntelliSense in VS Code) to be aware of the transformation and show the correct types.

For example in this code:

interface Vector2 { x: number, y: number }
declare function op_add(x: Vector2, y: Vector2): Vector2
declare let a: Vector2, b: Vector2

let c = a + b

I would expect that when I hover my mouse over c, it would show Vector2.

The Plan

In order to achieve this, I'll have to:

  1. Create a program that exposes the same API as typescript – in the same way that ttypescript does.
  2. Make that program modify the source code before passing it to typescript
  3. Make VS Code (or whatever editor) use my package instead of typescript

The Exectuion

I started by creating a short script called compile.ts that uses the Compiler API to parse a file called sample.ts into AST. Then it directly modifies the AST and changes Binary(x, PlusToken, y) to Call(op_add, x, y). Finally it prints the modified code to console and then tries to emit. This alone isn't enough for an IDE integration, but it is a good start.

compile.ts:

import * as ts from "typescript"
import { possibleChildProperties } from "./visit";

let program = ts.createProgram(['sample.ts'], { target: ts.ScriptTarget.ES5, module: ts.ModuleKind.CommonJS })
let inputFiles = program.getSourceFiles()
const printer = ts.createPrinter({ newLine: ts.NewLineKind.LineFeed })

let outputCode: string

for (let input of inputFiles) {
  if (input.fileName === 'sample.ts') {
    ts.visitNode(input, visitor) // modifies input's AST
    outputCode = printer.printNode(ts.EmitHint.Unspecified, input, input)
    break
  }
}

console.log(outputCode) // works
let emitResult = program.emit() // fails



function visitor(node: ts.Node): ts.Node {
  if (node.kind === ts.SyntaxKind.BinaryExpression) {
    let expr = node as ts.BinaryExpression

    if (expr.operatorToken.kind === ts.SyntaxKind.PlusToken) {
      return ts.createCall(ts.createIdentifier('op_add'), [], [expr.left, expr.right])
    }
  }

  return visitChildren(node, visitor)
}

function visitChildren(node: ts.Node, visitor: ts.Visitor) {
  for (const prop of possibleChildProperties) {
    if (node[prop] !== undefined) {
      if (Array.isArray(node[prop]))
        node[prop] = node[prop].map(visitor)
      else
        node[prop] = visitor(node[prop])
    }
  }

  return node
}

sample.ts:

let a = { a: 4 }
let b = { b: 3 }
let c = a + b

console.log output:

let a = { a: 4 };
let b = { b: 3 };
let c = op_add(a, b);

The Problem

While the code printer works fine and outputs the correct code, calling program.emit() results in an unspecified internal error. This probably means that I'm modifying the AST in an non-supported way.

/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:100920
                throw e;
                ^

Error: start < 0
    at createTextSpan (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:10559:19)
    at Object.createTextSpanFromBounds (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:10568:16)
    at getErrorSpanForNode (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:13914:19)
    at createDiagnosticForNodeInSourceFile (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:13808:20)
    at Object.createDiagnosticForNode (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:13799:16)
    at error (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:35703:22)
    at resolveNameHelper (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:36602:29)
    at resolveName (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:36274:20)
    at getResolvedSymbol (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:52602:21)
    at checkIdentifier (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:54434:26)

The Question

What is the correct way to modify a program's AST before running the type checker? I understand that AST should preferably be read-only, but the standard ts.visitEachChild can be only used after type-checking. And deep-cloning the nodes myself also doesn't seem like an viable option, as there isn't any way to create a Program from code-generated AST.

Updates

EDIT 1: As @jdaz noticed, my sample.ts was missing a declaration for op_add, which could be causing problems. I added this line to the top of the file:

declare function op_add(x: {}, y: {}): string

Now there's a different error – the generation of file diagnostic fails:

/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:100920
                throw e;
                ^

Error: Debug Failure. Expected -2 >= 0
    at Object.createFileDiagnostic (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:17868:18)
    at grammarErrorAtPos (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:69444:36)
    at checkGrammarForAtLeastOneTypeArgument (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:68771:24)
    at checkGrammarTypeArguments (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:68777:17)
    at checkCallExpression (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:59255:18)
    at checkExpressionWorker (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:61687:28)
    at checkExpression (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:61597:38)
    at checkExpressionCached (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:61275:38)
    at checkVariableLikeDeclaration (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:63983:69)
    at checkVariableDeclaration (/home/m93a/Dokumenty/tsc-experiments/node_modules/typescript/lib/typescript.js:64051:20)
Improve this question
5
  • 1
    Could the error just be coming from the fact that you haven't defined op_add in either of your files? So in typescript.js when it calls node.getStart it can't find anything and returns -1. Commented Aug 2, 2020 at 18:55
  • @jdaz Great observation! But sadly, one error disappeared and another one appeared. I edited the question. Commented Aug 2, 2020 at 19:09
  • Hmm do you need to actually define the entire function instead of only declaring it? Commented Aug 2, 2020 at 19:16
  • Tried that too, it does exactly the same as only delcaring. Commented Aug 2, 2020 at 21:18
  • I am trying to do literally the exact same thing, did you make any progress on this including the IDE integration parts? Commented Jun 12, 2022 at 16:03

3 Answers 3

Reset to default
3
+500

You got close with your code. The first issue you seem to be having is source code file checks that occur, essentially the Debug Failure. Expected -2 >= 0 error is saying that when trying to match the AST to source code it failed.

The second issue is that you need to modify the existing AST tree whereas visitNode is generating a new AST tree. This also must be done as early as possible (before emit is called AFAIK) otherwise the TypeChecker might use the original AST instead of your updated AST.

Below is an example of your visitor function that should solve both problems. Note that this is really hacky and fragile, expect it to break often.

OLD:

function visitor(node: ts.Node): ts.Node {
  if (node.kind === ts.SyntaxKind.BinaryExpression) {
    let expr = node as ts.BinaryExpression

    if (expr.operatorToken.kind === ts.SyntaxKind.PlusToken) {
      return ts.createCall(ts.createIdentifier('op_add'), [], [expr.left, expr.right])
    }
  }

  return visitChildren(node, visitor)
}

NEW:

function visitor(node: ts.Node): ts.Node {
  if (node.kind === ts.SyntaxKind.BinaryExpression) {
    let expr = node as ts.BinaryExpression;

    if (expr.operatorToken.kind === ts.SyntaxKind.PlusToken) {
      const newIdentifierNode = ts.createIdentifier('op_add');
      const newCallNode = ts.createCall(newIdentifierNode, [], [expr.left, expr.right]);
      newCallNode.flags = node.flags;
      newCallNode.pos = node.pos;
      newCallNode.end = node.end;
      newCallNode.parent = node.parent;
      newCallNode.typeArguments = undefined;

      Object.getOwnPropertyNames(node).forEach((prop) => {
          delete node[prop];
      });
      Object.getOwnPropertyNames(newCallNode).forEach((prop) => {
          node[prop] = newCallNode[prop];
      });
      return node;
    }
  }

  return visitChildren(node, visitor);
}
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thanks, this is amazing! Actually the newCallNode.typeArguments = undefined; part was very important too. I had no idea that you have to pass undefined for non-generic functions.
1

This might be a hacky way but since you already have the modified source code, why not build a new AST from that? For example:

const newSource = ts.createSourceFile(
        'newSource.ts',
        outputCode,
        ts.ScriptTarget.ES5,
        true
)
const newProgram = ts.createProgram(['newSource.ts'], { target: ts.ScriptTarget.ES5, module: ts.ModuleKind.CommonJS })
let emitResult = newProgram.emit()

This avoids changes to the original AST and runs without errors.

Improve this answer

1 Comment

Hey, thanks for a reply! This is nice, but hooking up the language services would be a lot harder if I had to translate the file path/name to a different one. Modifying the AST directly as Stefan Charsley suggested will make my work easy :)
1

Continuing from your selected answer and the problem you apparently did not overcome:

After replacing an AST node, or creating a new one, you can synchronize the new virtual text and each node with

    ts.setSourceMapRange(newnode, ts.getSourceMapRange(node));
    ts.setCommentRange(newnode, ts.getCommentRange(node));

That would be called from within then visitor.

However, any errors that appeared would (probably) then not be in sync with the original text - the same problem as in the answer you didn't accept suggesting creating a new non-virtual real intermediate text file.

Here is one possible workaround. I have a uses a partially similar approach in a partially related problem, and I have adapted it to what I think should work in your problem.

Write a transformer tf-overload that performs the specified lexical transformation you specified.

  • Pass 1
    • Compile: AST of orig source and report errors. Filter out errors about incompatible arguments to '+'. (Do not think it is a good approach, ... but). Other errors display.
    • Do not emit.
  • Pass 2 (optional)
    • transform orignal AST to new AST with tf-overload.
    • Compile: new AST, if there are any errors, report, but the positions could be off.
  • Pass 3
    • Call emit with the transformer tf-overload in the before slot.
    • The source map transformation is performed for you when it is called from within emit. So your debugger will display the correct position relative to the original source file. Emit won't do symbolic checking, but any warnings/errors from the emit phase will show correct positions in the editor.

You won't get UI special info you hoped for hovering over 'c'. For that you would need to integrate with ttypescript or ts-patch or some equivalent (or roll your own), and also make sure your editor is pointing to the correctly modified typescript libraries. Furthermore, for it to be useful for others they would also have the correct setup. No small feat.

However, that depends on the details and practical applications of your plan. Calling vector2 for every addition could result in a lot of overhead. You might want to use symbolic info after compile typeChecker to make sure you only call vector2 on arrays/tuples of numbers. Then you wouldn't need to filter errors, but would be creating you own. That's a lot more work. You can check the array lengths at runtime.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.