abstract syntax tree – hangaroundtheweb

In this article, we are going to take a look at a tool called ‘AST Finder’ which will significantly improve the developer experience for writing codemods.

This is a continuation post for AST Builder which is a tool to generate AST nodes automatically from source code.

If you want to know more about codemods, their building blocks and how they work, please check out this detailed post about codemods.

What?

AST Finder is actually a playground for finding AST nodes using source code. Because ASTs play a big role in writing codemods, this tool will assist the developers to a great extent in writing codemods. Because codemods actually do AST-to-AST transformation on your source code, and that’s one of the primary reasons, why codemods are more resilient in making effective code transformations.

Why?

Never memorize something that you can look up.
― Albert Einstein

The first task in writing codemods is to find the appropriate nodes in the AST and work on them. And we definitely need a tool which makes it easy to find nodes. The problem is, there is no proper documentation on finding AST nodes using the jscodeshift api. All you have to do is learn from other codemods out there and sift through the code and find out how you can query nodes.

Now let’s say you want to replace a CallExpression, foo.bar() with a new one like foo(). The AST representation for the above two expressions will be like:

// AST Node for foo.bar()
"expression": {
  "type": "CallExpression",
  "callee": {
    "type": "MemberExpression",
    "computed": false,
    "object": {
      "type": "Identifier",
      "name": "foo"
    },
    "property": {
      "type": "Identifier",
      "name": "bar"
    }
  },
  "arguments": []
}

// AST Node for foo.bar()

"expression": {

"type": "CallExpression",

"callee": {

"type": "MemberExpression",

"computed": false,

"object": {

"type": "Identifier",

"name": "foo"

"property": {

"type": "Identifier",

"name": "bar"

}

"arguments": []

}

I have omitted a lot of information in the above code for clarity and readability purposes. It only contains the relevant information for the actual CallExpression AST node. If you want to explore the full tree structure of the AST, you can check it in ast-explorer.

// AST Node for foo()
"expression": {
  "type": "CallExpression",
  "callee": {
    "type": "Identifier",
    "name": "foo"
  },
  "arguments": []
}

// AST Node for foo()

"expression": {

"type": "CallExpression",

"callee": {

"type": "Identifier",

"name": "foo"

"arguments": []

}

As you can see from the above two AST nodes, the only difference between the two is the callee object which is a simple Identifier in foo() and a MemberExpression in foo.bar(). Usually with codemods, we will be replacing the original expression with the new one. Hence here, we will be replacing the original CallExpression with a new one like this.

module.exports = function transformer(file, api) {
  const j = api.jscodeshift;
  const root = j(file.source);
  const body = root.get().value.program.body;

  root.find(j.CallExpression, {
    callee: {
      object: {
        name: "foo"
      },
      property: {
        name: "bar"
      }
    }
  }).replaceWith(path => {
    return j.callExpression(j.identifier("foo"), []);
  });

  return root.toSource();
};

module.exports = function transformer(file, api) {

const j = api.jscodeshift;

const root = j(file.source);

const body = root.get().value.program.body;

root.find(j.CallExpression, {

callee: {

object: {

property: {

}

}).replaceWith(path => {

return j.callExpression(j.identifier("foo"), []);

});

return root.toSource();

};

In order to replace the old CallExpression with a new one, first we need to find the existing CallExpression. From the above codemod you can see we are querying the AST using jscodeshift api like this:

root.find(j.CallExpression, {
  callee: {
    object: {
      name: "foo"
    },
    property: {
      name: "bar"
    }
  }
})

root.find(j.CallExpression, {

callee: {

object: {

property: {

}

})

If you try to find the above CallExpression within ast-explorer transform editor, you will be having a tough time if you are doing it for the first-time. Because you are not very familiar with the find api in the first place, and you don’t know the right order and type of parameters you need to supply to correctly find the AST node. And don’t forget the typos and punctuation errors you make while typing the code.

There are also some subtle nuances with the jscodeshift api which beginners won’t know for example, the api j.callExpression is a constructor for building CallExpression nodes, whereas j.CallExpression is an instance of the type CallExpression which is basically used to find nodes of the type CallExpression.

This is where AST Finder comes into the picture, it is acting as a reference guide for find apis to easily query your AST nodes. Just input the code in the input editor (see the image above to identify the input editor which is always at the top left pane in the layout) you will get the find api automatically generated for you without any mistakes. So if you input foo.bar() into the AST Finder, it will give you something like:

root.find(j.CallExpression, {
  callee: {
    object: {
      name: "foo"
    },
    property: {
      name: "bar"
    }
  }
})

root.find(j.CallExpression, {

callee: {

object: {

property: {

}

})

Now, you can simply copy the query from AST Finder and use it in your codemods. How cool is that?

How?

AST Finder uses ‘ast-node-finder‘ an npm package underneath, which provides the apis for finding AST nodes through jscodeshift. All the api’s takes an object as a parameter and returns the find api in string format, which you can use with jscodeshift to query new nodes. The object which is passed to the api as a parameter is actually a node in the AST generated by the respective parser. When you feed the node to the ast-node-finder api, you get back the jscodeshift api to find that node.

This allows developers to easily and effectively find AST nodes from source code. All you have to do is just enter or copy paste the source code into the input editor and you can see the jscodeshift api automatically generated for you in the output editor.

import { findQuery } from 'ast-node-finder';
import { parse } from 'recast';

const source = `foo.bar()`;

const ast = parse(source);

// Pass the node from ast and get the find api
console.log(findQuery(ast.program.body[0].expression));

import { findQuery } from 'ast-node-finder';

import { parse } from 'recast';

const source = `foo.bar()`;

const ast = parse(source);

// Pass the node from ast and get the find api

console.log(findQuery(ast.program.body[0].expression));

The above snippet will generate some thing like this:

root.find(j.CallExpression, {
  callee: {
    object: {
      name: "foo"
    },
    property: {
      name: "bar"
    }
  }
})

root.find(j.CallExpression, {

callee: {

object: {

property: {

}

})

You can also use the AST Finder to visualize your AST on the top right pane without all the noise and clutter of meta information. We deliberately filter out the loc nodes from the AST and also the tokens, since we feel it is not of much use for working with codemods. To dig deep into the finder you can take a look at the source code here, it is built in Ember.js.

And if you are a dark-theme fan, AST Finder also allows to switch your editor themes to dark mode. Please use the Report issues link at the footer if you want to report any issues or feedback, you can tell us how we can improve the tool and what additional languages we need to support.

Stay tuned or subscribe to the newsletter at the bottom to know more about the exciting tools we are building around ASTs and Codemods.

References

In this article, we are going to take a look at a tool called ‘AST Builder’ which will significantly improve the developer experience for writing codemods.

Codemod is a tool/library to assist you with large-scale codebase refactors that can be partially automated but still require human oversight and occasional intervention. Codemod was developed at Facebook and released as open source.

If you want to know more about codemods, their building blocks and how they work, please check out this detailed post about codemods.

What?

AST Builder is actually a playground for building AST nodes using source code. Because ASTs play a big role in writing codemods, this tool will assist the developers to a great extent in writing codemods. Because codemods actually do AST-to-AST transformation on your source code, and that’s one of the primary reasons, why codemods are more resilient in making effective code transformations.

It currently supports Javascript (ES5, ES6 and some ES7 constructs) , JSX and Glimmer.js handlebars syntax. Please take a look at these issues, Cover core api, Cover ES6 api for coverage information. And I am planning to include more language syntax and semantics.

Why?

We already have an well-established and battle-tested tool called “ast-explorer” for visualizing abstract syntax trees or ASTs. Why we need a new tool then? Because AST explorer is only for exploring your ASTs, it doesn’t tell how to create AST nodes. Even though, ast-explorer offers intellisense in their editor for the jscodeshift apis, it doesn’t work for all the parsers like you can only be using the autocomplete api for recast parser. If you choose any other parser other than recast, you won’t get the intellisense in the codemod editor.

And most of the time, you will be creating nodes for transforming code using codemods. And we definitely need a tool which makes it easy to create nodes. The problem is, there is no proper documentation on creating AST nodes using the jscodeshift api. All you have to do is learn from other codemods out there and sift through the code and find out how you can create new nodes.

For that you need to understand the parser internals, the Node schema and the types of node builders available for the language you are working.

If you are still not convinced why this tool will make a difference in developer experience for building codemods, listen to what others say here.

Say for example, for Javascript, you need to know the ESTree spec or the node builder definition in ast-types. This module provides an efficient, modular, Esprima-compatible implementation of the abstract syntax tree type hierarchy pioneered by the Mozilla Parser API.

Now let’s say you want to replace a CallExpression, foo() with a new one like foo.bar(). The AST representation for the above two expressions will be like:

// AST Node for foo()
{
  "type": "ExpressionStatement",
  "expression": {
    "type": "CallExpression",
    "callee": {
      "type": "Identifier",
      "name": "foo",
    }
  }
}

// AST Node for foo()

{

"type": "ExpressionStatement",

"expression": {

"type": "CallExpression",

"callee": {

"type": "Identifier",

"name": "foo",

}

// AST Node for foo.bar()
{
      "type": "ExpressionStatement",
      "expression": {
        "type": "CallExpression",
        "callee": {
          "type": "MemberExpression",
          "computed": false,
          "object": {
            "type": "Identifier",
            "name": "foo",
          }
        }
      }
  }

// AST Node for foo.bar()

{

"type": "ExpressionStatement",

"expression": {

"type": "CallExpression",

"callee": {

"type": "MemberExpression",

"computed": false,

"object": {

"type": "Identifier",

"name": "foo",

}

module.exports = function transformer(file, api) {
  const j = api.jscodeshift;
  const root = j(file.source);
  const body = root.get().value.program.body;

  root.find(j.CallExpression, {
    callee: {
      name: "foo"
    }
  }).replaceWith(path => {
    return j.callExpression(
      j.memberExpression(
        j.identifier("foo"), j.identifier("bar"), false), []);
  });

  return root.toSource();
};

module.exports = function transformer(file, api) {

const j = api.jscodeshift;

const root = j(file.source);

const body = root.get().value.program.body;

root.find(j.CallExpression, {

callee: {

}

}).replaceWith(path => {

return j.callExpression(

j.memberExpression(

j.identifier("foo"), j.identifier("bar"), false), []);

});

return root.toSource();

};

In order to replace the old CallExpression with a new one, we need to build the new one. From the above codemod you can see we are building the new one using jscodeshift api like this:

return j.callExpression(
  j.memberExpression(
    j.identifier("foo"), j.identifier("bar"), false), []);

return j.callExpression(

j.memberExpression(

j.identifier("foo"), j.identifier("bar"), false), []);

If you try to build the above CallExpression within the ast-explorer transform editor, you will be having a tough time if you are doing it for the first-time. Because you are not very familiar with the the builder api in the first place, and you don’t know the correct order and type of parameters you need to supply to correctly build the AST node. And don’t forget the typos and punctuation errors you make while typing the code.

There are also some subtle nuances with the jscodeshift api which beginners won’t know for example, the api j.callExpression is a constructor for building CallExpression nodes, whereas j.CallExpression is an instance of the type CallExpression which is basically used to find nodes of the type CallExpression.

This is where AST Builder comes into the picture, it is acting as a reference guide for builder apis to easily build your AST nodes. Just input the expected code in the input editor (see the image above to identify the input editor which is always at the top left pane in the layout) you will get the builder api automatically generated for you without any mistakes. So if you input foo.bar() into the AST Builder it will give you something like:

j.expressionStatement(
  j.callExpression(
    j.memberExpression(
      j.identifier("foo"), j.identifier("bar"), false), [])
);

j.expressionStatement(

j.callExpression(

j.memberExpression(

j.identifier("foo"), j.identifier("bar"), false), [])

);

You can safely omit the ExpressionStatement wrapper if you are just replacing the nodes.

Now, you can simply copy the builder api from AST Builder and use it in your codemods. How easy is that?

How?

AST Builder uses ‘ast-node-builder‘ an npm package underneath, which provides the apis for building AST nodes through jscodeshift. All the api’s takes an object as a parameter and returns the builder api in string format, which you can use with jscodeshift to create new nodes. The object which is passed to the api as a parameter is actually a node in the AST generated by the respective parser. When you feed the node to the ast-node-builder api, you get back the jscodeshift api to build that node.

This allows developers to easily and effectively create AST nodes from source code, instead of tinkering with the autocomplete api in ast-explorer. All you have to do is just enter or copy paste the source code into the input editor and you can see the jscodeshift api automatically generated for you in the output editor.

const { buildAST } = require('ast-node-builder');
const { parse }  = require('recast');
const code = `foo.bar()`;

let ast = parse(code);

let pseudoAst =  buildAST(ast);
console.log(pseudoAst);

const { buildAST } = require('ast-node-builder');

const { parse } = require('recast');

const code = `foo.bar()`;

let ast = parse(code);

let pseudoAst = buildAST(ast);

console.log(pseudoAst);

The above snippet will generate some thing like this:

j.expressionStatement(
  j.callExpression(
    j.memberExpression(
      j.identifier("foo"), j.identifier("bar"), false), [])
);

j.expressionStatement(

j.callExpression(

j.memberExpression(

j.identifier("foo"), j.identifier("bar"), false), [])

);

You can also use the AST Builder to visualize your AST on the top right pane without all the noise and clutter of meta information. We deliberately filter out the loc nodes from the AST and also the tokens, since we feel it is not of much use for working with codemods. To dig deep into the builder you can take a look at the source code here, it is built in Ember.js.

And if you are a dark-theme fan, AST Builder also allows to switch your editor themes to dark mode. You can play around with the list of languages and list of parsers. Please use the Report issues link at the footer if you want to report any issues or feedback, you can tell us how we can improve the tool and what additional languages we need to support.

Stay tuned or subscribe to the newsletter at the bottom to know more about the exciting tools we are building around ASTs and Codemods.