Categories
Articles Javascript

JARVIS – Write me a Codemod

In this article, we are going to take a look at a tool called JARVIS which will profoundly transform the way how you write codemods.

This is a continuation post for AST Finder which is a tool to generate an api to find AST nodes automatically from source code.

If you want to know more about codemods, their building blocks and how they work, please check out this detailed post about codemods.

What?

JARVIS is actually a playground for writing codemods, where in you can specify the input and output code, choose the appropriate transformation you want to apply and the codemod is automatically generated for you.

JARVIS currently supports Javascript and Glimmer.js templates. For Javascript, you can use parsers like recast and babel. For Glimmer.js templates which is a more advanced form of Handlebars templates used by Ember.js framework, it is using the ember-template-recast parser to parse and transform the code.

It gives a playground to write code transformations just like AST Explorer but with a more fine-grained control and flexibility to write the transformations because it automatically figures out a lot of things for you like, how you can build new nodes, how you can find nodes from the AST, how to replace, remove or insert nodes into the tree, which visitor function you need to use for writing transformations.

Why?

If the goal is to enhance software development throughput, you can either:

• get faster people to do the work,
• get more people to do the work, or
• automate the work.

― Chad Fowler in The Pragmatic Programmer

We already have a very good tool called “ast-explorer” for visualizing abstract syntax trees or ASTs and writing codemods. Why we need a new tool then? Because AST explorer is only for exploring your ASTs, it doesn’t tell how to find or build AST nodes. You have to figure out a lot of things before you can start writing a codemod.

I spent a lot of time figuring out the jscodeshift apis, by looking at their documentation, learning from existing codemods, searching in google, and so on. The learning curve is kind of steep. And my colleagues have to tread down the same path. Whenever we tell any new developer in our team to learn about codemods, they face the same challenges we had in learning about them. Hence I wanted to make their lives easier and approach, learning about codemods a painless task. And that’s why JARVIS is created.

The first task in writing codemods is to find the appropriate nodes in the AST and work on them. And we definitely need a tool which makes it easy to find nodes. The problem is, there is no proper documentation on finding AST nodes using the jscodeshift api. All you have to do is learn from other awesome-codemods out there and sift through the code and find out how you can query nodes.

And the second thing, when you are replacing nodes, you need to know how you can build new nodes. While writing codemods, most of the time we will be creating new nodes. JARVIS can take care of these things for you so that you can focus on building or tweaking the transformation rather than wasting most of your precious time in finding the reference or documentation for building and finding AST nodes.

Once you are done with the transformation, you can simply copy the codemod from JARVIS, put it in a file, let’s say for example transform.js and run it through with jscodeshift.

How?

JARVIS uses two npm packages underneath. One for building AST nodes from code and the other one for finding AST nodes from code. It is pretty much a combination of both AST-Builder and AST-Finder working together to automatically generate code transforms.

First you need to specify the input code in the top-left panel (please see the annotated screenshot above for reference), then you choose the appropriate AST Node operation below. You can choose to remove the node, replace , insert the new node before or after the existing node.

Second, if you are replacing code you can enter the destination code how it is going to change in the bottom-left panel. Then JARVIS will automatically generate a boilerplate code for your transformation in the top-right panel. You can tweak the transformation according to your requirements and verify the transformation is working properly or not by looking at the output in the bottom-right panel.

Let’s consider a simple example in JS. Let’s say you want to replace all your foo() calls with foo.bar() in your code. The first task here is you need to identify the type of AST node you want to manipulate, in our case replace. This is an ExpressionStatement with an expression of type CallExpression. Whereas, the expected output node type is a MemberExpression. Hence you need to understand the syntax and semantics of finding a CallExpression and create a MemberExpression AST node, which will be a very difficult endeavor for someone who has just starting with codemods.

What JARVIS does here is, it figures out the type of the input node and the output node, the node manipulation operation you want to do with, and then automatically generates a codemod boilerplate for the same.

The codemod for the above transformation will look something like this:

To know more about how JARVIS works, you can take a look at the source code here, it is built in Ember.js.

And if you are a dark-theme fan, JARVIS also allows to switch your editor themes to dark mode. Please use the Report issues link at the footer if you want to report any issues or feedback, you can tell us how we can improve the tool and what additional languages / parsers we need to support.

Stay tuned or subscribe to the newsletter at the bottom to know more about the exciting tools we are building around ASTs and Codemods.

References

Categories
Articles Javascript

AST Finder – Finding AST nodes from code

In this article, we are going to take a look at a tool called ‘AST Finder’ which will significantly improve the developer experience for writing codemods.

This is a continuation post for AST Builder which is a tool to generate AST nodes automatically from source code.

If you want to know more about codemods, their building blocks and how they work, please check out this detailed post about codemods.

What?

AST Finder is actually a playground for finding AST nodes using source code. Because ASTs play a big role in writing codemods, this tool will assist the developers to a great extent in writing codemods. Because codemods actually do AST-to-AST transformation on your source code, and that’s one of the primary reasons, why codemods are more resilient in making effective code transformations.

Why?

Never memorize something that you can look up.

― Albert Einstein

We already have an well-established and battle-tested tool called “ast-explorer” for visualizing abstract syntax trees or ASTs. Why we need a new tool then? Because AST explorer is only for exploring your ASTs, it doesn’t tell how to find AST nodes.

The first task in writing codemods is to find the appropriate nodes in the AST and work on them. And we definitely need a tool which makes it easy to find nodes. The problem is, there is no proper documentation on finding AST nodes using the jscodeshift api. All you have to do is learn from other codemods out there and sift through the code and find out how you can query nodes.

Now let’s say you want to replace a CallExpressionfoo.bar() with a new one like foo(). The AST representation for the above two expressions will be like:

I have omitted a lot of information in the above code for clarity and readability purposes. It only contains the relevant information for the actual CallExpression AST node. If you want to explore the full tree structure of the AST, you can check it in ast-explorer.

As you can see from the above two AST nodes, the only difference between the two is the callee object which is a simple Identifier in foo() and a MemberExpression in foo.bar(). Usually with codemods, we will be replacing the original expression with the new one. Hence here, we will be replacing the original CallExpression with a new one like this.

In order to replace the old CallExpression with a new one, first we need to find the existing CallExpression. From the above codemod you can see we are querying the AST using jscodeshift api like this:

If you try to find the above CallExpression within ast-explorer transform editor, you will be having a tough time if you are doing it for the first-time. Because you are not very familiar with the find api in the first place, and you don’t know the right order and type of parameters you need to supply to correctly find the AST node. And don’t forget the typos and punctuation errors you make while typing the code.

There are also some subtle nuances with the jscodeshift api which beginners won’t know for example, the api j.callExpression is a constructor for building CallExpression nodes, whereas j.CallExpression is an instance of the type CallExpression which is basically used to find nodes of the type CallExpression.

This is where AST Finder comes into the picture, it is acting as a reference guide for find apis to easily query your AST nodes. Just input the code in the input editor (see the image above to identify the input editor which is always at the top left pane in the layout) you will get the find api automatically generated for you without any mistakes. So if you input foo.bar() into the AST Finder, it will give you something like:

Now, you can simply copy the query from AST Finder and use it in your codemods. How cool is that?

How?

AST Finder uses ‘ast-node-finder‘ an npm package underneath, which provides the apis for finding AST nodes through jscodeshift. All the api’s takes an object as a parameter and returns the find api in string format, which you can use with jscodeshift to query new nodes. The object which is passed to the api as a parameter is actually a node in the AST generated by the respective parser. When you feed the node to the ast-node-finder api, you get back the jscodeshift api to find that node.

This allows developers to easily and effectively find AST nodes from source code. All you have to do is just enter or copy paste the source code into the input editor and you can see the jscodeshift api automatically generated for you in the output editor.

The above snippet will generate some thing like this:

You can also use the AST Finder to visualize your AST on the top right pane without all the noise and clutter of meta information. We deliberately filter out the loc nodes from the AST and also the tokens, since we feel it is not of much use for working with codemods. To dig deep into the finder you can take a look at the source code here, it is built in Ember.js.

And if you are a dark-theme fan, AST Finder also allows to switch your editor themes to dark mode. Please use the Report issues link at the footer if you want to report any issues or feedback, you can tell us how we can improve the tool and what additional languages we need to support.

Stay tuned or subscribe to the newsletter at the bottom to know more about the exciting tools we are building around ASTs and Codemods.

References

Categories
Articles Javascript

AST Builder – Building AST nodes from code

In this article, we are going to take a look at a tool called ‘AST Builder’ which will significantly improve the developer experience for writing codemods.

Codemod is a tool/library to assist you with large-scale codebase refactors that can be partially automated but still require human oversight and occasional intervention. Codemod was developed at Facebook and released as open source.

If you want to know more about codemods, their building blocks and how they work, please check out this detailed post about codemods.

What?

AST Builder is actually a playground for building AST nodes using source code. Because ASTs play a big role in writing codemods, this tool will assist the developers to a great extent in writing codemods. Because codemods actually do AST-to-AST transformation on your source code, and that’s one of the primary reasons, why codemods are more resilient in making effective code transformations.

It currently supports Javascript (ES5, ES6 and some ES7 constructs) , JSX and Glimmer.js handlebars syntax. Please take a look at these issues, Cover core apiCover ES6 api for coverage information. And I am planning to include more language syntax and semantics.

Why?

We already have an well-established and battle-tested tool called “ast-explorer” for visualizing abstract syntax trees or ASTs. Why we need a new tool then? Because AST explorer is only for exploring your ASTs, it doesn’t tell how to create AST nodes. Even though, ast-explorer offers intellisense in their editor for the jscodeshift apis, it doesn’t work for all the parsers like you can only be using the autocomplete api for recast parser. If you choose any other parser other than recast, you won’t get the intellisense in the codemod editor.

And most of the time, you will be creating nodes for transforming code using codemods. And we definitely need a tool which makes it easy to create nodes. The problem is, there is no proper documentation on creating AST nodes using the jscodeshift api. All you have to do is learn from other codemods out there and sift through the code and find out how you can create new nodes.

For that you need to understand the parser internals, the Node schema and the types of node builders available for the language you are working.

If you are still not convinced why this tool will make a difference in developer experience for building codemods, listen to what others say here.

Say for example, for Javascript, you need to know the ESTree spec or the node builder definition in ast-types. This module provides an efficient, modular, Esprima-compatible implementation of the abstract syntax tree type hierarchy pioneered by the Mozilla Parser API.

Now let’s say you want to replace a CallExpressionfoo() with a new one like foo.bar(). The AST representation for the above two expressions will be like:

I have omitted a lot of information in the above code for clarity and readability purposes. It only contains the relevant information for the actual CallExpression AST node. If you want to explore the full tree structure of the AST, you can check it in ast-explorer.

As you can see from the above two AST nodes, the only difference between the two is the callee object which is a simple Identifier in foo() and a MemberExpression in foo.bar(). Usually with codemods, we will be replacing the original expression with the new one. Hence here, we will be replacing the original CallExpression with a new one like this.

In order to replace the old CallExpression with a new one, we need to build the new one. From the above codemod you can see we are building the new one using jscodeshift api like this:

If you try to build the above CallExpression within the ast-explorer transform editor, you will be having a tough time if you are doing it for the first-time. Because you are not very familiar with the the builder api in the first place, and you don’t know the correct order and type of parameters you need to supply to correctly build the AST node. And don’t forget the typos and punctuation errors you make while typing the code.

There are also some subtle nuances with the jscodeshift api which beginners won’t know for example, the api j.callExpression is a constructor for building CallExpression nodes, whereas j.CallExpression is an instance of the type CallExpression which is basically used to find nodes of the type CallExpression.

This is where AST Builder comes into the picture, it is acting as a reference guide for builder apis to easily build your AST nodes. Just input the expected code in the input editor (see the image above to identify the input editor which is always at the top left pane in the layout) you will get the builder api automatically generated for you without any mistakes. So if you input foo.bar() into the AST Builder it will give you something like:

You can safely omit the ExpressionStatement wrapper if you are just replacing the nodes.

Now, you can simply copy the builder api from AST Builder and use it in your codemods. How easy is that?

How?

AST Builder uses ‘ast-node-builder‘ an npm package underneath, which provides the apis for building AST nodes through jscodeshift. All the api’s takes an object as a parameter and returns the builder api in string format, which you can use with jscodeshift to create new nodes. The object which is passed to the api as a parameter is actually a node in the AST generated by the respective parser. When you feed the node to the ast-node-builder api, you get back the jscodeshift api to build that node.

This allows developers to easily and effectively create AST nodes from source code, instead of tinkering with the autocomplete api in ast-explorer. All you have to do is just enter or copy paste the source code into the input editor and you can see the jscodeshift api automatically generated for you in the output editor.

The above snippet will generate some thing like this:

You can also use the AST Builder to visualize your AST on the top right pane without all the noise and clutter of meta information. We deliberately filter out the loc nodes from the AST and also the tokens, since we feel it is not of much use for working with codemods. To dig deep into the builder you can take a look at the source code here, it is built in Ember.js.

And if you are a dark-theme fan, AST Builder also allows to switch your editor themes to dark mode. You can play around with the list of languages and list of parsers. Please use the Report issues link at the footer if you want to report any issues or feedback, you can tell us how we can improve the tool and what additional languages we need to support.

Stay tuned or subscribe to the newsletter at the bottom to know more about the exciting tools we are building around ASTs and Codemods.

References

Categories
Articles Javascript

Codemods – The New Age Saviors for JS Developers

In this article, we take a look at Codemods, a tool for making sweeping changes across your code with ease and effectiveness. Codemods are invented by a team of developers from Facebook to assist large-scale migrations in code-base, and we are going to take a deep look into a particular codemod tool called jscodeshift.

Refactoring

Before we dive into the codemod tools like jscodeshift, we need to understand why we need to do large-scale refactorings and what is really meant by refactoring. The term refactoring was first introduced by Martin Fowler from ThoughtWorks and he also wrote a seminal book about the topic. If you want to get better at writing good code or to do better refactorings, this is the one book you should definitely look into. According to Fowler,

Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior.

Its heart is a series of small behavior preserving transformations. Each transformation (called a “refactoring”) does little, but a sequence of these transformations can produce a significant restructuring. Since each refactoring is small, it’s less likely to go wrong. The system is kept fully working after each refactoring, reducing the chances that a system can get seriously broken during the restructuring.

But the real motivation behind code refactorings are the smells in your code or some hidden issues lurking in the dark for a very long time, which will eventually pop up during one of the critical moments when your application is trying to do some important task, this code can prove costly or it might bring you a maintenance nightmare time and again. Hence we need to understand these intricate patterns of code causing all these troubles in the first place and then apply proper refactoring mechanisms to remove them completely.

Code smells

Bad smells in code are certain structures in the code that suggest (sometimes they scream for) the possibility of refactoring. Determining what is and is not a code smell is subjective, and varies by language, developer, and development methodology. In Refactoring, Martin Fowler along with Kent Beck have identified about 20 different kinds of code smells. They have also catalogued the various refactoring methods that can be applied to remove these code smells from your source.

Tools

jscodeshift

jscodeshift is a toolkit for running codemods over multiple JavaScript or Typescript files. It provides:

  • A runner, which executes the provided transform for each file passed to it. It also outputs a summary of how many files have (not) been transformed.
  • A wrapper around recast, providing a different API. recast is an AST-to-AST transform tool and also tries to preserve the style of original code as much as possible.
    For making transforms using jscodeshift, all you need to do is install the package through npm and run it with a list of transforms against your source files.$ npm i -g jscodeshift $ jscodeshift -t myTransforms fileA fileB –foo=bar

recast

recast is a JavaScript syntax tree transformer, non-destructive pretty-printer, and automatic source map generator. Recast exposes two essential interfaces, one for parsing JavaScript code (require(“recast”).parse) and the other for reprinting modified syntax trees (require(“recast”).print).

Here’s a simple but non-trivial example of how you might use .parse and .print:

Now do whatever you want to ast. Really, anything at all! When you finish manipulating the AST, let recast.print work its magic:

High level overview of Codemods work

At a very high level, codemods do a very simple processing with your source code like converting it to AST representations, manipulating nodes in the AST and then finally converting the AST back to source code.

Different stages of Codemods

Below you can find the details working of how the jscodeshift and recast tools transform your source code into AST and back to code again.

AST

An abstract syntax tree (AST), or just syntax tree, is a tree representation of the abstract syntactic structure of source code written in a programming language. Each node of the tree denotes a construct occurring in the source code.

What you can do with AST

Understanding AST representations of your source code gives you a great leverage in doing a lot of awesome stuff like the following:

  • You can write your own eslint rules to enforce code conventions
  • You can write your own JS transpiling code
  • You can write powerful code mods to refactor your codebase

Codemods in Frameworks

A lot of JS frameworks nowadays relies heavily on codemods to make their migrations seamless, so that their consumers or the developers using the frameworks don’t have to rewrite a lot of stuff while moving on to newer apis or a newer version of the framework itself. Frameworks likeReactVueAngular have published a lot of codemod tools to assist the developers in carrying out large-scale code migrations for their applications written in those frameworks.

Ember ❤ Codemods

Ember.js, a popular MVC model JS framework relies heavily one codemods for their migration paths from one version to the next. Whenever there is a deprecation or breaking change introduced in the framework, the core team will ensure that they have proper backward compatibility and clearly defined migration paths with codemods.

ember-cli-update

ember-cli-update is a command-line tool which updates Ember CLI Ember.js apps and addons (and Glimmer.js apps). You can run this either as a global executable available to all projects or an Ember CLI command in a single project. It also fetches list of codemods and instructions from ember-cli-update-codemods-manifest.

To update to the latest version of Ember CLI:

To update to a certain version of Ember CLI:

To run codemods: (This should be run after running the normal update shown above, and after you’ve resolved any conflicts.)

Reference

Videos

Articles

Tools