Category: Articles

Categories
Articles Javascript

JARVIS – Write me a Codemod

In this article, we are going to take a look at a tool called JARVIS which will profoundly transform the way how you write codemods.

This is a continuation post for AST Finder which is a tool to generate an api to find AST nodes automatically from source code.

If you want to know more about codemods, their building blocks and how they work, please check out this detailed post about codemods.

What?

JARVIS is actually a playground for writing codemods, where in you can specify the input and output code, choose the appropriate transformation you want to apply and the codemod is automatically generated for you.

JARVIS currently supports Javascript and Glimmer.js templates. For Javascript, you can use parsers like recast and babel. For Glimmer.js templates which is a more advanced form of Handlebars templates used by Ember.js framework, it is using the ember-template-recast parser to parse and transform the code.

It gives a playground to write code transformations just like AST Explorer but with a more fine-grained control and flexibility to write the transformations because it automatically figures out a lot of things for you like, how you can build new nodes, how you can find nodes from the AST, how to replace, remove or insert nodes into the tree, which visitor function you need to use for writing transformations.

Why?

If the goal is to enhance software development throughput, you can either:

• get faster people to do the work,
• get more people to do the work, or
• automate the work.

― Chad Fowler in The Pragmatic Programmer

We already have a very good tool called “ast-explorer” for visualizing abstract syntax trees or ASTs and writing codemods. Why we need a new tool then? Because AST explorer is only for exploring your ASTs, it doesn’t tell how to find or build AST nodes. You have to figure out a lot of things before you can start writing a codemod.

I spent a lot of time figuring out the jscodeshift apis, by looking at their documentation, learning from existing codemods, searching in google, and so on. The learning curve is kind of steep. And my colleagues have to tread down the same path. Whenever we tell any new developer in our team to learn about codemods, they face the same challenges we had in learning about them. Hence I wanted to make their lives easier and approach, learning about codemods a painless task. And that’s why JARVIS is created.

The first task in writing codemods is to find the appropriate nodes in the AST and work on them. And we definitely need a tool which makes it easy to find nodes. The problem is, there is no proper documentation on finding AST nodes using the jscodeshift api. All you have to do is learn from other awesome-codemods out there and sift through the code and find out how you can query nodes.

And the second thing, when you are replacing nodes, you need to know how you can build new nodes. While writing codemods, most of the time we will be creating new nodes. JARVIS can take care of these things for you so that you can focus on building or tweaking the transformation rather than wasting most of your precious time in finding the reference or documentation for building and finding AST nodes.

Once you are done with the transformation, you can simply copy the codemod from JARVIS, put it in a file, let’s say for example transform.js and run it through with jscodeshift.

How?

JARVIS uses two npm packages underneath. One for building AST nodes from code and the other one for finding AST nodes from code. It is pretty much a combination of both AST-Builder and AST-Finder working together to automatically generate code transforms.

First you need to specify the input code in the top-left panel (please see the annotated screenshot above for reference), then you choose the appropriate AST Node operation below. You can choose to remove the node, replace , insert the new node before or after the existing node.

Second, if you are replacing code you can enter the destination code how it is going to change in the bottom-left panel. Then JARVIS will automatically generate a boilerplate code for your transformation in the top-right panel. You can tweak the transformation according to your requirements and verify the transformation is working properly or not by looking at the output in the bottom-right panel.

Let’s consider a simple example in JS. Let’s say you want to replace all your foo() calls with foo.bar() in your code. The first task here is you need to identify the type of AST node you want to manipulate, in our case replace. This is an ExpressionStatement with an expression of type CallExpression. Whereas, the expected output node type is a MemberExpression. Hence you need to understand the syntax and semantics of finding a CallExpression and create a MemberExpression AST node, which will be a very difficult endeavor for someone who has just starting with codemods.

What JARVIS does here is, it figures out the type of the input node and the output node, the node manipulation operation you want to do with, and then automatically generates a codemod boilerplate for the same.

The codemod for the above transformation will look something like this:

To know more about how JARVIS works, you can take a look at the source code here, it is built in Ember.js.

And if you are a dark-theme fan, JARVIS also allows to switch your editor themes to dark mode. Please use the Report issues link at the footer if you want to report any issues or feedback, you can tell us how we can improve the tool and what additional languages / parsers we need to support.

Stay tuned or subscribe to the newsletter at the bottom to know more about the exciting tools we are building around ASTs and Codemods.

References

Categories
Articles Javascript

AST Finder – Finding AST nodes from code

In this article, we are going to take a look at a tool called ‘AST Finder’ which will significantly improve the developer experience for writing codemods.

This is a continuation post for AST Builder which is a tool to generate AST nodes automatically from source code.

If you want to know more about codemods, their building blocks and how they work, please check out this detailed post about codemods.

What?

AST Finder is actually a playground for finding AST nodes using source code. Because ASTs play a big role in writing codemods, this tool will assist the developers to a great extent in writing codemods. Because codemods actually do AST-to-AST transformation on your source code, and that’s one of the primary reasons, why codemods are more resilient in making effective code transformations.

Why?

Never memorize something that you can look up.

― Albert Einstein

We already have an well-established and battle-tested tool called “ast-explorer” for visualizing abstract syntax trees or ASTs. Why we need a new tool then? Because AST explorer is only for exploring your ASTs, it doesn’t tell how to find AST nodes.

The first task in writing codemods is to find the appropriate nodes in the AST and work on them. And we definitely need a tool which makes it easy to find nodes. The problem is, there is no proper documentation on finding AST nodes using the jscodeshift api. All you have to do is learn from other codemods out there and sift through the code and find out how you can query nodes.

Now let’s say you want to replace a CallExpression, foo.bar() with a new one like foo(). The AST representation for the above two expressions will be like:

I have omitted a lot of information in the above code for clarity and readability purposes. It only contains the relevant information for the actual CallExpression AST node. If you want to explore the full tree structure of the AST, you can check it in ast-explorer.

As you can see from the above two AST nodes, the only difference between the two is the callee object which is a simple Identifier in foo() and a MemberExpression in foo.bar(). Usually with codemods, we will be replacing the original expression with the new one. Hence here, we will be replacing the original CallExpression with a new one like this.

In order to replace the old CallExpression with a new one, first we need to find the existing CallExpression. From the above codemod you can see we are querying the AST using jscodeshift api like this:

If you try to find the above CallExpression within ast-explorer transform editor, you will be having a tough time if you are doing it for the first-time. Because you are not very familiar with the find api in the first place, and you don’t know the right order and type of parameters you need to supply to correctly find the AST node. And don’t forget the typos and punctuation errors you make while typing the code.

There are also some subtle nuances with the jscodeshift api which beginners won’t know for example, the api j.callExpression is a constructor for building CallExpression nodes, whereas j.CallExpression is an instance of the type CallExpression which is basically used to find nodes of the type CallExpression.

This is where AST Finder comes into the picture, it is acting as a reference guide for find apis to easily query your AST nodes. Just input the code in the input editor (see the image above to identify the input editor which is always at the top left pane in the layout) you will get the find api automatically generated for you without any mistakes. So if you input foo.bar() into the AST Finder, it will give you something like:

Now, you can simply copy the query from AST Finder and use it in your codemods. How cool is that?

How?

AST Finder uses ‘ast-node-finder‘ an npm package underneath, which provides the apis for finding AST nodes through jscodeshift. All the api’s takes an object as a parameter and returns the find api in string format, which you can use with jscodeshift to query new nodes. The object which is passed to the api as a parameter is actually a node in the AST generated by the respective parser. When you feed the node to the ast-node-finder api, you get back the jscodeshift api to find that node.

This allows developers to easily and effectively find AST nodes from source code. All you have to do is just enter or copy paste the source code into the input editor and you can see the jscodeshift api automatically generated for you in the output editor.

The above snippet will generate some thing like this:

You can also use the AST Finder to visualize your AST on the top right pane without all the noise and clutter of meta information. We deliberately filter out the loc nodes from the AST and also the tokens, since we feel it is not of much use for working with codemods. To dig deep into the finder you can take a look at the source code here, it is built in Ember.js.

And if you are a dark-theme fan, AST Finder also allows to switch your editor themes to dark mode. Please use the Report issues link at the footer if you want to report any issues or feedback, you can tell us how we can improve the tool and what additional languages we need to support.

Stay tuned or subscribe to the newsletter at the bottom to know more about the exciting tools we are building around ASTs and Codemods.

References

Categories
Articles Javascript

AST Builder – Building AST nodes from code

In this article, we are going to take a look at a tool called ‘AST Builder’ which will significantly improve the developer experience for writing codemods.

Codemod is a tool/library to assist you with large-scale codebase refactors that can be partially automated but still require human oversight and occasional intervention. Codemod was developed at Facebook and released as open source.

If you want to know more about codemods, their building blocks and how they work, please check out this detailed post about codemods.

What?

AST Builder is actually a playground for building AST nodes using source code. Because ASTs play a big role in writing codemods, this tool will assist the developers to a great extent in writing codemods. Because codemods actually do AST-to-AST transformation on your source code, and that’s one of the primary reasons, why codemods are more resilient in making effective code transformations.

It currently supports Javascript (ES5, ES6 and some ES7 constructs) , JSX and Glimmer.js handlebars syntax. Please take a look at these issues, Cover core api, Cover ES6 api for coverage information. And I am planning to include more language syntax and semantics.

Why?

We already have an well-established and battle-tested tool called “ast-explorer” for visualizing abstract syntax trees or ASTs. Why we need a new tool then? Because AST explorer is only for exploring your ASTs, it doesn’t tell how to create AST nodes. Even though, ast-explorer offers intellisense in their editor for the jscodeshift apis, it doesn’t work for all the parsers like you can only be using the autocomplete api for recast parser. If you choose any other parser other than recast, you won’t get the intellisense in the codemod editor.

And most of the time, you will be creating nodes for transforming code using codemods. And we definitely need a tool which makes it easy to create nodes. The problem is, there is no proper documentation on creating AST nodes using the jscodeshift api. All you have to do is learn from other codemods out there and sift through the code and find out how you can create new nodes.

For that you need to understand the parser internals, the Node schema and the types of node builders available for the language you are working.

If you are still not convinced why this tool will make a difference in developer experience for building codemods, listen to what others say here.

Say for example, for Javascript, you need to know the ESTree spec or the node builder definition in ast-types. This module provides an efficient, modular, Esprima-compatible implementation of the abstract syntax tree type hierarchy pioneered by the Mozilla Parser API.

Now let’s say you want to replace a CallExpression, foo() with a new one like foo.bar(). The AST representation for the above two expressions will be like:

I have omitted a lot of information in the above code for clarity and readability purposes. It only contains the relevant information for the actual CallExpression AST node. If you want to explore the full tree structure of the AST, you can check it in ast-explorer.

As you can see from the above two AST nodes, the only difference between the two is the callee object which is a simple Identifier in foo() and a MemberExpression in foo.bar(). Usually with codemods, we will be replacing the original expression with the new one. Hence here, we will be replacing the original CallExpression with a new one like this.

In order to replace the old CallExpression with a new one, we need to build the new one. From the above codemod you can see we are building the new one using jscodeshift api like this:

If you try to build the above CallExpression within the ast-explorer transform editor, you will be having a tough time if you are doing it for the first-time. Because you are not very familiar with the the builder api in the first place, and you don’t know the correct order and type of parameters you need to supply to correctly build the AST node. And don’t forget the typos and punctuation errors you make while typing the code.

There are also some subtle nuances with the jscodeshift api which beginners won’t know for example, the api j.callExpression is a constructor for building CallExpression nodes, whereas j.CallExpression is an instance of the type CallExpression which is basically used to find nodes of the type CallExpression.

This is where AST Builder comes into the picture, it is acting as a reference guide for builder apis to easily build your AST nodes. Just input the expected code in the input editor (see the image above to identify the input editor which is always at the top left pane in the layout) you will get the builder api automatically generated for you without any mistakes. So if you input foo.bar() into the AST Builder it will give you something like:

You can safely omit the ExpressionStatement wrapper if you are just replacing the nodes.

Now, you can simply copy the builder api from AST Builder and use it in your codemods. How easy is that?

How?

AST Builder uses ‘ast-node-builder‘ an npm package underneath, which provides the apis for building AST nodes through jscodeshift. All the api’s takes an object as a parameter and returns the builder api in string format, which you can use with jscodeshift to create new nodes. The object which is passed to the api as a parameter is actually a node in the AST generated by the respective parser. When you feed the node to the ast-node-builder api, you get back the jscodeshift api to build that node.

This allows developers to easily and effectively create AST nodes from source code, instead of tinkering with the autocomplete api in ast-explorer. All you have to do is just enter or copy paste the source code into the input editor and you can see the jscodeshift api automatically generated for you in the output editor.

The above snippet will generate some thing like this:

You can also use the AST Builder to visualize your AST on the top right pane without all the noise and clutter of meta information. We deliberately filter out the loc nodes from the AST and also the tokens, since we feel it is not of much use for working with codemods. To dig deep into the builder you can take a look at the source code here, it is built in Ember.js.

And if you are a dark-theme fan, AST Builder also allows to switch your editor themes to dark mode. You can play around with the list of languages and list of parsers. Please use the Report issues link at the footer if you want to report any issues or feedback, you can tell us how we can improve the tool and what additional languages we need to support.

Stay tuned or subscribe to the newsletter at the bottom to know more about the exciting tools we are building around ASTs and Codemods.

References

Categories
Articles EmberJS

Creating runtime assisted Codemods using Telemetry helpers

In this article, we are going to take a glimpse at the ember-codemods-telemetry-helpers package and how it helps to create more advanced codemods for Ember.js.

If you want a more in-depth introduction to codemods, you can checkout this post detailing the why’s and how’s of codemods.

Telemetry for layman

First we will take a look at what Telemetry is all about. According to Wikipedia,

Telemetry is the collection of measurements or other data at remote or inaccessible points and their automatic transmission to receiving equipment for monitoring.

Software Telemetry

In software, telemetry is used to gather data on the use and performance of applications and its components, e.g. how often certain features are used, measurements of start-up time and processing time, hardware, application crashes and general usage statistics and/or user behavior. In some cases, very detailed data is reported like individual window metrics, counts of used features and individual function timings.

This kind of telemetry can be essential to software developers to receive data from a wide variety of endpoints that can’t possibly all be tested in-house.

Privacy concerns

Due to concerns about privacy, since software telemetry can easily be used to profile users, telemetry in user software is often an user choice, commonly presented as an opt-in feature (requiring explicit user action to enable it) or during the software installation process.

We had enough look about what really Telemetry is about. Now it’s time to look at the more specific use-case of creating codemods using telemetry helpers.

Telemetry helpers for Codemods

Telemetry helpers runs the app, grabs basic info about all of the modules at runtime. This allows the codemod to know the names of every helper, component, route, controller, etc. in the app without guessing / relying on static analysis. They basically help you to create “runtime assisted codemods”.

Below you can find the package composition of the ember-codemods-telemetry-helpers package.

The goal of this project is to enable each codemod to manage its own type of data gathering and to provide the harness to run that custom gathering function.

This package exports six functions for gathering telemetry information which can be used in the codemods.

Using the telemetry helpers for codemods

Assuming you are authoring a codemod with codemod-cli, ember-codemods-telemetry-helpers allows you the freedom to assign your own “telemetry gathering” function while provide one of its own out of the box.

All invocations of gatherTelemetryForUrl internally returns an object enumerated with properties named after all possible entries within window.require.entries. The values of each property is the value returned from within the gathering function. Using the example above, the output might be:

This package provides one gathering function: analyzeEmberObject. This function does a “best effort” analysis of the app runtime, returning such things as Components, Helpers, Routes, etc. and their particular properties.

It parses Ember meta data object, collects the runtime information and returns the following list of properties :

Gathering runtime data

Let’s see how the codemods use the telemetry helpers to gather data at runtime. For that, let’s take an example to see how the ember-native-class-codemod is invoked.

The first argument that you must pass to the codemod is the URL of a running instance of your application. The codemod opens up your application using the URL passed as the argument and analyzes the classes directly in order to transform them. Any classes that were not analyzed will not be transformed. This includes classes that are private to a module and never exported.

If you have any lazily loaded modules, such as modules from Ember Engines, you need to ensure that the URL you provide loads these modules as well. Otherwise, the codemod will not be able to detect or analyze them.

Origins of the Telemetry helpers

This project was extracted from ember-native-class-codemod. That codemod uses Puppeteer through this package to visit the Ember app and gather telemetry necessary to convert to native classes.

The idea for the extraction was to put the harness in this package (extracted from the native class codemod), but have the actual “telemetry gathering” live in each individual codemod project because the things that they need are quite different, for example, for implicit this codemod and angle brackets codemod all we need to know is an array of the helpers and components in the app, but for native class codemod it needs much more information such as names and types of methods, properties, etc on each default export.

Codemods already using the helpers

Currently there are two codemods : ember-native-class-codemod and ember-no-implicit-this-codemod, which are already making use of the telemetry helpers to gather information from the app during runtime. And there is an open pull request by Ryan Mark for the ember-angle-brackets-codemod to make use of the telemetry helpers.

References