Categories
Articles Javascript

AST Builder – Building AST nodes from code

In this article, we are going to take a look at a tool called ‘AST Builder’ which will significantly improve the developer experience for writing codemods.

Codemod is a tool/library to assist you with large-scale codebase refactors that can be partially automated but still require human oversight and occasional intervention. Codemod was developed at Facebook and released as open source.

If you want to know more about codemods, their building blocks and how they work, please check out this detailed post about codemods.

What?

AST Builder is actually a playground for building AST nodes using source code. Because ASTs play a big role in writing codemods, this tool will assist the developers to a great extent in writing codemods. Because codemods actually do AST-to-AST transformation on your source code, and that’s one of the primary reasons, why codemods are more resilient in making effective code transformations.

It currently supports Javascript (ES5, ES6 and some ES7 constructs) , JSX and Glimmer.js handlebars syntax. Please take a look at these issues, Cover core apiCover ES6 api for coverage information. And I am planning to include more language syntax and semantics.

Why?

We already have an well-established and battle-tested tool called “ast-explorer” for visualizing abstract syntax trees or ASTs. Why we need a new tool then? Because AST explorer is only for exploring your ASTs, it doesn’t tell how to create AST nodes. Even though, ast-explorer offers intellisense in their editor for the jscodeshift apis, it doesn’t work for all the parsers like you can only be using the autocomplete api for recast parser. If you choose any other parser other than recast, you won’t get the intellisense in the codemod editor.

And most of the time, you will be creating nodes for transforming code using codemods. And we definitely need a tool which makes it easy to create nodes. The problem is, there is no proper documentation on creating AST nodes using the jscodeshift api. All you have to do is learn from other codemods out there and sift through the code and find out how you can create new nodes.

For that you need to understand the parser internals, the Node schema and the types of node builders available for the language you are working.

If you are still not convinced why this tool will make a difference in developer experience for building codemods, listen to what others say here.

Say for example, for Javascript, you need to know the ESTree spec or the node builder definition in ast-types. This module provides an efficient, modular, Esprima-compatible implementation of the abstract syntax tree type hierarchy pioneered by the Mozilla Parser API.

Now let’s say you want to replace a CallExpressionfoo() with a new one like foo.bar(). The AST representation for the above two expressions will be like:

I have omitted a lot of information in the above code for clarity and readability purposes. It only contains the relevant information for the actual CallExpression AST node. If you want to explore the full tree structure of the AST, you can check it in ast-explorer.

As you can see from the above two AST nodes, the only difference between the two is the callee object which is a simple Identifier in foo() and a MemberExpression in foo.bar(). Usually with codemods, we will be replacing the original expression with the new one. Hence here, we will be replacing the original CallExpression with a new one like this.

In order to replace the old CallExpression with a new one, we need to build the new one. From the above codemod you can see we are building the new one using jscodeshift api like this:

If you try to build the above CallExpression within the ast-explorer transform editor, you will be having a tough time if you are doing it for the first-time. Because you are not very familiar with the the builder api in the first place, and you don’t know the correct order and type of parameters you need to supply to correctly build the AST node. And don’t forget the typos and punctuation errors you make while typing the code.

There are also some subtle nuances with the jscodeshift api which beginners won’t know for example, the api j.callExpression is a constructor for building CallExpression nodes, whereas j.CallExpression is an instance of the type CallExpression which is basically used to find nodes of the type CallExpression.

This is where AST Builder comes into the picture, it is acting as a reference guide for builder apis to easily build your AST nodes. Just input the expected code in the input editor (see the image above to identify the input editor which is always at the top left pane in the layout) you will get the builder api automatically generated for you without any mistakes. So if you input foo.bar() into the AST Builder it will give you something like:

You can safely omit the ExpressionStatement wrapper if you are just replacing the nodes.

Now, you can simply copy the builder api from AST Builder and use it in your codemods. How easy is that?

How?

AST Builder uses ‘ast-node-builder‘ an npm package underneath, which provides the apis for building AST nodes through jscodeshift. All the api’s takes an object as a parameter and returns the builder api in string format, which you can use with jscodeshift to create new nodes. The object which is passed to the api as a parameter is actually a node in the AST generated by the respective parser. When you feed the node to the ast-node-builder api, you get back the jscodeshift api to build that node.

This allows developers to easily and effectively create AST nodes from source code, instead of tinkering with the autocomplete api in ast-explorer. All you have to do is just enter or copy paste the source code into the input editor and you can see the jscodeshift api automatically generated for you in the output editor.

The above snippet will generate some thing like this:

You can also use the AST Builder to visualize your AST on the top right pane without all the noise and clutter of meta information. We deliberately filter out the loc nodes from the AST and also the tokens, since we feel it is not of much use for working with codemods. To dig deep into the builder you can take a look at the source code here, it is built in Ember.js.

And if you are a dark-theme fan, AST Builder also allows to switch your editor themes to dark mode. You can play around with the list of languages and list of parsers. Please use the Report issues link at the footer if you want to report any issues or feedback, you can tell us how we can improve the tool and what additional languages we need to support.

Stay tuned or subscribe to the newsletter at the bottom to know more about the exciting tools we are building around ASTs and Codemods.

References

Categories
Articles EmberJS

Creating runtime assisted Codemods using Telemetry helpers

In this article, we are going to take a glimpse at the ember-codemods-telemetry-helpers package and how it helps to create more advanced codemods for Ember.js.

If you want a more in-depth introduction to codemods, you can checkout this post detailing the why’s and how’s of codemods.

Telemetry for layman

First we will take a look at what Telemetry is all about. According to Wikipedia,

Telemetry is the collection of measurements or other data at remote or inaccessible points and their automatic transmission to receiving equipment for monitoring.

Software Telemetry

In software, telemetry is used to gather data on the use and performance of applications and its components, e.g. how often certain features are used, measurements of start-up time and processing time, hardware, application crashes and general usage statistics and/or user behavior. In some cases, very detailed data is reported like individual window metrics, counts of used features and individual function timings.

This kind of telemetry can be essential to software developers to receive data from a wide variety of endpoints that can’t possibly all be tested in-house.

Privacy concerns

Due to concerns about privacy, since software telemetry can easily be used to profile users, telemetry in user software is often an user choice, commonly presented as an opt-in feature (requiring explicit user action to enable it) or during the software installation process.

We had enough look about what really Telemetry is about. Now it’s time to look at the more specific use-case of creating codemods using telemetry helpers.

Telemetry helpers for Codemods

Telemetry helpers runs the app, grabs basic info about all of the modules at runtime. This allows the codemod to know the names of every helper, component, route, controller, etc. in the app without guessing / relying on static analysis. They basically help you to create “runtime assisted codemods”.

Below you can find the package composition of the ember-codemods-telemetry-helpers package.

The goal of this project is to enable each codemod to manage its own type of data gathering and to provide the harness to run that custom gathering function.

This package exports six functions for gathering telemetry information which can be used in the codemods.

Using the telemetry helpers for codemods

Assuming you are authoring a codemod with codemod-cliember-codemods-telemetry-helpers allows you the freedom to assign your own “telemetry gathering” function while provide one of its own out of the box.

All invocations of gatherTelemetryForUrl internally returns an object enumerated with properties named after all possible entries within window.require.entries. The values of each property is the value returned from within the gathering function. Using the example above, the output might be:

This package provides one gathering function: analyzeEmberObject. This function does a “best effort” analysis of the app runtime, returning such things as Components, Helpers, Routes, etc. and their particular properties.

It parses Ember meta data object, collects the runtime information and returns the following list of properties :

Gathering runtime data

Let’s see how the codemods use the telemetry helpers to gather data at runtime. For that, let’s take an example to see how the ember-native-class-codemod is invoked.

The first argument that you must pass to the codemod is the URL of a running instance of your application. The codemod opens up your application using the URL passed as the argument and analyzes the classes directly in order to transform them. Any classes that were not analyzed will not be transformed. This includes classes that are private to a module and never exported.

If you have any lazily loaded modules, such as modules from Ember Engines, you need to ensure that the URL you provide loads these modules as well. Otherwise, the codemod will not be able to detect or analyze them.

Origins of the Telemetry helpers

This project was extracted from ember-native-class-codemod. That codemod uses Puppeteer through this package to visit the Ember app and gather telemetry necessary to convert to native classes.

The idea for the extraction was to put the harness in this package (extracted from the native class codemod), but have the actual “telemetry gathering” live in each individual codemod project because the things that they need are quite different, for example, for implicit this codemod and angle brackets codemod all we need to know is an array of the helpers and components in the app, but for native class codemod it needs much more information such as names and types of methods, properties, etc on each default export.

Codemods already using the helpers

Currently there are two codemods : ember-native-class-codemod and ember-no-implicit-this-codemod, which are already making use of the telemetry helpers to gather information from the app during runtime. And there is an open pull request by Ryan Mark for the ember-angle-brackets-codemod to make use of the telemetry helpers.

References

Categories
Uncategorized

The Craftsman – Mind Map

In this post, we are going to share you a mind map of the book called The Craftsman by Richard Sennett.

Richard Sennett is a prime observer of society … one of his great strengths, the thing that makes his narrative so gripping, is the sheer range of his thinking and his brilliance in relating the past to the present.

The Craftsman is a complex but very rewarding read about craftsmanship, pride in one’s work and the differences between individual work and mechanical production, and the intellectual investment in the former versus the more mechanistic approach in the latter.

The Craftsman by Richard Sennett


Sennett uses many examples from the world of music to illustrate his thesis and he finishes with an explanation of the philosophical underpinnings of his ideas, pragmatism in this case.
This book has an extensive index and bibliography and his examples are well-chosen and illustrative and evidence of his wide understanding of the matter. Not a book to read on the beach or a plane or even at home in a week or so. It requires time and patience and is rewarding even when dipping into individual sections but the real benefit comes with the final section brings together the threads and leaves the reader with much more understanding of this field of human endeavor than one had before.

Categories
Articles

Mind maps in Spacemacs

This post is the second in the series of posts called PlantUML in Spacemacs. In this post we are going to see how we can create awesome Mind maps in Spacemacs using PlantUML. Mind map diagrams is the new feature recently rolled out in PlantUML and they are still in the beta* stage.

And you will find only minimal documentation in the official site related to mind map diagrams. This post gives you some additional bonus information such as adding colors to your mind map, creating high resolution mind maps and so on. You can find all these details in the Advanced Usage section in the bottom of the post.

If you want to know how to setup PlantUML in Spacemacs, please visit our previous post in this series.

Spacemacs

Spacemacs is a community-driven Emacs distribution. It is a new way to experience Emacs with a sophisticated and polished set-up focused on ergonomics, mnemonics and consistency.

Mind maps

A mind map is a diagram used to visually organize information. A mind map is hierarchical and shows relationships among pieces of the whole. It is often created around a single concept, drawn as an image in the center of a blank page, to which associated representations of ideas such as images, words and parts of words are added. Major ideas are connected directly to the central concept, and other ideas branch out from those major ideas.

1200px-Tennis-mindmap.png

Although the term “mind map” was first popularized by British popular psychology author and television personality Tony Buzan, the use of diagrams that visually “map” information using branching and radial maps traces back centuries.

OrgMode syntax

Mind maps in PlantUML support the org-mode syntax also, which means you can easily create mind maps if you know understand the standard org-mode conventions for outlining. This will be an added advantage for people who are already familiar with Emacs and org-mode syntax.

Removing box

You can remove the box drawing using an underscore.

Arithmetic notation

You can use the following notation to choose diagram side.

Markdown syntax

This syntax is compatible with Markdown.

Changing diagram direction

It is possible to use both sides of the diagram.

Complete Example

Here is the complete example of a mind map with all the options which can be provided for your mind maps like caption, title, header, footer and legends.

Advanced Usage

So far we have only seen the examples provided by the official docs for Mind maps in PlantUML. Now we move into more advanced usage like colors, style and other stuff which will be very helpful for people instead of boring monotonously colored mind maps.

First we will jump into the colors section which is my favorite one, because I want my mind maps to be more colorful because that was the original idea proposed by Tony Buzan in his book called The Speed Reading Book.

Colors

With PlantUML, you can use specify a color using any one of the below methods:

  • with its standard name
  • using HEX value #AABBCC
  • using short HEX value #ABC

You can visit the official colors page in PlantUML for the list of color names you can use.

Changing background color

You can also change the entire background color of your mind maps other than the default white color using skinparam. skinparam is a command to change colors and fonts for the drawing.

You can use this command:

  • In the diagram definition, like any other commands,
  • In an included file,
  • In a configuration file, provided in the command line or the ANT task

Changing color of each node

If you want to change the color of each of the nodes in your mind map tree, you can do so by giving the color option within each node like below. Again you can use any of the three options such as color names, short and long HEX values.

Using color names:

Using HEX values:

Shorthand hex notation abbreviates 6-character RRGGBB CSS colors into 3-character RGB shorthand. So the color value ##FF6600 becomes #F60 in the shorthand notation.

Using short version HEX values:

Creating Monochrome Mind Map

If you are a person who cares about the environment, who don’t want to waste the color inks in your printer, you can choose to generate your mind maps in monochrome version like black/white with the monochrome skinparam command.

Handwritten style

If you want your mind maps to look realistic or you are not comfortable with computer-generated mind maps, or you want a natural feel to your mind maps so that they will look like they are hand drawn, then you need to configure your mind map generation with the handwritten skinparam.

By default this property will be false and if you want handwritten style mind maps you can set it to true.

Disabling Shadows

You can disable shadows from your mind map items with the option called shadowing. Again it is a skinparam command with a Boolean value. To disable shadows you need to explicitly set them to false.

Creating Print resolution Mind Map

By default, the default resolution for the image generated using PlantUML will be 72 which is the default resolution for digital media and the size of your mind maps will be determined by the content it has such as how big the tree is or how many number of nodes or leafs you have in it.

This option called dpi will come in handy if you want to take a print out of your mind maps and hang it in your favorite place so that you can take a look at them at your convenience.

DPI or Dots per inch is a measure of spatial printing or video or image scanner dot density, in particular the number of individual dots that can be placed in a line within the span of 1 inch.

For print material, the ideal resolution needed is 300 which you can set it via the dpi skinparam command.

Hope you enjoyed the post and it will help you to create awesome Mind maps from within your favorite editor. Please let me know your feedback or queries in the comments. Stay tuned for more in this series on other cool stuff you can do with PlantUML.

References