Table of Contents

Hopc Development

Note: This documentation page is intended for export developers only, in particular those that are willing to experiment with new compilation optimizations or analysis.

Hopc is the Hop multi-language compiler. It accepts has source languages Scheme and JavaScript and it can generates any of these two languages as target. This section describes how to modify the compiler when the source language is JavaScript.

The Hopc JavaScript compiler is multi-staged. As of branch 3.3 it contains about 50 stages. The compiler can be tweaked in two ways:

Multi-staged compilation

In the compiler terminology, a driver is the compiler entity that controls how stages flow one from the other. The compiler exports a list of builtin drivers. This list can be obtained with:

$ hopc --js-drivers-list

A particular builtin driver can be selected with:

$ hopc --js-driver DRIVER-NAME

The list and order of compilation stages that correspond to that driver are obtained with:

$ hopc --js-driver DRIVER-NAME --js-show-driver

Custom drivers can be used to compile a source file by explicitly listing the stages to use. Example:

$ hopc --js-driver syntax,hopscript-header,loopexit,bestpractice,symbol,this,read-only,return,property,scheme foo.js -v2

Note that when the option -v2 or higher is used on the command line, the compiler displays all the compilation stages it executes.

If the shell HOPTRACE environment variable is bound to a string prefixed with j2s:, for instance HOPTRACE=j2s:stage, the compiler will dump its internal abstract syntax tree after executing each stage. These dumps will be located in /tmp/$USER/J2S/your-source/....

A dedicated syntax enables users to insert a custom stage in a predefined driver. The syntax is as follow

$ hopc --js-driver ...,ANCHOR,USER foo.js -v2

Using this syntax, USER stage will be executed after ANCHOR stage, found in the normal driver. For instance,

$ hopc -Ox --js-driver ...,arguments,stats.hop foo.js -v2

Will execute the stats.hop user stage after the arguments stage of the optim driver (because of the use of the -Ox option).

External Compilation Stages

External compilation stages can be used to specify a custom compilation driver. For that the URL of the stage is used instead of a the name of a builtin stage. For instance, let us assume that a Hop server is running and accepting connections on port 8888 and let us assume that it implements a compilation stage accessible with a service named js2http, then a source compilation can be obtained with:

$ hopc --js-driver syntax,hopscript-header,loopexit,bestpractice,symbol,this,read-only,return,property,http://localhost:8888/hop/js2http,scheme foo.js -v2

The service invoked by the compiler will receive one object with two properties:

The service will have to return a valid AST that will be used by the following compilation stages.

Several external compilation stages can be used in one compilation.

The Abstract Syntax Tree

The Hopc Abstract Syntax Tree (AST henceforth) is defined as a Scheme class hierarchy declared in the file hop/js2scheme/ast.scm. The Hop hop.hopc module enables easy access and manipulation from within JavaScript. This modules maps all the Scheme class to JavaScript classes and it provides two functions used to import and export the AST.

hopc.intern(ast)

This function accepts as input an AST as received by the service implementing the stage and it re-constructs the sharing between AST nodes that have been serialized by the compiler. This is named an interned ast.

A compilation must return an interned ast, so the simplest possible compilation stage is:

const hopc = require(hop.hopc);

service js2http({ ast, config }) {
    return hopc.intern(ast);
}

A compilation stage can modify or annotate the ast it receives. These modifications will be visible to the following compilation stages.

hopc.extern(ast)

Interned ast cannot be serialized into the JSON format using the regular toJSON function as these trees are cyclic data structure. The function hopc.extern serializes the tree into a JSON representation handling the sharing so that they could be re-established by the hopc.intern function.

Here is an example of a compilation stage that dumps its tree in a json file and read it again for its return value.

js2json.js

const fs = require('fs');
var exec = require('child_process').exec;
const hopc = require(hop.hopc);

service js2http({ ast, config }) {
   const prg = hopc.intern(ast);
   
   return new Promise((res, rej) => {
         fs.writeFile("/tmp/test.json", hopc.extern(prg), function(err) {
            // should call the analyzer here.
            var child = exec('cat /tmp/test.json');
            var result = '';

            child.stdout.on('data', function(data) {
               // analyser result.
               result += data;
            });

            child.on('close', function() {
               var o = JSON.parse(result);
               // pass the result to the next step.
               res(hopc.intern(o));
            });
         });
      });
}

Ast Walker

The hop.hopc module provides a facility for traversing ast.

new hopc.HiopcAstWalker()

Builds an ast walker.

walker.walk(prg)

Walks along the ast using the depth-first traversal. That is, this function traverses all the ast node scanning all nodes fields.

The traversal of each node can be individually modified by declaring new methods to a walker. Methods are named after class names. For instance, the following declares a custom walker for the nodes that correspond to JavaScript object accesses:

const w = new hopc.HopcAstWalker();
w.J2SAccess = function(node) {
   console.log("an access ", node.loc);
   return node;
}
w.walk(prg);

A Complete Example

This example shows how to implement an external Hopc compilation stage that modifies the ast it receives (by replacing + with `` and by displaying some information about the program.)*

To execute this example, a server must be executed with:

% hop -v -g -p 8888 js2http.js

The compiler should be invoked with:

% hopc -v fact.js --js-driver syntax,hopscript-header, \
   loopexit,bestpractice,symbol,this,read-only,return, \
   property,http://localhost:8888/hop/js2http, \
   scheme
% ./a.out

js2http/js2http.js

var hopc = require("hopc");
var ast = require(hopc.ast);

service js2http({ ast, config }) {
   var prg = hopc.intern(ast);
   var w = new hopc.HopcAstWalker();

   w.J2SDeclFun = function(node) {
      console.log("scanning function ", node.id.__symbol__);
      w.walk(node);
      return node;
   }

   w.J2SDeclSvc = function(node) {
      console.log("scanning service ", node.id.__symbol__);
      w.walk(node);
      return node;
   }

   w.J2SBinary = function(node) {
      w.walk(node);
      if (node.op.__symbol__ === "*") {
         console.log('transforming operator "*" into "+"');
         node.op = { __symbol__: '+' };
      }
      return node;
   }

   w.J2SAccess = function(node) {
      console.log('profiling access');

      var sym = { __symbol__: "console" };
      var con = new ast.J2SUnresolvedRef(node.loc, undefined, false, sym);
      var log = new ast.J2SString(node.loc, undefined, "log", false);
      var access = new ast.J2SAccess(node.loc, undefined, false, con, log)
      var name = new ast.J2SString(node.loc, undefined, accessName(node.obj), false);
      var call = new ast.J2SCall(node.loc, undefined, access, [ name ]);
      
      var exprs = [ call, w.walk(node) ];

      return new ast.J2SSequence(node.loc, undefined, exprs);
   }

   w.walk(prg);

   return prg;
}

function accessName(node) {
   if (node.__node__ == "J2SUnresolvedRef") {
      return node.id;
   } else if (node.__node__ == "J2SRef") {
      return node.decl.id.__symbol__;
   } else {
      return "???";
   }
}