shift-refactor

Shift Refactor
--------------

shift-refactor is a suite of utility functions designed to analyze and modify JavaScript source files.

It originated as a tool to reverse engineer obfuscated JavaScript but is general-purpose enough for arbitrary transformations.

$3

Anyone who works with JavaScript ASTs (Abstract Syntax Trees). If you're not familiar with ASTs, here are a few use cases where they come in useful:

- Automatic refactoring, making sweeping changes to JavaScript source files (Developers, QA).
- Analyzing JavaScript for linting, complexity scoring, etc (Developers, QA).
- Extracting API details to auto-generate documentation or tests (Developers, QA).
- Scraping JavaScript for information or security vulnerabilities (Pen Testers, QA, Security Teams, Hacker types).
- Programmatically transforming malicious or obfuscated JavaScript (Reverse Engineers).

Status

Stable.

Installation

``
$ npm install shift-refactor
`

Usage

The script below finds and prints all literal strings in a script.

`js
// Read 'example.js' as text
const fs = require('fs');
const src = fs.readFileSync('example.js', 'utf-8');

const { refactor } = require('.');

// Create a refactor query object
const $script = refactor(src);

// Select all LiteralStringExpressions
const $stringNodes = $script('LiteralStringExpression')

// Turn the string AST nodes into real JS strings
const strings = $stringNodes.codegen();

// Output the strings to the console
strings.forEach(string => console.log(string));

`

$3

This script takes the obfuscated source and turns it into something much more readable.

`js
const { refactor } = require('.'); // require('shift-refactor');
const Shift = require('shift-ast');

// Obfuscated source
const src = var a=['\x74\x61\x72\x67\x65\x74','\x73\x65\x74\x54\x61\x72\x67\x65\x74','\x77\x6f\x72\x6c\x64','\x67\x72\x65\x65\x74','\x72\x65\x61\x64\x65\x72'];var b=function(c,d){c=c-0x0;var e=a[c];return e;};(function(){class c{constructor(d){this[b('0x0')]=d;}['\x67\x72\x65\x65\x74'](){console'\x6c\x6f\x67']);}b('0x1'){this['\x74\x61\x72\x67\x65\x74']=e;}}const f=new c(b('0x2'));f[b('0x3')]();fb('0x1'));f[b('0x3')]();}());;

const $script = refactor(src);

const strings = $script(Script > :first-child ArrayExpression > .elements);

const destringifyDeclarator = $script(VariableDeclarator[binding.name="b"][init.params.items.length=2]);

destringifyDeclarator.rename('destringify');

const destringifyOffset = destringifyDeclarator.$(BinaryExpression > LiteralNumericExpression);

const findIndex = (c, d) => c - destringifyOffset.first().value;

$script(CallExpression[callee.name="destringify"]).replace(
node => {
return new Shift.LiteralStringExpression({
value: strings.get(findIndex(node.arguments[0].value)).value
})
}
)

$script([binding.name="a"]).delete();
$script([binding.name="destringify"]).delete();

$script.convertComputedToStatic();

console.log($script.print());
`

Query Syntax

The query syntax is from shift-query (which is a port of esquery) and closely resemble CSS selector syntax.

The following selectors are supported:
* AST node type: FunctionDeclaration
wildcard:
* attribute existence: [attr]
* attribute value: [attr="foo"] or [attr=123]
attribute regex: [attr=/foo./]
* attribute conditons: [attr!="foo"], [attr>2], [attr<3], [attr>=2], or [attr<=3]
* nested attribute: [attr.level2="foo"]
* field: FunctionDeclaration > IdentifierExpression.name
* First or last child: :first-child or :last-child
* nth-child (no ax+b support): :nth-child(2)
* nth-last-child (no ax+b support): :nth-last-child(1)
* descendant: ancestor descendant
* child: parent > child
* following sibling: node ~ sibling
* adjacent sibling: node + adjacent
* negation: :not(ExpressionStatement)
* matches-any: :matches([attr] > :first-child, :last-child)
* subject indicator: !IfStatement > [name="foo"]
* class of AST node: :statement, :expression, :declaration, :function, or :target

Useful sites & tools

- Shift-query's online sandbox to test queries quickly.
- Shift-query CLI tool to query JavaScript on the command line.
- AST Explorer to explore JavaScript AST's visually (make sure to select "shift" on the top menu bar).
- Shift-AST.org - home of the Shift JavaScript tool suite.

API

$3

Create a refactor query object.

#### Note:

This function assumes that it is being passed complete JavaScript source or a root AST node (Script or Module) so that it can create and maintain global state.

#### Example

`js
const { refactor } = require('shift-refactor');

const $script = refactor(/ JavaScript Source \/);
`

Refactor Query Object

The API is meant to look and feel like jQuery since – like jQuery – it works with CSS-style queries and regularly accesses nodes on a tree. Each query object is both a function and an instance of the internal RefactorSession class.

Calling the query object as a function will produce a new query object, You can call a refactor query with a query to produce a new query object with the new nodes or you can call methods off the object to act on the nodes already selected. The examples prefix refactor query objects with a $ to indicate they are refactor query objects and not naked Nodes or other objects.

$3

`js
const { refactor } = require('shift-refactor');

const $script = refactor(src);
const $variableDecls = $script('VariableDeclarationStatement')
const $bindingIdentifiers = $variableDecls('BindingIdentifier');
const names = $bindingIdentifiers.map(node => node.name);
`

$3

- .$(queryOrNodes)
- .append(replacer)
- .closest(closestSelector)
- .codegen()
- .declarations()
- .delete()
- .filter(iterator)
- .find(iterator)
- .findMatchingExpression(sampleSrc)
- .findMatchingStatement(sampleSrc)
- .findOne(selectorOrNode)
- .first(selector)
- .forEach(iterator)
- .get(index)
- .logOut()
- .lookupVariable()
- .lookupVariableByName(name)
- .map(iterator)
- .nameString()
- .parents()
- .prepend(replacer)
- .print()
- .query(selector)
- .raw()
- .references()
- .rename(newName)
- .replace(replacer)
- .replaceAsync(replacer)
- .replaceChildren(query, replacer)
- .statements()
- .toJSON()
- .type()

#### .$(queryOrNodes)

Sub-query from selected nodes

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
let a = 1;
function myFunction() {
let b = 2, c = 3;
}

$script = refactor(src);

const funcDecl = $script('FunctionDeclaration[name.name="myFunction"]');
const innerIdentifiers = funcDecl.$('BindingIdentifier');
// innerIdentifiers.nodes: myFunction, b, c (note: does not include a)
`

#### .append(replacer)

Inserts the result ofreplacerafter the selected statement.

#### Note:

Only works on Statement nodes.

#### Example

`js
const { refactor } = require('shift-refactor');
const Shift = require('shift-ast');

const src =
var message = "Hello";
console.log(message);

$script = refactor(src);

$script('LiteralStringExpression[value="Hello"]').closest(':statement').append('debugger');
`

#### .closest(closestSelector)

Finds the closest parent node that matches the passed selector.

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
function someFunction() {
interestingFunction();
}
function otherFunction() {
interestingFunction();
}

$script = refactor(src);

// finds all functions that call interestingFunction
const fnDecls = $script('CallExpression[callee.name="interestingFunction"]').closest('FunctionDeclaration');
`

#### .codegen()

Generates JavaScript source for the first selected node.

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
for (var i=1; i < 101; i++){
if (i % 15 == 0) console.log("FizzBuzz");
else if (i % 3 == 0) console.log("Fizz");
else if (i % 5 == 0) console.log("Buzz");
else console.log(i);
}

$script = refactor(src);

const strings = $script("LiteralStringExpression")

console.log(strings.codegen());
`

#### .declarations()

Finds the declaration for the selected Identifier nodes.

#### Note:

Returns a list of Declaration objects for each selected node, not a shift-refactor query object.

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
const myVariable = 2, otherVar = 3;
console.log(myVariable, otherVar);

$script = refactor(src);

// selects the parameters to console.log() and finds their declarations
const decls = $script('CallExpression[callee.object.name="console"][callee.property="log"] > .arguments').declarations();
`

#### .delete()

Delete nodes

#### Example

`js
const { refactor } = require('shift-refactor');

$script = refactor('foo();bar();');

$script('ExpressionStatement[expression.callee.name="foo"]').delete();
`

#### .filter(iterator)

Filter selected nodes via passed iterator

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
let doc = window.document;
function addListener(event, fn) {
doc.addEventListener(event, fn);
}

$script = refactor(src);

const values = $script('BindingIdentifier').filter(node => node.name === 'doc');
`

#### .find(iterator)

Finds node via the passed iterator iterator

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
const myMessage = "He" + "llo" + " " + "World";

$script = refactor(src);

$script('LiteralStringExpression')
.find(node => node.value === 'World')
.replace('"Reader"');
`

#### .findMatchingExpression(sampleSrc)

Finds an expression that closely matches the passed source.

#### Note:

Used for selecting nodes by source pattern instead of query. The passed source is parsed as a Script and the first statement is expected to be an ExpressionStatement.Matching is done by matching the properties of the parsed statement, ignoring additional properties/nodes in the source tree.

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
const a = someFunction(paramOther);
const b = targetFunction(param1, param2);

$script = refactor(src);

const targetCallExpression = $script.findMatchingExpression('targetFunction(param1, param2)');
`

#### .findMatchingStatement(sampleSrc)

Finds a statement that matches the passed source.

#### Note:

Used for selecting nodes by source pattern vs query. The passed source is parsed as a Script and the first statement alone is used as the statement to match. Matching is done by matching the properties of the parsed statement, ignoring additional properties/nodes in the source tree.

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
function someFunction(a,b) {
var innerVariable = "Lots of stuff in here";
foo(a);
bar(b);
}

$script = refactor(src);

const targetDeclaration = $script.findMatchingStatement('function someFunction(a,b){}');
`

#### .findOne(selectorOrNode)

Finds and selects a single node, throwing an error if zero or more than one is found.

#### Note:

This is useful for when you want to target a single node but aren't sure how specific your query needs to be to target that node and only that node.

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
let outerVariable = 1;
function someFunction(a,b) {
let innerVariable = 2;
}

$script = refactor(src);

// This would throw, because there are multiple VariableDeclarators
// $script.findOne('VariableDeclarator');

// This won't throw because there is only one within the only FunctionDeclaration.
const innerVariableDecl = $script('FunctionDeclaration').findOne('VariableDeclarator');
`

#### .first(selector)

Returns the first selected node. Optionally takes a selector and returns the first node that matches the selector.

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
func1();
func2();
func3();

$script = refactor(src);

const func1CallExpression = $script('CallExpression').first();
`

#### .forEach(iterator)

Iterate over selected nodes

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
let a = [1,2,3,4];

$script = refactor(src);

$script('LiteralNumericExpression').forEach(node => node.value *= 2);
`

#### .get(index)

Get selected node at index.

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
someFunction('first string', 'second string', 'third string');

$script = refactor(src);

const thirdString = $script('LiteralStringExpression').get(2);
`

#### .logOut()

console.log()s the selected nodes. Useful for inserting into a chain to see what nodes you are working with.

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
let a = 1, b = 2;

$script = refactor(src);

$script("VariableDeclarator").logOut().delete();
`

#### .lookupVariable()

Looks up the Variable from the passed identifier node

#### Note:

ReturnsVariableobjects from shift-scope, that contain all the references and declarations for a program variable.

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
const someVariable = 2, other = 3;
someVariable++;
function thisIsAVariabletoo(same, as, these) {}

$script = refactor(src);

// Finds all variables declared within a program
const variables = $script('BindingIdentifier').lookupVariable();
`

#### .lookupVariableByName(name)

Looks up Variables by name.

#### Note:

There may be multiple across a program. Variable lookup operates on the global program state. This method ignores selected nodes.

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
const someVariable = 2, other = 3;

$script = refactor(src);

const variables = $script.lookupVariableByName('someVariable');
`

#### .map(iterator)

Transform selected nodes via passed iterator

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
let doc = window.document;
function addListener(event, fn) {
doc.addEventListener(event, fn);
}

$script = refactor(src);

const values = $script('BindingIdentifier').map(node => node.name);
`

#### .nameString()

Retrieve the names of the first selected node. Returns undefined for nodes without names.

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
var first = 1, second = 2;

$script = refactor(src);
const firstName = $script('BindingIdentifier[name="first"]').nameString();
`

#### .parents()

Retrieve parent node(s)

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
var a = 1, b = 2;

$script = refactor(src);
const declarators = $script('VariableDeclarator');
const declaration = declarators.parents();
`

#### .prepend(replacer)

Inserts the result ofreplacerbefore the selected statement.

#### Note:

Only works on Statement nodes.

#### Example

`js
const { refactor } = require('shift-refactor');
const Shift = require('shift-ast');

const src =
var message = "Hello";
console.log(message);

$script = refactor(src);

$script('ExpressionStatement[expression.type="CallExpression"]').prepend(new Shift.DebuggerStatement());
`

#### .print()

Generates JavaScript source for the first selected node.

#### Example

`js
const { refactor } = require('shift-refactor');
const Shift = require('shift-ast');

const src =
window.addEventListener('load', () => {
lotsOfWork();
})

$script = refactor(src);

$script("CallExpression[callee.property='addEventListener'] > ArrowExpression")
.replace(new Shift.IdentifierExpression({name: 'myListener'}));

console.log($script.print());
`

#### .query(selector)

Sub-query from selected nodes

#### Note:

synonym for .$()

#### .raw()

Returns the raw Shift node for the first selected node.

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
const a = 2;

$script = refactor(src);

const declStatement = $script('VariableDeclarationStatement').raw();
`

#### .references()

Finds the references for the selected Identifier nodes.

#### Note:

Returns a list of Reference objects for each selected node, not a shift-refactor query object.

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
let myVar = 1;
function someFunction(a,b) {
myVar++;
return myVar;
}

$script = refactor(src);

const refs = $script('BindingIdentifier[name="myVar"]').references();
`

#### .rename(newName)

Rename all references to the first selected node to the passed name.

#### Note:

Uses the selected node as the target, but affects the global state.

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
const myVariable = 2;
myVariable++;
const other = myVariable;
function unrelated(myVariable) { return myVariable }

$script = refactor(src);

$script('VariableDeclarator[binding.name="myVariable"]').rename('newName');
`

#### .replace(replacer)

Replace selected node with the result of the replacer parameter

#### Example

`js
const { refactor } = require('shift-refactor');
const Shift = require('shift-ast');

const src =
function sum(a,b) { return a + b }
function difference(a,b) {return a - b}

$script = refactor(src);

$script('FunctionDeclaration').replace(node => new Shift.VariableDeclarationStatement({
declaration: new Shift.VariableDeclaration({
kind: 'const',
declarators: [
new Shift.VariableDeclarator({
binding: node.name,
init: new Shift.ArrowExpression({
isAsync: false,
params: node.params,
body: node.body
})
})
]
})
}))
`

#### .replaceAsync(replacer)

Async version of .replace() that supports asynchronous replacer functions

#### Example

`js
const { refactor } = require('shift-refactor');

$script = refactor('var a = "hello";');

async function work() {
await $script('LiteralStringExpression').replaceAsync(
(node) => Promise.resolve("goodbye")
)
}
`

#### .replaceChildren(query, replacer)

Recursively replaces child nodes until no nodes have been replaced.

#### Example

`js
const { refactor } = require('shift-refactor');
const Shift = require('shift-ast');

const src =
1 + 2 + 3

$script = refactor(src);

$script.replaceChildren(
'BinaryExpression[left.type=LiteralNumericExpression][right.type=LiteralNumericExpression]',
(node) => new Shift.LiteralNumericExpression({value: node.left.value + node.right.value})
);
`

#### .statements()

Returns the selects the statements for the selected nodes. Note: it will "uplevel" the inner statements of nodes with a.bodyproperty.Does nothing for nodes that have no statements property.

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
console.log(1);
console.log(2);

$script = refactor(src);

const rootStatements = $script.statements();
`

#### .toJSON()

JSON-ifies the current selected nodes.

#### Example

`js
const { refactor } = require('shift-refactor');

const src =
(function(){ console.log("Hey")}())

$script = refactor(src);

const json = $script.toJSON();
`

#### .type()

Return the type of the first selected node

#### Example

`js
const { refactor } = require('shift-refactor');
const Shift = require('shift-ast');

const src =
myFunction();

$script = refactor(src);

const type = $script('CallExpression').type();
``