dt-python-parser

[![NPM version][npm-image]][npm-url]

[npm-image]: https://img.shields.io/npm/v/dt-python-parser.svg?style=flat-square
[npm-url]: https://www.npmjs.com/package/dt-python-parser

English | 简体中文

dt-python-parser is a Python Parser project based on ANTLR4 for the big data field. Through ANTLR4 the default generated Parser, Visitor and Listener objects, we can easily achieve the Syntax Validation (Syntax Validation), of Python statements Lexical analysis (Tokenizer), traverse AST nodes and other functions. In addition, several auxiliary methods are provided, for example, to filter comments of type # and """ in Python statements.

Supported Python syntax version:

- Python2
- Python3

> Tip: The current Parser is the Javascript language version, if necessary, you can try to compile the Grammar file to other target languages

Installation

``bash
// use npm
npm i dt-python-parser --save

// use yarn
yarn add dt-python-parser
`

Use

$3

First, you need to declare the corresponding Parser object. Different Python syntax versions need to introduce different Parser object processing. For example, if it is for Python2, you need to introduce Python2 Parser separately, here we use Python3 As an example:

`javascript
import {Python3Parser} from'dt-python-parser';

const parser = new Python3Parser();

const correctPython = print('abc')\nprint('abc')\n;
const errors = parser.validate(correctPython);
console.log(errors);
`

Output:

`javascript
/*
[]
*/
`

Example of verification failure:

`javascript
const incorrectPython =
'! a = 10\nif a> 5:\n print("a bigger than 5")\nelse:\n print("a smaller than 5")';
const errors = parser.validate(incorrectPython);
console.log(errors);
`

Output:

`javascript
/*
[
{
startLine: 1,
endLine: 1,
startCol: 0,
endCol: 1,
message: "extraneous input'!' expecting {,'def','return','raise','from','import','import','global','nonlocal','assert', 'if','while','for','try','with','lambda','not','None','True','False','class','yield','yield ','del','pass','continue','break','break', NEWLINE, NAME, STRING_LITERAL, BYTES_LITERAL, DECIMAL_INTEGER, OCT_INTEGER, HEX_INTEGER, BIN_INTEGER, FLOAT_NUMBER, IMAG_NUMBER, DECIMAL_INTEGER, HCTEXGER, HCTEXGER, FLOAT_NUMBER, IMAG_NUMBER,'...','*','(','[','+','-','~','{','@'}"
}
]
*/
`

First instantiate the Parser object, and then use the validate method to verify the Python statement. If the verification fails, an array containing the error information is returned.

$3

In necessary scenarios, you can perform lexical analysis on Python sentences separately to obtain all Tokens objects:

`javascript
import {Python3Parser} from'dt-python-parser';

const parser = new Python3Parser();
const python ='for i in range(5):\n print(i)';
const tokens = parser.getAllTokens(python);
console.log(tokens);

/*
[
CommonToken {
source: [[Python3Lexer], [InputStream] ],
type: 14,
channel: 0,
start: 0,
stop: 2,
tokenIndex: -1,
line: 1,
column: 0,
_text: null
},
...
]
*/
`

$3

Use Visitor mode to visit the specified node in the AST

`javascript
import {Python3Parser, Python3Visitor} from'dt-python-parser';

const parser = new Python3Parser();
const python = import sys\nfor i in sys.argv:\n print(i);
// parseTree
const tree = parser.parse(python);
class MyVisitor extends Python3Visitor {
// Override visitImport_name method
visitImport_name(ctx): void {
let importName = ctx
.getText()
.toLowerCase()
.match(/(?<=import).+/)?.[0];
console.log('ImportName', importName);
}
}
const visitor = new MyVisitor();
visitor.visit(tree);

/*
ImportName sys
*/
`

> Tip: When using Visitor mode, the method name of the node can be found in the Visitor file in the corresponding Python directory

$3

In Listener mode, use the ParseTreeWalker object provided by ANTLR4 to traverse the AST and call the corresponding method when entering each node.

`javascript
import {Python3Parser, Python3Listener} from'dt-python-parser';

const parser = new Python3Parser();
const python ='import sys\nfor i in sys.argv:\n print(i)';
// parseTree
const tree = parser.parse(python);
class MyListener extends Python3Listener {
enterImport_name(ctx): void {
let importName = ctx
.getText()
.toLowerCase()
.match(/(?<=import).+/)?.[0];
console.log('ImportName', importName);
}
}
const listenTableName = new MyListener();
parser.listen(listenTableName, tree);

/*
ImportName sys
*/
`

> Tip: When using Listener mode, the method name of the node can be found in the Listener file in the corresponding Python directory

$3

Clear comments and leading and trailing spaces

`javascript
import {cleanPython} from'dt-python-parser';

const python = #it is for test\nfor i in range(5):\n print(i);
const cleanedPython = cleanPython(python);
console.log(cleanedPython);

/*
for i in range(5):
print(i)
*/
`

$3

Get comments of type # or """

`javascript
import { lexer } from 'dt-python-parser';

const python = """it is for test"""\nvar1 = "Hello World!"\nfor i in range(5):\n print(i);
const commentTokens = lexer(python);
console.log(commentTokens);

/*
[
{
type: 'Comment',
value: '"""it is for test"""',
start: 0,
lineNumber: 1,
end: 20
}
]
*/
`

$3

- parserTreeToString (input: string):
Parse Python into List-like` style tree string, generally used for testing

Roadmap

- Auto-complete
- Code formatting
- Grammar structure optimization
- Execution efficiency optimization

License

MIT