Generating a Java Parser in Python 3 using ANTLR 4

Question

Using the Lexer and the Parser from here:

https://raw.githubusercontent.com/antlr/grammars-v4/master/java/JavaLexer.g4

https://raw.githubusercontent.com/antlr/grammars-v4/master/java/JavaParser.g4

with antlr-4.6 to generate Python3 targets

java -jar ./antlr-4.6-complete.jar -Dlanguage=Python3 ./JavaLexer.g4

java -jar ./antlr-4.6-complete.jar -Dlanguage=Python3 ./JavaParser.g4

However, im unable to run the compilationUnit() method on the generated parser. It errors out saying

ipdb> parser.compilationUnit()

File "/home/sviyer/onmt-fresh/java/JavaParser.py", line 1063, in compilationUnit
    localctx = JavaParser.CompilationUnitContext(self, self._ctx, self.state)
  File "/home/sviyer/.conda/envs/allennlp/lib/python3.6/site-packages/antlr4/error/ErrorStrategy.py", line 223, in sync
    raise InputMismatchException(recognizer)
antlr4.error.Errors.InputMismatchException: None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "TestAntlr.py", line 13, in <module>
    parser.compilationUnit()
  File "/home/sviyer/onmt-fresh/java/JavaParser.py", line 1063, in compilationUnit
    localctx = JavaParser.CompilationUnitContext(self, self._ctx, self.state)
  File "/home/sviyer/.conda/envs/allennlp/lib/python3.6/site-packages/antlr4/error/ErrorStrategy.py", line 126, in reportError
    self.reportInputMismatch(recognizer, e)
  File "/home/sviyer/.conda/envs/allennlp/lib/python3.6/site-packages/antlr4/error/ErrorStrategy.py", line 266, in reportInputMismatch
    + " expecting " + e.getExpectedTokens().toString(recognizer.literalNames, recognizer.symbolicNames)
  File "/home/sviyer/.conda/envs/allennlp/lib/python3.6/site-packages/antlr4/error/ErrorStrategy.py", line 522, in getTokenErrorDisplay
    s = t.text
AttributeError: 'int' object has no attribute 'text'

The Lexer works fine though and the parser parses it. My code is:

stream = antlr4.InputStream(code)

lexer = JavaLexer(stream)

toks = antlr4.CommonTokenStream(lexer)

parser = JavaParser(stream)

Ivan Kochurkin · Accepted Answer · 2017-10-29 14:07:38Z

2

Your code is incorrect. Try this one:

code = open('sample.java', 'r').read()
codeStream = InputStream(code)
lexer = JavaLexer(codeStream)

# First lexing way
tokensStream = CommonTokenStream(lexer)
parser = JavaParser(tokensStream)

# Second lexing way
'''tokens = lexer.getAllTokens()
tokensSource = ListTokenSource(tokens)
tokensStream = CommonTokenStream(tokensSource)
parser = JavaParser(tokensStream)'''

tree = parser.compilationUnit()
print "Tree " + tree.toStringTree(recog=parser);

Also, use the latest stable ANTLR version (4.7).

answered Oct 29, 2017 at 14:07

Ivan Kochurkin

4,5318 gold badges51 silver badges87 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

greenberet123 Over a year ago

This is exactly the same as mine. Whats the difference?

Ivan Kochurkin Over a year ago

Not exactly. You create lexer and parser from the same stream.

greenberet123 Over a year ago

Oh! I didn't notice that! Thanks! I made that change and everything worked!

abulka Over a year ago

I had to make a few small changes to get the sample to run. tokensSource = ListTokenSource.ListTokenSource(tokens) and tree = parser.compilation_unit() and for Python 3 needed print("Tree", tree.toStringTree(recog=parser))

Collectives™ on Stack Overflow

Generating a Java Parser in Python 3 using ANTLR 4

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related