Supporting new syntax for refactoring

Audience: rope developers

Area: patchedast, extract refactoring, restructuring

patchedast is a crucial part of rope refactoring. This is where rope annotates the AST (abstract syntax tree) generated by the ast module with location and ordering information so that it can generate the refactored code.

Among others, these refactorings depend heavily on the information generated by patchedast:

• extract method/variable
• restructuring

For example, suppose that rope had not yet supported the global keyword, and your want to add support for this syntax to rope. The global keyword is used inside functions to declare that the function mutates a global variable:

myvar = 1
def foo():
    global myvar
    myvar += 1

Step one, you'll need to figure out the ast node that Python uses to represent this syntax. Explore what the AST for that syntax looks like in the REPL:

>>> import ast
>>> ast.dump(ast.parse(code))

Step two, write a test in patchedasttest.py, for example for global keyword, this should look like this:

def test_global_node(self):
    source = "global a, b\n"
    ast_frag = patchedast.get_patched_ast(source, True)
    checker = _ResultChecker(self, ast_frag)
    checker.check_region("Global", 0, len(source) - 1)
    checker.check_children("Global", ["global", " ", "a", "", ",", " ", "b"])

If you're not quite sure what the checker.check_children() should look like, it may often be easier to write the implementation for patchedast.py first, then verify that the output has the right shape manually, then copy it to checker.check_children().

Third step, simply add a node handler with the new AST node's name to _PatchingASTWalker. In this case, we're adding support for ast.Global, so we will create a new function named _PatchingASTWalker._Global.

class _PatchingASTWalker:
    ...
    def _Global(self, node):
        children = ["global", *self._child_nodes(node.names, ",")]
        self._handle(node, children)

You don't have to call this function yourself, _PatchingASTWalker.__call__ will call the appropriate handler functions when it encounters the node with that name.

Let's dissect the _PatchingASTWalker._Global a little bit. Here, we are simply creating a sorted list of children of the ast.Global node:

children = ["global", "a", "b"]

children can contain either strings, which is atomic, or other ast node classes. For example, in _PatchingASTWalker._Dict, for the dictionary syntax, we have this:

children.extend([key, ":", value])

key and value are itself can be complex expressions, and they can have arbitrarily complex ast node, so they can't just be simple strings. In this case, we just added the ast nodes themselves to the children:

children.extend([<ast.Constant>, ":", <ast.Number>])

At the end of a node handler, you have to call self._handle(node, children) so that _PatchingASTWalker would recursively annotate any ast node in children that aren't simple strings.

This children will eventually be set to the node's sorted_children.

What we're constructing here is basically just a list of string or other ast nodes, in the order the they would appear in the source code.