Pyparsing: Parsing nested plaintext data with a key=value

This should help you out. See comments for details.

from pyparsing import *

test_string ='''
object1 {
        key1 = value1
        key2
        #key3 = value3
        key4
        #key5 = value5
        key6 = value6
        subobject1 {
            key1 = value1
            key2 = value2
            key3 = value3
        }
}'''

# interpret inline 'string' as Suppress('string'), 
# instead of LBRACE,EQ,RBRACE,HASH = map(Suppress, '{=}#')
ParserElement.inlineLiteralsUsing(Suppress)  

# be sure to exclude special characters when using printables
name_expr = Word(printables, excludeChars='{}')
key_val_expr = '=' + Word(printables)

# p1('name') is equivalent to p1.setResultsName('name')
# p1 | p2 is equivalent to MatchFirst(p1, p2)
# if lineEnd() matches first, there is no value. 
# then use a parse action to return the string 'NONE' as value instead
# else, match a regular key_value
# also, you have to use Group because key_val_line is a repeating element
key_val_line = Group(name_expr('key') + (lineEnd().setParseAction(lambda t: 'NONE') | key_val_expr)('val'))
key_val_lines = OneOrMore(key_val_line)('key_val_lines')

obj = Forward()
obj << Group(name_expr('obj_name') + '{' + OneOrMore(key_val_lines | obj) + '}')('objects')

parse_results = obj.parseString(test_string)
print(parse_results.dump())

This prints the following:

[['object1', ['key1', 'value1'], ['key2', 'NONE'], ['#key3', 'value3'], ['key4', 'NONE'], ['#key5', 'value5'], ['key6', 'value6'], ['subobject1', ['key1', 'value1'], ['key2', 'value2'], ['key3', 'value3']]]]
- objects: ['object1', ['key1', 'value1'], ['key2', 'NONE'], ['#key3', 'value3'], ['key4', 'NONE'], ['#key5', 'value5'], ['key6', 'value6'], ['subobject1', ['key1', 'value1'], ['key2', 'value2'], ['key3', 'value3']]]
  - key_val_lines: [['key1', 'value1'], ['key2', 'NONE'], ['#key3', 'value3'], ['key4', 'NONE'], ['#key5', 'value5'], ['key6', 'value6']]
    [0]:
      ['key1', 'value1']
      - key: 'key1'
      - val: 'value1'
    [1]:
      ['key2', 'NONE']
      - key: 'key2'
      - val: 'NONE'
    [2]:
      ['#key3', 'value3']
      - key: '#key3'
      - val: 'value3'
    [3]:
      ['key4', 'NONE']
      - key: 'key4'
      - val: 'NONE'
    [4]:
      ['#key5', 'value5']
      - key: '#key5'
      - val: 'value5'
    [5]:
      ['key6', 'value6']
      - key: 'key6'
      - val: 'value6'
  - obj_name: 'object1'
  - objects: ['subobject1', ['key1', 'value1'], ['key2', 'value2'], ['key3', 'value3']]
    - key_val_lines: [['key1', 'value1'], ['key2', 'value2'], ['key3', 'value3']]
      [0]:
        ['key1', 'value1']
        - key: 'key1'
        - val: 'value1'
      [1]:
        ['key2', 'value2']
        - key: 'key2'
        - val: 'value2'
      [2]:
        ['key3', 'value3']
        - key: 'key3'
        - val: 'value3'
    - obj_name: 'subobject1'

Recursive parsers are not an easy first start with pyparsing, and your optional bits make things more complicated too. I think this code mostly does what you want - hopefully it will be more meaningful to you now that you have done some of your own wrestling with pyparsing thus far:

import pyparsing as pp

LBRACE, RBRACE, EQ = map(pp.Suppress, "{}=")
# convert parsed '#' to a bool that you can test on
disabled_marker = pp.Literal("#").addParseAction(lambda: True)
identifier = pp.pyparsing_common.identifier
key = identifier()

# try to parse a numeric value first, might be interesting
# pyparsing_common.number will auto-convert string to float or int at parse time,
# so you won't have to detect and do the conversion later
value = pp.pyparsing_common.number | pp.Word(pp.printables)

obj_item = pp.Forward()
obj_expr = pp.Group(identifier("name")
                    + pp.Group(LBRACE
                               + pp.ZeroOrMore(obj_item)
                               + RBRACE)("attributes"))

key_with_value = pp.Group(pp.Optional(disabled_marker)("disabled")
                          + key("key") + EQ + value("value"))
# use empty() to inject a None for the value
key_without_value = pp.Group(pp.Optional(disabled_marker)("disabled")
                             + key("key") 
                             + pp.empty().addParseAction(lambda: [None])("value"))

# now define an item that can be used in an object - this order is important!
obj_item <<= obj_expr | key_with_value | key_without_value

To parse your string2 input:

zz = obj_expr.parseString(string2)
print(zz[0].dump())

Gives:

['object1', [['key1', 'value1'], ['key2', None], [True, 'key3', 'value3'], ['key4', 'value4'], ['subobject1', [['key1', 'value1'], ['key2', 'value2'], ['key3', 'value3']]], [True, 'key5', 'value5'], ['key6', 'v_a_l_u_e_6'], ['subobject2', [['key1', 'value1']]], ['key7', 'value7'], ['key8', None]]]
- attributes: [['key1', 'value1'], ['key2', None], [True, 'key3', 'value3'], ['key4', 'value4'], ['subobject1', [['key1', 'value1'], ['key2', 'value2'], ['key3', 'value3']]], [True, 'key5', 'value5'], ['key6', 'v_a_l_u_e_6'], ['subobject2', [['key1', 'value1']]], ['key7', 'value7'], ['key8', None]]
  [0]:
    ['key1', 'value1']
    - key: 'key1'
    - value: 'value1'
  [1]:
    ['key2', None]
    - key: 'key2'
    - value: None
  [2]:
    [True, 'key3', 'value3']
    - disabled: True
    - key: 'key3'
    - value: 'value3'
  [3]:
    ['key4', 'value4']
    - key: 'key4'
    - value: 'value4'
  [4]:
    ['subobject1', [['key1', 'value1'], ['key2', 'value2'], ['key3', 'value3']]]
    - attributes: [['key1', 'value1'], ['key2', 'value2'], ['key3', 'value3']]
      [0]:
        ['key1', 'value1']
        - key: 'key1'
        - value: 'value1'
      [1]:
        ['key2', 'value2']
        - key: 'key2'
        - value: 'value2'
      [2]:
        ['key3', 'value3']
        - key: 'key3'
        - value: 'value3'
    - name: 'subobject1'
  [5]:
    [True, 'key5', 'value5']
    - disabled: True
    - key: 'key5'
    - value: 'value5'
  [6]:
    ['key6', 'v_a_l_u_e_6']
    - key: 'key6'
    - value: 'v_a_l_u_e_6'
  [7]:
    ['subobject2', [['key1', 'value1']]]
    - attributes: [['key1', 'value1']]
      [0]:
        ['key1', 'value1']
        - key: 'key1'
        - value: 'value1'
    - name: 'subobject2'
  [8]:
    ['key7', 'value7']
    - key: 'key7'
    - value: 'value7'
  [9]:
    ['key8', None]
    - key: 'key8'
    - value: None
- name: 'object1'

EDIT: I removed the Dict constructs, as they actually make the output more difficult to process.

@Jeppi's answer has some excellent suggestions. I would add:

• Word(printables) is always a risky construct, since it will match as much non-whitespace as there is. For instance, if a line contained "color=red" with no spaces, then it would be interpreted as a key "color=red" with no value. You would be better to define key with something like Word(alphanums) or Word(alphas, alphanums+"_"). To allow for the possible leading '#', use Word(alphas+'#', alphanums+"_").

• Your idea about conditionalizing the presence of '#' with if LineStart() == HASH is interesting, but not how pyparsing works. At this point in the code, you are still building the parser itself, which happens separate from any input text. The actual determination of whether a particular line starts with '#' happens during parsing, which is done later when your code calls collection.parseString. That is, you build up all the parser bits, and then point them at the source text. Any "if character X is present" logic needs to be represented using some alternation or optional construct in the parser itself, not with Python if-then code.

• Consider using pyparsing's Optional class for elements that may or may not be present. This applies to the possible key-value without a value, and could also be another way to handle the possible leading '#' character in key names.