Approach for parsing file and creating dynamic data structure for use by another program

1. Create a sqlite database to represent the data in the properties file.

2. Write a script to dump the data from the properties file into the sqlite database.

3. Write a script that can regenerate the properties file based on the data in the sqlite database. New hosts can now be added by inserting rows into the sqlite database. Every time an update or insert is done, run this script to keep the old scripts happy, until step 5 is completed.

4. Have all new scripts interact with the sqlite database instead of the properties file.

5. Rewrite the old scripts to use the sqlite database. For example, looping through results from sql queries instead of adding more if statements.

Since your property names have some internal structure to them, you might try using pyparsing to process your properties file:

from pyparsing import Word, alphas, alphanums, delimitedList, restOfLine

props = """
projectName.ClusterNameServer1.sslport=443
projectName.ClusterNameServer1.port=80
projectName.ClusterNameServer1.host=myHostA
projectName.ClusterNameServer2.sslport=443
projectName.ClusterNameServer2.port=80
projectName.ClusterNameServer2.host=myHostB
"""

# define format of a single property definition line
ident = Word(alphas, alphanums+'_')
propertyName = delimitedList(ident,'.')
propertyDefn = propertyName("name") + '=' + restOfLine("value")

# sample code that parses the properties and accesses the
# name and value fields
for prop in propertyDefn.searchString(props):
    print '.'.join(prop.name), '->', prop.value

Prints:

projectName.ClusterNameServer1.sslport -> 443
projectName.ClusterNameServer1.port -> 80
projectName.ClusterNameServer1.host -> myHostA
projectName.ClusterNameServer2.sslport -> 443
projectName.ClusterNameServer2.port -> 80
projectName.ClusterNameServer2.host -> myHostB

If you can rely on the project-server-parameter structure of the properties, you can build a hierarchical object using defaultdicts (from the collections module):

# use defaultdicts to accumulate these values into a nested structure
from collections import defaultdict
properties = defaultdict(lambda:defaultdict(dict))

# build hierarchical property structure
# (assumes very rigid project-server-parameter naming)
for prop in propertyDefn.searchString(props):
    properties[prop.name[0]][prop.name[1]][prop.name[2]] = prop.value

# show data in nice outline form
for project in properties:
    print '-', project
    for cluster in properties[project]:
        print '  -', cluster
        for param in properties[project][cluster]:
            print '    -', param, properties[project][cluster][param]

Prints:

- projectName
  - ClusterNameServer1
    - host myHostA
    - sslport 443
    - port 80
  - ClusterNameServer2
    - host myHostB
    - sslport 443
    - port 80

Executing the following code creates a dictionnary whose keys are server's number and values are (sslport,port,host)

import re

RE = ('projectName.ClusterNameServer(\d+)\.sslport=(\d+)\s+'
      'projectName.ClusterNameServer\\1\.port=(\d+)\s+'
      'projectName.ClusterNameServer\\1\.host=(\S+)(?:\s+|\Z)')

propertyfile_name = 'servers.txt'

with open(propertyfile_name) as f:
    dico = dict( (server,(s,p,h)) for (server,s,p,h) in re.findall(RE,f.read()) )

print dico

result

dico== {'1': ('443', '80', 'myHostA'), '2': ('443', '80', 'myHostB')}

Then the dico data can be passed " to the deployment scripts "; I don't know what are your possibilities to pass this data. It may be done as a flat file, or, better, a pickle file.

The simplest solution would be to include the lines of the above code directly into the "deployment scripts" so as they would have the data without the need to receive the data from outside.

. EDIT

Another solution, that I find better:

import csv
from itertools import ifilter

propertyfile_name = 'servers.txt'

with open(propertyfile_name,'rb') as f:
    dico = dict( ifilter(None,csv.reader(f,delimiter='=')) )    
print dico

The ifilter() function is necessary because of blank lines in the file.

result

{'projectName.ClusterNameServer2.host': 'myHostB', 'projectName.ClusterNameServer2.port': '80', 'projectName.ClusterNameServer1.host': 'myHostA', 'projectName.ClusterNameServer1.port': '80', 'projectName.ClusterNameServer2.sslport': '443', 'projectName.ClusterNameServer1.sslport': '443'}

.

EDIT

RE = ('(projectName.ClusterNameServer\d+\.(?:sslport|port|host))=(\S+)(?:\s+|\Z)')

propertyfile_name = 'servers.txt'

with open(propertyfile_name) as f:
    dico = dict( re.findall(RE,f.read()) )

print dico

Same result

If you want result

dico== {'1': ('443', '80', 'myHostA'), '2': ('443', '80', 'myHostB')}

I will do tomorrow