Error in looping through a text file in python

The cause of your error is that you are misusing the file object genrator by calling next on it twice as often as you think. Each call to next gets a line and returns it. Therefore, while '---' not in next(file): fields1 = next(file).split(',') gets a line, checks it for ---, then gets another line and tries to parse it. This means that you are able to skip a line containing the --- by having it come up in the second next. In that case you will get to the end of the file before you find the line you are looking for. StopIteration is how iterators normally indicate that their input has been exhausted.

There are a couple of other issues you may want to address in your code:

1. Using next on a generator like a file when you are already inside a for loop may cause undefined behavior. You may be getting away with it this time, but it is not good practice in general. The main reason you are getting away with it, by the way, is possibly that you never actually return control to the for loop once the while is triggered, and not that files are particularly permissive in this regard.
2. The inner with that dumps your data to a file is inside your while loop. That means that the file you open with 'w' permissions will get truncated for every iteration of the while (i.e., each line in the file). As the array grows, the output will actually appear fine, but you probably want to move that out of the inner loop.

The simplest solution would be to rewrite the code in two loops: one to find the start of the part you care about, and the other to process it until the end is found.

Something like this:

test_arr = []
with open(fileName, 'r') as file:
    for line in file:
        if line.startswith('---') and 'section1' in line:
            break

    for line in file:
        if '---' in line:
            break
        fields1 = line.split(',')
        for item in fields1:
            testarr.append({config.get('test','test'): item})

with open(test_dir,'w') as test_file:
    json.dump(testarr, test_file)

EDIT:

Taking @tripleee's advice, I have removed the regex check for the start line. While regex gives great precision and flexibility for finding a specific pattern, it is really overkill for this example. I would like to point out that if you are looking for a section other than section1, or if section1 appears after some other lines with dashes, you will absolutely need this two-loop approach. The one-loop solutions in the other answers will not work in a non-trivial case.

Looks like you are overcomplicating matters massively. The next inside the inner while loop I imagine is tripping up the outer for loop, but that's just unnecessary anyway. You are already looping over lines; pick the ones you want, then quit when you're done.

with open(fileName,'r') as inputfile:
    for line in inputfile:
        if line.startswith('-') and 'section1' in line:
            continue
        elif line.startswith('-'):
            break
        else:
            testarr.append({config.get('test', 'test'): x
                for x in line.split(',')})

with open(test_dir,'w') as test_file:
    json.dump(testarr, test_file)

I hope I got the append right, as I wanted to also show you how to map the split fields more elegantly, but I'm not sure I completely understand what your original code did. (I'm guessing you'll want to trim the \n off the end of the line before splitting it, actually. Also, I imagine you want to trim the quotes from around each value. x.strip('"') for x in line.rstrip('\n').split(','))

I also renamed file to inputfile to avoid clashing with the reserved keyword file.

If you want to write more files, basically, add more states in the loop and move the write snippet back inside the loop. I don't particularly want to explain how this is equivalent to a state machine but it should not be hard to understand: with two states, you are skipping or collecting; to extend this, add one more state for the boundary when flipping back, where you write out the collected data and reinitialize the collected lines to none.

next() raises a StopIteration exception when the iterator is exhausted. In other words, your code gets to the end of the file, and you call next() again, and there's nothing more for it to return, so it raises that exception.

As for how to solve your problem, I think this might be what you want:

with open(fileName, 'r') as file:
    for line in file:
        if line.startswith('---'):
            if 'section1' in line:
                continue
            else:
                break
        fields1 = line.split(',')
        for x in range(len(fields1)):
            testarr.append({
                config.get('test', 'test'): fields1[x]
            })

with open(test_dir, 'w') as test_file:
    json.dump(testarr, test_file)