3

TL;DR

What would be a good way, in C++ and using STL idioms, to iterate through a binary file to read, transform, and then again write out the data? The files can be pretty large (several hundred MB) so I don't want to load the entire file into memory at one time.

More context

I am trying to improve a utility which performs various operations on binary files. These files contain set of records consisting of a header and then the data. The utility provides options to dump the file to text, filter out certain records, extract certain records, append records etc. Unfortunately all of these functions have the code to read and write from the file copied and pasted into every function so the single source file contains a lot of redundant code and is starting to get out of hand.

I'm only just getting up to speed with using C++ and the STL but this is something that seems should be doable with some sort of template/iterator magic but I can't find a good example explaining this scenario. The other strategy I may pursue is to wrap the file access in a class which provides GetNextRecord and WriteNextRecord methods.

Below is a self-contained/(extremely) simplified version of what I'm working on. Is there a good way to write a function to read the data in the file created by WriteMyDataFile and create a new output file that removes all the records containing an 'i' character? I'm looking to abstract away the reading/writing of the file so that the function can mainly be about working with the data.

#include <fstream>
#include <iostream>
#include <sstream>
#include <string>
#include <vector>

using namespace std;

const int c_version = 1;

struct RecordHeader
{
    int length;      
    int version;
};

void WriteMyDataFile(char* recordFile, char* data)
{
    ofstream output (recordFile, ios::out | ios::binary);

    stringstream records(data);

    while(records)
    {
        string r;
        records >> r;

        if(r.length() < 1)
        {
            continue;
        }

        RecordHeader header;
        header.length = r.length();
        header.version = c_version;

        output.write((char*)&header, sizeof(header));
        output.write(r.data(), header.length);
    }

    output.close();
}

 vector<string> ReadDataFile(char* recordFile)
 {
    vector<string> records;
    ifstream input (recordFile, ios::in | ios::binary);

    while(!input.eof())
    {
        RecordHeader header;
        input.read((char*)&header, sizeof(header));

        if(!input.eof())
        {
            char* buffer = new char[header.length + 1];

            input.read(buffer, header.length);
            buffer[header.length] = '\0';

            string s(buffer);
            records.push_back(s);

            delete[] buffer;
        }
    }
    return records;
}


int main(int argc, char *argv[])
{
    WriteMyDataFile(argv[1], argv[2]);
    vector<string> records = ReadDataFile(argv[1]);

    for(int i=0; i < records.size(); i++)
    {
        cout << records[i] << endl;
    }

    return 0;
}

To run this:

C:\>RecordUtility.exe test.bin "alpha bravo charlie delta"

Output:
alpha
bravo
charlie
delta

Improve this question
3
  • Have you tried istreambuf_iterators and ostreambuf_iterators? Commented Jun 27, 2011 at 17:40
  • Generically, stream iterators seem like a good idea, but depending on your particular needs memory mapping the file might be another worthwhile approach. Commented Jun 27, 2011 at 17:48
  • I've tried reading about them but most of the websites I've stumbled on don't really explain how one should actually use these. Do I just need a class which wraps my record header/data and then overloads << and >> to use these? Commented Jun 27, 2011 at 17:52

4 Answers 4

Reset to default
3

I'd handle this by overloading operator>> and operator<< for your Record type:

struct Record { 
    struct header {
        int length;
        int version;
    }

    header h;
    std::vector<char> body;
};

std::istream &operator>>(std::istream &is, Record &r) {
    is.read((char *)&r.h, sizeof(r.h));
    body.resize(h.length);
    is.read(&body[0], h.length);
    return is;
}

std::ostream &operator<<(std::ostream &os, Record const &r) { 
    os.write((char *)r.h, sizeof(r.h));
    os.write(r.body, r.body.size());
    return OS;
}

Once you've done that, you can use istream_iterator and ostream_iterator with a stream of those structures. For example, to do the copy roughly equivalent to what you have above would be something like:

std::ifstream in("some input file");

std::copy(std::istream_iterator<Record>(in), 
          std::istream_iterator<Record>(),
          std::ostream_iterator<Record>(std::cout, "\n"));

Or if, for example, you wanted to copy only those records with a Version number of 2 or higher, you could do something like:

struct filter { // or use a lambda in C++0x
    bool operator()(Record const &r) { return r.h.Version < 2; }
};

std::remove_copy_if(std::istream_iterator<Record>(in),
                    std::istream_iterator<Record>(),
                    std::ostream_iterator<Record>(std::cout, "\n"),
                    filter());
Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

+1 I don't think ` header h` should be part of the DataRecord object. It is solely there to help in streaming the data and to validate its correctness. When you read it in you should not need to persist this data.
@Martin: You can avoid storing the Length explicitly, but primarily because the vector already stores it. You need it stored somewhere, however, to be able to create another file in the expected input format. It's harder to guess whether the Version needs to be stored or not (or is needed at all).
Thanks, I had most of the pieces but this helped put it together.
2

Instead of this:

while(!input.eof())

It is easier (and clearer) to write:

RecordHeader header;
while(input.read((char*)&header, sizeof(header)))
{

To do the template magic you want is to us std::istream_iterator and std::ostream_iterator.

This basically requires uyou to write the operator >> and operator << for your class.

PS. I hate the use of binary objects (RecordHeader). It makes the code harder to maintain. Stream the object so that it knows how to read itself back in. Which leads back to the operators >> and <<

Improve this answer

5 Comments

i don't see where this is "easier" or "clearer". And, it simply not the answer to the question.
@Mike Caron: What would be considered more standard by the majority of languages. The read operation should be the condition of continuation of the loop. If the read fails (because of EOF or other error) then the loop is not even entered. Also because it is more standard it then makes it easier to introduce the [io]stream_iterator into the code.
@Thomas Berger: I agree. I had to step away before I could add more. I would suggest reading Jerry Coffin who has written what I would have done.
@Martin: More correct, sure. But, it's not easier to write (more typing, pointers and casting) nor clearer ("When will it stop?").
@Mike Caron: Totally disagree. I think the preferred form is much easier to write/read (as it is the same as all other languages (and the way it is taught in Universities (at least when I went to university))). It is also less typing/ less pointers and zero explicit casting. If the class is designed correctly (which IOstream libraries are).
1

The code you posted and you idea about a wrapper class,looks like the best way to do this with the STL for me.

If you want to provide the plain data to your main program, you may have a look at boost::iostream. It provides some good ways to implement filters "into" a stream (for example a zlib filter) and maybe what you are looking for.

Improve this answer

Comments

0

You can create your own stream operators operator<< and operator>> to manage the reading/writing of your Record structures from a stream. Then you can run things through your vector of record, applying whatever filtering you desire (perhaps with std::remove_if, for the example in the question) and write it back similar to below...

#include <algorithm>
#include <vector>
#include <iostream>
#include <iterator>
#include <stdexcept>
#include <sstream>

namespace {
    template <class Type>
    void WriteBinary(const Type& data, std::ostream& os)
    {
        const char *binaryData = reinterpret_cast<const char*>(&data);
        os.write(binaryData, sizeof(data));
    }

    template <class Type>
    Type ReadBinary(std::istream& is)
    {
        Type data;
        is.read(reinterpret_cast<char*>(&data), sizeof(data));
        return data;
    }
}

struct Record
{
    int               mVersion;
    std::vector<char> mData;
};

std::ostream& operator<<(std::ostream& os, const Record& record)
{
    WriteBinary(record.mData.size(), os);
    WriteBinary(record.mVersion, os);

    std::copy(record.mData.begin(), 
              record.mData.end(), 
              std::ostream_iterator<char>(os)); 

    return os;
}

std::istream& operator>>(std::istream& is, Record& record)
{
    if (std::char_traits<char>::not_eof(is.peek()))
    {
        typedef std::vector<char>::size_type size_type;

        size_type length = ReadBinary<size_type>(is);
        record.mVersion = ReadBinary<int>(is);

        if (record.mVersion != 1)
        {
            throw std::runtime_error("Invalid version number.");
        }

        record.mData.clear();
        record.mData.resize(length);
        is.read(&record.mData.front(), length);
    }
    else
    {
        // Read the EOF char to invalidate the stream.
        is.ignore();
    }

    return is;
}

int main()
{
    // Create a Record
    std::string str = "Hello";

    Record rec;
    rec.mVersion = 1;
    rec.mData.assign(str.begin(), str.end());

    // Write two copies of the record to the stream.
    std::stringstream ss;
    ss << rec << rec;

    // Read all the records in the "file"
    std::vector<Record> records((std::istream_iterator<Record>(ss)),
                                std::istream_iterator<Record>());

    std::cout << "Read " << records.size() << " records." << std::endl;

    // Manipulate records here...then write all of them back to a file.
    std::stringstream myNewFile;
    std::copy(records.begin(), 
              records.end(), 
              std::ostream_iterator<Record>(myNewFile));

    return 0;
}
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.