Read large csv file into array c++

Question

I am trying to read a csv file with over 170000 rows with 10 columns each entry. I wrote this code using c++ (in visual studio 2017) to read it, but it only reads 3600 entries before failing.

// Trial1.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include <iostream>
#include <cstdlib>
#include <fstream>
#include <sstream>

using namespace std;
int main()
{

    ifstream file("Brightest Day.csv");

    if (!file.is_open())
    {
        cout << "ERROR: File Open" << "\n";
    }

    string data[3000][10];

    for (long i = 0; i < 3000; i++)
    {

            for (int j = 0; j < 10; j++)
            {
                getline(file, data[i][j], ',');

            }

    }

    for (long i = 0; i < 3000; i++)
    {

        for (int j = 0; j < 10; j++)
        {
            cout<<data[i][j]<<" | ";
            if (j == 10)
            {
                cout << "\n";
            }
        }
    }
    return 0;
}

Even if it could only read around 10000 entries, I'd call it a success

Rubber ducky wants to know what the significance of 3000 is. — user4581301
– user4581301, Commented May 14, 2018 at 19:43
if (!file.is_open()) { cout << "ERROR: File Open" << "\n"; } looks like botched error handling to me. If the file is not open you print an error but continue anyway. — Jesper Juhl
– Jesper Juhl, Commented May 14, 2018 at 19:45
In what way does this program fail? It's not enough to simply say "it reads 3600 before failing". Are you getting an error message? Is the program crashing? Is it failing to compile? What kind of failure is occurring? — Xirema
– Xirema, Commented May 14, 2018 at 19:46
Uhhh... for (long i = 0; i < 3000; i++)... over 170000 rows .... i < 3000... — Daniel R. Livingston
– Daniel R. Livingston, Commented May 14, 2018 at 19:49
With over 170000 rows you are most likely going to run out of stack space if you keep trying to store your strings there (string data[3000][10]; - besides, that array is never going to hold 170000 strings, no matter how hard you really want it to) - use a std::vector. — Jesper Juhl
– Jesper Juhl, Commented May 14, 2018 at 19:58

Drew Dormann · Accepted Answer · 2018-05-14 20:19:58Z

5

You are overflowing your stack. Welcome to this website.

Your call stack is designed for small objects whose sizes are known at compile time. That's why your rubber ducky is wondering where 3000 came from. It's a guess, and anyone creating a 3001-line csv will likely crash your program. If you think 10000 lines is a success, then 10001 lines is a crash.

Use std::vector. It's an array-like structure. It manages its own size. And it doesn't store your data on the limited stack.

answered May 14, 2018 at 20:19

Drew Dormann

65.5k14 gold badges133 silver badges200 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user4581301 Over a year ago

Addendum: The math goes down something like this: in MSVC 32 bit, a std::string is 28 bytes 28 *10 = 280 bytes per row. 3600 rows is 1008000 bytes. The standard Windows stack is 1MB, so very little wiggle room is left for anything else. If compiling for 64 bit, pointers are bigger so std::string is bigger. You can probably only get about 2600 rows in 64 bit.

Collectives™ on Stack Overflow

Read large csv file into array c++

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related