How to read from a binary file and load the data into a 3d vector in C++?

Question

I currently have some .bin files, each of which contains a matrix of size AxBxC. I would like to load it into a 3d vector vector<vector<vector<float>>> in C++, but encountered some problems.

I tried to flatten the matrix and load it into a buffer first, but the size of the buffer was weird so I got stuck there.

My code was:

vector<vector<vector<float>>> load_3d_bin(const std::string& path){
    vector<vector<vector<float>>> r;
    std::ifstream binary_feat(path.c_str(),std::ios::in | std::ios::binary);
    std::vector<float> flattened_feat((std::istream_iterator<char>(binary_feat)),std::istream_iterator<char>());
    for(auto i: flattened_feat){
        int value = i;
    }
    std::cout<<flattened_feat.size()<<endl;

// code to be written

   return r;
}

Ideally r would be a 3d vector, but I am stuck at the iterator part as the .size() outputs differently from total length = AxBxC, which was what I expected...

Any solution or feedback? Thanks in advance!

--Edit:

Sorry for not being clear enough! In my original .bin files, the floats are encoded as 3d matrices, i.e., in each file they are in an array of AxBxC size. I would like to first flatten the original 3d matrices and then reshape the flattened vectors into 3d vectors of size 'AxBxC'.

I managed to do the flattening part myself as follows:

typedef vector<vector<vector<float>>> t_f
typedef vector<vector<float>> m_f

t_f load_3d_bin(const std::string& path){
    t_f r;
    std::ifstream binary_feat(path.c_str(), std::ios::in | std::ios::binary);
    std::vector<float> flattened_feat;
    float tmp = 0;
    binary_feat.read(reinterpret_cast<char*>(&tmp), sizeof(float));
    while(!binary_feat.eof()) {
        binary_feat.read(reinterpret_cast<char*>(&tmp), sizeof(float));
        flattened_feat.push_back(tmp); 
    }
    std::cout<<"file size:::"<<flattened_feat.size()<< std::endl;

# the reshaping code to be written

return r;
}

And now with the flattened vector I have, I tried to reshape it into a 3d vector, which is the # the reshaping code to be written part above:

t_f r(A, m_f(B));
for (int i=0; i<A; i++){
    for (int j=0; j<B; j++){
        for (int k=0; k<C; k++){
            r[i][j][k] = flattened_feat[(i*B+j)*C+k];
        }
    }
}

But I got segmentation fault... Any idea why I got the error and how to solve it? Or is it better to directly load the 3d matrices in .bin files to 3d vectors? Thanks!

3d arrays (array in the general sense) can be realized by using a 1d array (e.g. std::vector) and managing the 3d strides and indices yourself. This is IMHO a better way than a "real" 3d data structure. It is the method used by opencv, numpy and others. And in addition it makes reading a file rather trivial. — wohlstad
– wohlstad, Commented Sep 23, 2022 at 9:44
@TedLyngmo The original 3d .bin files were written in python... simply np.array(3dfeatures).astype(np.float32).tofile(3dbin_path), where 3dfeatures is a 3d np array. — Ziggy1209
– Ziggy1209, Commented Sep 26, 2022 at 1:44
@Ziggy1209 Perhaps you should include the shortest possible python script that produces such a file in the question. It could give the those interested something to test on when trying to deserialize it in a different language. — Ted Lyngmo
– Ted Lyngmo, Commented Sep 26, 2022 at 16:15

Botje · Accepted Answer · 2022-09-23 10:01:55Z

0

Assuming your bin files just contain a flat concatenation of floats in binary format, in row-major order:

vector<vector<vector<float>>> load_3d_bin(const std::string& path){
  vector<vector<vector<float>>> ret(A, vector<vector<float>>(B));
  std::ifstream binary_feat(path.c_str(),std::ios::in | std::ios::binary);

  for (int i = 0; i < A; i++) {
    for (int j = 0; j < B; j++) {
      for (int k = 0; k < C; k++) {
        auto& row = ret[i][j].emplace_back(C);
        binary_feat.read(row.data(), row.size() * sizeof(float));
      }
    }
  }
  return ret;
}

Alternatively, you could use the fantastic armadillo library:

fcube c;
c.load(path.c_str(), raw_binary);
c.reshape(A, B, C);

answered Sep 23, 2022 at 10:01

Botje

32.1k4 gold badges36 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ziggy1209 · Accepted Answer · 2022-09-26 04:26:23Z

0

In the end I managed to do this myself.

The solution is as follows:

typedef vector<vector<vector<float>>> t_f
typedef vector<vector<float>> m_f
typedef vector<float> v_f

t_f load_3d_bin(const std::string& path){
    std::ifstream binary_feat(path.c_str(), std::ios::in | std::ios::binary);
    std::vector<float> flattened_feat;
    float tmp = 0;
    binary_feat.read(reinterpret_cast<char*>(&tmp), sizeof(float));
    while(!binary_feat.eof()) {
        binary_feat.read(reinterpret_cast<char*>(&tmp), sizeof(float));
        flattened_feat.push_back(tmp); 
    }

    t_f r(A, m_f(B,v_f(C)));
    for (int i=0; i<A; i++){
        for (int j=0; j<B; j++){
            for (int k=0; k<C; k++){
                r[i][j][k] = flattened_feat[(i*B+j)*C+k];
            }
        }
    }

    return r;
}

The returned r is a 3d vector<vector<vector<float>>> of size AxBxC.

answered Sep 26, 2022 at 4:26

Ziggy1209

115 bronze badges

1 Comment

Ted Lyngmo Over a year ago

while(!binary_feat.eof()) is wrong. You'll likely read one float too many that way. Why do you read and discard a float at the start? Just do while(binary_feat.read(buf, sizeof(float))) { ... }. It risky reading directly into the float. I'd recommend reading into a char buf[sizezof(float)] then memcpy the buf into the float.

Collectives™ on Stack Overflow

How to read from a binary file and load the data into a 3d vector in C++?

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related