4

I need to write C code (not C# or C++) to get data from hardware analyzer (through 100Mb TCP/IP network) and write it to disk.

Why I say "Low latency", well, Hardware analyser have a 9KB internal buffer, this is 2ms of data storing, after that it runs into buffer overflow and I lose information packets.

My application is able to get this buffer without losing any packet, but I noticed I wasn't able to write data to the disk at this rate.

My code looks like this:

int main (int argc, char * argv [] )
{
  pthread_t th_rx; /* Thread for receiving packets */

  outputfile = fopen ("output.log", "wb");

  link_open(); // open device link

  pthread_create ( &th_rx, NULL, read_packets, 0 );

  // running loop
  fclose (outputfile);
  pthread_exit(NULL);
  link_close(); // close device link 
  return 0; 
}

static thread_result read_packets (void *arg)
{
  char  rxbuf[40960];

  while (receiving)
  {
    bytes_received = read_packet(); //read packet from device buffer
    rxbuf = extract_data(bytes_received);
    fwrite (rxbuf, 1, bytes_received, outputfile); 
  }
  return thread_return;
}

What I need here are ideas to how to do that.

  • 1: don't write packet upon received, create a circular? buffer
  • 2: Creating 2 thread circular buffer?

Any ideas of how to improve my code and what I can do to get stable writes?

Code examples will be very appreciated because I'm feeling lost :(

Thanks

3
  • Two threads, as suggested by @nos, yes. Also, use a producer-consumer queue of buffer object references/pointers to avoid the continual, and avoidable, bulk-copying. Commented Nov 29, 2013 at 12:06
  • This stackoverflow question can be relevant - stackoverflow.com/questions/20186859/… Commented Nov 29, 2013 at 12:07
  • Lazin, this method uses Boost library and it's C++ Commented Nov 29, 2013 at 12:35

3 Answers 3

2

As you guessed correctly, the firs thing you should do is, do not write to disk while you are reading from device.

  • create circular/ ring buffer (or just list of buffers) to store data read from device. This operation will be fast enough, so that device memory does not fill up.

    • However, if device read is non-blocking you may want o decide to sleep so that your loop do not hog the cpu.
  • One thread reads data from device and puts in buffer.

  • In another thread, keep writing the buffer to disk.
Sign up to request clarification or add additional context in comments.

2 Comments

Thanks Rohan, please can you enlighten me with some code examples?
On quick google I got couple of useful links, you need to improve them for multiple threads en.wikipedia.org/wiki/Circular_buffer c.learncodethehardway.org/book/ex44.html
1

As you said hw analyzer has 9 kb memory for only 2 ms. Then speed required for writing to HDD is 9*1024*500 = 4608000 b/s or 4.6 Mb/s. I believe that test computer has speed of writing at HDD at least 10 Mb/s. So the only one problem here is to write to HDD with such a big portions of data which can give enough time to do writing. For instance, buffer can be 1 sec or 4.6 Mb. Storing of such portion of data will be faster then collecting of 1 s of data from hw device and "writing thread" takes no more than 500 ms to do its job. In real program this time can differ but in any way it must be enough for single-core systems as well.
Therefore you may use two ping-pong (one is filled with data from hw, one is stored to HDD) buffers with two threads and without circular buffer. Thus you can use producer/consumer model which is well described and has a lot of examples.


3 Comments

Michael, you have a error in your calcs, because we recive 9KB not 9Kb, this is 9*8*1024*500 = 36Mb/s, that's all, by the way, I will need a better throughput from HDD.
@hiroru Yep, sorry, I've missed it for some reason, 8x difference... But anyway, 40-50 Mb/s must not be a problem for modern HDD/SSD. Thus the same idea should work only with a greater size of ram.
I am coding a 1 producer - 1 consumer model with N queues as you described.
0

You should look into asynchronous I/O. However Async-I/O APIs are operating system dependent and you didn't specify which OS you're using. Also your use of threads makes no sense. If you want to use threads (instead of async I/O) then you'd have one thread doing the packet reading and the other thread should to the writing. Putting it both into a single thread gives you no benefit.

1 Comment

This must be cross-platform (Win/Linux)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.