PHP Stream Write Buffer

Question

I am trying to write a large amount of data (~250K) to a stream in a non-blocking fashion.

Abstracting away the complexities and object structure, this is what I am running:

$fp = fsockopen('host', 80);
socket_set_blocking( $fp, false );
fwrite( $fp, $string, $length ); // Where $string is a 250K string

However the data doesn't all get written. Assuming this was PHP's write buffer coming into play, I set stream_set_write_buffer( $fp, 0 ) but that didn't solve the problem either.

I broke my fwrite into chunks of 4096B - and it looks like the client sends 3 complete batches (of 4096 bytes) and ~1500B of the fourth batch. Any and all successive calls to fwrite return 0 bytes written.

Does anyone have any idea how I can queue this data to all be sent out in a non-blocking fashion? If I remove socket_set_blocking( $fp, false ); - it all works fine. So clearly it's an issue with running it asynchronously.

What are your thoughts? Would the sockets extension help here at all? Does it handle buffers differently?

Note: I am intentionally writing this socket transport layer to avoid using curl for various reasons. Using curl_multi() is not an option.

You should use socket_select() to find out whether the socket is ready for writing. — Ja͢ck
– Ja͢ck, Commented Nov 15, 2012 at 2:34
Can't use socket_select without recompiling PHP due to file limit issues. The select() system call is relatively poor for this. - Is that my only option? — Colin M
– Colin M, Commented Nov 15, 2012 at 2:35
What do you mean by file limit issues? How many sockets do you have? — Ja͢ck
– Ja͢ck, Commented Nov 15, 2012 at 2:36
This is running on a web server that is capable of serving hundreds of websites and tons of requests. We are well beyond the default-limit of 1024 max open files, even though I'm only opening one socket. It's a limitation of the select() system call. — Colin M
– Colin M, Commented Nov 15, 2012 at 2:37
You have a string that is 1/4 a megabyte? The concation to make that is probably going to hurt your servers performance sometime. — Cole Tobin
– Cole Tobin, Commented Nov 15, 2012 at 3:01

user895378 · Accepted Answer · 2012-11-15 03:12:50Z

5

Your problem is almost certainly due to the fact that fwrite operations on non-blocking socket streams can be interrupted by the arrival of new packets. As a result, you can't count on an fwrite to be atomic. In such cases you MUST rely on the return value of your fwrite invocation to tell you exactly how many bytes were written to the stream in that pass and continue writing until all your data is sent.

For example ...

$dataToWrite = 'my data';
$bytesToWrite = strlen($dataToWrite);
$totalBytesWritten = 0;

while ($totalBytesWritten < $bytesToWrite) {
    $bytes = fwrite($mySock, substr($dataToWrite, $totalBytesWritten));
    $totalBytesWritten += $bytes;
}

Obviously, a robust treatment of this problem must also account for situations where the socket connection goes away, etc.

answered Nov 15, 2012 at 3:12

user895378

Sign up to request clarification or add additional context in comments.

10 Comments

Colin M Over a year ago

Yes but after the first fwrite returns a partial write, ALL subsequent calls to fwrite appear to return 0 - although the socket reports that it's still open.

user895378 Over a year ago

@ColinMorelli Then you should be using stream_select and only writing when it tells you that the stream can accept a write operation without blocking. You may also need to verify that the socket really is still alive. You can do this with is_resource($stream) && !feof($stream)

Colin M Over a year ago

See my comment above to my own answer just now - stream_select unfortunately doesn't work in our environment. Also, I am positive that the socket connection is available. is_resource($stream) && !feof($stream) returns true and by just making the socket synchronous everything works as expected. The connection is not being dropped.

Colin M Over a year ago

I may have lied in my first comment. I just realized that I wasn't using the return value of fwrite to decrement my $bytesWritten, I was just subtracting off the total number of bytes I tried to write. I will make these changes tomorrow and be back to comment.

Colin M Over a year ago

You were right on this - this method did work. We didn't end up going this route but I still accepted this as the answer.

|

Michael · Accepted Answer · 2012-11-15 03:12:41Z

0

Are you accounting for blocking and retries? I would assume you need to account for EAGAIN/EWOULDBLOCK/EINTR states otherwise you are going to end up blocking yourself.

answered Nov 15, 2012 at 3:12

Michael

10.5k1 gold badge39 silver badges51 bronze badges

Collectives™ on Stack Overflow

PHP Stream Write Buffer

2 Answers 2

10 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

10 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related