How to read a stream of data fetched on client side - JavaScript, Node JS

Question

I have created a simple proxy using Node (Express), where I am reading a pdf file from a remote server. I am able to read the file in chunks, and sending the chunks in response as res.write(). When I log the data, I can see all the streaming, but I can't receive the chunks on the front end. I am calling the endpoints correctly, but I want to get a response for each chunk is read.

Client Side code

    fetch('http://localhost:8003/pdf-cors')
    .then(response => response.text())
    .then(data => console.log(data));

Node Express code

app.get('/pdf-cors', (req, res) => {
    https.get('https://www.mazda.com/globalassets/en/assets/csr/download/2017/2017_all.pdf')
        .on('response', response => {
        response.on('data', data => {
            console.log("[Reading File]...");
            res.write(data);
        })
        response.on('end', _ => {
            console.log("File reading done !");
        })
    })
});

Note: When I put res.end() right after res.write(data) I am able to see the first chunk passed to the client in the console, but then I get an error says Error [ERR_STREAM_WRITE_AFTER_END]: write after the end.

The only thing that I want is to see each chunk being passed to the front end.

If you never close the write stream (with res.end()) - you won't get the output sent to the client. If reading the file has ended (at the end), the writing has ended at that same point. — Randy Casburn
– Randy Casburn, Commented Oct 29, 2020 at 20:27
You could (should maybe) use pipe like this: req.pipe(request.get(target)).pipe(res); — Randy Casburn
– Randy Casburn, Commented Oct 29, 2020 at 20:29

Estus Flask · Accepted Answer · 2020-10-30 17:20:45Z

6

Fetch API allows to stream data with ReadableStream.

On client side a stream can be read like:

  let res = await fetch('...');
  let reader = res.body.getReader();
  let result;
  let decoder = new TextDecoder('utf8');
  while (!result?.done) {
    result = await reader.read();
    let chunk = decoder.decode(result.value);
    console.log(chunk);
  }

answered Oct 30, 2020 at 17:20

Estus Flask

226k79 gold badges478 silver badges620 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

mwilcox Over a year ago

This helped me parse a variety of errors - thanks!!

Iman Emadi Over a year ago

how can we turn it into a blob url and use it on a video element ? i mean , to assign it to video element , must either create a blob url , or create a mediaStream of it to assign to the video's srcObject . none are possible as i tried .

Zebadix · Accepted Answer · 2023-08-03 11:00:16Z

0

If you only need server to client events you can use EventStream with server-sent events.

export const apiEventSourceStream = async (onMessage) => {
  const url = `${API_IRL}/query-stream`;
  const eventSource = new EventSource(url);

  eventSource.onmessage = (event) => {
    onMessage(event.data);
  };

  eventSource.onerror = (error) => {
    console.error("EventSource error: ", error);
    eventSource.close();
  };

  return () => eventSource.close();
};

edited Aug 3, 2023 at 11:00

answered Jul 24, 2023 at 9:13

Zebadix

162 bronze badges

Collectives™ on Stack Overflow

How to read a stream of data fetched on client side - JavaScript, Node JS

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related