-1

having large amount of browser data - json sent to the server side, so wanna to chunk the data and send to the server sequentially with sleep 1 seconds between each sending.

have tried two ways: way 1:

  let url = "http://localhost:3000/api/v1/test";
  let chunkSize = 100;
  const chunks = data.reduce((acc, _, i) => {
    if (i % chunkSize === 0) acc.push([]);
    acc[acc.length - 1].push(array[i]);
    return acc;
  }, [] as T[][]);

  console.log('chunks',chunks);

  for (const chunk of chunks) {
    console.log( 'chunk',chunk.length);
     await fetch( url , {
                  method: 'POST',
                  body:  JSON.stringify( chunk ) ,
                  "Content-type": "application/json; charset=UTF-8"
                  }
    });
    await setTimeout(function(){},1000);    
    console.log( 'chunk',chunk.length);

however when testing, the process gets stuck when sending the first chunk, no error showing up, just stuck.

second way:

    let url = "http://localhost:3000/api/v1/test";
    let chunkSize = 100;
    const chunks = data.reduce((acc, _, i) => {
        if (i % chunkSize === 0) acc.push([]);
        acc[acc.length - 1].push(array[i]);
        return acc;
      }, [] as T[][]);

  return Promise.all(chunks.map(async chunk => {
     console.log( 'chunk',chunk.length);
     await setTimeout(function(){},1000);  // not working at all
     return await fetch( url , {
                  method: 'POST',
                  body:  JSON.stringify( chunk ) ,
                  headers: {
                  "Content-type": "application/json; charset=UTF-8"
                  }
    });

  }));

the problem with this above is await setTimeout(function(){},1000); never sleeps.

2
  • The only thing setTimeout() does is tell the browser to run a function in 1 second's time - in essence, setTimeout() will probably return almost instantaneously (well, within the number of milliseconds that it takes for the browser to confirm that the timer has been set). So, your "await" of setTimeout() is not going to wait 1 second - it's only going to way a very small number of milliseconds (which is obviously what you're seeing in practice). You probably want to investigate using either "setInterval" instead, or google "javascript sleep" (there are many different approaches) Commented Jul 28 at 23:39
  • 1
    await only makes sense with a Promise, therefore await new Promise(resolve => setTimeout(resolve, 1000)); in the FIRST code, the second code is a complete waste of effort Commented Jul 28 at 23:42

2 Answers 2

3

await doesn't work here because setTimeout doesn't return a promise to resolve.

to fix this you could create a sleep function that returns a promise and await that instead:

create a sleep function:

const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));

and replace your setTimeout with this sleep function:

for (const chunk of chunks) {
    console.log( 'chunk',chunk.length);
     await fetch( url , {
                  method: 'POST',
                  body:  JSON.stringify( chunk ) ,
                  "Content-type": "application/json; charset=UTF-8"
                  }

    await sleep(1000);
    console.log( 'chunk',chunk.length);
    });
Sign up to request clarification or add additional context in comments.

Comments

0

Here is my preferred version. Easier to read and to grasp what is going on

I have some chunk alternatives too; here I kept yours

let url = 'http://localhost:3000/api/v1/test';
let chunkSize = 100;

// Split data into chunks
const chunks = data.reduce((acc, _, i) => {
  if (i % chunkSize === 0) acc.push([]);
  acc[acc.length - 1].push(data[i]);
  return acc;
}, [] as T[][]);

const nodChunks = chunks.length
console.log('# of chunks', nofChunks);

let cnt = 0;

const sendChunk = () => {
  if (cnt >= nofChunks) {
    console.log('All chunks sent.');
    return;
  }

  const chunk = chunks[cnt];
  console.log('Sending chunk', cnt, 'with length', chunk.length);

  fetch(url, {
      method: 'POST',
      body: JSON.stringify(chunk),
      headers: {
        'Content-type': 'application/json; charset=UTF-8',
      },
    })
    .then(response => response.json()) 
    .then(result => {
      console.log('Success:', result); // for example
      cnt++;
      setTimeout(sendChunk, 1000); // Call next chunk after 1s
    })
    .catch(error => {
      console.error('Error:', error);
      // Here you can retry after delay if needed, or stop:
      // setTimeout(sendChunk, 1000);
    });
}

// Start sending
sendChunk();

JSPerf of the below chunkers vs your version

Alternative using slice (faster)

const chunks = Array.from({ length: Math.ceil(data.length / chunkSize) }, (_, i) =>
  data.slice(i * chunkSize, (i + 1) * chunkSize)
);

Alternative using a generator (fastest)

function* chunkArray(data, chunkSize) {
  for (let i = 0; i < data.length; i += chunkSize) {
    yield data.slice(i, i + chunkSize);
  }
}

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.