Javascript: How can i upload to S3 directly without saving output file locally?

Question

I have a problem with developing a crawler using nodejs/puppeteer. The old crawler was:

Crawl Pages
Store the output file locally with the fs module

Since i'm going to introduce UI on the server, have set up the scenario to upload it to S3 instead of storing it locally, and show the result as a UI.

Crawl Pages
Stream output files to the server with the fs module
Get the output file back and upload it to the S3 bucket

The above is a scenario that i know of as a knowledge, and i'd like to know if it is possible as below.

Crawl Pages
Upload the stream stored data to memory to the S3 bucket

If you have a scenario like this, I would like to receive a guide. I would really appreciate if you would comment or reply :)

TomLV · Accepted Answer · 2019-08-27 18:08:33Z

0

This is definitely possible. If you just pipe from your input stream to your server and pipe up to the S3 it should complete the loop.

This is possible because you can stream uploads to S3, even without knowing the size of the file beforehand.

This answer should help you out: S3 file upload stream using node js

If you post some code we can answer this a little bit better. But hope this puts you in the right direction.

answered Aug 27, 2019 at 18:08

TomLV

3852 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Javascript: How can i upload to S3 directly without saving output file locally?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related