Project using Python Webscraping

Question

I'm working on a project and I just wanted to know what route (If I'm not going down the correct route) I should go down in order to successfully complete this project.

Ferry Time: This application uses boat coordinates and ETA algorithms to determine when a boat will be approaching a certain destination.

Now, I've worked up a prototype that works, but not the way I want it to. In order for my ETA's to show up on my website accurately, I have a python script that runs every minute to webscrape these coordinates from a specific site, do an algorithm, and spit out an ETA. The ETA is then sent to my database, where I display it on my website using PHP and SQL.

The ETA's only update as long as this script is running (I literally run the script on Eclipse and just leave it there)

Now my question: Is there a way I can avoid running the script? Almost like an API. Or to not use a database at all?

Thanks !

Wells Burke · Accepted Answer · 2017-05-30 20:32:58Z

1

If the result of your algorithm only depends on the LAST scrape and not a history of several scrapes, then you could just scrape "on demand" and deploy your algo as an AWS lambda function.

answered May 30, 2017 at 20:32

Wells Burke

261 bronze badge

Sign up to request clarification or add additional context in comments.

1 Comment

Jonathan Sanchez Over a year ago

Thanks Wells! I'm going to give AWS a try

lpiner · Accepted Answer · 2017-05-30 19:54:11Z

0

Even if they had an API you'd still need to run something against it to get the results. If you don't want to leave your IDE open you could use cron to call your script.

answered May 30, 2017 at 19:54

lpiner

4654 silver badges13 bronze badges

Collectives™ on Stack Overflow

Project using Python Webscraping

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related