Crawler in python

Question

I have a website (my college website) for which each student has a username/password to login into his account. The website consists of many files uploaded in different formats. I want to programmatically enter my credentials and login to my account and download all files that have been uploaded in the site.

Is there any python module that would authenticate html form and provide access to the pages and then allow downloading files present in those pages?

All that i want to do is download files that are present in a website that can be accessed only by providing credentials. — nikhil
– nikhil, Commented May 10, 2011 at 17:32
ok..is there any python module that would authenticate html form and provide access to the pages and then allow downloading files present in those pages? — nikhil
– nikhil, Commented May 10, 2011 at 17:40
"Is there any python module...?" Yes. The answer is almost always "Yes". You should -- perhaps -- ask your real question. Merely testing for existence isn't very informative. Perhaps you'd like to know which module? Or perhaps you'd like to search Stack Overflow for existing questions that are just like your question. For example, all of these may already answer your question: stackoverflow.com/search?q=%5Bpython%5D+crawler — S.Lott
– S.Lott, Commented May 10, 2011 at 20:32

Philip Southam · Accepted Answer · 2011-05-10 17:41:44Z

2

I would look at http://scrapy.org. It's python based crawler/scraper that you can customize to perform any workflow you want. Here's an example of how user authentication is handled: http://doc.scrapy.org/topics/request-response.html?highlight=username#request-usage-examples

answered May 10, 2011 at 17:41

Philip Southam

16.6k6 gold badges30 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Crawler in python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related