How to insert seperate Python scripts in Django

Question

I have created a webscraper in Python and now I want to insert this file into my views.py and execute them using the HTML button created on the HTML page.

My scraper name is maharera.py and it is saved in same folder where I have saved views.py

My views.py looks like this:

from django.shortcuts import render
from django.conf.urls import url
from django.conf.urls import include
from django.http import HttpResponse
# Create your views here.

def index(request):
    first = {"here":"will enter more details"}
    return render(request, "files/first-page.html", context=first)
    #return HttpResponse("<em>Rera details will be patched here</em>")

After inserting it in views.y I want to execute that file using html HTML I created. How can I do that?

DisappointedByUnaccountableMod · Accepted Answer · 2021-06-08 06:39:41Z

1

Actual answer to question

Lets say the contents of maharera.py are as follows

def scraper(*args, **kwargs):
    #the scraper code goes here

then you'll need to import it as follows in the views.py file

from django.shortcuts import render
from django.conf.urls import url
from django.conf.urls import include
from django.http import HttpResponse
# Create your views here.
import maharera

def index(request):
    first = {"here":"will enter more details"}
    return render(request, "files/first-page.html", context=first)
    #return HttpResponse("<em>Rera details will be patched here</em>") 

def scraper_view(request):
    maharera.scraper()
    return HttpResponse("<em>Scraper started</em>")

It is advisable to not run a web scraper through a http requests like these. Http requests are supposed to return response within fraction of seconds and should not take long.

When you hit scraper_view it will start executing the code inside it. In scraper view, there is call to the scraper and we don't know how long will it take for that function to end. Till that function doesn't end, the response of the page will not be returned to the user.

For such long running tasks, you should look into task queues. Looking into celery

edited Jun 8, 2021 at 6:39

DisappointedByUnaccountableMod

6,8444 gold badges21 silver badges23 bronze badges

answered Jan 19, 2018 at 7:02

Anuj

1,01411 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

James Over a year ago

I am learning django and trying to experiment it with my office work. Suppose my boss wants to get data he will press button and scrape the data. Thanks for you response and advice , once i become strong in django i will definitely Follow your advice.Now Code is running. One thing how can i execute this scraper using the html button. I have create button and when i press it should starts the scraper in DJANGO.

Anuj Over a year ago

So since you're learning django this is the perfect time to learn things right. Here's a small thing that could help you. When your boss hits the scrapper button, Create a variable in database which checks if the the scrapper completed the task. Start a celery task which scraps. when the the scrapper ends, let the task update database variable. Create a new view which checks if the scrapper completed the task, if its completed show data, else show scrapping in progress

Anuj Over a year ago

Also don't forget to mark this answer as answer to the question. :)

Collectives™ on Stack Overflow

How to insert seperate Python scripts in Django

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related