Python Scrapy Click on html button

Question

I am new to scrapy and using scrapy with python 2.7 for web automation. I want to click on a html button on a website which opens a login form. My problem is that I just want to click on a button and trasfer control to new page. I have read all similar questions but none found satisfactory because they all contain direct login or using selenium.

Below is HTML Code for button and I want to visit http://example.com/login where there is login page.

<div class="pull-left">
    <a href="http://example.com/login" class="emplink">Employers</a>

I have written code for to extract link. But how to visit that link and carry out next process. Below is My code.

import scrapy

class QuotesSpider(scrapy.Spider):
    name = 'pro'
    url =  "http://login-page.com/"


def start_requests(self):
    yield scrapy.Request(self.url, self.parse_login)


def parse_login(self, response):
    employers = response.css("div.pull-left a::attr(href)").extract_first()
    print employers

Do I need to use "yield" Everytime and callback to new fuction for just visiting a link or there is other way to do it.

Wilfredo · Accepted Answer · 2018-07-04 20:09:50Z

2

What you need is to yield a new request or easier make a response.follow like in the docs:

def parse_login(self, response):
    next_page = response.css("div.pull-left a::attr(href)").extract_first()
    if next_page is not None:
        yield response.follow(next_page, callback=self.next_page_parse)

About the callback, it depends basically on how easily can the page gets parsed, for example, check the general spiders section on the docs

answered Jul 4, 2018 at 20:09

Wilfredo

1,5481 gold badge9 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python Scrapy Click on html button

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related