parse directly on pure html source with selenium in python

Question

I'm trying to test the selenium program that I wrote by giving it an HTML source as a string for some reasons such as speed. I don't want it to get the URL and I don't want it to open a file I just want to pass it a string that contains whole DIV part of that site and do parsing stuff on it. this is part of a module that i wrote:

source = driver.page_source
return {'containers': source}

and in another module,

def get_rail_origin(self):
    return self.data['containers'].find_element_by_id('o_outDepName')...

I'm trying to do parsing stuff on it but I get

AttributeError: 'str' object has no attribute 'find_element_by_id'

So how can I parse on pure HTML source without opening any file or URL

Andersson · Accepted Answer · 2018-11-03 13:22:59Z

1

Selenium works with live HTML DOM. If you want to get source and then parse it, you can try, for instance, lxml.html:

def get_rail_origin(self):
    source = html.fromstring(self.data['containers'])
    return source.get_element_by_id('o_outDepName')

P.S. I assumed that self.data['containers'] is HTML source code

answered Nov 3, 2018 at 13:22

Andersson

52.8k18 gold badges83 silver badges132 bronze badges

Sign up to request clarification or add additional context in comments.

14 Comments

Andersson Over a year ago

@iman_sh77 , yes, exactly. Selenium is a tool that allows to interact with browser and a DOM while you need a tool to parse HTML code...

Andersson Over a year ago

@iman_sh77 , hmm... That's interesting. But note that execute_script is method-container that just allows to inject JavaScript passed as argument. If it works for you you can execute every required operation as JavaScript

Andersson Over a year ago

@iman_sh77 If your goal is to scrape some data from web then you don't need Selenium at all! I prefer to use python-requests + lxml.html

Andersson Over a year ago

@iman_sh77 , not exactly. Even if you need to fill some forms and click some buttons it's all can be done via direct HTTP request (you might need to pass some parameters to your request). But if your goal is to test web-UI, then you should do this with Selenium

Andersson Over a year ago

@iman_sh77 , it's hard to tell how to solve the issue best as I don't know how exactly that page behaves

|

Collectives™ on Stack Overflow

parse directly on pure html source with selenium in python

1 Answer 1

14 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

14 Comments

Your Answer

Sign up or log in

Post as a guest

Related