How to open each url in new tab in chrome driver using python

Question

I have list of URLs and i need to scrape data from them. The website refusing connection when opening each url in new driver, so i decided to open each url in new tab(the website allowing this way). Below code i am using

from selenium import webdriver
import time
from lxml import html

driver = webdriver.Chrome()
driver.get('https://www.google.com/')

file = open('f:\\listofurls.txt', 'r')

for aa in file:
    aa = aa.strip()
    driver.execute_script("window.open('{}');".format(aa))
    soup = html.fromstring(driver.page_source)
    name = soup.xpath('//div[@class="name"]//text()')
    title = soup.xpath('//div[@class="title"]//text()')
    print(name, title)
    time.sleep(3)

But the problem is all URLs are opening at a time instead of one after one.

is opening drivers one by one allowed? I mean opening one driver, then close it and after that opening another one. the website will refuse it or is it ok? — 0xM4x
– 0xM4x, Commented Nov 13, 2019 at 12:34

Hamza Lachi · Accepted Answer · 2019-11-13 12:51:40Z

1

You can try this code:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
from lxml import html

driver = webdriver.Chrome()
driver.get('https://www.google.com/')

file = open('f:\\listofurls.txt', 'r')

for aa in file:
    #open tab
    driver.find_element_by_tag_name('body').send_keys(Keys.COMMAND + 't') 
# You can use (Keys.CONTROL + 't') on other OSs

    # Load a page 
    driver.get(aa)
# Make the tests...
    soup = html.fromstring(driver.page_source)
    name = soup.xpath('//div[@class="name"]//text()')
    title = soup.xpath('//div[@class="title"]//text()')
    print(name, title)
    time.sleep(3)


driver.close()

answered Nov 13, 2019 at 12:51

Hamza Lachi

1,0641 gold badge8 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Srinath Neela Over a year ago

Dear Hamza lachi, thank you very much, it is working

0xM4x · Accepted Answer · 2019-11-13 12:22:23Z

0

I think you have to strip before the loop like this:

driver = webdriver.Chrome()
driver.get('https://www.google.com/')

file = open('f:\\listofurls.txt', 'r')
aa = file.strip()

for i in aa:
    driver.execute_script("window.open('{}');".format(i))
    soup = html.fromstring(driver.page_source)
    name = soup.xpath('//div[@class="name"]//text()')
    title = soup.xpath('//div[@class="title"]//text()')
    print(name, title)
    time.sleep(3)

answered Nov 13, 2019 at 12:22

0xM4x

4702 gold badges9 silver badges23 bronze badges

1 Comment

Srinath Neela Over a year ago

not working, i think 'strip' is not issue here, new tab should be performed after extracting data from current tab.

Collectives™ on Stack Overflow

How to open each url in new tab in chrome driver using python

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related