Print text output into single list with Selenium web scraper for loop

Question

I am running the following program which scrapes this website. The program uses a list which fills 3 search fields on the website then prints the text of the selected page. It does this over and over again until the list_2.txt comes to an end.

Here is the code:

list_2 = [['7711564', '14', '93'], ['0511442', '7', '27']]

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from time import sleep


driver = webdriver.Firefox()
driver.get("https://www.airdrie.ca/index.cfm?serviceID=284")

for query in list_2:

   driver.find_element_by_name("whichPlan").send_keys(query[0])
   driver.find_element_by_name("whichBlock").send_keys(query[1])
   driver.find_element_by_name("whichLot").send_keys(query[2])

   driver.find_element_by_name("legalSubmit").click()
   sleep(3)

   text_element = driver.find_elements_by_xpath("//div[@class='datagrid']")
   text_element2 =
   driver.find_elements_by_xpath("//table[@class='quickkey_tbl ']")

   txt = [x.text for x in text_element]
   print(txt, '\n')
   txt2 = [x.text for x in text_element2]
   print(txt2, '\n')

   driver.back()
   driver.refresh()
   sleep(2)

I want to be able to print ALL the results from each loop/iteration into a single list. I tried using += but this ends up printing double outputs for the first item on my list only.

Yes, txt and txt2 for each iteration/loop should be inside one consolidated list. — Bisbot
– Bisbot, Commented Nov 26, 2018 at 20:33

Ratmir Asanov · Accepted Answer · 2018-11-26 20:40:37Z

2

You can try something like this:

results_list = []

for query in list_2:
   ...

   txt = [x.text for x in text_element]
   print(txt, '\n')
   txt2 = [x.text for x in text_element2]
   print(txt2, '\n')
   results_list.append(txt + txt2)
   ...

Hope it helps you!

answered Nov 26, 2018 at 20:40

Ratmir Asanov

6,4795 gold badges30 silver badges44 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Print text output into single list with Selenium web scraper for loop

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related