1

How to make my Selenium script faster, guys. I have this Selenium script:

import time                                     
from selenium import webdriver                
from selenium.webdriver.support.wait import WebDriverWait                                       
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support import expected_conditions as EC                                
from selenium.webdriver.common.by 
import By from selenium.webdriver.common.keys import Keys 
from xvfbwrapper import Xvfb                                                                                                                    
display = Xvfb()                                
display.start()                                                                                 
option = Options()                              
prefs = {'profile.default_content_setting_values': {'images': 2}}                                
option.add_experimental_option('prefs', prefs) 
driver = webdriver.Chrome(options = option)      
driver.get("https://www.nike.com.br/chuteira-nike-premier-2-sala-unissex-153-169-171-309321")
wait = WebDriverWait(driver, 5)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, '.cc-allow'))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, "//label[@for='tamanho__idM40F395']"))).click()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'button#anchor-acessar-unite-oauth2'))).click()

Guys, the problem is that this script is slow. How can I wait only for the time in this scenario instead of waiting longer than the time in this scenario to click on an element? Because I think that because of this excerpt wait = WebDriverWait (driver, 5) the page expects more than it should to click on the elements, so I just wanted to wait for the time in this scenario, and also notice that in this script I took the load of images, I did this to optimize the script, is it possible to remove the loading of images and text? I would like to remove some html and css text elements to load faster. Can someone tell me how can I do this?

Improve this question
12
  • What are you trying to target with this selector "//label[@for='tamanho__id40']"? I looked at all the label elements on the page and none of them have a for attribute like that. Commented May 24, 2021 at 20:30
  • also, youve tagged both python-3.x and python-2.7 for your question. Which version are you using? Commented May 24, 2021 at 20:31
  • @C.Peck there are dozens of label elements with the for attribute attached to them on the page, but none of them have tamanho__id40 as a value Commented May 24, 2021 at 20:58
  • 1
    @JD2775, yes that is more clearly what I saw too. (or anything even resembling tamanho__id40 for that value) Commented May 24, 2021 at 20:59
  • @JD2775 I edited the question had put the URL and the wrong is right now :) Commented May 24, 2021 at 21:10

1 Answer 1

Reset to default
3

I've made some research and below are my results. You have few options to try:

1 Headless mode:

It is believed that headless mode is faster. Check this answer Running Selenium with Headless Chrome Webdriver and here Difference of Headless browsers for automation Your code would look like (loading pictures not disabled):

from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

import time
start_time = time.time()
options = webdriver.ChromeOptions()
options.headless = True  # running in headless mode
options.add_argument("--disable-blink-features")
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(chrome_options=options, executable_path='/snap/bin/chromium.chromedriver')

driver.get("https://www.nike.com.br/chuteira-nike-premier-2-sala-unissex-153-169-171-309321")
wait = WebDriverWait(driver, 15)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, '.cc-allow'))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, "//label[@for='tamanho__idM40F395']"))).click()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'button#anchor-acessar-unite-oauth2'))).click()
end_time = time.time()
print("Total execution time: {} seconds".format(end_time - start_time))

Total execution time (3 runs):

12.759594202041626 seconds
13.571481466293335 seconds
17.36514186859131 seconds

2 Disable pictures Reference: Python: Disable images in Selenium Google ChromeDriver

from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

import time
start_time = time.time()
options = webdriver.ChromeOptions()

prefs = {'profile.default_content_setting_values': {'images': 2}}
options.add_experimental_option("prefs", prefs)  # disable loading pictures
options.add_argument("--disable-blink-features")
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(chrome_options=options, executable_path='/snap/bin/chromium.chromedriver')

driver.get("https://www.nike.com.br/chuteira-nike-premier-2-sala-unissex-153-169-171-309321")
wait = WebDriverWait(driver, 15)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, '.cc-allow'))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, "//label[@for='tamanho__idM40F395']"))).click()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'button#anchor-acessar-unite-oauth2'))).click()
end_time = time.time()
print("Total execution time: {} seconds".format(end_time - start_time))

Total execution time: 10.57245922088623 seconds
Total execution time: 13.770843982696533 seconds
Total execution time: 10.908315896987915 seconds

3 Disable CSS Reference How to disable CSS in Python selenium using ChromeOptions - Not sure this option is correct now because it does not improve waiting times and works only in headless mode:

from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

import time
start_time = time.time()
options = webdriver.ChromeOptions()
prefs = {'profile.default_content_setting_values': {'cookies': 2, 'images': 2, 'javascript': 2,
                            'plugins': 2, 'popups': 2, 'geolocation': 2,
                            'notifications': 2, 'auto_select_certificate': 2, 'fullscreen': 2,
                            'mouselock': 2, 'mixed_script': 2, 'media_stream': 2,
                            'media_stream_mic': 2, 'media_stream_camera': 2, 'protocol_handlers': 2,
                            'ppapi_broker': 2, 'automatic_downloads': 2, 'midi_sysex': 2,
                            'push_messaging': 2, 'ssl_cert_decisions': 2, 'metro_switch_to_desktop': 2,
                            'protected_media_identifier': 2, 'app_banner': 2, 'site_engagement': 2,
                            'durable_storage': 2}}
options.add_experimental_option("prefs", prefs)  # disable loading pictures
options.headless = True  # running in headless mode
options.add_argument("--disable-blink-features")
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(chrome_options=options, executable_path='/snap/bin/chromium.chromedriver')

driver.get("https://www.nike.com.br/chuteira-nike-premier-2-sala-unissex-153-169-171-309321")
wait = WebDriverWait(driver, 15)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, '.cc-allow'))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, "//label[@for='tamanho__idM40F395']"))).click()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'button#anchor-acessar-unite-oauth2'))).click()
end_time = time.time()
print("Total execution time: {} seconds".format(end_time - start_time))


Total execution time: 13.649555444717407 seconds
Total execution time: 12.698347806930542 seconds
Total execution time: 13.572114706039429 seconds

Conclusion

From what I see using prefs = {'profile.default_content_setting_values': {'images': 2}} is the optimal solution, at least for your case. Headless mode does not add any improvements in speed.

However, results may be different depending on Selenium code and other factors.

Also, here you can find some good advices on how to make tests faster. List:

  • Use fast selectors
  • Use fewer locators
  • Create atomic tests
  • Don't test the same functionality twice
  • Use only explicit waits
  • Use the chrome driver
  • Use drivers for headless browsers
  • Re-use the browser instance
  • Run scripts in parallel
  • Use HTTP parse libraries
  • Pre-populate cookies do not load images in the web page

You are not using implicit waits, using css selectors which are faster.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.