Python: Headless Selenium without admin access

Question

I have a crawler in python that uses Selenium webdriver. I would like to start it in a cluster and leave it running for about 10 days. The problem is:

I do not have an X display!!!!

I have done some searching and reading. Normally this would be solved by using Xfvb and pyvirtualdisplay. It is not yet installed in the clusters. Now another problem pops up:

I do not have admin access in clusters!!!!

Although I can install pyvirtualdisplay in python VE, I cannot run

sudo apy-get install xvfb

I don't own a personal desktop. Any suggestion?

KeepCalmAndCarryOn · Accepted Answer · 2013-10-09 00:01:25Z

1

You can connect phantomjs to Selenium.

It needs no X display at all http://phantomjs.org/

connect it to your selenium grid server like this

java -jar selenium-server-standalone-2.33.0.jar -role hub &

#bit flaky if selenium isn't up quietly goes away
sleep 5
phantomjs --webdriver=4001 --webdriver-selenium-grid-hub=http://127.0.0.1:4444 &

To add more to it add them on extra ports

phantomjs --webdriver=4002 --webdriver-selenium-grid-hub=http://127.0.0.1:4444 &

answered Oct 9, 2013 at 0:01

KeepCalmAndCarryOn

9,0962 gold badges34 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Patrick the Cat Over a year ago

Forgive me if I'm asking something stupid. I simply don't have selenium-server-standlone-x.xx.x.jar. I import selenium.webdriver and do all I want in Python. Can I still use this javascript lib?

KeepCalmAndCarryOn Over a year ago

I think the answer here is what you need stackoverflow.com/questions/13287490/… - specifically the answer by @Pykler

Collectives™ on Stack Overflow

Python: Headless Selenium without admin access

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related