I want to write an address in the address bar of a browser as well as click on a link using java Robot class. How can I track the different objects in a certain window?
1 Answer
Just giving a look at the API http://docs.oracle.com/javase/7/docs/api/java/awt/Robot.html, anyone who do this should know that via Robot Class there is no "trackComponent(Component specificComponent)" method, you got 2 things that may help you:
1-getPixelColor (more than help, seems useless for you by now, maybe i'm wrong).
2-createScreenCapture.
the second method is maybe the answer for your problem, you could take a picture of the screen and with some image processor (javaCV could help you on this: https://code.google.com/p/javacv/) you could then track the components on the screen you took (for instance: from pixels xxx to pixels yyy is the Address bar of browser), of course you need to read some documentation about javaCV (OpenCV) for get this done, after that just use the method for move cursor and enter keys for fill the components, hope someone give a simpler way to do this, but i think this way you learn a bit of JavaCV a really powerful tool.
4 Comments
PeakGen
JavaCV is not powerful, but OpenCV is. JavaCV is a wrapper. Instead of moving with JavaCV, this can be done by using OpenCV in C++ and connecting it to Java via JNI, because very low learning resources are there for JavaCV. Apart from that, x and y locations of the image will not help if the image got resized. Instead, move wih thresholding, edge find, or to the best - template matching
jac1013
@Knight topic says java, i know OpenCV is the real thing (as you said JavaCV is only a wrapper) but since he only will need the image processor feature JavaCV could do the work (poor documentation on this? maybe yes, but if he doesn't know how to C++ it's better to give it a try than trying to learn C++ head first) and about thresholding, edge find or template matching, you are more than right + 1, x and y locations is not the best approach but what you mean with image being resized? is a screenshot to be processed, why should exist a resizing there? resolution issue? that could be fixed.
PeakGen
By resize what I mean is, think the address bar location is coded as x=125, y=124 when the web browser is "maximized". User decides re-size the browser. Now the x=125,y=124 theory is not going to work.
jac1013
ok, you are right but why would i take the screenshot then wait let's say 10 seconds and then analize the screenshot and do the Robot things? if we are going to do the Robot Class things is an instant process, take screenshot-analizePicture-doMovesAndKeyPress, this occurs in miliseconds, i mean the user won't even notice that cursor moved (if we move it back to the initial position before the automated movements and clicks).
CTRL-LandALT-Dseem to be the most standard shortcuts to get to the address bar (CMD-Lfor Mac, of course).