Scraping data dynamically generated by JavaScript

Question

I tried using PhantomJSDriver for scraping this URL: http://www.tsetmc.com/loader.aspx?ParTree=151311&i=67126881188552864

1) First, in this URL we must click on the upper blue link named: سابقه

2) Then I want scrap this XPath: //*[@id="trade"]/div[2]/table/tbody/tr[9]/td[7]

Here is my code but this code can't find required XPath:

var driverService = PhantomJSDriverService.CreateDefaultService();
var driver = new PhantomJSDriver(driverService);

driver.Url = "http://www.tsetmc.com/loader.aspxParTree=151311&i=67126881188552864";
driver.Navigate();
var source = driver.PageSource;
driver.FindElementByXPath("//*[@id=\"tabs\"]/div/ul/li[3]/a").Click();

Thread.Sleep(5000);

try
{
    var pathElement = driver.FindElementByXPath("//[@id=\"trade\"]/div[2]/table/tbody/tr[9]/td[7]");
    MessageBox.Show(pathElement.Text);
}
catch
{
    driver.Dispose();
    this.Close();
}
driver.Dispose();

confusedMind · Accepted Answer · 2018-08-30 13:49:47Z

4

I would recommend to directly hit this url:

http://members.tsetmc.com/tsev2/data/InstTradeHistory.aspx?i=67126881188552864&Top=999999&A=0

And then parse the values as this contains all the values basically on all the 209 pages so it will save you also the trouble of moving from page 1 -209.

And it looks easy split by ; gives you all the rows and then split by @ gives you the values per column.

answered Aug 30, 2018 at 13:49

confusedMind

2,6637 gold badges37 silver badges78 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Scorpian Z Over a year ago

but how can you find this URL?

confusedMind Over a year ago

@ScorpianZ open firefox (open console) or fiddler hit the URL when it runs you will see the above URL getting loaded (ajax call ).

Collectives™ on Stack Overflow

Scraping data dynamically generated by JavaScript

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related