crawling web data using python html error

Question

i want to crawling data using python i tried tried again but it didn't work i can not found code's error i wrote code like this:

import re
import requests
from bs4 import BeautifulSoup

url='http://news.naver.com/main/ranking/read.nhn?mid=etc&sid1=111&rankingType=popular_week&oid=277&aid=0003773756&date=20160622&type=1&rankingSectionId=102&rankingSeq=1'
html=requests.get(url)
#print(html.text)
a=html.text
bs=BeautifulSoup(a,'html.parser')
print(bs)
print(bs.find('span',attrs={"class" : "u_cbox_contents"}))

i want to crawl reply data in news

as you can see, i tried to searing this:

span, class="u_cbox_contents" in bs

but python only say "None"

None

so i check bs using function print(bs)

and i check up bs variable's contents

but there is no span, class="u_cbox_contents"

why this happing?

i really don't know why

please help me

thanks for reading.

Community · Accepted Answer · 2017-05-23 12:08:29Z

1

Requests will fetch the URL's contents, but will not execute any JavaScript.

I performed the same fetch with cURL, and I can't find any occurrence of u_cbox_contents in the HTML code. Most likely, it's injected using JavaScript, which explains why BeautifulSoup can't find it.

If you need the page's code as it would be rendered in a "normal" browser, you could try Selenium. Also have a look at this SO question.

edited May 23, 2017 at 12:08

CommunityBot

11 silver badge

answered Jun 30, 2016 at 10:26

DocZerø

8,60511 gold badges44 silver badges75 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

crawling web data using python html error

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related