Unable to loop through navigable string with BeautifulSoup CSS Selector

Question

I would like to extract the content inside the p tag below.

<section id="abstractSection" class="row">
<h3 class="h4">Abstract<span id="viewRefPH" class="pull-right hidden"></span>
</h3>
<p> Variation of the  (<span class="ScopusTermHighlight">EEG</span>), has functional and. behavioural effects in sensory <span class="ScopusTermHighlight">EEG</span We can interpret our. Individual <span class="ScopusTermHighlight">EEG</span> text to extract <span class="ScopusTermHighlight">EEG</span> power level.</p>
</section>

A one line Selenium as below,

document_abstract = WebDriverWait(self.browser, 20).until(
    EC.visibility_of_element_located((By.XPATH, '//*[@id="abstractSection"]/p'))).text

can extract easily the p tag content and provide the following output:

Variation of the EEG, has functional and. behavioural effects in sensoryEEG. We can interpret our. Individual EEG text to extract EEG power level.

Nevertheless, I would like to employ the BeautifulSoup due to speed consideration.

The following bs by referring to the css selector (i.e.,#abstractSection ) was tested

url = r'scopus_offilne_specific_page.html'
with open(url, 'r', encoding='utf-8') as f:
    page_soup = soup(f, 'html.parser')
    home=page_soup.select_one('#abstractSection').next_sibling
    for item in home:
        for a in item.find_all("p"):
            print(a.get_text())

However, the compiler return the following error:

AttributeError: 'str' object has no attribute 'find_all'

Also, since Scopus require login id, the above problem can be reproduce by using the offline html which is accessible via this link.

May I know where did I do wrong, appreciate for any insight

rpb · Accepted Answer · 2020-06-16 19:39:43Z

0

Thanks to this OP, the problem issued above apparently can be solve simply as below

document_abstract=page_soup.select('#abstractSection > p')[0].text

answered Jun 16, 2020 at 19:39

rpb

3,3073 gold badges32 silver badges72 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Unable to loop through navigable string with BeautifulSoup CSS Selector

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related