Python Extract Specific Data from Webscraping

Question

I'm trying to extract specific data from a webpage using this code:

from bs4 import BeautifulSoup
import requests

source = requests.get('https://www.dailyfx.com/sentiment-report').text

soup = BeautifulSoup(source, 'lxml')
#print(soup.prettify())
price = soup.find_all('span', {'class' : 'gsstx'})


print(price)

but it just prints out everything in the gsstx span class like this:

, Retail trader data shows 76.23% of traders are net-long with the ratio of traders long to short at 3.21 to 1. , We typically take a contrarian view to crowd sentiment, and the fact traders are net-long suggests USD/CHF prices may continue to fall. , Retail trader data shows 31.96% of traders are net-long with the ratio of traders short to long at 2.13 to 1. , We typically take a contrarian view to crowd sentiment, and the fact traders are net-short suggests USD/JPY prices may continue to rise. , **Retail trader data shows 44.10% of traders are net-long with the ratio of traders short to long at 1.27 to 1.** , We typically take a contrarian view to crowd sentiment, and the fact traders are net-short suggests Wall Street prices may continue to rise. ]

How can I just print out this part?

Retail trader data shows 44.10% of traders are net-long with the ratio of traders short to long at 1.27 to 1.

Danish Bansal · Accepted Answer · 2021-12-23 16:43:53Z

1

For extracting and saving in csv, use following code (SPECIFIC TO THIS USE CASE AS SKIP MACHANSIM WILL NOT BE SAME IN ALL CASES)

from bs4 import BeautifulSoup
import requests

source = requests.get('https://www.dailyfx.com/sentiment-report').text

soup = BeautifulSoup(source, 'lxml')
#print(soup.prettify())
price = soup.find_all('span', {'class' : 'gsstx', 'style':"font-weight:bold;"})
lis = []
skip = 0
for i in price:
    if skip%2==0:
        lis.append(i.get_text())
    skip+=1

df = pd.DataFrame({"Data":lis})
df.to_csv("Data.csv",index=False)

answered Dec 23, 2021 at 16:43

Danish Bansal

7202 gold badges8 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

IsDobbyFree Over a year ago

Thanks again Danish, works perfectly, just one more thing is it possible to parse the data so I'm only getting the percentage figures and the figures at the end so it would be something like 45.48% 1.20:1?

Danish Bansal Over a year ago

You can always use regex

Danish Bansal · Accepted Answer · 2021-12-23 16:23:42Z

Following is the solution to get the bolded text

from bs4 import BeautifulSoup
import requests

source = requests.get('https://www.dailyfx.com/sentiment-report').text

soup = BeautifulSoup(source, 'lxml')
#print(soup.prettify())
price = soup.find_all('span', {'class' : 'gsstx', 'style':"font-weight:bold;"})
for i in price:
    print(i.get_text())

Output is as below

Retail trader data shows 45.48% of traders are net-long with the ratio of traders short to long at 1.20 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-short suggests AUD/JPY prices may continue to rise. 
Retail trader data shows 56.52% of traders are net-long with the ratio of traders long to short at 1.30 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-long suggests AUD/USD prices may continue to fall. 
Retail trader data shows 58.91% of traders are net-long with the ratio of traders long to short at 1.43 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-long suggests Oil - US Crude prices may continue to fall. 
Retail trader data shows 44.67% of traders are net-long with the ratio of traders short to long at 1.24 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-short suggests Germany 30 prices may continue to rise. 
Retail trader data shows 76.68% of traders are net-long with the ratio of traders long to short at 3.29 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-long suggests EUR/CHF prices may continue to fall. 
Retail trader data shows 73.68% of traders are net-long with the ratio of traders long to short at 2.80 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-long suggests EUR/GBP prices may continue to fall. 
Retail trader data shows 39.30% of traders are net-long with the ratio of traders short to long at 1.54 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-short suggests EUR/JPY prices may continue to rise. 
Retail trader data shows 59.76% of traders are net-long with the ratio of traders long to short at 1.49 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-long suggests EUR/USD prices may continue to fall. 
Retail trader data shows 28.01% of traders are net-long with the ratio of traders short to long at 2.57 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-short suggests France 40 prices may continue to rise. 
Retail trader data shows 34.28% of traders are net-long with the ratio of traders short to long at 1.92 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-short suggests FTSE 100 prices may continue to rise. 
Retail trader data shows 39.83% of traders are net-long with the ratio of traders short to long at 1.51 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-short suggests GBP/JPY prices may continue to rise. 
Retail trader data shows 60.04% of traders are net-long with the ratio of traders long to short at 1.50 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-long suggests GBP/USD prices may continue to fall. 
Retail trader data shows 78.26% of traders are net-long with the ratio of traders long to short at 3.60 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-long suggests Gold prices may continue to fall. 
Retail trader data shows 62.12% of traders are net-long with the ratio of traders long to short at 1.64 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-long suggests NZD/USD prices may continue to fall. 
Retail trader data shows 96.23% of traders are net-long with the ratio of traders long to short at 25.54 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-long suggests Silver prices may continue to fall. 
Retail trader data shows 49.36% of traders are net-long with the ratio of traders short to long at 1.03 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-short suggests US 500 prices may continue to rise. 
Retail trader data shows 48.09% of traders are net-long with the ratio of traders short to long at 1.08 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-short suggests USD/CAD prices may continue to rise. 
Retail trader data shows 77.47% of traders are net-long with the ratio of traders long to short at 3.44 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-long suggests USD/CHF prices may continue to fall. 
Retail trader data shows 32.86% of traders are net-long with the ratio of traders short to long at 2.04 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-short suggests USD/JPY prices may continue to rise. 
Retail trader data shows 41.61% of traders are net-long with the ratio of traders short to long at 1.40 to 1. 
We typically take a contrarian view to crowd sentiment, and the fact traders are net-short suggests Wall Street prices may continue to rise. 
1

For skipping use following code (Specific to the case)

from bs4 import BeautifulSoup
import requests

source = requests.get('https://www.dailyfx.com/sentiment-report').text

soup = BeautifulSoup(source, 'lxml')
#print(soup.prettify())
price = soup.find_all('span', {'class' : 'gsstx', 'style':"font-weight:bold;"})
skip = 0
for i in price:
    if skip%2==0:
        print(i.get_text())
    skip+=1

New Output

Retail trader data shows 45.48% of traders are net-long with the ratio of traders short to long at 1.20 to 1. 
Retail trader data shows 56.52% of traders are net-long with the ratio of traders long to short at 1.30 to 1. 
Retail trader data shows 58.91% of traders are net-long with the ratio of traders long to short at 1.43 to 1. 
Retail trader data shows 44.67% of traders are net-long with the ratio of traders short to long at 1.24 to 1. 
Retail trader data shows 76.68% of traders are net-long with the ratio of traders long to short at 3.29 to 1. 
Retail trader data shows 73.68% of traders are net-long with the ratio of traders long to short at 2.80 to 1. 
Retail trader data shows 39.30% of traders are net-long with the ratio of traders short to long at 1.54 to 1. 
Retail trader data shows 59.76% of traders are net-long with the ratio of traders long to short at 1.49 to 1. 
Retail trader data shows 28.01% of traders are net-long with the ratio of traders short to long at 2.57 to 1. 
Retail trader data shows 34.28% of traders are net-long with the ratio of traders short to long at 1.92 to 1. 
Retail trader data shows 39.83% of traders are net-long with the ratio of traders short to long at 1.51 to 1. 
Retail trader data shows 60.04% of traders are net-long with the ratio of traders long to short at 1.50 to 1. 
Retail trader data shows 78.26% of traders are net-long with the ratio of traders long to short at 3.60 to 1. 
Retail trader data shows 62.12% of traders are net-long with the ratio of traders long to short at 1.64 to 1. 
Retail trader data shows 96.23% of traders are net-long with the ratio of traders long to short at 25.54 to 1. 
Retail trader data shows 49.36% of traders are net-long with the ratio of traders short to long at 1.03 to 1. 
Retail trader data shows 48.09% of traders are net-long with the ratio of traders short to long at 1.08 to 1. 
Retail trader data shows 77.47% of traders are net-long with the ratio of traders long to short at 3.44 to 1. 
Retail trader data shows 32.86% of traders are net-long with the ratio of traders short to long at 2.04 to 1. 
Retail trader data shows 41.61% of traders are net-long with the ratio of traders short to long at 1.40 to 1.

Collectives™ on Stack Overflow

Python Extract Specific Data from Webscraping

2 Answers 2

2 Comments

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related