I'm grabbing a series of links from a website with python and BS4 but i need to clean them up so I only get the URL in the string.
the links i get look like this:
javascript:changeChannel('http://some-server.com/with1234init.also', 20);
and i need it to look like this
Well, if all the links are like that one you can do it with a very simple approach:
s.split("'")[1]
For example:
>>>s="javascript:changeChannel('http://some-server.com/with1234init.also', 20);"
>>>s.split("'")
['javascript:changeChannel(',
'http://some-server.com/with1234init.also',
', 20);']
str = javascript:changeChannel('http://some-server.com/with1234init.also', 20);
formattedtext ="http://" + str.split("http://")[1].split(',')[0].strip("'")
A reasonably robust way is to take your chunk of text and search it with a URL-matching regex pattern.
See also:
Python regular expression again - match url
which links to here: http://daringfireball.net/2010/07/improved_regex_for_matching_urls
Extracting URL link using regular expression re - string matching - Python
Using regex...
import re
re.search(pattern, text)
... or
re.findall(pattern, text)
A full example...
>>> p = re.compile(r'(?i)\b((?:https?://|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:\'".,<>?«»“”‘’]))')
or
>>> p = re.compile('(?i)\\b((?:https?://|www\\d{0,3}[.]|[a-z0-9.\\-]+[.][a-z]{2,4}/)(?:[^\\s()<>]+|\\(([^\\s()<>]+|(\\([^\\s()<>]+\\)))*\\))+(?:\\(([^\\s()<>]+|(\\([^\\s()<>]+\\)))*\\)|[^\\s`!()\\[\\]{};:\\\'".,<>?\xc2\xab\xc2\xbb\xe2\x80\x9c\xe2\x80\x9d\xe2\x80\x98\xe2\x80\x99]))')
>>> m = p.search("javascript:changeChannel('http://some-server.com/with1234init.also', 20);")
>>> m.group()
'http://some-server.com/with1234init.also'
the pattern used is from the web URL version in the above link
Note the use of the r prefix and the escaped ' quote towards the end in the first pattern.
using re.compile caches the regex pattern
