Strip URL in Python

Question

I'm quite new to python. I'm trying to parse a file of URLs to leave only a specific part (bold part) of URL.

Here are some example of URL's i am working with:

http://www.mega.pk/**washingmachine**-dawlance/
http://www.mega.pk/**washingmachine**-haier/
http://www.mega.pk/**airconditioners**-acson/
http://www.mega.pk/**airconditioners**-lg/
http://www.mega.pk/**airconditioners**-samsung/

I have tried some regular expression but it gets very complicated. What I have in mind is remove this "http://www.mega.pk/" from all urls as it is common and then remove anything that is after "-" including all slashes. But know no way of doing it.

nichinungas · Accepted Answer · 2020-05-13 09:49:50Z

5

Use the urllib (formerly urlparse) module. It's built specifically for this purpose.

from urllib.parse import urlparse

url = "http://www.mega.pk/washingmachine-dawlance/"

path = urlparse(url).path  # get the path from the URL ("/washingmachine-dawlnace/")
path = path[:path.index("-")]  # remove everything after the '-' including itself
path = path[1:]  # remove the '/' at the starting of the path (just before 'washing')

path variable will have the value washingmachine

Cheers!

edited May 13, 2020 at 9:49

nichinungas

136 bronze badges

answered Oct 6, 2015 at 9:46

narendranathjoshi

3,1462 gold badges21 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Avinash Raj · Accepted Answer · 2015-10-06 09:27:45Z

3

Use re.sub

re.sub(r'^.*\/([^/]*)-.*', r'\1', line)

DEMO

Example:

>>> s = '''http://www.mega.pk/washingmachine-dawlance/
    http://www.mega.pk/washingmachine-haier/'''.splitlines()
>>> for line in s:
    print(re.sub(r'^.*\/([^/]*)-.*', r'\1', line))


washingmachine
washingmachine
>>>

answered Oct 6, 2015 at 9:27

Avinash Raj

175k32 gold badges247 silver badges289 bronze badges

Comments

Alex · Accepted Answer · 2015-10-06 09:35:17Z

You can achieve the same without using a regular expression. The solution proposed by Avinash is cleaner, but the below approach might be easier to understand, especially if you want to modify it at some point:

s = '''http://www.mega.pk/washingmachine-dawlance/
http://www.mega.pk/washingmachine-haier/'''.splitlines()
for line in s:    
   cleanedUrl = line.replace('http://www.mega.pk/**','').replace('/','')
   urlParameters = cleanedUrl.split('-')
   print urlParameters[-1]

Or if you prefer, you can use a compacter version:

s = '''http://www.mega.pk/washingmachine-dawlance/
http://www.mega.pk/washingmachine-haier/'''.splitlines()
for line in s:    
    print line.replace('http://www.mega.pk/**','').replace('/','').split('-')[-1]

Mayur Koshti · Accepted Answer · 2015-10-06 12:40:53Z

1

You can also use the following regex:

>>> import re
>>> regex = r"(?is)^\s*https?\:\/\/[^\/]+\/(.*?)(?:\-|\/).*"
>>> res = re.sub(regex, r"\1", PROVIDE_URL)

You will get desire result.

answered Oct 6, 2015 at 12:40

Mayur Koshti

1,89218 silver badges21 bronze badges

Comments

Eugene Soldatov · Accepted Answer · 2015-10-06 09:38:26Z

0

use re.sub in couple with lambda function:

re.sub(r'^http://www.mega.pk/(\w+)-.*', lambda x: x.group(), line)

answered Oct 6, 2015 at 9:38

Eugene Soldatov

10.2k2 gold badges39 silver badges44 bronze badges

Comments

Mirko · Accepted Answer · 2021-08-20 16:37:48Z

0

You can also do this using urljoin:

import urllib.parse


url = "http://www.mega.pk/**washingmachine**-dawlance/"

website = urllib.parse.urljoin(url, '/')

print(website)

answered Aug 20, 2021 at 16:37

Mirko

2,5112 gold badges23 silver badges19 bronze badges

1 Comment

uak Over a year ago

result of your solution is the base url without the path, whoever it was useful for me as I only needed the base url

Collectives™ on Stack Overflow

Strip URL in Python

6 Answers 6

Comments

Comments

Comments

Comments

Comments

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

Comments

Comments

Comments

Comments

Comments

1 Comment

Related