Constructing a url with query strings

Question

How do I properly construct urls with query strings?

For example, from a website, I scrape the value www.abc.com/SomethingHere?x=1&y=2 however, the value I get uplon scraping is www.abc.com/SomethingHere?x=1&y=2 sometimes there's wierd %xx at the end I don't understand. Requests made with these modified strings fail (but are ok if I manually remove the amp and percentage wierdness). It also makes me afraid of adding more query parameters with just www.abc.com/SomethingHere?x=1&y=2&z=3

How do I make sure I get the proper urls?

Aidan Kane · Accepted Answer · 2014-12-18 02:19:26Z

2

Do it in two steps:

import urllib

# first parse the url
>>> parsed = urllib.parse.urlparse('www.abc.com/SomethingHere?x=1&amp;y=2')
>>> parsed
ParseResult(scheme='', netloc='', path='www.abc.com/SomethingHere', params='', query='x=1&amp;y=2', fragment='')

# the parse the query string component (into a dictionary)
>>> q = parsed.query
>>> urllib.parse.parse_qs(q)
{'y': ['2'], 'x': ['1']}

answered Dec 18, 2014 at 2:19

Aidan Kane

4,0162 gold badges29 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

nitishagar · Accepted Answer · 2014-12-18 01:12:22Z

0

You can have a look at urlparse in python (here). Calling urlparse on your query, we get something like:

urlparse('www.abc.com/SomethingHere?x=1&amp;y=2&z=3')
Output: ParseResult(scheme='', netloc='', path='www.abc.com/SomethingHere', params='', query='x=1&amp;y=2&z=3%%xx', fragment='')

For modifying query params you can further use urljoin, as follows:

urljoin('www.abc.com/SomethingHere?x=1&amp;y=2&z=3%%xx', '?x=2')
Output: 'www.abc.com/SomethingHere?x=2'

edited Dec 18, 2014 at 1:12

answered Dec 18, 2014 at 1:07

nitishagar

9,4213 gold badges32 silver badges41 bronze badges

1 Comment

yayu Over a year ago

Thank you. How to get rid of amp; and other idiosyncracies sytematically (i.e, other than str.replace)

Collectives™ on Stack Overflow

Constructing a url with query strings

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related