192

Suppose I was given a URL.
It might already have GET parameters (e.g. http://example.com/search?q=question) or it might not (e.g. http://example.com/).

And now I need to add some parameters to it like {'lang':'en','tag':'python'}. In the first case I'm going to have http://example.com/search?q=question&lang=en&tag=python and in the second — http://example.com/search?lang=en&tag=python.

Is there any standard way to do this?

Improve this question
0

16 Answers 16

Reset to default
239

There are a couple of quirks with the urllib and urlparse modules. Here's a working example:

try:
    import urlparse
    from urllib import urlencode
except: # For Python 3
    import urllib.parse as urlparse
    from urllib.parse import urlencode

url = "http://stackoverflow.com/search?q=question"
params = {'lang':'en','tag':'python'}

url_parts = list(urlparse.urlparse(url))
query = dict(urlparse.parse_qsl(url_parts[4]))
query.update(params)

url_parts[4] = urlencode(query)

print(urlparse.urlunparse(url_parts))

ParseResult, the result of urlparse(), is read-only and we need to convert it to a list before we can attempt to modify its data.

Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

You probably want to use urlparse.parse_qs instead of parse_qsl. The latter returns a list whereas you want a dict. See docs.python.org/library/urlparse.html#urlparse.parse_qs.
@florian : At least in python 2.7 you then need to call urlencode as urllib.urlencode(query, doseq=True). Otherwise, parameters that existed in the original url are not preserved correctly (because they are returned as tuples from @parse_qs@
I've rewritten this to work in Python 3 as well. Code here.
The results of urlparse() and urlsplit() are actually namedtuple instances. Thus you can assign them directly to a variable and use url_parts = url_parts._replace(query = …) to update it.
Caution - this implementation removes repeated query parameters that some RESTful services use. With a little modification this can be fixed. query = urlparse.parse_qsl(url_parts[4]) query += params.items() But then if you want to replace exiting query params using dict, takes a little more.
|
105

Outsource it to the battle tested requests library.

This is how I will do it:

from requests.models import PreparedRequest
url = 'http://example.com/search?q=question'
params = {'lang':'en','tag':'python'}
req = PreparedRequest()
req.prepare_url(url, params)
print(req.url)
Improve this answer

2 Comments

It doesn't merge query string parameters in case if they should be overriden, so the url can be like http://example.com/search?q=question&q=someval.
there is not a reason to override the value of a given query parameter as multiple values for a parameter are widely supported. query params wiki.
73

Why

I've been not satisfied with all the solutions on this page (come on, where is our favorite copy-paste thing?) so I wrote my own based on answers here. It tries to be complete and more Pythonic. I've added a handler for dict and bool values in arguments to be more consumer-side (JS) friendly, but they are yet optional, you can drop them.

How it works

Test 1: Adding new arguments, handling Arrays and Bool values:

url = 'http://stackoverflow.com/test'
new_params = {'answers': False, 'data': ['some','values']}

add_url_params(url, new_params) == \
    'http://stackoverflow.com/test?data=some&data=values&answers=false'

Test 2: Rewriting existing args, handling DICT values:

url = 'http://stackoverflow.com/test/?question=false'
new_params = {'question': {'__X__':'__Y__'}}

add_url_params(url, new_params) == \
    'http://stackoverflow.com/test/?question=%7B%22__X__%22%3A+%22__Y__%22%7D'

Talk is cheap. Show me the code.

Code itself. I've tried to describe it in details:

from json import dumps

try:
    from urllib import urlencode, unquote
    from urlparse import urlparse, parse_qsl, ParseResult
except ImportError:
    # Python 3 fallback
    from urllib.parse import (
        urlencode, unquote, urlparse, parse_qsl, ParseResult
    )


def add_url_params(url, params):
    """ Add GET params to provided URL being aware of existing.

    :param url: string of target URL
    :param params: dict containing requested params to be added
    :return: string with updated URL
    
    >> url = 'https://stackoverflow.com/test?answers=true'
    >> new_params = {'answers': False, 'data': ['some','values']}
    >> add_url_params(url, new_params)
    'https://stackoverflow.com/test?data=some&data=values&answers=false'
    """
    # Unquoting URL first so we don't lose existing args
    url = unquote(url)
    # Extracting url info
    parsed_url = urlparse(url)
    # Extracting URL arguments from parsed URL
    get_args = parsed_url.query
    # Converting URL arguments to dict
    parsed_get_args = dict(parse_qsl(get_args))
    # Merging URL arguments dict with new params
    parsed_get_args.update(params)

    # Bool and Dict values should be converted to json-friendly values
    # you may throw this part away if you don't like it :)
    parsed_get_args.update(
        {k: dumps(v) for k, v in parsed_get_args.items()
         if isinstance(v, (bool, dict))}
    )

    # Converting URL argument to proper query string
    encoded_get_args = urlencode(parsed_get_args, doseq=True)
    # Creating new parsed result object based on provided with new
    # URL arguments. Same thing happens inside urlparse.
    new_url = ParseResult(
        parsed_url.scheme, parsed_url.netloc, parsed_url.path,
        parsed_url.params, encoded_get_args, parsed_url.fragment
    ).geturl()

    return new_url

Please be aware that there may be some issues, if you'll find one please let me know and we will make this thing better

Improve this answer

4 Comments

Perhaps add a try except with from urllib.parse to include Python 3 support? Thanks for the snippet, very useful!
Maybe add imports too ?
Unencodes encoded urls such as http://stackoverflow.com/with%2Fencoded?data=some&data=values&answe%2rs=false. Also, use three chevrons >>> to help doctests pick up your doctests
Why not change parsed_get_args = dict(parse_qsl(get_args)) to parsed_get_args = parse_qs(get_args)
66

You want to use URL encoding if the strings can have arbitrary data (for example, characters such as ampersands, slashes, etc. will need to be encoded).

Check out urllib.urlencode:

>>> import urllib
>>> urllib.urlencode({'lang':'en','tag':'python'})
'lang=en&tag=python'

In python3:

from urllib import parse
parse.urlencode({'lang':'en','tag':'python'})
Improve this answer

1 Comment

In python 3, this has been moved to urllib.parse.urlencode
29

You can also use the furl module https://github.com/gruns/furl

>>> from furl import furl
>>> print furl('http://example.com/search?q=question').add({'lang':'en','tag':'python'}).url
http://example.com/search?q=question&lang=en&tag=python
Improve this answer

1 Comment

add would add duplicated params, not overwrite existing param
28

If you are using the requests lib:

import requests
...
params = {'tag': 'python'}
requests.get(url, params=params)
Improve this answer

2 Comments

@chefhose the question is ... relative to what ? You are not in a web page, there is no context to be relative to.
Warning: requests.get actually performs the GET request. Might be obvious when you know requests but this answer is amongst all these other answers that are giving a string :)
17

Based on this answer, one-liner for simple cases (Python 3 code):

from urllib.parse import urlparse, urlencode


url = "https://stackoverflow.com/search?q=question"
params = {'lang':'en','tag':'python'}

url += ('&' if urlparse(url).query else '?') + urlencode(params)

or:

url += ('&', '?')[urlparse(url).query == ''] + urlencode(params)
Improve this answer

1 Comment

I know you mentioned "simple cases", but to clarify: it won't work properly if there is an ? in the anchor (#?stuff).
17

I find this more elegant than the two top answers:

from urllib.parse import urlencode, urlparse, parse_qs

def merge_url_query_params(url: str, additional_params: dict) -> str:
    url_components = urlparse(url)
    original_params = parse_qs(url_components.query)
    # Before Python 3.5 you could update original_params with 
    # additional_params, but here all the variables are immutable.
    merged_params = {**original_params, **additional_params}
    updated_query = urlencode(merged_params, doseq=True)
    # _replace() is how you can create a new NamedTuple with a changed field
    return url_components._replace(query=updated_query).geturl()

assert merge_url_query_params(
    'http://example.com/search?q=question',
    {'lang':'en','tag':'python'},
) == 'http://example.com/search?q=question&lang=en&tag=python'

The most important things I dislike in the top answers (they are nevertheless good):

  • Łukasz: having to remember the index at which the query is in the URL components
  • Sapphire64: the very verbose way of creating the updated ParseResult

What's bad about my response is the magically looking dict merge using unpacking, but I prefer that to updating an already existing dictionary because of my prejudice against mutability.

Improve this answer

1 Comment

I was about to write an answer and found this. (y)
11

Yes: use urllib.

From the examples in the documentation:

>>> import urllib
>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
>>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query?%s" % params)
>>> print f.geturl() # Prints the final URL with parameters.
>>> print f.read() # Prints the contents
Improve this answer

5 Comments

Can you please give some brief example?
f.read() will show you the HTML page. To see the calling url, f.geturl()
-1 for using a HTTP request for parsing a URL (which is actually basic string manipulation). Plus the actual problem is not considered, because you need to know how the URL looks like to be able to append the query string correctly.
Either the author edited question either this answer is not related to it.
for python 3 this is now: urllib.request.urlopen and urllib.parse.urlencode
11

python3, self explanatory I guess

from urllib.parse import urlparse, urlencode, parse_qsl

url = 'https://www.linkedin.com/jobs/search?keywords=engineer'

parsed = urlparse(url)
current_params = dict(parse_qsl(parsed.query))
new_params = {'location': 'United States'}
merged_params = urlencode({**current_params, **new_params})
parsed = parsed._replace(query=merged_params)

print(parsed.geturl())
# https://www.linkedin.com/jobs/search?keywords=engineer&location=United+States
Improve this answer

4 Comments

Watch out! This approach is using an internal function (indicated by the "_" before the function name) : _replace. It is not recommended to do so, because behavior of these internal functions may change or they may be removed without warning.
According to another stack overflow comment, admittedly from a few years ago, that's not true @GrazingScientist: stackoverflow.com/questions/21628852/…
@yellow-saint: You are absolutly right, even with Python 3.9 _replace is still a valid public method. I did not know that. So, thanks for pointing this out. Still, this is not the usual case for methods starting with an underscore ("_").
Just to add, _replace is part of the publicly advertised API, as it's documented in the docs: docs.python.org/3/library/… (One such method is _replace(). The _replace() method will return a new ParseResult object replacing specified fields with new values)
7

I liked Łukasz version, but since urllib and urllparse functions are somewhat awkward to use in this case, I think it's more straightforward to do something like this:

params = urllib.urlencode(params)

if urlparse.urlparse(url)[4]:
    print url + '&' + params
else:
    print url + '?' + params
Improve this answer

1 Comment

How about .query instead of [4] ?
5

Use the various urlparse functions to tear apart the existing URL, urllib.urlencode() on the combined dictionary, then urlparse.urlunparse() to put it all back together again.

Or just take the result of urllib.urlencode() and concatenate it to the URL appropriately.

Improve this answer

Comments

4

Yet another answer:

def addGetParameters(url, newParams):
    (scheme, netloc, path, params, query, fragment) = urlparse.urlparse(url)
    queryList = urlparse.parse_qsl(query, keep_blank_values=True)
    for key in newParams:
        queryList.append((key, newParams[key]))
    return urlparse.urlunparse((scheme, netloc, path, params, urllib.urlencode(queryList), fragment))
Improve this answer

Comments

2

In python 2.5

import cgi
import urllib
import urlparse

def add_url_param(url, **params):
    n=3
    parts = list(urlparse.urlsplit(url))
    d = dict(cgi.parse_qsl(parts[n])) # use cgi.parse_qs for list values
    d.update(params)
    parts[n]=urllib.urlencode(d)
    return urlparse.urlunsplit(parts)

url = "http://stackoverflow.com/search?q=question"
add_url_param(url, lang='en') == "http://stackoverflow.com/search?q=question&lang=en"
Improve this answer

Comments

2

Here is how I implemented it.

import urllib

params = urllib.urlencode({'lang':'en','tag':'python'})
url = ''
if request.GET:
   url = request.url + '&' + params
else:
   url = request.url + '?' + params

Worked like a charm. However, I would have liked a more cleaner way to implement this.

Another way of implementing the above is put it in a method.

import urllib

def add_url_param(request, **params):
   new_url = ''
   _params = dict(**params)
   _params = urllib.urlencode(_params)

   if _params:
      if request.GET:
         new_url = request.url + '&' + _params
      else:
         new_url = request.url + '?' + _params
   else:
      new_url = request.url

   return new_ur
Improve this answer

1 Comment

urllib has been split up in Python 3. The urllib.urlencode() function is now urllib.parse.urlencode(),
0

Using yarl:

from yarl import URL

url = 'http://stackoverflow.com/search?q=question'
params = {'lang': 'en', 'tag': 'python'}

url_obj = URL(url).with_query(params)
new_url = str(url_obj)
print(new_url)

(yarl is a dependency of aiohttp, so if you are using aiohttp, you already have yarl installed.)

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.