You can use urlparse (called urllib.parse in 3.x) to separate the URL into each of its components:
>>> from urlparse import urlparse
>>> ipv4address = urlparse("http://1.2.3.4:80")
>>> ipv4address
ParseResult(scheme='http', netloc='1.2.3.4:80', path='', params='', query='', fragment='')
>>> ipv6address = urlparse("http://[fec2::10]:80")
>>> ipv6address
ParseResult(scheme='http', netloc='[fec2::10]:80', path='', params='', query='', fragment='')
Then you can split the port off by finding the index of the last colon using rfind:
>>> ipv4address.netloc.rfind(':')
7
>>> ipv4address.netloc[:7], ipv4address.netloc[8:]
('1.2.3.4', '80')
>>> ipv6address.netloc.rfind(':')
10
>>> ipv6address.netloc[:10], ipv6address.netloc[11:]
('[fec2::10]', '80')
Identifying which type it is should then be as simple as if ':' in that_split_tuple[0], right? (Not 100% sure because it's been a while since I learned about how to write IPv6 addresses in URLs.)
Finally, removing the brackets from your IPv6 address is simple, there are many ways to do it:
>>> ipv6address.netloc[:10].replace('[', '').replace(']', '')
'fec2::10'
>>> ipv6address.netloc[:10].strip('[]')
'fec2::10'
Edit: since you expressed concern about not always having port numbers, you could simplify significantly by using a regular expression:
>>> import re
>>> f = lambda(n): re.split(r"(?<=\]):" if n.startswith('[') else r"(?<=\d):", n)
>>> f(ipv4address.netloc)
['1.2.3.4', '80']
>>> f(ipv6address.netloc)
['[fec2::10]', '80']
>>> f("1.2.3.4")
['1.2.3.4']
>>> f("[fec2::10]")
['[fec2::10]']
(I'm having trouble being more clever with my regular expression, hence the inline ternary.)