Rui CarmoOpinions on shortened URLs are a dime a dozen these days, but the basic facts are:
- They’re awfully convenient for passing around (and this was true even before Twitter came about)
- They are, by nature, short-lived (either the services or the URLs)
- You should never rely on their being around later on
So basically you have absolutely no excuse to not be able to handle them. I decided to mess around with the concept a few weeks back to see how simple I could make it all work, and came up with a couple of useful Python classes that I can share with the world:
Creating short URLs
The trouble with creating short URLs is that there are entirely too many shortening services, and far too many variations on APIs – in fact, nearly all of them suffer from “not invented here” syndrome and try to “enhance” their APIs to give you a lot of stuff that you basically don’t (ever) need, and wrap their results in JSON or XML
Me, I refuse to put up with that kind of crap.
So I poked around a bit, found the simplest services to work against and created the following class, which will try all its known services in turn until it gives you a working URL:
import urllib, urllib2, urlparse, httplib
BITLY_AUTH = 'login=foo&apiKey=bar'
class URLShortener:
services = {
'api.bit.ly':
"http://api.bit.ly/shorten?version=2.0.1&%s&format=text&longUrl=" % BITLY_AUTH,
'api.tr.im': '/api/trim_simple?url=',
'tinyurl.com': '/api-create.php?url=',
'is.gd': '/api.php?longurl='
}
def query(self, url):
for shortener in self.services.keys():
c = httplib.HTTPConnection(shortener)
c.request("GET", self.services[shortener] + urllib.quote(url))
r = c.getresponse()
shorturl = r.read().strip()
if ("Error" not in shorturl) and ("http://" + urlparse.urlparse(shortener)[1] in shorturl):
return shorturl
else:
continue
raise IOError
Yes, the error handling is naïve – any network exceptions and stuff ought to be caught upstream from this – but it works fine so far.
Expanding short URLs
This is the really fun bit, because it is not immediately obvious whether or not a short URL will actually be immediately useful – there are plenty of times when you’ll actually be redirected to something else, and while fooling around with the Google Reader API (something I’ll eventually write about alter), I found that also applied (in spades) to Feedburner links and whatnot.
So I decided to build some smarts into the process and have it not only ping some known hosts twice, but also turn it into a link checker of sorts, and learning which hosts were actually redirecting to other places:
import urllib, urllib2, urlparse, httplib
class URLExpander:
# known shortening services
shorteners = ['tr.im','is.gd','tinyurl.com','bit.ly','snipurl.com','cli.gs',
'feedproxy.google.com','feeds.arstechnica.com']
twofers = [u'\u272Adf.ws']
# learned hosts
learned = []
def resolve(self, url, components):
""" Try to resolve a single URL """
c = httplib.HTTPConnection(components.netloc)
c.request("GET", components.path)
r = c.getresponse()
l = r.getheader('Location')
if l == None:
return url # it might be impossible to resolve, so best leave it as is
else:
return l
def query(self, url, recurse = True):
""" Resolve a URL """
components = urlparse.urlparse(url)
# Check weird shortening services first
if (components.netloc in self.twofers) and recurse:
return self.query(self.resolve(url, components), False)
# Check known shortening services first
if components.netloc in self.shorteners:
return self.resolve(url, components)
# If we haven't seen this host before, ping it, just in case
if components.netloc not in self.learned:
ping = self.resolve(url, components)
if ping != url:
self.shorteners.append(components.netloc)
self.learned.append(components.netloc)
return ping
# The original URL was OK
return url
This one’s a bit more convoluted but has turned out to be very useful indeed, and you can simply pickle the whole object to preserve its learned hosts.