Tao of Mac

May 11th 2012 · 1 min read · #cache #cli #http #varnish

Varnish

Overview

Varnish is one of my favorite building blocks for scalable web services – it can dramatically improve performance, and with a few configuration tweaks it can even provide per‑session caching and other niceties.

It is used as this site’s primary front-end and has served well so far.

Capturing varnishncsa Output For Analysis

I often explore access logs with . HTTP logs are easy to parse but can be hard to mine for meaningful patterns, so I capture structured data including cache handling and back‑end response times.

Capture Script

import os, sys, csv, time
from subprocess import *

# the fields we want
fields = ['timestamp','hostname','namespace','page','result','size','responsetime','cache']

child = Popen(["varnishncsa","-F","%r,%s,%b,%{Varnish:time_firstbyte}x,%{Varnish:handling}x"], stdout=PIPE)

line = child.stdout.readline()
o = csv.writer(open('output.csv','wb+'))

# output the header
o.writerow(fields)

while line:
    (req,result,size,responsetime,cache) = line.strip().split(',')
    (method,url,protocol) = req.split(' ')
    try:
        (dummy,dummy,hostname,namespace,page) = url.split('/',4)
    except:
        (dummy,dummy,hostname,dummy) = url.split('/',3)
        namespace = page = ''
    timestamp = str(time.time())
    if size == '-':
        size = ''
    row = [globals()[x] for x in fields]
    print ','.join(row)
    o.writerow(row)
    line = child.stdout.readline()

Example Output

timestamp,hostname,namespace,page,result,size,responsetime,cache
1336745775.29,taoofmac.com,space,HOWTO/Setup/daapd,200,9076,0.038134813,miss
1336745775.39,taoofmac.com,themes,serif/css/serif-min.css,200,5997,0.000071287,hit
1336745775.49,taoofmac.com,themes,serif/js/site-min.js,200,41993,0.000078678,hit
1336745775.84,taoofmac.com,themes,serif/img/noise.png,200,8431,0.000062704,hit
1336745775.84,taoofmac.com,themes,serif/img/sitelogo_2011.png,200,16346,0.000038862,hit
1336745775.94,taoofmac.com,themes,serif/img/error.png,200,666,0.000059605,hit
1336745784.61,the.taoofmac.com,space,RecentChanges?format=rss,302,167,0.000052452,hit
1336745787.52,planet.taoofmac.com,,,404,358,0.000077963,hit
1336745797.29,the.taoofmac.com,,,200,129190,0.094822168,miss
1336745802.3,the.taoofmac.com,space,HOWTO/Merge%20Folders,304,0,0.006964445,pass

This page is referenced in: