Go and wash your mouth with SOAP

It’s been a long while since I posted something with any amount of code, so I thought I’d try my hand at it again.

Almost a year ago I put to words what I thought about Go, and I’m pleased to say I things have improved markedly – to the point where I found myself using it of late.

Besides the strong points I outlined earlier, it’s matured beyond the shortcomings I came across, with a few extras rolled into the bargain:

Stupendous ease in cross-compiling¹
Noticeable improvements in performance
More (battle-tested) libraries

It’s not, however, a breathtaking language, or one that I feel comfortable using when a modicum of abstraction is required. Even with first-order functions and fairly flexible maps, it’s hard to achieve the kind of lofty high-level judo moves that I’ve come to cherish of late in LISP country – in fact, most of them are impossible by design, which may or may not be your cup of tea.

In a nutshell, and drawing a parallel with “conventional” IT shop Java, Go feels like a blue collar language that you can build stuff on and maintain according to broadly consensual, standardized patterns. “Coding in the large”, right?

Although I still find it somewhat horrific to fish dependencies directly out of Github and Launchpad as a matter of course, the terse, focused tooling and mostly uniform formatting make it easy to get into. So much so, in fact, that after you get used to its particular idioms it’s almost boring in its straightforwardness.

So I tried using it to do really boring stuff – stuff that I’ve done many times already in Objective-C, Java and Python, and that I hate with a passion like the fire of a thousand suns:

I used it to invoke a SOAP service on our Enterprise Service Bus.

This is a sort of masochistic kata I invariably end up doing in every programming language I use, partly because I have to and partly because it’s the kind of real-life problem that hipster developers underestimate. It also usually requires you to bang your head repeatedly against a number of walls before it works properly, which is still considered (by and large) to be a valid, character-forming approach to education.

SOAP isn’t something Go does out of the box², so I had to dig a fair bit in order to figure out how to achieve what I wanted. As far as I know, and given Go‘s focus on more modern approaches to web services, there isn’t any “best practice” on how to go about doing this (if there is, by all means write about it someplace, like I’m doing now).

For Python, I have a staple solution that serves me very well – pysimplesoap, to which I contributed in the past, and which will parse just about any WSDL file you throw at it and dynamically create any number of proxy classes, readily accessible through late binding. It’s terrifying and wonderful and profoundly disturbing all at the same time, but works extremely well and (after it builds all the proxies) plenty fast enough.

For Java, there’s wsimport – which I use with Clojure/Java interop – or heavy-handed approaches like “The Axis Of Pain” and other enterprisey libraries. All told, your mileage may vary, but it’s usually quite long.

But let’s get back to Go. I picked a service I know very well – I won’t tell you what it is exactly, but let’s call it QueryEntity for the sake of argument (as always, names were changed to protect the guilty).

Here’s a typical QueryEntity request:

<?xml version="1.0" encoding="UTF-8"?>
<Envelope xmlns="http://schemas.xmlsoap.org/soap/envelope/" xmlns:envelope="http://schemas.xmlsoap.org/soap/envelope/" envelope:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
    <Header xmlns="http://schemas.xmlsoap.org/soap/envelope/">
        <ESBCredentials xmlns="http://esb/definitions">
            <ESBUsername xmlns="http://esb/definitions">user@esb</ESBUsername>
            <ESBPassword xmlns="http://esb/definitions">changeme</ESBPassword>
        </ESBCredentials>
    </Header>
    <Body xmlns="http://schemas.xmlsoap.org/soap/envelope/">
        <QueryEntity xmlns="http://esb/queryservice">
            <ClientGUID>CA55E77E-1962-0000-2014-DEC1A551F1ED</ClientGUID>
            <QueryString>wally</QueryString>
        </QueryEntity>
    </Body>
</Envelope>

For the sake of clarity, I’m using the non-namespace-prefixed tag style, and I’ve also considerably simplified the payload. You’ll notice right away that this uses XML header auth (the whole thing goes atop SSL anyway and the services usually support multiple auth methods like temporary tokens and whatnot, so this is actually A Good Thing).

Now, how does one go about doing this in Go, especially considering that you want to invoke several services with different payloads in the envelope Body?

The standard answer in Go is to use struct field tagging, like so:

type Envelope struct {
    XMLName       xml.Name       `xml:"http://schemas.xmlsoap.org/soap/envelope/ Envelope"`
    EncodingStyle string         `xml:"http://schemas.xmlsoap.org/soap/envelope/ encodingStyle,attr"`
    Header        EnvelopeHeader `xml:"http://schemas.xmlsoap.org/soap/envelope/ Header"`
    Body          EnvelopeBody   `xml:"http://schemas.xmlsoap.org/soap/envelope/ Body"`
}

This is what the Go xml/encoding package uses to generate corresponding tags for each field (via reflection). And it is fine and good until you realize that you’re going to have multiple distinct XML Body payloads.

It would be borderline insane to go about defining as many Envelope subtypes as SOAP services, so what do you do?

Well, you use an empty interface{}, which allows you to assign other types:

// This has to change depending on which service we invoke
type EnvelopeBody struct {
    Payload interface{} // the empty interface lets us assign any other type
}

But there’s a catch – if you just go in blindly and assign a QueryEntity struct to the Payload field , you’ll get this after you run it through the XML encoder:

...
    <Body xmlns="http://schemas.xmlsoap.org/soap/envelope/">
        <Payload>
            <QueryEntity>
                <ClientGUID>CA55E77E-1962-0000-2014-DEC1A551F1ED</ClientGUID>
                <QueryString>wally</QueryString>
            </QueryEntity>
        </Payload>
    </Body>
</Envelope>

…which the ESB will merrily tell you to go stuff somewhere else, because it doesn’t match the service contract.

The solution is to define your types like so:

import (
    "./esb" // the full ESBCredentials type is defined elsewhere
    ...
)

// Sample data
var ESBUsername = "user@esb"
var ESBPassword = "changeme"
var ClientGUID = "CA55E77E-1962-0000-2014-DEC1A551F1ED"

type Envelope struct {
    XMLName       xml.Name       `xml:"http://schemas.xmlsoap.org/soap/envelope/ Envelope"`
    EncodingStyle string         `xml:"http://schemas.xmlsoap.org/soap/envelope/ encodingStyle,attr"`
    Header        EnvelopeHeader `xml:"http://schemas.xmlsoap.org/soap/envelope/ Header"`
    Body          EnvelopeBody   `xml:"http://schemas.xmlsoap.org/soap/envelope/ Body"`
}

// This is fixed independently of whatever port we call
type EnvelopeHeader struct {
    Credentials *esb.ESBCredentials `xml:"http://esb/definitions ESBCredentials"`
}

// This has to change depending on which service we invoke
type EnvelopeBody struct {
    Payload interface{} // the empty interface lets us assign any other type
}

type QueryEntity struct {
    // having an XMLName field lets you re-tag (and rename) the Payload
    XMLName       xml.Name `xml:"http://esb/queryservice QueryEntity"`
    ClientGUID   *string
    QueryString  *string
}

The XMLName field and accompanying tag is an idiomatic construct whose only purpose is to override the XML generation logic so that the Payload field is never mapped into the resulting XML.

This took me a fair while to figure out, and is the reason for the fairly lengthy example – which isn’t quite finished yet, for I’ve yet to fill in the whole thing and perform the actual request, which is done like so:

func CreateQuery(query string) *Envelope {
    // Build the envelope (this could be farmed out to another func)
    retval := &Envelope{}
    retval.EncodingStyle = "http://schemas.xmlsoap.org/soap/encoding/"
    retval.Header = EnvelopeHeader{}
    retval.Header.Credentials = &esb.ESBCredentials{}
    retval.Header.Credentials.ESBUsername = &ESBUsername
    retval.Header.Credentials.ESBPassword = &ESBPassword

    // Build the payload that matches our desired SOAP port
    payload := QueryEntity{}
    payload.ClientGUID = &HouseholdId
    payload.QueryString = query

    // ...and in the darkness bind it
    retval.Body.Payload = payload
    return retval
}


func InvokePitsOfDarkness() {
    ...
    buffer := &bytes.Buffer{}
    encoder := xml.NewEncoder(buffer)
    envelope := CreateQuery("wally")
    err := encoder.Encode(envelope)
    // this is just a test, so let's panic freely
    if err != nil {
        log.Panic("Could not encode request")
    }

    client := http.Client{}
    req, err := http.NewRequest("POST", "https://esb/service", buffer)
    if err != nil {
        log.Panic(err.Error())
    }

    req.Header.Add("SOAPAction", "\"http://esb/service/QueryEntity\"")
    req.Header.Add("Content-Type", "text/xml")
    resp, err := client.Do(req)
    if err != nil {
        log.Panic(err.Error())
    }
    if resp.StatusCode != 200 {
        log.Panic(resp.Status)
    }

    result, err := GoElsewhereToDecode(resp.Body)
    ...
}

Decoding the actual result is left as an exercise to the reader – and I haven’t finished it myself for all the possible return values, but I’m in the process of learning how to use reflection to save me the trouble of setting up various kinds of structs to bind the return data to.

It bears noting that, for the moment, I’m still using Python to do this in “production”, since there are no noticeable benefits from switching languages at this point (and no point in doing silly benchmarks in something that doesn’t have any timing or resource constraints). But I can see myself taking that down and using Go instead in a few months, once I have a better feel for the language³.

Incidentally, the equivalent code in Clojure isn’t pretty either – wsimport works for me, but a “nicer” approach using clojure.data.xml didn’t go very far because it currently doesn’t know about XML namespaces – I had to hack those in after the fact, but if I ever find a nice, clean and reusable solution I’ll post about it.

And now, back to less technical stuff. I’ve been mulling product management and business development again, and I’m likely to have something to say on those during the upcoming weeks.

Which comes in especially handy if, like me, you like to tinker with resource-constrained ARM devices, for which Go seems to be a natural fit – I can build a binary on my Mac and run it directly on a development board without the usual gcc shenanigans, which makes me very happy. ↩︎
Realistically, it’s not something anything ought to do out of the box, unless you belong to that group of people who believed the likes of COBOL, ASN.1 and X.400 were spiffing ideas. As far as my experience goes, XML shares a usage pattern with violence – even when it’s not the right solution or it plain doesn’t work, people just try to use more of it. ↩︎
There’s also the matter of Go being anointed as suitable for inclusion in our ecosystem, but that’s another story entirely. When I joined up Python was a niche thing, and now every time I muster the time to deliver a talk on it the room fills to capacity… ↩︎

Tao of Mac

Go and wash your mouth with SOAP