What I learned about the semantic web by traveling too much

Last year I had the ‘privilege’ of amassing enough air miles on Delta Airlines to ‘earn’ the status of Diamond Medallion – with a few miles to spare to help me get it again this year, with (maybe) a little less travel. The quotes are not meant to demean the benefits you get with the status at all – I sincerely appreciate the shorter line at security, access to the lounge and the fairly regular upgrades – but rather to point out that traveling that much, well, sort of sucks. One of my colleagues (who doesn’t get the joy of traveling as much as I do) recently asked me if there was a level above Diamond, to which I replied, yes….it’s called Divorce 😉
However, all clouds have a silver lining and in this case its a very practical lesson about how far we’ve come in getting software to do real work for us, instead of making more work for us to do. It all started with a little application for my iPhone and iPad called Tripit. I discovered Tripit through the use of another app that a fellow traveller in a long forgotten airport lounge told me about, Flight Tracker. The Flight Tracker app gives you a map with an overlay of where your flight is at any given time and has and integration with Tripit to feed it all your flight number info. I’ve long ago deleted Flight Tracker from my devices (ended up being pretty useless since I am the one flying and can’t track myself without a network – even if I could, what’s the point?), but use Tripit on a (far too) frequent basis.
My basic description of Tripit is an app that consolidates all of my travel information into one itinerary. Sounds pretty simple and to the casual observer it actually is. But if you look deeper, you realize that there’s some magic in there somewhere. The way Tripit gets my travel details is by me forwarding all of the confirmation emails I get from the flights I book, hotels I reserve and cars I rent. As soon as the hit my inbox, I forward them along to trips@tripit.com and forget about them. Tripit knows where to stick my information based on the reply-address in the email I receive. It infers a relationship between the data in the email and my account by looking at the email’s meta data. Pretty simplistic semantic leap, I know, but we’re just getting started.
What Tripit does next is where the real magic starts. Without having any prior knowledge of the format, structure or data in the email I am sending it, it is able to deconstruct the message into its core elements, and stick them into its data structure. It finds flight numbers, seat numbers, confirmation numbers and departure / arrival times from confirmation emails sent by United, Delta, Southwest and American (all the ones I have tried so far). It finds check-in, check-out, rate and confirmation info from every random hotel I have stayed at for the last year in America, Europe and Asia. It combines all this info with rental car, restaurant, tour reservations, and anything else that I can cram into my agenda by reserving ahead of time into one chronologically ordered agenda. What’s more, all the info looks the same – an airline booking with Delta looks just like one from United. A hotel reservation with the Westin looks just like one from the Hilton. As icing on the cake, it adds in a google map from airport to hotel with estimated drive times and any tolls.
Based on my experience with Tripit and its ability to digest anything I send it and turn it into structured data, I started to think about potential applications of the same principles to manufacturing (the whole reason I have to travel in the first place). Where is the product development process do we have massive amounts of unstructured data that would benefit from being put into a more structured format? The first spot that came to mind is requirements gathering. In the requirements gathering phase (which BTW I think is a giant misnomer since it shouldn’t really be a phase, but a continuous process) there are hundreds, thousands (or even millions depending on how large the market / customer base is and how long the product development cycles are) of suggestions, complaints and requests about new or existing product features.
These inputs come in the form of emails, call logs, and increasingly tweets and blog posts. Imagine a tool that would let you dump all of that input into a giant hopper and the most requested capabilities and changes would be assembled and prioritized – structured – for you in an easy to consume dashboard. Take it a step further and rather than a batch process (dumping all the inputs in at one time and seeing what comes out the other side), imagine a continuous process where new ideas, comments and suggestions where always being fed into the hopper and the top priorities and ideas where constantly in touch with the real time needs of the market. Clearly this will require some advancement from the sort of engine that Tripit uses that maps unstructured data to a predefined structured model to one that can analyze the unstructured information, define the categories based on what it finds and then map everything it finds in the rest of the input data into those categories. That might seem like a big leap, but don’t fear – search engines like Google and Bing are doing very similar things today, just to web pages rather than emails.
Terrible idea? Never work? Am I full of shit? Maybe…but this is one of those ideas that got stuck in my head like a bad song – it just keeps coming back up no matter how hard I try to think about something else. History is on my side as well: the number of ideas that have moved from the consumer to the business space in the last 10 years would make this post insufferably long if I tried to list them all here. So I am going to keep toying with it and trying to figure out if there is something there. Feel free to help me out – if you have other ideas about other spots in the product development / manufacturing loop where a magic box to convert a mess into something structured would be helpful, chime in with a comment below. On the flip side, if you see some big holes – hit me up with those as well. From 36,000 feet over Denver, that’s all for now.