Deep packet inspection (DPI) hardware can identify an astonishing array of protocols passing across the Internet—up to and including protocols that are rare even to us in the Orbiting HQ (Gadu-Gadu? Manolito? Feidian?). But if you've ever wondered just how this can be done, and done at wire speed, wonder no more: Europe's leading DPI vendor has open-sourced a version of its traffic detection engine.

OpenDPI.org is the new home for ipoque's open source project; anyone interested can take a look at the code or contribute patches. The goal in this case, though, isn't so much about crowdsourcing product development but about easing consumer fears about DPI technology.

Klaus Mochalski, CEO of ipoque, explains that "transparency was important for us from the beginning. The lack of transparency from the vendors' side is widespread in the DPI business. Our thoughts are a bit different and that is why we decided to push this project."

The OpenDPI engine, released under the LGPL license, differs from ipoque's commercial scanning engine in its high-priced DPI hardware. The open-source version is much slower and (more importantly) doesn't reveal ipoque's methods for identifying encrypted transmissions. DPI vendors all claim high levels of success at identifying such traffic based on the flow patterns and handshake signatures common to protocols like BitTorrent and Skype, even if they cannot crack the encryption and examine the content of those transmissions.

The OpenDPI engine will identify a huge list of non-encrypted protocols, however:

P2P File Sharing: BitTorrent, eDonkey , KaZaa/Fasttrack, Gnutella, WinMX, DirectConnect, AppleJuice, Soulseek, XDCC, Filetopia, Manolito, iMesh, Pando

BitTorrent, eDonkey , KaZaa/Fasttrack, Gnutella, WinMX, DirectConnect, AppleJuice, Soulseek, XDCC, Filetopia, Manolito, iMesh, Pando Voice over IP: SIP, IAX, RTP

SIP, IAX, RTP Instant Messaging: Yahoo, Oscar, IRC, unencrypted Jabber, Gadu!Gadu, MSN

Yahoo, Oscar, IRC, unencrypted Jabber, Gadu!Gadu, MSN Streaming Protocols : ORB, RTSP, Flash, MMS, MPEG, Quicktime, Joost, WindowsMedia, RealMedia, TVAnts, SOPCast, TVUPlayer, PPStream, PPLive, QQLive, Zattoo, VeohTV, AVI, Feidian, Ececast, Kontiki, Move, RTSP, SCTP, SHOUTcast

: ORB, RTSP, Flash, MMS, MPEG, Quicktime, Joost, WindowsMedia, RealMedia, TVAnts, SOPCast, TVUPlayer, PPStream, PPLive, QQLive, Zattoo, VeohTV, AVI, Feidian, Ececast, Kontiki, Move, RTSP, SCTP, SHOUTcast Tunnel Protocols: IPsec,GRE, SSL, SSH, IP in IP

IPsec,GRE, SSL, SSH, IP in IP Standard Protocols: HTTP, Direct download links (1-click file hosters), POP, SMTP, IMAP, FTP, BGP, DHCP, DNS, EGP, ICMP, IGMP, MySQL, NFS, NTP, OSPF, pcAnywhere, PostgresSQL, RDP, SMB, SNMP, SSDP, STUN, Telnet, Usenet, VNC, IPP, MDNS, NETBIOS, XDMCP, RADIUS, SYSLOG, LDAP

HTTP, Direct download links (1-click file hosters), POP, SMTP, IMAP, FTP, BGP, DHCP, DNS, EGP, ICMP, IGMP, MySQL, NFS, NTP, OSPF, pcAnywhere, PostgresSQL, RDP, SMB, SNMP, SSDP, STUN, Telnet, Usenet, VNC, IPP, MDNS, NETBIOS, XDMCP, RADIUS, SYSLOG, LDAP Gaming Protocols: World of Warcraft, Half-Life, Steam, Xbox, Quake, Second Life

ipoque apparently wants to convince people that its detection code doesn't store or examine the actual content being transmitted. The company made the same point in a white paper released last week. "DPI as such has no negative impact on online privacy," it says. "It is, again, only the applications that may have this impact. Prohibiting DPI as a technology would be just as naive as prohibiting automatic speech recognition because it can be used to eavesdrop on conversations based on content. Although DPI can be used as a base technology to look at and evaluate the actual content of a network communication, this goes beyond what we understand as DPI as it is used by Internet bandwidth management—the classification of network protocols and applications."

DPI can (and does) go much further than this, of course; it is used by law enforcement to grab complete copies of particular users' Internet datastreams in investigations, and companies like NebuAd (now defunct) and Phorm (still funct) use it to examine the URLs being visited by users in order to better target advertising to them. ipoque's paper admits to such uses, but calls them "beyond the scope of this paper."

Releasing its detection engine for analysis is meant to allay fears that ipoque's traffic management DPI is a "bad" application of the technology. "By giving the general public access to parts of our DPI engine, we want to demonstrate that many of the alleged privacy violations simply do not happen in DPI bandwidth management systems," says the company, though plenty of Internet users dislike DPI for reasons that have little to do with privacy and have much more to do with concerns over things like network neutrality (however one defines that idea).

But at least we now know how to identify a Second Life connection:

>if ((ntohs(packet->udp->dest) == 12035 || ntohs(packet->udp->dest) == 12036 || (ntohs(packet->udp->dest) >= 13000 && ntohs(packet->udp->dest) <= 13050)) //port && packet->payload_packet_len > 6 // min length with no extra header, high frequency and 1 byte message body && get_u8(packet->payload, 0) == 0x40 // reliable packet && ntohl(get_u32(packet->payload, 1)) == 0x00000001 // sequence number equals 1 //ntohl (get_u32 (packet->payload, 5)) == 0x00FFFF00 // no extra header, low frequency message - can't use, message may have higher frequency ) { IPQ_LOG(IPOQUE_PROTOCOL_SECONDLIFE, ipoque_struct, IPQ_LOG_DEBUG, "Second Life detected.n"); ipoque_int_secondlife_add_connection(ipoque_struct); return;