I’ve recently been collaborating with Christian Hansen on HTTP::Tiny, a minimalist, HTTP/1.1 client library for Perl. For basic web client tasks like grabbing a single page or mirroring a file, it does the job in a fraction of the code that would be needed to install LWP::UserAgent. Because it has no non-core dependencies, it is ideal for what I need to get CPAN.pm to bootstrap itself with pure Perl.

Here’s a quick look at how you would use it for the two common tasks I mentioned:

use HTTP::Tiny; my $http = HTTP::Tiny->new; my $response; # get a single page $response = $http->get('http://example.com/'); die "Failed!

" unless $response->{success}; print $response->{content}; # mirror a file $response = $http->mirror('http://example.com/file.tar.gz', 'file.tar.gz'); die "Failed!

" unless $response->{success}; print "Unchanged!

" if $response->{status} eq '304';

The example above is almost the same as you’d get using LWP::UserAgent with one big exception. HTTP::Tiny returns the response as a hash reference rather than as an object. Just like LWP::UserAgent, the mirror method will send an If-Modified-Since header for an existing file to skip downloading if the file is unchanged. HTTP::Tiny doesn’t (yet) handle query parameters – you have to prepare those yourself, but for simple downloads, you don’t generally need those anyway.

Where did HTTP::Tiny come from? When I was working on getting CPAN.pm to support a pure-Perl HTTP bootstrap, I started with HTTP::Lite. When I discussed it on #p5p, Christian Hansen pointed out a number of serious shortcomings and decided that it would be easier for him to write a new, lightweight HTTP/1.1 client from scratch rather than try to redo the plumbing in HTTP::Lite.

The result is HTTP::Tiny and I’ve been collaborating with Christian to get it ready for use by CPAN.pm and ready for the Perl core. Unlike HTTP::Lite, HTTP::Tiny is a conditionally conforming HTTP/1.1 client. It supports both redirection and mirroring, which HTTP::Lite does not, both of which are important features for a CPAN client.

As a “Tiny” module, HTTP::Tiny achieves its HTTP/1.1 conformance in a just a fraction the code required for LWP::UserAgent and its non-core dependents. Don’t get me wrong – LWP::UserAgent is a great piece of software. But sometimes – like for the Perl core or for a fatpacked application – something much smaller and simpler will do just as well.

Let’s see what a difference there is. CPANdeps shows us all of LWP::UserAgent’s dependencies and highlights the ones that are non-core (as of Perl 5.10.1):

I used David A. Wheeler’s SLOCCOUNT to count lines of code in each of the distributions containing these modules as well as for HTTP::Lite and HTTP::Tiny (based on the “soon-to-be” 0.007 release). In all cases, I excluded files in t/ and examples directories. (Lines of Perl includes any programs distributed with the distribution.)

Here are the results:

Distribution .pm files Lines (Perl) Lines (C) Total Lines ----------------- --------- ------------ ---------- ----------- libwww-perl-5.837 52 10258 0 10258 HTML-Parser-3.68 7 883 1972 2855 HTML-Tagset-3.20 1 139 0 139 URI-1.56 52 2715 0 2715 ----- ----- ----- ----- TOTAL LWP & friends 112 13995 1972 15967 versus Distribution .pm files Lines (Perl) Lines (C) Total Lines ----------------- --------- ------------ ---------- ----------- HTTP-Lite-2.3 1 634 0 634 HTTP-Tiny-0.007 1 603 0 603

Wow! Both HTTP::Lite and HTTP::Tiny accomplish simple HTTP tasks with less than 4% of the SLOC of LWP and its non-core dependencies.

What about memory usage? Here’s the “null” case that shows memory usage of just loading the three client modules:

VSZ RSS COMMAND 30440 5512 perl -MLWP::UserAgent -e 1 while 1 26700 3960 perl -MHTTP::Tiny -e 1 while 1 26040 3168 perl -MHTTP::Lite -e 1 while 1

HTTP::Lite is smallest, but it also doesn’t have the features or conformance I need.

That’s not much of a real-world test, so let’s try something else – downloading the 950K CPAN 02packages.details.txt.gz file. Here is the test code for HTTP::Tiny and LWP::UserAgent side by side:

# http-tiny-mirror.pl # lwp-useragent-mirror.pl #!/usr/bin/env perl #!/usr/bin/env perl use strict; use strict; use warnings; use warnings; use HTTP::Tiny; use LWP::UserAgent; my ($url, $file) = @ARGV; my ($url, $file) = @ARGV; die unless $url & $file; die unless $url & $file; my $http = HTTP::Tiny->new; my $http = LWP::UserAgent->new; my $res = $http->mirror($url, $file); my $res = $http->mirror($url, $file); die "Failed!

" die "Failed!

" unless $res->{success}; unless $res->is_success; 1 while 1; 1 while 1;

What about HTTP::Lite? It doesn’t have a mirror method and doesn’t do redirection, which I get for free with HTTP::Tiny and LWP::UserAgent. I’d have to write dozens of lines of code to emulate that for a “fair” test, so I skipped it.

Let’s add those two programs (after downloading the 02packages file to files with different names) to our memory benchmarks:

VSZ RSS COMMAND 50916 9824 perl ./lwp-useragent-mirror.pl [...] 35376 4348 perl ./http-tiny-mirror.pl [...] 30440 5512 perl -MLWP::UserAgent -e 1 while 1 26700 3960 perl -MHTTP::Tiny -e 1 while 1 26040 3168 perl -MHTTP::Lite -e 1 while 1

HTTP::Tiny wins – not by a huge amount in absolute terms, admittedly, but if you don’t need the extra features that LWP::UserAgent offers, HTTP::Tiny might just be all you need.

Thank you, Christian, for HTTP::Tiny!