I work in the NYTimes.com feeds team. We handle retrieving, parsing and transforming incoming feeds from whatever strange proprietary format our partners choose to give us into something that our CMS can digest. As you can imagine, we deal with a huge amount of text processing. To handle all of these transformations as efficiently as possible we rely heavily on the magic of Perl. Recently, as feeds become more and more important, we have begun to feel pains caused by past impromptu segments of inefficient code written to meet quick, episodic deadlines. A situation that we are especially prone to as a fast moving news organization.

I am a relatively new employee here at NYTimes.com and one of my responsibilities is to create tools to help ensure the integrity and scalability of our code. To this end, I would like to introduce you to The New York Times Perl Profiler, or Devel::NYTProf. The purpose of this tool is to allow developers to easily profile Perl code line-by-line with minimal computational overhead and highly visual output. With only one additional command, developers can generate robust color-coded HTML reports that include some useful statistics about their Perl program. Here is the typical usage:

perl -d:NYTProf myslowcode.pl

nytprofhtml

See? It’s easy! nytprofhtml is an implementation of the included reporting interface (Devel::NYTProf::Reader). If you don’t want HTML reports, you can implement your own format with relative ease. If you create something cool, be sure to let me know via CPAN patch request or open@nytimes.com. Detailed instructions can be found in the documentation and source code on CPAN.

You can see sample screen shots of the html report’s index pageand a single module report.

Similar tools exist to profile Perl code. Devel::DProf is the ubiquitous profiler, but it only collects information about subroutine calls. Because of this limitation, its not all that helpful in finding that elusive broken regex in a 75-line subroutine of regex transforms. Devel::FastProf is another per-line profiler, however I found its output difficult to coerce into HTML. It also doesn’t support non-Linux systems (we need at least Solaris and Ubuntu/Linux support).

Devel::NYTProf is available as a distribution on the CPAN. You may install by typing “install Devel::NYTProf” in the ‘cpan’ command-line application, or manually by downloading the tarball from CPAN.

We were able to reduce the long runtime on one particular application by 20% (about a minute) after the very first test run of our profiler. We hope that you will find our tool as useful as we have. Of course, any comments and suggestions are welcome!