Startup benchmarks for Mojo, Catalyst, Titanium, HTTP::Engine, CGI::Application and CGI.pm

What’s the minimum time it could take a serve a web page using a given Perl-based solution using CGI? What’s the minimum amount of memory it would take?

To check the relative differences between several options listed below, I made the simplest possible “Hello World” page, and benchmarked the time and memory used to serve it. To create a base line, I also measured the results for an empty Perl script that just prints “Hello World”.

The result summary is after the jump.

If it’s not already clear, these benchmarks are not meant to represent real world applications or web servers. Of course, physics dictate that would be difficult of any application to perform any faster than these times, using the same systems and hardware.

System Version Startup*Memory**
Empty Script Perl 5.8.8 0.01s 4.7M
CGI::Simple 1.103 0.08s 5.5M
CGI.pm 3.42 0.11s 5.7M
CGI::Application 4.20 0.21s 6.0M
Mojo 0.8.9 0.29s 8.3M
Mojolicious 0.8.9 0.31s 8.8M
Titanium 1.0 0.14s 6.0M
Titanium w/Dispatch 1.0 0.39s 10.1M
Catalyst 5.8000_03 0.80s 13.5M
HTTP::Engine 0.0.18 1.50s 14.9M
HTTP::Engine/Shika 0.0.99_01 0.19s 6.6M
* The “real” time in seconds as timed with `time -p perl foo.pl` on a 1.1 Ghz ThinkPad T23 laptop with Ubuntu 8.04 Linux. ** The virtual memory size when the script is about to exit, as timed by printing the output of this at the end of the script run: `ps -o pid,user,rss,vsize,ucomm -C perl`;

Let’s go through some of the numbers.

CGI::Simple, a lighter-weight CGI.pm alternative turns in the best time and memory consumption. Before considering using it, note that it has a different interface in a few places, like file upload handling, and there are still some cases that CGI.pm can handle but it can’t, like some kinds of file uploads.

Next there’s much-maligned CGI.pm, still appearing here with a respectable second-fastest start time and low memory consumption. Both CGI::Simple and CGI.pm help process an HTTP request and prepare an HTTP response, which is functionality targeted by most of the options listed here.

CGI::Application is next on the list. It explicitly doesn’t provide any of the functionality of CGI.pm itself, and I don’t think CGI.pm would even be loaded for this bare-bones “Hello World” case. CGI::Application provides basic application structure, control flow, and a callback system for plugins. As a long time user, It’s no surprise to confirm that CGi::Application is a very lightweight solution.

Mojo is next. It’s functionality includes supporting several server backends and abstracting the HTTP server response and request. This is a bit similar to what CGI.pm does, since CGI.pm also supports mod_perl and FastCGI as well has helping with the HTTP request and response. (Of course the Mojo and CGI approaches couldn’t be more different!). Mojo also provides a similar amount of functionality as HTTP::Engine. Mojo’s result shows that it can be a rather lightweight solution itself, perhaps suitable for running in a CGI environment. Overall, the performance of Mojo is impressive here.

Next up we have Mojolicious and Titanium, which had very similar speed benchmarks. Mojolicious adds dispatching and a file-extension based templating system to Mojo, including the ability to serve static pages and a simple built-in templating system. It’s perhaps similar to the amount of functionality that CGI::Application covers when combined with CGI.pm and HTML::Template, but again the approach is very different.

Of the group of systems tested here, I would say Titanium covers the largest scope of functionality, since it goes beyond usual framework offerings, including additional methods for managing database handles, config files, sessions, form validation and form filling. I often run Titanium-style apps under CGI with good performance. In a persistent environment, the smaller memory footprint would be a more significant win here, allowing more application processes to be run concurrently.

Second to last on the list we have Catalyst, the Perl framework which needs no introduction. It’s slower startup time is not particularly considered a concern here, as it is not targeted at being deployed in CGI environments. That it uses a bit more memory does show some kind of additional overhead, although for many deployments a bit extra memory per-process won’t matter.

Finally, taking far more time and still more memory, we have HTTP::Engine. Considering its scope of functionality competes with Mojo, this performance and memory overhead is disappointing. HTTP::Engine explicitly has a CGI interface, but taking 1.5 seconds and almost 15 megs of memory just to print “Hello World” is dismal. Update 12/1/2008- HTTP::Engine replaced “Moose” with “Shika” and massive speed and memory gains, apparently without any reduction in functionality. The run time of 0.19 seconds and memory usage of 6.6 megs is rather respectable.

I can already anticipate the feedback to this article saying that the startup times don’t matter because much of this is compile-time work, which doesn’t apply to persistent-environments, because compilation would only then happen once at server start-up time, and is not per-request. And there’s the perception I expect to hear echo’ed back again: No one uses straight CGI anymore, right?

As a professional website developer with over a decade of experience, nearly all the web applications I’ve written have run in CGI, and have performed just fine this way. Why do anything more complicated, when the simple solution works?

A web framework is a bit like an operating system, it’s there to prop up the layer where the real value-added work happens. For an OS, that’s the application. For a web-app, that’s the custom application logic. If your web application is going to be slow and hog memory it should because your custom application does that, not because the framework itself takes almost two seconds to boot up and uses 15 megabytes of memory before your application really gets started.

If your application is not expected to be high-traffic, several of the solutions mentioned here are candidates for running under CGI, saving the complexity and overhead of deploying in FastCGI or modperl. If you creating your applications with best practices like avoiding global variables, it will likely be easy to convert it to run under modperl or FastCGI later.

As a disclosure, some of the testing scripts I used are below. For Catalyst, Mojo, and Mojolicious, I used the default CGI scripts that came bundled with them.

CGI.pm

use CGI;
my $q = CGI->new;
print $q->header;
print "Hello from CGI\n";

CGI::Application

package HelloWorld;
use base 'CGI::Application';

sub setup {
    my $self = shift;
    $self->start_mode('hello_world');
    $self->run_modes([qw/hello_world/]);

}

sub hello_world {
    my $self = shift;
    return "Hello World\n".
}

HelloWorld->new->run;


# Titanium with Dispatching, dispatch.pl
use CGI::Application::Dispatch;
CGI::Application::Dispatch->dispatch(
    default => 'hello-world/hello_world',
);

Titanium with Dispatching, HelloWorld.pm

package HelloWorld;
use base 'Titanium';

sub setup {
    my $self = shift;
    $self->start_mode('hello_world');
    $self->run_modes([qw/hello_world/]);

}

sub hello_world {
    my $self = shift;
    return "Hello World\n";     
}
1;


# HTTP::Engine
use HTTP::Engine;
my $engine = HTTP::Engine->new(
      interface => {
          module => 'CGI',
          request_handler => 'main::handle_request',
      },
  );
$engine->run;

sub handle_request {
      my $req = shift;
      HTTP::Engine::Response->new( body => "Hello from HTTP::Engine\n\n");
}

Empty.pll

print "Hello World\n";

6 Comments

you might want to check this out: http://svn.coderepos.org/share/lang/perl/App-Benchmark-WAF/trunk/

it uses Apache::Test and hence mod_perl, but the relative figures shouldn't be that different

Thanks. Has anyone published their results from this benchmark?

Hey.

How come you are using a developer snapshot of Catalyst rather than the latest release version (5.7015)?

Marcus,

Thanks for stopping by. I wanted the article to be relevant for as long as possible and also reflect the latest work of the team, so I chose the developer snapshot. I expected the team had already done some benchmarking with it to confirm it reflects a direction desired to head it.

diasuke,

You say:

the relative figures shouldn't be that different.

But I would be surprised if they would be very similar, because the compile-time effort wouldn't be a factor, which should improve the relative performance of Catalyst and HTTP::Engine considerably, I think.

I tried to run the benchmark myself but it soon failed because it couldn't find a Catalyst dependency...."parent.pm".

Mark

Honestly, you are comparing apples to oranges here and your benchmarks are heavily stacked in favor of your argument.

CGI.pm is so far away from HTTP::Engine it is not even funny. The most they have in common is that they are both meant to be used for web programming, beyond that they both have very different goals and approaches. Additionally Catalyst does so much more then CGI::Application, which is primarily concerned only with simple dispatch, and HTTP::Engine, which doesn't even do dispatch, can not usefully compared to either. Titanium, Mojolicious and Catalyst do share some degree of similarity, but they all tend to approach what they do in different ways and therefore make different tradeoffs which result in different types of performance in different types of environments.

No one uses straight CGI anymore, right?

I have been doing web development for over a decade as well, for several different companies and I have never done straight CGI. I have converted straight CGI scripts to mod_perl and occasionally had to fix a CGI script that broke, but I have never actually written something that ran as vanilla CGI. If you take a look around outside of the Open Source world, you will see pretty much no one at all is doing plain CGI and hasn't been for years. Even Classic ASP and Cold Fusion (oh the horror) had a persistent interpreter of some kind.

In my opinion, there is no excuse to still use vanilla CGI as it is simply just a waste of resources, even the simplest CGI script would likely run better/faster on FCGI or mod_perl with minimal effort to get runing. There really are just much more interesting problems out there to solve then making something run fast under vanilla CGI.

- Stevan

Leave a comment

Recent Entries

Close