Application server performance testing, includes Django, ErlyWeb, Rails and others

The following represents a set of toy benchmark results for various web application servers including ErlyWeb, Hunchentoot, Django and other application servers. The goal was to test out of the box performance through a simple VIEW page. For example, I was trying to avoid static files but going through the controller and the view. In the Rails example, there is a simple controller that then activates the view template. The same approach is used with the ErlyWeb example. Ideally, we are trying to avoid the server loading the static files but in our case the server may be loading cached content. Overall, Erlang's ErlyWeb/Yaws had great performance and out of the box Django did very well. Tomcat and my various low-level simple JSP/Servlet apps also had stable performance.

Software and Hardware Configuration used in the Benchmark

Two machines were used in the benchmark, a Win32 dual core machine and a single CPU Linux machine.
Win32 Box:

• Microsoft Windows XP, Vers 2002, Service Pack 2
  
• Intel Core 2 CPU
  
• 6300 @ 1.86GHz, 3.49 GB of Ram
  
• Tomcat: Apache Tomcat Version 5.5.26
  
• Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_11-b03)
  
• Java HotSpot(TM) Client VM (build 1.5.0_11-b03, mixed mode)
  
• Ruby: ruby 1.8.7 (2008-06-20 patchlevel 22) [i386-cygwin]
  
• Ruby: Rails 2.1.0
  
• JRuby: 1.1.13
  
• Python: Python 2.5.1, Cygwin
  
• Python: Django 0.96
  
• Erlang: Eshell V5.6, R12B-3 Jun 9, 2008
  
• Erlang: Yaws/Erlyweb: 1.77 ; erlyweb-0.7.2
  
• Lisp: SBCL 1.0.13 Win32 (WITHOUT THREAD SUPPORT)
  
• Lisp: Hunchentoot: 0.15.7

Linux Box:

• Linux houston 2.6.24-19-generic #1 SMP Fri Jul 11 23:41:49 UTC 2008 i686 GNU/Linux
  
• Ubuntu Hardy Heron 8.04
  
• GNOME 2.22.2
  
• 3.0 GB of Ram
  
• AMD Athlon 64 Processor 3200+ (running at 2.0GHz)


• Python: Python 2.5.2
  
• Ruby: ruby 1.8.6 (2007-09-24 patchlevel 111) [i486-linux]
  
• java version "1.6.0_06"
  
• Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
  
• Java HotSpot(TM) Client VM (build 10.0-b22, mixed mode, sharing)
  
• Apache Tomcat Version 5.5.23
  
• Erlang: Eshell V5.6, R12B-3 Jun 9, 2008
  
• Lisp: SBCL 1.0.14
  
• Lisp: Hunchentoot: 0.15.7

Overview of the tests run

There were 10 tests run on windows and 5 tests run on linux. Most of the source and setups for each test are at the bottom of this entry. The tests are only used to demonstrate the out of the box configurations. There weren't any added optimizations or additional proxy servers used. I literally downloaded the entire compiler, interpreter environments and setup the servers 20 minutes later. (In the event you can't read the bar graphs, use the following order from left to right for those charts).

1. Erlyweb_erl-1 - Yaws and Erly Web Test, simple controller with Lorem Ipsum data.
   
2. Django_py-2 - Simple view Lorem Ipsum, using built-in python Django server
   
3. JSP-3 - Simple JSP page on Tomcat (Lorem Ipsum data)
   
4. Hunch_Lisp-4 - Simple Hunchentoot template (Lorem Ipsum data). No threading support.
   
5. Rails-5 - With WebBrick, simple controller/view with Lorem Ipsum data
   
6. JRuby-6 - Simple Botlist JRuby Framework, Simple View.
   
7. JRuby_Rails_Tom-7 - JRuby Rails running under Tomcat (similar to Rails-5)
   
8. JRuby_Rails_Brick-8 - JRuby Rails running under Web Brick (similar to Rails-5)
   
9. JRuby_Rails_Jetty-9 - JRuby Rails running under Jetty (similar to Rails-5)
   
10. Simple_HTTP-10 - Custom "Do it yourself" web server in 150 lines of Java IO code. This test should give us the best results because of the low overhead. You wouldn't want to use this as your default application server but could be used for other messaging or middleware purposes.

Linux Tests:

1. LIN_HTTP-1 - Simple HTTP server, run on linux
   
2. LIN_PHP-2 - Apache/PHP page
   
3. LIN_LISP-3 - Hunchentoot on Linux with threading support
   
4. LIN_JRUBY-4 - Simple Botlist JRuby/JSP page
   
5. LIN_BOTLIST-5 - Botlist web-application listing page (demonstrates more complete application). Botlist uses JRuby, Spring, Hibernate with MySQL.
   

Testing software used

Two pieces of software were used to test these servers, Apache's JMeter and Apache Bench. Apache Bench and JMeter were used on the Linux box.

• JMeter 2.3.2 - Generates the Summary report out, shown below. Most of these tests were run with 20 concurrent users with a slight delay of 20ms between each request. 1000-2000 requests per user. For slower servers on the Win32 side (like with Hunchentoot) could only handle 5 users and a fewer number of requests.
  
• Apache Bench
  

Table Summary of Results, Win32 and Linux

Label	# Samples	Average	Min	Max	Std. Dev.	Error %	Throughput	KB/sec	Avg. Bytes
Erlyweb_erl-1	24000	0	0	42	1	0	360.57	2236.28	6351
Django_py-2	30000	24	4	1123	84.59	6.67E-004	152.89	1121.16	7508.88
JSP-3	24000	0	0	28	1.03	0	259.35	1556.86	6147
Hunch_Lisp-4	6000	77	2	267723	4136.62	0	13.02	104.39	8208
Rails-5	30000	170	7	1059	56.18	0	85.53	616.81	7385
JRuby-6	24000	10	3	84	10.04	0	217.33	1304.62	6147
JRuby_Rails_Tom-7	4500	93	5	29509	514.56	0	17.33	22.57	1334
JRuby_Rails_Brick-8	27000	458	32	1048	66.83	0	30.18	2	68
JRuby_Rails_Jetty-9	3600	17	5	11327	257.83	0	60.33	4.01	68
Simple_HTTP-10	40000	6	1	284	8.69	0	402.76	663.92	1688
Label	# Samples	Average	Min	Max	Std. Dev.	Error %	Throughput	KB/sec	Avg. Bytes
LIN_HTTP-1	30000	129	1	1010	92.63	0	94.3	155.44	1688
LIN_PHP-2	30000	3	1	136	4.93	0	215.21	486.96	2317
LIN_LISP-3	30000	115	5	21653	348.18	0	98.14	786.68	8208
LIN_JRUBY-4	30000	45	3	1811	74.31	0	152.67	916.45	6147
LIN_BOTLIST-5	30000	195	8	5203	156.65	0	71.08	4080.69	58789

• # Samples - Total number of HTTP requests
  
• Average - Average time to complete request in ms
  
• Min - Min time to complete request in ms
  
• Max - Max time to complete request in ms
  
• Throughput - Number of requests per second
  
• KB/sec - KB output per second
  
• Avg. Bytes - Average bytes per request. There is some variance in the data that is output for each view page.
  

Charts and Graphs

Note: Click on the chart thumbnail for a the full-size image (thanks for ruining my images blogger.com!)

Chart-1: Throughput on Win32 Box. From left to right: Erlyweb_erl-1 | Django_py-2 | JSP-3 | Hunch_Lisp-4 | Rails-5 | JRuby-6 | JRuby_Rails_Tom-7 | JRuby_Rails_Brick-8 | JRuby_Rails_Jetty-9 | Simple_HTTP-10

Chart-2: KB per second on Win32 Box. From left to right: Erlyweb_erl-1 | Django_py-2 | JSP-3 | Hunch_Lisp-4 | Rails-5 | JRuby-6 | JRuby_Rails_Tom-7 | JRuby_Rails_Brick-8 | JRuby_Rails_Jetty-9 | Simple_HTTP-10

Chart-3: Average time per request on Win32 Box. From left to right: Erlyweb_erl-1 | Django_py-2 | JSP-3 | Hunch_Lisp-4 | Rails-5 | JRuby-6 | JRuby_Rails_Tom-7 | JRuby_Rails_Brick-8 | JRuby_Rails_Jetty-9 | Simple_HTTP-10

Chart-9: Throughput (Req/Sec) on Linux. From left to right: LIN_HTTP-1 | LIN_PHP-2 | LIN_LISP-3 | LIN_JRUBY-4 | LIN_BOTLIST-5

Chart-4: Lisp throughput over time on Linux Box.

Chart-5: Apache/PHP time per request over time on Linux Box.

Chart-6: Lisp/Hunchentoot time per request over time on Linux Box.

Chart-7: Django Time per Requests Summary Win32

Chart-8: Rails/Webbrick Time per Requests Summary Win32

Apache Bench Output (Hunchentoot, PHP, and SimpleHTTP)

Here is console output from apache bench for some of the tests above. I wanted to use three different load testing suites; ApacheBench, JMeter, and Erlang's Tsung to see if they returned the same results. But AB and JMeter were only used.

------------------------------------
Server Software:        Hunchentoot
------------------------------------
Server Hostname:        localhost
Server Port:            4242
Document Length:        8208 bytes
Concurrency Level:      20
Time taken for tests:   47.605169 seconds
Complete requests:      5000
Total transferred:      41890000 bytes
HTML transferred:       41040000 bytes
Requests per second:    105.03 [#/sec] (mean)
Time per request:       190.421 [ms] (mean)
Time per request:       9.521 [ms] (mean, across all concurrent requests)
Transfer rate:          859.32 [Kbytes/sec] received

--------------------------------------
Server Software:        Apache/2.2.8 (PHP)
--------------------------------------
Server Hostname:        localhost
Server Port:            80
Document Length:        2317 bytes
Concurrency Level:      20
Time taken for tests:   5.182021 seconds
Complete requests:      5000
Total transferred:      12747822 bytes
HTML transferred:       11601219 bytes
Requests per second:    964.87 [#/sec] (mean)
Time per request:       20.728 [ms] (mean)
Time per request:       1.036 [ms] (mean, across all concurrent requests)
Transfer rate:          2402.34 [Kbytes/sec] received

--------------------------------------
Server Software:        Simple-HTTP-Server-In-Java
--------------------------------------
Server Hostname:        localhost
Server Port:            9980
Document Length:        1688 bytes
Concurrency Level:      20
Time taken for tests:   19.169574 seconds
Complete requests:      5000
Total transferred:      8860000 bytes
HTML transferred:       8440000 bytes
Requests per second:    260.83 [#/sec] (mean)
Time per request:       76.678 [ms] (mean)
Time per request:       3.834 [ms] (mean, across all concurrent requests)
Transfer rate:          451.34 [Kbytes/sec] received

Environment Setups and Source Code

Setting up the different environments involved out of the box (download and run) configurations.

Yaws and ErlyWeb (Erlyweb_erl-1):

1. Download and install the most recent version of Erlang. For Windows, I went through
the one-click installer. For Ubuntu-Linux, I compiled the most recent source.
2. Download Yaws; I installed Yaws using the steps detailed in the README files:
tar -xzf yaws-1.76.tar.gz
./configure --prefix=/cygdrive/c/erl_stuff/yaws
make
make install
3. Download Erlyweb (as of Aug 10, 0.7.2) and following the install instructions:
http://code.google.com/p/erlyweb/wiki/IntroductionTutorial

(Ease of Setup: 2.5/4.0 - where 4.0 is the easiest to configure) - Erlyweb is easy to setup, but I needed to perform some code changes to get the crypto library to compile. Also, the MySQL driver did not work on windows.

Here is the simple view code:
http://haskellnotebook.googlecode.com/svn/trunk/benchmarks/crud_blog/erlyweb/riki/src/components/

Python and Django (Hunch_Lisp-4):

I assume that you have Python already installed. It is available to most Linux systems. For Win32, there is a one-click installer executable as well as the cygwin version. For this test, I used the cygwin version.

For the Django install, I used the step-by-step guide.
http://www.djangoproject.com/documentation/install/

To run the server, used the built-in server:
python manage.py runserver 9980

The django view code:
http://haskellnotebook.googlecode.com/svn/trunk/benchmarks/crud_blog/django/ghostblog/templates/

Tomcat (JSP-3):

On Win32, I didn't perform a full installation. Download Tomcat and unzip the archive file, enter the bin directory and click on "startup.bat".

All web applications (archived war files) are deployed to the TOMCAT_HOME/webapps/ directory.

Rails Setup for running Webbrick (Rails-5):

1. Install Ruby
2. You can install Rails with Gems as described on the Rails homepage.
http://www.rubyonrails.org/down

3. gem install rails --http-proxy http://proxy.com:9999 (if you are behind a web proxy)
4. Run the rails command to generate the skeleton code:
rails blah

5. I created a generic controller:
ruby script/generate controller home index

6. Run the WebBrick server:
ruby script/server

Additional steps to run the JRuby applications.
To run the JRuby WebBrick application, just use the JRuby commands and executables as opposed to the (C)ruby commands.

To build a JRuby/Rails web-application, use the Warbler plugin:
(http://weblogs.java.net/blog/arungupta/archive/2008/04/rails_and_java_1.html)

./gem.bat install warbler --http-proxy http://localhost:9999
../jruby.bat -S warble

Copy the jblah.war file to the tomcat or jetty webapps directory

Hunchentoot on SBCL (Common Lisp) (Hunch_Lisp-4)

1. Install SBCL
2. Hunchentoot:
I created a setup guide for Hunchentoot, you can use these steps or if you are on Linux you can just use ASDF-INSTALL:
Enterprise Common Lisp Setup With Hunchentoot

3. After Hunchentoot is installed, I use the following commands to compile and load my application:
* (asdf:operate 'asdf:compile-op 'ghost-trinity)
* (asdf:operate 'asdf:load-op 'ghost-trinity)
4. To run the server:

* (hunchentoot:start-server :port 4242)

Simple JRuby Interpreter Application through Botlist (JRuby-6):

Simple HTTP Server in Java (Simple_HTTP-10):

The multi-threaded Java HTTP server is 150 lines of code. It is very basic code for processing a web client and then returning only one response.

http://haskellnotebook.googlecode.com/svn/trunk/benchmarks/crud_blog/simple_server/

BufferedReader bufReader = new BufferedReader(new InputStreamReader(in));
            while(running) {
                // read in a line from the client
                line = bufReader.readLine();
                if (line == null)
                    break;
                // and write out the reversed line
                //System.out.println("[server/" + line.length() + "]" + line);
                if (line.length() == 0)
                    break;
            }
    // Write a html response back
            StringBuffer buf = new StringBuffer();
            buf.append("HTTP/1.1 200 Ok\r\n");
            buf.append("Server: Apache-Test\r\n");
            buf.append("Connection: close\r\n");
            buf.append("Content-Type: text/html\r\n");
...
...
...

Conclusion Bullet Points

• Erlang, ErlyWeb and Yaws had the best performance. I performed other side tests and went to 100 and 300 concurrent connections and there was no lapse in performance. Others have tested Yaws with 50000, 80000 connections and seen similar results.
  
• Django had solid performance with the default server. 152.89 request per second. Average of 24 milliseconds a request.
  
• Rails with WebBrick did ok. Normally this configuration is used for development only. 85.5 request per second and the average time to complete a request was 170 ms.
  
• Based on "my" tests, the JRuby/Rails setup had the lowest performance, especially with a J2EE server like Tomcat or Jetty. With the JRuby WebBrick server. Each request took 458ms or 0.5 seconds at 30 requests a second.
  
• I had trouble testing JRuby Rails with Tomcat and Jetty. Jetty gave the best performance of those two, but it was hard to run more than 5000 requests. I also had to reduce the number of concurrent users to four or five.
  For example, Tomcat would output the following errors after so many requests:
  
  Aug 9, 2008 11:02:03 AM org.apache.catalina.core ContainerBase$ContainerBackgroundProcessor processChildren
  SEVERE: Exception invoking periodic operation:
  java.lang.OutOfMemoryError: Java heap space
  

Full Source Download and Data Results

http://haskellnotebook.googlecode.com/files/benchmarks_webapps.tar.gz

Resources

http://yarivsblog.com/articles/2007/12/09/erlyweb-vs-ruby-on-rails-ec2-performance-showdown/

http://docs.codehaus.org/display/GRAILS/Grails+vs+Rails+Benchmark

----