Tuesday, July 8, 2008

Enterprise Common Lisp: Setting up HUNCHENTOOT, the common lisp webserver

Overview

"Common Lisp is also an excellent language for exploratory programming--if you don't know exactly how your program is going to work when you first sit down to write it, Common Lisp provides several features to help you develop your code incrementally and interactively." -- Peter Seibel (Practical Common Lisp)

Hunchentoot is a powerful web application server with support for other Lisp web libraries including HTML template parsing (server pages) and SSL support. This entry provides a step by step guide for setting up the hunchentoot web server on WindowsXP using SBCL as the common lisp implementation (and running from cygwin).

Note: all downloads are provided at the end of the blog entry.


Getting the Available Dependencies

One thing I have learned so far about Lisp and the Lisp community, there are many different ways to configure and use it. I generally like using the most obvious, intuitive approaches and then over time, using "best practices". Basically, everything I present may not be the standard way of configuring Hunchnentoot, but each step worked successfully with my environment. It took me two to three hours to install SBCL, get all of the libraries, compile the libraries and then setup the system. If you have worked with J2EE applications, these steps are nothing more than adding a Java jar library to your servlet oriented web application.

Get SBCL

I downloaded the most recent SBCL, 1.0.13 for Win32. Apparently, the port is still in progress. Download and install the application using the installer. Also, make sure that the sbcl command works with cygwin. Ideally, the steps mentioned in this setup tutorial should work with WindowsXP's default command line environment, I much prefer cygwin. So the setup steps are oriented towards that linux environment.



Get Weitz's Lisp Starter Pack



"The Lisp Starter Pack is an attempt to help Common Lisp newcomers getting started. The aim is to quickly set up a comfortable working environment which includes a couple of useful and/or popular open source libraries" -- Ed Weitz

I decided to use the collection of Lisp libraries provided by the Lisp Starter Pack. It includes the source for the Hunchentoot server, CLSQL database libraries and all of the dependencies. Select all of the library options and wait for them to be downloaded to your install directory.


Note: I am behind a web proxy and the Starter Pack proxy download worked fine. Apparently the application will query the Windows registry for the static proxy settings and use those host/port values.




Once you have all the dependencies, the next set of steps simply involve compiling all of the dependencies and running Hunchentoot.

Compile the Hunchentoot Dependencies

Asdf is "Another System Definition Facility". It fills a similar role for CL Development as make. We will use a one line command to go through and compile all of the dependencies with asdf.


For example:
(asdf:operate 'asdf:compile-op :hunchentoot)

The order in which you compile the Lisp libraries is not important unless the library has additional dependencies. For example, Hunchentoot depends on all these 15 libraries. Chunga depends on flexi-streams. Here are the library directories from the Lisp Starter Pack and the order that I compiled them.

  1. cl-who-0.11.1
  2. rfc2388
  3. trivial-gray-streams-2006-09-16
  4. flexi-streams-1.0.3
  5. chunga-0.4.3
  6. md5-1.8.5
  7. url-rewrite-0.1.1
  8. cl-base64-3.3.2
  9. cl-fad-0.6.2
  10. cl-ppcre-1.4.1
  11. uffi-1.6.0
  12. cffi-0.9.2
  13. clsql-4.0.3
  14. html-template-0.9.1
  15. hunchentoot-0.15.7

Before compiling, put the following listing in your SBCL configuration file:
~/.sbclrc

(require '#:asdf)
(pushnew :hunchentoot-no-ssl *features*)
(dolist (dir
(directory
(merge-pathnames "site/*/"
(sb-ext:posix-getenv "SBCL_HOME"))))
(pushnew dir asdf:*central-registry*))

This local configuration adds the asdf package, disables SSL support for hunchentoot, and adds the [windows_environment_path:SBCL_HOME]/site directory to the ASDF registry. Also, copy the Lisp library dependencies from the Starter Pack install directory to the SBCL_HOME site directory. This is a screenshot of my SBCL_HOME/site directory.



Change to the site directory, where you copied all of the lisp libraries. Change to the first Hunchentoot dependency directory, cl-who-0.11.1. At the cygwin prompt, invoke sbcl.

You are going to have compile 15 libraries, if you don't want to exit the REPL and then compile the code, you can modify the Lisp default path. Overwrite the *DEFAULT-PATHNAME-DEFAULTS* and then use truename function to check the value.


This is experimental prerelease support for the Windows platform: use
at your own risk. "Your Kitten of Death awaits!"
* (setf *DEFAULT-PATHNAME-DEFAULTS* #P"C:\\projects\\tools\\home\\projects\\down
loads_main\\main\\1.0.13\\site")

#P"C:\\projects\\tools\\home\\projects\\downloads_main\\main\\1.0.13\\site"
* (truename ".")




Compile the first library, CL-WHO

At the sbcl prompt, enter the following command:

(asdf:operate 'asdf:compile-op :cl-who)

Make sure that your default path is at the cl-who-0.11.1 directory.



Change into each of the Lisp library directories and compile the code with the asdf:operate compile command. Compile in the order mentioned above.

cffi-0.9.2, chunga-0.4.3, cl-base64-3.3.2,cl-fad-0.6.2, cl-ppcre-1.4.1
cl-who-0.11.1,clsql-4.0.3, flexi-streams-1.0.3, html-template-0.9.1, hunchentoot-0.15.7, md5-1.8.5, rfc2388, trivial-gray-streams-2006-09-16, uffi-1.6.0, url-rewrite-0.1.1


Hopefully, all of the libraries will compile and you will be able finally compile Hunchentoot. You may have an issue with clsql. I got an FFI function "strtoul" is not found. To resolve, open the /site/clsql-4.0.3/uffi/clsql-uffi-loader.lisp source and add the following lisp code to the last line in that file.

(uffi:load-foreign-library "msvcrt.dll")

At the cl-sql directory, re-run:

(asdf:operate 'asdf:compile-op :clsql)

You may also want to run the command for the specific database driver you are working with.

(asdf:operate 'asdf:compile-op :clsql-mysql)

Launch Hunchentoot

At this point, you should have already compiled Hunchentoot and now all you have to do is launch the server. And open your browser to localhost:4242 (and whatever port you decide to bind Hunchentoot to).

# sbcl
...
...
* (require :hunchentoot)
* (hunchentoot:start-server :port 4242)



A Simple Web Application
I am going to rush through the example web app. Essentially, here is the code and you are on your own. Here is the source for a basic MVC-1 application. The view uses the html-template parser. The Hunchentoot controller code is mostly contained in the trinity.lisp file.
We first need to associate the action controller URLs with the lisp dispatcher functions. Consider this code from trinity.lisp:

(setq hunchentoot:*dispatch-table*
(list (hunchentoot:create-regex-dispatcher
"^/$" 'generate-index-page)
(hunchentoot:create-regex-dispatcher
"^/trinity/$" 'generate-index-page)
(hunchentoot:create-regex-dispatcher
"^/trinity/edit/$" 'handle-edit-page)))

(defun generate-index-page ()
"Generate the index page showing all the blog posts."
(with-output-to-string (stream)
(html-template:fill-and-print-template
#P"index.html" '() :stream stream)))

To handle the POST request (controller code), you simply check for the "POST" method type with the Hunchentoot request-method function. Use hunchentoot:post-parameter to get the form values:

(defun handle-edit-page ()
(cond ((eq (hunchentoot:request-method) :GET)
(generate-page #P"edit.html"))
((eq (hunchentoot:request-method) :POST)
(send-riki-server))))

Compiling and Running the Code
Enter into your web application directory and run sbcl, then compile and run Hunchentoot:

(require :asdf)
(asdf:operate 'asdf:load-op 'ghost-trinity)
(hunchentoot:start-server :port 4242)

The Source
Here is all of the source. The asd file contain the ASDF definition. Packages.lisp simple contains the ghost-trinity package definition. Trinity.lisp contains all of the source.

  1. ghost-trinity.asd - ASDF file
  2. packages.lisp - Define ghost-trinity package
  3. trinity.lisp - Main Hunchentoot oriented Lisp source
  4. index.html - Index template view page
  5. edit.html - Edit (on form post) template view page
  6. confirm.html - Confirmation page


(defpackage #:ghost-trinity-system (:use #:asdf #:cl))
(in-package :ghost-trinity-system)
(asdf:defsystem :ghost-trinity
:name "ghost-trinity"
:author "Berlin Brown"
:version "0.1"
:maintainer "Berlin Brown <berlin.brown@gmail.com>"
:licence "BSD"
:description "Ghost Trinity Web Front End"
:long-description "Ghost Trinity Web Front End"
:depends-on (:hunchentoot
:cl-who
:html-template
:clsql
:clsql-mysql
:sb-bsd-sockets)
:components ((:file "packages")
(:file "trinity" :depends-on ("packages"))
))
;;; End of File

;; trinity.lisp - simple web application
;; References:
;; [1] http://clsql.b9.com/manual/with-database.html
(in-package :ghost-trinity)
(require :cl-who)
(require :hunchentoot)
(require :html-template)
(require :clsql)
(require :clsql-mysql)

(clsql:locally-enable-sql-reader-syntax)
(setf clsql:*default-caching* nil)

(defun generate-index-page ()
"Generate the index page showing all the blog posts."
(with-output-to-string (stream)
(html-template:fill-and-print-template
#P"index.html" '() :stream stream)))
(defun generate-page (page)
"Generate the index page showing all the blog posts."
(with-output-to-string (stream)
(html-template:fill-and-print-template
page '() :stream stream)))
(defun handle-edit-page ()
(cond ((eq (hunchentoot:request-method) :GET)
(generate-page #P"edit.html"))
((eq (hunchentoot:request-method) :POST)
(send-riki-server))))
(defun send-riki-server ()
(let ((a (hunchentoot:post-parameter "message")))
(print a)
(generate-page #P"confirm.html")))
;;------------------------------------------------
;; Hunchentoot server settings
;;------------------------------------------------
(setq hunchentoot:*catch-errors-p* nil)
;; Set the web server dispatch table
(setq hunchentoot:*dispatch-table*
(list (hunchentoot:create-regex-dispatcher
"^/$" 'generate-index-page)
(hunchentoot:create-regex-dispatcher
"^/trinity/$" 'generate-index-page)
(hunchentoot:create-regex-dispatcher
"^/trinity/edit/$" 'handle-edit-page)))
;; Make sure html-template looks for files in the right directory
(setq html-template:*default-template-pathname*
#P"...webapp\\")
;; Start the web server utilities
(defvar *ht-server* nil)
(defun start-app ()
"Start the web server"
(defvar *ht-server*
(hunchentoot:start-server :port 4242)
))
(defun stop-app ()
(hunchentoot:stop-server *ht-server*))
;; End of the File




Database Code

The listing below is not unlike other database connect routines in other programming languages. Establish the connection with the database using the host name, user name and password. Then invoke the read or update operation. Here is a separate example driver application for demonstrating those tasks against a MySQL database:

Create the following schema and run in MySQL:

CREATE TABLE entity_links (
id int(11) NOT NULL auto_increment,
main_url varchar(255) NOT NULL,
url_title varchar(128) NOT NULL,
url_description varchar(255) default NULL,
keywords varchar(255) default NULL,
views int(11) default '0',
created_on datetime NOT NULL default '0000-00-00 00:00:00',
rating int(11) NOT NULL default '0',
user_id int(11) default NULL,
full_name varchar(128) NOT NULL,
hostname varchar(128) default NULL,
process_count int(11) NOT NULL default '0',
updated_on datetime default '0000-00-00 00:00:00',
link_type varchar(20) default NULL,
bot_rating decimal(5,2) default '0.00',
generated_obj_id varchar(60) default NULL,
user_up_votes int(11) default '0',
user_down_votes int(11) default '0',
request_time int(11) default '0',
object_id_status tinyint(4) default '0',
para_tag_ct int(11) default '0',
PRIMARY KEY (id),
UNIQUE KEY main_url (main_url),
UNIQUE KEY generated_obj_id (generated_obj_id)
);

And we simply run this clsql/clsql-mysql code in SBCL (load "test_db_connect.lisp"):

(require :clsql)
(require :clsql-mysql)
(clsql:locally-enable-sql-reader-syntax)
(setf clsql:*default-caching* nil)
;; Data structure to store our links
(clsql:def-view-class entity_links ()
((main_url
:reader main_url
:initarg :main_url
:type string)
(url_title
:reader url_title
:initarg :url_title
:type string)
))
;; Ensure that the references to the entity links table, uses
;; the correct tablename and case.
(setf (clsql:view-table (find-class 'entity_links)) '|entity_links|)

(defmacro with-db ((database) &body body)
`(clsql:with-database
;; Supply database connect str
;; For mysql => URL DATABASE USER PASSWORD
(,database '("localhost" "botlist_development" "USER" "PASSWORD")
:database-type :mysql
:pool t
:if-exists nil)
,@body))
(defun test-db ()
"Wrapper method for the test case; connect to the database
and run a simple query"

(with-db (db)
(setf clsql:*default-database* db)
(clsql:status t)
(let ((z (clsql:query
"select main_url from entity_links limit 0,2"
:database db))
(d (clsql:select 'entity_links
:database db
:flatp t)))
(print z)))
(format t "After with-db~%")
(clsql:status t))
(defun main ()
"Main entry point for the example"
(format t "Running - ~%")
(test-db)
(format t "Done - ~%"))
(main)
;; End of the File


Full Source Download

http://haskellnotebook.googlecode.com/files/lisp_webapp_example.tar.gz

Resources

I didn't cover the code in much detail, so just use the list of resources to learn more about Hunchentoot, the database code and anything else that was mentioned.

http://www.newartisans.com/blog_files/common.lisp.with.apache.php
http://www.weitz.de/hunchentoot/
http://clsql.b9.com/
http://common-lisp.net/project/asdf-install/
http://www.sbcl.org/
http://www.weitz.de/starter-pack/
http://haskellnotebook.googlecode.com/files/lisp_webapp_example.tar.gz
http://en.wikipedia.org/wiki/Buddhism - Also useful

No comments: