Tuesday, July 8, 2008

Enterprise Common Lisp: Setting up HUNCHENTOOT, the common lisp webserver

Overview

"Common Lisp is also an excellent language for exploratory programming--if you don't know exactly how your program is going to work when you first sit down to write it, Common Lisp provides several features to help you develop your code incrementally and interactively." -- Peter Seibel (Practical Common Lisp)

Hunchentoot is a powerful web application server with support for other Lisp web libraries including HTML template parsing (server pages) and SSL support. This entry provides a step by step guide for setting up the hunchentoot web server on WindowsXP using SBCL as the common lisp implementation (and running from cygwin).

Note: all downloads are provided at the end of the blog entry.


Getting the Available Dependencies

One thing I have learned so far about Lisp and the Lisp community, there are many different ways to configure and use it. I generally like using the most obvious, intuitive approaches and then over time, using "best practices". Basically, everything I present may not be the standard way of configuring Hunchnentoot, but each step worked successfully with my environment. It took me two to three hours to install SBCL, get all of the libraries, compile the libraries and then setup the system. If you have worked with J2EE applications, these steps are nothing more than adding a Java jar library to your servlet oriented web application.

Get SBCL

I downloaded the most recent SBCL, 1.0.13 for Win32. Apparently, the port is still in progress. Download and install the application using the installer. Also, make sure that the sbcl command works with cygwin. Ideally, the steps mentioned in this setup tutorial should work with WindowsXP's default command line environment, I much prefer cygwin. So the setup steps are oriented towards that linux environment.



Get Weitz's Lisp Starter Pack



"The Lisp Starter Pack is an attempt to help Common Lisp newcomers getting started. The aim is to quickly set up a comfortable working environment which includes a couple of useful and/or popular open source libraries" -- Ed Weitz

I decided to use the collection of Lisp libraries provided by the Lisp Starter Pack. It includes the source for the Hunchentoot server, CLSQL database libraries and all of the dependencies. Select all of the library options and wait for them to be downloaded to your install directory.


Note: I am behind a web proxy and the Starter Pack proxy download worked fine. Apparently the application will query the Windows registry for the static proxy settings and use those host/port values.




Once you have all the dependencies, the next set of steps simply involve compiling all of the dependencies and running Hunchentoot.

Compile the Hunchentoot Dependencies

Asdf is "Another System Definition Facility". It fills a similar role for CL Development as make. We will use a one line command to go through and compile all of the dependencies with asdf.


For example:
(asdf:operate 'asdf:compile-op :hunchentoot)

The order in which you compile the Lisp libraries is not important unless the library has additional dependencies. For example, Hunchentoot depends on all these 15 libraries. Chunga depends on flexi-streams. Here are the library directories from the Lisp Starter Pack and the order that I compiled them.

  1. cl-who-0.11.1
  2. rfc2388
  3. trivial-gray-streams-2006-09-16
  4. flexi-streams-1.0.3
  5. chunga-0.4.3
  6. md5-1.8.5
  7. url-rewrite-0.1.1
  8. cl-base64-3.3.2
  9. cl-fad-0.6.2
  10. cl-ppcre-1.4.1
  11. uffi-1.6.0
  12. cffi-0.9.2
  13. clsql-4.0.3
  14. html-template-0.9.1
  15. hunchentoot-0.15.7

Before compiling, put the following listing in your SBCL configuration file:
~/.sbclrc

(require '#:asdf)
(pushnew :hunchentoot-no-ssl *features*)
(dolist (dir
(directory
(merge-pathnames "site/*/"
(sb-ext:posix-getenv "SBCL_HOME"))))
(pushnew dir asdf:*central-registry*))

This local configuration adds the asdf package, disables SSL support for hunchentoot, and adds the [windows_environment_path:SBCL_HOME]/site directory to the ASDF registry. Also, copy the Lisp library dependencies from the Starter Pack install directory to the SBCL_HOME site directory. This is a screenshot of my SBCL_HOME/site directory.



Change to the site directory, where you copied all of the lisp libraries. Change to the first Hunchentoot dependency directory, cl-who-0.11.1. At the cygwin prompt, invoke sbcl.

You are going to have compile 15 libraries, if you don't want to exit the REPL and then compile the code, you can modify the Lisp default path. Overwrite the *DEFAULT-PATHNAME-DEFAULTS* and then use truename function to check the value.


This is experimental prerelease support for the Windows platform: use
at your own risk. "Your Kitten of Death awaits!"
* (setf *DEFAULT-PATHNAME-DEFAULTS* #P"C:\\projects\\tools\\home\\projects\\down
loads_main\\main\\1.0.13\\site")

#P"C:\\projects\\tools\\home\\projects\\downloads_main\\main\\1.0.13\\site"
* (truename ".")




Compile the first library, CL-WHO

At the sbcl prompt, enter the following command:

(asdf:operate 'asdf:compile-op :cl-who)

Make sure that your default path is at the cl-who-0.11.1 directory.



Change into each of the Lisp library directories and compile the code with the asdf:operate compile command. Compile in the order mentioned above.

cffi-0.9.2, chunga-0.4.3, cl-base64-3.3.2,cl-fad-0.6.2, cl-ppcre-1.4.1
cl-who-0.11.1,clsql-4.0.3, flexi-streams-1.0.3, html-template-0.9.1, hunchentoot-0.15.7, md5-1.8.5, rfc2388, trivial-gray-streams-2006-09-16, uffi-1.6.0, url-rewrite-0.1.1


Hopefully, all of the libraries will compile and you will be able finally compile Hunchentoot. You may have an issue with clsql. I got an FFI function "strtoul" is not found. To resolve, open the /site/clsql-4.0.3/uffi/clsql-uffi-loader.lisp source and add the following lisp code to the last line in that file.

(uffi:load-foreign-library "msvcrt.dll")

At the cl-sql directory, re-run:

(asdf:operate 'asdf:compile-op :clsql)

You may also want to run the command for the specific database driver you are working with.

(asdf:operate 'asdf:compile-op :clsql-mysql)

Launch Hunchentoot

At this point, you should have already compiled Hunchentoot and now all you have to do is launch the server. And open your browser to localhost:4242 (and whatever port you decide to bind Hunchentoot to).

# sbcl
...
...
* (require :hunchentoot)
* (hunchentoot:start-server :port 4242)



A Simple Web Application
I am going to rush through the example web app. Essentially, here is the code and you are on your own. Here is the source for a basic MVC-1 application. The view uses the html-template parser. The Hunchentoot controller code is mostly contained in the trinity.lisp file.
We first need to associate the action controller URLs with the lisp dispatcher functions. Consider this code from trinity.lisp:

(setq hunchentoot:*dispatch-table*
(list (hunchentoot:create-regex-dispatcher
"^/$" 'generate-index-page)
(hunchentoot:create-regex-dispatcher
"^/trinity/$" 'generate-index-page)
(hunchentoot:create-regex-dispatcher
"^/trinity/edit/$" 'handle-edit-page)))

(defun generate-index-page ()
"Generate the index page showing all the blog posts."
(with-output-to-string (stream)
(html-template:fill-and-print-template
#P"index.html" '() :stream stream)))

To handle the POST request (controller code), you simply check for the "POST" method type with the Hunchentoot request-method function. Use hunchentoot:post-parameter to get the form values:

(defun handle-edit-page ()
(cond ((eq (hunchentoot:request-method) :GET)
(generate-page #P"edit.html"))
((eq (hunchentoot:request-method) :POST)
(send-riki-server))))

Compiling and Running the Code
Enter into your web application directory and run sbcl, then compile and run Hunchentoot:

(require :asdf)
(asdf:operate 'asdf:load-op 'ghost-trinity)
(hunchentoot:start-server :port 4242)

The Source
Here is all of the source. The asd file contain the ASDF definition. Packages.lisp simple contains the ghost-trinity package definition. Trinity.lisp contains all of the source.

  1. ghost-trinity.asd - ASDF file
  2. packages.lisp - Define ghost-trinity package
  3. trinity.lisp - Main Hunchentoot oriented Lisp source
  4. index.html - Index template view page
  5. edit.html - Edit (on form post) template view page
  6. confirm.html - Confirmation page


(defpackage #:ghost-trinity-system (:use #:asdf #:cl))
(in-package :ghost-trinity-system)
(asdf:defsystem :ghost-trinity
:name "ghost-trinity"
:author "Berlin Brown"
:version "0.1"
:maintainer "Berlin Brown <berlin.brown@gmail.com>"
:licence "BSD"
:description "Ghost Trinity Web Front End"
:long-description "Ghost Trinity Web Front End"
:depends-on (:hunchentoot
:cl-who
:html-template
:clsql
:clsql-mysql
:sb-bsd-sockets)
:components ((:file "packages")
(:file "trinity" :depends-on ("packages"))
))
;;; End of File

;; trinity.lisp - simple web application
;; References:
;; [1] http://clsql.b9.com/manual/with-database.html
(in-package :ghost-trinity)
(require :cl-who)
(require :hunchentoot)
(require :html-template)
(require :clsql)
(require :clsql-mysql)

(clsql:locally-enable-sql-reader-syntax)
(setf clsql:*default-caching* nil)

(defun generate-index-page ()
"Generate the index page showing all the blog posts."
(with-output-to-string (stream)
(html-template:fill-and-print-template
#P"index.html" '() :stream stream)))
(defun generate-page (page)
"Generate the index page showing all the blog posts."
(with-output-to-string (stream)
(html-template:fill-and-print-template
page '() :stream stream)))
(defun handle-edit-page ()
(cond ((eq (hunchentoot:request-method) :GET)
(generate-page #P"edit.html"))
((eq (hunchentoot:request-method) :POST)
(send-riki-server))))
(defun send-riki-server ()
(let ((a (hunchentoot:post-parameter "message")))
(print a)
(generate-page #P"confirm.html")))
;;------------------------------------------------
;; Hunchentoot server settings
;;------------------------------------------------
(setq hunchentoot:*catch-errors-p* nil)
;; Set the web server dispatch table
(setq hunchentoot:*dispatch-table*
(list (hunchentoot:create-regex-dispatcher
"^/$" 'generate-index-page)
(hunchentoot:create-regex-dispatcher
"^/trinity/$" 'generate-index-page)
(hunchentoot:create-regex-dispatcher
"^/trinity/edit/$" 'handle-edit-page)))
;; Make sure html-template looks for files in the right directory
(setq html-template:*default-template-pathname*
#P"...webapp\\")
;; Start the web server utilities
(defvar *ht-server* nil)
(defun start-app ()
"Start the web server"
(defvar *ht-server*
(hunchentoot:start-server :port 4242)
))
(defun stop-app ()
(hunchentoot:stop-server *ht-server*))
;; End of the File




Database Code

The listing below is not unlike other database connect routines in other programming languages. Establish the connection with the database using the host name, user name and password. Then invoke the read or update operation. Here is a separate example driver application for demonstrating those tasks against a MySQL database:

Create the following schema and run in MySQL:

CREATE TABLE entity_links (
id int(11) NOT NULL auto_increment,
main_url varchar(255) NOT NULL,
url_title varchar(128) NOT NULL,
url_description varchar(255) default NULL,
keywords varchar(255) default NULL,
views int(11) default '0',
created_on datetime NOT NULL default '0000-00-00 00:00:00',
rating int(11) NOT NULL default '0',
user_id int(11) default NULL,
full_name varchar(128) NOT NULL,
hostname varchar(128) default NULL,
process_count int(11) NOT NULL default '0',
updated_on datetime default '0000-00-00 00:00:00',
link_type varchar(20) default NULL,
bot_rating decimal(5,2) default '0.00',
generated_obj_id varchar(60) default NULL,
user_up_votes int(11) default '0',
user_down_votes int(11) default '0',
request_time int(11) default '0',
object_id_status tinyint(4) default '0',
para_tag_ct int(11) default '0',
PRIMARY KEY (id),
UNIQUE KEY main_url (main_url),
UNIQUE KEY generated_obj_id (generated_obj_id)
);

And we simply run this clsql/clsql-mysql code in SBCL (load "test_db_connect.lisp"):

(require :clsql)
(require :clsql-mysql)
(clsql:locally-enable-sql-reader-syntax)
(setf clsql:*default-caching* nil)
;; Data structure to store our links
(clsql:def-view-class entity_links ()
((main_url
:reader main_url
:initarg :main_url
:type string)
(url_title
:reader url_title
:initarg :url_title
:type string)
))
;; Ensure that the references to the entity links table, uses
;; the correct tablename and case.
(setf (clsql:view-table (find-class 'entity_links)) '|entity_links|)

(defmacro with-db ((database) &body body)
`(clsql:with-database
;; Supply database connect str
;; For mysql => URL DATABASE USER PASSWORD
(,database '("localhost" "botlist_development" "USER" "PASSWORD")
:database-type :mysql
:pool t
:if-exists nil)
,@body))
(defun test-db ()
"Wrapper method for the test case; connect to the database
and run a simple query"

(with-db (db)
(setf clsql:*default-database* db)
(clsql:status t)
(let ((z (clsql:query
"select main_url from entity_links limit 0,2"
:database db))
(d (clsql:select 'entity_links
:database db
:flatp t)))
(print z)))
(format t "After with-db~%")
(clsql:status t))
(defun main ()
"Main entry point for the example"
(format t "Running - ~%")
(test-db)
(format t "Done - ~%"))
(main)
;; End of the File


Full Source Download

http://haskellnotebook.googlecode.com/files/lisp_webapp_example.tar.gz

Resources

I didn't cover the code in much detail, so just use the list of resources to learn more about Hunchentoot, the database code and anything else that was mentioned.

http://www.newartisans.com/blog_files/common.lisp.with.apache.php
http://www.weitz.de/hunchentoot/
http://clsql.b9.com/
http://common-lisp.net/project/asdf-install/
http://www.sbcl.org/
http://www.weitz.de/starter-pack/
http://haskellnotebook.googlecode.com/files/lisp_webapp_example.tar.gz
http://en.wikipedia.org/wiki/Buddhism - Also useful

11 comments:

Mathias said...

Thanks to this guide I got around porting a Hunchentoot application I made to run under Windows as well as Ubuntu. So now I don't have to keep a VMWare instance running in order to use that app :) However, as SBCL on Windows does not use threads yet I assume I will get problems with high trafic.

Anyway, thanks again!

Berlin Brown said...

Yea, I don't understand why the SBCL developers don't acknowledge windows . The build is many iterations behind the linux builds. You don't have the ability to add various options like "thread support", etc (unless I am missing something).

It is nothing short of arrogance.

Mathias said...

I think it is really unfair to call them arrogant. After all they provide a totally free, good and respectable CL implementation. I would call it "unfortunate" maybe, or boring, but arrogant, come on! The source code is free for all to hack on if you want to contribute. In fact, I think I heard that there is a Google Summer of Code project to add threads for the win32 version.

Berlin Brown said...

Good point. And, it is unfortunate.

I remember reading on a various mailing list that they just don't have many developers that use windows. I guess that makes sense.

eGlyph said...

Hello Berlin,

I was using qemu+slime over ssh for development purposes and decided to give a try to your method. I have to note that in addition to steps you've listed I had to compile sb-posix, but it went fine after all.
One quirk is that the line like (defvar *server* (hunchentoot:start-server :port 8080)) blocks the interpreter. Is it related to the absence of threads support in sbcl on Windows?

Berlin Brown said...

"Is it related to the absence of threads support in sbcl on Windows?"

yes, that is correct. And I can't remember off-hand if CTRL-C should get you back to the SBCL prompt.

You are probably right; sb-posix should be included.
Is sb-posix included in the LispStarterPack?

eGlyph said...

Sb-posix is a part of sbcl package, if I'm not mistaken.

Ctrl-c kills the running instance, at least when using SLIME.

In a couple of days I'll try describe my way of setting development environment for Hunchentoot on WindowsXP.

Mathias said...

Thanks to Anton at Hunch' mailing list I got the tip to use THROW-CATCH to stop the server. This is what I have:

(defun start-hunch ()
(format t "~%~%** Starting Hunchentoot ** ~%~%")
(force-output)
(catch 'stop
(hunchentoot:start-server :port 3000))
(format t "~%~%** Hunchentoot stopped ** ~%~%Start it again with ~%~% (start-hunch)~%~%"))

And to that I have a handler that THROWs the tag 'stop.

Works nicely. Not as nice as under GNU/Linux, but still OK.

Berlin Brown said...

Thanks mathias, I am going to try that

sk said...

Excellent post. If you could also tell how you managed to make sbcl+clsql talk to mysql it would be just perfect.

I've already spent several days trying to accomplish this and feel really stupid, since I keep reading that people could run this setup without problems.

This is a most recent post I found about somebody using this setup, so I thought I would ask for tips here.

Here is what I tried:

-- First - I installed mysql, connected to it, setup user password and empty database.

-- Next I copied all necessary libraries to my local folder, pushed their paths into asdf search path and started compiling.

-- Everything worked ok until I tried to compile clsql-mysql. It complained that it couldn't find libmysql.dll and libmysqlclient.dll. I found libmysql.dll in mysql\bin folder but I couldn't find libmysqlclient.dll anywhere. Some googling tipped me to make a copy of libmysql.dll to libmysqlclient.dll. Which I did.

-- Trying to compile clsql-mysql again - again cannot find these libraries. Pushing their location to clsql-sys:*foreign-library-search-paths* didn't help.
I ended up copying them into clsql/uffi folder.

-- Another compilation try: it didn't complain about libmysal.dll and libmysqlclient.dll! Good!

-- But, now it complained that it couldn't find clsql_mysql.dll. I found it in clsql/db-mysql folder. Copied to clsql/uffi folder - this didn't help. Pushing db-mysql folder to *foreign-library-search-paths* didn't help either. Copying it into windows/system32 folder also had no effect.

What am I doing wrong? Reading your post and comments it looks like many people were able to use clsql with sbcl and mysql without any problems.

Sorry for long and off topic post, but I was trying to resolve this for so long that I would appreciate any tip to try.

Thanks

Sergey

sk said...

Figured this out.

Turns out it was because I tried to run it from D: drive.

http://lists.b9.com/pipermail/clsql/2008-September/001618.html