Tuesday, November 27, 2007

Web requests with the Haskell HTTP library


Web development work can exist at the server level but it can also exist at the client level. You may need to build a simple HTTP load test framework or test if a particular web application or page is available. It is also possible that you need to upload files. I have built a couple of simple HTTP frameworks in Java and Python but this be a first attempt at Haskell based framework. Actually, in this small example; there isn't much of a framework but just an example on how to use the most recent HTTP library update.

Install the HTTP library

It seems that GHC Haskell release 6.8 encourages that libraries exist outside of the base Haskell build. For example, the parsec parser library requires that you download parsec from haskell.org or other location or extract the source from darcs. Basically, development will continue on haskell libraries outside of the typical GHC development. With that being said, the same holds true for the HTTP library. You can pull the source from darcs or at the time of this blog entry, the latest HTTP library is HTTP-3001.0.2. I downloaded and did a Setup.hs configure/build/install against GHC version 6.8.

If you attempted to compile haskell source with out installing this library, you will get a package not found error.

ghc --make -package HTTP ManClient.hs

What does and do and how

The goal of this simple application is to post data or actually could be used to post to any website. Also, if you simply replace the POST string type in the code to GET, you can use this code to read content from a website and print the web page to the console.
Get Function

get :: URI -> IO String
get uri =
eresp <- simpleHTTPProxy (manRequest uri)
resp <- handleErr (err . show) eresp
case rspCode resp of
(2,0,0) -> return (rspBody resp)
_ -> err (httpError resp)
showRspCode (a,b,c) = map intToDigit [a,b,c]
httpError resp =
showRspCode (rspCode resp) ++ " " ++ rspReason resp

Build the Request Type

manRequest :: URI -> Request
manRequest uri = Request { rqURI = uri,
rqMethod = POST,
rqHeaders = [
Header HdrContentLength
(show (length forumRequestBody)),
Header HdrUserAgent userAgentMimicIE
rqBody = forumRequestBody }

Download Source


Sunday, November 25, 2007

Blog Spam: Cat programming language

This is my first blog spam post but I hope to post more in the future. Basically, blog spam is a post that does not have any real content and is generally used to promote the blog. In my case, I am not promoting my blog but I just thought this post on the Cat programming language is interesting and I didn't really feel like trying to come up with some witty entry so instead I will just copy the project description and resource links:

"Cat is a functional stack-based programming language inspired by the Joy programming language. The primary differences is that Cat provides a static type system with type inferencing (like ML or Haskell), and a term rewriting macro language extension language called MetaCat."


What I also thought was interesting was that Cat is embeddable.


I hope to make OctaneLang embeddable as well.

Wednesday, November 21, 2007

Science behind botlist

I haven't done a lot of work on botlist recently. I have been captivated by other projects. But I have a couple of doable features that I want to implement.

First I need to develop the push-pull system a little bit further. This allows for botlist to be updated remotely. Right now, it is working fine but I want to ensure data integrity for when the client is communicating with the server. Basically, when the client sends data uploads to the server, I want to ensure that the server has all the right data. It will probably be a simple MD5 scheme to validate a binary upload or something similar. As of right now, the push pull system just ignores invalid records when they are encountered.

Recommendation System

Botlist is a really simple system, right now. New links are submmitted by users or by the remote bots and added to the table. I am currently working on a system for the main page to be filled with popular links and actual content. Recently I picked up a USA today newspaper and amazed at the layout. Who decides and under what conditions articles should be placed at the front. How much text is too much, too little, what is captivating to the reader. It would be nice mimic such functionality. If you have visited the Drudge Report, it would be similar to how that works (minus Drudge manually updating his popular links).

Sunday, November 18, 2007

Additional Resources (link blog entry: haskell and parsing links)

My old take on a blog was to write useful entries. That didn't go so well, I mean I didn't really get that many readers. So this time around, I am just going to post whatever I feel like posting. A free for all of sorts. In this entry, here are some useful haskell and other functional programming resources and blogs. Blogs and articles that I have been reading over the last couple of days:



Wednesday, November 14, 2007

More practical Scala, searchable help document system

I go into more detail on the google code site, but here is a tool for creating searchable help documents:

"This is a small utility for indexing developer help documents with Lucene. And because some botlist documents have already been indexed, you can use it to search some of those example docs."


Tuesday, November 13, 2007

Practical Scala; Walking a tree directory

One of my requirements for botlist is to have a searchable help system. E.g. instead of man pages or Java API. I want to be able to add files to a directory. Index the files in the directory and then have the data as searchable. All of this through a command line interface. I started some initial code and used the following Scala idioms:

First Version, use of File Class:

class DocWalkFile(file: File) {
def children = new Iterable[File] {
def elements =
if (file.isDirectory) file.listFiles.elements else Iterator.empty;
def andTree : Iterable[File] = (
Seq.single(file) ++ children.flatMap(child => new DocWalkFile(child).andTree))
def listDocuments(dir: File): List[File] =
(new DocWalkFile(dir)).andTree.toList filter (f => (f.getName.endsWith(".java") || f.getName.endsWith(".txt")))

Second Version:

import java.io.File

object DocWalkFile {
def andTree(dir: File): List[File] = {
val fileList = dir.listFiles.toList
(fileList filter(f => f.isFile && f.getName.endsWith(".txt"))) :::
(fileList filter(f => f.isDirectory) flatMap (f => andTree(f) ))
def main(args: Array[String]): Unit = {
if (args length == 0)
println("usage: DocWalkFile [dir]")
else {
println("using dir " + args.apply(0));
File(args.apply(0))).foreach(f => println(">> " + f))


Sunday, November 11, 2007

Short primer, importing a SVN repository into a GIT one

I wanted to convert parts of the Botlist project to GIT. Basically, I did just that. I saved the history from the Subversion repository and was able to import that into a new GIT repository. My entry will discuss the steps that I took to make that happen, also I work with several different remote machines so I hacked together a startup script for git-daemon.

Get GIT and install

I am working with an Ubuntu Dapper machine (yes, it is an older distro) and I didn't want to go through the hassle of using a very out of date version of git, so we are going to download the most recent git from kernel.org.

wget http://kernel.org/pub/software/scm/git/git-
tar -xvf git-

Pre configure and Installation

Actually, before running configure on the git source; you may need to get two

On Ubuntu, I did the following:
  • sudo apt-cache search libsvn-core-perl
  • sudo apt-get install libsvn-core-perl
  • sudo apt-get install libcurl3-dev
In order for the git-svn to work, you will need the libsvn perl libraries. For working with http operations, you will need libcurl.

Run ./configure

At this point, you can start to build the git source; type ./configure at the command prompt and the subsequent make/install commands. Basically, standard steps for compiling and installing a typical linux source package.
  • ./configure
  • make
  • sudo make install
Git is installed, now get the subversion project (history)

Assuming git was built properly, you should be able to import a subversion
project. I suggest creating a directory and then have git create the .git repository information in that particular folder.

  • sudo mkdir -p /usr/local/var/gitscm/botlist.git
  • cd /usr/local/var/gitscm/botlist.git
  • sudo git-svn clone http://openbotlist.googlecode.com/svn/trunk/openbotlist/
At this point, you will see verbose output of git creating the botlist repository.
That is all that is needed to create a repository from subversion, notice that the svn (.svn) working directory folder is not included in your git project.

Running the git-daemon at machine startup

Unlike other linux daemons, setting up git-daemon is pretty simple. At least for running the git daemon server under the git protocol. Basically, git-daemon is a server for hosting git repositories and this particular server defaults to running on port 9418. GIT also supports hosting repositories through HTTP.

Run whereis git-daemon to find out where the executable is located. It is up to you how to pass the git-daemon arguments, I used a simple wrapper bash script and launched git from there.

/usr/local/bin/git-daemon --reuseaddr --verbose --base-path=${BASE_PATH} --export-all -- ${BASE_PATH}

[End of Script]
Change the base-path path according to where you want to store your git repositories. In the script above, gitscm is just a regular directory. The subdirectories (for example, botlist) are actual git repositories (that contain the .git directory). Create a symbolic link in your init.d directory, something along the lines of this:

ln -s /usr/local/bin/git-daemon2 git-daemon2

With ubuntu, you can run update-rc.d to ensure that the git-daemon2 script gets launched at startup:

  • sudo update-rc.d git-daemon2 defaults

Run and Test

Now, you can start the git-daemon2 script
  • git-daemon2 &
  • git ls-remote git://
  • To get the project, cd to some directory
  • git clone git://

Tuesday, November 6, 2007

Now Playing: Brandenburg Concerto

In my effort to find the perfect development music, I stumbled upon these concertos by Bach, in particular "Brandenburg Concerto No. 2 in F - 1 Allegro"


They are widely regarded as among the finest musical compositions of the Baroque era.

Friday, November 2, 2007

13949712720901ForOSX Vote! I am actually helping Java

I am not really a big Mac user, but I use it every once in a while. And I still develop in Java, so I am helping the Java cause by posting this vote to get Java6 OSX.

Personally, I feel that Apple has a right to leave out whatever software they choose. Plus, they already have Java5 working with their operating system so it is not like they are totally ignoring the Java community. But, if enough momentum is created for adding the new Java; would it hurt for them to listen to some of their users.

So, I am doing as the blog instructed and here is my vote:


Thursday, November 1, 2007

Google, we are watiing?? on opensocial


Google, what is the deal? Where is opensocial. I am actually kind of excited about this technology. I heard reports that library was supposed to be released on Thurs. It is 6:42PM (PST) , you have a couple more hours left. Don't be evil by lying to us.

I have a new perspective, Firefox and Flash bugs

After a little bit of research I eventually ended up on at the bugzilla.mozilla.org site. It seems like there are various issues with flash and FF. So hopefully, I will do some work in trying to resolve my problems as opposed to just ranting.