Friday, June 20, 2008

Off Topic - Iron Man Sucks (*** MAJOR SPOILERS ***)

Iron Man Movie Sucks (Note; there are major spoilers, if you don't want to know intimate details about the movie then don't continue reading)

Ok, it doesn't suck, but it is not nearly as good as people say it is. Reviewers are raving over this comic book adaption. The American Idol populous is going nuts; "Iron Man is the best movie ev4r!!!!". I strongly disagree. There are certainly some good CGI moments and scenes of good comic book heroism, but at the end of the day, this is yet another "silly" comic book to movie translation. There are a couple of problems that I had with this film.

  1. Simplistic Plot - You can figure out the plot in the first 15-20 minutes. Even from the first 5 minutes I could kind of get an idea on what is going to happen. "Rich guy in the desert with army soldiers? Let me guess...he gets captured.", "Rich guy in the desert is detained and left alone with a bunch of electronics. Let me guess, he is going to be build a suit, the Ironman suit". And why are the terrorists watching him on camera, it is clear he isn't building a missle, why don't they just shoot his ass. We see many scenes where Robert Downey Jr's character is testing out the robotic limbic system and the helmet. Isn't it blatantly obvious that is building something other than the missile he was asked to build. Give me a break.

  2. Tony Stark (Ironman) is a misogynistic, elitist tool - Robert Downey plays a great a**hole, but I will get to him later. The Tony Stark character is a horrible rich guy turned super hero or at the very least Robert screws up the role. There are so many acts of fake misogyny. Stewardess suddenly turn into strippers and dance on poles. The reporter character goes from doing an interview with Tony Stark to suddenly sleeping with him. The subservient Pepper is not only a beaten down secretary but then she falls in love with Stark. Give me a break. Christian Bale in Batman Begins and Michael Keaton in the original Batman do a much better job.

  3. Silly Scenes and Plotlines designed for teenagers - The overall plot is not too terribly bad, but the implementation of the story is so dumbed down. I already mentioned how Tony Stark was allowed to build a complete Ironman suit in the dessert while the captors watch on camera? Silly. He walks out of the terrorist camp and then is suddenly shot into the air, lands and his Airforce friend just happens to pick him up in the dessert. He comes home to build a better Ironman suit. This is what I find strange. He goes out of his way to be this secret hero and hide what he is working on. Where does he hide? In his workshop in his house? No secret lair or hideaway in some third world country; Tony Stark hides from the rest of the world by going downstairs into his basement. Not even to mention the fact that his arch nemesis seems to have keys to his house and can freely come and go. I found this kind of strange. The bad guy in the film goes out of his way to send Tony Stark to some terrorist camp to be killed but it seems like it would have been easier to just kill Tony while he slept. "What are you WORKING Tony?? What are you WORKING Tony?? ..." Why do you have to ask, why don't you just sneak into his basement!!!!!

  4. Pepper Potts Role and other acts of Sexism - Clearly this film was an all male production. The feminist movement will have a lot to complain about with this film. Pepper's role in the film just made me squirm. "Why are you working for this a-hole?"

  5. Robert Downey Jr. OMFG would you shutup? - I will be honest. I like Robert and he is good in a lot of his roles. And maybe sobering up had a lot to do with it. But Robert Downey was terribly annoying in this film. Every line had to add some witty sarcasm. He also had to spit out as many words as he can as fast as can and then end with some witty punchline. "I am the richest, smartest, weapons arms dealer that can build robots while fixing on cars and dining and seeing women, blahhaahalhalhahlahlhalhflahlhlahlahlahlala...would you sleep with me?" Every single line in the movie. It drove me nuts.

  6. No real enemy threat or bad guys - Who is the enemy in this film? The terrorists in pajamas and AK-47s that live thousands of miles away? His old, fat-ass former partner? Ironman gets to the end of the movie without really fighting anyone.

  7. In summary, I thought this movie was yet another over-hyped Hollywood film (shocker!!!).

Saturday, June 14, 2008

Is Java the new COBOL? Yes. What does that mean, exactly? (Part 1)

"If you want to know the past, look at your present. If you want to know the future, look at your present." -- Gautama Buddha

Java is the new COBOL, whaaaat?

    If you are software developer that mainly writes applications for the web; even if you aren't a Java/J2EE developer, you have probably heard the new meme circulating the web and blogosphere. "Java is the new COBOL". Most blogs and technical writers regurgitate that statement and follow-up by saying that EJBs suck, and some companies are considering Ruby on Rails. I didn't really think much of these posts. I get the impression that a lot of these "managers" don't write any Java or COBOL code. They are really not highlighting the seriousness of the problem. COBOL development has remained stagnant for decades. Java may see the same problem in the future. The Pentagon/DOD has a trillion dollar COBOL problem because it is so difficult to change their ancient infrastructure. Anyone who is exposed to the Java world will become frustrated with some aspects of the environment. The beginner will certainly become frustrated with the amount of work it takes to perform simple tasks like opening a file or working with Swing to create a small GUI application and they will find it much easier to perform these tasks in a language like Python. The seasoned developer will find fault with Java's stability or the complexity of the various web frameworks (XML hell). The programming language designer will find fault with all aspects of Java. I believe that the "Java is the new COBOL" is a politically correct way for saying "Java sucks". I won't disagree with those statements. I am not overly partial to any technology, I just want to get work done and find the technology that makes that work as painless as possible. As an added bonus, I want to ensure that my time is spent on a project is worthwhile. What is worthwhile? If I had a choice, I would much rather work on or learn more about the software used to work with the Phoenix Mars Lander [1]. The lander was sent 422 millions miles and the robotic systems must operate autonomously, sending data and images back to Earth. The software used to build this software is probably more complicated than what is on the other side of the spectrum, for example MySpace or a MySpace application. Some software is more complex, more interesting, and generally more useful than other software. I am sure that the JPL team had to plan for almost any contingency from the physical complexities of the Mars environment, from the hardware components as well as within the Phoenix's internal software components. Not everyone can or would want to work at the JPL, creating robotic software. There are always going to be different software needs and different requirements, but the software development community should make an effort to plan for contingencies and seek out the best technical solutions for their users. Does a Java architecture provide the best solution? Does a COBOL one? I guess depends on the requirement.

What about actual COBOL developers? What do they think?

"The use of COBOL cripples the mind; its teaching should, therefore, be regarded as a criminal offence." -- Dijkstra, 1975 [22]

    What about critical applications like in the banking industry? Use your imagination to come up with the possible requirements for web oriented banking software for internal and external applications. These applications might need to keep up with credit-card, or debit-card transactions for online payments. They might want to store personal profiles of the users. Users will need to login into, register for, pay for and maintain their accounts. The application has to be secure and follow all of the SEC guidelines. Depending on the company, you might have hundreds of thousands of users with a running database that has been collecting data for 40-50 years. Imagine the number of transactions that must be logged in order to run your business. This activity must be logged and must be accurate 24/7. I can only imagine if I logged into my SunTrust account and my bank statement said I had $40 when I really had $40,000. There are issues with these systems and errors due occur, but they are kept to a minimum. Imagine the requirements for the external application used by members of the public and then imagine the number of internal applications needed to maintain the public facing tools. You could have log-monitors of the logs. You could have sophisticated, custom build tools. You could have custom requirements document management systems. I remember laughing and reading a recent article about DreamHost's billing issue. "On January 15, 2008, the billing system was mistakenly used to bill users up to December 2008." That couldn't happen for all of the users at Bank of America or SunTrust.

    Back to COBOL. COBOL was created in 1959 and is a mainstay of business programming in the finance industry. It is normally being used for the back-end and data processing work and various Java and .NET middleware libraries are being used to connect web interfaces to the running mainframe COBOL software. "In 1997, the Gartner Group reported that 80% of the world's business ran on COBOL with over 200 billion lines of code in existence and with an estimated 5 billion lines of new code annually.[4] IBM is one of the larger mainframe hardware vendors. So, if you hear about a mainframe shop, you can safely assume they are running a bunch of IBM hardware and software. There are still COBOL jobs available, there are still software and hardware vendors like IBM pushing their products. And it doesn't have to be an z/OS or AIX system. You can run SUSE linux on a virtualized LPAR (logical partition) or any other supported operating system all on the same machine. I don't understand any of the terminology and architecture of COBOL/Mainframe systems. Most COBOL (COmmon Business-Oriented Language) interact with large databases, that might include launching stored procedures and manipulating. A mainframe JCL (Job Control Language) script that will be placed on the queue and run as a job on the mainframe. It can compile and link a COBOL program and then execute the program.

    The listing below is a brief snippet of COBOL code, without spending much time with COBOL you can get a glimpse at why it isn't something one would consider to be a beautiful language.



    Actually, these systems are fairly stable. Like I said before, a bank system has to run 24/7, collecting and manipulating data. In a J2EE/Linux/Unix environment, if the your JVM lock-ups or crashes due to memory leak issues or unforeseen NullpointerException bugs; normally you have to call someone, identify the problem and immediately get your server running again. A mainframe zSeries system may only experience a couple of minutes of downtime a year and all of your hardware sub-systems can be hot-swapped.

Why are you telling me this?

    That is COBOL. So, why isn't it used for new and exciting web-development today? It is obvious once you start working with the language. Here is a good explanation that a hybrid Java/COBOL developer told me; "It is too basic, there isn't a lot of new development or methodologies for COBOL. It isn't exciting to work with. COBOL doesn't have many modern web development libraries or frameworks". From my talking with a working COBOL developer, he wasn't excited about working with it. You don't even have to build any COBOL applications or look at any code, you can already tell that the language is dead. There aren't any new books being written for the language. There are no conferences or training. A COBOL program 10 years ago is written in the same manner that one is written today. The same methodologies are used, same best practices, etc. In 2008, COBOL could not keep up with the demand for the user's requirements. I can't say for sure (I haven't written or deployed any COBOL code either). User's want sophisticated web user interfaces and GUI applications, modern 3D games. A COBOL application is ok for moving 100 credit-card transactions from one database to a separate log history table but it is not the best tool to build a game like Halo 3 or software systems for the Phoenix Mars Lander (using my original example). Essentially, new development with COBOL is dead.

What about Java, is Java the new COBOL?

    I haven't taken a lot of the blogosphere seriously on the "Java is the new COBOL" meme because of the way that a lot of technology leaders frame the statement. The question is, "Should Java be used for new development and applications?". If you are a die-hard Java fanboy and your answer is yes under all circumstances; then you are doing a disservice to your users and your career as a technical professional. You are moving in the right direction, if your answer is no or maybe; you are dedicated to looking at:

  • The various software technologies and evaluating them with a rational, objective analysis on how the technology can benefit your user's needs
  • The available developer resources (this may include only one developer)
  • The overall future maintenance of your application

What is wrong with Java?

"One does not accumulate but eliminate. It is not daily increase but daily decrease. The height of cultivation always runs to simplicity." -- Bruce Lee

    I won't add more to what has already been said about Java and J2EE, later I am going to post some of my favorite "Java Sucks" quotes. You might also see some misinformation on the blogsphere with their use of the term Java. Some use it to describe the Java programming language, some use it describe the entire Java platform. Some people even take it a step further to include Java the programming language, the virtual machine runtime created by Sun or other software vendors, J2EE (Enterprise Edition, including EJBs, Servlet Spec, JMS, JTA, etc), or any libraries built with Java. In my blog post, I am talking about Java-everything (including third-party applications) unless I specifically target the "programming language" or "virtual machine", for example. From some people, if they say "Java Sucks", you can always respond with; "Well, Sun's HotSpot just-in-time compiler is used to achieve great runtime speed. And just a note, some mistakenly criticize Java's(the JVM) performance based on Sun's early Sun's implementation. In 2000, Sun released the HotSpot VM with just-in-time complication which including much improved performance, compared to the bytecode interpreter released in 1995. [12] And Java is still used in many complex production systems. Plus, you are not always tied to using the Java programming language to write your applications. Many other programming languages run on the JVM, including.

  • JRuby (The Ruby language on the java virtual machine) [9]
  • Jython (Python on the JVM)
  • Scala (A multi-paradigm (OO and functional programming) strongly-typed programming language [8]
  • ABCL (Common Lisp)
  • Clojure (A lisp implementation on the JVM

Interesting quotes about Java

  • "The programmers you'll be able to hire to work on a Java project won't be as smart as the ones you could get to work on a project written in Python....frankly, the fact that good hackers prefer Python to Java should tell you something about the relative merits of those languages." -- Paul Graham [16]

  • "What in the world is application-level crap like checkPrintJobAccess() doing in the base language class library? There's all kinds of special-case abstraction-breaking garbage like this. ", "Interfaces seem a huge, cheesy copout for avoiding multiple inheritance; they really seem like they were grafted on as an afterthought. Maybe there's a good reason for them being the way they are, but I don't see it; it looks like they were just looking for a way to multiply-inherit methods without allowing call-next-method and without allowing instance variables?" -- jwz on the java language [17]

  • "Scala is a far superior language to Java...The status of “Java” has been artificially elevated by its groupthink members such that any attack or suggestion of inferiority is reviled, even if the suggestion is perfectly rational and able to be supported with evidence" -- Tony Morris, Offending Religiosity [18]

  • "Static typing is limiting", "Arrays are broken", "A double-whammy when combined with static typing. There is this weird distinction between object types and primitive types", "Instead of adding expressive features to the language, the Java designers decided to focus on promoting code duplication, aka design patterns instead". "No REPL". -- Slava Pestov (November 21, 2004) [19]

  • "WebWork, Tapestry, Spring, and pretty much every non-Struts framework users scoff and laugh at JSF. It’s ugly, it’s not intuitive, and it is hellbent on the Microsoft style approach; fuck the users and force them to use clever tools." -- Hani Suleiman, author of the BileBlog

  • "Deployment is still the painful arduous hope-and-pray-you-get-a-helpful-hint process we’ve all come to know and love. The smallest hiccup will cause the server to flail about wildly, spewing error messages with a frequency that is only matched by the number of new pointless codehaus projects every week. The error message themselves are as delightfully whimsical as they’ve always been. Random ObjectNames, with tantalisingly empty ‘I depend on:’ proclamations. The ‘Depends on me’ of course still impressively manages to insist that ejb’s within a module depend on a seemingly random other ejb in a far flung module that has absolutely no dependencies to anything. Hot redeployment of apps that failed to start up doesn’t work worth crap, the stacktraces seem longer, and the whole thing just makes you run to your nearest j2ee vendor and beg them to let you fling cash at them just to stop the pain." -- Hani Suleiman on JBoss

  • "Java desktop apps succeed only in niches where UI design and usability don't matter: development tools and enterprise software. Programmers expect things to be crude and complicated... and the poor users of enterprise software don't have a choice..." [27]

XML Configuration Hell

XML and property configuration hell does not plague all Java development, this is one misconception that permeates the blogosphere. If you aren't writing web applications, you can certainly avoid XML oriented frameworks. And if you are working with web applications, you can use lighter frameworks like Wicket. One of Wicket's selling points is a "refreshing lack of XML". But there is a certainly a tendency with Java web frameworks to be configured by needlessly complex XML configuration files. You can attribute some of this universal standard to Sun's default J2EE XML configurations. The J2EE web.xml, application.xml, WEB-INF/*. One or two simple system XML configurations are not bad. But, you start to enter XML hell when you want to combine technologies. By default, you have to deploy a web.xml, application.xml. If you are using and combining common frameworks like Struts2 MVC, Hibernate ORM and Spring2, Log4j, JSF, Google Web Toolkit, Axis you could have one to many different configuration files per framework. Essentially, you could have a minimum of five or six different configuration files to setup even the simplest web application (if you mistakenly use those frameworks). The practice of over-engineering every solution is very prevalent in the Java community, this is why some may consider Java to be the new COBOL. COBOL was so stuck to its niche that it couldn't really evolve to match the ever changing software world. It wouldn't make sense to provide COBOL web portals or portable GUI applications even though the back-end systems were originally developed in COBOL. Java is falling into the same trap. Why should a developer have to waste time worrying about five to six hundred line configuration files, on top of a poorly designed, inconsistent Java API when web frameworks like Rails or Django can accomplish similar tasks in twenty or thirty lines of code? More code means that more things can go wrong.

Web 2.0 and New Requirements

    Grace Hopper, the author of COBOL, helped standardize programming languages used by the Navy to integrate incompatible embedded systems. Hopper helped build many of the Navy's onboard computers; "Hopper was tasked with developing a program for shipboard ADP equipment support, building compilers for shipboard minicomputers was a major part of her workload". [15] Web development is not the end all of software development, but a lot companies understand the benefits and safety net associated with this ubiquitous style of computing. And a lot of companies and users don't just want a basic MVC-1 database driven application. They want more, they want applications that interoperate with other successful services. For example, the popular web application [14] contains user-driven restaurant and business reviews in and around large cities. The application also links the location of these businesses with google maps. You can pinpoint on a particular restaurant and if you don't like that restaurant, you can highlight similar restaurants in the area on the map.

    These are the new wave of applications that users are wanting. They have seen it done and they want more of it. Is Java capable of keeping up with new requirements? Are other languages and platforms better and easier to use to meet these software demands? Here are just a couple of applications that are being developed in the Web 2.0 world. And yes, the term Web 2.0 doesn't have any real meaning; it is yet another tech buzzword created by the consultants and trade magazines to describe trendy software technology. Essentially, these software problems are not complicated and certainly not new.

  • Ajax and JS Widget Support
  • Wiki Sites
  • Web Forums
  • Portlets
  • Mashups
  • Social Networking and User Driven Applications
  • Web Services
  • Semantic Web (E.g. FOAF)
  • Rich Internet Applications
  • Cloud Computing
  • Clustered Applications

*GASP* Where are the libraries?

    Building modern web applications around the Java platform is skirting on the edges of web-development, programming, and network administration. I don't think the term Java programmer is an accurate term to describe a Java developer's functions and workload. Java developer's aren't always programming to put together an application for the user. I was once tasked with converting HTML output to PDF documents. Most of the work involved finding good opensource libraries that supported at least HTML4 tags, XHTML and CSS. Most of the libraries required an input HTML string and streamed the output PDF to a file or some other medium. There was a requirement and a need; I researched various tools and integrated the library with our web application. I didn't do much programming. A lot of Java work follows a similar work flow. Traditional programming is not always required. How do you add Lucene full-text search to your application? Add Lucene to your classpath, come up with strategies for indexing your documents, provide text edit box for user's to query the system and you are essentially done. Once again, programming is not required for you to add this functionality. Let me say it another way; imagine if the Lucene library wasn't available and you still needed to provide full text search. If you wanted to build a search library, more programming work will be involved building the API then using an existing one.

What about the other Programming Language Alternatives? Could you use more than one General Purpose Language in your Architecture?

(Actually, to say languages like Lisp, Scala and Haskell are alternatives to Java is kind of an insult to these technologies. Java is really a corporation inspired alternative to existing software systems that garnished a lot of momentum over the last ten years.)

    One thing I never understand about the Java group-think community is the fear of integrating new programming languages into their environment. At a minimum, most Java web developers have to work with Javascript, XML/XSLT, HTML/XHTML, SQL and Java itself, sometimes even C/C++. Designing an application with one or more general purpose languages is not totally a foreign concept. Twitter attempted to solve some of their scalability issues by integrating their Ruby on Rails web platform with Ejabberd Jabber/XMPP server as their instant messaging platform. Erlang is the most widely used programming language that has been known to use a message-passing concurrency model. [21] Erlang is strong at parallel, distributed applications. If your requirements change or you see an increase in demand that your current set of resources can't meet, then shouldn't investigate other options. Other programming languages? Here are just a dozen or so stable languages that have vibrant developer communities and capable libraries that can even be used for enterprise development:
Popular and Mature Modern Programming Languages

  • Python/Jython
  • Ruby/JRuby
  • Erlang
  • Haskell
  • Scala
  • Clojure
  • Smalltalk/Seaside
  • Lisp (Common Lisp/Scheme; SBCL, CLisp, PLT Scheme)
  • Factor
  • OCaml
  • F#

To boldly go where no man has gone before, The Enterprise

    The requirements for web development are not going to change and will increasingly get more complex. In defense of the Java developer, he may believe that his only option is to architect his application with a J2EE platform because the application server may have all of the libraries that he or she needs to complete his requirements. Normally, this is not the case. He probably doesn't need JCA, JMS or all of JAAS or EJBs. Yet, a compliant J2EE server will provide all of this whether you need it or not. But, if the enterprise developer is considering new languages and platforms, he might need a minimum set of libraries. If we go back to the requirements of the online banking software; you aren't going to consider a web platform that doesn't support HTTP over SSL. And you probably don't have the resources to implement SSL yourself. Most modern languages have some of the features listed below. They are mature and have been a part of the language for years. If the language doesn't have the particular feature, there is nothing preventing you from using a combination of general purpose languages in your architecture.

Some Typical Enterprise Requirements

  • Mature IDE (E.g. Eclipse, RAD, IntelliJ IDEA)
  • Unicode Support
  • Stable XML Parsing Libraries
  • Database drivers (E.g. JDBC)
  • SSL support and Security platforms (E.g. JAAS)
  • Email Libraries
  • MVC Web Frameworks
  • (Soap, REST, RPC) Webservices libraries
  • Search (E.g. Lucene)
  • Legacy communication libraries (E.g. JCA, CICS, POS
  • Chart generation libraries
  • Image manipulation libraries (E.g. JAI)
  • Scalable - handle an increase in users or demand
  • Good Speed Performance
  • Technical Support?


    So the question for the 2008 age developer is this; to all the millions of one language Java developers that are out there, are you going to continue to only work with the Java language and platform, five, ten, twenty years into the future? Are you going to consider the alternatives? Do you think your users might benefit from other technologies? And I am sorry but if the answer is no, fear of change is not the most adequate response. I have been reading a lot about Richard Stallman, the free software advocate. He has one central theme that he uses to defend his free software movement; free software is not about the developers, it is about the users. If you are dedicated to providing robust software, wanting to make a lot of money is not going to be one your goals. It really boils down to what you want to accomplish as a software developer. Is your goal to defend some programming language religious ideology or provide useful computing applications to users and the software community?

[12] (Image from wikipedia)
[15] Grace Hopper: Admiral of the Cyber Sea", Kathleen Broome Williams 2004
[27] In Which I Think About Java Again, But Only For A Moment


[23] A lot of developers have many suggestions for what and what not to teach in computer science. I recommend a course on the History of Computing/Software. I wouldn't have been so happy-go-lucky about Java and Fortran languages if I knew the entire history of software at the time.

[24] Why did you disable commenting? I am intentionally disabling comments. I don't want to get into the "Java Sucks", "Rails is awesome", "Lambda the Ultimate laughs at the Java community" argument. There are a couple of people that can bash Java and get away with. Slava Pestov (creator of Factor), Tony Morris (developer on ScalaCheck and works with Scala), Hani the Bile Blogger can get away with criticizing Java. They are fluent in the language and are great programmers. However, my blog post may encourage a bunch of wanna-be Rails fan-boys or Java drones to tell me how I have gone wrong. I had two goals with my post. First; I wanted to respond to the "Java is the new COBOL comments". Second; I wanted to understand why we use X or Y programming language. In some respects, I was talking to myself. Why Java? Why Lisp? Why Haskell? I know I probably have flaws in my posts and criticism is certainly welcome. Disabling comments will encourage healthy criticism. If you want to add something to the blog post or other comments, you can email me: berlin dot brown at

Euler Problem Number 1 in Scheme (Comparison with CL and Java)

I know the Euler project administrators try discourage to users from posting Euler solutions, but I hope there isn't any harm in posting an solution implementation for the first couple of problems. I plan on posting possible solutions in a variety of different programming languages. Here are three implementations in Scheme, Common Lisp and procedural Java.

;; Euler problem, number 1
;; Environment: mzscheme
;; If we list all the natural numbers below 10 that
;; are multiples of 3 or 5, we get 3, 5, 6 and 9.
;; The sum of these multiples is 23.
;; Find the sum of all the multiples of 3 or 5 below 1000.
;; References:
;; [1]

(define (euler1 n)
(let ((x 0))
(do ([i 0 (+ i 1)]) [(= i n)]
(if (or (= 0 (modulo i 5))
(= 0 (modulo i 3)))
(set! x (+ x i)))))

(define (main)
(display "Running\n")
(display (euler1 999))
(display "\nDone\n")


Common Lisp

(loop for x from 3 to 999
when (or (zerop (mod x 3))
(zerop (mod x 5)))
sum x)


public class euler1 {
public static int euler1(final int n) {
int sum = 0;
for (int i = 0; i < n; i++) {
if (((i % 3) == 0) || ((i % 5) == 0)) {
sum += i;
return sum;
public static void main(final String [] args) {
System.out.println("-->" + euler1(1000));

Friday, June 13, 2008

Simple Code Snippet: simple http client in common lisp (oriented for clisp)

One of my early tasks for every language I have worked with has always been to write a http client. An simple http client will simply send a GET/POST request to a web server and interpret some of the results. Here is some common lisp code to do just that. Once again, this uses more of a "procedural" style of programming as opposed to a real "lispy" one. I suggest investigating drakma from Dr. Weitz; he has a much more robust implementation. [1]

And don't worry, I expect Rainer to set me down the right path.


;; Author: Berlin Brown <berlin dot brown at>
;; Date: 6/6/2008
;; File: httpforum.lisp
;; ---------------------------
;; Short Description:
;; ---------------------------
;; Simple HTTP client for communicating with web forums.
;; Environment: Tested with GNU CLISP 2.44 (2008-02-02) Win32
;; Full Description:
;; ---------------------------
;; Approaching for posting to the forum:
;; ---------------------------
;; Posting to a forum is going to happen in three HTTP requests.
;; 1. Request the main page - open this page to get the cookie/session
;; 2. Request the POST page
;; 3. Using the POST HTTP request to transmit the data to the server7.
;; ---------------------------
;; Example 200 response from server:
;; (first request)
;; --------------------------
;; HTTP/1.1 200 OK
;; Date: Sun, 08 Jun 2008 17:51:09 GMT
;; Server: Apache 3
;; X-Powered-By: PHP/4.4.3
;; Set-Cookie: SESSION=e7c1aefdb2c7137963a5ff94592f5f66; \
;; expires=Mon, 08 Jun 2009 17:51:09 GMT
;; Expires: Sat, 07 Jun 2008 17:51:09 +0000GMT
;; Last-Modified: Sun, 08 Jun 2008 17:51:09 +0000
;; Cache-Control: private
;; Connection: close
;; Transfer-Encoding: chunked
;; Content-Type: text/html
;; References:
;; [1]
;; [2]
;; [3]
;; [4]

(defparameter *DEFAULT_USER_AGENT_IE*
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1)")
(defparameter *DEFAULT_USER_AGENT_FF*
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/20070914 Firefox/")

(defparameter *postdata-hash* nil)
(defparameter *postdata*
'(("username" "myuser")
("password" "mypwd")))

(defclass http-headers ()
; To access slots:
; (setf (http-status headers) 200)"
; (http-status headers)
((status :accessor http-status :initarg status :initform nil)
(status-code :accessor http-status-code :initarg status-code :initform -1)
(date :accessor http-date :initarg date :initform nil)
(server :accessor http-server :initarg server)
(cookie :accessor http-cookie :initarg cookie :initform nil)
(cookie-data :accessor http-cookie-data :initform nil)
(session :accessor http-session :initform nil)
(expires :accessor http-expires :initarg expires :initform nil)
(modified :accessor http-modified :initarg modified :initform nil)
(connection :accessor http-connection :initarg connection :initform nil)
(content :accessor http-content :initarg content)))

(defun join (lst)
"Using reduce to join, note O(n^2)
(with-output-to-string (stream) (dolist (string strings)
(write-string string stream)))"

(reduce #'(lambda (x y) (concatenate 'string x y)) lst))

(defun list->hash (lst hashdata)
"Convert key value list data into a hash table"
(dolist (key-val lst)
(setf (gethash (first key-val) hashdata)
(second key-val)))

(defun init-postdata ()
(load "")
(setf *postdata-hash* (make-hash-table :test #'equal))
(list->hash *postdata* *postdata-hash*))

(defun print-hash-entry (key value)
(format t "key=~S | value=~S~%" key value))
(defun print-postdata->str (hash)
"Pretty print a hashmap of URL form POST data into
a list string data structure"

(with-hash-table-iterator (it hash)
(multiple-value-bind (entry-p key value) (it)
(if entry-p (print-hash-entry key value) (return))))))

(defun postdata->list (hash)
"Convert a hashmap of URL form POST data into a list string data structure"
(let ((lst nil))
(with-hash-table-iterator (it hash)
do (multiple-value-bind (entry-p key value) (it)
(if entry-p (setf
lst (cons (format nil "&~a=~a" key value) lst))
(return lst)))))))

(defun postdata->str (hash)
"Wrapper function, Convert a hashmap of URL form POST data into a string
for posting to the server"

(subseq (join (postdata->list *postdata-hash*)) 1))

(defmethod set-cookies ((headers http-headers))
" Create a hashtable data structure from the collection of cookies
expires=Mon, 08 Jun 2009 23:59:52 GMT"

(let* ((data-str (http-cookie headers))
(data-lst (split-by-one data-str #\;))
(tabl (make-hash-table :test #'equal)))
(dolist (key-val data-lst)
(let* ((kv (split-by-one key-val #\=))
(key (string-trim " " (first kv)))
(val (second kv)))
(setf (gethash key tabl) val)))
(setf (http-cookie-data headers) tabl)
(setf (http-session headers)
(gethash "SESSION" tabl))

(defun print-headers (headers)
"Pretty print the HTTP header information"
(format t "~%-----------------~%")
(format t " Status - ~a~%" (http-status headers))
(format t " Status Code - [~a]~%" (http-status-code headers))
(format t " Cookies - ~a~%" (http-cookie-data headers))
(format t " Session - ~a~%" (http-session headers))
(format t " Date - ~a~%" (http-date headers))
(format t " Server - [~a]~%" (http-server headers))
(format t " Expires - ~a~%" (http-expires headers))
(format t " Modified - ~a~%" (http-modified headers))
(format t " Connection - ~a~%" (http-connection headers))
(format t " Content - [~a]~%" (http-content headers))
(format t "-----------------~%") headers)

(defun join (lst)
"Using reduce to join, note O(n^2)
(with-output-to-string (stream) (dolist (string strings)
(write-string string stream)))"

(reduce #'(lambda (x y) (concatenate 'string x y)) lst))

(defun split-by-one (string delim)
"Returns a list of substrings of string
divided by ONE space each.
Note: Two consecutive spaces will be seen as
if there were an empty string between them."

(loop for i = 0 then (1+ j)
as j = (position delim string :start i)
collect (subseq string i j)
while j))

(defun get-headers (data)
"Loop till we get all the header data. We are using a
bad approach to detect when to quit. Look for a zero length string"

for line in (split-by-one data #\Newline)
when (= (length line) 0)
return lines
collect line into lines
finally lines))

(defun header-data (line)
"Convert a header string into a simple data structure.
(list 'Header:' start-of-text-int)"

(let* ((lst (split-by-one line #\Space))
(header (first lst))
(pos (length header)))
(list header (1+ pos))))

(defun http-parse-headers (data)
"Parse the HTTP headers and a return a HTTP header data object"
(let ((headers (make-instance 'http-headers)))
(dolist (header-str (get-headers data))
(let* ((h (first (header-data header-str)))
(pos (second (header-data header-str)))
(htxt (subseq header-str pos)))
(cond ((string-equal "HTTP/1.1" h)
(setf (http-status headers) htxt)
(setf (http-status-code headers)
(funcall (lambda (str)
(first (split-by-one str #\Space)))
((string-equal "Date:" h)
(setf (http-date headers) htxt))
((string-equal "Server:" h)
(setf (http-server headers) htxt))
((string-equal "Set-Cookie:" h)
(setf (http-cookie headers) htxt))
((string-equal "Expires:" h)
(setf (http-expires headers) htxt))
((string-equal "Last-Modified:" h)
(setf (http-modified headers) htxt))
((string-equal "Connection:" h)
(setf (http-connection headers) htxt))
((string-equal "Content-Type:" h)
(setf (http-content headers) htxt)))))

(defun http-client-request (socket host page
&key (method "GET") (postmap nil))
"Send a valid http 1.1 request to the server"
(let* ((p (string-equal "POST" method))
(post-str (postdata->str postmap))
(post-len (if (and post-str p)
(length post-str) 0))
(content-type (when p
"Content-Type: application/x-www-form-urlencoded~%")))
(format socket "~a ~a HTTP/1.1~%" method page)
(format socket "Host: ~a~%" host)
(format socket "User-Agent: ~a~%" *DEFAULT_USER_AGENT_FF*)
(format socket "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.81~%")
(format socket "Accept-Language: en-us,en;q=0.51~%")
(format socket "Accept-Encoding: gzip,deflate~%")
(format socket "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.71~%")
(format socket "Keep-Alive: 0~%")
(format socket "Connection: close~%")
(when p
(format socket "Content-Length: ~a~%" post-len)
(format socket content-type))
(format socket "Keep-Alive: 0~2%")
(when (and post-str p)
(format socket post-str))))

(defun http-connect (host page &optional (port 80)
&key (method "GET") (postmap nil))
"Use of common lisp keyword arguments [(defun i (&key x &key y) (list x y))]"
;; HTTP requires the :DOS line terminator
(with-open-stream (socket
(format t "~%###############~%")
(format t "INFO: Sending Request~%")
(format t "###############~%~%")
;; Print REQUEST data to file and to STDOUT
(http-client-request t host page
:method method :postmap postmap)
(ostream "request_data.log"
:direction :output
(http-client-request ostream host page
:method method :postmap postmap))
(format t "~%###############~%")
(http-client-request socket host page)
:for line = (read-line socket nil nil)
:while line
:collect (concatenate 'string line (string #\Newline)))))

(defun http-connect-data (host page &optional (port 80)
&key (method "GET") (postmap nil))
(join (http-connect host page port
:method method :postmap postmap)))

(defun http-main (host page &optional (port 80)
&key (method "GET") (postmap nil))
"Entry point for a HTTP get request, return the http header object"
(let* ((http-data (http-connect-data host page port
:method method :postmap postmap))
(headers (http-parse-headers http-data)))
(print http-data)
(print-headers (set-cookies headers))))

(defun main ()
"Main entry point for the application"
(format t "INFO: httpforum - connecting~%")
;;(print (http-main "localhost" "/test/index.jsp" 9080))
;; Post data from the file to localhost
;; post.php.
(print (http-main "localhost" "/fruit/post.php" 80
:method "POST" :postmap *postdata-hash*))
(format t "~%INFO: done~%"))

;; End of File

Saturday, June 7, 2008

Simple code snippet; with common lisp (sbcl) use html-template to generate html documents

As a compliment to my earlier post. Here is an example using html-template like any other server page like Active Server Pages or Java Server Pages. Pass variables to the template parser and the parser will output a full HTML document.

;; Berlin Brown
;; Simple prototype example; using commmon lisp and html-template,
;; Read an input html document template file and use the
;; template parser to generate a new html document with the
;; populated variables.
;; Filename: test_html_templ.lisp
;; Environment: (SBCL 1.0.14) common lisp with libraries.
;; References:
;; [1]

(require :html-template)

(defun parse-template ()
(let* ((rows
(loop for i below 49 by 7
collect (list :cols
(loop for j from i below (+ i 7)
for string = (format nil "~R" j)
collect (list :content string
(oddp j))))))
(values (list :rows rows)))
(with-open-file (stream "output.html"
:direction :output
:if-exists :supersede)
#p"./index_templ.html" values
:stream stream))))

(defun main ()
(format t "Running~%")
(format t "Done~%"))

;; End of the file.

Simple code snippet; with common lisp (sbcl) connect to database and query with CLSQL

;; Author: Berlin Brown
;; Short Description: Database Test - connect to the entity links database
;; and run a query
;; Environment: (SBCL 1.0.14) common lisp with libraries:
;; clsql-mysql, uffi
;; Description: The following snippet contains common lisp code
;; connect to a simple URL database called botlist_development and returns
;; two records from a table called entity_links.
;; References:
;; [1]

(require :clsql)
(require :clsql-mysql)

(setf clsql:*default-caching* nil)

;; Data structure to store our links
(clsql:def-view-class entity_links ()
:reader main_url
:initarg :main_url
:type string)
:reader url_title
:initarg :url_title
:type string)
;; Ensure that the references to the entity links table, uses
;; the correct tablename and case.
(setf (clsql:view-table (find-class 'entity_links)) '|entity_links|)

(defmacro with-db ((database) &body body)
;; Supply database connect str
(,database '("localhost" "botlist_development" "USER" "PASSWORD")
:database-type :mysql
:pool t
:if-exists nil)
(defun test-db ()
"Wrapper method for the test case; connect to the database
and run a simple query"

(with-db (db)
(setf clsql:*default-database* db)
(clsql:status t)
(let ((z (clsql:query
"select main_url from entity_links limit 0,2"
:database db))
(d (clsql:select 'entity_links
:database db
:flatp t)))
(print z)))
(format t "After with-db~%")
(clsql:status t))
(defun main ()
"Main entry point for the example"
(format t "Running - ~%")
(format t "Done - ~%"))

;; End of the File

Wednesday, June 4, 2008

Better error reporting with Websphere 404/500 error .jsp page

It is always a pain to detect J2EE errors. Here is a code snippet for a 404/500 type JSP error file to give you a better look at generic errors.

// <%@ page import=",*" %>
// <%
// Place in a 404/500 JSP error page file.
if (pageContext.getAttribute("ErrorReport", PageContext.REQUEST_SCOPE) != null) {
// Better error reporting in a J2EE/Websphere application.
ServletErrorReport errReport = (ServletErrorReport) pageContext.getAttribute("ErrorReport",

out.println("Error Code: " + errReport.getErrorCode());
if (errReport.getRootCause() != null) {
StringWriter sw = new StringWriter();
PrintWriter pw = new PrintWriter(sw);
if (errReport.getRootCause() instanceof ServletException) {
ServletException se = (ServletException) errReport.getRootCause();
if (se.getMessage() != null) {
} else {
} // End of If
if (se.getRootCause() != null) {
} else {
} // End of If
} else {
else {
// Populate the StringWriter
StringWriter sw = new StringWriter();
PrintWriter pw = new PrintWriter(sw);
StackTraceElement elem [] = errReport.getStackTrace();
for (int i = 0; i < errReport.getStackTrace().length; i++) {
out.println("<br />ERR: (rootcause==null) SE: /"
+ elem[i].getFileName() + " /" + elem[i].getClassName() + " /" + elem[i].getMethodName());
out.println("<br/>ERR: (rootcause==null)|" + errReport.getMessage());
out.println("<br/>ERR: (rootcause==null)|" + errReport.getStackTrace().toString());
// %>