Sunday, April 20, 2008

botlist-ghost network on Google's AppEngine and the botlist git repo

I was busy this weekend. I setup the git repo for botlist on Also, I am creating another django based front-end for botlist. Time permitting, I will discuss the changes in future posts.

New git repos: - botlist source, git mirror - glAnts Mech Game

Botlist/GhostNetwork on App Engine

Saturday, April 19, 2008

A java oriented approach for working with ABCL (common lisp on the jvm)

ABCL-PROMPT>>> (load "swing_example.lisp")

Use with ABCL (known as Armed Bear Common Lisp)
Author: Berlin Brown <berlin dot brown at>
Date: 4/15/2008

"The opposite of a correct statement is a false statement." -- Niels Bohr

*LICENSE* (new bsd license):

*Short Overview*
Swing Client View (an swing example implemented for ABCL lisp)

*Full Overview*
I always have trouble finding the right tool to create simple, throw-away
UIs. I created this code to launch a set of perl test scripts.
I needed basic menu-item/button action handlers and a text-area
to view and/or parse the incoming text that would have normally
been piped to standard output.

I ended up choosing common-lisp as opposed to python or
ruby because the lisp syntax is
just as light, and powerful a general purpose language;
it is perfectly suited for my task.
I commonly use emacs, so I am already equipped with an editor.
I decided to use Swing and ABCL for the
UI toolkit, and for no real particular reason. If I can
create this UI using swing running on the JVM,
it will be 100 times easier using another toolkit. Plus,
swing and the jvm is reliatively
portable. In the future, I hope to convert this to a more
portable MVC architecture.
This Swing widget creation code could operate as the view
for a larger system and then I could
easily switch with another lisp gui toolkit library.

*Setup and Environment*
ABCL runs like any other java application. Once, you compile
the single jar and resource
bundled *.cls lisp binary files, a script is available
that you use to launch the ABCL main class:


You need a working java runtime and JDK
(preferablly Sun's jre/jdk 1.5 or greater),
a common lisp implementation (e.g. CMUCL, SBCL, or CLISP)
and the ABCL package.
Download the latest ABCL from sourceforge [1]
Any of the common lisp implemenations will work.
In the past, I have compiled
ABCL on win32 cygwin with clisp and Ubuntu Linux with SBCL.
I didn't experience any issues. You may have some
trouble configuring the customization
file to point to your JDK. I had to modify the script
and just removed the operating
system detect code.

I added this to the buttom of the ABCL customizations.lisp file
to override any pre-existing definitions:
(setq *jdk* "C:\\Program Files\\Java\\jdk1.5.0_11\\")
(setq *java-compiler* "javac")

Follow the steps in the README file:
Start your common lisp implemenation and get to the lisp REPL prompt:

Load build-abcl.lisp:

(load "build-abcl.lisp")

Then do:

(build-abcl:build-abcl :clean t :full t)

Launch the ABCL startup script:
$$$ abcl.bat

Armed Bear Common Lisp 0.0.10 (built Tue Apr 8 2008 12:43:18 -0500)
Java 1.5.0_11 Sun Microsystems Inc.
Java HotSpot(TM) Client VM
Low-level initialization completed in 0.656 seconds.
Startup completed in 2.171 seconds.
Type ":help" for a list of available commands.

CL-USER(1): (load "swing_example.lisp")

*Source Code*
This code can be a little hard to follow. There is a lot of typing needed
to setup and instantiate the java classes or access the java fields.
So, I suggest you use lisp syntactic sugar to
beautify and cleanup the example
and remember to send me your patches.

*Useful tests from the ABCL java test suite*
[a] Example to create an instance of a java object
with the String constructor
(deftest jnew.1
(let ((constructor (jconstructor "java.lang.String" "java.lang.String")))
(jclass-of (jnew constructor "foo")))

[b] Accessing the 'int' java primitive
(deftest jclass-name.8
(jclass-name (jclass "int"))

[c] Invoking a java method.
(deftest jclass.4
(let ((class1 (jcall (jmethod "java.lang.Object" "getClass") "foo"))
(class2 (jclass "java.lang.String")))
(jcall (jmethod "java.lang.Object" "equals" "java.lang.Object")
class1 class2))

(deftest jclass.3
(equal (jclass '|java.lang.String|) (jclass "java.lang.String"))

Example static method call "System.getenv('KEY')
(jstatic "getenv" (jclass "java.lang.System") "KEY")

Java Static Fields:
(jfield-raw "java.lang.Boolean" "TRUE")

*Synonymous java code (see the lisp function initTextAreaLayout)*:
I created a 100% java skeleton version to see how to
transform that code to lisp. Here is
some of the java source.

import java.awt.BorderLayout;
import java.awt.event.ActionEvent;
import javax.swing.AbstractAction;
import javax.swing.BoxLayout;
import javax.swing.JButton;
import javax.swing.JFrame;
import javax.swing.JMenu;
import javax.swing.JMenuBar;
import javax.swing.JMenuItem;
import javax.swing.JPanel;
import javax.swing.JScrollPane;
import javax.swing.JTextArea;
import javax.swing.JTextField;
import javax.swing.ScrollPaneConstants;
import javax.swing.UIManager;

private void initTextAreaLayout() {
pathField = new JTextField(DEFAULT_PATH, 40);
final JPanel topPanel = new JPanel();
topPanel.setLayout(new BoxLayout(topPanel, BoxLayout.Y_AXIS));

final JPanel buttonPanel = new JPanel();
this.readRequestsButton = new JButton("Run Action");


contentArea = new JTextArea(25, 60);
JScrollPane scrollPane = new JScrollPane(contentArea,

getContentPane().setLayout(new BorderLayout());
getContentPane().add(topPanel, BorderLayout.NORTH);
getContentPane().add(scrollPane, BorderLayout.CENTER);

*Tested with JDK/JRE version*:
Java version "1.5.0_11"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_11-b03)
Java HotSpot(TM) Client VM (build 1.5.0_11-b03, mixed mode)

[2] Peter Graves - creator of ABCL

** ABCL Lisp Helpers **
Example static method call "System.getenv('KEY')
(jstatic "getenv" (jclass "java.lang.System") "KEY")

(jfield-raw "java.lang.Boolean" "TRUE")

** Useful Java Swing Constructors **
Also see: [3]

JTextField() / JTextField(String text, int columns)
/ JScrollPane(Component view) /
JScrollPane(Component view, int vsbPolicy, int hsbPolicy)

[4] ABCL is not forgiving with passing the wrong type to a java method. Make
sure the right type is passed to the right argument.
ABCL returns generic errors:

Debugger invoked on condition of type JAVA-EXCEPTION:

[5] I defined some constants and parameters at the top of the code to reduce
some redundant code.

(defconstant j-string "java.lang.String")

(defparameter *new-jmenu-item* (jconstructor j-jmenuitem j-string))



(defpackage :swing-view
(:use :common-lisp :java))
(in-package :swing-view)

;; ** Java Class String Constant Definitions **
(defconstant *default-path* "file://./latin_text.txt")
(defconstant j-string "java.lang.String")
(defconstant j-component "java.awt.Component")
(defconstant j-container "java.awt.Container")
(defconstant j-layout-manager "java.awt.LayoutManager")
(defconstant j-borderlayout "java.awt.BorderLayout")
(defconstant j-actionevent "java.awt.event.ActionEvent")
(defconstant j-abstractaction "javax.swing.AbstractAction")
(defconstant j-boxlayout "javax.swing.BoxLayout")
(defconstant j-jbutton "javax.swing.JButton")
(defconstant j-jframe "javax.swing.JFrame")
(defconstant j-jmenu "javax.swing.JMenu")
(defconstant j-jmenubar "javax.swing.JMenuBar")
(defconstant j-jmenuitem "javax.swing.JMenuItem")
(defconstant j-jpanel "javax.swing.JPanel")
(defconstant j-jscrollpane "javax.swing.JScrollPane")
(defconstant j-jtextarea "javax.swing.JTextArea")
(defconstant j-jtextfield "javax.swing.JTextField")
(defconstant j-scrollpaneconstants "javax.swing.ScrollPaneConstants")
(defconstant j-uimanager "javax.swing.UIManager")

(defparameter *scroll-h-always* (jfield-raw j-scrollpaneconstants
(defparameter *scroll-v-always* (jfield-raw j-scrollpaneconstants

(defparameter *bl-north* (jfield j-borderlayout "NORTH")
"Definition for swing constants")
(defparameter *bl-east* (jfield j-borderlayout "EAST")
"Definition for swing constants")
(defparameter *bl-center* (jfield j-borderlayout "CENTER")
"Definition for swing constants")
(defparameter *bl-west* (jfield j-borderlayout "WEST")
"Definition for swing constants")
(defparameter *bl-south* (jfield j-borderlayout "SOUTH")
"Definition for swing constants")
(defparameter *box-y-axis* (jfield j-boxlayout "Y_AXIS")
"Definition for swing constants")
(defparameter *box-x-axis* (jfield j-boxlayout "X_AXIS")
"Definition for swing constants")

;; ** Java method definitions **
(defparameter *method-set-layout*
(jmethod j-container "setLayout" j-layout-manager))
(defparameter *method-jmenu-add* (jmethod j-jmenu "add" j-jmenuitem))
(defparameter *method-jpanel-add* (jmethod j-jpanel "add" j-component))
(defparameter *method-container-add* (jmethod j-container "add" j-component))
(defparameter *method-container-add-2* (jmethod j-container "add"
j-component "java.lang.Object"))

;; Java constructor definitions
(defparameter *new-jmenu-item* (jconstructor j-jmenuitem j-string))
(defparameter *new-scroll-pane*
(jconstructor j-jscrollpane j-component "int" "int"))
(defparameter *new-scroll-pane-2* (jconstructor j-jscrollpane "int" "int"))
(defparameter *new-textarea* (jconstructor j-jtextarea "int" "int"))

;; Function implementations
(defun j-true () (make-immediate-object t :boolean))
(defun j-false () (make-immediate-object nil :boolean))

(defun to-java-string (s)
(jnew (jconstructor "java.lang.String" "java.lang.String") s))

(defun init-swing (jframe)
"Define the frame properties. Add the button panel and menu"
(progn (setNativeLookUI)
(initTextAreaLayout (get-content-pane jframe))))

(defun get-content-pane (jframe)
"Translated to natural language: Using the instance of
the JFrame object called jframe, invoke the getContentPane method
and return an instance of the Container class"

(jcall (jmethod j-jframe "getContentPane") jframe))

(defun new-borderlayout ()
"Create an instance of the border layout class"
(jnew (jconstructor j-borderlayout)))

(defun new-box-layout (panel axis)
(jnew (jconstructor j-boxlayout j-container "int") panel axis))

(defun path-textfield ()
(jnew (jconstructor j-jtextfield j-string "int")
*default-path* 40))

(defun initTextAreaLayout (content-pane)
"Attach the content text area widget and other components
@content-pane (java.awt.Container) - Container we are adding widgets
@see java.awt.Container"

(let* ((text-field (path-textfield))
(contentArea (jnew *new-textarea* 25 60 ))
(topPanel (jnew (jconstructor j-jpanel)))
(buttonPanel (jnew (jconstructor j-jpanel)))
(scrollPane (jnew *new-scroll-pane* contentArea
*scroll-v-always* *scroll-h-always*)))
(format t "INFO: content-pane obj:= [ ~a | ~a ]~%" content-pane topPanel)
(jcall *method-set-layout* topPanel
(new-box-layout topPanel *box-y-axis*))
;; Add TO the top panel; in java this will look
;; like: topPanel.add(pathField)
(jcall *method-jpanel-add* topPanel text-field)
(jcall *method-set-layout* content-pane (new-borderlayout))
(jcall *method-container-add-2* content-pane topPanel *bl-north*)
(jcall *method-container-add-2* content-pane scrollPane *bl-center*)))

(defun setNativeLookUI ()
"Use the operating system native UI look and feel, do not use the Swing oriented look"
(jstatic "setLookAndFeel"
(jclass j-uimanager)
(defun createReaderFrame ()
" Create the simple reader frame "
(let ((simpleFrame (jnew (jconstructor j-jframe))))
(init-swing simpleFrame)
(jcall (jmethod j-jframe "pack") simpleFrame)
(jcall (jmethod j-jframe "setVisible" "boolean")
simpleFrame (j-true))))

(defun lisp-main ()
"Main entry point, create the jframe and attach the widget components then
start the jframe thread"

(format t "INFO: creating panel objects~%")


;; End of Script

Wednesday, April 16, 2008

Botlist has competition; topix

"The answer seemed simple in 2004: aggregate news from thousands of sources, create thousands of topically driven news web pages and populate each of those pages with only news about that particular topic.

The Topix team did just that. We created artificial intelligence algorithms that continuously monitor breaking news from over 50,000 sources, 24 hours a day.

These algorithms read every story as it is released and then categorize each one into one or more of the 360,000 news pages, including pages for all 32,500 US zip codes, 5,500 public company and industry verticals, 48,000 celebrities and musicians, 1,500 sports teams and personalities, and many, many more. Pretty soon, millions of people were visiting the site every month.

But visitors wanted more: they wanted to tell us what they thought about the news on the site. They jammed our email and voice mail with comments about the articles and stories that mattered to them. In response, we added forums to the site in December 2005.
And then it got really interesting."

-- This is about the topix application.

Botlist is similar, but clearly doesn't have 360,000 news pages.

Sunday, April 13, 2008

FAQ: What is the botlist application?

FAQ: What is the botlist web application?

Botlist, aka the-ghost-network(ghostnet for short) is about web content. This application is designed to find and extract relevant content from the web. And the data will be accompanied by intelligent agents to mine that data.

The idea is based loosely on a lot of the aggregation web 2.0 applications there are out there. I really liked the idea of That started botlist in 2003-2004. Then reddit came along and really motivated me to start redesigning botlist.

However, botlist is different. These social-networking news applications are driven by users. Botlist is driven by aggregate bots. The bots (aka intelligent agents/crawlers) are spawned to crawl useful information on the web and aggregate a public database and web application.

Also, botlist is a collection of different programs. I am working on a specification for the web front-end. Basically the web front-end is synonymous with the "reddit/digg" clone. As new data comes in, data is viewable on the botlist front-end. I am also working on other front-ends in different frameworks, including a rails/django front-end. The second aspect of botlist is the crawler/backend bot agents. The third aspect is archiving the content.

FAQ: What is the purpose of botlist? Why couldn't I have just used Yahoo or Google? Or Reddit?

Or, you could use any of the other public repositories. Botlist is different in two ways. First, all source and all data will be under some form of liberal free/open/cc license. The source code for all projects will be openly available, as well as all data crawled and mined by botlist. Secondly, botlist is just different. There is a lot of content on the web. Some analysts found that there are 170 million websites on the net and billions and billions of web-pages. That is a lot of content. It doesn't make sense for us to only use 2-3 portals to look at that information. Botlist is just another tool used to look at the web.

Botlist and EC2 and S3

Botlist is officially running off of EC2 and S3. This is the initial test, I have some more things to clear out.

Basically, it means that the botlist content will be archived on S3.

More details in future posts.

Software Developer Oscar Movie Review: No Country vs There will be Blood

This is easy.

There will be blood is one the best films in a decade. There will be blood is 10 times better than No Country for Old Men.

Why? Right off the bat, you are given way more to work with. There will be blood opens a lot of questions but doesn't drown you with pedestrian answers.

No country is just a classical shoot-em-up with a very cunning bad character. After about 30 minutes, you can already guess the plot, outcome and everything else.

The oscars got it wrong, as usual.

Friday, April 11, 2008

Ubuntu, screen resolution and graphic cards, is still kind of complicated

I have spent most of my life in front of computers. 20 years later, Things on Ubuntu still scare me. I can only imagine what it must be like for a complete novice computer user. That would be a sadistic joke, put an Ubuntu machine in front of them.

Anyway, I had some issues with screen resolution again. This time I just wanted to increase my dual monitor resolution from 1280x800 to 1600x1024. Somehow, between nvidia and xorg; it won't let me keep my settings. Not just that, one monitor is fine...the other is not so fine. The only difference between the dual monitors. I am using a Nvidia 5300 on one and a 5200 on another. Same monitors.


Here is my post on Ubuntu. Maybe, in 5-6 years this will get fixed.

hey should have Mensa/GRE/College entrance exams and have a question, "How do you set your screen resolution on Ubuntu"

Anyway, I am having trouble setting my screen resolution and I had it working fine, now I going nuts trying to increase it.

Dual Monitor My setup:
OS: Ubuntu 7.10
Cards: Nvidia 5300 and 5200.
Monitors: Acer 20inches max res at 1680x1050.

What I had orginally (this worked fine for several months)
Resolution in xorg.conf: 1280x800.

This resolution worked fine but then I decided to increase my resolution to the preferred LCD settings, closer to 1680x1050.

I open nvidia-settings.

sudo nvidia-settings.

It has the resolutions:

1600x1024. I click on both of those monitors and set the resolutions. Everything works fine. I say overwrite my xorg.conf. So far, everything looks good.

But, when I logout or reboot, it doesn't keep the setting? It sets one monitor to 1280x800 and the other to 1600x1024? Where would it keep that setting?

A bug?


Section "Device"
Identifier "Videocard0"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BoardName "GeForce PCX 5300"
BusID "PCI:2:0:0"

Section "Device"
Identifier "Videocard1"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BoardName "GeForce FX 5200"
BusID "PCI:5:9:0"

Section "Screen"
Identifier "Screen0"
Device "Videocard0"
Monitor "Monitor0"
DefaultDepth 24
Option "TwinView" "0"
Option "metamodes" "1600x1024 +0+0"
SubSection "Display"
Depth 24

Section "Screen"
Identifier "Screen1"
Device "Videocard1"
Monitor "Monitor1"
DefaultDepth 24
Option "TwinView" "0"
Option "metamodes" "1600x1024 +0+0"
SubSection "Display"
Depth 24


Edited: Ok, I think I figured out my screen resolution issues. Apparently there are 15,000 ways to configure the resolution: 1. nvidia-settings. 2. xorg.conf. 3. gnome resolution screen settings 4. the graphics and screen settings? wtf?

But I still don't know which one is the default. Also, which one setting has priority.

Thursday, April 10, 2008

Regex fun

Here is a session of testing regex. The goal is to find how to detect a java import statement at the beginning of a line.

For example;

Detect this line:

"import java.blah.Com;"

>>> print re.match("^(import).$", a)
>>> print re.match("^import.$", a)
>>> print re.match("^import$", a)
>>> a
'import lkjksdjlfksjldkfjsd '
>>> print re.match("^[import]$", a)
>>> print re.match("^[import]$", a)
>>> a = "import"
>>> print re.match("^[import]$", a)
>>> print re.match("[import]", a)
<_sre.SRE_Match object at 0x7ff35790>
>>> a = "import" ; print a
>>> a = "import" ; print re.match("[import]", a)
<_sre.SRE_Match object at 0x7ff35790>
>>> a = "import " ; print re.match("[import]", a)
<_sre.SRE_Match object at 0x7ff35678>
>>> a = "import " ; print re.match("^[import]", a)
<_sre.SRE_Match object at 0x7ff35758>
>>> a = " import " ; print re.match("^[import]", a)
>>> a = " import " ; print re.match("^[import]$", a)
>>> a = "import " ; print re.match("^[import]$", a)
>>> a = "import" ; print re.match("^[import]$", a)
>>> a = "import" ; print re.match("^[import].", a)
<_sre.SRE_Match object at 0x7ff35678>
>>> a = "import " ; print re.match("^[import].", a)
<_sre.SRE_Match object at 0x7ff35758>
>>> a = "import " ; print re.match("^[import].", a).end()
>>> a = "import " ; print re.match("^[import].", a)
<_sre.SRE_Match object at 0x7ff35790>
>>> a = "import " ; print re.match("^[import].", a)
<_sre.SRE_Match object at 0x7ff35678>
>>> a = "import " ; print re.match("^[import]", a)
<_sre.SRE_Match object at 0x7ff35758>
>>> a = "import " ; print re.match("^[import].\Z", a)
>>> a = "import " ; print re.match("^[import].\$", a)
>>> a = "import " ; print re.match("^[import].\Z", a)
>>> a = "import" ; print re.match("^[import].\Z", a)
>>> a = "import" ; print re.match("[import].\Z", a)
>>> a = "import " ; print re.match("[import].\Z", a)
>>> a = "import " ; print re.match("^[import].*\Z", a)
<_sre.SRE_Match object at 0x7ff35790>
>>> a = "import " ; print re.match("^[import].*$", a)
<_sre.SRE_Match object at 0x7ff35678>
>>> a = "import" ; print re.match("^[import].*$", a)
<_sre.SRE_Match object at 0x7ff35758>
>>> a = "impor" ; print re.match("^[import].*$", a)
<_sre.SRE_Match object at 0x7ff35790>
>>> a = "import " ; print re.match("^[import].*$", a)
<_sre.SRE_Match object at 0x7ff35678>
>>> a = "ort " ; print re.match("^[import].*$", a)
<_sre.SRE_Match object at 0x7ff35758>
>>> a = "ort " ; print re.match("^import.*$", a)
>>> a = "import " ; print re.match("^import.*$", a)
<_sre.SRE_Match object at 0x7ff35678>
>>> a = "ximport " ; print re.match("^import.*$", a)
>>> a = "import " ; print re.match("^import.*$", a)
<_sre.SRE_Match object at 0x7ff35790>
>>> a = "import " ; print re.match("^import.*\;$", a)
>>> a = "import ;" ; print re.match("^import.*\;$", a)
<_sre.SRE_Match object at 0x7ff35758>
>>> a = "import " ; print re.match("^import.*\;$", a)

Saturday, April 5, 2008

Botlist metrics with stat svn

Here is an interesting look at the botlist project. You can see everything, lines of code, most active time of development, and many different metrics:

Friday, April 4, 2008

Thesis notes; adding factor article from Chris Double

"Factor Articles - Work in Progress Chris Double April 4, 2008"

I will be adding notes from Chris Double's recent article on factor. Here is a collection of his work.

Eclipse and why I continue to use it

I use Eclipse. I don't deny it. I also use Emacs and VIM. But there are some features in Eclipse that I like and as far as I know, it is not a crime to use the IDE. Here is yet another forum discussion where I describe why Eclipse is not Java.

First, I think these comments from two posters clearly define why Eclipse doesn't do some tasks as you might expect.

"Eclipse doesn't make assumptions about which JRE library you're building against. That information comes from the Build Path information in a Java Project, and while that may be a hassle for the "Hello, World" kicking-the-tires scenario, assumptions can wreck the reproducibility of problems you run across as you scale to more complex scenarios. Past the simple cases, you want this amount of control over what you're building against"

"Eclipse uses incremental builder to build the project. It doesn't matter you have one file or hundereds of files. It just compiles the modified files.And if you have one and only one file,and if it has dependencies on lots of jar files,it is still easier to manage this with eclipse, rather than setting the classpaths and use javac from command prompt." -- biju thomas

Eclipse operates (in some respects) like the squeak smalltalk environment where your code is connected not just to the syntax-highlighting editor but to the other tools that Eclipse has. Eclipse has a hierarchy tool that maps out all the subclass and super-class implementations. If I subclass the Map class, Eclipse can connect to the source code "" and I hit F3 to navigate to the constructor source code if I need to look up some of the implementation. I can go all the way up the class hierarchy. I can check other classes in the package and see the other inter-dependencies. If it is a junit test case, I can launch the junit GUI to check if all my tests passed or not. I can hover a method and see the super-classes's javadoc comment.

All of that goes away if you are just treating a java class as a simple text file.

IBM's VisualAge was written in Smalltalk. VisualAge turned into IBM's Eclipse.

I said that Eclipse was not designed around a single Java source file. The Eclipse development "environment" designers over the last couple of years and many milestones decided to couple the java dependencies like the java system classpath and user classpath and other project dependencies in terms of a project. For example, you may compile an Eclipse project with Sun's JDK or you may decide to compile your project using IBM's JDK. And, for whatever reason they decided that it is better to combine the ENTIRE java system runtime environment (rt.jar) and user classpath (developer libraries like mylibrary.jar) together as a whole.

For example, when you are using the text editor and syntax highlighting. It is probably better to know if you are using Java 1.6 version of a String or Java 1.4 version the String class. If you compile your code without knowing that information, then the text editor may highlight the syntax differently.

Thursday, April 3, 2008

Describing HTTP requests

I was having a chat on an internet forum. Basically, a person was wondering how HTTP connections work, also, how do they apply to HTTPs and proxy server requests. Here is my take, written in forum casual dialog.


I think I understand what you are trying to do. But I don't understand why you need the proxy software. I will ignore that and concentrate on your goal.

Your goal is to successfully log into a HTTPS site. First, to the developer on the client side; there really isn't that much difference between HTTP and HTTPS. The data is encrypted during transmission and protects against people trying to eavesdrop on the transmission between A and B. Basically, you should see data (HTML string data) requests between client and server. The client does have to make sure that it loads the SSL libraries properly for the HTTPS request to work.

I do this a lot with java. I have to do this to initialize the SSL aspect of the http request;

Part A: sf = (; sock = null;
X509TrustManager tm = new MyX509TrustManager();
HostnameVerifier hm = new MyHostnameVerifier();

And then I do this:
Part B:
URL url = getSSLURL(fullURL);
conn = (HttpsURLConnection) url.openConnection();
My point, Part A is kind of complicated, that is required to initialize SSL for a HTTPS request. You would have to do something similar on the .NET side. I bet it is easier.

(Example 1, assuming no proxy)
Assuming you don't have the proxy in between the client and the server.

Stateless request from Client

Client -> Data -> Server
Server -> HTML Response -> Client

1. Client sends request to https://site (GET / HTTP ... blah)
2. Server sends the HTML data back to the client.

Once again, it shouldn't matter that this is HTTPS. If you don't see data, you didn't load the SSL client libraries properly. Or you aren't fully reading the data from the server.


The proxy.

A proxy is a funny sounding word for middle-man. Talk to the middle-man only to get to the server. With the proxy, it isn't that much different. First, you have to assume that the proxy is working properly. Our office web-proxy server is reliable and works properly. A cheap proxy server may give funky results. Your proxy may not even support HTTPS. I would check that first.

Once again, on the client, there really isn't that much difference. But, there are two situations with the proxies. One is where the proxy is configurable and the other is where the proxy is supposed to act transparently, just like you would any other request.

Scenario 2 is simple. Lets cover that one first. Proxy acts like an office, firewall proxy that you might find at a big office.

In this scenario, you are making a socket connection to the (HTTP/HTTPS works on a socket connection) the proxy server. But you are putting the FULL URL in your GET request.

Example request.

Client socket to myproxy:9999

Adding absolute path to the GET request


That is a pseudo example, just make sure that all requests use the "" as opposed to the relative path.

Also, in java; I can use a library to build the HTTP request content for me including constructing that header. So, ideally I don't have to build that request for working with a proxy server. I use the java settings to enable proxying.

E.g. This sets the proxy and any client requests are taken care of. .NET make work the same.

System.getProperties().put("proxySet", "" + getTestClient().isEnableProxy()); System.getProperties().put("proxyHost", getTestClient().getProxyHost());
System.getProperties().put("proxyPort", getTestClient().getProxyPort());

Also, you still have to probably make a secure socket connection to the proxy server.


Scenario 1. where the proxy is configurable.

This is like Scenario 2. Except, you may not have to make a HTTPS request to the proxy server and you can make a HTTP request. If the proxy server is configurable, sometimes it makes it easier on the developer.

Back to your original question.

To know that you connected to your site. It is simple, send POST data to the HTTPS site and you should get a configuration page.

I don't want to bring up botlist, but I have an automated test framework where I automatically register a user and automatically login and the user clicks on various links.


You may run into all kinds of issues with your custom client.

1. Redirects. Make sure you have redirects enabled. the HTTPS may ask for a redirect from the client (that is you).
2. Unsupported clients (not Firefox or Internet Explorer). Change the user-agent on your client to resemble IE or Firefox. Some servers monitor check your user-agent to make sure you aren't a bot.
3. Proper HTTP 1.1 headers. If you don't build a correct header, the server may not like what you and nullify the request.

Also, what is the goal of the proxy server? Modify the client requests? Are you running the proxy server on a remote server somewhere to mask where the client requests are coming from?

"I managed to post a modified string of HTML to a server. I set the requestURI, which I believe is the URL I want to be redirected to after my successful post to the original URL, and I got some piece of javascript back within an HTML tag."

I don't understand what you are saying here. I will paraphrase.

You want to make a POST request to a server and send data that server. I don't understand why you want to send HTML data to a server? Typically, you would send just "data" to the server. For example, I bet CoT is using a POST request and the data may contain "Full Name", "Email", "Home Page" and the "Message".

A post request to CoT might look like (this is pseudo code, I am going on memory).

User-Agent: Firefox
Set-Cookie: "remember-me"

---- The data
full_name=Bot Berlin&

And from that post page, the server will send back the HTML confirmation page data for the client to read.

"Thanks, data saved"

It could also send a redirect. The client has to handle either or.

On the proxy server stuff, think like this.

Your client is making a "socket" connection to the proxy server and building a HTTP request that will be used against the target web server. That is all you need to worry about (assuming the proxy server isn't crap).

Client: 1. Hello proxy web server at host: my-office-proxy, port 9999.
Client: 2. I want to send this HTTP POST data to login into my gmail account. Can you help me out? By the way, it is a HTTPs request. Here is the top header:


Proxy-Server: 1. I just got a client request, what does he want? Oh, he wants me to send a request to gmail.
Proxy-Server connects to gmail with the request from the client.

Proxy-Server to Gmail: Hey gmail, I have this request, do you accept?

Gmail to Proxy-Server: Sir, here is the confirmation page for the client.

Proxy-Server to Client: Gmail just sent me the confirmation page data.

Have a nice day.