Couldn’t quite find the perfect java-based web stack

As I mentioned in this article, I wanted to find a nice lightweight java back-end that would:

  • Allow me to write a REST api on the java-side using as much of POJO as possible.
  • Be able to serve static HTML, CSS and JS files easily.
  • Be able to combine individual HTML, CSS and JS into a single file and serve them using unique MD5 file names so caching wouldn’t be an issue (this I didn’t mention in the earlier article).

At first I thought Apache Tapestry would do the job but now after deploying two apps I’ve decided against it. It mucks around too much with the front-end. You HAVE to serve static content under a specific folder, and you have no control over that. Yuck. It also forces you to put a bunch of their javascript on your html like Prototype.js. No, I don’t want that.

In the end I just built a framework using Jetty and Servlet 3.0. Of course, I made it do exactly what I want, but I feel my requirements are pretty common for today’s apps. I hope I can open-source it and put it on GitHub (and put it on the Maven repository as well). But that would require a bit of work so we’ll see!

Using Tropo to send SMS messages

(Special thanks to Mark Headd, developer evangelist from Tropo, for helping me out).

We’ve been using Twilio for a while now to make phone calls and send text messages. However, they removed their international SMS support, and I started looking around for another SMS provider.

So naturally I tried out Tropo, since I come across them once in a while. I did manage to get SMSes out, though there are some traps. Their API is also slightly confusing, but it works in the end.

Here are the steps, and some things to look out for, if you want to just get an SMS out.

Create a Tropo Scripting application. Associate it with a hosted javascript file that looks like this:

message(theMessage, {to:numberToDial, network:'sms'})
hangup()

I accidentally copied a Groovy example from the Tropo blog rather than a javascript example, and that totally threw me off. In that example, the message parameter is called “message”, which of course interferes with the message() function. And the associative array is specified using square brackets. Yes, I know, not javascript, but when you’re in “I want to copy the example and make this work” the brain turned off a little 🙂

Add a phone number to that Tropo application.

You can now send a message with a web browser:

http://api.tropo.com/1.0/sessions?action=create&token=<YOUR TOKEN>&theMessage=Hello&numberToDial=+1<YOUR NUMBER>

Note: The phone number must have NO spaces! It doesn’t work with spaces. Weird.

The java library also works (again, no spaces!):

String token = <YOUR TOKEN>;

Tropo tropo = new Tropo();
Map&lt;String, String&gt; params = new HashMap();

//params.put("numberToDial", "&lt;YOUR NUMBER NO SPACES&gt;");
params.put("theMessage", "Test from Tropo, Java. v5");
TropoLaunchResult result = tropo.launchSession(token, params);

Hookbox testing going well

So I’ve put Hookbox through more testing. I’m still using Hosted Hookbox, and that really helps me focus on testing prior to deployment.

Here are some positive findings:

  • The best thing about using Hosted Hookbox during its early days is that I’m always testing with the latest versions.
  • It fully supports publishing internationalized characters. I tested with chinese and european characters.
  • It does a good job of escaping special characters. I sent a fairly complicated XML through, and received everything perfectly.
  • Integrating with Actionscript was a breeze, using watched variables.

On the negative side, I did experience one server hang. That was after publishing some really large xml numerous times. I had to restart the hosted hookbox server and all was good. HOWEVER, the clients don’t auto-connect, which will be a big concern moving forward.

Since testing is going well, we’re going to go live soon (within a month). Which makes me nervous since Hookbox is not yet version 1.0. It’s still at v 0.3.4dev.

Hookbox is THE fastest way to add comet functionality in an existing webapp

I have a web application that runs Flash in the browser client, and a JBoss 4.0.4 J2EE server on the server side. I don’t know a lot about comet technologies, but after poring through reading material over 1 day, it sounded complicated.

Thankfully I discovered Hookbox while reading CometDaily.

It’s still an early beta, but in my initial tests, I found it to be the perfect solution that I was looking for. You install hookbox on a server, and it’s a self-contained server that has provides basic comet (i.e. ajax push) functionality. It comes with a javascript library which you install on the client-side, and it allows your web server to communicate with the hookbox server via a RESTy api.

I managed to get a full-loop test going in about an hour, especially since there is a hosted version of hookbox that you can sign up instantaneously for development testing. To see server push happen so effortlessly in my existing web app, it’s beautiful!

The application which I’m going to build around hookbox will be fairly complex: It is a K-12 school scheduling software. Why I need comet is that it will be a multi-user interactive scheduling tool, i.e. many users can log on at the same time, and schedule classes, and everyone sees updates as they happen. So it will be interesting to see how the beta-release of hookbox handles this.

I’ll write more as I discover things. But for now, two quick notes:

  • The document says that javascript can connect to hookbox via a call like hookbox.connect(“http://<hookboxserverandport>/csp&#8221;); Actually, the /csp at the end has to be dropped. So mine is: hookbox.connect(“http://aris.hosted.hookbox.org&#8221;);
  • I couldn’t get the subscribe.publish() to work. The publish method doesn’t exist on the subscribe object. But I don’t need this right now, since only my webserver publishes events.

Update after 1 week of testing is in my next post.

How we used the PipelineDeals.com API to integrate sales information

As a new (and already satisfied) user of PipelineDeals.com, I ventured into using its API soon enough. QuickSchools.com, the SaaS school management system that we’re selling, generates sales leads from quite a few channels, and I want all our sales lead contacts to be tracked on PipelineDeals.com.  Here are some of those different channels:

  • They can fill out a “contact us” form.
  • They can sign up for a demo account.
  • They can start a 30-day free trial.
  • They can chat with us via live chat feature.
  • They can call us.
  • They can email us.

Having a holistic view of those leads would certainly help us sell better. I don’t want to lose a sale simply because we didn’t have our processes in order. I also want to see how we perform across those various channels so that we can know what we’re doing right and what we’re doing wrong.

For some of these leads, I want the data to be automatically inserted into our PipelineDeals.com system. PipelineDeals.com offers 2 integration options:

  • You can import leads using a Web-2-leads feature. Simply create a web form, and when it is submitted to PipelineDeals.com, it will create a lead. However, I wanted to send the data from our Java application, so this didn’t work.
  • You can import leads (and much more) using a REST API. Now, if you’ve used a REST API before, you’ll know it’s pretty simple. And this was no exception. Within a couple of hours, I had the data from our Java system sending leads information to our PipelineDeals.com database. In fact, it took so long because I spent some time creating an effective framework within our application for calling such REST APIs.

But unfortunately, the PipelineDeals.com API is not comprehensive. It misses certain key abilities. For example, I can’t seem to change the tags for the leads. And it’s hard to make my voice heard that I want such things, because most of the votes on the PipelineDeals.com UserVoice forum is for user features, not for API enhancements. Oh well. In any case, it’s just about workable for my purposes.

Speeding up Internet access to US sites on Streamyx

My company operates QuickSchools.com, and with it we’re trying hard to break into the US market. Our servers are located in Seattle. But accessing and managing those servers from Malaysia is an absolute nightmare, due to the dismal performance of Streamyx broadband when we try to access sites in the US. Go ahead, try to access our QuickSchools.com website. If you’re on Streamyx, you’ll see how frustratingly slow it is. (Thankfully, the site is blazingly fast when accessed from the US.)

My company would not mind paying for a higher grade of Streamyx, but I have no confidence this would solve our problem. The issue seems to be with the international links that Streamyx uses, and not my last mile connection.

Happily, we’ve figured out a working solution which allows us to connect to US sites with a more consistent, and faster connection. I’m seeing download speeds increase by 10 times. How about that eh!

I developed the solution based on a hunch. The local streamyx network itself is not a major problem. Connection to other ISPs in Malaysia and to Singapore is just fine. However, I guessed that Singapore’s link to the US would be much faster.

So I rented a virtual private server (VPS) in Singapore (gplhost.com). It took only minutes to sign-up online, and Debian was installed within 15 minutes. So, from the Singapore server, I pinged our US servers. True enough, I got good response times! Ping took about 200 ms. Compare that with 400 ms using my home or office Streamyx connection.

Next, I set up an OpenVPN server on the Singapore server, and configured it so that it would route all client traffic through the VPN. It’s pretty much the default configuration, with these additional configurations, as detailed here.

  • It would act as a gateway for all VPN clients (push “redirect-gateway”)
  • It should push the DNS address that the Singapore server uses (push “dhcp-option DNS x.x.x.x”).
  • I turned on IP Forwarding on Debian.
  • I configured the iptables to run NAT on vpn traffic:
    • iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE
    • iptables -t nat -L -n will show the result

Next, I installed OpenVPN-GUI on my Vista laptop. To make it work, I had to do these additional steps:

  • Set the bin\openvin.exe to run as administrator (right-click -> Properties -> Compatability tab -> set run as administrator)
  • Added these two lines to the end of the client.ovpn file
    • route-method exe
    • route-delay 2
  • Set up Windows Firewall so that it does not firewall the Windows TAP virtual adapter. To do this, I first checked to see which is the Windows TAP virtual adapter by opening Device Manager, under Network Adapters. Then run ipconfig /all in a command prompt window to find out which network connection uses the TAP device. Finally, in Windows Firewall -> Advanced tab, I looked for the network connection corresponding to the TAP virtual adapter, and unchecked it.

It took a little bit of debugging to get everything going. Once it did, I set about testing.

Initial results were not mind-blowing. Using speedtest.net, I got these results.

 

Without VPN

With VPN

Ping

Generally above 350.

Can be above 320 sometimes.

Generally below 350.

Transfer of a 1.5MB file via WinSCP

1min 17s.

1min 23s.

SpeedTest.net (to Seattle, US)

Download/Upload/Ping

Try 1: 700/350/372

Try 2: 652/343/372

Upload/Upload/Ping

Try 1: 499/548/319

Try 2: 548/827/328

So it would appear that with VPN via Singapore, ping times improve more than 10%, upload speeds improve by 100%, but download speeds get worse by about 20%.

However, in practice, I find the VPN solution to be an excellent backup for those frequent times where Streamyx just chokes. For example, just today my staff tried to download a 6MB file, and after 10 minutes it appeared it would still take another 50 minutes over Streamyx (groan). I whipped up my VPN connection, and the file downloaded in less than 5 minutes. We can all agree that’s an appreciable improvement.

As another example, there are some sites which simply die when trying to load over Streamyx. One example is www.visiblebody.com whose servers are located in Massachusetts,  It’s an awesome site that allows you to explore the human body in 3D (for free!). For an amateur anatomist like I am, it’s truly a gift. However, with Streamyx I simply could not load the entire 3D model, despite trying more than 10 times. But with the VPN solution, I got the model downloaded and ready for exploration in less than 15 minutes (it’s a BIG model).

Now, this solution is not cheap. I elected for the VPS package which allows for 150GB of monthly transfer for USD30/month. But for those who are desperate for a faster and more consistent link to the US, and only have Streamyx as a viable option, than the USD30/month is absolutely worth it.

Last note: I understand HTTP proxy can also be a viable solution, but for me I wanted a complete TCP/IP solution because I use other protocols like WinSCP, CVS, etc.