A while ago, I released cl-async, a library for non-blocking programming in Common Lisp. I’ve been updating it a lot over the past month or two to add features and fix bugs, and it’s really coming along.

My goal for this project is to create a general-purpose library for asynchronous programming in lisp. I think I have achieved this. With the finishing of the futures implementation, not only is the library stable, but there is now a platform to build drivers on top of. This will be my next focal point over the next few months.

There are a few reasons I decided to build something new. Here’s an overview of the non-blocking libraries I could find:

• IOLib – An IO library for lisp that has a built-in event loop, only works on *nix.
• Hinge – General purpose, non-blocking library. Only works on *nix, requires libev and ZeroMQ.
• Conserv – A nice layer on top of IOLib (so once again, only runs on *nix). Includes TCP client/server and HTTP server implementations. Very nice.
• teepeedee2 – A non-blocking, performant HTTP server written on top of IOLib.

I created cl-async because of all the available libraries, they are either non-portable, not general enough, have too many dependencies, or a combination of all three. I wanted a library that worked on Linux and Windows. I wanted a portable base to start from, and I also wanted tools to help make drivers.

Keeping all this in mind, I created bindings for libevent2 and built cl-async on top of them. There were many good reasons for choosing libevent2 over other libraries, such as libev and libuv (the backend for Node.js). Libuv would have been my first choice because it supports IOCP in Windows (libevent does not), however wrapping it in CFFI was like getting a screaming toddler to see the logic behind your decision to put them to bed. It could have maybe happened if I’d written a compatibility layer in C, but I wanted to have a maximum of 1 (one) dependency. Libevent2 won. It’s fast, portable, easy to wrap in CFFI, and on top of that, has a lot of really nice features like an HTTP client/server, TCP buffering, DNS, etc etc etc. The list goes on. That means less programming for me.

Like I mentioned, my next goal is to build drivers. I’ve already built a few, but I don’t consider them stable enough to release yet. Drivers are the elephant in the room. Anybody can implement non-blocking IO for lisp, but the real challenge is converting everything that talks over TCP/HTTP to be async. If lisp supported coroutines, this would be trivial, but alas, we’re stuck with futures and the lovely syntax they afford.

I’m going to start with drivers I use every day: beanstalk, redis, cl-mongo, drakma, zs3, and cl-smtp. These are the packages we use at work in our queue processing system (now threaded, soon to be evented + threaded). Once a few of these are done, I’ll update the cl-async drivers page with best practices for building drivers (aka wrapping async into futures). Then I will take over the world.

Another goal I have is to build a real HTTP server on top of the bare http-server implementation provided by cl-async. This will include nice syntax around routing (allowing REST interfaces), static file serving, etc.

Cl-async is still a work in progress, but it’s starting to become stabilized (both in lack of bugs and the API itself), so check out the docs or the github project and give it a shot. All you need is a lisp and libevent =].

So my brother Jeff and I are building to Javascript-heavy applications at the moment (heavy as in all-js front-end). We needed a framework that provides loose coupling between the pieces, event/message-based invoking, and maps well to our data structures. A few choices came up, most notably Backbone.js and Spine. These are excellent frameworks. It took a while to wrap my head around the paradigms because I was so used to writing five layers deep of embedded events. Now that I have the hang of it, I can’t think of how I ever lived without it. There’s just one large problem…these libraries are for jQuery.

jQuery isn’t bad. We’ve always gravitated towards Mootools though. Mootools is a framework to make javascript more usable, jQuery is nearly a completely new language in itself written on top of javascript (and mainly for DOM manipulation). Both have their benefits, but we were always good at javascript before the frameworks came along, so something that made that knowledge more useful was an obvious choice for us.

I’ll also say that after spending some time with these frameworks and being sold (I especially liked Backbone.js) I gave jQuery another shot. I ported all of our common libraries to jQuery and I spent a few days getting used to it and learning how to do certain things. I couldn’t stand it. The thing that got me most was that there is no distinction between a DOM node and a collection of DOM nodes. Maybe I’m just too used to Moo (4+ years).

Composer.js »

So we decided to roll our own. Composer.js was born. It merges aspects of Spine and Backbone.js into a Mootools-based MVC framework. It’s still in progress, but we’re solidifying a lot of the API so developers won’t have to worry about switching their code when v1 comes around.

Read the docs, give it a shot, and let us know if you have any problems or questions.

Also, yes, we blatantly ripped off Backbone.js in a lot of places. We’re pretty open about it, and also pretty open about attributing everything we took. They did some really awesome things. We didn’t necessarily want to do it differently more than we wanted a supported Mootools MVC framework that works like Backbone.

I’ve been using Google Analytics for quite some time. I do love it. It’s easy, it’s pretty, it’s addicting to look at the charts and graphs. Lately I’ve been trying out an alternative to Google Analytics, Piwik.

I’m not going to list a bunch of bullet points with features of both compared. I am, however, going to tell you my experiences with both. I started with Google Analytics. The setup and install is so easy, a blind ape could do it. Once tracking, the graphs, maps, numbers are all easy to understand. You can compare your site to others like it, and you can set up different segments of visitors and display the graphs according to your segments.

Google Analytics has been fun and easy. I do have some hangups about it. Although there is no cost, Google owns your data. They are giving you a service, you are selling them your data in return. They know about your visitors and can track them based on their interest in various pages of yours. They can do whatever they want with this information. For some people, this is fine. I personally don’t give much of a rat’s ass what Google thinks they know about me. For others, this is a privacy issue.

I also ran into some limitations with Google Analytics. It doesn’t track downloads very easily, and getting any sort of report that it doesn’t already give you is impossible because you only get what you see.

I decided to try Piwik. It’s open source, they advertise themselves as an alternative to GA, and you own all of your data. I threw up a new site on NearlyFreeSpeech.net, installed Piwik (literally a 5 min. install, just as they say), and started playing with it. A lot of the graphs are the same, the dashboard is completely customizable and is not jenky at all. After tracking some sites with it, I’m convinced it’s actually more accurate than Google. It picks up more visitors and keywords from search engines.

So I’ve been using Piwik regularly for about 3 or 4 months. There are some things I miss. Goals in Piwik do not have the awesome funnel that GA does. Not even close. The goals are pretty stupid, honestly, and all the ones I track are done manually through javascript. It’s nice to be able to track them, but it’s something I can get through any of my apps anyway. So from what I’ve noticed, Piwik is missing the funnel view (although they are working on it), and it’s missing an IP filter: half the visits are from me sometimes, and it’d be nice to be able to browse my own apps without having to worry about messing up the analytics.

Aside from those two points, Piwik is the winner for me. I really never actually check Google Analytics anymore. Piwik really has stepped up and provided a service that’s comparable in features to GA, but free as free for me to use it without having to give away info about my awesome users. I would definitely urge anyone who likes Google Analytics to check out Piwik. The best part is, they’re actively developing it and there more and more features to look forward to as new releases come out.

I recently read a post on a web development firm’s blog (anonymous to protect them and myself). It was talking about how open-source web software is inferior to closed-source. The main reasoning was that open-source allows attackers to find vulnerabilities just by sifting through the code. The company touts their proprietary CMS as better than Drupal or WordPress because only they (and their customers, heh) see the source code. Therefore it’s rock solid.

I was kind of blown away by this. Obviously it’s a marketing ploy to scare unknowing customers into using them instead of doing a simple WordPress install, but it’s blatantly wrong and I feel the need to respond. Oddly enough, their blog is in WordPress. Hmm.

First off, all software has vulnerabilities. All servers have vulnerabilities. Yes, it’s easier to find them if you know the setup or know the code, but from what I’ve seen in my lifetime of computer work is this: if someone wants to hack your site, they will. If there is a vulnerability, they will find it. And as I just said, all software has vulnerabilities. It’s stupid to assume that because the source is only readily available to people who pay you money and the people who work on their site after you that no vulnerabilities will ever be found. They will be found. Look at Google. They were just hacked by China. Does Google open source their Gmail app? No, completely closed-source. But someone wanted to hack them, so they got hacked. That’s what happens. Also, if your proprietary CMS is written in PHP, Python, Ruby, Perl, etc etc…you’re still using open source. Someone could attack the site at the language level. Does it make sense to now develop your own closed-source programming language so nobody will ever be able to hack it?

Secondly, most well-known open-source software has been around a very long time and has had hundreds of thousands (if not millions) of people using it. This means that over time, it gets battle-hardened. The common and not-so-common vulnerabilities are found, leaving the users with the latest versions a rock-solid code base that has gone through thousands of revisions to be extremely secure. With open-source, you’ve got hundreds of eyes looking over everything that’s added/changed/removed at all times. With proprietary code, you get a few pairs of eyes at best, with much fewer installs, much fewer revisions to harden and secure.

Is open-source better than proprietary? If you’re poor, most likely, but otherwise they both have their good and bad points. The main point of this article isn’t to bash proprietary software at all, it’s to refute the claim that because the source is open the product is less secure. I believe the exact opposite, in fact. If your code is open for everyone to look at, you damn well better be good at seeing vulnerabilities before they even get deployed…and if you don’t catch it, someone else developing the project probably will.

Is open source too open? Hell no.