I’ll keep this brief. I’ve been thinking a lot lately about SETI. What happens when we do, at some point in our existence, come into contact with another “intelligent” life form? Ideally we’d communicate with it. I think this part is a bit overlooked. How do we communicate with other life forms?

I think the best way to start is to learn to communicate with the life forms that are right here on Earth. If we can’t communicate fluently with mammals, birds, reptiles, etc then how can we expect ourselves to communicate with life forms that have (in all likeliness) evolved under completely different circumstances than us?

I think the main problem is humanity’s bar for “intelligence.” Humans, I believe, are not intelligent. We know stuff and we can do stuff and over our short existence we’ve managed to dramatically alter our evolutionary path, but I believe about %0.01 of what we’ve done is intelligent. The rest is driven by fear and greed. We’re very quick to shout our intelligence from the mountaintops whilst viewing other beings that we share this planet with as simpler or lesser.

Perhaps all the beings of this planet aren’t lesser. Perhaps they all have their own languages. Perhaps we could have talked to them at one point in our existence but now we are so incredibly intelligent that we can’t understand them anymore.

The Search for Extraterrestrial Intelligence needs to start here. We need to forget about what we know about ourselves and the rest of our world. We need to learn the language of our world and its inhabitants before we go searching for languages of other worlds.

We need to learn to listen before we talk.

So my brother Jeff and I are building to Javascript-heavy applications at the moment (heavy as in all-js front-end). We needed a framework that provides loose coupling between the pieces, event/message-based invoking, and maps well to our data structures. A few choices came up, most notably Backbone.js and Spine. These are excellent frameworks. It took a while to wrap my head around the paradigms because I was so used to writing five layers deep of embedded events. Now that I have the hang of it, I can’t think of how I ever lived without it. There’s just one large problem…these libraries are for jQuery.

jQuery isn’t bad. We’ve always gravitated towards Mootools though. Mootools is a framework to make javascript more usable, jQuery is nearly a completely new language in itself written on top of javascript (and mainly for DOM manipulation). Both have their benefits, but we were always good at javascript before the frameworks came along, so something that made that knowledge more useful was an obvious choice for us.

I’ll also say that after spending some time with these frameworks and being sold (I especially liked Backbone.js) I gave jQuery another shot. I ported all of our common libraries to jQuery and I spent a few days getting used to it and learning how to do certain things. I couldn’t stand it. The thing that got me most was that there is no distinction between a DOM node and a collection of DOM nodes. Maybe I’m just too used to Moo (4+ years).

Composer.js

So we decided to roll our own. Composer.js was born. It merges aspects of Spine and Backbone.js into a Mootools-based MVC framework. It’s still in progress, but we’re solidifying a lot of the API so developers won’t have to worry about switching their code when v1 comes around.

Read the docs, give it a shot, and let us know if you have any problems or questions.

Also, yes, we blatantly ripped off Backbone.js in a lot of places. We’re pretty open about it, and also pretty open about attributing everything we took. They did some really awesome things. We didn’t necessarily want to do it differently more than we wanted a supported Mootools MVC framework that works like Backbone.

I was looking around for Riak information when I stumbled (not via stumble, but actually doing my own blundering) across a blog post that mentioned a Riak GUI. I checked it out. Install is simple and oddly enough, the tool uses only javascript and Riak (no web server needed). I have to say I’m thoroughly impressed by it. Currently the tool doesn’t do a ton besides listing buckets, keys, and stats, but you can edit your data inline and delete objects. It also supports Luwak, which I have no first-hand experience with and was unable to try out.

One thing I thought that was missing was a way to run a map-reduce on the cluster via text input boxes for the functions. It would make writing and testing them a bit simpler I think, but then again it would be easy enough to write this myself in PHP or even JS, so maybe I’ll add it in. Search integration would be nice too, although going to 127.0.0.1:8098/solr/[bucket]search?… is pretty stupid easy.

All in all, a great tool.

John Resign created some incredibly wonderful Javascript code for templating. It’s so terse that it almost shouldn’t work…but it does. I’ve been using it on a lot of front-end JS apps lately, and realized I could make a few changes and improvements.

First off, I don’t like <% asp style tags %>. It reminds me of programming ASP. It reminds me of a trip through hell I’ve taken too many times. I changed it to use PHP-style tags instead:

<ul class="<?=myclass?>">
<? for(var i = 0; i < items.length; i++) { ?>
    <li><?=items[i].name?>
<? } ?>
</ul>

This makes it easier for me to type. I also made one further modification. Adding $ in front of your variables will check if they are undefined before using them, and if not defined will return them as null:

Hello, <?=$name?>.
<? if($user.friends) { ?>
    You have <?=$user.friends.length?> friends.
<? } ?>

The if statement above will compile to

if((typeof(user.friends) == 'undefined' ? null : user.friends)) { ... 

This allows some simple usages of undefined variables, such as “if(undefined_var) { … } else { … }” which actually pops up a lot. You still can’t access non-existent properties of variables that aren’t defined, but this should catch a lot of errors that would otherwise turn your code into a bunch of if(typeof …)’s.

Here’s the code (for brevity, I left out all the caching stuff that makes this fast):

var template = '<h1>My Template</h1> ...';
new Function(
	"obj",
	"var p=[],print=function(){p.push.apply(p,arguments);};" +
	// Introduce the data as local variables using with(){}
	"with(obj) {p.push('" +
	// Convert the template into pure JavaScript
	template.replace(/[\r\t\n]/g, " ")
		// find any code blocks (not html, not <?=print_var?>... anything inside a
		// <? ... ?>
		.replace(/<\?(.*?)\?>/g, function(match) {
			// look for any string starting with a "$" and wrap it in a ternary typeof op
			return match.replace(/\$([a-z_][a-z0-9_\.]+)/gi, '(typeof($1) == "undefined" ? null : $1)');
		})
		.split("<?").join("\t")
		.replace(/((^|\?>)[^\t]*)'/g, "$1\r")
		.replace(/\t=\$?(.*?)\?>/g, "',(typeof($1) != 'undefined' ? $1 : ''),'")
		.split("\t").join("');")
		.split("?>").join("p.push('")
		.split("\r").join("\\'") + "');}"
		//+ "console.log('Loading template " + name + "');"
		+ "return p.join('');"
);

This has been working for me for a bit now, and has saved me countless annoying declarations at the top if my templates. If you run into any problems, please let me know.

I just did a writeup about MongoDB’s performance in the last big app we did. Now it’s time to rip Mono a new one.

Mono has been great. It’s .NET for linux. We originally implemented it because it’s noted for being a fast, robust compiled language. I didn’t know C# before starting the project, but afterwards I feel I have a fairly good grasp on it (10 months of using it constantly will do that). I have to say I like it. Coming from a background in C++, C# is very similar except the biggest draw is you don’t separate out your definitions from your code. Your code is your definition. No header files. I understand this is a requirement if you’re going to link code in C/C++ to other C/C++ code, but I hate doing it.

Back to the point, mono is great in many ways. It is fast, compiles from source fairly easily (although libgdiplus is another story, if you want to do image processing), and easy to program in.

We built out a large queuing system with C#. You enter jobs into a queue table in MongoDB, and they get processed based on priority/time entered (more or less) by C#. Jobs can be anything from gathering information from third-parties to generating images and layering them all together (I actually learned first-hand how some of these Photoshop filters work). The P/Invoke system allowed us to integrate with third party libraries where the language failed (such as simple web requests with timeouts or loading custom fonts,  for instance).

As with any project, it started off great. Small is good. Once we started processing large numbers of items in parallel, we’d get horrible crashes with native stacktraces. At first glance, it looked like problems with the Boehm garbage collector. We recompiled Mono with –enable-big-arrays and –with-large-heap. No luck. We contacted the Mono guys and, probably in lieu of all the political shenanigans happening with Mono at the moment, didn’t really have a good response for us. Any time the memory footprint got greater than 3.5G, it would crash. It didn’t happen immediately though, it seems random. Keep in mind Mono and the machines running it were 64bit…4G is not the limit!

Our solution was two fold:

• Put crash-prone code into separate binaries and call them via shell. If the process crashes, oh well, try again. The entire queue doesn’t crash though. This is especially handy with the image libraries, which seem to have really nasty crashes every once in a while (not related to the garbage collection).
• Make sure Monit is watching at all times.

We also gave the new sgen GC a try, but it was much too slow to even compare to the Boehm. It’s supposed to be faster, but pitting the two against each other in a highly concurrent setting crowned Boehm the clear winner.

All in all, I like C# the language and Mono seemed very well put together at a small to medium scale. The garbage collector shits out at a high memory/concurrency level. I wouldn’t put Mono in a server again until the GC stuff gets fixed, which seems low priority from my dealings with the devs. Still better than Java though.

Let me set the background by saying that I currently (until the end of the week anyway) work for a large tech company. We recently launched a reader app for iPad. On the backend we have a thin layer of PHP, and behind that a lot of processing via C# with Mono. I, along with my brother Jeff, wrote most of the backend (PHP and C#). The C# side is mainly a queuing system driven off of MongoDB.

Our queuing system is different from others in that it supports dependencies. For instance, before one job completes, its four children have to complete first. This allows us to create jobs that are actually trees of items all processing in parallel.

On a small scale, things went fairly well. We built the entire system out, and tested and built onto it over the period of a few months. Then came time for production testing. The nice thing about this app was that most of it could be tested via fake users and batch processing. We loaded up a few hundred thousand fake users and went to town. What did we find?

Without a doubt, MongoDB was the biggest bottleneck. What we really needed was a ton of write throughput. What did we do? Shard, of course. Problem was that we needed even distribution on insert…which would give us almost near-perfect balance for insert/update throughput. From what we found, there’s only one way to do this: give each queue item a randomly assigned “bucket” and shard based on that bucket value. In other words, do your own sharding manually, for the most part.

This was pretty disappointing. One of the whole reasons for going with Mongo is that it’s fast and scales easily. It really wasn’t as painless as everyone led us to believe. If I could do it all over again, I’d say screw dependencies, and put everything into Redis, but the dependencies required more advanced queries than any key-value system could do. I’m also convinced a single MySQL instance could have easily handled what four MongoDB shards could barely keep up with…but at this point, that’s just speculation.

So there’s my advice: don’t use MongoDB for evenly-distributed high-write applications. One of the hugest problems is that there is a global write lock on the database. Yes, the database…not the record, not the collection. You cannot write to MongoDB while another write is happening anywhere. Bad news bears.

On a more positive note, for everything BUT the queuing system (which we did get working GREAT after throwing enough servers at it, by the way) MongoDB has worked flawlessly. The schemaless design has cut development time in half AT LEAST, and replica sets really do work insanely well. After all’s said and done, I would use MongoDB again, but for read-mostly data. Anything that’s high-write, I’d go Redis (w/client key-hash sharding, like most memcached clients) or Riak (which I have zero experience in but sounds very promising).

TL,DR; MongoDB is awesome. I recommend it for most usages. We happened to pick one of the few things it’s not good at and ended up wasting a lot of time trying to patch it together. This could have been avoided if we picked something that was built for high write throughput, or dropped our application’s “queue dependency” requirements early on. I would like if MongoDB advertised the global write lock a bit more prominently, because I felt gypped when one of their devs mentioned it in passing months after we’d started. I do have a few other projects in the pipeline and plan on using MongoDB for them.

I looked long and hard (reallly long and reallly hard) for an x86_64 version of the SDL.dll file (Simple DirectMedia Layer) but found it nowhere. I ended up compiling it myself with mingw64 and figured maybe some other people might benefit from my hard work and dedication.

Get it here, and place the DLL in %WINDOWS%\System32\

Keep in mind this was compiled on my system, so it may not work on yours, although I’ve tested it on three different x86_64 Windows 7 machines and it works fine, just fine. Lispbuilder-sdl loads the DLL and I’m able to do OpenGL programming in Common Lisp with it (at least in ClozureCL).

Check out my self-organizing map project on github for example usage (info on SOMs).

Wow, I can’t believe I missed this…nobody seems to be talking about it at all. Ever since PHP 5.3, I can finally do non-generic callbacks.

UPDATE: Check out this description of PHP lambdas (much better than what I’ve done in the following).

function do_something($value)
{
    // used >= 2 times, but only in this function, so no need for a global
    $local_function = function($value) { ... };

    // use our wonderful anonymous function
    $result = $local_function($value);
    ...
    // and again
    $result = $local_function($result);
    return $result;
}

There’s also some other great stuff you can do:

$favorite_songs = array(
    array('name' => 'hit me baby one more time', 'artist' => 'britney'),
    array('name' => 'genie in a bottle', 'artist' => 'xtina'),
    array('name' => 'last resort', 'artist' => 'papa roach')
);
$song_names = array_map(function($item) { return $item['name']; }, $favorite_songs);

GnArLy bra. If PHP was 20 miles behind Lisp, it just caught up by about 30 feet. This has wonderful implications because there are a lot of functions that take a callback, and the only way to use them was to define a global function and send in an array() callback. Terrible. Inexcusable. Vomit-inducing.

Not only can you now use anonymous functions for things like array_map() and preg_replace_callback(), you can define your own functions that take functions as arguments:

function do_something_binary($fn_success, $fn_failed)
{
    $success = ...
    if($success)
    {
        return $fn_success();
    }
    return $fn_failed();
}

do_something_binary(
    function() { echo "I successfully fucked a goat!"; },
    function() { echo "The goat got away..."; }
);

Sure, you could just return $success and call whichever function you need after that, but this is just a simple example. It can be very useful to encapsulate code and send it somewhere, this is just a demonstration of the beautiful new world that just opened for PHP.

So drop your crap shared host (unless it has >= 5.3.0), get a VPS, and start using this wonderful new feature.

After coming to terms with how completely idiotic the rapid over-consumption of anti-radiation pills was after a recent nuclear energy scare, the American public realized it had to turn in another direction to fix all of its problems: anti-stupidity pills. The new drug, dubbed dioxide-enisium-rectopeptide (or “Derp”), recently gained FDA approval and hit the shelves of drug stores last week.

Since the phenomenon started, the average IQ rates of Americans has rocketed from 16.4 to a staggering 16.7. Not only are 99% of Americans now able to withstand a 400-year nuclear holocaust, but most can recite the entire English alphabet or even distinguish between highly complex shapes, such as squares and triangles. With this increase in smarts, scientists predict that within two weeks, 99% of the population will be scientists.

Plans to build over 100 million nuclear bombs and detonate them around the globe are being drafted by Congress; the idea being that only Americans are worthy of living because they are so incredibly intelligent and amazing.

“Just think! Without the other countries, where will be no war, no fighting, no trade embargoes, no failing world economy. Just peace,” Republic Senator Rick Harris stated. “Americans can finally hold hands in circles and dance and laugh and sing. And with everyone here taking anti-radiation pills, the nuclear fallout won’t even affect us. We can live like kings! KINGS!!! HAHAHAHA!!”

President Obama has signed the bill and the motion is currently in action.

Although things are looking good for the U.S. right now, there is a dark side to being the smartest nation in the universe and all containing dimensions. Top thinkers worry that nobody will want to empty trash cans, sweep floors, pay taxes, or any of the other horribly mundane tasks that years of failure and stupidity have forced us into. Some have suggested that a select few people be banned from taking the smart pills, effectively forcing them into perpetual state of drooling idiocy. Others think we could spare Canada from the final bombing and enslave the entire country. A general consensus has not been reached.

Just when you thought Americans couldn’t be any more intelligent, fun-loving, passionate, humble, caring, progressive, radiation-resistant, cultured, intelligent, helpful, and intelligent enough already, they proved the world wrong again.

One thing that annoys me in Vim is that in normal mode, the cursor defaults to being at the end of a tab character. When I hit “Home” I expect the cursor to go all the way to the left, but instead it hovers 4 spaces to the right of where I expect it to. I stumbled across the answer after reading a mailing list thread for vim.

set list lcs=tab:\ \ 
" Note the extra space after the second \

You can put this in your .vimrc to automatically set this behavior. Very useful.