Memory searching

Every game has variables that store information: your ammo, your health, how large your penis is, etc. Each of these variables generally has a dedicated position in memory where it’s stored. If you can find the location of this value, you can modify it to your liking. But how to you find it? This is the process of memory searching.

The tool we’re going to use is called TSearch. I don’t think it has an official site anymore (assuming it ever did), but you can get it off this completely legit Angelfire website (direct link). While I can’t guarantee the executable is free of pathogens of the digital persuasion, I can say that I’ve scanned it with several tools and come up blank. UPDATE: There are probably better tools you can use, for instance Cheat Engine, an open source program in the same vein as TSearch. I Chose TSearch because that’s what I used in the old days, and had a copy laying around on my machine.

The way memory searching works is you pick a value you want to modify. We’re going to do health. So open start Terraria, open your favorite world, and let’s go to town. Once Terraria is loaded, open TSearch. On the top-left, you’ll see a button called “Open process.” Click it and select Terraria.

Now that the process is open, TSearch has access to Terraria’s memory, including your health value >=]. We’ll start our search for the location soon, but first we need a starting point.

The next step is to hurt yourself in Terraria. Yes, we’re going to to be playing the part of the masochist (but only for a little while). So jump off that cliff, dip your toe in lava, tango with a skeleton, whatever it takes…you want your health value to not be in its default position.

We’re going to search for that health value in Tsearch and as a starting point to try and find its exact location in memory.

Once we have lowered our health, open Tsearch. Make note of the value your health is in Terraria (and also make sure the game is paused).

Open TSearch and click on the little magnifying glass icon under the “Open process” button (the one furthest to the left). It will bring up a dialog that allows us to search for locations in memory based on value. We’re going to pull down the menu and select “range.” We use “range” instead of “exact value” because the health value regenerates, meaning at any given point it might be between two integers.

Let’s say our health is at 268 (like the image to the right). For value one, we enter 267, and value two 269. Then select Type: “2-byte.” This means we’re searching for a 2-byte value between 267 and 269. Hit “Ok.”

It’s going to find probably tens of thousands of values. Don’t worry, we can narrow these down. Open Terraria again and let your health regenerate (or hurt yourself again, being careful not to kill yourself). Once your health changes by a few points, open TSearch again.

We’re going to search again, but search within our current results. Click the magnifying glass directly to the right of the first one (it has “…” on it). This will open the same dialog before. Do a ranged search. Let’s say our new health value is 259, we’re going to search between 258 and 260. Run the search.

If you’re lucky, you’ll get only one result. If you’re unlucky and you get more than one search result, repeat the above filtering step one more time (change the health value, filter withing the current results, rinse & repeat until only one value remains). Sometimes you’ll have to do many searches in a row to get the results you want.

Once you have your one search result, click on that entry in the search box and then click the green plus icon. It should add it to the list on the right. Double click under the “Description” field and name the value “health.” On the right, you’ll notice the value 259. Change it to 400. The click the box next to the value’s name (it should turn blue). You just froze your health at 400.

Go back into the game and hurt yourself. You’ll notice that your health goes down, but immediately bounces back after about half a second. Ladies and gentlemen, enjoy your infinite health =].

Notes

Hopefully this can answer some more questions about memory searching.

• This method only works while TSearch is open. Once you close it, the game will start tracking health again. If you want your hacks to persist, it’s a whole nother set of tools and processes, worthy of at least a few more posts. Keep in mind that if you use Steam, it will probably overwrite any changes you make to the exe. Such is life.
• Memory searching works for values besides health. Be creative! You can get unlimited ammo, unlimited money, etc.
• In Terraria, using this method for money can be tricky. If you move money from one of the money slots, it will change memory locations and you’ll lose your hack.
• This method works for other games as well. I chose Terraria because it’s the only game I currently use it on.

Have fun!

./configure --allow-fetch
make ALLOW_WARNINGS=1

This will build RethinkDB without a hitch.

UPDATE: Check out Samuel Hughes’ comment on compiling in Slack, which may make some of the below process simpler. Specifically, the section about editing the Makefile to not use static libraries (apparently you can pass RECOMMEND_STATIC=0 to the make process to do this instead).

So I’ve had a theoretical boner for RethinkDB ever since reading about it. I decided to try and give it a go, but have had problems compiling. I’m going to try and give an overview of how to get it running. These instructions are aimed at people who don’t have a linux with the targeted packaging systems that RethinkDB currently supports (in other words, you’re stuck with compiling it yourself). The build process is slightly annoying (which is why I’m writing this guide). Slava from the RethinkDB team told me that they are working on a new build system, so hopefully we’ll soon be able to just to a make.

Installing V8

See the instructions for building and installing V8. They are pretty simple, but I believe you need Python (since they use GYP for the build):

cd v8
make dependencies
make native
# do a manual installation
sudo mkdir -p /usr/local/v8/include /usr/local/v8/lib
sudo cp include/* /usr/local/v8/include
sudo cp out/native/lib.target/libv8.so /usr/local/v8/lib

Done (yeah, I know, my syntax highlighting is annoying).

Building RethinkDB

This was a bit trickier, but hey I’ve compiled alpha versions of Compiz on top of Slack before, this should be a cakewalk. It took some Makefile tweaking to get it running, so here’s how I did it. Note that you’ll need Python >= 2.7 to do the full make process (or else you’ll get “AttributeError: ‘module’ object has no attribute ‘check_output’”). Slack 13.1 comes with 2.6.x so I had to compile it. Guess I need to upgrade soon. So grab the latest source:

git clone git://github.com/rethinkdb/rethinkdb.git
cd rethinkdb

First things first, src/Makefile adds the option -Werror to the build. This is great, but causes the build to fail when it includes v8.h since there are unused variables. So for now, we’ll have to trust them that there are no other warnings/errors and remove this from the Makefile. So open src/Makefile in your favorite editor and change:

RT_CXXFLAGS+=-Wall -Wextra -Werror -Wnon-virtual-dtor

to

RT_CXXFLAGS+=-Wall -Wextra -Wnon-virtual-dtor

Also the Makefile tries to use static versions of some of the boost libs, but I only have dynamic versions on my system. So let’s comment out that line. Find this and comment it out (UPDATE: per Samuel Hughes’ comment below, you can skip this step and pass RECOMMEND_STATIC=0 to the make commend instead of hacking up the Makefile):

STATIC_RECOMMENDS_INDIFFERENT:=boost_serialization boost_program_options

becomes

#STATIC_RECOMMENDS_INDIFFERENT:=boost_serialization boost_program_options

Now let’s write a “simple” build script that wraps the make process:

#!/bin/bash

BIN="`pwd -P`/support/usr/bin"
export PATH=$PATH:$BIN
export RT_CXXFLAGS="-I./include -I./src -I/usr/local/v8/include"
export RT_LDFLAGS="-L/usr/local/v8/lib ../support/usr/lib/libprotobuf.a ../support/usr/lib/libtcmalloc_minimal.a -lboost_program_options -lboost_serialization"
export STATIC_LIBRARY_PATHS=$RT_LDFLAGS
make \
        VERBOSE=1 \
        FETCH_INTERNAL_TOOLS=1 \
        RECOMMEND_STATIC=0 \
        STATIC_LIBRARY_PATHS="$RT_LDFLAGS"

Save it to compile.sh (or just c like I do because life is just too short to be typing “ompile.sh” all over the place) and run it.

That should do it! If all the post-build stuff (like installing js modules and crap) works fine, you should be able to start the db like so:

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/v8/lib/ ./build/release/rethinkdb

If that’s too cumbersome for you, you can either link /usr/local/v8/lib/libv8.so to /usr/local/lib or add /usr/local/v8/lib to your /etc/ld.so.conf file (and, of course, run ldconfig as root) and run rethinkdb freely, without the worries of library paths.

However, upon starting Xorg, my keyboard mouse would not work. I tried many things: following the steps in the handbook, enabling/disabling hald, reconfiguring Xorg, etc. No luck. My Xorg.0.log was telling me that it couldn’t load the kdb/mouse drivers. After snooping around some forums, I found the solution:

• Install the x11-drivers/xf86-input-keyboard port
• Install the x11-drivers/xf86-input-mouse port

After doing this, all was right with the world. Just to clarify, I am using dbus/hald and more or less using the default configuration that Xorg -configure gave me.

My goal for this project is to create a general-purpose library for asynchronous programming in lisp. I think I have achieved this. With the finishing of the futures implementation, not only is the library stable, but there is now a platform to build drivers on top of. This will be my next focal point over the next few months.

There are a few reasons I decided to build something new. Here’s an overview of the non-blocking libraries I could find:

• IOLib – An IO library for lisp that has a built-in event loop, only works on *nix.
• Hinge – General purpose, non-blocking library. Only works on *nix, requires libev and ZeroMQ.
• Conserv – A nice layer on top of IOLib (so once again, only runs on *nix). Includes TCP client/server and HTTP server implementations. Very nice.
• teepeedee2 – A non-blocking, performant HTTP server written on top of IOLib.

I created cl-async because of all the available libraries, they are either non-portable, not general enough, have too many dependencies, or a combination of all three. I wanted a library that worked on Linux and Windows. I wanted a portable base to start from, and I also wanted tools to help make drivers.

Keeping all this in mind, I created bindings for libevent2 and built cl-async on top of them. There were many good reasons for choosing libevent2 over other libraries, such as libev and libuv (the backend for Node.js). Libuv would have been my first choice because it supports IOCP in Windows (libevent does not), however wrapping it in CFFI was like getting a screaming toddler to see the logic behind your decision to put them to bed. It could have maybe happened if I’d written a compatibility layer in C, but I wanted to have a maximum of 1 (one) dependency. Libevent2 won. It’s fast, portable, easy to wrap in CFFI, and on top of that, has a lot of really nice features like an HTTP client/server, TCP buffering, DNS, etc etc etc. The list goes on. That means less programming for me.

Like I mentioned, my next goal is to build drivers. I’ve already built a few, but I don’t consider them stable enough to release yet. Drivers are the elephant in the room. Anybody can implement non-blocking IO for lisp, but the real challenge is converting everything that talks over TCP/HTTP to be async. If lisp supported coroutines, this would be trivial, but alas, we’re stuck with futures and the lovely syntax they afford.

I’m going to start with drivers I use every day: beanstalk, redis, cl-mongo, drakma, zs3, and cl-smtp. These are the packages we use at work in our queue processing system (now threaded, soon to be evented + threaded). Once a few of these are done, I’ll update the cl-async drivers page with best practices for building drivers (aka wrapping async into futures). Then I will take over the world.

Another goal I have is to build a real HTTP server on top of the bare http-server implementation provided by cl-async. This will include nice syntax around routing (allowing REST interfaces), static file serving, etc.

Cl-async is still a work in progress, but it’s starting to become stabilized (both in lack of bugs and the API itself), so check out the docs or the github project and give it a shot. All you need is a lisp and libevent =].

The way cl-mongo’s connections work is that it has a global hash table that holds connections: cl-mongo::*mongo-registry*. When the threads are all running and communicating with MongoDB, they are using the same hash table without any inherent locking or synchronization. There are a few options to fix this. You can implement a connection pool that supports access from multiple threads (complicated), you can give each thread its own connection and force the each thread to use its connection when communicating, or you can take advantage of special variables in lisp (the easiest, simplest, and most elegant IMO). Let’s check out the last option.

Although it’s not in the CL spec, just about all implementations allow you to have global thread-local variables by using (defparameter) or (defvar), both of which create special variables (read: dynamic variables, as opposed to lexical). Luckily, cl-mongo uses defvar to create *mongo-registry*. This means in our worker, we can re-bind this variable above the top level loop using (let) and all subsequent calls to MongoDB will use our new thread-local version of *mongo-registry* instead of the global one that all the threads we’re bumping into each other using:

;; Main worker loop, using global *mongo-registry* (broken)
(defun start-worker ()
  (loop
    (let ((job (get-job)))
      (let ((results (process-job job)))
        ;; this uses the global registry. not good if running multiple threads.
        (with-mongo-connection (:db "musio")
          (db.save "scraped" results))))))

New version:

;; Replace *mongo-registry* above worker loop, creating a local version of the
;; registry for this thread.
(defun start-worker ()
  ;; setting to any value via let will re-create the variable as a local thread
  ;; variable. nil will do just fine.
  (let ((cl-mongo::*mongo-registry* nil))
    (loop
      (let ((job (get-job)))
        (let ((results (process-job job)))
          ;; with-mongo-connection now uses the local registry, which stops the
          ;; threads from touching each other.
          (with-mongo-connection (:db "musio")
            (db.save "scraped" results)))))))

BOOM everything works great after this change, and it was only a one line change. It may not be as efficient as connection pooling, but that’s a lot more prone to strange errors and synchronization issues than just segregating the connections from each other and calling it a day. One issue: *mongo-registry* is not exported by cl-mongo, which is why we access it via cl-mongo::*mongo-registry* (notice the double colon instead of single). This means in future versions, the variable name may change, breaking our above code. So, don’t update cl-mongo without testing. Not hard.

Hopefully this helps a few people out, let me know if you have better solutions to this issue!

People are saying it has outgrown it’s original usage, or it contains bad error messages, or it’s not smart about the messages received.

These are very smart people, with real observations. The problem is, their observations are misplaced.

What email is

Email is a distributed, asynchronous messaging protocol. It does this well. It does this very well. So well, I’m getting a boner thinking about it. You send a message and it either goes where it’s supposed to, or you get an error message back. That’s it, that’s email. It’s simple. It works.

There’s no company controlling all messages and imposing their will on the ecosystem as a whole. There’s no single point of failure. It’s beautifully distributed and functions near-perfectly.

The problem

So why does it suck so much? It doesn’t. It’s awesome. The problem is the way people view it. Most of the perceived suckiness comes from its simplicity. It doesn’t manage your TODOs. It doesn’t have built-in calendaring. It doesn’t give you oral pleasure (personally I think this should be built into the spec though). So why don’t we build all these great things into it if they don’t exist? We could add TODOs and calendaring and dick-sucking to email!!

Because that’s a terrible idea. People are viewing email as an application; one that has limited features and needs to be extended so it supports more than just stupid messages.

This is wrong.

We need to view email as a framework, not an application. It is used for sending messages. That’s it. It does this reliably and predictably.

Replacing email with “smarter” features will inevitably leave people out. I understand the desire to have email just be one huge TODO list. But sometimes I just want to send a fucking message, not “make a TODO.” Boom, I just “broke” the new email.

Email works because it does nothing but messaging.

How do we fix it then?

We fix it by building smart clients. Let’s take a look at some of our email-smart friends.

Outlook has built-in calendaring. BUT WAIT!!!!! Calendaring isn’t part of email!!1 No, it’s not.

Gmail has labels. You can categorize your messages by using tags essentially. Also, based on usage patterns, Gmail can give weight to certain messages. That’s not part of email either!! No, my friend, it’s not.

Xobni also has built incredible contact-management and intelligence features on top of email. How do they know it’s time to take your daily shit before you do? Defecation scheduling is NOT part of the email spec!!

How have these companies made so much fucking money off of adding features to email that are not part of email?

It’s all in the client

They do it by building smart clients! As I said, you can send any message your heart desires using email. You can send JSON messages with a TODO payload and attach a plaintext fallback. If both clients understand it, then BAM! Instant TODO list protocol. There, you just fixed email. Easy, no? Why, with the right client, you could fly a fucking space shuttle with email. That’s right, dude, a fucking space shuttle.

If your client can create a message and send it, and the receiving client can decode it, you can build any protocol you want on top of email.

That’s it. Use your imaginations. I’ll say it one more time:

There’s nothing to fix

Repeat after me: “There’s nothing to fix!” If you have a problem with email, fork a client or build your own! Nobody’s stopping you from “fixing” email. Many people have made a lot of cash by “fixing” email.

We don’t have to sit in fluorescent-lit, university buildings deliberating for hours on end about how to change the spec to fit everyone’s new needs. We don’t need 100 stupid startups “disrupting” the “broken” email system with their new protocols, that will inevitably end up being  a proprietary, non-distributed, “ad hoc, informally-specified, bug-ridden, slow implementation of half of” the current email system.

Please don’t try to fix email, you’re just going to fuck it up!! Trust me, you can’t do any better. Instead, let’s build all of our awesome new features on top of an already beautifully-working system by making smarter clients.

To make a long story short, some of the Mongo data files in the datadir were owned by root, not the “mongodb” user. I chowned them back to “mongodb” and everything went back to normal. Why this happened, I don’t know, but at least there’s a fix ;-).

1 chown -R mongodb:mongodb /srv/mongo-datadir

Here’s a sample (created with vim’s :TOhtml command):

 1 /**
 2  * Trigger an event for this object, which in turn runs all callbacks for that
 3  * event WITH all parameters passed in to this function.
 4  *
 5  * For instance, you could do:
 6  * mymodel.bind("destroy", this.removeFromView.bind(this));
 7  * mymodel.trigger("destroy", "omg", "lol", "wtf");
 8  *
 9  * this.removeFromView will be called with the arguments "omg", "lol", "wtf".
10  *
11  * Note that any trigger event will also trigger the "all" event. the idea
12  * being that you can subscribe to anything happening on an object.
13  */
14 trigger: function(ev)
15 {
16     var args   =   shallow_array_clone(Array.from(arguments));
17     [ev, 'all'].each(function(type) {
18         if(!this._events[type]) return;
19         Array.clone(this._events[type]).each(function(callback) {
20             callback.apply(this, (type == 'all') ? args : args.slice(1));
21         }, this);
22     }, this);
23 
24     return this;
25 }

This is javascript, but I also use it for HTML, CSS, php, and lisp. Note that all code highlighting on this blog is done via vim with this color scheme.

There’s a security flaw though. Let’s say I log in as user “andrew” and attach to my previous TMUX session: tmux attach. Now I have to run a number of commands as root. Well, prefixing every command with sudo and manually typing in all the /sbin/ paths to each executable it a pain in the ass. I know this is a bad idea, but I’ll often spawn a root shell. Let’s say I spawn a root shell in a TMUX session, then go do something else, fully intending log out later, but I forget. My computer disconnects, and I forget there’s a root shell sitting there.

If someone manages to compromise the machine, and gain access to my user account, getting a root shell is as easy as doing tmux attach. Oops.

Well, I just found out you can timeout a shell after X seconds of inactivity, which is perfect for this case. As root:

1 echo -e "\n# logout after 5 minutes of inactivity\nexport TMOUT=300\n" >> /root/.bash_profile

Now I can open root shells until my ass bleeds, and after 5 minutes of inactivity, it will log out back into my normal user account.

A good sysadmin won’t make mistakes. A great sysadmin will make mistakes self-correct ;-].

I just installed QuickCache which uses flat files installed into the wp-content/cache/ folder. Here was my installation:

• Upload the plugin to wp-content/plugins/
• In SSH:

  chgrp web wp-content/
  mkdir wp-content/cache
  chgrp web wp-content/cache

• Enable plugin in admin.
• Done!

Stupid easy, works really well. It seems commonplace for people to develop wordpress plugins with their error reporting set to the absolute lowest and display_errors=0. I don’t appreciate this as it usually produces broken code. I’m thankful that the QuickCache authors put in the effort to, like, actually make it work. Thanks!

jQuery isn’t bad. We’ve always gravitated towards Mootools though. Mootools is a framework to make javascript more usable, jQuery is nearly a completely new language in itself written on top of javascript (and mainly for DOM manipulation). Both have their benefits, but we were always good at javascript before the frameworks came along, so something that made that knowledge more useful was an obvious choice for us.

I’ll also say that after spending some time with these frameworks and being sold (I especially liked Backbone.js) I gave jQuery another shot. I ported all of our common libraries to jQuery and I spent a few days getting used to it and learning how to do certain things. I couldn’t stand it. The thing that got me most was that there is no distinction between a DOM node and a collection of DOM nodes. Maybe I’m just too used to Moo (4+ years).

Composer.js »

So we decided to roll our own. Composer.js was born. It merges aspects of Spine and Backbone.js into a Mootools-based MVC framework. It’s still in progress, but we’re solidifying a lot of the API so developers won’t have to worry about switching their code when v1 comes around.

Read the docs, give it a shot, and let us know if you have any problems or questions.

Also, yes, we blatantly ripped off Backbone.js in a lot of places. We’re pretty open about it, and also pretty open about attributing everything we took. They did some really awesome things. We didn’t necessarily want to do it differently more than we wanted a supported Mootools MVC framework that works like Backbone.

One thing I thought that was missing was a way to run a map-reduce on the cluster via text input boxes for the functions. It would make writing and testing them a bit simpler I think, but then again it would be easy enough to write this myself in PHP or even JS, so maybe I’ll add it in. Search integration would be nice too, although going to 127.0.0.1:8098/solr/[bucket]search?… is pretty stupid easy.

All in all, a great tool.

First off, I don’t like <% asp style tags %>. It reminds me of programming ASP. It reminds me of a trip through hell I’ve taken too many times. I changed it to use PHP-style tags instead:

1 <ul class="<?=myclass?>">
2     <? for(var i = 0; i < items.length; i++) { ?>
3         <li><?=items[i].name?>
4     <? } ?>
5 </ul>

This makes it easier for me to type. I also made one further modification. Adding $ in front of your variables will check if they are undefined before using them, and if not defined will return them as null:

1 Hello, <?=$name?>.
2 <? if($user.friends) { ?>
3     You have <?=$user.friends.length?> friends.
4 <? } ?>

The if statement above will compile to

1 if((typeof(user.friends) == 'undefined' ? null : user.friends)) { ...

This allows some simple usages of undefined variables, such as “if(undefined_var) { … } else { … }” which actually pops up a lot. You still can’t access non-existent properties of variables that aren’t defined, but this should catch a lot of errors that would otherwise turn your code into a bunch of if(typeof …)’s.

Here’s the code (for brevity, I left out all the caching stuff that makes this fast):

 1 var template    =   '<h1><?=$title?></h1> ...';
 2 template        =   template.replace(/(\r\n|\n\r)/g, "\n");
 3 var fnstr       =
 4     "var ___p=[],print=function(){___p.push.apply(___p,arguments);};" +
 5     "with(obj) {___p.push('" +
 6     template
 7         // fix single quotes in html (escape them)
 8         .replace(/(^|\?>)([\s\S]*?)($|<\?)/g, function(match) {
 9             return match.replace(/'/g, '\\\'');
10         })
11         // implement safe usage of $varname
12         .replace(/<\?([\s\S]*?)\?>/g, function(match) {
13             return match.replace(/\$([a-z_][a-z0-9_\.]+)/gi, '(typeof($1) == "undefined" ? null : $1)').replace(/[\r\n]+/g, ' ');
14         })
15         .replace(/\r?\n/g, '___::NEWLINE::___')
16         .split("<?").join('___::TABBBBB::___')
17         .replace(/((^|\?>)(?!=___::TABBBBB::___))'/g, "$1___::SLASHR::___")
18         .replace(/___::TABBBBB::___=\s*\$?(.*?)\?>/g, "',$1,'")
19         .split('___::TABBBBB::___').join("');")
20         .split('?>').join("___p.push('")
21         .split("___::SLASHR::___").join("\\'")
22         .replace(/___::NEWLINE::___/g, '\'+ "\\n" + \'') +
23         "');}" + "return ___p.join('');";
24 var tpl_fn  =   new Function("obj", fnstr);

This has been working for me for a bit now, and has saved me countless annoying declarations at the top if my templates. If you run into any problems, please let me know.

Mono has been great. It’s .NET for linux. We originally implemented it because it’s noted for being a fast, robust compiled language. I didn’t know C# before starting the project, but afterwards I feel I have a fairly good grasp on it (10 months of using it constantly will do that). I have to say I like it. Coming from a background in C++, C# is very similar except the biggest draw is you don’t separate out your definitions from your code. Your code is your definition. No header files. I understand this is a requirement if you’re going to link code in C/C++ to other C/C++ code, but I hate doing it.

Back to the point, mono is great in many ways. It is fast, compiles from source fairly easily (although libgdiplus is another story, if you want to do image processing), and easy to program in.

We built out a large queuing system with C#. You enter jobs into a queue table in MongoDB, and they get processed based on priority/time entered (more or less) by C#. Jobs can be anything from gathering information from third-parties to generating images and layering them all together (I actually learned first-hand how some of these Photoshop filters work). The P/Invoke system allowed us to integrate with third party libraries where the language failed (such as simple web requests with timeouts or loading custom fonts,  for instance).

As with any project, it started off great. Small is good. Once we started processing large numbers of items in parallel, we’d get horrible crashes with native stacktraces. At first glance, it looked like problems with the Boehm garbage collector. We recompiled Mono with –enable-big-arrays and –with-large-heap. No luck. We contacted the Mono guys and, probably in lieu of all the political shenanigans happening with Mono at the moment, didn’t really have a good response for us. Any time the memory footprint got greater than 3.5G, it would crash. It didn’t happen immediately though, it seems random. Keep in mind Mono and the machines running it were 64bit…4G is not the limit!

Our solution was two fold:

• Put crash-prone code into separate binaries and call them via shell. If the process crashes, oh well, try again. The entire queue doesn’t crash though. This is especially handy with the image libraries, which seem to have really nasty crashes every once in a while (not related to the garbage collection).
• Make sure Monit is watching at all times.

We also gave the new sgen GC a try, but it was much too slow to even compare to the Boehm. It’s supposed to be faster, but pitting the two against each other in a highly concurrent setting crowned Boehm the clear winner.

All in all, I like C# the language and Mono seemed very well put together at a small to medium scale. The garbage collector shits out at a high memory/concurrency level. I wouldn’t put Mono in a server again until the GC stuff gets fixed, which seems low priority from my dealings with the devs. Still better than Java though.

Our queuing system is different from others in that it supports dependencies. For instance, before one job completes, its four children have to complete first. This allows us to create jobs that are actually trees of items all processing in parallel.

On a small scale, things went fairly well. We built the entire system out, and tested and built onto it over the period of a few months. Then came time for production testing. The nice thing about this app was that most of it could be tested via fake users and batch processing. We loaded up a few hundred thousand fake users and went to town. What did we find?

Without a doubt, MongoDB was the biggest bottleneck. What we really needed was a ton of write throughput. What did we do? Shard, of course. Problem was that we needed even distribution on insert…which would give us almost near-perfect balance for insert/update throughput. From what we found, there’s only one way to do this: give each queue item a randomly assigned “bucket” and shard based on that bucket value. In other words, do your own sharding manually, for the most part.

This was pretty disappointing. One of the whole reasons for going with Mongo is that it’s fast and scales easily. It really wasn’t as painless as everyone led us to believe. If I could do it all over again, I’d say screw dependencies, and put everything into Redis, but the dependencies required more advanced queries than any key-value system could do. I’m also convinced a single MySQL instance could have easily handled what four MongoDB shards could barely keep up with…but at this point, that’s just speculation.

So there’s my advice: don’t use MongoDB for evenly-distributed high-write applications. One of the hugest problems is that there is a global write lock on the database. Yes, the database…not the record, not the collection. You cannot write to MongoDB while another write is happening anywhere. Bad news bears.

On a more positive note, for everything BUT the queuing system (which we did get working GREAT after throwing enough servers at it, by the way) MongoDB has worked flawlessly. The schemaless design has cut development time in half AT LEAST, and replica sets really do work insanely well. After all’s said and done, I would use MongoDB again, but for read-mostly data. Anything that’s high-write, I’d go Redis (w/client key-hash sharding, like most memcached clients) or Riak (which I have zero experience in but sounds very promising).

TL,DR; MongoDB is awesome. I recommend it for most usages. We happened to pick one of the few things it’s not good at and ended up wasting a lot of time trying to patch it together. This could have been avoided if we picked something that was built for high write throughput, or dropped our application’s “queue dependency” requirements early on. I would like if MongoDB advertised the global write lock a bit more prominently, because I felt gypped when one of their devs mentioned it in passing months after we’d started. I do have a few other projects in the pipeline and plan on using MongoDB for them.

Get it here, and place the DLL in %WINDOWS%\System32\

Keep in mind this was compiled on my system, so it may not work on yours, although I’ve tested it on three different x86_64 Windows 7 machines and it works fine, just fine. Lispbuilder-sdl loads the DLL and I’m able to do OpenGL programming in Common Lisp with it (at least in ClozureCL).

Check out my self-organizing map project on github for example usage (info on SOMs).

UPDATE: Check out this description of PHP lambdas (much better than what I’ve done in the following).

 2 function do_something($value)
 3 {
 4     // used >= 2 times, but only in this function, so no need for a global
 5     $local_function = function($value) { ... };
 6 
 7     // use our wonderful anonymous function
 8     $result = $local_function($value);
 9     ...
10     // and again
11     $result = $local_function($result);
12     return $result;
13 }

There’s also some other great stuff you can do:

2 $favorite_songs = array(
3     array('name' => 'hit me baby one more time', 'artist' => 'britney'),
4     array('name' => 'genie in a bottle', 'artist' => 'xtina'),
5     array('name' => 'last resort', 'artist' => 'papa roach')
6 );
7 $song_names = array_map(function($item) { return $item['name']; }, $favorite_songs);

GnArLy bra. If PHP was 20 miles behind Lisp, it just caught up by about 30 feet. This has wonderful implications because there are a lot of functions that take a callback, and the only way to use them was to define a global function and send in an array() callback. Terrible. Inexcusable. Vomit-inducing.

Not only can you now use anonymous functions for things like array_map() and preg_replace_callback(), you can define your own functions that take functions as arguments:

 2 function do_something_binary($fn_success, $fn_failed)
 3 {
 4     $success = ...
 5     if($success)
 6     {
 7         return $fn_success();
 8     }
 9     return $fn_failed();
10 }
11 
12 do_something_binary(
13     function() { echo "I successfully fucked a goat!"; },
14     function() { echo "The goat got away..."; }
15 );

Sure, you could just return $success and call whichever function you need after that, but this is just a simple example. It can be very useful to encapsulate code and send it somewhere, this is just a demonstration of the beautiful new world that just opened for PHP.

So drop your crap shared host (unless it has >= 5.3.0), get a VPS, and start using this wonderful new feature.

Since the phenomenon started, the average IQ rates of Americans has rocketed from 16.4 to a staggering 16.7. Not only are 99% of Americans now able to withstand a 400-year nuclear holocaust, but most can recite the entire English alphabet or even distinguish between highly complex shapes, such as squares and triangles. With this increase in smarts, scientists predict that within two weeks, 99% of the population will be scientists.

Plans to build over 100 million nuclear bombs and detonate them around the globe are being drafted by Congress; the idea being that only Americans are worthy of living because they are so incredibly intelligent and amazing.

“Just think! Without the other countries, where will be no war, no fighting, no trade embargoes, no failing world economy. Just peace,” Republic Senator Rick Harris stated. “Americans can finally hold hands in circles and dance and laugh and sing. And with everyone here taking anti-radiation pills, the nuclear fallout won’t even affect us. We can live like kings! KINGS!!! HAHAHAHA!!”

President Obama has signed the bill and the motion is currently in action.

Although things are looking good for the U.S. right now, there is a dark side to being the smartest nation in the universe and all containing dimensions. Top thinkers worry that nobody will want to empty trash cans, sweep floors, pay taxes, or any of the other horribly mundane tasks that years of failure and stupidity have forced us into. Some have suggested that a select few people be banned from taking the smart pills, effectively forcing them into perpetual state of drooling idiocy. Others think we could spare Canada from the final bombing and enslave the entire country. A general consensus has not been reached.

Just when you thought Americans couldn’t be any more intelligent, fun-loving, passionate, humble, caring, progressive, radiation-resistant, cultured, intelligent, helpful, and intelligent enough already, they proved the world wrong again.

set list lcs=tab:\ \ 
" Note the extra space after the second \

You can put this in your .vimrc to automatically set this behavior. Very useful.

Not anymore. Starting a few years back, I’ve done all of my development in Eclipse. It has wonderful plugins for PHP, C++, Javascript, etc. The past week or so I’ve been weening myself off of it and diving into vim. What actually got me started is I bought a Droid 2 off ebay for various hacking projects (I’m planning on reviewing it soon). Well, it was really easy to get vim working in it (sorry, lost the link already). I thought, well, shit, I’ve got vim, what the hell can I do with it? First things first, let’s get a plugin for syntax coloring/indentation for a few of my favorite languages. What?! It has all of them already.

Ok, now I’m interested. I installed vim for Windows (gvim), which was followed by a slow-but-steady growing period of “well, how do I do this” and “HA…I bet vim can’t do THI…oh, it can.” There are “marks” for saving your place in code, you can open the same file in multiple views (aka “windows”), you can bind just about any key combination to run any command or set of commands, etc. I even discovered tonight there’s a “windows” mode for vim that mimics how any normal editor works. I hate to admit it, but I’ll be using that a lot. One feature that blew my mind is the undo tree. Not stack, tree. Make a change, undo, make a new change, and the first change you did before your undo is still accessible (:undolist)!

The nice thing about vim is that it saves none of its settings. Every change you make to it while inside the editor is lost after a restart. This sounds aggravating, but it actually makes playing with the editor really fun and easy. If I open 30 windows and don’t know how to close them, just restart the editor. There are literally hundreds of trillions of instances when I was like “oh, shit” *restart*.

Once you have a good idea of what you want your environment to be like, you put all your startup commands in .vimrc (_vimrc on Windows) and vim runs it before it loads. Your settings file uses the same syntax as the commands you run inline in the editor, which is awesome and makes it easy to remember how to actually use vim.

So far I’m extremely impressed. The makers of vim have literally thought of everything you could possibly want to do when coding. And if they haven’t thought of it, someone else has and has written a plugin you can drop into your plugins directory and it “just works.” Speaking of plugins, vim.org’s plugin list seems neverending. I was half expecting to see most plugins have a final mod date of 2002 or something, but a good portion have newer version released within the past two weeks. It seems the ones that are from 2002 never get updated because they’re mostly perfect. Excellent.

I do miss a few things though. First off, the project file list most editors have on the left side. I installed NERDTree to alleviate that pain, but honestly it’s not the same as having my right click menus and pretty icons. I’m slowly getting used to it though. The nice thing about a text-only file tree is that in those instances where you only have shell access and need to do some coding, there isn’t a dependency on a GUI.

Tabs are another thing I miss. Gvim has tabs, but they aren’t one tab == one file (aka “buffer”) like most editors. You can hack it to do this, sort of, but it works really jenky. Instead I’m using MiniBufExplorer, which takes away some of the pain. I actually hacked it a bit because I didn’t like the way it displays the tabs, which gave me a chance to look at some real vim script. It’s mostly readable to someone who’s never touched it before.

That about does it for my rant. Vim is fast, free, customizable, extendable, scriptable, portable, wonderful, etc…and I’ve barely scratched the surface.

A Google tech writer recently accused Twitter engineer of sexual assault on her blog, and given the responses shot at both sides (Noirin Shirley, accuser, and Florian Leibert, accused) I thought I’d inject my personal thoughts on both the actual report given by Noirin and the responses to the incident.

First off, it’s a big deal to make an accusation like this. Careers hang in the balance, and blah blah blah, we’ve all heard this already. That said, a lot of women are sexually assaulted and never mention it. A lot tell a few people and it never goes anywhere. A lot try to get help but it never comes.

I think it’s not only amazing, but brave that Noirin had the guts to stand up to her assaulter and accuse him in public. It takes brass balls to do this. It also takes brass balls to do this knowing full well the responses you’re going to get because of it. I’m not one to not take stands on things, so I will say I think she’s awesome. I’m sick of women getting pushed around and there being no consequences for the men doing it.

I also do know that women make false accusations, but in my experience the ones who do so have a history of doing so and don’t start off doing it later on in life.

Now, at least one publication is saying that although it’s great to be public about this matter, it’s not ok to be public about the assailant’s name. I have to disagree. So many assaults go unresolved because it’s hard to prove unless you have a police officer right there watching, or at least 10 witnesses. Something like this wouldn’t hold up in court. It’s important that the person who did it be publicly recognized for his actions, because otherwise there very well may be no consequences, ever.

A lot of people are saying that she should say absolutely nothing until the police investigate and the courts make a decision. I have to wonder if they are batshit insane. First off, the police generally have “more important” things to worry about than “hey sum guy jus touched my privatz,” unfortunately. And without any material evidence, it will never hold up in court. What I’m getting at is that even though I love our justice system here in good old USA, there are many things that will fall through the cracks. Does Noirin really need the police or court system to validate what she experienced that night? That’s fucking insane! She knows what happened better than the police or courts, and has every right to talk about it. Plus, she’s opening herself up to a world of legal trouble by doing this, which is just one more reason she’s brave for doing it (and one more incentive to NOT do it falsely).

Let me put it this way: If somebody assaults you, you have the right to fucking let the world know who did it and what happened! Just because it won’t hold up in court (and believe me, it won’t) doesn’t mean it didn’t happen, and doesn’t mean the assailant shouldn’t suffer the social consequences. If a rape happens in the woods and nobody is there to witness it, did it happen? The courts, rightfully so, say “No.” But it still happened, and the aggressor needs to pay for it in some way.

If she lied about it, then that’s another issue entirely. If it did happen, as she said it did, then good for her for letting the world know and making the world that much safer for women.

Either way, there are some very good counter-arguments and discussion on the reddit comments page for the post, which I spent a good amount of time reading before making this post.

Now, there’s no inherent problem with multicast. It’s generally known for being unreliable, but when all you’re sending is “Hi!” over the wire, data integrity isn’t a high priority. The problem with multicast in reality is that most “cloud” (VPS) providers (AWS, Linode, Slicehost, Rackspace, etc) don’t support it on their networks. You can send a multicast message to a group, but your other machines listening on that group won’t hear it. The other problem with multicast is that the failover tools mentioned above ONLY support multicast. There is no way to tell them to listen to another machine directly over unicast, which is supported by cloud providers.

One way you can solve this is by using GRE tunnels, which allow you to create a tunnel to another computer with everything inside encrypted. This allows multicast communications to pass between two computers, even if the router blocks them normally.

I recently tried to get this set up on my current host, Linode. I was not successful, even with the help of another member who had the same problem (but solved it with GRE). I just could not get two machines to talk to eachother over a GRE tunnel with keepalived.

The solution

I posted my question to serverfault.com in a last resort (video). I’d asked more or less the same question there before, but didn’t get the answer I wanted. This time, I hit a jackpot though.

Willy Tarreau, creator of HAProxy, responded with a patch to keepalived that allows it to communicate over unicast. I applied it, recompiled, set up the new options the patch gives (“vrrp_unicast_bind” &”vrrp_unicast_peer“), and spun it up on both machines.

Yesss!! It works! Stopping HAProxy on the first server made the second machine take the shared IP.

Now, ideally there would be a bunch of machines, namely all my web servers that would be standing by ready to take the shared IP. This patch only allows me two machines. Failover is failover though, when one instance goes down, I get an email and can go in and investigate.  I’d still like to know if there is a way to do failover on a cluster of servers without multicast, but for now this works great.

I did some reading and found that the preg* functions cache up to 4096 regex results in a request. This is the problem…a pretty stupid one too. It would be nice if they made this a configurable option or at least let you turn it off when, say, you are running a regex on a different string every time (why the hell would I run the same regex on the same string twice…isn’t that what variables are for?) Unless I’m misunderstanding and PHP caches the compiled regex (but not its values)…but either way, memory was climbing based on the length of the string.

Since the regex was only looking at the beginning of the string and disregarding the rest (thank god), the fix was easy (although a bit of a hack):

$val = preg_replace('/.../', '', $long_string);

Becomes:

$short_string = substr($long_string, 0, 128);
$val = preg_replace('/.../', '', $short_string);

PHP guys: how about an option to make preg* NOT have memory leaks =).

It seems that since widespread adoption of the internet, although more and more people have become seemingly connected through social networking and other mediums, people are drifting further and further apart. A friend is no longer a friend. A real friend is now what a friend was, and a friend is someone you say “damn we haven’t talked in years, how r u?” to. Communities are popping up everywhere online that replace the communities around us physically.

This in itself I don’t feel is bad. A lot of people who never would have met are meeting and sharing new ideas. Information spreads more rapidly. Cultural consciousness is more global, which in most cases is a very good thing.

I think things start to go wrong when people get addicted to this information overload though. They use it as a fuel for everyday distraction, a replacement for the communities they live in, and a tool to deliver opinions and beliefs to them when they would have otherwise had to think (although this last item is true of most media).

Also, it’s one thing to not be in front of someone when you talk to them. A voice conversation can have emotion and depth, but it can also be quick and effortless. The fact that it’s being replaced by one-off messages that are 100% ignorable and have no real content to them is kind of sickening. I’ve heard arguments that “I text someone when it doesn’t make sense to have a whole conversation,” but I’ll see the same person texting back and forth with someone for 10 minutes straight. Or a text is delivered and the person who sent it squirms in anticipation for the reply, which may never come.

What’s wrong with a phone call? Granted, if you’re in a bar and it’s very loud, texting would be appropriate. If you call someone and they don’t pick up, either they don’t want to talk or, god forbid, they aren’t right next to their phone all times of the day. If you want to talk to someone, just call them. I don’t believe texting is a viable replacement for what was the last string of human contact we had.

That all said, I know it’s a giant ball and it rolls where it rolls and there’s no stopping it. There’s no problem with being aware of things that are going on around us though. I feel like each time a real connection between two real people is replaced with something artificial, our culture as a whole goes just a little bit more insane. I’m interested to see how this all pans out, mainly because I don’t have a whole lot of attachment to what our culture is now.

The tool is called monit, and I’m surprised I ever lived without it. Not only does it monitor your services and keep them running, it can restart them if they fail, use too much memory/cpu, stop responding on a certain port, etc. Not only that, but it will email you every time something happens.

While perusing monit’s site, I saw M/Monit which allows you to monitor monit over web, essentially. The only thing I scratched my head about was that M/Monit uses port 8080 (which is fine) but NginX already uses port 8080, and I wasn’t about to change that, so I opened conf/server.xml and looked for 8080, replaced with 8082 (monit runs on 8081 =)). Then I reconfigured monit to communicate with M/Monit and vice versa, and now I have a kickass process monitor that alerts me when things go wrong, and also sends updates to a service that allows me to monitor the monitor.

I can’t look at things like queries/sec as I can with Cacti (which is awesome but a little clunky) but I can see which important services are running on each of my servers, and even restart them if I need to straight from M/Monit. The free download license allows to use M/Monit on one server, which is all I need anyway.

Great job monit team, you have gone above and beyond.

• Any server can go down. Assuming you have more than one replica of your data, you can lose any server in your setup and still be up and running. This can be achieved with replication, but it’s not as easy. You have to have some form of Master-Master replication, perhaps with DRDB, and some form of failover (usually heartbeat).
• Your data  set scales. If you start running out of disk space with a cluster, just add a few more data nodes and your data will be spread out over them. With replication, each replicated server has to have enough storage to fit the entire database. That means if your dataset grows too large, you have to either partition (a hack, essentially) or upgrade your servers.
• Your bandwidth scales. With a cluster, if you are running out of bandwidth, you can add more mysqld processes on your www servers or add more data nodes and your bandwidth scales almost linearly. With replication, you can only add so many slaves before your writes are the bottleneck. Then, once again, you have to look into things like circular replication (dangerous) or partitioning your data set (large updates to your app unless you have an insanely good ORM, big infrastructure change).

These are the main points that helped me decide. Historically, with a clustered approach, the entire dataset would have to fit in the memory of all the data nodes, which is somewhat restrictive if the dataset gets too large. Nowadays, the cluster only needs to store indexes in memory, and can store all non-indexed data on disk. There is talk of having completely disk-based store as well.

All that being said, I set up cluster, which was surprisingly easy. I’m not going to go over how to set it up or anything, just read the manual. After some benchmarking with the web API for beeets.com, the cluster setup appeared to be running about the same speed as the InnoDB setup when testing various commands…a pleasant surprise. It also appeared to handle concurrency a bit better.

Obviously once the dataset grows past a few megs and the traffic bumps up, we’ll revisit the benchmarking, but my hope is that what cluster loses in speed from your everyday general query, it gains in speed by having ability for higher concurrency.

I had a few problems though. In PHP (just PHP, and nothing else) hosts were not resolving. The linux OS was resolving hosts just fine, but PHP couldn’t. It was frustrating. Also, I was unable to sudo. I kept checking permissions on all my files in /etc, rebooting, checking again, etc.

The fix

Then I looked again. /etc itself was owned by andrew:users. Huh? I changed permissions back root:root, chmod 755. Everything works. Now some background.

A while back, I wrote some software (bash + php) that makes it insanely easy to install software to several servers at once, and sync configurations for different sets of servers. It’s called “ssync.” It’s not ready for release yet, but I can say without it, I’d have about 10% of the work done that I’d finished already. Ssync is a command-line utility that lets you set up servers (host, internal ip, external ip) and create groups. Each group has a set of install scripts and configuration files that can be synced to /etc. The configuration files are PHP scriptable, so instead of, say, adding all my hosts by hand to the /etc/hosts file, I can just loop over all servers in the group and add them automatically. Same with my www group, I can add a server to the “www” group in ssync, and all of a sudden the HAproxy config knows about the server.

Here’s the problem. When ssync was sending configuration files to /etc on remote servers, it was also setting permissions on those files (and folders) by default. This was because I was using -vaz, which attempts to preserve ownership, groupship, and permissions from the source (not good). I added some new params (so now it’s “-vaz –no-p –no-g –no-o”). Completely fixed it.

At the time, I thought “What’s the point of keep-alive for front-end? By the time the user navigates to the next page of your site, the timeout has expired, meaning a connection was left open for nothing.” This assumed that a user downloads the HTML for a site, and doesn’t download anything else until their next page request. I forgot about how some websites actually have things other than HTML, namely images, CSS, javascript, etc.

Well in a recent “omg I want everything 2x faster” frenzy, I decided for once to focus on the front-end. On beeets, we’re already using S3 with CloudFront (a CDN), aggressive HTTP caching, etc. I decided to try the latest HAProxy (1.4.4) with keep-alive.

I got it, compiled it, reconfigured:

defaults
	...
	option httpclose

became:
defaults
	...
	timeout client  5000
	option http-server-close

Easy enough…that tells HAProxy to close the server-side connection, but leave the client connection open for 5 seconds.

Well, a quick test and site load times were down by a little less than half…from about 1.1s client load time (empty cache) to 0.6s. An almost instant benefit. How does this work?

Normally, your browser hits the site. It requests /page.html, and the server says “here u go, lol” and closes the connection. Your browser reads page.html and says “hay wait, I need site.css too.” It opens a new connection and the web server hands the browser site.css and closes the connection. The browser then says “darn, I need omfg.js.” It opens another connection, and the server rolls its eyes, sighs, and hands it omfg.js.

That’s three connections, with high latency each, your browser made to the server. Connection latency is something that, no matter how hard you try, you cannot control…and there is a certain amount of latency for each of the connections your browser opens. Let’s say you have a connection latency of 200ms (not uncommon)…that’s 600ms you just waited to load a very minimal HTML page.

There is hope though…instead of trying to lower latency, you can open fewer connections. This is where keep-alive comes in.

With the new version of HAProxy, your browser says “hai, give me /page.html, but keep the connection open plz!” The web server hands over page.html and holds the connection open. The browser reads all the files it needs from page.html (site.css and omfg.js) and requests them over the connection that’s already open. The server keeps this connection open until the client closes it or until the timeout is reached (5 seconds, using the above config). In this case, the latency is a little over 200ms, the total time to load the page 200ms + the download time of the files (usually less than the latency).

So with keep-alive, you just turned a 650ms page-load time into a 250ms page-load time… a much larger margin than any sort of back-end tweaking you can do. Keep in mind most servers already support keep-alive…but I’m compelled to write about it because I use HAProxy and it’s now fully implemented.

Also keep in mind that the above scenario isn’t necessarily correct. Most browsers will open up to 6 concurrent connections to a single domain when loading a page, but you also have to factor in the fact that the browser blocks downloads when it encounters a javascript include, and then attempts to download and run the javascript before continuing the page load.

So although your connection latency with multiple requests goes down with keep-alive, you won’t get a 300% speed boost, more likely a 100% speed boost depending on how many scripts are loading in your page along with any other elements…100% is a LOT though.

So for most of us webmasters, keep-alive is a wonderful thing (assuming it has sane limits and timeouts). It can really save a lot of page load time on the front-end, which is where users spend the most of their time waiting. But if you happen to have a website that’s only HTML, keep-alive won’t do you much good =).

Everyone (who’s anyone) knows that gzipping your content is a great way to reduce download time for your users. It can cut the size of html, css, and javascript by about 60-90%. Everyone also knows that gzipping can be very cpu intensive. Not anymore.

I just installed nginx’s Gzip Static Module (compile nginx with –with-http_gzip_static_module) on beeets.com. It allows you to pre-cache your gzip files. What?

Let’s say you have the file /css/beeets.css. When a request for beeets.css comes through. the static gzip module will look for /css/beeets.css.gz. If it finds it, it will serve that file as gzipped content. This allows you to gzip your static files using the highest compression ratio (gzip -9) when deploying your site. Nginx then has absolutely no work to do besides serving the static gzip file (it’s very good at serving static content).

Wherever you have a gzip section in your nginx config, you can do:

gzip_static on;

That’s it. Note that you will have to create the .gz versions of the files yourself, and it’s mentioned in the docs that it’s better if the original and the .gz files have the same timestamp; so it may be a good idea to “touch” the files after both are created. It’s also a good idea to turn the gzip compression down (gzip_comp_level 1..3). This will minimally compress dynamic content without putting too much strain on the server.

This is a great way to get the best of both worlds: gzipping (faster downloads) without the extra load on the server. Once again, nginx pulls through as the best thing since multi-cellular life. Keep in mind that this only works on static content (css, javascript, etc etc). Dynamic pages can and should be gzipped, but with a lower compression ratio to keep load off the server.

That’s the stupidest thing I’ve ever seen. What kind of business limits the usage of its products to upstanding citizens only? Last I checked it was the government’s job to impose its views on businesses, not businesses imposing their views on their customers.

I have to say, it’s nice that someone is using their business to take a stand, I guess I’d just prefer it to be in defense of free speech and expression. Sure ALL porn is smutty and violent, but that’s expression in itself. Also, can you fight basic human nature? Perhaps, on a personal level. Repression of sexual tendencies is a lot different than acceptance and non-action though. My point is that pornography is the one place where sexual fantasies are allowed to exist in any way shape and/or form, and Americans, being extremely sexually self-repressed, need that outlet, not more repression.

I also think it’s funny when someone tries to be exclusive because they’re SO awesome when someone else is doing it way better…