Hello All, an update from Alex/Snipa:

I've landed in vegas and we're setup in the house to start building out Goo 3.0  At this time, s0up is making massive strides in the re-write of the site to make it more web two point oh-ish-kinda-sorta-nastyness?  I'm rewriting the main file indexer, and DrMacinysha is handling all of the questions/comments blipped to our support e-mail, and is working as our public face.

Oh yeah, and we've got 24 2Tb hard-drives, and 10 server chassis' being built downstairs in my house.  So, lets go over the plan as it stands now, so everyone can see where their donations went, and why we're looking for one last solid push.

Goo's revamp is a full re-imagining of our stack, from top to bottom with the core goals of stability, and making those donator accounts worth something.  Right now, as anyone who has followed this project can attest, goo has grown from it's roots as a simple mirror of the Google Apps package, into a full fledged cornerstone of the community, that's been slowly crumbling because it was slapped together.  It's current home in Dallas has represented it's single longest stay in a single location.  We've moved from Michigan, to Phoenix (IOFlood is lovely, and we still keep our build server, and my personal dedi there!), to Chicago (FDC servers, didn't work well for us :(), Back to Phoenix, and the great team at IOFlood, and finally to Dallas, our current location.  As many remember, we started out on a small 400mbit pipe, and we're currently sitting on 2gbps.  Our goals have changed over time, as we found out what worked and didn't work for us.  Our system is designed around the flexibility of networked file systems, with the ability to move files to/from a central location though some hardlinks and linux/MySQL trickery.  The past couple of weeks have shown us the issue with our current implementation, which is Lustre, and it's not really Lustre's fault, but our fault for creating an environment it wasn't designed to run in.  As anyone in the HPC industry can tell you, Lustre is a really, really great FS for huge deployments with lots of hardware.  For us, it was handy because it was fast, and could be upgraded later.

So, now that we find our lynchpin, our filesystem, how do we fix this?

Distribution across nodes works extremely well for our file set, and upon further review of the available FS' out there, a very clear contender came into view, which supported everything we needed, including self-repair of lost files though multiple copies, the ability to pull data from multiple storage nodes for speed purposes, and the ability to run everything at wire-speed on gige connections, without coming into high CPU/memory contention.  Goo is being rebuilt around Ceph's FS, and we're going to be taking advantage of every last thing we can manage to make this as high performance as possible.  Our current layout involves the following:

2x PHP processing servers - This has been a major bottleneck in the past, where our web servers would kill everything in the name of keeping PHP running, so we're going to give PHP 2 servers to play with.

1x Frontend Web Server - This is to handle all of the static images/js/etc for the site, and handle SSL offloading (Yes, we're finally putting SSL on the login and user admin pages, sorry it's taken so long!)

6x Storage/Distro Nodes - We're rebuilding these around dual quad xeons with 16 Gb of ram each.  We'll be dedicating a single processor to the Ceph FS handling for 1 core/drive, then 4 threads for Lighttpd, letting us reach our theoretical maximum of 1gbit/server.

1x Frontend Node - This is the server that everyone knows as upload.goo.im, our public upload server, this will be turned into a simple frontend box, which is also likely to become the core of the new build system over time, but more on that later.

With these 10 servers, and a couple of switches we need to buy still, we'll be looking at revamping the entire goo-stack into something that we can expand easily out to 10gbps, which is what kind of connection we're going to be getting.  We're faced with the very simple matter that Goo itself is a large entity, and it needs to be treated as such, so we're going to be handling it the proper way from this point forwards.  Our goal now is to help our users understand what kind of transitional period that goo is going though, it's a bit like a teenager starting to rebel.  It's not responsible enough to live on it's own, but it wants it so bad.

So what does this mean?  The 6 servers are going to be split into donator/non-donator, with the latter being lower priority all around, we're looking at the possibility of speed-limits on the non-donator, but set high enough that it really wouldn't be a burden to anyone, simply so there's a solid difference between donator and non-donator.  The biggest difference will be that donators will find themselves on less-crowded servers, and able to get higher speeds due to this.  Donator servers are also going to have additional RAM to handle more files in cache for popular downloads.

Our monthly hosting costs at the moment are in the 4000-5000$/mo range, due to extra power for our older servers, the high amount of bandwidth we use, etc.  We're looking at moving to 5000-6500$/mo, and thats where we need help.  For one month, we're looking at space rental costs in excess of 10,000$ for a single month, while we do the final transfer and DNS/IP swaps, so we're looking for one final push to get us over the edge.  We'll be digging into our own pockets to help make sure we get there, and we're sorry we have to keep asking, but this is the last move for goo for the foreseeable future, and it will help re-cement us into the cornerstone that we were.

If you have comments/suggestions for the team, like suggestions for what to do to help differentiate between a donator and a non-donator, feel free to drop us a mail at contact@snipanet.com