The Life and Times of Michael Shadle

Blog | Projects | Netflix

Large File Uploads Over HTTP - The Final Solution (I Think)

Tuesday, August 26th, 2008 at 8:18 pm

Problem statement: HTTP sucks for file uploads.

You know it. I know it. The problems?

What would the ideal file upload experience support?

With all this in mind, I somehow stumbled across the idea (roughly posted here) based on the time-tested learnings from Usenet and NZB files, and BitTorrent. The main idea? Splitting the file up into manageable segments. There's also some other logic too, but that's the main idea.

Why do I claim this is the final solution?

What's required, how does it work?

As of right now, this is what I have down (it has changed already since the PHP post):

  1. The client contacts the server to begin the transaction. It supplies the following information:
    • Action = "begin"
    • Final filesize
    • Final filename
    • Final file hash (SHA256 or MD5, still haven't determined which one)
    • A list of all the segments - their segment id, byte size, hash (again SHA256 or MD5) - XML or JSON or something
  2. Server sends back "server ready, here's your $transaction_id"
  3. Client starts sending the file, one segment at a time, with the following information:
    • Action = "process"
    • Transaction ID = $transaction_id
    • Segment ID = $segment_id
    • Content = base64 or uuencoded segment (for safe transit)
  4. Server replies back "segment received, transaction id $transaction_id, segment id $segment_id, checksum $checksum"
  5. Client compares the checksum for $segment_id, if it matches, move on to the next segment. If not, retransmit.
  6. When the client is done sending all the segments, client sends message to the server:
    • Action = "finish"
    • Transaction ID = $transaction_id
  7. Server assembles all the segments (if they're separate) and sends to the client:
    • Transaction ID = $transaction_id
    • Checksum = $checksum of the final file
  8. Client compares the checksum to it's own checksum. If it matches, client sends message to server:
    • Action = "complete"
    • Transaction ID = $transaction_id

Viola, done. I think the "protocol" transmits some extra information that isn't needed; so some of this might need to be cleaned up. This is the initial idea though. Props to Newzbin for inventing NZB files which was a big influence in this concept.

I'm somewhat rushing this post out, hopefully it solicits some feedback. I'm going to be revising this and working with a Java developer to work on a client written in Java. Hopefully someday we'll get one with less overhead. I'll post PHP code as I write it too to handle the server portion of it.

Posted in Development | 1 Comment »


Updates on Home Storage Solutions

Sunday, August 17th, 2008 at 2:25 am

For a while I was looking into and hoping to go the eSATA route. Immature chipsets and lack of OS support have somewhat kept that idea frozen.

I want to use ZFS for a filesystem. Or a filesystem -like- ZFS. Currently there are no others out there like it. There is Btrfs, and there is another one I thought picking up steam (although I can't remember it now for the life of me) - both of those however aren't stable yet. ZFS is still not as stable as I wish on FreeBSD. It won't run natively on Linux, and I don't think it's very stable either. The only true way to get a stable filesystem like ZFS is to in fact run ZFS on Solaris.

I was not excited to try Solaris. It used to be a joke to call it "Slowaris" - I remember the old days of using random UNIX shells and hating Solaris boxes because I couldn't run hardly anything or compile anything. However, that's changed somewhat now. I took the plunge and installed SXCE (Nevada build 94) since Solaris 10u5 did not support the new CIFS implementation. So far, I've learned a little bit here and there about Solaris system administration and I've been using ZFS to create some snapshots, filesystems, etc. It is so easy even my mom could handle it. Not to mention Solaris has some pretty neat tools like the Solaris Fault Manager, which I have crontabbed to run every 30 minutes and email me if -any- hardware/faults get reported. So I have this great box sitting there running the best filesystem possible integrity-wise, and it is also damn quiet. It's not a small form factor which I would have liked, though.

So I begin looking into trying to get a small form factor ZFS box. I might be able to, if I want to hack up a Shuttle style case (see Udat at Mashie Design - there are mini-itx motherboards now with 6 SATA ports onboard which would allow for a 5 drive RAIDZ1 + maybe use a Compact Flash card for a boot drive. However, that requires case modding and can only fit 5 drives. I'm not sure I really want to try all that.

Instead, even Mashie himself has admitted to moving into larger form factors for storage boxes (I believe he's using a CM Stacker nowadays) - and from a space perspective, it probably does make the most sense.

Currently I'm exploring going with a full-size case that could hold 15 drives, or a mid-size case that could hold 10 drives (not including optical + boot)

I think I may have found a winner, for the mid-size option. Lian-li has a self-proclaimed "silent" chassis that has 9 bays (which means 6 bays for 2x5-in-3 modules) + optical + 1 boot disk in the spare 5.25" bay. Roughly 8-9TB usable in a mid-size case that would be about as silent as it can get. It even has a front door on it. Actually, there's a second place one - this one is much more extensible, but has no door on it, which I think would help shield any noise coming from these 5-in-3 modules. See here. Cooler Master also has a case like that too - again, no door on the front. I wish I had local access to all of these cases to try each of them out. Right now I have to order them online, and then pay possible restocking fees, and at least the cost of shipping the product back. I'm tired of that back and forth game. I've had to do it too many times in the past.

Lian-li also has a full-size chassis that already includes 10 internal drive bays, + 5x 5.25" front bays. Those extra bays could be used for more drives too. So many options... I'm trying to determine the amount of space I want to use in my office and how large and bulky I want these machines to be. Ideally I would like as little CPUs as possible - no need to have full-out operating systems installed and having to manage all that. I'm just tired of all my equipment making noise, getting hot, failing, and data corrupting due to failure or bit-rot. It's time to upgrade and streamline.

I've pretty much examined every chassis at Lian-li, Coolermaster and Antec. I have an Antec P182 right now. It's great and quiet, but does not support as many drives as I want for these next boxes. Stay tuned as I pull the trigger and build another storage box soon. Perhaps I'll share some pictures and specifications with my existing storage box I just built, which is for "off site" daily snapshots of my hosting infrastructure and some other servers I administer.

These next machines will be for my own personal use... and now I've gained some good knowledge on what to expect. It's been a while since I've built a normal-sized machine, as I've been a Shuttle XPC user for years now :)

Posted in Uncategorized | No Comments »


nginx + WordPress - Redux

Sunday, August 17th, 2008 at 1:52 am

There's been a minor tweak required in my original WordPress+nginx rewrite rule post.

I think we've finally got it right now. I've been exchanging emails ad nauseum with Igor and I believe the behavior now works properly (part of the exchange was regarding PHP+basic http auth, there were some bugs around the regexps/nested location blocks or something)

Anyway, here is a perfect working example, running this very website:

server {
   listen 80;
   server_name michaelshadle.com;
   index index.php;
   root /home/mike/web/michaelshadle.com/;
   include /etc/nginx/defaults.conf;
   include /etc/nginx/expires.conf;
   error_page 404 = /wordpress/index.php?q=$request_uri;
   location ^~ /wordpress/wp-admin {
      auth_basic "wordpress";
      auth_basic_user_file /home/mike/web/michaelshadle.com/.htpasswd;
      location ~ \.php$ {
         fastcgi_pass 127.0.0.1:11000;
      }
   }
   location ~ \.php$ {
      fastcgi_pass 127.0.0.1:11000;
   }
}

Likewise, you can also omit the basic HTTP authentication if you don't think you need it:

server {
   listen 80;
   server_name michaelshadle.com;
   index index.php;
   root /home/mike/web/michaelshadle.com/;
   include /etc/nginx/defaults.conf;
   include /etc/nginx/expires.conf;
   error_page 404 = /wordpress/index.php?q=$request_uri;
   location ~ \.php$ {
      fastcgi_pass 127.0.0.1:11000;
   }
}

Note: this does require a patched version of 0.7.10 - I assume he will put these changes into 0.7.11. Some of the changes required include the basic HTTP auth stuff. Also, when using error_page 404, it generated logfile noise in the error log. That was fixed as of 0.7.9 or 0.7.10, so no longer will you receive script-handled 404's in your error log.

This should be the last and final need to run WordPress properly under nginx. This has the approval of Igor, the creator - you cannot get better than that. (Note: this is WordPress 2.6.1, but I have not seen any reason the rewrite rule would be different since even before 2.x)

Let me know if it doesn't work! Or better yet, let the nginx mailing list know :)

Posted in Uncategorized | No Comments »


Simple WordPress Hack - Redirect the index.php!

Saturday, August 16th, 2008 at 1:05 am

I don't know how or when, but I wound up getting indexed with index.php in some of my URLs.

For some reason, WordPress hasn't decided that they should parse and remove that. So for the interim, I've decided to finally throw in a couple quick lines of code to do the trick. Throw it in any plugin you want or make a standalone file, it works fine with my 2.6.1 so far.

add_action('init', 'chop_index');

function chop_index() {
   if(preg_match('/index\.php$/', $_SERVER['REQUEST_URI'])) {
        $url = preg_replace('/index\.php$/', '/', $_SERVER['REQUEST_URI']);
        $url = preg_replace('/\/\/$/', '/', $url);
        header("Location: http://".$_SERVER['HTTP_HOST'].$url, true, 301); exit();
   }
}

Posted in WordPress | No Comments »


Netflix - The Company That Actually Gives A Crap

Wednesday, August 13th, 2008 at 11:53 pm

You know, Netflix is neat. Not only is their website pretty simple and clean, the company itself seems to "do right" by its customers too. Today is the second time they've made me feel warm and fuzzy being a customer. The first time was when they decided to change their rate plans to be cheaper and automatically started charging me less. A lot of companies drop the price on their monthly rates and you have to re-signup or call and complain, etc...

Dear Michael,

Great news! We're lowering the price of your 3 DVDs out at-a-time plan to $16.99 a month plus applicable taxes. Now you can enjoy Netflix for less!

You don't need to do a thing - except pay less. Your membership will automatically move to the lower price and be reflected in your Membership Terms and Details. The lower price will take effect beginning with your statement on or after July 23, 2007.

Membership Terms and Details: http://www.netflix.com/Terms

Your $16.99 plan not only gives you 3 DVDs out at-a-time but you can also watch 17 hours of movies and TV episodes instantly on your PC each month - for no additional charge.

Check it out: http://www.netflix.com/WatchNow

Enjoy!
Your friends at Netflix

Now that is cool. But then they've given me another reason to praise them. They're reporting disc shipment issues, and being nice about it, they've just decided to issue a credit since we're technically losing a few days of service. They didn't have to do that, but they did.

I just received this email today:

We're Sorry DVD Shipments Are Delayed

Dear Michael,

Our shipping system is unexpectedly down. We received a DVD back from you and should have shipped you a DVD, but we likely have not. Our goal is to ship DVDs as soon as possible, and we will keep you posted on the status of your DVD shipments.

We are sorry for any inconvenience this has caused. If your DVD shipment is delayed, we will be issuing a credit to your account in the next few days. You don't need to do anything. The credit will be automatically applied to your next billing statement.

Again, we apologize for the delay and thank you for your understanding. If you need further assistance, please call us at 1-888-638-3549.

-The Netflix Team

Posted in Consumerism | No Comments »


Windows Never Ceases To Amaze...

Sunday, June 15th, 2008 at 11:04 pm

I think I keep my machines pretty well up to date. I'm fixing up my girlfriend's computer right now, and I finally got all the updates applied except apparently one got missed. Okay, simple enough right?

Step #1: Install .NET Framework 1.0, click "Check for updates" - what? there's another update?

Step #2: Install .NET Framework 1.1 Service Pack 1, click "Check for updates" - what? ANOTHER one?

Step #3: Install .NET Framework 1.1 Service Pack 1 security update, click "Check for updates" ...

Really? Why couldn't we have a pre-packaged full install? Why do I have to run this "check for updates" over and over (and why does it take so goddamn long nowadays?)

Okay, looks like that really was the last update and this machine is now 100% up to date. At least for the moment...

Posted in Consumerism, Software | 2 Comments »


RK: Igor Sysoev

Saturday, June 14th, 2008 at 9:41 am

At first I thought Lighttpd was the cat's meow. After talking with Andrei (who maintains PHP-FPM) I thought - if he got PHP-FPM right, he must know his stuff. He recommended I try nginx - and boy am I glad I did.

Igor is an interesting character to me. He is very matter-of-fact, he has no problem issuing patches almost instantly to enhance his product, and he also has no problem being short with people when rejecting an idea or informing them they're wrong. To me it seems like sometimes people who maintain projects try to be more politically correct, but from what I've seen, Igor seems to be extremely technical by heart, and does not really stop to smell the flowers (at least on the mailing list...)

Igor gets mad props for creating nginx - quite possibly the most efficient web/proxy server on earth. For a bit I was using it to proxy 4+ million web requests a day (small php, html, graphics and even larger file and video downloads) through a single server, doing gzip as well - and it handled it all without using more than 14 megs of physical RAM. Nginx is such an engineering feat that I've actually started contemplating how to re-do my architecture since it is totally viable now to have a single frontend server proxying all the dynamic requests to dedicated FastCGI servers. No need really to be running multiple nginx instances anymore... one handles everything!

Anyway, back to Igor - my new goal in life is not only to help promote PHP-FPM but now nginx as well. There's only a couple minor things I wish nginx would do a bit better, but otherwise, it is my web solution and possibly even my proxy solution to replace LVS. Who would have thought a userland daemon could be so efficient? Even though Igor maintains nginx almost 100% by himself, releases come out often, he can produce patches to fix bugs or add features within a couple hours, and he usually replies to emails within the same day on the mailing list. I haven't seen that level of support from any other open source project, much less commercial products. "Want a patch? Wait for our next release in six months!"

Igor's website is at http://sysoev.ru/en/, however it doesn't really have much info on it - and you'll probably be looking for nginx at http://nginx.net/ anyway. Thanks Igor :)

Posted in Respect Knuckles | 3 Comments »


WordPress+nginx Rewrite Rules - Stop The Insanity!

Thursday, May 1st, 2008 at 8:00 am

Note: this has been outdated now. I have an updated post with the latest and Igor-approved method here.

When I was switching over to nginx, I found a handful of random and overkill config examples to make it work. I thought I had found the simplest solution already, but Igor (the creator of nginx) actually gave me an even "better" solution in nginx.

This is assuming WordPress is physically installed in /wordpress, and pages are served up using friendly URLs, /2008/02/03/post-title/ - just like this site. All the rewrites are off the root. I assume you've already got PHP setup to parse properly and you have a working server {} block. I'll post that info too if people really need it.

1st example (seems like overkill, can't figure out a reason why people are splitting up the rewrite rules - I guess because they assume "wp-anything" is all static?):

if (!-e $request_filename) {
      rewrite ^([_0-9a-zA-Z-]+)?(/wp-.*) $2  break;
      rewrite ^([_0-9a-zA-Z-]+)?(/.*\.php)$ $2 last;
      rewrite ^ /index.php last;
}

2nd example (this was as good as I thought it could be, but I was wrong):

if (!-e $request_filename) {
   rewrite ^(.+)$ /wordpress/index.php?q=$1 last;
}

3rd example (this is the best, according to Igor):

error_page 404 = /wordpress/index.php?q=$uri;

(Ref: http://article.gmane.org/gmane.comp.web.nginx.english/4739)

Posted in Software, WordPress | 4 Comments »


Disabling IPV6 in Linux 2.6.24

Saturday, April 19th, 2008 at 2:19 am

In the past, I've always been able to disable IPV6 in Linux with a simple line in any /etc/modprobe.d file:

alias net-pf-10 off

It appears though it has no effect anymore. I had to change it to this line:

blacklist net-pf-10

How annoying.

Posted in Software | 4 Comments »


The (Current) State of (e)SATA Port Multipliers

Monday, April 14th, 2008 at 10:47 pm

For the most part, open source has been good to me. I can run my business on free software that is for the most part better than its commercial counterparts. I'm able to use often-updated and relatively bug-free software in all facets of my life. However, there are a couple things that have been bugging me - one of them has been SATA Port Multipler (PMP, not to be confused with PM - Power Management!) support. 1TB drives hit an all time low last week (at least in my ads) and it got me motivated to check in to the status of PMP support once again. Previously there weren't a lot of people using it and not a lot of feedback or information. Today, I have a bunch of good information, but still not the magic combination, which for me, includes ZFS (as it is the only filesystem out there still to offer all the good checksumming, self-healing, etc.) This post isn't going to talk about why ZFS rocks. Most people already know why. I want a filesystem that has been built for integrity. This is for home archiving, and I'll be damned if bit rot is going to ruin my memories.

My goal was to find an OS that has ZFS support, which currently is between Solaris (or OpenSolaris, Nexenta, etc.) and FreeBSD 7 (which has "experimental" but usable ZFS support) - I'd prefer Linux as it has the fastest rate of bug fixes and driver support and I am already using Linux on one of my home machines, but ZFS doesn't run natively on it (FUSE is not native enough.)

The following below are the current status I am aware of taken from direct emails exchanged with Adriaan de Groot (who said he might port PMP support to FreeBSD 7) and Tejun Heo (who maintains the PMP driver libata-pmp for Linux) and Google searches, mailing list and forum postings as of today.

Windows

I'm not even going to bother here. Most drivers for the controllers are coded for Windows first, and then if we're lucky an open source Linux/etc. version is put out. So Windows support is probably 100% - but you're stuck to using NTFS or something (blech.)

FreeBSD

According to Adriaan:

"Don't bother, as port multipliers are not supported. I had some PMP support hacked in at some point, but without NCQ there's not much (performance) point. It *might* be that Soren has added NCQ / PMP in the meantime, but I'm not aware of it."

Also:

"PMP requires NCQ to be useful; otherwise you end up queuing all the requests for the different disks and performance goes down a fair bit - but like I said, it's *possible* and if all you're after is big storage, it will work. Anyway, it's not the cards' fault: there just isn't any NCQ in the (FreeBSD) kernel."

Tejun's comment:

Although NCQ and PMP supports don't really have to go together. Non-NCQ PMP should work as good as direct non-NCQ. Nothing more to lose there because it's attached via PMP. Maybe they were talking about pushing multiple commands to a port. If the driver can't do that, PMP won't work too nice. 

As of now according to Adriaan, there is no PMP support in FreeBSD (except possibly some highly experimental patches.)

Solaris (and variants)

From Adriaan:

"I run OSOL ... but OSOL also does not support PMP"

Additional links:

OS X

Possibly? I did not look into this. It would be a shame if some were supported with NCQ and not given back to FreeBSD... this guy claims to have what sounds like two 4:1 multipliers on his Mac:

http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg13749.html

Linux

(There is a site that tries to keep up to date with SATA support in the Linux kernel. This is probably the best place to look. As of writing, it still appears up to date. http://linux-ata.org/driver-status.html)

According to Tejun, who was extremely helpful answering my barrage of questions gracefully, it does look like Linux has some PMP options (now if only we had ZFS or BTRFS...!) with the following notes for each chipset:

Controller chipsets: SiI3124, SiI3132 - both will work. His comments:

"3124/3132 controllers are good"

"Performance-wise, all the PMPs on the market should be okay but 3124/3132s have certain limitations on PCI bus side and can't deliver full bandwidth concurrently from all drives attached through PMP. I don't remember the exact numbers but it maxes out after three drives or so."

"I have second gen chips from SIMG and they behave really good."

Note: SiI3124-1 is only 1.5Gbps. SiI3124-2 supports 3Gbps.

Enclosure chipsets: SiI3726, SiI4726, Marvell 88SM4140  - all should work great. His comments:

"3726/4726 PMPs are the first gen PMPs and both are a bit quirky"

"Marvell port multipliers behave really well."

"That said, 3726/4726's work okay too. The only problem is that they always need their first slot occupied to operate correctly. As long as you occupy the first slot, there shouldn't be much problem."

Update: "There have been reports of 3726/4726 PMPs having trouble with 3Gbps link speed under certain configurations. I still don't know what's the exact cause.  It doesn't seem common but if you can maybe trying out with one drive before ordering the whole array is a good idea."

Note: the Marvell is only a 4:1 multiplier, the SiI ones are 5:1.

So the big winner here for the time being is Linux - looks like I will be purchasing a SiI3124-2-based controller (probably this one, which claims it is a 3124A, a successor of the 3124-2) and a SiI3726 or SiI4726 based PMP enclosure. I am trying to determine which one will be the quietest right now. I don't believe that hardware RAID-5 is supported on any PMP, only RAID 0, 1, and 10. It gets confusing because most advertise RAID but it leverages some software RAID stuff. So I will probably using mdadm for RAID5 at home.

Hopefully this is a useful (and correct) summary. I invite anyone with updated information to leave a comment, and I will correct it. Especially if you have experience with eSATA PMP enclosures and have anything to add! Not many people have ventured out into these waters yet, but it looks pretty neat - off a single PCI-X or PCI-e card you could chain 20 drives all with decent bandwidth and only have one cable for each set of five drives...

Posted in Software, Toys | 1 Comment »


« Previous Entries

Entries (RSS) and Comments (RSS). 12 queries. 0.224 seconds.