When I did my Red Hat Certified Engineer training, the second time around (when I updated from Red Hat 7.1 to Enterprise Linux 3), the trainer mentioned this Ext3 vs Reiserfs benchmark that Gurulabs performed. Probably because it had such a memorable name, I've remembered it and referred people to it, and referred to it myself, a few times over the years since.
So it seemed only fair to try out their benchmarking methods (i.e. use NetApp's postmark with a few different settings) on the uber-disk as well, while I had it idle.
So I did.
Firstly, I ran it locally on minotaur, the piddling little 866Mhz Pentium III, with 320Mb of RAM (but with 1.5Tb of disk attached directly via USB), and then I ran the exact same tests from teevee, the box destined to become my MythTV server (3.0Ghz Pentium 4, 1Gb of RAM), with the disks mounted with ATA over Ethernet. This was with a JFS filesystem on a single logical volume striped across four physical volumes.
I ran the tests three times, just like the Gurulabs benchmark, first time around, it was with:
- 50,000 transactions (set transactions 50000)
- 2,000 simultaneous files (set number 2000)
- files in the range of 1,000 to 9,000 bytes (set size 1000 9000)
The idea being that this will generate around 150Mb of data, which should all fit in the RAM (of either system)
The second time around was with:
- 50,000 transactions
- 2,000 simultaneous files
- files in the range of 1,000 to 90,000 bytes
The idea being that this, generating 1400Mb of data, will exceed the RAM on both systems (but not uniformally, obviously).
The final run was a total thrash with:
- 200,000 transactions
- 4,000 simultaneous files
- files in the range of 1,000 to 300,000 bytes
This one causes about 19Gb of data to be written, and absolutely hammers the living daylights out of the disks (or the network, and the disks, in the case of ATAoE).
So, now for the results:
- Test case 1, with locally attached disks
-
Time: 22 seconds total 21 seconds of transactions (2380 per second) Files: 27170 created (1235 per second) Creation alone: 2000 files (2000 per second) Mixed with transactions: 25170 files (1198 per second) 24986 read (1189 per second) 24697 appended (1176 per second) 27170 deleted (1235 per second) Deletion alone: 2340 files (2340 per second) Mixed with transactions: 24830 files (1182 per second) Data: 149.53 megabytes read (6.80 megabytes per second) 161.85 megabytes written (7.36 megabytes per second) - Test case 1, with ATAoE attached disks
-
Time: 27 seconds total 25 seconds of transactions (2000 per second) Files: 27170 created (1006 per second) Creation alone: 2000 files (2000 per second) Mixed with transactions: 25170 files (1006 per second) 24986 read (999 per second) 24697 appended (987 per second) 27170 deleted (1006 per second) Deletion alone: 2340 files (2340 per second) Mixed with transactions: 24830 files (993 per second) Data: 149.53 megabytes read (5.54 megabytes per second) 161.85 megabytes written (5.99 megabytes per second) - Findings
-
A degradation in performance, sure, but not a huge one.
- Test case 2, with locally attached disks
-
Time: 218 seconds total 209 seconds of transactions (239 per second) Files: 27144 created (124 per second) Creation alone: 2000 files (400 per second) Mixed with transactions: 25144 files (120 per second) 24830 read (118 per second) 25094 appended (120 per second) 27144 deleted (124 per second) Deletion alone: 2288 files (572 per second) Mixed with transactions: 24856 files (118 per second) Data: 1405.24 megabytes read (6.45 megabytes per second) 1535.91 megabytes written (7.05 megabytes per second) - Test case 2, with ATAoE attached disks
-
Time: 165 seconds total 158 seconds of transactions (316 per second) Files: 27144 created (164 per second) Creation alone: 2000 files (1000 per second) Mixed with transactions: 25144 files (159 per second) 24830 read (157 per second) 25094 appended (158 per second) 27144 deleted (164 per second) Deletion alone: 2288 files (457 per second) Mixed with transactions: 24856 files (157 per second) Data: 1405.24 megabytes read (8.52 megabytes per second) 1535.91 megabytes written (9.31 megabytes per second) - Findings
-
Whoa, who'd have thought this? Better performance over the wire? I'm blaming the fact that minotaur is a bit of a weakling compared to teevee, and teevee had a better caching advantage with more RAM. Mind you, you'd expect to see the same for the first test as well, unless size is a factor. Inconclusive results I guess.
- Test case 3, with locally attached disks
-
Time: 7356 seconds total 7310 seconds of transactions (27 per second) Files: 104099 created (14 per second) Creation alone: 4000 files (117 per second) Mixed with transactions: 100099 files (13 per second) 100246 read (13 per second) 99560 appended (13 per second) 104099 deleted (14 per second) Deletion alone: 4198 files (349 per second) Mixed with transactions: 99901 files (13 per second) Data: 18995.06 megabytes read (2.58 megabytes per second) 19738.85 megabytes written (2.68 megabytes per second) - Test case 3, with ATAoE attached disks
-
Time: 12320 seconds total 12111 seconds of transactions (16 per second) Files: 104099 created (8 per second) Creation alone: 4000 files (32 per second) Mixed with transactions: 100099 files (8 per second) 100246 read (8 per second) 99560 appended (8 per second) 104099 deleted (8 per second) Deletion alone: 4198 files (49 per second) Mixed with transactions: 99901 files (8 per second) Data: 18995.06 megabytes read (1.54 megabytes per second) 19738.85 megabytes written (1.60 megabytes per second) - Findings
-
So presumably the network latency made its presence felt with data volumes of this magnitude.
