CentOS - Oct 2014 - Filesystem writes unexpectedly slow (CentOS 6.4)

I have a rather large box (2x8-core Xeon, 96GB RAM) where I have a couple of 
disk arrays connected on an Areca controller. I just added a new external array,
8 3TB drives in RAID5, and the testing I'm doing right now is on this array,
but
this seems to be a problem on this machine in general, on all file systems 
(even, possibly, NFS, but I'm not sure about that one yet).

So, if I use iozone -a to test write speeds on the raw device, I get results in 
the 500-800MB/sec range, depending on write sizes, which is about what I'd
expect.

However, when I have an ext4 filesystem on this device, mounted with noatime and
data=writeback, (the filesystem is completely empty) and I test with dd, the 
results are less encouraging:

dd bs=1M if=/dev/zero of=/Volumes/data_10-2/test.bin count=40000
40000+0 records in
40000+0 records out
41943040000 bytes (42 GB) copied, 292.288 s, 143 MB/s

Now, I'm not expecting to get the raw device speeds, but this seems at least
to
be 2-3 times slower than what I'd expect.

Using conv=fsync oflag=direct makes it utterly pathetic:

dd bs=1M if=/dev/zero of=/Volumes/data_10-2/test.bin oflag=direct conv=fsync 
count=5000
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 178.791 s, 29.3 MB/s

Now, I'm sure there can be many reasons for this, but I wonder where I
should
start looking to debug this.

-- 
Joakim Ziegler  -  Supervisor de postproducci?n  -  Terminal
joakim at terminalmx.com   -   044 55 2971 8514   -   5264 0864

On 2014-10-14, Joakim Ziegler <joakim at terminalmx.com>
wrote:
>
> So, if I use iozone -a to test write speeds on the raw device, I get
results in
> the 500-800MB/sec range, depending on write sizes, which is about what
I'd expect.
>
> However, when I have an ext4 filesystem on this device, mounted with
noatime and
> data=writeback, (the filesystem is completely empty) and I test with dd,
the
> results are less encouraging:
My first question would be, why not test the filesystem with iozone too?
(And/or, test the device with dd.)  You may or may not come up with the
same results, but at least someone can't come back and blame your
testing methodology for the odd results.

(Just as an aside, if your 6.4 box is on a public network, you should
probably consider updating it as well, since many security and bug fixes
have been issued since 6.4 was released.)

If you are still getting poor results from ext4, you have at least two
more options.

==Check with the ext4 mailing list; they're usually pretty helpful.
==Try your tests against xfs.  Try to make sure your tests are
replicating your use cases as closely as you can manage; you wouldn't
want to pick a filesystem based on a test that doesn't actually
replicate how you're going to use the fs.

--keith


-- 
kkeller at wombat.san-francisco.ca.us

On 10/14/2014 02:15 PM, Joakim Ziegler wrote:
> I have a rather large box (2x8-core Xeon, 96GB RAM) where I have a
> couple of disk arrays connected on an Areca controller. I just added a
> new external array, 8 3TB drives in RAID5, and the testing I'm doing
> right now is on this array, but this seems to be a problem on this
> machine in general, on all file systems (even, possibly, NFS, but I'm
> not sure about that one yet).
The first thing I would check is that you have a BBU installed on the
areca controller and that it is functioning properly (check the cli, I
don't know the exact commands off the top of my head), also make sure
that write caching is enabled on the controller (after you've checked
the BBU, of course).  Without a working BBU in place hardware RAID
controllers, such as areca, disable write caching (by default) and this
will have a significant impact on write speeds.

Note that newer controllers use a type of flash memory instead of a BBU.


Peter

On Mon, 13 Oct 2014 20:15:11 -0500
Joakim Ziegler <joakim at terminalmx.com> wrote:

...
> So, if I use iozone -a to test write speeds on the raw device, I get
> results in the 500-800MB/sec range, depending on write sizes, which
> is about what I'd expect.
> 
> However, when I have an ext4 filesystem on this device, mounted with
> noatime and data=writeback, (the filesystem is completely empty) and
> I test with dd, the results are less encouraging:
...
> Now, I'm sure there can be many reasons for this, but I wonder where
> I should start looking to debug this.
 
First I'd suggest comparing apples to apples. That is try doing the dd
test on the raw device and compare to dd on ext4.

Then you may want to try changing io scheduler from the default cfq to
deadline. This typically works better for many raid controllers but
ymmv.

Also testing with xfs instead of ext4 is probably worth it. xfs usually
outperform ext4 in streaming writes (like dd). Of course this raises
the question of whether that dd is a useful metric for your actual
load... xfs may infact be needed (3T * 7 = 21 TB > ext4 max (if I
remember correctly, refer to rh online data for rhel6 to make sure)).

Good luck,
 Peter K