Benchmarking SSD with MongoDB and CouchDB, Part 4

As noted in part 1, benchmarking is a black art. Most of the time it is unclear, what you want to measure and what you are measuring.

Setting Up a Durable Document Store

For today’s test, I want to investigate the following setup: I need a document store, which guarantees the client that the data is durably stored. This rules out MongoDB and leaves CouchDB as a good candidate because you can configure CouchDB to fsync data to disk before answering the client.

The File-System

The first obstacle is the file-system. It is clear from part 3 that there are huge performance differences between the different file systems and configurations. In order to get an reliable answer from the file-system, when data has been committed to disk, I chose the following setup:

  • ext4 with journal
  • barrier enabled
  • journal_data enabled

This should – in principle – guarantee that data is acknowledge after it has been written to disk. However, it seems that at least the ADATA is somewhat ignoring the barriers. So, I’ve used our OCZ and the Western Digital VelociRaptor hard-disk.

In order to prepare the disk, format it using ext4 and then – before mounting – issue as root:

tune2fs -O ‘has_journal’ -o ‘^nobarrier,journal_data’ /dev/sdXX

The FSync-Test

The first tests uses the fsync-bench program to establish a base-line. Using the hard-disk

./fsync-bench –no-dots  /raptor/testfile 4096 100
insert time: 1.497353 sec for 100 documents (66.784519 docs / sec, 0.014974 secs / doc)
273549.390157 bytes / sec, 0.260877 mbyte / sec

Using the OCZ

./fsync-bench –no-dots  /samsung/testfile 4096 1000
insert time: 1.268481 sec for 1000 documents (788.344484 docs / sec, 0.001268 secs / doc)
3229059.008373 bytes / sec, 3.079471 mbyte / sec

So the score is

768 docs/sec versus 67 docs/sec

The SSD is a factor of 11 faster. The ADATA gives 2300 docs/sec – but it makes no difference if you switch on or off the barrier. The OCZ without barriers is comparable to the ADATA. This leaves the question if the ADATA is really honoring the barriers. Our brand-new Samsung SSD 830 can write 210 docs/sec with barriers and 2200 docs/sec without barriers. Which is a bit disappointing.

The CouchDB

Part 1 mentioned the insert benchmark by Felix Geisendörfer. I’m now going to rerun this benchmark on both disks. But first of all, we need to configure CouchDB in such a way that it commits each POST to disk. Open the configuration file and make sure that it contains a line

delayed_commits = false

This will ensure that each document is written to disk before the client gets an acknowledgment. Then start Felix’s tests using the VelociRaptor as storage:

> php insert-bench.php

-> single_insert 1000 docs:
doc count (before compact): 1000
doc count (after compact): 1000
insert time: 57.9287 sec
insert time / doc: 57.93 ms
insert doc / sec: 17.26
compact time: 1.0064 sec
compact time / doc: 1.01 ms

Using the OCZ as storage:

> php insert-bench.php

-> single_insert 1000 docs:
doc count (before compact): 1000
doc count (after compact): 1000
insert time: 6.5044 sec
insert time / doc: 6.5 ms
insert doc / sec: 153.74
compact time: 1.0095 sec
compact time / doc: 1.01 ms

So the score is

154 docs/sec versus 17 docs/sec

CouchDB on an SSD is a factor of 9 faster. Which is in line with the fsync test.

Apache Bench

Felix’s tests only uses a single connection. CouchDB speaks HTTP, so it is possible using ab2  to simulate parallel client access. I’ve checked in a shell script append-only.sh to generate documents in parallel.

For the raptor this gives:

concurrency docs/sec
1 11
2 19
4 37
8 72
16 141
32 292
64 565
128 1087
256 1797

For the OCZ:

concurrency docs/sec
1 21
2 42
4 83
8 170
16 321
32 608
64 1194
128 2090
256 3824

So there are two mysteries here:

  1. Why is the Apache Bench slower when using concurrency 1?
  2. Why is the Apache Bench much faster than the fsync-bench when using concurrency 256?

To be continued….