Cosmin Lehene, has a wonderful pair of posts about how a team in Adobe selected, tested and implemented an HBase based datastore for production use.
Its interesting how much they thought about failure and about the backup for backups. And in spite of all that how things still break. Building something new based on cutting-edge technology is not for the faint hearted. It needs to be supported by the organization, lead by those who can see the future, and backed by a team of experts who are always ready for challenges.