Tales From the Cloudera Field
Kevin O’Dell, Kate Ting, Aleks Shulman
{kevin, kate, aleks}@cloudera.com
?2014 Cloudera, Inc. All rights reserved.
?2014 Cloudera, Inc. All rights reserved.
Who Are We?
Kevin O’Dell
- Previously HBase Support Team Lead
- Currently Systems Engineer with a focus on HBase deployments
Kate Ting
- Technical Account Manager of Cloudera’s largest HBase deployments
- Co-author of O’Reilly’s Apache Sqoop Cookbook
Aleks Shulman
- HBase Test Engineer focused on ensuring HBase is enterprise ready
- Primary focus on building compatibility frameworks for rolling upgrades
?2014 Cloudera, Inc. All rights reserved.
?2014 Cloudera, Inc. All rights reserved.
Cloudera Internal HBase Metrics
? Cloudera uses HBase internally for the Support Team
? We ingest Tickets, Cluster Stats, and Apache Mailing Lists
? Cloudera has ~20K HBase nodes under management
? Over 60% of my accounts use HBase
?2014 Cloudera, Inc. All rights reserved.
?2014 Cloudera, Inc. All rights reserved.
Agenda
● Tales Getting Production Started
● Tales Fixing Production Bugs
● Tales Upgrading Production Clusters
?2014 Cloudera, Inc. All rights reserved.
?2014 Cloudera, Inc. All rights reserved.
Agenda
● Tales Getting Production Started
● Tales Fixing Production Bugs
● Tales Upgrading Production Clusters
?2014 Cloudera, Inc. All rights reserved.
?2014 Cloudera, Inc. All rights reserved.
HBase Deployment Mistakes
? Cluster Sizing
?Managing Your Regions
?General Recommendations
?2014 Cloudera, Inc. All rights reserved.
?2014 Cloudera, Inc. All rights reserved.
Why Cluster Sizing Matters
? Jobs Failing
? Writes Blocking
? Performance Issues
?2014 Cloudera, Inc. All rights reserved.
?2014 Cloudera, Inc. All rights reserved.
Heavy Write Sizing
java_max_h ap 16GB
memstore_upper .50
java_max_heap * memstore = memstore_total_size
Calculating Total Available Memstore
desired_flush_size 128MB
repl_factor 3 (default)
max_file_size 20GB
Calculating Max Regions
memstore_total_size / desired_flush_size = total_regions_per_rs
max_file_size * (total_regions_per_rs * repl_factor) = raw_storage_per_node
X-axis = Flush_Size
Y-axis = Region_Count
?2014 Cloudera, Inc. All rights reserved.
?2014 Cloudera, Inc. All rights reserved.
Update for Known Writes Sizing
write_throughput 20MBs
total_data_size 350TB
hlog_size * number_of_hlogs = amount_of_data_before_flush
Calculating force flushes
hlog_size 128MBs
number_of_hlogs 64
(write_throughput * 60 * 60) / amount_of_data_before_flush =
number_nodes_before_flush
Calculating Max Regions
total_data_size 350TB
maxfile_size 20GB
((total_data_size * 1024) / maxfile_size) / desired_RS_count =
total_regions_per_rs
?2014 Cloudera, Inc. All rights reserved.
?2014 Cloudera, Inc. All rights reserved.
Why is Region Management Important
? Initial loads are failing
? Region Servers are crashing from overload
?2014 Cloudera, Inc. All rights reserved.
?2014 Cloudera, Inc. All rights reserved.
Region Management Best Practices
Region Split Policy
ConstantSize Split on Max Filesize Use when pre-splitting all
tables
UpperBoundSplitPolicy Split on smarter intervals Use when not able to pre-split
all tables
Balancer Policy
SimpleLoadBalancer Aimlessly balance regions Use with lots of tables with low
region count
ByTable Balance by table Use with few tables with high
region count
?2014 Cloudera, Inc. All rights reserved.
?2014 Cloudera, Inc. All rights reserved.
General Recommendations
Feature Benefit When to Enable
Short Circuit Reads (SCR) Speed up read times by bypassing
datanode layer
Always
Snappy Compression Speed up read times and lower data
consumption
On heavily accessed tables
Bloom Filters Speed up read times when numerous
HFiles are present
Row should always be used,
Row+Column is more accurate but
higher in memory usage
HLog Compression Speed up writes and recovery times Always
Data Block Encoding compress long keys to store more in
cache
Best for short/tall table