Tales From the Cloudera Field

Tales From the Cloudera Field Kevin O’Dell, Kate Ting, Aleks Shulman {kevin, kate, aleks}@cloudera.com ?2014 Cloudera, Inc. All rights reserved. ?2014 Cloudera, Inc. All rights reserved. Who Are We? Kevin O’Dell - Previously HBase Support Team Lead - Currently Systems Engineer with a focus on HBase deployments Kate Ting - Technical Account Manager of Cloudera’s largest HBase deployments - Co-author of O’Reilly’s Apache Sqoop Cookbook Aleks Shulman - HBase Test Engineer focused on ensuring HBase is enterprise ready - Primary focus on building compatibility frameworks for rolling upgrades ?2014 Cloudera, Inc. All rights reserved. ?2014 Cloudera, Inc. All rights reserved. Cloudera Internal HBase Metrics ? Cloudera uses HBase internally for the Support Team ? We ingest Tickets, Cluster Stats, and Apache Mailing Lists ? Cloudera has ~20K HBase nodes under management ? Over 60% of my accounts use HBase ?2014 Cloudera, Inc. All rights reserved. ?2014 Cloudera, Inc. All rights reserved. Agenda ● Tales Getting Production Started ● Tales Fixing Production Bugs ● Tales Upgrading Production Clusters ?2014 Cloudera, Inc. All rights reserved. ?2014 Cloudera, Inc. All rights reserved. Agenda ● Tales Getting Production Started ● Tales Fixing Production Bugs ● Tales Upgrading Production Clusters ?2014 Cloudera, Inc. All rights reserved. ?2014 Cloudera, Inc. All rights reserved. HBase Deployment Mistakes ? Cluster Sizing ?Managing Your Regions ?General Recommendations ?2014 Cloudera, Inc. All rights reserved. ?2014 Cloudera, Inc. All rights reserved. Why Cluster Sizing Matters ? Jobs Failing ? Writes Blocking ? Performance Issues ?2014 Cloudera, Inc. All rights reserved. ?2014 Cloudera, Inc. All rights reserved. Heavy Write Sizing java_max_h ap 16GB memstore_upper .50 java_max_heap * memstore = memstore_total_size Calculating Total Available Memstore desired_flush_size 128MB repl_factor 3 (default) max_file_size 20GB Calculating Max Regions memstore_total_size / desired_flush_size = total_regions_per_rs max_file_size * (total_regions_per_rs * repl_factor) = raw_storage_per_node X-axis = Flush_Size Y-axis = Region_Count ?2014 Cloudera, Inc. All rights reserved. ?2014 Cloudera, Inc. All rights reserved. Update for Known Writes Sizing write_throughput 20MBs total_data_size 350TB hlog_size * number_of_hlogs = amount_of_data_before_flush Calculating force flushes hlog_size 128MBs number_of_hlogs 64 (write_throughput * 60 * 60) / amount_of_data_before_flush = number_nodes_before_flush Calculating Max Regions total_data_size 350TB maxfile_size 20GB ((total_data_size * 1024) / maxfile_size) / desired_RS_count = total_regions_per_rs ?2014 Cloudera, Inc. All rights reserved. ?2014 Cloudera, Inc. All rights reserved. Why is Region Management Important ? Initial loads are failing ? Region Servers are crashing from overload ?2014 Cloudera, Inc. All rights reserved. ?2014 Cloudera, Inc. All rights reserved. Region Management Best Practices Region Split Policy ConstantSize Split on Max Filesize Use when pre-splitting all tables UpperBoundSplitPolicy Split on smarter intervals Use when not able to pre-split all tables Balancer Policy SimpleLoadBalancer Aimlessly balance regions Use with lots of tables with low region count ByTable Balance by table Use with few tables with high region count ?2014 Cloudera, Inc. All rights reserved. ?2014 Cloudera, Inc. All rights reserved. General Recommendations Feature Benefit When to Enable Short Circuit Reads (SCR) Speed up read times by bypassing datanode layer Always Snappy Compression Speed up read times and lower data consumption On heavily accessed tables Bloom Filters Speed up read times when numerous HFiles are present Row should always be used, Row+Column is more accurate but higher in memory usage HLog Compression Speed up writes and recovery times Always Data Block Encoding compress long keys to store more in cache Best for short/tall table