NoSQLStorageResearch < LCG

LCG Web>WLCGMonitoringConsolidation>NoSQLStorageResearch (2014-04-23, IvanKadochnikov) (raw view)
%TOC%

---+ HBase

Open-source BigTable implementation on top of HDFS or other distributed FS in the Hadoop ecosystem. Column-based NOSQL database. A stored value is addressed by rowkey, column family, column qualifier, timestamp. From the point of view of the DB rowkey, family, qualifier and value are untyped byte arrays.

   $ Scan: A query to the HBase database returning multiple rows. Can be limited by a range of rowkeys, by columns and by timestamp. Can use filters for more control.

---++ Queries

---+++ Interfaces

---++++ HBase Shell

The easiest tool to test HBase, query a table or debug a custom filter or coprocessor.

---++++ Java client

The fastest way to query HBase. See [[#TestResults][test results]].

---++++ REST interface

HBase provides a REST server that can be run on any node in the cluster (or outside the cluster if desired). It acts as an HBase client and a REST server to provide access to HBase data for non-java clients.

See [[#AppendixREST][appendix]] for more. Advantages:
   * Cross-platform and universal
Main problems:
   * Is supposedly slower than Thrift, not tested
   * Filters and scans can be used with REST, but not custom filters and not Endpoints.

---++++ Thrift interface

HBase also provides a Thrift server that can be run on any node in the cluster (or outside the cluster if desired). It acts as an HBase client and a Thrift server to provide access to HBase data for non-java clients.

Thrift is an interface definition language and automatically generates language-specific bindings for the interface.

Advantages:
   * Cross-platform
   * Reported as faster than REST
   * Custom filters can be used if configured on the Thrift server
Disadvantages:
   * Is slower than a Java client for large scans
   * Language-specific bindings need to be built
   * Endpoints (stored procedure Coprocessors) can't be run with Thrift

---++++ Phoenix

Phoenix is a tool to provide an interface very close to SQL to query HBase tables. Not for batch processing, not based on MapReduce.

[[https://github.com/forcedotcom/phoenix/][Readme]]

[[http://forcedotcom.github.io/phoenix/index.html][Query language reference]]

Things we can learn from Phoenix:
   * Key salt
   * Using filters
   * Using coprocessors
Reasons for us not to use Phoenix:
   * JDBC interface, which means Java client. Our existing code is Python. 
      * Ways to use JDBC in Python are sketchy, but may warrant another look.
   * Needs to be installed on the cluster. 
      * But if we use custom filters and coprocessors ourselves, they would need to be installed too.

---+++ Filters

Fliters limit the amount of data retrieved by an HBase scan based on some filtering criteria. Some useful examples: PageFilter (N) is great for debugging, limits the amount of rows returned from a scan to N.

Custom filter classes can be created. [[#AppendixFilters][More info]]

---+++ Coprocessors

[[https://blogs.apache.org/hbase/entry/coprocessor_introduction][Introduction]]

Coprocessor types:
   * Observers ~ triggers
   * Endpoints ~ stored procedures
Endpoints can't be called through Thrift out of the box: https://issues.apache.org/jira/browse/HBASE-5600

---++++ Phoenix observers

Observers are used by Phoenix for many things, including aggregation.

We may possibly use their coprocessors directly for custom scans instead of going through jdbc. This approach suffers from the fact, that Phoenix coprocessors are very tightly coupled with Phoenix metadata and utilities. So setting the correct Scan custom attributes without Java and Phoenix classes is almost impossible.

We may also just mimic the way Phoenix uses observers to do aggregation. They were not really designed for that so it would not be trivial at all, but it is possible (Phoenix works after all). Using observers instead of endpoints wold mean aggregation is possible from any client that support custom Scan attributes, including Thift.

---++ Processing

---+++ HBase from !MapReduce jobs

[[https://hbase.apache.org/book/mapreduce.example.html][Examples]]

---+++ Pig
---++++ UDF (user-defined functions)

#UserDefinedFunction UDF are used to do any data processing within PIG more complex that restructuring bags and straightforward aggregation.

[[https://pig.apache.org/docs/r0.11.1/udf.html#python-udfs][Python UDFs]]. Actually Jython, take care. For example, Java date libraries need to be used for any date manipulation.

Not convenient to construct rowkeys because of Unicode issues. Pig byte arrays are strings in the jython UDF, but jython only has unicode strings. As a result creating a rowkey byte array with binary values results in some bytes being encoded into pairs of bytes.

---++++ HBase from Pig

Works:
<verbatim>STORE data INTO 'ssbtest' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:etime, cf:value, cf:color, cf:url, cf:status');</verbatim>

First value in the tuple used as the key. To make a compound key either use several calls to CONCAT or write a [[#UserDefinedFunction][UDF]]

---++++ Phoenix from Pig

Phoenix provide a Pig storage class to store to a Phoenix table from Pig. This should work but fails with an error:
<verbatim>STORE data INTO 'hbase://ssbtest' USING com.salesforce.phoenix.pig.PhoenixHBaseStorage('dashb-ai-410', '-batchSize 10');</verbatim>

[[https://github.com/forcedotcom/phoenix/blob/1.2.1/src/main/java/com/salesforce/phoenix/pig/PhoenixHBaseStorage.java][Source]] of the class in question.

One can always use HBase storage for Pig instead. Phoenix tables are just HBase tables with some parameters and coprocessors set. The main disadvantage is concatenating the rowkey value needs to be done manually with HBase storage.

---+ Appendix

[[http://opentsdb.net/schema.html][OpenTSDB]] - an interesting monitoring project to (possibly) take some schema ideas from.

---++ Benchmark scan Java vs Thrift

#TestResults
| *data* | 1 month aggregated transfer data, 1.7M rows |
| *key format* | bTtttVvvvVvvvSsssSsssSsssSsssDdddDdddDdddDddd |
| ^ | salt = hash mod 16, 10 min time bin, vo, src, dst  |
| *scan parameters* | caching = 1000 batch = 100 |

java client on cluster (dashb-ai-410):
| *time range* | *vo src dst* | *row count* | *prep* | *scan* | *total* |
| 1072637400 1672810200 | "" "" "" | 1770598 | | | 22659 ms |
| 1372637400 1372810200 | "" "" "" | 105812 | 6 ms | 831 ms | 837 ms |
| 1372637400 1372810200 | "atlas" "" "" | 68970 | 5 ms | 624 ms | 629 ms |
| 1372637400 1372810200 | "atlas1" "" "" | 0 | 5 ms | 103 ms | 108 ms |
| 1372637400 1372810200 | "atlas" "GRIF" "" | 11 | 5 ms | 134 ms | 139 ms |
| 1372637400 1372810200 | "atlas" "GRIF" "" | 11 | 5 ms | 134 ms | 139 ms |
| 1372637400 1372810200 | "atlas" "" "GRIF" | 1418 | 5 ms | 270 ms | 275 ms |
| 1072637400 1672810200 | "" "" "GRIF" | 26456 | 8 ms | 4743 ms | 4751 ms |
| 1072637400 1672810200 | "" "GRIF" "" | 1400 | 10 ms | 1611 ms | 1621 ms |
| 1072637400 1672810200 | "atlas" "GRIF" "" | 417 | 9 ms | 1195 ms | 1204 ms |
| 1372637400 1372810200 | "atlas" "GRIF" "GRIF" | 2 | 10 ms | 142 ms | 152 ms |

WARNING: some rows are for some reason left out when src and/or dst is set. This bug needs to be found. The python+thrift code works more correctly

java client on desktop
| *not tested* | exception when creating scan |

thrift client on cluster (dashb-ai-410)
| *time range* | *vo src dst* | *row count* | *walltime* |
| 1372637400 1372810200 | "atlas" "" "" | 68970 | 11.0466649532 |

---++ More info on the REST interface

#AppendixREST In python [[http://docs.python-requests.org/en/latest/index.html][requests]] is a good library for working with REST. Happybase is supposed to be a more high-level HBase-specific library, I have not had time to try it.

[[https://wiki.apache.org/hadoop/Hbase/Stargate][Rest interface reference]]

Filters and scans can be used with REST, but not custom filters and not Endpoints.

[[https://gist.github.com/stelcheck/3979381][Filter parameters reference]]

Example:
<verbatim><Scanner batch="5"> <filter>{
  "type": "FamilyFilter",
  "op": "EQUAL",
  "comparator": {
    "type": "BinaryComparator",
    "value": "dGVzdHJvdw\u003d\u003d"
  }
}</filter> </Scanner></verbatim>

Yes, json-like inside xml. Rest also supports application/protobuf instead of XML, but filter definition probably still uses the same format. Note that values are b64 encoded.

---++ Details about filters

#AppendixFilters
---+++ Custom filters

Any filter can be used from HBase shell, provided it's in the classpath. hbase&gt; scan 't1', {FILTER =&gt; org.apache.hadoop.hbase.filter.ColumnPaginationFilter.new(1, 0)}

To be able to use filter through Thrift or REST it needs to provide
<verbatim>createFilterFromArguments(ArrayList<byte[]> filterArguments) </verbatim>

#ShellCustomFilter This provides a way to create a filter from string parameters, which is exactly what Thrift/Rest provide. To debug this method for your custom filter from HBase shell, register the method (so that hbase can look up method class name from a short method name) and try a scan like this:
<verbatim>hbase> org.apache.hadoop.hbase.filter.ParseFilter.registerFilter("FuzzyRowFilterFromText", "org.apache.hadoop.hbase.filter.FuzzyRowFilterFromText")
scan 'SSB3', {FILTER => "(FuzzyRowFilterFromText ( '\x00\x00\x00\x00\x00\x00ssb_metric_18\x00\x00\x00VICTORIA-LCG2\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', '\x01\x01\x01\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01') AND (PageFilter (10)) )" }
</verbatim>

To use the custom filter with other interfaces, it needs to be registered too:

[[https://issues.apache.org/jira/browse/HBASE-5428][Adding custom filters]] to Thrift server configuration.

#ClouderaRegisterFilter In CDH4 Cloudera manager:

Services-&gt;Hbase1-&gt;Configuration-&gt;View and edit-&gt;Default Category-&gt;HBaseThrift Server (Default)-&gt;Advanced-&gt;HBase Thrift Server Configuration Safety Valve for hbase-site.xml:
<verbatim>
<property>
  <name>hbase.thrift.filters</name><value>FuzzyRowFilterFromText:org.apache.hadoop.hbase.filter.FuzzyRowFilterFromText</value>
</property>
</verbatim>

This is [[https://issues.apache.org/jira/browse/HBASE-7115][not implemented]] for REST yet.

---+++ Filters with skip hints

Easy to imagine, harder to understand and implement. Faster than simple filters that just check one row. http://hadoop-hbase.blogspot.ch/2012/01/hbase-intra-row-scanning.html http://hadoop-hbase.blogspot.ch/2012/01/filters-in-hbase-or-intra-row-scanning.html

---+++ !FuzzyRowFilter

!FuzzyRowFilter is a rowkey filter allowing filter by subsets of the rowkey bytes. Advantages:
   * Fast: provides skip hints to the scanner
   * Structurally simple: just provide keyvalues and byte masks
   * Part of HBase
Disadvantages:
   * Not very friendly: construct keyvalues and byte masks yourself
   * Does not support ranges for rowkey parts, only discrete values. 
      * It's OK in our use-case, we only have one part of the key that needs range filtering: timestamp. We want it to be at the front of the key anyway, so we can use scan's startRow and stopRow to filter by timestamp. Then this filter would only be used to filter by resource and by metric, which don't need range filtering.
   * Does not allow for variable-length rowkey parts. 
      * Either we use fixed length of resource and metric strings
      * Or we use binary resource id and metric id to save space
   * Does not implement createFilterFromArguments 
      * This is pretty easy to fix with a subclass that implements this method
   * Consequently [[https://issues.apache.org/jira/browse/HBASE-7983][can't be used with REST]] or Thrift out of the box (they need createFilterFromArguments)

---+++ !SkipScanFilter

[[https://github.com/forcedotcom/phoenix/wiki/Skip-Scan][SkipScanFilter]] is a rowkey filter allowing to filter by key columns in Phoenix tables. Used iternally by Phoenix queries. Advantages:
   * Fast: provides skip hints to the scanner
   * Supports ranges for key column values.
   * Supports variable-length and empty key columns.
   * Easier to use: for each key column provide a set of value ranges 
      * But key columns are defined in Phoenix
Disadvantages:
   * Part of Phoenix
   * Needs Phoenix metadata (key column parameters) to construct the filter
   * Does not implement createFilterFromArguments 
      * Which is harder to implement ourselves because of depending on Phoenix metadata

---++ Cloudera CDH4

Ports used:

[[http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/4.5.4/Configuring-Ports-for-Cloudera-Manager-Enterprise-Edition/cmeecp_topic_4.html][Used by the manager stack]]

[[http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/4.5.4/Configuring-Ports-for-Cloudera-Manager-Enterprise-Edition/cmeecp_topic_3.html][Used by actual servers]]

Thrift moved from the default 9090 to 7182 (manager port) because we have a lot of ports open as it is. Thrift is running on host 414, manager is on host 410 so there is no conflict as of now.

---++ FuzzyRowFilter in Thrift howto:

This filter does not provide the necessary method so we create our own. The class definition is [[https://twiki.cern.ch/twiki/pub/LCG/NoSQLStorageResearch/FuzzyRowFilterFromText.java][here]]

To use this filter, build this class into a jar. Eclipse can be used if HBase and Hadoop jars are added to the build path. Otherwise one can build it on any cluster node.

Then this jar must be added to Thrift server classpath, the easiest way is to put it in /usr/lib/hbase/lib with the other HBase jars.

Then the filter must be registered through Thrift server configuration. See [[#ClouderaRegisterFilter]] for details on how to do that in Cloudera CDH4

After restarting the thrift server the filter should be available through Thrift.

To test the FuzzyRowFilterFromText wrapper see [[#ShellCustomFilter]]

---++ SSB howto:

#SsbPutData Put data into HDFS: If "ssb" is a local dir with your data and hdfs://user/root/ is a hdfs dir where you have write permissions

=hadoop fs -put ssb /user/root=

Put this data into HBase:

[[https://twiki.cern.ch/twiki/pub/LCG/NoSQLStorageResearch/SsbStore.java][Example MapReduce job]]

Not necessary to use a MapReduce job here. This one does not even have a reducer. If necessary it should not be hard to rewrite as a standalone utility. When it's not a MapReduce job there would be no need to copy input data to HDFS first ether.

To build and run this MapReduce job on a node in the cluster:
<verbatim>
cd /root/ssbMapReduce/src/
javac -cp $HADOOP_CLASSPATH:`hbase classpath` SsbStore.java &&
jar cf mr.jar *.class && 
HADOOP_CLASSPATH=$HADOOP_CLASSPATH:`hbase classpath` hadoop jar mr.jar SsbStore /user/root/ssb SSB3
</verbatim>

To request SSB data from HBase

[[https://twiki.cern.ch/twiki/pub/LCG/NoSQLStorageResearch/scan.py.txt][REST interface scan]]

[[https://twiki.cern.ch/twiki/pub/LCG/NoSQLStorageResearch/smartscan.py.txt][REST interface scan with FuzzyRowFilter]] (does not work, REST does not support this filter)

[[https://twiki.cern.ch/twiki/pub/LCG/NoSQLStorageResearch/thrifttscan.py.txt][Thrift interface scan with FuzzyRowFilter]]

---++ Transfers howto:

Put data into HDFS just like [[#SsbPutData][SSB]]

Put this data into HBase: [[https://twiki.cern.ch/twiki/pub/LCG/NoSQLStorageResearch/TransfersStore.java][Example MapReduce job]]

<verbatim>
cd transfersMapReduce/src/org/cern/dashboard/hadoop/ &&
mkdir -p _classes &&
javac -cp `hbase classpath` -d _classes *.java &&
jar -cvf transfers_mr.jar -C _classes . &&
HADOOP_CLASSPATH=$HADOOP_CLASSPATH:`hbase classpath` hadoop jar transfers_mr.jar org.cern.dashboard.hadoop.TransfersStore transfers TEST_TFRS2
</verbatim>

[[https://twiki.cern.ch/twiki/pub/LCG/NoSQLStorageResearch/thriftscantransfers.py.txt][Thrift client scan ]]

[[https://twiki.cern.ch/twiki/pub/LCG/NoSQLStorageResearch/TransfersScan.java][Java client scan ]]

Running java client:
<verbatim>
cd transfersMapReduce/src/org/cern/dashboard/hadoop/ &&
mkdir -p _classes &&
javac -cp `hbase classpath` -d _classes *.java &&
jar -cvf transfers.jar -C _classes . &&
java -cp `hbase classpath`:transfers.jar org.cern.dashboard.hadoop.TransfersScan \
16 1372637400 1372810200 \
"atlas" "" ""
</verbatim>

New topic [[ElasticsearchKibanaAuthenticationNotes]]
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who
java	FuzzyRowFilterFromText.java	r1	manage	1.2 K	2013-08-16 - 11:42	IvanKadochnikov
txt	smartscan.py.txt	r1	manage	1.4 K	2013-08-16 - 11:02	IvanKadochnikov
txt	scan.py.txt	r1	manage	1.7 K	2013-08-16 - 11:02	IvanKadochnikov
java	TransfersStore.java	r1	manage	3.5 K	2013-08-16 - 11:42	IvanKadochnikov
txt	thriftscan.py.txt	r1	manage	4.2 K	2013-08-16 - 11:02	IvanKadochnikov
txt	thriftscantransfers.py.txt	r1	manage	5.0 K	2013-08-16 - 11:44	IvanKadochnikov
java	TransfersScan.java	r1	manage	5.1 K	2013-08-16 - 11:42	IvanKadochnikov
java	SsbStore.java	r1	manage	7.2 K	2013-08-16 - 11:02	IvanKadochnikov
Topic revision: r12 - 2014-04-23 - IvanKadochnikov
LCG Wikis
LCG Service Coordination
LCG Grid Deployment
LCG Applications Area
Public webs
Welcome Guest
- Cern Search
- TWiki Search
- Google Search
LCG All webs