21 May 2013 ... Apache Accumulo User Manual. Version 1.5 ..... Apache Accumulo is a highly
scalable structured store based on Google's BigTable. Accumulo.
Benchmarking Apache Accumulo BigData Distributed Table Store Using Its.
Continuous Test ... performance figures for Apache Accumulo found in the study.
Introduction to Sqrrl and Accumulo. 2. Security In The Wild. 3. Sqrrl and Accumulo
Technology. 4. The Data-Centric Security Ecosystem. In our September ...
Page 1. Derby Reference Manual. Version 10. Derby Document build: December
11, 2006, 7:09:24 AM (PST). Page 2. Page 3. Contents.
Page 1 of 1. The manual was found. Link is available only 30 minutes. Click - free download. Page 1 of 1. Manual chevrol
... table design and best practices PDF File Accumulo Application Development ... applications for several use cases Acc
Get up to speed on Apache Accumulo Application Development Table Design and Best Practices Author Download PDF 13 76 MB
TABELLA ACCUMULO MIGLIA. BONUS. Descrizione. Miglia. Benvenuto Hi Fly
Now*. 500 miglia. Benvenuto Hi Fly Advantage. 1000 miglia. Presenta un amico*
...
Apache Camel ª is a versatile open-source integration framework based on
known ... Apache Camel uses URIs to work directly with any kind of Transport or ...
Apache Thrift. Scalable Cross Language Services Implementation. 김병곤
[email protected]. JBoss User Group. 共感#5 ...
A sample of other features include SSL and TLS support (mod_ssl), a proxy ...
Virtual hosting allows one Apache installation to serve many different actual ...
Aug 20, 2007 ... use Apache::Util qw(:all);. 1.2 Description. This module provides a Perl interface
to some of the C utility functions available in Apache.
study an Apache Accumulo–based big data system designed for a network ... We
then characterize the corresponding Accumulo performance bottlenecks.
Licensed to the Apache Software Foundation (ASF) under one or more ... The
ASF licenses this file to you under the Apache License, ... Creating a new server .
parsed. 2. Digester uses SAX to do the parsing, XML processing with Digester happens. in an event-driven manner. An even
il cliente richieda il sistema di accumulo di energia elettrica. ... La profonda
esperienza nell'uso dei vari tipi di Batteria di Accumulo ha inoltre permesso a.
Developing Distributed Applications that use ZooKeeper by ...... There are two
packages that make up the ZooKeeper Java binding: org.apache.zookeeper.
public class Executor implements Watcher, Runnable, DataMonitor. ... The
Watcher interface is defined by the ZooKeeper Java API. ZooKeeper .... dead =
true;.
Mar 11, 2012 ... Apache Shindig is an OpenSocial container and helps you to start hosting ...
Apache Shindig is a container for hosting social application ...
1. Purpose. This document comprehensively describes all user-facing facets of
the Hadoop MapReduce framework and serves as a tutorial. 2. Prerequisites.
Licensed to the Apache Software Foundation (ASF) under .... The Editor for
Apache DS 1.5.0 allows you to edit the 'server.xml' file of an Apache DS version.
What is ZooKeeper. • A general wait-free coordination service. • membership,
leader election, lock… • Provide order guarantees for operations. • Linearizable ...
ZooKeeper Programmer's Guide. Developing Distributed Applications that use
ZooKeeper by. Table of contents. 1 Introduction.
28 Jun 2012 ... Apache Accumulo is a highly scalable structured store based on Google's
BigTable. ... Accumulo supports efficient storage and retrieval of.
Introduction Apache Accumulo is a highly scalable structured store based on Google's BigTable. Accumulo is written in Java and operates over the Hadoop Distributed File System (HDFS), which is part of the popular Apache Hadoop project.
Accumulo supports e cient storage and retrieval of
structured data, including queries for ranges, and provides support for using Accumulo tables as input and output for MapReduce jobs. Accumulo features automatic load-balancing and partitioning, data compression and ne-grained security labels.
5
Chapter 2
Accumulo Design 2.1 Data Model Accumulo provides a richer data model than simple key-value stores, but is not a fully relational database. Data is represented as key-value pairs, where the key and value are comprised of the following elements: Key Row ID
Column Family
Quali er
Visibility
Timestamp
Value
All elements of the Key and the Value are represented as byte arrays except for Timestamp, which is a Long.
Accumulo sorts keys by element and lexicographically in ascending order.
Timestamps are sorted in descending order so that later versions of the same Key appear rst in a sequential scan. Tables consist of a set of sorted key-value pairs.
2.2 Architecture Accumulo is a distributed data storage and retrieval system and as such consists of several architectural components, some of which run on many individual servers. Much of the work Accumulo does involves maintaining certain properties of the data, such as organization, availability, and integrity, across many commodity-class machines.
6
2.3 Components An instance of Accumulo includes many TabletServers, write-ahead Logger servers, one Garbage Collector process, one Master server and many Clients.
2.3.1 Tablet Server The TabletServer manages some subset of all the tablets (partitions of tables). This includes receiving writes from clients, persisting writes to a write-ahead log, sorting new key-value pairs in memory, periodically ushing sorted key-value pairs to new les in HDFS, and responding to reads from clients, forming a merge-sorted view of all keys and values from all the les it has created and the sorted in-memory store. TabletServers also perform recovery of a tablet that was previously on a server that failed, reapplying any writes found in the write-ahead log to the tablet.
2.3.2 Loggers The Loggers accept updates to Tablet servers and write them to local on-disk storage. Each tablet server will write their updates to multiple loggers to preserve data in case of hardware failure.
2.3.3 Garbage Collector Accumulo processes will share les stored in HDFS. Periodically, the Garbage Collector will identify les that are no longer needed by any process, and delete them.
2.3.4 Master The Accumulo Master is responsible for detecting and responding to TabletServer failure. It tries to balance the load across TabletServer by assigning tablets carefully and instructing TabletServers to migrate tablets when necessary. The Master ensures all tablets are assigned to one TabletServer each, and handles table creation, alteration, and deletion requests from clients. The Master also coordinates startup, graceful shutdown and recovery of changes in write-ahead logs when Tablet servers fail.
7
2.3.5 Client Accumulo includes a client library that is linked to every application. The client library contains logic for nding servers managing a particular tablet, and communicating with TabletServers to write and retrieve key-value pairs.
2.4 Data Management Accumulo stores data in tables, which are partitioned into tablets. Tablets are partitioned on row boundaries so that all of the columns and values for a particular row are found together within the same tablet. The Master assigns Tablets to one TabletServer at a time. This enables row-level transactions to take place without using distributed locking or some other complicated synchronization mechanism. As clients insert and query data, and as machines are added and removed from the cluster, the Master migrates tablets to ensure they remain available and that the ingest and query load is balanced across the cluster.
8
2.5 Tablet Service When a write arrives at a TabletServer it is written to a Write-Ahead Log and then inserted into a sorted data structure in memory called a MemTable.
When the MemTable reaches a
certain size the TabletServer writes out the sorted key-value pairs to a le in HDFS called Indexed Sequential Access Method (ISAM) le. This process is called a minor compaction. A new MemTable is then created and the fact of the compaction is recorded in the Write-Ahead Log. When a request to read data arrives at a TabletServer, the TabletServer does a binary search across the MemTable as well as the in-memory indexes associated with each ISAM le to nd the relevant values. If clients are performing a scan, several key-value pairs are returned to the client in order from the MemTable and the set of ISAM les by performing a merge-sort as they are read.
2.6 Compactions In order to manage the number of les per tablet, periodically the TabletServer performs Major Compactions of les within a tablet, in which some set of ISAM les are combined into one le.
The previous les will eventually be removed by the Garbage Collector.
This also pro-
vides an opportunity to permanently remove deleted key-value pairs by omitting key-value pairs suppressed by a delete entry when the new le is created.
2.7 Fault-Tolerance If a TabletServer fails, the Master detects it and automatically reassigns the tablets assigned from the failed server to other servers. Any key-value pairs that were in memory at the time the TabletServer are automatically reapplied from the Write-Ahead Log to prevent any loss of data. The Master will coordinate the copying of write-ahead logs to HDFS so the logs are available to all tablet servers. To make recovery e cient, the updates within a log are grouped by tablet. The sorting process can be performed by Hadoops MapReduce or the Logger server. TabletServers can quickly apply the mutations from the sorted logs that are destined for the tablets they have now been assigned. TabletServer failures are noted on the Master's monitor page, accessible via http://master-address:50095/monitor.
9
10
Chapter 3
Accumulo Shell Accumulo provides a simple shell that can be used to examine the contents and con guration settings of tables, insert/update/delete values, and change con guration settings. The shell can be started by the following command:
$ACCUMULO_HOME/bin/accumulo shell -u [username] The shell will prompt for the corresponding password to the username speci ed and then display the following prompt:
Shell - Apache Accumulo Interactive Shell - version 1.3 - instance name: myinstance - instance id: 00000000-0000-0000-0000-000000000000 - type 'help' for a list of available commands -
3.1 Basic Administration The Accumulo shell can be used to create and delete tables, as well as to con gure table and instance speci c options.
root@myinstance mytable> root@myinstance mytable> tables !METADATA mytable root@myinstance mytable> createtable testtable root@myinstance testtable> root@myinstance junk> deletetable testtable root@myinstance> The Shell can also be used to insert updates and scan tables. This is useful for inspecting tables.
root@myinstance mytable> scan root@myinstance mytable> insert row1 colf colq value1 insert successful root@myinstance mytable> scan row1 colf:colq [] value1 The value in brackets "[]" would be the visibility labels. Since none were used, this is empty for this row. You can use the "-t" option to scan to see the timestamp for the cell, too.
3.2 Table Maintenance The
compact command instructs Accumulo to schedule a compaction of the table during which
les are consolidated and deleted entries are removed.
root@myinstance mytable> compact -t mytable 07 16:13:53,201 [shell.Shell] INFO : Compaction of table mytable scheduled for 20100707161353EDT The
ush
command instructs Accumulo to write all entries currently in memory for a given
table to disk.
root@myinstance mytable> ush -t mytable 07 16:14:19,351 [shell.Shell] INFO : Flush of table mytable initiated...
12
3.3 User Administration The Shell can be used to add, remove, and grant privileges to users.
root@myinstance mytable> createuser bob Enter new password for 'bob': ********* Please con rm new password for 'bob': ********* root@myinstance mytable> authenticate bob Enter current password for 'bob': ********* Valid root@myinstance mytable> grant System.CREATE_TABLE -s -u bob root@myinstance mytable> user bob Enter current password for 'bob': ********* bob@myinstance mytable> userpermissions System permissions: System.CREATE_TABLE Table permissions (!METADATA): Table.READ Table permissions (mytable): NONE bob@myinstance mytable> createtable bobstable bob@myinstance bobstable> bob@myinstance bobstable> user root Enter current password for 'root': ********* root@myinstance bobstable> revoke System.CREATE_TABLE -s -u bob
13
Chapter 4
Writing Accumulo Clients 4.1 Running Client Code There are multiple ways to run Java code that uses Accumulo. Below is a list of the dierent ways to execute client code.
•
using java executable
•
using the accumulo script
•
using the tool script
Inorder to run client code written to run against Accumulo, you will need to include the jars that Accumulo depends on in your classpath. Accumulo client code depends on Hadoop and Zookeeper. For Hadoop add the hadoop core jar, all of the jars in the Hadoop lib directory, and the conf directory to the classpath. For Zookeeper 3.3 you only need to add the Zookeeper jar, and not what is in the Zookeeper lib directory. You can run the following command on a con gured Accumulo system to see what its using for its classpath.
$ACCUMULO_HOME/bin/accumulo classpath Another option for running your code is to put a jar le in $ACCUMULO_HOME/lib/ext. After doing this you can use the accumulo script to execute your code.
For example if you
create a jar containing the class com.foo.Client and placed that in lib/ext, then you could use the command $ACCUMULO_HOME/bin/accumulo com.foo.Client to execute your code. If you are writing map reduce job that access Accumulo, then you can use the bin/tool.sh script to run those jobs. See the map reduce example.
14
4.2 Connecting All clients must rst identify the Accumulo instance to which they will be communicating. Code to do this is as follows:
4.3 Writing Data Data are written to Accumulo by creating Mutation objects that represent all the changes to the columns of a single row. The changes are made atomically in the TabletServer. Clients then add Mutations to a BatchWriter which submits them to the appropriate TabletServers. Mutations can be created thus:
Text rowID = new Text("row1"); Text colFam = new Text("myColFam"); Text colQual = new Text("myColQual"); ColumnVisibility colVis = new ColumnVisibility("public"); long timestamp = System.currentTimeMillis(); Value value = new Value("myValue".getBytes()); Mutation mutation = new Mutation(rowID); mutation.put(colFam, colQual, colVis, timestamp, value);
4.3.1 BatchWriter The BatchWriter is highly optimized to send Mutations to multiple TabletServers and automatically batches Mutations destined for the same TabletServer to amortize network overhead. Care must be taken to avoid changing the contents of any Object passed to the BatchWriter since it keeps objects in memory while batching. Mutations are added to a BatchWriter thus:
long memBuf = 1000000L; // bytes to store before sending a batch long timeout = 1000L; // milliseconds to wait before sending int numThreads = 10; 15
BatchWriter writer = conn.createBatchWriter("table", memBuf, timeout, numThreads) writer.add(mutation); writer.close(); An example of using the batch writer can be found at accumulo/docs/examples/README.batch
4.4 Reading Data Accumulo is optimized to quickly retrieve the value associated with a given key, and to e ciently return ranges of consecutive keys and their associated values.
4.4.1 Scanner To retrieve data, Clients use a Scanner, which provides acts like an Iterator over keys and values. Scanners can be con gured to start and stop at particular keys, and to return a subset of the columns available.
// specify which visibilities we are allowed to see Authorizations auths = new Authorizations("public"); Scanner scan = conn.createScanner("table", auths); scan.setRange(new Range("harry","john")); scan.fetchFamily("attributes"); for(Entry entry : scan) { String row = e.getKey().getRow(); Value value = e.getValue(); }
4.4.2 Isolated Scanner Accumulo supports the ability to present an isolated view of rows when scanning. There are three possible ways that a row could change in accumulo :
16
•
a mutation applied to a table
•
iterators executed as part of a minor or major compaction
•
bulk import of new les
Isolation guarantees that either all or none of the changes made by these operations on a row are seen. Use the IsolatedScanner to obtain an isolated view of an accumulo table. When using the regular scanner it is possible to see a non isolated view of a row. For example if a mutation modi es three columns, it is possible that you will only see two of those modi cations. With the isolated scanner either all three of the changes are seen or none. The IsolatedScanner buers rows on the client side so a large row will not crash a tablet server. By default rows are buered in memory, but the user can easily supply their own buer if they wish to buer to disk when rows are large. For an example, look at the following src/examples/src/main/java/org/apache/accumulo/examples/isolation/InterferenceTest.java
4.4.3 BatchScanner For some types of access, it is more e cient to retrieve several ranges simultaneously.
This
arises when accessing a set of rows that are not consecutive whose IDs have been retrieved from a secondary index, for example. The BatchScanner is con gured similarly to the Scanner; it can be con gured to retrieve a subset of the columns available, but rather than passing a single Range, BatchScanners accept a set of Ranges. It is important to note that the keys returned by a BatchScanner are not in sorted order since the keys streamed are from multiple TabletServers in parallel.
ArrayList ranges = new ArrayList(); // populate list of ranges ... BatchScanner bscan = conn.createBatchScanner("table", auths, 10); bscan.setRanges(ranges); bscan.fetchFamily("attributes"); for(Entry entry : scan) System.out.println(e.getValue()); An example of the BatchScanner can be found at accumulo/docs/examples/README.batch
17
4.5 Proxy The proxy API allows the interaction with Accumulo with languages other than Java. A proxy server is provided in the codebase and a client can further be generated.
4.5.1 Prequisites The proxy server can live on any node in which the basic client API would work. That means it must be able to communicate with the Master, ZooKeepers, NameNode, and the Data nodes. A proxy client only needs the ability to communicate with the proxy server.
4.5.2 Con guration The con guration options for the proxy server live inside of a properties le. At the very least, you need to supply the following properties:
protocolFactory=org.apache.thrift.protocol.TCompactProtocol$Factory tokenClass=org.apache.accumulo.core.client.security.tokens.PasswordToken port=42424 instance=test zookeepers=localhost:2181 You can nd a sample con guration le in your distribution:
$ACCUMULO_HOME/proxy/proxy.properties. This sample con guration le further demonstrates an abilty to back the proxy server by MockAccumulo or the MiniAccumuloCluster.
4.5.3 Running the Proxy Server After the properties le holding the con guration is created, the proxy server can be started using the following command in the Accumulo distribution (assuming you your properties le is named con g.properties):
$ACCUMULO_HOME/bin/accumulo proxy -p con g.properties
18
4.5.4 Creating a Proxy Client Aside from installing the Thrift compiler, you will also need the language-speci c library for Thrift installed to generate client code in that language.
Typically, your operating system's
package manager will be able to automatically install these for you in an expected location such as /usr/lib/python/site-packages/thrift. You can nd the thrift le for generating the client:
$ACCUMULO_HOME/proxy/proxy.thrift. After a client is generated, the port speci ed in the con guration properties above will be used to connect to the server.
4.5.5 Using a Proxy Client The following examples have been written in Java and the method signatures may be slightly dierent depending on the language speci ed when generating client with the Thrift compiler. After initiating a connection to the Proxy (see Apache Thrift's documentation for examples of connecting to a Thrift service), the methods on the proxy client will be available.
The rst
thing to do is log in:
Map password = new HashMap(); password.put("password", "secret"); ByteBuer token = client.login("root", password); Once logged in, the token returned will be used for most subsequent calls to the client. Let's create a table, add some data, scan the table, and delete it. First, create a table.
client.createTable(token, "myTable", true, TimeType.MILLIS); Next, add some data:
// rst, create a writer on the server String writer = client.createWriter(token, "myTable", new WriterOptions()); // build column updates Map cellsToUpdate = //... // send updates to the server client.updateAndFlush(writer, "myTable", cellsToUpdate); client.closeWriter(writer); 19
Scan for the data and batch the return of the results on the server:
String scanner = client.createScanner(token, "myTable", new ScanOptions()); ScanResult results = client.nextK(scanner, 100); for(KeyValue keyValue : results.getResultsIterator()) { // do something with results } client.closeScanner(scanner);
20
Chapter 5
Development Clients Normally, Accumulo consists of lots of moving parts. Even a stand-alone version of Accumulo requires Hadoop, Zookeeper, the Accumulo master, a tablet server, etc. If you want to write a unit test that uses Accumulo, you need a lot of infrastructure in place before your test can run.
5.1 Mock Accumulo Mock Accumulo supplies mock implementations for much of the client API. It presently does not enforce users, logins, permissions, etc. It does support Iterators and Combiners. Note that MockAccumulo holds all data in memory, and will not retain any data or settings between runs. While normal interaction with the Accumulo client looks like this:
Instance instance = new ZooKeeperInstance(...); Connector conn = instance.getConnector(user, passwd); To interact with the MockAccumulo, just replace the ZooKeeperInstance with MockInstance:
Instance instance = new MockInstance(); In fact, you can use the "fake" option to the Accumulo shell and interact with MockAccumulo:
- type 'help' for a list of available commands root@mock-instance> createtable test root@mock-instance test> insert row1 cf cq value root@mock-instance test> insert row2 cf cq value2 root@mock-instance test> insert row3 cf cq value3 root@mock-instance test> scan row1 cf:cq [] value row2 cf:cq [] value2 row3 cf:cq [] value3 root@mock-instance test> scan -b row2 -e row2 row2 cf:cq [] value2 root@mock-instance test> When testing Map Reduce jobs, you can also set the Mock Accumulo on the AccumuloInputFormat and AccumuloOutputFormat classes:
// ... set up job con guration AccumuloInputFormat.setMockInstance(job, "mockInstance"); AccumuloOutputFormat.setMockInstance(job, "mockInstance");
5.2 Mini Accumulo Cluster While the Mock Accumulo provides a lightweight implementation of the client API for unit testing, it is often necessary to write more realistic end-to-end integration tests that take advantage of the entire ecosystem. The Mini Accumulo Cluster makes this possible by con guring and starting Zookeeper, initializing Accumulo, and starting the Master as well as some Tablet Servers. It runs against the local lesystem instead of having to start up HDFS. To start it up, you will need to supply an empty directory and a root password as arguments:
File tempDirectory = // JUnit and Guava supply mechanisms for creating temp directories MiniAccumuloCluster accumulo = new MiniAccumuloCluster(tempDirectory, "password"); accumulo.start(); Once we have our mini cluster running, we will want to interact with the Accumulo client API:
Instance instance = new ZooKeeperInstance(accumulo.getInstanceName(), accumulo.getZooKeepers()); Connector conn = instance.getConnector("root", "password"); Upon completion of our development code, we will want to shutdown our MiniAccumuloCluster:
accumulo.stop() // delete your temporary folder 22
Chapter 6
Table Con guration Accumulo tables have a few options that can be con gured to alter the default behavior of Accumulo as well as improve performance based on the data stored.
These include locality
groups, constraints, bloom lters, iterators, and block cache.
6.1 Locality Groups Accumulo supports storing of sets of column families separately on disk to allow clients to scan over columns that are frequently used together e cient and to avoid scanning over column families that are not requested. After a locality group is set Scanner and BatchScanner operations will automatically take advantage of them whenever the fetchColumnFamilies() method is used. By default tables place all column families into the same default" locality group. Additional locality groups can be con gured anytime via the shell or programmatically as follows:
6.1.1 Managing Locality Groups via the Shell usage: setgroups ={,}{ ={,}} [-?] -t
6.1.2 Managing Locality Groups via the Client API Connector conn; HashMap localityGroups = new HashMap(); HashSet metadataColumns = new HashSet(); metadataColumns.add(new Text("domain")); metadataColumns.add(new Text("link")); HashSet contentColumns = new HashSet(); contentColumns.add(new Text("body")); contentColumns.add(new Text("images")); localityGroups.put("metadata", metadataColumns); localityGroups.put("content", contentColumns); conn.tableOperations().setLocalityGroups("mytable", localityGroups); // existing locality groups can be obtained as follows Map groups = conn.tableOperations().getLocalityGroups("mytable"); The assignment of Column Families to Locality Groups can be changed anytime. The physical movement of column families into their new locality groups takes place via the periodic Major Compaction process that takes place continuously in the background. Major Compaction can also be scheduled to take place immediately through the shell:
user@myinstance mytable> compact -t mytable
6.2 Constraints Accumulo supports constraints applied on mutations at insert time. This can be used to disallow certain inserts according to a user de ned policy.
Any mutation that fails to meet the
requirements of the constraint is rejected and sent back to the client. Constraints can be enabled by setting a table property as follows:
user@myinstance mytable> con g -t mytable -s table.constraint.1=com.test.ExampleConstraint user@myinstance mytable> con g -t mytable -s table.constraint.2=com.test.AnotherConstraint user@myinstance mytable> con g -t mytable -f constraint ---------+--------------------------------+---------------------------24
SCOPE | NAME | VALUE ---------+--------------------------------+---------------------------table | table.constraint.1............ | com.test.ExampleConstraint table | table.constraint.2............ | com.test.AnotherConstraint ---------+--------------------------------+---------------------------Currently there are no general-purpose constraints provided with the Accumulo distribution.
New constraints can be created by writing a Java class that implements the org.apache.accumulo.core.constraints.Con interface. To deploy a new constraint, create a jar le containing the class implementing the new constraint and place it in the lib directory of the Accumulo installation. New constraint jars can be added to Accumulo and enabled without restarting but any change to an existing constraint class requires Accumulo to be restarted. An example of constraints can be found in accumulo/docs/examples/README.constraints with corresponding code under accumulo/src/examples/simple/main/java/accumulo/examples/simple/constraints .
6.3 Bloom Filters As mutations are applied to an Accumulo table, several les are created per tablet. If bloom lters are enabled, Accumulo will create and load a small data structure into memory to determine whether a le contains a given key before opening the le. This can speed up lookups considerably. To enable bloom lters, enter the following command in the Shell:
user@myinstance> con g -t mytable -s table.bloom.enabled=true An extensive example of using Bloom Filters can be found at accumulo/docs/examples/README.bloom .
6.4 Iterators Iterators provide a modular mechanism for adding functionality to be executed by TabletServers when scanning or compacting data. This allows users to e ciently summarize, lter, and aggregate data. In fact, the built-in features of cell-level security and column fetching are implemented using Iterators.
Some useful Iterators are provided with Accumulo and can be
found in the org.apache.accumulo.core.iterators.user package.
6.4.2 Setting Iterators Programmatically scanner.addIterator(new IteratorSetting( 15, // priority "myiter", // name this iterator "com.company.MyIterator" // class name )); Some iterators take additional parameters from client code, as in the following example:
IteratorSetting iter = new IteratorSetting(...); iter.addOption("myoptionname", "myoptionvalue"); scanner.addIterator(iter) Tables support separate Iterator settings to be applied at scan time, upon minor compaction and upon major compaction. For most uses, tables will have identical iterator settings for all three to avoid inconsistent results.
6.4.3 Versioning Iterators and Timestamps Accumulo provides the capability to manage versioned data through the use of timestamps within the Key. If a timestamp is not speci ed in the key created by the client then the system will set the timestamp to the current time. Two keys with identical rowIDs and columns but dierent timestamps are considered two versions of the same key. If two inserts are made into accumulo with the same rowID, column, and timestamp, then the behavior is non-deterministic. Timestamps are sorted in descending order, so the most recent data comes rst. Accumulo can be con gured to return the top k versions, or versions later than a given date. The default is to return the one most recent version. The version policy can be changed by changing the VersioningIterator options for a table as follows:
user@myinstance mytable> con g -t mytable -s table.iterator.scan.vers.opt.maxVersions=3 26
user@myinstance mytable> con g -t mytable -s table.iterator.minc.vers.opt.maxVersions=3 user@myinstance mytable> con g -t mytable -s table.iterator.majc.vers.opt.maxVersions=3
Logical Time Accumulo 1.2 introduces the concept of logical time.
This ensures that timestamps set by
accumulo always move forward. This helps avoid problems caused by TabletServers that have dierent time settings. The per tablet counter gives unique one up time stamps on a per mutation basis. When using time in milliseconds, if two things arrive within the same millisecond then both receive the same timestamp. When using time in milliseconds, accumulo set times will still always move forward and never backwards. A table can be con gured to use logical timestamps at creation time as follows:
user@myinstance> createtable -tl logical
Deletes Deletes are special keys in accumulo that get sorted along will all the other data. When a delete key is inserted, accumulo will not show anything that has a timestamp less than or equal to the delete key. During major compaction, any keys older than a delete key are omitted from the new le created, and the omitted keys are removed from disk as part of the regular garbage collection process.
6.4.4 Filters When scanning over a set of key-value pairs it is possible to apply an arbitrary ltering policy through the use of a Filter. Filters are types of iterators that return only key-value pairs that satisfy the lter logic. Accumulo has a few built-in lters that can be con gured on any table: AgeO, ColumnAgeO, Timestamp, NoVis, and RegEx. More can be added by writing a Java class that extends the org.apache.accumulo.core.iterators.Filter class. The AgeO lter can be con gured to remove data older than a certain date or a xed amount of time from the present. The following example sets a table to delete everything inserted over 30 seconds ago:
27
user@myinstance> createtable ltertest user@myinstance ltertest> setiter -t ltertest -scan -minc -majc -p 10 -n my lter -ageo AgeOFilter removes entries with timestamps more than milliseconds old ----------> set org.apache.accumulo.core.iterators.user.AgeOFilter parameter negate, default false keeps k/v that pass accept m ----------> set org.apache.accumulo.core.iterators.user.AgeOFilter parameter ttl, time to live (milliseconds): 3000 ----------> set org.apache.accumulo.core.iterators.user.AgeOFilter parameter currentTime, if set, use the given value as the abs user@myinstance ltertest> user@myinstance ltertest> scan user@myinstance ltertest> insert foo a b c user@myinstance ltertest> scan foo a:b [] c user@myinstance ltertest> sleep 4 user@myinstance ltertest> scan user@myinstance ltertest> To see the iterator settings for a table, use:
6.4.5 Combiners Accumulo allows Combiners to be con gured on tables and column families. When a Combiner is set it is applied across the values associated with any keys that share rowID, column family, and column quali er.
This is similar to the reduce step in MapReduce, which applied some
function to all the values associated with a particular key. For example, if a summing combiner were con gured on a table and the following mutations were inserted:
28
Row Family Quali er Timestamp Value rowID1 colfA colqA 20100101 1 rowID1 colfA colqA 20100102 1 The table would re ect only one aggregate value:
rowID1 colfA colqA
-
2
Combiners can be enabled for a table using the setiter command in the shell.
Below is an
example.
root@a14 perDayCounts> setiter -t perDayCounts -p 10 -scan -minc -majc -n daycount -class org.apache.accumulo.core.iterators.user.SummingCombiner TypedValueCombiner can interpret Values as a variety of number encodings (VLong, Long, or String) before combining ----------> set SummingCombiner parameter columns, [:]{,[:]} : day ----------> set SummingCombiner parameter type, : STRING root@a14 perDayCounts> insert foo day 20080101 1 root@a14 perDayCounts> insert foo day 20080101 1 root@a14 perDayCounts> insert foo day 20080103 1 root@a14 perDayCounts> insert bar day 20080101 1 root@a14 perDayCounts> insert bar day 20080101 1 root@a14 perDayCounts> scan bar day:20080101 [] 2 foo day:20080101 [] 2 foo day:20080103 [] 1 Accumulo includes some useful Combiners out of the box. To nd these look in the
org.apache.accumulo.core.iterators.user package.
Additional Combiners can be added by creating a Java class that extends
org.apache.accumulo.core.iterators.Combiner
and adding a jar containing that class to
Accumulo's lib/ext directory. An example of a Combiner can be found under accumulo/src/examples/simple/main/java/org/apache/accumulo/examples/simple/combiner/StatsCombiner.java
6.5 Block Cache In order to increase throughput of commonly accessed entries, Accumulo employs a block cache. This block cache buers data in memory so that it doesn't have to be read o of disk.
29
The
RFile format that Accumulo prefers is a mix of index blocks and data blocks, where the index blocks are used to nd the appropriate data blocks. Typical queries to Accumulo result in a binary search over several index blocks followed by a linear scan of one or more data blocks. The block cache can be con gured on a per-table basis, and all tablets hosted on a tablet server share a single resource pool.
To con gure the size of the tablet server's block cache, set the
following properties: tserver.cache.data.size: Speci es the size of the cache for le data blocks. tserver.cache.index.size: Speci es the size of the cache for le indices. To enable the block cache for your table, set the following properties: table.cache.block.enable: Determines whether le (data) block cache is enabled. table.cache.index.enable: Determines whether index cache is enabled. The block cache can have a signi cant eect on alleviating hot spots, as well as reducing query latency. It is enabled by default for the !METADATA table.
6.6 Compaction As data is written to Accumulo it is buered in memory.
The data buered in memory is
eventually written to HDFS on a per tablet basis. Files can also be added to tablets directly by bulk import. In the background tablet servers run major compactions to merge multiple les into one.
The tablet server has to decide which tablets to compact and which les within a
tablet to compact. This decision is made using the compaction ratio, which is con gurable on a per table basis. To con gure this ratio modify the following property: table.compaction.major.ratio Increasing this ratio will result in more les per tablet and less compaction work. More les per tablet means more higher query latency. So adjusting this ratio is a trade o between ingest and query performance. The ratio defaults to 3. The way the ratio works is that a set of les is compacted into one le if the sum of the sizes of the les in the set is larger than the ratio multiplied by the size of the largest le in the set. If this is not true for the set of all les in a tablet, the largest le is removed from consideration, and the remaining les are considered for compaction. This is repeated until a compaction is triggered or there are no les left to consider. The number of background threads tablet servers use to run major compactions is con gurable. To con gure this modify the following property:
30
tserver.compaction.major.concurrent.max Also, the number of threads tablet servers use for minor compactions is con gurable.
To
con gure this modify the following property: tserver.compaction.minor.concurrent.max The numbers of minor and major compactions running and queued is visible on the Accumulo monitor page.
This allows you to see if compactions are backing up and adjustments to the
above settings are needed. When adjusting the number of threads available for compactions, consider the number of cores and other tasks running on the nodes such as maps and reduces. If major compactions are not keeping up, then the number of les per tablet will grow to a point such that query performance starts to suer. One way to handle this situation is to increase the compaction ratio. For example, if the compaction ratio were set to 1, then every new le added to a tablet by minor compaction would immediately queue the tablet for major compaction. So if a tablet has a 200M le and minor compaction writes a 1M le, then the major compaction will attempt to merge the 200M and 1M le. If the tablet server has lots of tablets trying to do this sort of thing, then major compactions will back up and the number of les per tablet will start to grow, assuming data is being continuously written. Increasing the compaction ratio will alleviate backups by lowering the amount of major compaction work that needs to be done. Another option to deal with the les per tablet growing too large is to adjust the following property: table. le.max When a tablet reaches this number of les and needs to ush its in-memory data to disk, it will choose to do a merging minor compaction. A merging minor compaction will merge the tablet's smallest le with the data in memory at minor compaction time. Therefore the number of les will not grow beyond this limit. This will make minor compactions take longer, which will cause ingest performance to decrease. This can cause ingest to slow down until major compactions have enough time to catch up. When adjusting this property, also consider adjusting the compaction ratio. Ideally, merging minor compactions never need to occur and major compactions will keep up. It is possible to con gure the le max and compaction ratio such that only merging minor compactions occur and major compactions never occur. This should be avoided because doing only merging minor compactions causes major compactions is
O(N ∗ logR (N ))
O(N 2 )
where
R
work to be done. The amount of work done by is the compaction ratio.
Compactions can be initiated manually for a table. To initiate a minor compaction, use the ush command in the shell. To initiate a major compaction, use the compact command in the shell. The compact command will compact all tablets in a table to one le. Even tablets with one le are compacted. This is useful for the case where a major compaction lter is con gured for a table. In 1.4 the ability to compact a range of a table was added. To use this feature specify
31
start and stop rows for the compact command. This will only compact tablets that overlap the given row range.
6.7 Pre-splitting tables Accumulo will balance and distribute tables accross servers. Before a table gets large, it will be maintained as a single tablet on a single server. This limits the speed at which data can be added or queried to the speed of a single node. To improve performance when the a table is new, or small, you can add split points and generate new tablets. In the shell:
root@myinstance> createtable newTable root@myinstance> addsplits -t newTable g n t This will create a new table with 4 tablets. The table will be split on the letters g, n, and t which will work nicely if the row data start with lower-case alphabetic characters. If your row data includes binary information or numeric information, or if the distribution of the row information is not at, then you would pick dierent split points. Now ingest and query can proceed on 4 nodes which can improve performance.
6.8 Merging tablets Over time, a table can get very large, so large that it has hundreds of thousands of split points. Once there are enough tablets to spread a table across the entire cluster, additional splits may not improve performance, and may create unnecessary bookkeeping. The distribution of data may change over time. For example, if row data contains date information, and data is continually added and removed to maintain a window of current information, tablets for older rows may be empty. Accumulo supports tablet merging, which can be used to reduce delete the number of split points. The following command will merge all rows from A to Z into a single tablet:
root@myinstance> merge -t myTable -s A -e Z If the result of a merge produces a tablet that is larger than the con gured split size, the tablet may be split by the tablet server. Be sure to increase your tablet size prior to any merges if the goal is to have larger tablets:
root@myinstance> con g -t myTable -s table.split.threshold=2G
32
In order to merge small tablets, you can ask accumulo to merge sections of a table smaller than a given size.
root@myinstance> merge -t myTable -s 100M By default, small tablets will not be merged into tablets that are already larger than the given size. This can leave isolated small tablets. To force small tablets to be merged into larger tablets use the force option:
root@myinstance> merge -t myTable -s 100M --force Merging away small tablets works on one section at a time. If your table contains many sections of small split points, or you are attempting to change the split size of the entire table, it will be faster to set the split point and merge the entire table:
root@myinstance> con g -t myTable -s table.split.threshold=256M root@myinstance> merge -t myTable
6.9 Delete Range Consider an indexing scheme that uses date information in each row. For example 20110823-15:20:25.013 might be a row that speci es a date and time. In some cases, we might like to delete rows based on this date, say to remove all the data older than the current year. Accumulo supports a delete range operation which can e ciently removes data between two rows. For example:
root@myinstance> deleterange -t myTable -s 2010 -e 2011 This will delete all rows starting with 2010 and it will stop at any row starting 2011. You can delete any data prior to 2011 with:
root@myinstance> deleterange -t myTable -e 2011 --force The shell will not allow you to delete an unbounded range (no start) unless you provide the force option. Range deletion is implemented using splits at the given start/end positions, and will aect the number of splits in the table.
6.10 Cloning Tables A new table can be created that points to an existing table's data. metadata operation, no data is actually copied.
This is a very quick
The cloned table and the source table can
change independently after the clone operation. One use case for this feature is testing. For
33
example to test a new ltering iterator, clone the table, add the lter to the clone, and force a major compaction. To perform a test on less data, clone a table and then use delete range to e ciently remove a lot of data from the clone. Another use case is generating a snapshot to guard against human error.
To create a snapshot, clone a table and then disable write
permissions on the clone. The clone operation will point to the source table's les. This is why the ush option is present and is enabled by default in the shell.
If the ush option is not enabled, then any data the
source table currently has in memory will not exist in the clone. A cloned table copies the con guration of the source table.
However the permissions of the
source table are not copied to the clone. After a clone is created, only the user that created the clone can read and write to it. In the following example we see that data inserted after the clone operation is not visible in the clone.
root@a14> createtable people root@a14 people> insert 890435 name last Doe root@a14 people> insert 890435 name rst John root@a14 people> clonetable people test root@a14 people> insert 890436 name rst Jane root@a14 people> insert 890436 name last Doe root@a14 people> scan 890435 name: rst [] John 890435 name:last [] Doe 890436 name: rst [] Jane 890436 name:last [] Doe root@a14 people> table test root@a14 test> scan 890435 name: rst [] John 890435 name:last [] Doe root@a14 test> The du command in the shell shows how much space a table is using in HDFS. This command can also show how much overlapping space two cloned tables have in HDFS. In the example below du shows table ci is using 428M. Then ci is cloned to cic and du shows that both tables share 428M. After three entries are inserted into cic and its ushed, du shows the two tables still share 428M but cic has 226 bytes to itself. Finally, table cic is compacted and then du shows that each table uses 428M.
root@a14> du ci 428,482,573 [ci] root@a14> clonetable ci cic root@a14> du ci cic 34
428,482,573 [ci, cic] root@a14> table cic root@a14 cic> insert r1 cf1 cq1 v1 root@a14 cic> insert r1 cf1 cq2 v2 root@a14 cic> insert r1 cf1 cq3 v3 root@a14 cic> ush -t cic -w 27 15:00:13,908 [shell.Shell] INFO : Flush of table cic completed. root@a14 cic> du ci cic 428,482,573 [ci, cic] 226 [cic] root@a14 cic> compact -t cic -w 27 15:00:35,871 [shell.Shell] INFO : Compacting table ... 27 15:03:03,303 [shell.Shell] INFO : Compaction of table cic completed for given range root@a14 cic> du ci cic 428,482,573 [ci] 428,482,612 [cic] root@a14 cic>
35
Chapter 7
Table Design 7.1 Basic Table Since Accumulo tables are sorted by row ID, each table can be thought of as being indexed by the row ID. Lookups performed row ID can be executed quickly, by doing a binary search, rst across the tablets, and then within a tablet. Clients should choose a row ID carefully in order to support their desired application. A simple rule is to select a unique identi er as the row ID for each entity to be stored and assign all the other attributes to be tracked to be columns under this row ID. For example, if we have the following data in a comma-separated le:
userid,age,address,account-balance We might choose to store this data using the userid as the rowID and the rest of the data in column families:
Mutation m = new Mutation(new Text(userid)); m.put(new Text("age"), age); m.put(new Text("address"), address); m.put(new Text("balance"), account_balance); writer.add(m); We could then retrieve any of the columns for a speci c userid by specifying the userid as the range of a scanner and fetching speci c columns:
Range r = new Range(userid, userid); // single row Scanner s = conn.createScanner("userdata", auths); s.setRange(r); s.fetchColumnFamily(new Text("age")); 36
7.2 RowID Design Often it is necessary to transform the rowID in order to have rows ordered in a way that is optimal for anticipated access patterns. A good example of this is reversing the order of components of internet domain names in order to group rows of the same parent domain together:
com.google.code com.google.labs com.google.mail com.yahoo.mail com.yahoo.research Some data may result in the creation of very large rows - rows with many columns. In this case the table designer may wish to split up these rows for better load balancing while keeping them sorted together for scanning purposes. This can be done by appending a random substring at the end of the row:
com.google.code_00 com.google.code_01 com.google.code_02 com.google.labs_00 com.google.mail_00 com.google.mail_01 It could also be done by adding a string representation of some period of time such as date to the week or month:
com.google.code_201003 com.google.code_201004 com.google.code_201005 com.google.labs_201003 com.google.mail_201003 com.google.mail_201004 Appending dates provides the additional capability of restricting a scan to a given date range.
37
7.3 Indexing In order to support lookups via more than one attribute of an entity, additional indexes can be built. However, because Accumulo tables can support any number of columns without specifying them beforehand, a single additional index will often su ce for supporting lookups of records in the main table. Here, the index has, as the rowID, the Value or Term from the main table, the column families are the same, and the column quali er of the index table contains the rowID from the main table. Key Row ID Term
Column Family
Quali er
Field Name
MainRowID
Visibility
Timestamp
Value
Note: We store rowIDs in the column quali er rather than the Value so that we can have more than one rowID associated with a particular term within the index.
If we stored this in the
Value we would only see one of the rows in which the value appears since Accumulo is con gured by default to return the one most recent value associated with a key. Lookups can then be done by scanning the Index Table rst for occurrences of the desired values in the columns speci ed, which returns a list of row ID from the main table. These can then be used to retrieve each matching record, in their entirety, or a subset of their columns, from the Main Table. To support e cient lookups of multiple rowIDs from the same table, the Accumulo client library provides a BatchScanner. Users specify a set of Ranges to the BatchScanner, which performs the lookups in multiple threads to multiple servers and returns an Iterator over all the rows retrieved. The rows returned are NOT in sorted order, as is the case with the basic Scanner interface.
// rst we scan the index for IDs of rows matching our query Text term = new Text("mySearchTerm"); HashSet matchingRows = new HashSet(); Scanner indexScanner = createScanner("index", auths); indexScanner.setRange(new Range(term, term)); // we retrieve the matching rowIDs and create a set of ranges for(Entry entry : indexScanner) matchingRows.add(new Text(entry.getKey().getColumnQuali er()));
38
// now we pass the set of rowIDs to the batch scanner to retrieve them BatchScanner bscan = conn.createBatchScanner("table", auths, 10); bscan.setRanges(matchingRows); bscan.fetchFamily("attributes"); for(Entry entry : scan) System.out.println(entry.getValue()); One advantage of the dynamic schema capabilities of Accumulo is that dierent elds may be indexed into the same physical table. However, it may be necessary to create dierent index tables if the terms must be formatted dierently in order to maintain proper sort order. For example, real numbers must be formatted dierently than their usual notation in order to be sorted correctly. In these cases, usually one index per unique data type will su ce.
7.4 Entity-Attribute and Graph Tables Accumulo is ideal for storing entities and their attributes, especially of the attributes are sparse. It is often useful to join several datasets together on common entities within the same table. This can allow for the representation of graphs, including nodes, their attributes, and connections to other nodes. Rather than storing individual events, Entity-Attribute or Graph tables store aggregate information about the entities involved in the events and the relationships between entities. This is often preferrable when single events aren't very useful and when a continuously updated summarization is desired. The physical schema for an entity-attribute or graph table is as follows: Key Row ID
Column Visibility
Timestamp
Value
Family
Quali er
EntityID
Attribute Name
Attribute Value
Weight
EntityID
Edge Type
Related EntityID
Weight
For example, to keep track of employees, managers and products the following entity-attribute table could be used. Note that the weights are not always necessary and are set to 0 when not used.
39
RowID ColumnFamily ColumnQualifier Value E001 E001 E001 E001
name department hire_date units_sold
bob sales 20030102 P 001
0 0 0 780
E002 E002 E002 E002
name department manager_of manager_of
george sales E001 E003
0 0 0 0
E003 E003 E003 E003 E003
name department hire_date units_sold units_sold
harry accounts_recv 20000405 P 002 P 001
0 0 0 566 232
P 001 P 001 P 001 P 001
product_name product_type in_stock in_stock
nike_airs shoe germany brazil
0 0 900 200
P 002 P 002 P 002 P 002
product_name product_type in_stock in_stock
basic_jacket clothing usa germany
0 0 3454 700
To allow e cient updating of edge weights, an aggregating iterator can be con gured to add the value of all mutations applied with the same key. These types of tables can easily be created from raw events by simply extracting the entities, attributes, and relationships from individual events and inserting the keys into Accumulo each with a count of 1. The aggregating iterator will take care of maintaining the edge weights.
7.5 Document-Partitioned Indexing Using a simple index as described above works well when looking for records that match one of a set of given criteria.
When looking for records that match more than one criterion
40
simultaneously, such as when looking for documents that contain all of the words `the' and `white' and `house', there are several issues. First is that the set of all records matching any one of the search terms must be sent to the client, which incurs a lot of network tra c. The second problem is that the client is responsible for performing set intersection on the sets of records returned to eliminate all but the records matching all search terms. The memory of the client may easily be overwhelmed during this operation. For these reasons Accumulo includes support for a scheme known as sharded indexing, in which these set operations can be performed at the TabletServers and decisions about which records to include in the result set can be made without incurring network tra c. This is accomplished via partitioning records into bins that each reside on at most one TabletServer, and then creating an index of terms per record within each bin as follows: Key Row ID BinID
Column Family
Quali er
Term
DocID
Visibility
Value
Timestamp
Weight
Documents or records are mapped into bins by a user-de ned ingest application.
By storing
the BinID as the RowID we ensure that all the information for a particular bin is contained in a single tablet and hosted on a single TabletServer since Accumulo never splits rows across tablets. Storing the Terms as column families serves to enable fast lookups of all the documents within this bin that contain the given term. Finally, we perform set intersection operations on the TabletServer via a special iterator called the Intersecting Iterator.
Since documents are partitioned into many bins, a search of all
documents must search every bin. We can use the BatchScanner to scan all bins in parallel. The Intersecting Iterator should be enabled on a BatchScanner within user query code as follows:
Text[] terms = {new Text("the"), new Text("white"), new Text("house")}; BatchScanner bs = conn.createBatchScanner(table, auths, 20); IteratorSetting iter = new IteratorSetting(20, "ii", IntersectingIterator.class); IntersectingIterator.setColumnFamilies(iter, terms); bs.addScanIterator(iter); bs.setRanges(Collections.singleton(new Range())); for(Entry entry : bs) { System.out.println(" " + entry.getKey().getColumnQuali er()); } This code eectively has the BatchScanner scan all tablets of a table, looking for documents
41
that match all the given terms.
Because all tablets are being scanned for every query, each
query is more expensive than other Accumulo scans, which typically involve a small number of TabletServers. This reduces the number of concurrent queries supported and is subject to what is known as the `straggler' problem in which every query runs as slow as the slowest server participating. Of course, fast servers will return their results to the client which can display them to the user immediately while they wait for the rest of the results to arrive. If the results are unordered this is quite eective as the rst results to arrive are as good as any others to the user.
42
Chapter 8
High-Speed Ingest Accumulo is often used as part of a larger data processing and storage system. To maximize the performance of a parallel system involving Accumulo, the ingestion and query components should be designed to provide enough parallelism and concurrency to avoid creating bottlenecks for users and other systems writing to and reading from Accumulo. There are several ways to achieve high ingest performance.
8.1 Pre-Splitting New Tables New tables consist of a single tablet by default.
As mutations are applied, the table grows
and splits into multiple tablets which are balanced by the Master across TabletServers. This implies that the aggregate ingest rate will be limited to fewer servers than are available within the cluster until the table has reached the point where there are tablets on every TabletServer. Pre-splitting a table ensures that there are as many tablets as desired available before ingest begins to take advantage of all the parallelism possible with the cluster hardware. Tables can be split anytime by using the shell:
user@myinstance mytable> addsplits -sf /local_split le -t mytable For the purposes of providing parallelism to ingest it is not necessary to create more tablets than there are physical machines within the cluster as the aggregate ingest rate is a function of the number of physical machines. Note that the aggregate ingest rate is still subject to the number of machines running ingest clients, and the distribution of rowIDs across the table. The aggregation ingest rate will be suboptimal if there are many inserts into a small number of rowIDs.
43
8.2 Multiple Ingester Clients Accumulo is capable of scaling to very high rates of ingest, which is dependent upon not just the number of TabletServers in operation but also the number of ingest clients. This is because a single client, while capable of batching mutations and sending them to all TabletServers, is ultimately limited by the amount of data that can be processed on a single machine.
The
aggregate ingest rate will scale linearly with the number of clients up to the point at which either the aggregate I/O of TabletServers or total network bandwidth capacity is reached. In operational settings where high rates of ingest are paramount, clusters are often con gured to dedicate some number of machines solely to running Ingester Clients. The exact ratio of clients to TabletServers necessary for optimum ingestion rates will vary according to the distribution of resources per machine and by data type.
8.3 Bulk Ingest Accumulo supports the ability to import les produced by an external process such as MapReduce into an existing table.
In some cases it may be faster to load data this way rather than via
ingesting through clients using BatchWriters. This allows a large number of machines to format data the way Accumulo expects. The new les can then simply be introduced to Accumulo via a shell command. To con gure MapReduce to format data in preparation for bulk loading, the job should be set to use a range partitioner instead of the default hash partitioner. The range partitioner uses the split points of the Accumulo table that will receive the data. The split points can be obtained from the shell and used by the MapReduce RangePartitioner. Note that this is only useful if the existing table is already split into multiple tablets.
user@myinstance mytable> getsplits aa ab ac ... zx zy zz Run the MapReduce job, using the AccumuloFileOutputFormat to create the les to be introduced to Accumulo. Once this is complete, the les can be added to Accumulo via the shell:
Note that the paths referenced are directories within the same HDFS instance over which Accumulo is running. Accumulo places any les that failed to be added to the second directory speci ed. A complete example of using Bulk Ingest can be found at accumulo/docs/examples/README.bulkIngest
8.4 Logical Time for Bulk Ingest Logical time is important for bulk imported data, for which the client code may be choosing a timestamp.
At bulk import time, the user can choose to enable logical time for the set of
les being imported. When its enabled, Accumulo uses a specialized system iterator to lazily set times in a bulk imported le. This mechanism guarantees that times set by unsynchronized multi-node applications (such as those running on MapReduce) will maintain some semblance of causal ordering. This mitigates the problem of the time being wrong on the system that created the le for bulk import. These times are not set when the le is imported, but whenever it is read by scans or compactions. At import, a time is obtained and always used by the specialized system iterator to set that time. The timestamp assigned by accumulo will be the same for every key in the le. This could cause problems if the le contains multiple keys that are identical except for the timestamp. In this case, the sort order of the keys will be unde ned. This could occur if an insert and an update were in the same bulk import le.
8.5 MapReduce Ingest It is possible to e ciently write many mutations to Accumulo in parallel via a MapReduce job. In this scenario the MapReduce is written to process data that lives in HDFS and write mutations to Accumulo using the AccumuloOutputFormat. See the MapReduce section under Analytics for details. An example of using MapReduce can be found under accumulo/docs/examples/README.mapred
45
Chapter 9
Analytics Accumulo supports more advanced data processing than simply keeping keys sorted and performing e cient lookups. Analytics can be developed by using MapReduce and Iterators in conjunction with Accumulo tables.
9.1 MapReduce Accumulo tables can be used as the source and destination of MapReduce jobs.
To use an
Accumulo table with a MapReduce job (speci cally with the new Hadoop API as of version 0.20), con gure the job parameters to use the AccumuloInputFormat and AccumuloOutputFormat. Accumulo speci c parameters can be set via these two format classes to do the following:
•
Authenticate and provide user credentials for the input
•
Restrict the scan to a range of rows
•
Restrict the input to a subset of available columns
9.1.1 Mapper and Reducer classes To read from an Accumulo table create a Mapper with the following class parameterization and be sure to con gure the AccumuloInputFormat.
class MyMapper extends Mapper { public void map(Key k, Value v, Context c) { // transform key and value data here 46
}
}
To write to an Accumulo table, create a Reducer with the following class parameterization and be sure to con gure the AccumuloOutputFormat. The key emitted from the Reducer identi es the table to which the mutation is sent. This allows a single Reducer to write to more than one table if desired. A default table can be con gured using the AccumuloOutputFormat, in which case the output table name does not have to be passed to the Context object within the Reducer.
class MyReducer extends Reducer { public void reduce(WritableComparable key, Iterable values, Context c) { Mutation m; // create the mutation based on input key and value
}
}
c.write(new Text("output-table"), m);
The Text object passed as the output should contain the name of the table to which this mutation should be applied. The Text can be null in which case the mutation will be applied to the default table name speci ed in the AccumuloOutputFormat options.
9.1.2 AccumuloInputFormat options Job job = new Job(getConf()); AccumuloInputFormat.setInputInfo(job, "user", "passwd".getBytes(), "table", new Authorizations()); AccumuloInputFormat.setZooKeeperInstance(job, "myinstance", "zooserver-one,zooserver-two");
Optional settings:
To restrict Accumulo to a set of row ranges:
ArrayList ranges = new ArrayList(); // populate array list of row ranges ... AccumuloInputFormat.setRanges(job, ranges); 47
To restrict accumulo to a list of columns:
ArrayList columns = new ArrayList(); // populate list of columns AccumuloInputFormat.fetchColumns(job, columns); To use a regular expression to match row IDs:
AccumuloOutputFormat.setMaxLatency(job, 300); // milliseconds AccumuloOutputFormat.setMaxMutationBuerSize(job, 5000000); // bytes An example of using MapReduce with Accumulo can be found at accumulo/docs/examples/README.mapred
9.2 Combiners Many applications can bene t from the ability to aggregate values across common keys. This can be done via Combiner iterators and is similar to the Reduce step in MapReduce.
This
provides the ability to de ne online, incrementally updated analytics without the overhead or latency associated with batch-oriented MapReduce jobs. All that is needed to aggregate values of a table is to identify the elds over which values will be grouped, insert mutations with those elds as the key, and con gure the table with a combining iterator that supports the summarizing operation desired.
48
The only restriction on an combining iterator is that the combiner developer should not assume that all values for a given key have been seen, since new mutations can be inserted at anytime. This precludes using the total number of values in the aggregation such as when calculating an average, for example.
9.2.1 Feature Vectors An interesting use of combining iterators within an Accumulo table is to store feature vectors for use in machine learning algorithms. For example, many algorithms such as k-means clustering, support vector machines, anomaly detection, etc. use the concept of a feature vector and the calculation of distance metrics to learn a particular model. The columns in an Accumulo table can be used to e ciently store sparse features and their weights to be incrementally updated via the use of an combining iterator.
9.3 Statistical Modeling Statistical models that need to be updated by many machines in parallel could be similarly stored within an Accumulo table. For example, a MapReduce job that is iteratively updating a global statistical model could have each map or reduce worker reference the parts of the model to be read and updated through an embedded Accumulo client. Using Accumulo this way enables e cient and fast lookups and updates of small pieces of information in a random access pattern, which is complementary to MapReduce's sequential access model.
49
Chapter 10
Security Accumulo extends the BigTable data model to implement a security mechanism known as cell-level security.
Every key-value pair has its own security label, stored under the column
visibility element of the key, which is used to determine whether a given user meets the security requirements to read the value. This enables data of various security levels to be stored within the same row, and users of varying degrees of access to query the same table, while preserving data con dentiality.
10.1 Security Label Expressions When mutations are applied, users can specify a security label for each value. This is done as the Mutation is created by passing a ColumnVisibility object to the put() method:
Text rowID = new Text("row1"); Text colFam = new Text("myColFam"); Text colQual = new Text("myColQual"); ColumnVisibility colVis = new ColumnVisibility("public"); long timestamp = System.currentTimeMillis(); Value value = new Value("myValue"); Mutation mutation = new Mutation(rowID); mutation.put(colFam, colQual, colVis, timestamp, value);
50
10.2 Security Label Expression Syntax Security labels consist of a set of user-de ned tokens that are required to read the value the label is associated with. The set of tokens required can be speci ed using syntax that supports logical AND and OR combinations of tokens, as well as nesting groups of tokens together. For example, suppose within our organization we want to label our data values with security labels de ned in terms of user roles. We might have tokens such as:
admin audit system These can be speci ed alone or combined using logical operators:
// Users must have admin privileges: admin // Users must have admin and audit privileges admin&audit // Users with either admin or audit privileges admin|audit // Users must have audit and one or both of admin or system (admin|system)&audit When both | and & operators are used, parentheses must be used to specify precedence of the operators.
10.3 Authorization When clients attempt to read data from Accumulo, any security labels present are examined against the set of authorizations passed by the client code when the Scanner or BatchScanner are created. If the authorizations are determined to be insu cient to satisfy the security label, the value is suppressed from the set of results sent back to the client. Authorizations are speci ed as a comma-separated list of tokens the user possesses:
// user possess both admin and system level access Authorization auths = new Authorization("admin","system"); Scanner s = connector.createScanner("table", auths);
51
10.4 User Authorizations Each accumulo user has a set of associated security labels.
To manipulate these in the shell
use the setuaths and getauths commands. These may also be modi ed using the java security operations API. When a user creates a scanner a set of Authorizations is passed. If the authorizations passed to the scanner are not a subset of the users authorizations, then an exception will be thrown. To prevent users from writing data they can not read, add the visibility constraint to a table. Use the -evc option in the createtable shell command to enable this constraint.
For existing
tables use the following shell command to enable the visibility constraint. Ensure the constraint number does not con ict with any existing constraints.
con g -t table -s table.constraint.1=org.apache.accumulo.core.security.VisibilityConstraint Any user with the alter table permission can add or remove this constraint. This constraint is not applied to bulk imported data, if this a concern then disable the bulk import permission.
10.5 Secure Authorizations Handling For applications serving many users, it is not expected that an accumulo user will be created for each application user. In this case an accumulo user with all authorizations needed by any of the applications users must be created. To service queries, the application should create a scanner with the application users authorizations. These authorizations could be obtained from a trusted 3rd party. Often production systems will integrate with Public-Key Infrastructure (PKI) and designate client code within the query layer to negotiate with PKI servers in order to authenticate users and retrieve their authorization tokens (credentials). This requires users to specify only the information necessary to authenticate themselves to the system.
Once user identity is
established, their credentials can be accessed by the client code and passed to Accumulo outside of the reach of the user.
10.6 Query Services Layer Since the primary method of interaction with Accumulo is through the Java API, production environments often call for the implementation of a Query layer. This can be done using web services in containers such as Apache Tomcat, but is not a requirement. The Query Services
52
Layer provides a mechanism for providing a platform on which user facing applications can be built. This allows the application designers to isolate potentially complex query logic, and enables a convenient point at which to perform essential security functions. Several production environments choose to implement authentication at this layer, where users identi ers are used to retrieve their access credentials which are then cached within the query layer and presented to Accumulo through the Authorizations mechanism. Typically, the query services layer sits between Accumulo and user workstations.
53
Chapter 11
Administration 11.1 Hardware Because we are running essentially two or three systems simultaneously layered across the cluster: HDFS, Accumulo and MapReduce, it is typical for hardware to consist of 4 to 8 cores, and 8 to 32 GB RAM. This is so each running process can have at least one core and 2 - 4 GB each. One core running HDFS can typically keep 2 to 4 disks busy, so each machine may typically have as little as 2 x 300GB disks and as much as 4 x 1TB or 2TB disks. It is possible to do with less than this, such as with 1u servers with 2 cores and 4GB each, but in this case it is recommended to only run up to two processes per machine - i.e. DataNode and TabletServer or DataNode and MapReduce worker but not all three. The constraint here is having enough available heap space for all the processes on a machine.
11.2 Network Accumulo communicates via remote procedure calls over TCP/IP for both passing data and control messages.
In addition, Accumulo uses HDFS clients to communicate with HDFS. To
achieve good ingest and query performance, su cient network bandwidth must be available between any two machines.
54
11.3 Installation Choose a directory for the Accumulo installation.
This directory will be referenced by the
environment variable $ACCUMULO_HOME. Run the following:
$ tar xzf $ACCUMULO_HOME/accumulo.tar.gz Repeat this step at each machine within the cluster.
Usually all machines have the same
$ACCUMULO_HOME.
11.4 Dependencies Accumulo requires HDFS and ZooKeeper to be con gured and running before starting. Password-less SSH should be con gured between at least the Accumulo master and TabletServer machines. It is also a good idea to run Network Time Protocol (NTP) within the cluster to ensure nodes' clocks don't get too out of sync, which can cause problems with automatically timestamped data. Accumulo will remove from the set of TabletServers those machines whose times dier too much from the master's.
11.5 Con guration Accumulo is con gured by editing several Shell and XML les found in $ACCUMULO_HOME/conf. The structure closely resembles Hadoop's con guration les.
11.5.1 Edit conf/accumulo-env.sh Accumulo needs to know where to nd the software it depends on. Edit accumulo-env.sh and specify the following: 1. Enter the location of the installation directory of Accumulo for $ACCUMULO_HOME 2. Enter your system's Java home for $JAVA_HOME 3. Enter the location of Hadoop for $HADOOP_HOME 4. Choose a location for Accumulo logs and enter it for $ACCUMULO_LOG_DIR 5. Enter the location of ZooKeeper for $ZOOKEEPER_HOME
55
By default Accumulo TabletServers are set to use 1GB of memory.
You may change this by
altering the value of $ACCUMULO_TSERVER_OPTS. Note the syntax is that of the Java JVM command line options. This value should be less than the physical memory of the machines running TabletServers. There are similar options for the master's memory usage and the garbage collector process. Reduce these if they exceed the physical RAM of your hardware and increase them, within the bounds of the physical RAM, if a process fails because of insu cient memory. Note that you will be specifying the Java heap space in accumulo-env.sh.
You should make
sure that the total heap space used for the Accumulo tserver and the Hadoop DataNode and TaskTracker is less than the available memory on each slave node in the cluster.
On
large clusters, it is recommended that the Accumulo master, Hadoop NameNode, secondary NameNode, and Hadoop JobTracker all be run on separate machines to allow them to use more heap space. If you are running these on the same machine on a small cluster, likewise make sure their heap space settings t within the available memory.
11.5.2 Cluster Speci cation On the machine that will serve as the Accumulo master: 1. Write the IP address or domain name of the Accumulo Master to the $ACCUMULO_HOME/conf/masters le. 2. Write the IP addresses or domain name of the machines that will be TabletServers in $ACCUMULO_HOME/conf/slaves, one per line. Note that if using domain names rather than IP addresses, DNS must be con gured properly for all machines participating in the cluster. DNS can be a confusing source of errors.
11.5.3 Accumulo Settings Specify appropriate values for the following settings in $ACCUMULO_HOME/conf/accumulo-site.xml :
zookeeper zooserver-one:2181,zooserver-two:2181 list of zookeeper servers walog 56
/var/accumulo/walogs local directory for write ahead logs This enables Accumulo to nd ZooKeeper.
Accumulo uses ZooKeeper to coordinate settings
between processes and helps nalize TabletServer failure. Accumulo records all changes to tables to a write-ahead log before committing them to the table. The `walog' setting speci es the local directory on each machine to which write-ahead logs are written. This directory should exist on all machines acting as TabletServers. Some settings can be modi ed via the Accumulo shell and take eect immediately. However, any settings that should be persisted across system restarts must be recorded in the accumulo-site.xml le.
11.5.4 Deploy Con guration Copy the masters, slaves, accumulo-env.sh, and if necessary, accumulo-site.xml from the $ACCUMULO_HOME/conf/ directory on the master to all the machines speci ed in the slaves le.
11.6 Initialization Accumulo must be initialized to create the structures it uses internally to locate data across the cluster. HDFS is required to be con gured and running before Accumulo can be initialized. Once HDFS is started, initialization can be performed by executing $ACCUMULO_HOME/bin/accumulo init . This script will prompt for a name for this instance of Accumulo. The instance name is used to identify a set of tables and instance-speci c settings. The script will then write some information into HDFS so Accumulo can start properly. The initialization script will prompt you to set a root password. Once Accumulo is initialized it can be started.
57
11.7 Running 11.7.1 Starting Accumulo Make sure Hadoop is con gured on all of the machines in the cluster, including access to a shared HDFS instance. Make sure HDFS and ZooKeeper are running. Make sure ZooKeeper is con gured and running on at least one machine in the cluster.
Start Accumulo using the
bin/start-all.sh script. To verify that Accumulo is running, check the Status page as described under
Monitoring
. In
addition, the Shell can provide some information about the status of tables via reading the !METADATA table.
11.7.2 Stopping Accumulo To shutdown cleanly, run bin/stop-all.sh and the master will orchestrate the shutdown of all the tablet servers. Shutdown waits for all minor compactions to nish, so it may take some time for particular con gurations.
11.8 Monitoring The Accumulo Master provides an interface for monitoring the status and health of Accumulo components. This interface can be accessed by pointing a web browser to http://accumulomaster:50095/status
11.9 Logging Accumulo processes each write to a set of log les. By default these are found under $ACCUMULO/logs/.
11.10 Recovery In the event of TabletServer failure or error on shutting Accumulo down, some mutations may not have been minor compacted to HDFS properly. In this case, Accumulo will automatically reapply such mutations from the write-ahead log either when the tablets from the failed server
58
are reassigned by the Master, in the case of a single TabletServer failure or the next time Accumulo starts, in the event of failure during shutdown. Recovery is performed by asking the loggers to copy their write-ahead logs into HDFS. As the logs are copied, they are also sorted, so that tablets can easily nd their missing updates. The copy/sort status of each le is displayed on Accumulo monitor status page. Once the recovery is complete any tablets involved should return to an online" state. Until then those tablets will be unavailable to clients. The Accumulo client library is con gured to retry failed mutations and in many cases clients will be able to continue processing after the recovery process without throwing an exception. Note that because Accumulo uses timestamps to order mutations, any mutations that are applied as part of the recovery process should appear to have been applied when they originally arrived at the TabletServer that failed. This makes the ordering of mutations consistent in the presence of failure.
59
Appendix A
Shell Commands ? usage: ? [ ] [-?] [-np] [-nw] description: provides information about the available commands -?,--help display this help -np,--no-pagination disables pagination of output -nw,--no-wrap disables wrapping of output about usage: about [-?] [-v] description: displays information about this program -?,--help display this help -v,--verbose displays details session information addsplits usage: addsplits [ ] [-?] [-b64] [-sf ] [-t ] description: add split points to an existing table -?,--help display this help -b64,--base64encoded decode encoded split points -sf,--splits- le le with newline separated list of rows to add to table -t,--table name of a table to add split points to
60
authenticate usage: authenticate [-?] description: veri es a user's credentials -?,--help display this help bye usage: bye [-?] description: exits the shell -?,--help display this help classpath usage: classpath [-?] description: lists the current les on the classpath -?,--help display this help clear usage: clear [-?] description: clears the screen -?,--help display this help clonetable usage: clonetable [-?] [-e ] [-nf ] [-s ] description: clone a table -?,--help display this help -e,--exclude properties that should not be copied from source table. Expects , -nf,--noFlush
do not ush table data in memory before cloning.
-s,--set set initial properties before the table comes online. Expects =,= cls usage: cls [-?]
61
description: clears the screen -?,--help display this help compact usage: compact [-?] [-b ] [-e ] [-nf ] [-p | -t ] [-w] description: sets all tablets for a table to major compact as soon as possible (based on current time) -?,--help display this help -b,--begin-row begin row -e,--end-row end row -nf,--noFlush
do not ush table data in memory before compacting.
-p,--pattern regex pattern of table names to ush -t,--table name of a table to ush -w,--wait
wait for compact to nish
con g usage: con g [-?] [-d | -f | -s ] [-np] [-t
] description: prints system properties and table speci c properties -?,--help display this help -d,--delete delete a per-table property -f,-- lter show only properties that contain this string -np,--no-pagination disables pagination of output -s,--set
set a per-table property
-t,--table
display/set/delete properties for speci ed table createtable usage: createtable [-?] [-a ] [-b64] [-cc
] [-cs
| -sf ] [-evc] [-f ] [-ndi] [-tl | -tm] description: creates a new table, with optional aggregators and optionally pre-split -?,--help display this help -a,--aggregator comma separated column=aggregator -b64,--base64encoded decode encoded split points
62
-cc,--copy-con g
table to copy con guration from -cs,--copy-splits
table to copy current splits from -evc,--enable-visibility-constraint prevents users from writing data they can not read. When enabling this may want to consider disabling bulk import and alter table -f,--formatter default formatter to set -ndi,--no-default-iterators prevents creation of the normal default iterator set -sf,--splits- le le with newline separated list of rows to create a pre-split table -tl,--time-logical use logical time -tm,--time-millis use time in milliseconds createuser usage: createuser [-?] [-s ] description: creates a new user -?,--help display this help -s,--scan-authorizations scan authorizations debug usage: debug [ on | o ] [-?] description: turns debug logging on or o -?,--help display this help delete usage: delete [-?] [-l ] [-t ] description: deletes a record from a table -?,--help display this help -l,--authorization-label
formatted authorization label expression
-t,--timestamp timestamp to use for insert deleteiter usage: deleteiter [-?] [-majc] [-minc] -n [-scan] [-t
] description: deletes a table-speci c iterator -?,--help display this help -majc,--major-compaction applied at major compaction
63
-minc,--minor-compaction applied at minor compaction -n,--name iterator to delete -scan,--scan-time applied at scan time -t,--table
] description: scans a table and deletes the resulting records -?,--help display this help -b,--begin-row begin row (inclusive) -c,--columns comma-separated columns -e,--end-row end row (inclusive) -f,--force forces deletion without prompting -fm,--formatter fully quali ed name of the formatter class to use -np,--no-pagination disables pagination of output -r,--row row to scan -s,--scan-authorizations scan authorizations (all user auths are used if this argument is not speci ed) -st,--show-timestamps enables displaying timestamps -t,--table
table to be created deleterows usage: deleterows [-?] [-b ] [-e ] [-f ] [-t
] description: delete a range of rows in a table. Note that rows matching the start row ARE NOT deleted, but rows matching the end row ARE deleted. -?,--help display this help -b,--begin-row begin row -e,--end-row end row -f,--force delete data even if start or end are not speci ed -t,--tableName
table to delete row range deletescaniter
64
usage: deletescaniter [-?] [-a] [-n ] [-t
] description: deletes a table-speci c scan iterator so it is no longer used during this shell session -?,--help display this help -a,--all delete all for tableName -n,--name iterator to delete -t,--table
tableName deletetable usage: deletetable [-?] [-t ] description: deletes a table -?,--help display this help -t,--tableName deletes a table deleteuser usage: deleteuser [-?] description: deletes a user -?,--help display this help droptable usage: droptable [-?] [-t ] description: deletes a table -?,--help display this help -t,--tableName deletes a table dropuser usage: dropuser [-?] description: deletes a user -?,--help display this help du usage: du
[-?] [-p ] description: Prints how much space is used by les referenced by a table. When multiple tables are speci ed it prints how much space is used by les shared between tables, if any.
] description: searches each row, column family, column quali er and value, in parallel, on the server side (using a java Matcher, so put .* before and after your term if you're not matching the whole element) -?,--help display this help -b,--begin-row begin row (inclusive) -c,--columns comma-separated columns -e,--end-row end row (inclusive) -f,--show few Only shows certain amount of characters -fm,--formatter fully quali ed name of the formatter class to use -np,--no-pagination disables pagination of output -nt,--num-threads num threads -r,--row row to scan -s,--scan-authorizations scan authorizations (all user auths are used if this argument is not speci ed) -st,--show-timestamps enables displaying timestamps -t,--tableName
table to grep through exec le usage: exec le [-?] [-v] description: speci es a le containing accumulo commands to execute -?,--help display this help -v,--verbose displays command prompt as commands are executed exit usage: exit [-?] description: exits the shell -?,--help display this help
66
ush usage: ush [-?] [-b ] [-e ] [-p | -t ]
[-w]
description: ushes a tables data that is currently in memory to disk -?,--help display this help -b,--begin-row begin row -e,--end-row end row -p,--pattern regex pattern of table names to ush -t,--table name of a table to ush -w,--wait
wait for ush to nish
formatter usage: formatter [-?] -f | -l | -r [-t
] description: speci es a formatter to use for displaying table entries -?,--help display this help -f,--formatter fully quali ed name of the formatter class to use -l,--list display the current formatter -r,--remove remove the current formatter -t,--table
table to set the formatter on getauths usage: getauths [-?] [-u ] description: displays the maximum scan authorizations for a user -?,--help display this help -u,--user user to operate on getgroups usage: getgroups [-?] [-t
] description: gets the locality groups for a given table -?,--help display this help -t,--table
get locality groups for speci ed table getsplits usage: getsplits [-?] [-b64] [-m ] [-o ] [-t
] [-v] description: retrieves the current split points for tablets in the current table
67
-?,--help display this help -b64,--base64encoded encode the split points -m,--max speci es the maximum number of splits to create -o,--output speci es a local le to write the splits to -t,--tableName
table to get splits on -v,--verbose print out the tablet information with start/end rows grant usage: grant [-?] -p | -s | -t
-u description: grants system or table permissions for a user -?,--help display this help -p,--pattern regex pattern of tables to grant permissions on -s,--system grant a system permission -t,--table
grant a table permission on this table -u,--user user to operate on grep usage: grep [-?] [-b ] [-c ] [-e ] [-f ] [-fm ] [-np] [-nt ] [-r ] [-s ] [-st] [-t
] description: searches each row, column family, column quali er and value in a table for a substring (not a regular expression), in parallel, on the server side -?,--help display this help -b,--begin-row begin row (inclusive) -c,--columns comma-separated columns -e,--end-row end row (inclusive) -f,--show few Only shows certain amount of characters -fm,--formatter fully quali ed name of the formatter class to use -np,--no-pagination disables pagination of output -nt,--num-threads num threads -r,--row row to scan -s,--scan-authorizations scan authorizations (all user auths are used if this argument is not speci ed) -st,--show-timestamps enables displaying timestamps
68
-t,--tableName
table to grep through help usage: help [ ] [-?] [-np] [-nw] description: provides information about the available commands -?,--help display this help -np,--no-pagination disables pagination of output -nw,--no-wrap disables wrapping of output history usage: history [-?] [-c] description: Generates a list of commands previously executed -?,--help display this help -c,--Clears History, takes no arguments. Clears History File importdirectory usage: importdirectory true|false [-?] description: bulk imports an entire directory of data les to the current table. The boolean argument determines if accumulo sets the time. -?,--help display this help info usage: info [-?] [-v] description: displays information about this program -?,--help display this help -v,--verbose displays details session information insert usage: insert [-?] [-l ] [-t ] description: inserts a record -?,--help display this help -l,--authorization-label
] description: lists table-speci c iterators -?,--help display this help -majc,--major-compaction applied at major compaction -minc,--minor-compaction applied at minor compaction -n,--name iterator to delete -scan,--scan-time applied at scan time -t,--table
tableName listscans usage: listscans [-?] [-np] [-ts ] description: list what scans are currently running in accumulo. See the accumulo.core.client.admin.ActiveScan javadoc for more information about columns. -?,--help display this help -np,--no-pagination disables pagination of output -ts,--tabletServer list scans for a speci c tablet server masterstate usage: masterstate is deprecated, use the command line utility instead [-?] description: DEPRECATED: use the command line utility instead -?,--help display this help maxrow usage: maxrow [-?] [-b ] [-be] [-e ] [-ee] [-s ] [-t
] description: nd the max row in a table within a given range -?,--help display this help -b,--begin-row begin row -be,--begin-exclusive make start row exclusive, by defaults it inclusive -e,--end-row end row -ee,--end-exclusive make end row exclusive, by defaults it inclusive -s,--scan-authorizations scan authorizations (all user auths are used if this argument is not speci ed) -t,--table
] [-v] description: merge tablets in a table -?,--help display this help -b,--begin-row begin row -e,--end-row end row -f,--force merge small tablets to large tablets, even if it goes over the given size -s,--size merge tablets to the given size over the entire table -t,--tableName
table to be merged -v,--verbose verbose output during merge notable usage: notable [-?] [-t ] description: returns to a tableless shell state -?,--help display this help -t,--tableName Returns to a no table state o ine usage: o ine [-?] [-p | -t ] description: starts the process of taking table o ine -?,--help display this help -p,--pattern regex pattern of table names to ush -t,--table name of a table to ush online usage: online [-?] [-p | -t ] description: starts the process of putting a table online -?,--help display this help -p,--pattern regex pattern of table names to ush -t,--table name of a table to ush passwd usage: passwd [-?] [-u ]
71
description: changes a user's password -?,--help display this help -u,--user user to operate on quit usage: quit [-?] description: exits the shell -?,--help display this help renametable usage: renametable [-?] description: rename a table -?,--help display this help revoke usage: revoke [-?] -s | -t
-u
description: revokes system or table permissions from a user -?,--help display this help -s,--system revoke a system permission -t,--table
revoke a table permission on this table -u,--user user to operate on scan usage: scan [-?] [-b ] [-c ] [-e ] [-f ] [-fm ] [-np] [-r ] [-s ] [-st] [-t
] description: scans the table, and displays the resulting records -?,--help display this help -b,--begin-row begin row (inclusive) -c,--columns comma-separated columns -e,--end-row end row (inclusive) -f,--show few Only shows certain amount of characters -fm,--formatter fully quali ed name of the formatter class to use
72
-np,--no-pagination disables pagination of output -r,--row row to scan -s,--scan-authorizations scan authorizations (all user auths are used if this argument is not speci ed) -st,--show-timestamps enables displaying timestamps -t,--tableName
table to be scanned select usage: select [-?] [-np] [-s ] [-st] [-t
] description: scans for and displays a single record -?,--help display this help -np,--no-pagination disables pagination of output -s,--scan-authorizations scan authorizations -st,--show-timestamps enables displaying timestamps -t,--tableName
] description: scans a single row and displays all resulting records -?,--help display this help -np,--no-pagination disables pagination of output -s,--scan-authorizations scan authorizations -st,--show-timestamps enables displaying timestamps -t,--tableName
table to row select setauths usage: setauths [-?] -c | -s [-u ] description: sets the maximum scan authorizations for a user -?,--help display this help -c,--clear-authorizations clears the scan authorizations -s,--scan-authorizations set the scan authorizations -u,--user user to operate on setgroups
73
usage: setgroups =, =, [-?] [-t
] description: sets the locality groups for a given table (for binary or commas, use Java API) -?,--help display this help -t,--table
get locality groups for speci ed table setiter usage: setiter [-?] -ageo | -agg | -class | -regex | -reqvis | -vers [-majc] [-minc] [-n ] -p [-scan] [-t
] description: sets a table-speci c iterator -?,--help display this help -ageo,--ageo an aging o type -agg,--aggregator an aggregating type -class,--class-name a java class type -majc,--major-compaction applied at major compaction -minc,--minor-compaction applied at minor compaction -n,--name iterator to set -p,--priority the order in which the iterator is applied -regex,--regular-expression a regex matching type -reqvis,--require-visibility a type that omits entries with empty visibilities -scan,--scan-time applied at scan time -t,--table
] description: sets a table-speci c scan iterator for this shell session -?,--help display this help -ageo,--ageo an aging o type -agg,--aggregator an aggregating type -class,--class-name a java class type -n,--name iterator to set -p,--priority the order in which the iterator is applied -regex,--regular-expression a regex matching type -reqvis,--require-visibility a type that omits entries with empty visibilities
74
-t,--table
tableName -vers,--version a versioning type sleep usage: sleep [-?] description: sleep for the given number of seconds -?,--help display this help systempermissions usage: systempermissions [-?] description: displays a list of valid system permissions -?,--help display this help table usage: table [-?] description: switches to the speci ed table -?,--help display this help tablepermissions usage: tablepermissions [-?] description: displays a list of valid table permissions -?,--help display this help tables usage: tables [-?] [-l] description: displays a list of all existing tables -?,--help display this help -l,--list-ids display internal table ids along with the table name trace usage: trace [ on | o ] [-?] description: turns trace logging on or o -?,--help display this help
75
user usage: user [-?] description: switches to the speci ed user -?,--help display this help userpermissions usage: userpermissions [-?] [-u ] description: displays a user's system and table permissions -?,--help display this help -u,--user user to operate on users usage: users [-?] description: displays a list of existing users -?,--help display this help whoami usage: whoami [-?] description: reports the current user name -?,--help display this help
![[PDF] Download Accumulo: Application Development ... - Google Sites](https://m.moam.info/img/260x300/pdf-download-accumulo-application-development-goog_6477f0ac097c4786708c44b0.jpg)
