Category Archives: HBase

Delete snapshots older than 7 days

Posted on 2023 年 11 月 16 日 by cowman.chiang

To avoid running out of disk space in our test environment, we developed a plan to regularly execute shell scripts to clean up unnecessary snapshots.

#!/bin/bash

week=`date --date='7 days ago' +'%Y%m%d'`
echo "list_snapshots" | /home/webuser/hbase-1.2.9/bin/hbase shell | grep "pattern " | \
while read CMD; do
    filename=($CMD)
#    echo $filename
    date=`echo $filename | awk -F "_" '{print $2}'`
#    echo  "${filename#*_}"
#    echo $date
#    echo ${date:0:8}
    if [ "${date:0:8}" -lt $week ]
    then
        echo "delete_snapshot '$filename'" | /home/webuser/hbase-1.2.9/bin/hbase shell
    fi
done

#!/bin/bash

week=`date --date='7 days ago' +'%Y%m%d'`

echo "list_snapshots" | /home/webuser/hbase-1.2.9/bin/hbase shell | grep "pattern " | \

while read CMD; do

filename=($CMD)

# echo $filename

date=`echo $filename | awk -F "_" '{print $2}'`

# echo "${filename#*_}"

# echo $date

# echo ${date:0:8}

if [ "${date:0:8}" -lt $week ]

then

echo "delete_snapshot '$filename'" | /home/webuser/hbase-1.2.9/bin/hbase shell

done

Posted in HBase, Shell Script | Comments Off

HBase Client 2.5.5 in JDK 11

Posted on 2023 年 10 月 02 日 by cowman.chiang

Because of the vulnerability scan results of the project, the JDK version must be upgraded from 8 to 11 to complete the repair. The original HBase Client version 1.2 we used was incompatible with JDK 11, so we had to … Continue reading →

Posted in HBase, Java | Comments Off

[HBase] security issue

Posted on 2019 年 03 月 18 日 by cowman.chiang

error message: com.a.b.c.exception.BaseDAOException: org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user ‘Cowman’ (table=ABC, action=READ) in jvm conf. add “-DHADOOP_USER_NAME=webuser”

Posted in HBase, Java | Leave a comment

[HBase] convert long value to bytes

Posted on 2018 年 11 月 13 日 by cowman.chiang

Java code:

public static void main(String[] args) {
    String stringValue = Bytes.toStringBinary(Bytes.toBytes(1532080782183l));
    System.out.println(stringValue);
    Long longValue = Bytes.toLong(Bytes.toBytesBinary(stringValue));
    System.out.println(longValue);
}

public static void main(String[] args) {

String stringValue = Bytes.toStringBinary(Bytes.toBytes(1532080782183l));

System.out.println(stringValue);

Long longValue = Bytes.toLong(Bytes.toBytesBinary(stringValue));

System.out.println(longValue);

}

HBase shell command:

hbase(main):056:0> Bytes.toStringBinary(Bytes.to_bytes(1532080782183))
=> "\\x00\\x00\\x01d\\xB7!{g"
hbase(main):057:0> Bytes.toLong("\x00\x00\x01d\xB7!{g".to_java_bytes)
=> 1532080782183

hbase(main):056:0> Bytes.toStringBinary(Bytes.to_bytes(1532080782183))

=> "\\x00\\x00\\x01d\\xB7!{g"

hbase(main):057:0> Bytes.toLong("\x00\x00\x01d\xB7!{g".to_java_bytes)

=> 1532080782183

Posted in HBase, Java | Leave a comment

[HBase] Filters not working for negative integers

Posted on 2018 年 07 月 12 日 by cowman.chiang

[stackoverflow] HBase: Filters not working for negative integers Easility say: Since, Hbase has only BinaryComparators and not other ‘typed’ comparators, it fails to filter on Negative integers as it stores the 2’s compliment of the negative number. Further, the binary … Continue reading →

Posted in HBase | Leave a comment

[HBase] get the scan result without specific cq

Posted on 2018 年 06 月 06 日 by cowman.chiang

scan ‘TableName’, FILTER=>”QualifierFilter(!=,’binary:QUALIFY2′)”

Posted in HBase | Leave a comment

Install Hbase 5.9 in Mac OS X

Posted on 2017 年 03 月 10 日 by cowman.chiang

Download hbase package tar.gz file from https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball_59.html Untar tar.gz file edit conf/hbase-env.sh export JAVA_HOME={{JAVA_HOME Directory path}} edit conf/hbase-site.xml

  <property>
    <name>hbase.rootdir</name>
    <value>file:///{{location}}/data</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>{{location}}/zookeeper</value>
  </property>
  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>{{hostname}}</value>
  </property>

<name>hbase.rootdir</name>

<value>file:///{{location}}/data</value>

</property>

<name>hbase.zookeeper.property.dataDir</name>

<value>{{location}}/zookeeper</value>

</property>

<name>hbase.zookeeper.quorum</name>

<value>{{hostname}}</value>

</property>

start hbase service bin/start-hbase.sh run hbase shell bin/hbase shell stop hbase shell service bin/stop-hbase.sh

Posted in HBase, Mac | Leave a comment

HBase: put byte value

Posted on 2016 年 10 月 04 日 by cowman.chiang

put “TableName”, “rowkey”, “cf:fieldname”, [0].pack(“N”) N => byte array put “TableName”, “rowkey”, “cf:fieldname”, [0].pack(“Q>”) Q => 64-bit unsigned, native endian => change endian to big endian

Posted in HBase | Leave a comment

[HBase] Scan with filter by partial rowkey using HBase shell

Posted on 2016 年 05 月 26 日 by cowman.chiang

If we have these rowkey list data. xx_abc xx_abc_def xx_abd xx_abd_rfg To get rowkey which contain “abc” scan ‘TableName’, FILTER => “RowFilter(=, ‘substring:abc’)” We will get these. xx_abc xx_abc_def But if we want only get “xx_abc” scan ‘TableName’, FILTER => … Continue reading →

Posted in HBase | Leave a comment

[HBase] get/scan String or Integer value

Posted on 2016 年 04 月 22 日 by cowman.chiang

get String Value

get 'Table_Name', 'RowKey', {COLUMNS=&gt;'Column_Family:Qualify:toString'}

1 2	get 'Table_Name', 'RowKey', {COLUMNS=>'Column_Family:Qualify:toString'}

get Integer Value

get 'Table_Name', 'RowKey', {COLUMNS=&gt;'Column_Family:Qualify:toInt'}

1 2	get 'Table_Name', 'RowKey', {COLUMNS=>'Column_Family:Qualify:toInt'}

scan and get String Value

scan 'Table_Name', {COLUMNS=&gt;'Column_Family:Qualify:toString'}

1 2	scan 'Table_Name', {COLUMNS=>'Column_Family:Qualify:toString'}

scan and get Integer Value

scan 'Table_Name', {COLUMNS=&gt;'Column_Family:Qualify:toInt'}

1 2	scan 'Table_Name', {COLUMNS=>'Column_Family:Qualify:toInt'}

Posted in HBase | Leave a comment

Category Archives: HBase

Categories