How to quickly benchmark mysql with mysqlslap

MySQL client also provides a benchmarking tool called mysqlslap which could be installed on CentOS by:

sudo yum install mysql

If you just want to quickly benchmark some database with a simple query, you could try something like:

mysqlslap --user=[YourUsername] --password=[YourPassword] --host=[YourHostname] --verbose --concurrency 10 --iterations 10 --query 'SELECT * from [YourDatabaseName].lookup_en' --create-schema=[YourDatabaseName]

If you want to know more options, please refer to the Reference.


Reference:
http://dev.mysql.com/doc/refman/5.7/en/mysqlslap.html

How to use Eclipse with an existing sbt project

Assuming there is a sbt project, and now you want to use Eclipse to maintain the project.

(1) Install sbteclipse plugin in your sbt project to generate Eclipse project configs
You should follow the instruction on (use "sbt sbtVersion" to get your sbt version):
https://github.com/typesafehub/sbteclipse
but I did not manage to install the sbteclipse plugin globally for any sbt project, so I had to install it per project, i.e., adding

addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0")

to PROJECT_DIR/project/plugins.sbt in your sbt project.
After sbteclipse is installed, you could run:

sudo sbt eclipse

to generate project config files for Eclipse:

    .classpath
    .project
    .settings/

(2) Import the project into Eclipse
Use menu:

File → Import → Existing Projects into Workspace

(3) How to use it
On the command line, you should run:
sudo sbt ~compile
in the root directory of your sbt project such that sbt could automatically run when you make any changes.

How to find ranking of conferences and journals

(1) conference rankings

(1.1) Google Scholar offers conference rankings in various fields, e.g., the rankings of conferences and journals in the Computational Linguistics field could be found here (other fields could be found on the left category tree)

(1.2) Microsoft Academic Search also offers conference rankings in different fields, e.g., ranking of conferences in the Natural Language & Speech field could be found here.

(1.3) conference ranking in the Computer Science field could also be found on the Computing Research & Education Conference Portal

(2) journals

(2.1) free journal rankings
Scimago Journal & Country Rank

(2.2) non-free ones
Thomson Reuters Journal Citation Reports® (most of the universities have this)

How to use sqoop to copy MySQL tables to Hive

sqoop is useful when you need to copy MySQL tables to Hive

Here is an example to copy MySQL tables from different shards to one Hive table:

 #!/bin/bash  
   
 set -x  
 set -e  
   
 game=gameexample  
   
 mysql_host_prefix=userdb  
 mysql_host_suffix=myhostname.com  
 mysql_tab=user  
 mysql_database=mydatabase  
 mysql_user=myusername  
 mysql_pwd=xxxxx  
   
 hive_tab=user  
   
 echo "Log: `date` dropping Hive table $game.${hive_tab}"  
 hive -S -e "drop table \`$game.${hive_tab}\`;"   
   
 for SHARD_ID in {1..200}; do  
   
   mysql_host=${mysql_host_prefix}-${SHARD_ID}-${mysql_host_suffix}  
   mysql_conn="mysql --user=$mysql_user --password=$mysql_pwd -D${mysql_database} --host=${mysql_host} -s --skip-column-names"  
   hive_shard_tab=${hive_tab}_shard${SHARD_ID}  
   hdfs_dir=/user/mapr/${mysql_tab}_shard${SHARD_ID}  
   if ping -c 1 -W 1 "$mysql_host"; then  
     echo "Log: `date` $mysql_host is alive"  
   else  
     echo "Log: `date` $mysql_host is not alive"  
     exit 0  
   fi   
   
   echo "Log: `date` dropping Hive table $game.${hive_shard_tab}"  
   hive -S -e "drop table $game.${hive_shard_tab}"   
   echo "Log: `date` removing ${hdfs_dir} on HDFS"  
   hadoop fs -rm -r -f ${hdfs_dir}  
     
   sql="select count(*) from \`${mysql_tab}\`"  
   mysql_row_cnt=`echo "$sql" | $mysql_conn`  
   echo "Log: `date` found ${mysql_row_cnt} rows in the MySQL table ${mysql_tab} with query: $sql"  
     
   sqoop import \  
    --connect jdbc:mysql://$mysql_host/${mysql_database} \  
    --table "${mysql_tab}" \  
    --username $mysql_user \  
    --password $mysql_pwd \  
    --num-mappers 1 \  
    --hive-overwrite \  
    --hive-table $game.${hive_shard_tab} \  
    --hive-import \  
    --target-dir ${hdfs_dir} \  
    --hive-delims-replacement ' '   
     
   hive_row_cnt=`hive -S -e "select count(*) from $game.${hive_shard_tab}"`  
   echo "Log: `date` ended up with ${hive_row_cnt} rows in the Hive table $game.${hive_shard_tab} which are copied from the MySQL table ${mysql_tab} (${mysql_row_cnt} rows)"  
     
   # merging  
   if [ $SHARD_ID = 1 ]; then  
      sql_str="create table $game.\`$hive_tab\` as select * from $game.${hive_shard_tab};"  
      echo "Log: `date` creating the Hive table $game.${hive_tab} with the data from the first Shard with sql: $sql_str"  
      hive -S -e "$sql_str"   
   else  
      sql_str="insert into table $game.\`$hive_tab\` select * from $game.${hive_shard_tab};"  
      echo "Log: `date` merging into the Hive table $game.${hive_tab} the data from Shard $SHARD_ID with sql: $sql_str"  
      hive -S -e "$sql_str"   
   fi  
 done  
 exit 0  
   

How to use ngrep

ngrep is a very useful tool on Linux to capture TCP packages for a given host, a given port number, or a given key word.

(1) to capture packages (printed in hex format) from port 1234 with keyword "my-word" (using network device bond0 (see ifconfig to pick a device))

 sudo ngrep -l -t -d bond0 -q -x my-word port 1234

(2) to capture packages to a host my.hostname.com

sudo ngrep -l -t -d bond0 -q -W byline host my.hostname.com

How to find your citations

As a researcher, you may want to find all the papers which cite to your papers. Some ways are listed here:
(1) Google Scholar: is a good place but it may ignore some minor sources, e.g., some workshop papers;
(2) ResearchGate: you could upload your papers to it, and it could filter papers to find out what papers cite to your papers;
(3) Microsoft Academy: you could search for your papers first and then it could show you what papers cite to your papers;
(4) CiteSeerX

How to split terminal on remote server with background sessions

Screen

screen is a useful command to leave some processes running in the background on Linux.
We could actually split the screen terminal into sub-terminals with the following short-cuts:
(1) to split a terminal horizontally: press Ctrl+A, release them, press Shift+s
(2) to split a terminal vertically: press Ctrl+A, release them, press Shift+\
(3) to switch among sub-terminals: press Ctrl+A, release them, press Tab
But please note that the sub-terminals would be converted to background windows in screen if you detach (Ctrl+A+D) your screen session and then attach it again. To avoid this, you could use tmux instead.

Tmux

Tmux is more convenient than screen on this way.
type "tmux new -s sessionname" to create a new session
in the session, you could use:
(1) to split a terminal horizontally: press Ctrl+B, release them, press % (Shift+5);
(2) to split a terminal vertically: press Ctrl+B, release them, press " (Shift+');
(3) to detach the current session: press Ctrl+B, release them, press D;
(4) to attach to a session: "tmux attach -t sessionname";
(5) to switch to a pane: Ctrl+B, release them, press q (Show pane numbers, when the numbers show up type the key to go to that pane).

Reference:

http://fosshelp.blogspot.com/2014/02/how-to-linux-terminal-split-screen-with.html
https://www.youtube.com/watch?v=BHhA_ZKjyxo
https://gist.github.com/MohamedAlaa/2961058