Showing posts with label linux. Show all posts
Showing posts with label linux. Show all posts

May 6, 2013

偵測目錄變動並自動執行指令

最近比較少用 IDE 後,coding 少了些自動化的回饋機制總覺得缺了點什麼,手動執行不知不覺也浪費了許多時間

想了想, 有些重複流程是可以用程式自動化的, 只要有程式來偵測目錄變動並自動執行程式就夠了,在 linux 裡確沒找到相關的工具(inotify-tools), 於是用 Go 做了一個命令列工具 watchf ,能彈性組合命令, 也可以在多個作業系統上使用

Usage

Usage:
  watchf [options]
Options:
  -V=false: Show debugging messages
  -c=[]: Add arbitrary command (repeatable)
  -e=[all]: Listen for specific event(s) (comma separated list)
  -f=".watchf.conf": Specifies a configuration file
  -i=0: The interval limit the frequency of the command executions, if equal to 0, there is no limit (time unit: ns/us/ms/s/m/h)
  -p=".*": File name matches regular expression pattern (perl-style)
  -r=false: Watch directories recursively
  -s=false: Stop the watchf Daemon (windows is not support)
  -v=false: Show version and exit
  -w=false: Write command-line arguments to configuration file (write and exit)
Events:
     all  Create/Delete/Modify/Rename
  create  File/directory created in watched directory
  delete  File/directory deleted from watched directory
  modify  File was modified or Metadata changed
  rename  File moved out of watched directory
Variables:
  %f: The filename of changed file
  %t: The event type of file changes

Example 1

程式碼存檔時從另一個 terminal 看到程式碼錯誤分析、compiler 編譯訊息跟測試結果

以Go為例:

watchf -e "modify,delete" -c "go vet" -c "go test" -c "go install" -p "\.go$"

以Java為例:

watchf -r -i 5s -e "modify,delete" -c "mvn test" -p "\.java$"

Example 2

放置一個背景程式&偵測當前目錄與子目錄的檔案變動, 接著自動同步檔案到另一個目錄(或著遠端目錄)

watchf -r -c "rsync -aq $SRC $DST" &

停掉背景程式, 只要在同個執行目錄裡下watchf -s命令

Example 3

將常用參數寫入設定檔,之後只要在同個執行目錄裡下watchf命令不用給定參數

watchf -e "modify,delete" -c "go vet" -c "go test" -c "go install" -p "\.go$" -w

Apr 3, 2013

ab 漸進壓力測試工具

最近在學Go,也用Go寫一個 ab-like 的玩具(Go-HttpBench),為了要比較兩者效能另外寫了 bash script 用來轉換 ab 報表為行列數據,功能跟 autobench 相似,用階段性漸增的壓力來測試 http server,搜集到的數據可再用 gnuplot 或用 excel 繪制圖型

Example:

brandon@brandon-laptop:~/workspaces/go/gb$ ./ab+.sh -k http://localhost/
Start benchmark
#/usr/bin/ab -c CONCURRENCY=(100 to 10000 step 100) -n NUM=(100*CONCURRENCY) -k http://localhost/
#start time,concurrency,complete requests,failed requests,tps,min,mean,stddev,median,max,90%,95%,99%
Wed Apr  3 11:52:29 CST 2013,100,10000,0,31272.48,0,3,5.4,0,32,11,16,22
Wed Apr  3 11:52:29 CST 2013,200,20000,0,32050.72,0,6,14.2,0,113,23,40,64
Wed Apr  3 11:52:30 CST 2013,300,30000,0,33527.94,0,5,11.6,0,207,16,28,53
Wed Apr  3 11:52:31 CST 2013,400,40000,0,32396.87,0,12,25.0,0,189,48,70,105
Wed Apr  3 11:52:32 CST 2013,500,50000,0,32492.75,0,15,35.9,0,391,51,109,166
Wed Apr  3 11:52:34 CST 2013,600,60000,0,33188.85,0,18,38.8,0,448,73,110,166
Wed Apr  3 11:52:36 CST 2013,700,70000,0,33436.38,0,21,46.9,0,483,71,134,221
Wed Apr  3 11:52:38 CST 2013,800,80000,0,33370.64,0,24,54.5,0,1015,74,145,264
Wed Apr  3 11:52:40 CST 2013,900,90000,0,34093.27,0,26,66.4,0,1150,78,133,260
Wed Apr  3 11:52:43 CST 2013,1000,100000,0,34448.17,0,27,68.4,2,1251,85,136,266
Wed Apr  3 11:52:46 CST 2013,1100,110000,0,34072.87,0,32,78.1,0,1129,101,170,368
Wed Apr  3 11:52:49 CST 2013,1200,120000,0,33332.79,0,36,86.2,0,1032,144,228,386
Wed Apr  3 11:52:53 CST 2013,1300,130000,0,34580.66,0,37,96.0,5,3349,116,201,346
Wed Apr  3 11:52:57 CST 2013,1400,140000,0,33776.24,0,41,96.3,0,689,153,262,475
Wed Apr  3 11:53:01 CST 2013,1500,150000,0,31418.26,0,47,114.7,0,1031,164,295,533
Wed Apr  3 11:53:06 CST 2013,1600,160000,0,33285.65,0,48,111.8,0,629,217,340,492
Wed Apr  3 11:53:11 CST 2013,1700,170000,0,33441.14,0,50,121.5,0,825,211,356,559
Wed Apr  3 11:53:16 CST 2013,1800,180000,0,31452.71,0,52,166.0,0,4253,163,294,496
Wed Apr  3 11:53:22 CST 2013,1900,190000,0,33781.89,0,56,139.8,0,1018,234,425,627
Wed Apr  3 11:53:28 CST 2013,2000,200000,0,33766.12,0,59,137.8,1,1335,265,398,624
...


ab+.sh 使用方式跟 ab 一樣,除了 -c 跟 -n 是由 script 動態來調整外,其它 ab 參數可以從命令列尾端代入,另外測試前要修改壓力參數來給定測試壓力範圍
  • LOW_CONCURRENCY 初始的 concurreny 數
  • HIGH_CONCURRENCY 最大的 concurreny 數
  • RATE_CONCURRENCY 每回測試 concurreny 的累加值
  • NUM_CALL 單個 concurrent user 要發送的 request 次數
執行結果會顯示在畫面上並輸出至當前目錄下的 csv 檔 e.g. localhost_20130403_1152.csv

ab+.sh
#!/bin/bash
# ab+
#
# os and network tuning
# ulimit -n 100000
# sudo echo "2048 64512" > /proc/sys/net/ipv4/ip_local_port_range
# sudo echo "1" > /proc/sys/net/ipv4/tcp_tw_recycle
# sudo echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse
# sudo echo "10" > /proc/sys/net/ipv4/tcp_fin_timeout
# sudo echo "65535" > /proc/sys/net/core/somaxconn
# sudo echo "65535" > /proc/sys/net/ipv4/tcp_max_syn_backlog  

AB_PATH=/usr/bin/ab
SS_PATH=/bin/ss

LOW_CONCURRENCY=100
HIGH_CONCURRENCY=1000
CONCURRENCY_STEP=100
NUM_CALL=100

HOST=$(echo "$*" | perl -ne '/http.*?:\/\/([^\/?:#]*)/; print $1')
PREFIX=$HOST$(date +"_%Y%m%d_%H%M")
OUTPUT_FILE=$PREFIX.csv
MAX_OPEN_FILES=$(ulimit -n)

trap "exit;" SIGINT SIGTERM

if [ "$MAX_OPEN_FILES" -le $HIGH_CONCURRENCY ]; then
  echo "Warning: open file limit < HIGH_CONCURRENCY" 
  exit 1
fi

echo "Starting benchmark"

mkdir -p temp/"$PREFIX"

echo "#$AB_PATH -c CONCURRENCY=($LOW_CONCURRENCY to $HIGH_CONCURRENCY step $CONCURRENCY_STEP) -n NUM=($NUM_CALL*CONCURRENCY) $*" | tee "$OUTPUT_FILE"
echo "#start time,concurrency,complete requests,failed requests,tps,min,mean,stddev,median,max,90%,95%,99%" | tee -a "$OUTPUT_FILE"
for (( concurrency=LOW_CONCURRENCY; concurrency<=HIGH_CONCURRENCY; concurrency+=CONCURRENCY_STEP ))
do
  tempFile="temp/$PREFIX/c$concurrency-n$((NUM_CALL*concurrency)).log"
  echo "$AB_PATH -c $concurrency -n $((NUM_CALL*concurrency)) $*" > "$tempFile"
  
  startTime=$(date)
  $AB_PATH -c $concurrency -n $((NUM_CALL*concurrency)) "$@" >> "$tempFile" 2>&1

  if [ $? -ne 0 ]; then
    echo "Error: please check $tempFile"
    exit 1
  fi

  stats=$(egrep "Complete requests:|Failed requests:|Requests per second:|Total:" "$tempFile" | perl -pe 's/[^\d\.]+/ /g; s/^\s+//g;' | perl -pe 's/\s+$//g; s/\s+/,/g')
  histogram=$(egrep "90%|95%|99%" "$tempFile" | cut -c6- | perl -pe 's/[^\d\.]+/ /g; s/^\s+//g;' | perl -pe 's/\s+$//g; s/\s+/,/g')

  echo "$startTime,$concurrency,$stats,$histogram" | tee -a "$OUTPUT_FILE"

  tcpConnections=$($SS_PATH -s | egrep 'TCP:' | perl -ne '/(\d+)/; print $1')
  while [ $((tcpConnections+concurrency+CONCURRENCY_STEP)) -ge "$MAX_OPEN_FILES" ]
  do
    sleep 1
    tcpConnections=$($SS_PATH -s | egrep 'TCP:' | perl -ne '/(\d+)/; print $1')
  done
done

rm -rf temp

echo "Finished benchmark, see $OUTPUT_FILE"

Dec 20, 2011

[Java Performance]Chapter 2:Operating System Performance Monitoring

這篇談到如何在作業系統層級搜集數據(CPU、Memory、Network I/O、Disk I/O),從數據中分析可能的效能問題

名詞定義:

  • Performance monitoring: 使用非侵入式的方法從作業系統或應用程式搜集、觀察數據,如使用作業系統提供的工具(e.g. vmstat)搜集效能數據,但不會影響應用程式的行為與效能,這方法適用於大部份的環境(production / development/ testing ...)使用
  • Performance profiling: 使用侵入式方法搜集較特定目標數據,如某method的total cpu time/輸入參數,但可能會調整到應用程式行為或者降低效能,這方法比較適用於development/ testing 環境
  • Performance tuning: 通常是 monitoring與profiling之後,發覺問題進而修改程式碼 / 設定參數,增進應用程式效能

Feb 4, 2009

ubuntu configuration memo

Install gcin and liu

sudo apt-get install gcin
tar -zxf liu_gcin120.tar.gz
sudo ./liu_gcin120/install.sh
gedit ~/.gnomerc

#gcin environment

export GTK_IM_MODULE=gcin
export XMODIFIERS="@im=gcin"

gcin &

clip_image002

Launch System > Preferences > Sound > gcin Setup
click Setting for gtab input methods then unselect Auto-send when keycodes are filled
relogin x-window
Disable alert sound
System > Preferences > Sound > Soulds Tab
unselect “play alert sound”
Install stardic
sudo apt-get install stardict

Add firefox search plugin(/usr/lib/firefox-3.0.5/searchplugins)
Install windows applications
sudo apt-get install wine

Download the wine-door package from the link after that follow below commands to install
sudo dpkg -i wine-doors_0.1.2_all.deb

Install pdf-XChange right click on PDFXVwer.exe and select Open With Wine Windows Program Loader (This tool is useful to highlight and comment in pdf file)
Install IE6 just access link and fellow the instructions
Configuration Pidgin
right click on pidgin icon and select Blink on New Message. Again, click plugins and select below plugins.
  • History
  • Offline Mesasge Emulation
  • Message Notification
    Notify For
    ■ IM Windows
    ■ Chat windows
Accouns > select account which you are using > Modify
Enter Local alise and select New mail notifications
clip_image004


Install chm reader
sudo apt-get install gnochm

Issue: the default font size is small
Download the font-patch from the link after that follow below commands to apply.
sudo patch -b /usr/bin/gnochm < gnochm-fontsize.patch

Install Google gadget
Download the package from the link after that follow below commands to install.
sudo dpkg -i google-gadgets_0.9.3-0~getdeb1_i386.deb
Launch Applications > Accessories > Google Gadget (GTK)
Install Notes.
Download IBM Lotus Notes 8.5 Client for Linux (Debian Install) from IBM after that follow below commands to install.

mkdir notes
tar -xf '/home/brandon/Desktop/notes85_notes_linux_deb_beta2_prod.tar' -C notes
sudo dpkg -i notes/ibm_lotus_notes-8.5.i586.deb
/opt/ibm/lotus/notes/notes
Install JDK with Netbean / Eclipse
Download JDK from Sun after that follow below commands to install.
chmod u+x jdk-6u11-linux-i586.bin
./jdk-6u11-linux-i586.bin
gedit /home/brandon/.bashrc
export $JAVA_HOME=/home/brandon/jdk1.6.0_11
export PATH=$PATH:$JAVA_HOME/bin
clip_image006


Download Netbean from link after that follow below commands to install.
sudo '/media/Brandon'\''s HDD/untitled folder/netbeans-6.5-ml-java-linux.sh' --javahome $JAVA_HOME
Download Eclipse from link after that follow below commands to install.
sudo tar -zxvf eclipse-jee-ganymede-SR1-linux-gtk.tar.gz -C /opt/
gedit ~/.gnomerc

#JAVA Environment
export JAVA_HOME=/home/brandon/jdk1.6.0_11
export PATH=$PATH:$JAVA_HOME/bin 


clip_image008

Add new Launcher in deskop
clip_image010
Install Mail-Notification and enable SSL. (package)
(Reference : http://glyphobet.net/blog/essay/286)
sudo dpkg -i mail-notification_5.4.dfsg.1-1build1.deb

Sep 24, 2008

Linux remote desktop using VNC

Steps:
1.install vncserver

[root@hostname ~]#yum install vncserver

2.startup vncserver

[root@hostname ~]# service vncserver start
Starting VNC server: no displays configured [ OK ]

3.change environment who user you want to login

[root@hostname ~]#su – root

4.startup a vncserver instance and assign login password

[root@hostname ~]# vncserver

You will require a password to access your desktops.

Password:
Verify:

New 'hostname:1 (root)' desktop is hostname:1 http://xx.xx.xx.xx:5801

Creating default startup script /root/.vnc/xstartup
Starting applications specified in /root/.vnc/xstartup
Log file is /root/.vnc/hostname:1.log


5.stop the vncserver instance.

[root@hostname ~]# vncserver -kill :1


6.Modifiy default startup script /root/.vnc/xstartup


# Uncomment the following two lines for normal desktop:
unset SESSION_MANAGER <= remove the mark
exec /etc/X11/xinit/xinitrc <= remove the mark


7.Use Vnc Client as like OpenVnc and UltraVnc to connect the server.
Resources:
http://forum.icst.org.tw/phpbb/viewtopic.php?t=11500
http://linux.vbird.org/linux_server/0310telnetssh.php#vnc

May 12, 2007

Ldap Install Guide

  
yum install openldap-servers.i386 openldap-devel openldap-clients compat-openldap

#加入Samba.schema


 cd /etc/openldap/schema
wget http://www.samba.org/cgi-bin/cvsweb/samba/examples/LDAP/samba.schema?rev=1.1.4.1
mv samba.schema?rev=1.1.4.1 samba.schema

#編輯ldap設定檔



vi /etc/openldap/slapd.conf

#slapd.conf #add


  include /etc/openldap/schema/samba.schema 

#取得加密密碼


 
slappasswd -s root-dn-passwd {SSHA}F+zPYmGidD3HdO5eO3aidWTKjaRZDsUe

#modify


 suffix "dc=im,dc=nuu,dc=edu,dc=tw" rootdn "cn=Manager,dc=im,dc=nuu,dc=edu,dc=tw" rootpw {SSHA}n9G1cmwcsikHDF1OAByZzrjr1f836S3i 

#chown


 chown -R ldap:ldap /var/lib/ldap/  

#TLS


./CA -newcert
./CA –newca
openssl req -new -nodes -keyout newreq.pem -out newreq.pem
./CA -sign
mkdir /etc/openldap/ssl
mv demoCA/cacert.pem /etc/openldap/ssl
mv newcert.pem /etc/openldap/ssl/servercrt.pem
mv newreq.pem /etc/openldap/ssl/serverkey.pem
chown ldap:ldap /etc/openldap/ssl/serverkey.pem
chmod 600 /etc/openldap/ssl/serverkey.pem
scp /etc/openldap/ssl/cacert.pem
root@203.64.178.106:/etc/openldap/ssl/cacert.pem

在客戶端的/etc/openldap/ldap.conf 中加入
TLS_CACERT /etc/openldap/ssl/cacert.pem ,
注意此文件時從服務器上生成的。

Some Selinux Tips


vi /etc/selinux/targeted/booleans.local
allow_ypbind=0
ftpd_disable_trans=1
httpd_disable_trans=1
or
setsebool ftpd_disable_trans 1
setsebool httpd_disable_trans 1
for vsftpd
modprobe ip_nat_ftp 2> /dev/null
modprobe ip_conntrack_ftp 2> /dev/null
for openwebmail
touch /var/log/openwebmail.log
chcon -u system_u /var/log/openwebmail.log
chcon -t httpd_sys_script_rw_t /var/log/openwebmail.log
chcon -t httpd_unconfined_script_exec_t /var/www/cgi-bin/openwebmail/openwebmail*

for awtstats
chcon -t httpd_unconfined_script_exec_t /usr/local/awstats/wwwroot/cgi-bin/*

remote backup to tape

#Server

  useradd backup
su - backup
ssh-keygen -t rsa
mv .ssh/id_dsa.pub .ssh/authorized_keys
mt -f /dev/st0 rewind
chown backup /bin/st0

#copy your key to client
(use vi to copy and past) or scp etc..

#Client

  vi backup.sh
#!/bin/sh
#remote backup to tape
#Author: Yu-Jen Chen
filename="www_"`date +%Y%m%d_%H%M%S`".tar.gz"
remote="203.64.178.107"
tar -zcf $filename /var/www/html/
scp -i id_rsa $filename backup@$remote:~
ssh -i id_rsa backup@$remote "tar -rvf /dev/st0 "$filename

#set crontab
 crontab -e
0 3 15,30 * * /var/www/backup/backup.sh

Some SSH Tips

SSH的相關安全設定

http://www.study-area.org/tips/ssh_tips.htm

http://linux.vbird.org/linux_server/0310telnetssh.php#ssh_sshdconfig

關於Client 的登入工具
http://linux.vbird.org/linux_server/0310telnetssh.php#ssh_client

#密碼
#私鑰

http://linux.vbird.org/linux_server/0310telnetssh.php#ssh_nopasswd

若使用putty需使用PuTTYgen將ssh-keygen 的key轉換putty所用的格式


Load→Save private Key…

使用 Key Agent
若有太多key時於可以參考Pageant
#Kerberos & GSSAPI

請man sshd_config

#putty系列的工具
http://libai.math.ncu.edu.tw/bcc16/6/putty/
http://www.chiark.greenend.org.uk/~sgtatham/putty/

l PuTTY
提供 Telnet 以及 SSH client 的功能。

l PSCP
提供 SCP client 的功能(安全加密的網路檔案拷貝,使用命令提示列)。

l PSFTP
提供 PSFTP client 的功能。

l PuTTYtel
僅提供 Telnet client 的功能。

l Plink
提供 SSH client ,使用命令提示列。

l Pageant
SSH 認證通行碼的代理程式。  

l PuTTYgen
提供產生RSA金鑰的工具。

當訊息為亂碼時

請注意系統的charset與client 的charset


或換為Big5


當然一般狀況換為en的環境可以避免較多的亂碼或排版跳格問題

若要更改預設字元集則於

vi /etc/sysconfig/i18n


Some OpenLDAP Resourse

主要的站:

http://www.openldap.org/


電子書:

http://www.bspu.secna.ru/~swp/LDAP/

http://www.zytrax.com/books/ldap/

實體書(中文):

http://203.64.181.44/Webpac2/store.dll/?ID=169574&T=0&S=ASC&ty=ns

Tools:

ldapadmin

http://ldapadmin.sourceforge.net/

jxplorer

http://www.jxplorer.org/

CSV to LDIF Converter(perl)

http://www.novell.com/coolsolutions/tools/14462.html

CSV2LDIF

http://www.home.unix-ag.org/patrick/index.php?csv2ldif

http://www.bind9.net/ldap-tools

LDAP函式庫:

PHP_LDAP

http://tw.php.net/manual/tw/ref.ldap.php

JDNI

http://java.sun.com/products/jndi/

Perl-LDAP

http://ldap.perl.org/

參考網址:

http://del.icio.us/tag/ldap

http://en.wikipedia.org/wiki/LDAP

Introduction to LDAP

http://twistedmatrix.com/users/tv/ldap-intro/ldap-intro.html


Red Hat Enterprise Linux 4: Reference Guide:Chapter 13. Lightweight Directory Access Protocol (LDAP)

http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/ref-guide/ch-ldap.html

jiing's BlogLDAP相關資料整理
http://jiing.blogspot.com/2004/12/ldap.html

LDAP Howto, LDAP Links, LDAP Whitepapers

http://www.bind9.net/ldap/

Steven's Linux Note - Articles

http://ms.ntcb.edu.tw/~steven/

OpenLDAP-輕量級目錄存取協定前言
http://www.weithenn.idv.tw/cgi-bin/wiki.pl/OpenLDAP-%e8%bc%95%e9%87%8f%e7%b4%9a%e7%9b%ae%e9%8c%84%e5%ad%98%e5%8f%96%e5%8d%94%e5%ae%9a%e5%89%8d%e8%a8%80

RedHat Enterprise Linux Server 4.3 + OpenLDAP + DNS + Samba + Postfix + OpenWebmail
http://www.nseasy.net/

OpenLDAP: Configuring Access Control Lists

http://sapiens.wustl.edu/~sysmain/info/openldap/openldap_configure_acl.html

vbird網友投書

http://linux.vbird.org/somepaper

延伸閱讀:

Single Sign on

http://en.wikipedia.org/wiki/Single_sign-on

LdapInjection

http://www.owasp.org/index.php/LDAP_injection


LDAP Software, LDAP Tools, LDAP Utilities

Linux檔案存取權限

Permission

http://linux.vbird.org/linux_basic/0210filepermission.php#filepermission

SUID/SGID/Sticky Bit

http://linux.vbird.org/linux_basic/0220filemanager.php#fileperm

ACL

http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/File_Systems/ch-acls.html

http://linux.vbird.org/linux_server/0240network-secure-1.php#acl

Selinux

http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/selinux-guide/

http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Security_And_Authentication/index.html