ab 漸進壓力測試工具

最近在學Go，也用Go寫一個 ab-like 的玩具(Go-HttpBench)，為了要比較兩者效能另外寫了 bash script 用來轉換 ab 報表為行列數據，功能跟 autobench 相似，用階段性漸增的壓力來測試 http server，搜集到的數據可再用 gnuplot 或用 excel 繪制圖型

Example:

brandon@brandon-laptop:~/workspaces/go/gb$ ./ab+.sh -k http://localhost/
Start benchmark
#/usr/bin/ab -c CONCURRENCY=(100 to 10000 step 100) -n NUM=(100*CONCURRENCY) -k http://localhost/
#start time,concurrency,complete requests,failed requests,tps,min,mean,stddev,median,max,90%,95%,99%
Wed Apr  3 11:52:29 CST 2013,100,10000,0,31272.48,0,3,5.4,0,32,11,16,22
Wed Apr  3 11:52:29 CST 2013,200,20000,0,32050.72,0,6,14.2,0,113,23,40,64
Wed Apr  3 11:52:30 CST 2013,300,30000,0,33527.94,0,5,11.6,0,207,16,28,53
Wed Apr  3 11:52:31 CST 2013,400,40000,0,32396.87,0,12,25.0,0,189,48,70,105
Wed Apr  3 11:52:32 CST 2013,500,50000,0,32492.75,0,15,35.9,0,391,51,109,166
Wed Apr  3 11:52:34 CST 2013,600,60000,0,33188.85,0,18,38.8,0,448,73,110,166
Wed Apr  3 11:52:36 CST 2013,700,70000,0,33436.38,0,21,46.9,0,483,71,134,221
Wed Apr  3 11:52:38 CST 2013,800,80000,0,33370.64,0,24,54.5,0,1015,74,145,264
Wed Apr  3 11:52:40 CST 2013,900,90000,0,34093.27,0,26,66.4,0,1150,78,133,260
Wed Apr  3 11:52:43 CST 2013,1000,100000,0,34448.17,0,27,68.4,2,1251,85,136,266
Wed Apr  3 11:52:46 CST 2013,1100,110000,0,34072.87,0,32,78.1,0,1129,101,170,368
Wed Apr  3 11:52:49 CST 2013,1200,120000,0,33332.79,0,36,86.2,0,1032,144,228,386
Wed Apr  3 11:52:53 CST 2013,1300,130000,0,34580.66,0,37,96.0,5,3349,116,201,346
Wed Apr  3 11:52:57 CST 2013,1400,140000,0,33776.24,0,41,96.3,0,689,153,262,475
Wed Apr  3 11:53:01 CST 2013,1500,150000,0,31418.26,0,47,114.7,0,1031,164,295,533
Wed Apr  3 11:53:06 CST 2013,1600,160000,0,33285.65,0,48,111.8,0,629,217,340,492
Wed Apr  3 11:53:11 CST 2013,1700,170000,0,33441.14,0,50,121.5,0,825,211,356,559
Wed Apr  3 11:53:16 CST 2013,1800,180000,0,31452.71,0,52,166.0,0,4253,163,294,496
Wed Apr  3 11:53:22 CST 2013,1900,190000,0,33781.89,0,56,139.8,0,1018,234,425,627
Wed Apr  3 11:53:28 CST 2013,2000,200000,0,33766.12,0,59,137.8,1,1335,265,398,624
...

ab+.sh 使用方式跟 ab 一樣，除了 -c 跟 -n 是由 script 動態來調整外，其它 ab 參數可以從命令列尾端代入，另外測試前要修改壓力參數來給定測試壓力範圍

• LOW_CONCURRENCY 初始的 concurreny 數
• HIGH_CONCURRENCY 最大的 concurreny 數
• RATE_CONCURRENCY 每回測試 concurreny 的累加值
• NUM_CALL 單個 concurrent user 要發送的 request 次數

執行結果會顯示在畫面上並輸出至當前目錄下的 csv 檔 e.g. localhost_20130403_1152.csv

ab+.sh

#!/bin/bash
# ab+
#
# os and network tuning
# ulimit -n 100000
# sudo echo "2048 64512" > /proc/sys/net/ipv4/ip_local_port_range
# sudo echo "1" > /proc/sys/net/ipv4/tcp_tw_recycle
# sudo echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse
# sudo echo "10" > /proc/sys/net/ipv4/tcp_fin_timeout
# sudo echo "65535" > /proc/sys/net/core/somaxconn
# sudo echo "65535" > /proc/sys/net/ipv4/tcp_max_syn_backlog  

AB_PATH=/usr/bin/ab
SS_PATH=/bin/ss

LOW_CONCURRENCY=100
HIGH_CONCURRENCY=1000
CONCURRENCY_STEP=100
NUM_CALL=100

HOST=$(echo "$*" | perl -ne '/http.*?:\/\/([^\/?:#]*)/; print $1')
PREFIX=$HOST$(date +"_%Y%m%d_%H%M")
OUTPUT_FILE=$PREFIX.csv
MAX_OPEN_FILES=$(ulimit -n)

trap "exit;" SIGINT SIGTERM

if [ "$MAX_OPEN_FILES" -le $HIGH_CONCURRENCY ]; then
  echo "Warning: open file limit < HIGH_CONCURRENCY" 
  exit 1
fi

echo "Starting benchmark"

mkdir -p temp/"$PREFIX"

echo "#$AB_PATH -c CONCURRENCY=($LOW_CONCURRENCY to $HIGH_CONCURRENCY step $CONCURRENCY_STEP) -n NUM=($NUM_CALL*CONCURRENCY) $*" | tee "$OUTPUT_FILE"
echo "#start time,concurrency,complete requests,failed requests,tps,min,mean,stddev,median,max,90%,95%,99%" | tee -a "$OUTPUT_FILE"
for (( concurrency=LOW_CONCURRENCY; concurrency<=HIGH_CONCURRENCY; concurrency+=CONCURRENCY_STEP ))
do
  tempFile="temp/$PREFIX/c$concurrency-n$((NUM_CALL*concurrency)).log"
  echo "$AB_PATH -c $concurrency -n $((NUM_CALL*concurrency)) $*" > "$tempFile"
  
  startTime=$(date)
  $AB_PATH -c $concurrency -n $((NUM_CALL*concurrency)) "$@" >> "$tempFile" 2>&1

  if [ $? -ne 0 ]; then
    echo "Error: please check $tempFile"
    exit 1
  fi

  stats=$(egrep "Complete requests:|Failed requests:|Requests per second:|Total:" "$tempFile" | perl -pe 's/[^\d\.]+/ /g; s/^\s+//g;' | perl -pe 's/\s+$//g; s/\s+/,/g')
  histogram=$(egrep "90%|95%|99%" "$tempFile" | cut -c6- | perl -pe 's/[^\d\.]+/ /g; s/^\s+//g;' | perl -pe 's/\s+$//g; s/\s+/,/g')

  echo "$startTime,$concurrency,$stats,$histogram" | tee -a "$OUTPUT_FILE"

  tcpConnections=$($SS_PATH -s | egrep 'TCP:' | perl -ne '/(\d+)/; print $1')
  while [ $((tcpConnections+concurrency+CONCURRENCY_STEP)) -ge "$MAX_OPEN_FILES" ]
  do
    sleep 1
    tcpConnections=$($SS_PATH -s | egrep 'TCP:' | perl -ne '/(\d+)/; print $1')
  done
done

rm -rf temp

echo "Finished benchmark, see $OUTPUT_FILE"

ubuntu configuration memo

Install gcin and liu

sudo apt-get install gcin
tar -zxf liu_gcin120.tar.gz
sudo ./liu_gcin120/install.sh
gedit ~/.gnomerc

#gcin environment

export GTK_IM_MODULE=gcin
export XMODIFIERS="@im=gcin"

gcin &

Launch System > Preferences > Sound > gcin Setup
click Setting for gtab input methods then unselect Auto-send when keycodes are filled
relogin x-window
Disable alert sound
System > Preferences > Sound > Soulds Tab
unselect “play alert sound”
Install stardic

sudo apt-get install stardict

Add firefox search plugin(/usr/lib/firefox-3.0.5/searchplugins)
Install windows applications

sudo apt-get install wine

Download the wine-door package from the link after that follow below commands to install

sudo dpkg -i wine-doors_0.1.2_all.deb

Install pdf-XChange right click on PDFXVwer.exe and select Open With Wine Windows Program Loader (This tool is useful to highlight and comment in pdf file)
Install IE6 just access link and fellow the instructions
Configuration Pidgin
right click on pidgin icon and select Blink on New Message. Again, click plugins and select below plugins.

• History
• Offline Mesasge Emulation
• Message Notification
  Notify For
  ■ IM Windows
  ■ Chat windows

Accouns > select account which you are using > Modify
Enter Local alise and select New mail notifications



Install chm reader

sudo apt-get install gnochm

Issue: the default font size is small
Download the font-patch from the link after that follow below commands to apply.

sudo patch -b /usr/bin/gnochm < gnochm-fontsize.patch

Install Google gadget
Download the package from the link after that follow below commands to install.
sudo dpkg -i google-gadgets_0.9.3-0~getdeb1_i386.deb
Launch Applications > Accessories > Google Gadget (GTK)
Install Notes.
Download IBM Lotus Notes 8.5 Client for Linux (Debian Install) from IBM after that follow below commands to install.

mkdir notes
tar -xf '/home/brandon/Desktop/notes85_notes_linux_deb_beta2_prod.tar' -C notes
sudo dpkg -i notes/ibm_lotus_notes-8.5.i586.deb
/opt/ibm/lotus/notes/notes

Install JDK with Netbean / Eclipse
Download JDK from Sun after that follow below commands to install.

chmod u+x jdk-6u11-linux-i586.bin
./jdk-6u11-linux-i586.bin
gedit /home/brandon/.bashrc
export $JAVA_HOME=/home/brandon/jdk1.6.0_11
export PATH=$PATH:$JAVA_HOME/bin

Download Netbean from link after that follow below commands to install.

sudo '/media/Brandon'\''s HDD/untitled folder/netbeans-6.5-ml-java-linux.sh' --javahome $JAVA_HOME

Download Eclipse from link after that follow below commands to install.

sudo tar -zxvf eclipse-jee-ganymede-SR1-linux-gtk.tar.gz -C /opt/
gedit ~/.gnomerc

#JAVA Environment
export JAVA_HOME=/home/brandon/jdk1.6.0_11
export PATH=$PATH:$JAVA_HOME/bin 

Add new Launcher in deskop

Install Mail-Notification and enable SSL. (package)
(Reference : http://glyphobet.net/blog/essay/286)

sudo dpkg -i mail-notification_5.4.dfsg.1-1build1.deb

Google Notebook Gadget Resizer

The default height of notebook gadget is 200px , it’s too short that need often to scroll down/up , so that write a script to change the height.

Install Greasemonkey
https://addons.mozilla.org/en-US/firefox/addon/748

User Script:Google Notebook Gadget Resizer
http://userscripts.org/scripts/show/41679

Result:

Code:

// ==UserScript==
// @name          Google Notebook Gadget Resizer
// @description   To change height of gadgets in iGoogle
// @include       http://www.google.com/ig*
// @include       http://www.google.com.tw/ig*

// @author        parkghost@hotmail.com
// @version       1.0
// ==/UserScript==

// This script are working on Google notebook gadget and some similar gadgets which content type is url and not using dynamic-height feature.
//
// you can fellow below instruction to modify default height of Google notebook gadget or add more gadget to apply new height.
//
// 1.input "about:config" in url box
// 2.search "greasemonkey.scriptvals.userscripts.org/Google Notebook Gadget Resizer.oGadgets"
// 3.The properly is a array of url pattern with height that can modify of you want to.
//
// default:
// ({'http://www.google.com/notebook/ig':"600px", 'url pattern':"400px"})
//
// The url of gadget you could look at page source that place in iframe tag.(ex.http ://87.gmodules.com/ig/ifr?view=home&url=http://itszero.googlepages.com/iGoogleTVSchedule.xml&nocache=0&up_timeRange=0&up_showPrograms=6&up_fontSize=0.75&up_favoriteChannels=&up_defaultChannel=74&lang=en&country=us&.lang=en&.country=us&synd=ig&mid=87&ifpctok=-5100993727886241321&parent=http://www.google.com&extern_js=/extern_js/f/CgJlbhICdXMrMAo4ACwrMBA4ACwrMBI4ACwrMBM4ACwrMBU4ACw/GgLq_VyeDu0.js)
//
// The URL parameter is good to become a URL pattern.
// http://itszero.googlepages.com/iGoogleTVSchedule.xml

var oGadgets = {'http://www.google.com/notebook/ig':'600px','url pattern':'400px'};
var oGadgetsStr = GM_getValue('oGadgets');
if(oGadgetsStr){
 oGadgets = eval(oGadgetsStr);
}else{
GM_setValue('oGadgets',uneval(oGadgets));
}
var oFrames = document.getElementsByTagName('iframe');
for(var i=0 ; i < oFrames.length ; i++){ 
 for(var key in oGadgets){
   var re = new RegExp(key); 
   if(re.test(oFrames[i].src)){
     oFrames[i].style.height = oGadgets[key];
   }
 }
}

Resources:

Greasemonkey Manual:API
http://wiki.greasespot.net/API_reference

Managing Gadget Height
http://code.google.com/apis/gadgets/docs/ui.html#Dyn_Height

自訂副檔名搜尋

SearchFileByExt.vbs

start = Now
WScript.Echo("Start:" & start)
strComputer = "." 
Set objWMIService = GetObject("winmgmts:" _ 
& "{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2") 
 Set colFiles = objWMIService.ExecQuery _ 
("Select * from CIM_DataFile where Extension = 'mp3' OR Extension = 'wma'") 
For Each objFile in colFiles 
Wscript.echo(objFile.Caption)
'objFile
'objFile.Delete 
Next 
finish = Now
WScript.Echo("Finish:" & finish)
WScript.Echo("Cost:" & DateDiff("n",start,finish))

vbscript 版 wget

PingList.vbs

<package>
<job>
<runtime>
<named
name="SourceUrl"
helpstring="HTTP檔案位址"
type="string"
required="true"
/>
<named
name="DestinationFile"
helpstring="儲存檔案位址"
type="string"
required="true"
/>
<example>
Example : wget /SourceUrl:"http://xxx/xxx.exe" /DestinationFile:"C:\Documents and Settings\All Users\桌面\xxx.exe"
</example> 
<description>
用途：下載Web檔案
版本：1.0
作者：Parkghost</description>
</runtime>
<SCRIPT language="VBScript">
If WScript.Arguments.Named.Exists("SourceUrl") And WScript.Arguments.Named.Exists("DestinationFile") Then
strUrl = WScript.Arguments.Named.Item("SourceUrl")
strFile = WScript.Arguments.Named.Item("DestinationFile")
Const adTypeBinary = 1
Const adSaveCreateOverWrite = 2
Const ForWriting = 2
Dim web, varByteArray, strData, strBuffer, lngCounter, ado 
Err.Clear 
Set web = Nothing
Set web = CreateObject("WinHttp.WinHttpRequest.5.1")
If web Is Nothing Then Set web = CreateObject("WinHttp.WinHttpRequest") 
If web Is Nothing Then Set web = CreateObject("MSXML2.ServerXMLHTTP") 
If web Is Nothing Then Set web = CreateObject("Microsoft.XMLHTTP") 
web.Open "GET", strURL, False 
web.Send 
If Err.Number <> 0 Then 
SaveWebBinary = False 
Set web = Nothing 
WScript.Quit
End If 
If web.Status <> "200" Then
SaveWebBinary = False 
Set web = Nothing 
WScript.Quit
End If 
varByteArray = web.ResponseBody 
Set web = Nothing

'Save the file
'On Error Resume Next 
Set ado = Nothing 
Set ado = CreateObject("ADODB.Stream") 
If ado Is Nothing Then 
Set fs = CreateObject("Scripting.FileSystemObject") 
Set ts = fs.OpenTextFile(strFile, ForWriting, True) 
strData = "" 
strBuffer = "" 
For lngCounter = 0 to UBound(varByteArray) 
ts.Write Chr(255 And Ascb(Midb(varByteArray,lngCounter + 1, 1))) 
Next 
ts.Close 
Else 
ado.Type = adTypeBinary 
ado.Open 
ado.Write varByteArray 
ado.SaveToFile strFile, adSaveCreateOverWrite 
ado.Close 
End If 
SaveWebBinary = True 
Else
WScript.Arguments.ShowUsage()
End if
</SCRIPT>
</job>
</package>

解決vbscript len function 無法判斷 double byte 字元

strText = ("中文123")
WScript.Echo(strText  & ":" & Len(strText))
WScript.Echo(strText  & ":" & newLen(strText))

Function newLen(text)
oldlen = Len(text)
newLen = 0
For i = 1 To oldlen
If Asc(Mid(text,i,1)) > 0 Then
 newLen = newLen + 1
Else
 newLen = newLen + 2
End If
Next
End Function

網路連線測式(Ping by List)

PingList.vbs

Msg = "站點" & vbTab & paddingStr("位址",27) & vbTab & paddingStr("IP",15) & vbTab & "回應時間" & vbNewLine
Set objWMIService = GetObject("winmgmts:\\.\root\cimv2")
Set objXML = CreateObject("Microsoft.XMLDOM")
objXML.async = False
objXML.Load "ipList.xml"
If objXML.parseError.errorCode <> 0 Then
 WScript.Echo("設定檔載入錯誤")
End If

Set objList = objXML.getElementsByTagName("*")
Set nodes = objList.item(i).selectSingleNode("//site").childNodes

For Each node In nodes
 Dim ProtocolAddress
 Dim ResponseTime
 Set objNamedNodeMap = node.attributes  
 Set objPing = objWMIService.ExecQuery("select * from Win32_PingStatus where address = '" & node.text & "'")
 For Each objStatus in objPing
  ProtocolAddress = objStatus.ProtocolAddress
  ResponseTime = objStatus.ResponseTime
 Next 
 Msg = Msg & objNamedNodeMap.item(0).text & vbTab & paddingStr(node.text,25) & vbTab & paddingStr(ProtocolAddress,15) & vbTab & ResponseTime & vbNewLine 
Next
WScript.Echo(Msg)

Function paddingStr(str,lenght)
 If Len(str) > lenght then
  paddingStr = str
 Else
  'todo:double byte analysis
  paddingStr = str &  Space(lenght - Len(str))
 End If
End Function

ipList.xml

<?xml version="1.0" encoding="big5"?>
<site>
<item name="HiNet">www.hinet.net</item>
<item name="Yahoo">tw.yahoo.com</item>
<item name="Pchome">www.pchome.com</item>
</site>