Apr 3, 2013

ab 漸進壓力測試工具

最近在學Go,也用Go寫一個 ab-like 的玩具(Go-HttpBench),為了要比較兩者效能另外寫了 bash script 用來轉換 ab 報表為行列數據,功能跟 autobench 相似,用階段性漸增的壓力來測試 http server,搜集到的數據可再用 gnuplot 或用 excel 繪制圖型

Example:

brandon@brandon-laptop:~/workspaces/go/gb$ ./ab+.sh -k http://localhost/
Start benchmark
#/usr/bin/ab -c CONCURRENCY=(100 to 10000 step 100) -n NUM=(100*CONCURRENCY) -k http://localhost/
#start time,concurrency,complete requests,failed requests,tps,min,mean,stddev,median,max,90%,95%,99%
Wed Apr  3 11:52:29 CST 2013,100,10000,0,31272.48,0,3,5.4,0,32,11,16,22
Wed Apr  3 11:52:29 CST 2013,200,20000,0,32050.72,0,6,14.2,0,113,23,40,64
Wed Apr  3 11:52:30 CST 2013,300,30000,0,33527.94,0,5,11.6,0,207,16,28,53
Wed Apr  3 11:52:31 CST 2013,400,40000,0,32396.87,0,12,25.0,0,189,48,70,105
Wed Apr  3 11:52:32 CST 2013,500,50000,0,32492.75,0,15,35.9,0,391,51,109,166
Wed Apr  3 11:52:34 CST 2013,600,60000,0,33188.85,0,18,38.8,0,448,73,110,166
Wed Apr  3 11:52:36 CST 2013,700,70000,0,33436.38,0,21,46.9,0,483,71,134,221
Wed Apr  3 11:52:38 CST 2013,800,80000,0,33370.64,0,24,54.5,0,1015,74,145,264
Wed Apr  3 11:52:40 CST 2013,900,90000,0,34093.27,0,26,66.4,0,1150,78,133,260
Wed Apr  3 11:52:43 CST 2013,1000,100000,0,34448.17,0,27,68.4,2,1251,85,136,266
Wed Apr  3 11:52:46 CST 2013,1100,110000,0,34072.87,0,32,78.1,0,1129,101,170,368
Wed Apr  3 11:52:49 CST 2013,1200,120000,0,33332.79,0,36,86.2,0,1032,144,228,386
Wed Apr  3 11:52:53 CST 2013,1300,130000,0,34580.66,0,37,96.0,5,3349,116,201,346
Wed Apr  3 11:52:57 CST 2013,1400,140000,0,33776.24,0,41,96.3,0,689,153,262,475
Wed Apr  3 11:53:01 CST 2013,1500,150000,0,31418.26,0,47,114.7,0,1031,164,295,533
Wed Apr  3 11:53:06 CST 2013,1600,160000,0,33285.65,0,48,111.8,0,629,217,340,492
Wed Apr  3 11:53:11 CST 2013,1700,170000,0,33441.14,0,50,121.5,0,825,211,356,559
Wed Apr  3 11:53:16 CST 2013,1800,180000,0,31452.71,0,52,166.0,0,4253,163,294,496
Wed Apr  3 11:53:22 CST 2013,1900,190000,0,33781.89,0,56,139.8,0,1018,234,425,627
Wed Apr  3 11:53:28 CST 2013,2000,200000,0,33766.12,0,59,137.8,1,1335,265,398,624
...


ab+.sh 使用方式跟 ab 一樣,除了 -c 跟 -n 是由 script 動態來調整外,其它 ab 參數可以從命令列尾端代入,另外測試前要修改壓力參數來給定測試壓力範圍
  • LOW_CONCURRENCY 初始的 concurreny 數
  • HIGH_CONCURRENCY 最大的 concurreny 數
  • RATE_CONCURRENCY 每回測試 concurreny 的累加值
  • NUM_CALL 單個 concurrent user 要發送的 request 次數
執行結果會顯示在畫面上並輸出至當前目錄下的 csv 檔 e.g. localhost_20130403_1152.csv

ab+.sh
#!/bin/bash
# ab+
#
# os and network tuning
# ulimit -n 100000
# sudo echo "2048 64512" > /proc/sys/net/ipv4/ip_local_port_range
# sudo echo "1" > /proc/sys/net/ipv4/tcp_tw_recycle
# sudo echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse
# sudo echo "10" > /proc/sys/net/ipv4/tcp_fin_timeout
# sudo echo "65535" > /proc/sys/net/core/somaxconn
# sudo echo "65535" > /proc/sys/net/ipv4/tcp_max_syn_backlog  

AB_PATH=/usr/bin/ab
SS_PATH=/bin/ss

LOW_CONCURRENCY=100
HIGH_CONCURRENCY=1000
CONCURRENCY_STEP=100
NUM_CALL=100

HOST=$(echo "$*" | perl -ne '/http.*?:\/\/([^\/?:#]*)/; print $1')
PREFIX=$HOST$(date +"_%Y%m%d_%H%M")
OUTPUT_FILE=$PREFIX.csv
MAX_OPEN_FILES=$(ulimit -n)

trap "exit;" SIGINT SIGTERM

if [ "$MAX_OPEN_FILES" -le $HIGH_CONCURRENCY ]; then
  echo "Warning: open file limit < HIGH_CONCURRENCY" 
  exit 1
fi

echo "Starting benchmark"

mkdir -p temp/"$PREFIX"

echo "#$AB_PATH -c CONCURRENCY=($LOW_CONCURRENCY to $HIGH_CONCURRENCY step $CONCURRENCY_STEP) -n NUM=($NUM_CALL*CONCURRENCY) $*" | tee "$OUTPUT_FILE"
echo "#start time,concurrency,complete requests,failed requests,tps,min,mean,stddev,median,max,90%,95%,99%" | tee -a "$OUTPUT_FILE"
for (( concurrency=LOW_CONCURRENCY; concurrency<=HIGH_CONCURRENCY; concurrency+=CONCURRENCY_STEP ))
do
  tempFile="temp/$PREFIX/c$concurrency-n$((NUM_CALL*concurrency)).log"
  echo "$AB_PATH -c $concurrency -n $((NUM_CALL*concurrency)) $*" > "$tempFile"
  
  startTime=$(date)
  $AB_PATH -c $concurrency -n $((NUM_CALL*concurrency)) "$@" >> "$tempFile" 2>&1

  if [ $? -ne 0 ]; then
    echo "Error: please check $tempFile"
    exit 1
  fi

  stats=$(egrep "Complete requests:|Failed requests:|Requests per second:|Total:" "$tempFile" | perl -pe 's/[^\d\.]+/ /g; s/^\s+//g;' | perl -pe 's/\s+$//g; s/\s+/,/g')
  histogram=$(egrep "90%|95%|99%" "$tempFile" | cut -c6- | perl -pe 's/[^\d\.]+/ /g; s/^\s+//g;' | perl -pe 's/\s+$//g; s/\s+/,/g')

  echo "$startTime,$concurrency,$stats,$histogram" | tee -a "$OUTPUT_FILE"

  tcpConnections=$($SS_PATH -s | egrep 'TCP:' | perl -ne '/(\d+)/; print $1')
  while [ $((tcpConnections+concurrency+CONCURRENCY_STEP)) -ge "$MAX_OPEN_FILES" ]
  do
    sleep 1
    tcpConnections=$($SS_PATH -s | egrep 'TCP:' | perl -ne '/(\d+)/; print $1')
  done
done

rm -rf temp

echo "Finished benchmark, see $OUTPUT_FILE"

May 10, 2012

Finagle 概述

Finagle 是基於JVM的網路框架/堆疊,開發者能在之上使用Java,Scala或任何基於JVM程式語言來開發非同步運算的RPC Client/Server,其專案是 Twitter 總結了實務上遇到的問題,所設計的分散式系統方案,並在2011的7月開放源始碼,目前功能如下表


Protocols

  • HTTP
  • Streaming HTTP (Comet)
  • Thrift
  • Memcached/Kestrel
  • More to come!

Server Support

  • Backpressure (to defend against abusive clients)
  • Service Registration (e.g., via Zookeeper)
  • Native OpenSSL bindings

Client Support

  • Connection Pooling
  • Load Balancing
  • Failure Detection
  • Failover/Retry
  • Distributed Tracing (a la Dapper)
  • Service Discovery (e.g., viaZookeeper)
  • Rich Statistics
  • Native OpenSSL bindings
  • Sharding


Apr 7, 2012

ScalaSTM 試用

STM(software transactional memory)是種 concurrency control 機制,相對於用 locking 進行同步化,更安全與簡單,以下使用帳戶轉帳的例子實作比較locking與STM

帳號轉帳案例
多組帳號同時以隨機金額進行轉帳,轉帳過程要保證沒有 race condition 與避免 deadlock


Mar 18, 2012

Random / Math.random() / ThreadLocalRandom

java.util.Random

java.util.Random 亂數演算法會依賴 seed 數值產生亂數,並在產生亂數後重新產生 seed,
在 Multi-thread 下為了保證在各Thread取到不同的亂數, java.util.Random 在實作上使用 optimistic locking 來同步更換 seed ,問題是若所有Thread都使用同個Random物件產生亂數,在大量競爭下會造成不斷的loop計算,進而產生效能問題

Feb 17, 2012

TOTP authentication demo 開發工具與佈署環境(Heroku)

延續前篇 使用 TOTP 與 Google Authenticator 實作 Two-Factor authentication ,此篇說明使用到的工具與相關心得

Application
Back-end

  • Java
  • slf4j
  • Apache commons-codec
  • Spring framework 3.1
嘗試了些新版的功能
1. Java-based configuration 覺得一般使用沒比 XML schema-based 好用,做框架設計的時候會比較實用,動態組裝、設定樣版、模組化,不知道還能怎麼運用
2. Environment API 在啟動時依系統參數來區分"開發"或"佈署環境"的Profile,用於載入不同的 bean definition,若沒指定啟動參數則為 default 的 profile
e.g. JAVA_OPTS => -Dspring.profiles.active=heroku
Google 的 QRCode library,在這用來產生 QRCode image 傳遞 secret key
Redis Client 端的 Java Library
在這用來做 redis 的連線控制,避免階層操作下佔用多組連線
  • TOTP Algorithm
基於 RFC6238:TOTP: Time-Based One-Time Password Algorithm 附錄程式並做了些簡化

Front-end
  • Spring MVC
試用 Servlet 3.0 免宣告 web.xml 的功能,但從 Tomcat 搬到 Jetty 遇到些問題,在 Jetty 的根目錄(/) 會被 DefaultServlet 覆蓋, 導不到 spring mvc的DispatcherServlet,不想多加 Filter 只好換回 web.xml
Twitter 開發的前端工具庫