仙人掌世界

Apr 3, 2013

ab 漸進壓力測試工具

最近在學Go，也用Go寫一個 ab-like 的玩具(Go-HttpBench)，為了要比較兩者效能另外寫了 bash script 用來轉換 ab 報表為行列數據，功能跟 autobench 相似，用階段性漸增的壓力來測試 http server，搜集到的數據可再用 gnuplot 或用 excel 繪制圖型

Example:


brandon@brandon-laptop:~/workspaces/go/gb$ ./ab+.sh -k http://localhost/
Start benchmark
#/usr/bin/ab -c CONCURRENCY=(100 to 10000 step 100) -n NUM=(100*CONCURRENCY) -k http://localhost/
#start time,concurrency,complete requests,failed requests,tps,min,mean,stddev,median,max,90%,95%,99%
Wed Apr  3 11:52:29 CST 2013,100,10000,0,31272.48,0,3,5.4,0,32,11,16,22
Wed Apr  3 11:52:29 CST 2013,200,20000,0,32050.72,0,6,14.2,0,113,23,40,64
Wed Apr  3 11:52:30 CST 2013,300,30000,0,33527.94,0,5,11.6,0,207,16,28,53
Wed Apr  3 11:52:31 CST 2013,400,40000,0,32396.87,0,12,25.0,0,189,48,70,105
Wed Apr  3 11:52:32 CST 2013,500,50000,0,32492.75,0,15,35.9,0,391,51,109,166
Wed Apr  3 11:52:34 CST 2013,600,60000,0,33188.85,0,18,38.8,0,448,73,110,166
Wed Apr  3 11:52:36 CST 2013,700,70000,0,33436.38,0,21,46.9,0,483,71,134,221
Wed Apr  3 11:52:38 CST 2013,800,80000,0,33370.64,0,24,54.5,0,1015,74,145,264
Wed Apr  3 11:52:40 CST 2013,900,90000,0,34093.27,0,26,66.4,0,1150,78,133,260
Wed Apr  3 11:52:43 CST 2013,1000,100000,0,34448.17,0,27,68.4,2,1251,85,136,266
Wed Apr  3 11:52:46 CST 2013,1100,110000,0,34072.87,0,32,78.1,0,1129,101,170,368
Wed Apr  3 11:52:49 CST 2013,1200,120000,0,33332.79,0,36,86.2,0,1032,144,228,386
Wed Apr  3 11:52:53 CST 2013,1300,130000,0,34580.66,0,37,96.0,5,3349,116,201,346
Wed Apr  3 11:52:57 CST 2013,1400,140000,0,33776.24,0,41,96.3,0,689,153,262,475
Wed Apr  3 11:53:01 CST 2013,1500,150000,0,31418.26,0,47,114.7,0,1031,164,295,533
Wed Apr  3 11:53:06 CST 2013,1600,160000,0,33285.65,0,48,111.8,0,629,217,340,492
Wed Apr  3 11:53:11 CST 2013,1700,170000,0,33441.14,0,50,121.5,0,825,211,356,559
Wed Apr  3 11:53:16 CST 2013,1800,180000,0,31452.71,0,52,166.0,0,4253,163,294,496
Wed Apr  3 11:53:22 CST 2013,1900,190000,0,33781.89,0,56,139.8,0,1018,234,425,627
Wed Apr  3 11:53:28 CST 2013,2000,200000,0,33766.12,0,59,137.8,1,1335,265,398,624
...

ab+.sh 使用方式跟 ab 一樣，除了 -c 跟 -n 是由 script 動態來調整外，其它 ab 參數可以從命令列尾端代入，另外測試前要修改壓力參數來給定測試壓力範圍

LOW_CONCURRENCY 初始的 concurreny 數
HIGH_CONCURRENCY 最大的 concurreny 數
RATE_CONCURRENCY 每回測試 concurreny 的累加值
NUM_CALL 單個 concurrent user 要發送的 request 次數

執行結果會顯示在畫面上並輸出至當前目錄下的 csv 檔 e.g. localhost_20130403_1152.csv

ab+.sh

#!/bin/bash
# ab+
#
# os and network tuning
# ulimit -n 100000
# sudo echo "2048 64512" > /proc/sys/net/ipv4/ip_local_port_range
# sudo echo "1" > /proc/sys/net/ipv4/tcp_tw_recycle
# sudo echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse
# sudo echo "10" > /proc/sys/net/ipv4/tcp_fin_timeout
# sudo echo "65535" > /proc/sys/net/core/somaxconn
# sudo echo "65535" > /proc/sys/net/ipv4/tcp_max_syn_backlog  

AB_PATH=/usr/bin/ab
SS_PATH=/bin/ss

LOW_CONCURRENCY=100
HIGH_CONCURRENCY=1000
CONCURRENCY_STEP=100
NUM_CALL=100

HOST=$(echo "$*" | perl -ne '/http.*?:\/\/([^\/?:#]*)/; print $1')
PREFIX=$HOST$(date +"_%Y%m%d_%H%M")
OUTPUT_FILE=$PREFIX.csv
MAX_OPEN_FILES=$(ulimit -n)

trap "exit;" SIGINT SIGTERM

if [ "$MAX_OPEN_FILES" -le $HIGH_CONCURRENCY ]; then
  echo "Warning: open file limit < HIGH_CONCURRENCY" 
  exit 1
fi

echo "Starting benchmark"

mkdir -p temp/"$PREFIX"

echo "#$AB_PATH -c CONCURRENCY=($LOW_CONCURRENCY to $HIGH_CONCURRENCY step $CONCURRENCY_STEP) -n NUM=($NUM_CALL*CONCURRENCY) $*" | tee "$OUTPUT_FILE"
echo "#start time,concurrency,complete requests,failed requests,tps,min,mean,stddev,median,max,90%,95%,99%" | tee -a "$OUTPUT_FILE"
for (( concurrency=LOW_CONCURRENCY; concurrency<=HIGH_CONCURRENCY; concurrency+=CONCURRENCY_STEP ))
do
  tempFile="temp/$PREFIX/c$concurrency-n$((NUM_CALL*concurrency)).log"
  echo "$AB_PATH -c $concurrency -n $((NUM_CALL*concurrency)) $*" > "$tempFile"
  
  startTime=$(date)
  $AB_PATH -c $concurrency -n $((NUM_CALL*concurrency)) "$@" >> "$tempFile" 2>&1

  if [ $? -ne 0 ]; then
    echo "Error: please check $tempFile"
    exit 1
  fi

  stats=$(egrep "Complete requests:|Failed requests:|Requests per second:|Total:" "$tempFile" | perl -pe 's/[^\d\.]+/ /g; s/^\s+//g;' | perl -pe 's/\s+$//g; s/\s+/,/g')
  histogram=$(egrep "90%|95%|99%" "$tempFile" | cut -c6- | perl -pe 's/[^\d\.]+/ /g; s/^\s+//g;' | perl -pe 's/\s+$//g; s/\s+/,/g')

  echo "$startTime,$concurrency,$stats,$histogram" | tee -a "$OUTPUT_FILE"

  tcpConnections=$($SS_PATH -s | egrep 'TCP:' | perl -ne '/(\d+)/; print $1')
  while [ $((tcpConnections+concurrency+CONCURRENCY_STEP)) -ge "$MAX_OPEN_FILES" ]
  do
    sleep 1
    tcpConnections=$($SS_PATH -s | egrep 'TCP:' | perl -ne '/(\d+)/; print $1')
  done
done

rm -rf temp

echo "Finished benchmark, see $OUTPUT_FILE"

May 10, 2012

Finagle 概述

Finagle 是基於JVM的網路框架/堆疊，開發者能在之上使用Java，Scala或任何基於JVM程式語言來開發非同步運算的RPC Client/Server，其專案是 Twitter 總結了實務上遇到的問題，所設計的分散式系統方案，並在2011的7月開放源始碼，目前功能如下表

Protocols

HTTP
Streaming HTTP (Comet)
Thrift
Memcached/Kestrel
More to come!

Server Support

Backpressure (to defend against abusive clients)
Service Registration (e.g., via Zookeeper)
Native OpenSSL bindings

Client Support

Connection Pooling
Load Balancing
Failure Detection
Failover/Retry
Distributed Tracing (a la Dapper)
Service Discovery (e.g., viaZookeeper)
Rich Statistics
Native OpenSSL bindings
Sharding

ScalaSTM 試用

STM（software transactional memory）是種 concurrency control 機制，相對於用 locking 進行同步化，更安全與簡單，以下使用帳戶轉帳的例子實作比較locking與STM

帳號轉帳案例

多組帳號同時以隨機金額進行轉帳，轉帳過程要保證沒有 race condition 與避免 deadlock

Random / Math.random() / ThreadLocalRandom

java.util.Random

java.util.Random 亂數演算法會依賴 seed 數值產生亂數，並在產生亂數後重新產生 seed，
在 Multi-thread 下為了保證在各Thread取到不同的亂數， java.util.Random 在實作上使用 optimistic locking 來同步更換 seed ，問題是若所有Thread都使用同個Random物件產生亂數，在大量競爭下會造成不斷的loop計算，進而產生效能問題

TOTP authentication demo 開發工具與佈署環境(Heroku)

延續前篇使用 TOTP 與 Google Authenticator 實作 Two-Factor authentication ，此篇說明使用到的工具與相關心得

Application
Back-end

Java
slf4j
Apache commons-codec
Spring framework 3.1

嘗試了些新版的功能
1. Java-based configuration 覺得一般使用沒比 XML schema-based 好用，做框架設計的時候會比較實用，動態組裝、設定樣版、模組化，不知道還能怎麼運用
2. Environment API 在啟動時依系統參數來區分"開發"或"佈署環境"的Profile，用於載入不同的 bean definition，若沒指定啟動參數則為 default 的 profile

e.g. JAVA_OPTS => -Dspring.profiles.active=heroku

Zxing

Google 的 QRCode library，在這用來產生 QRCode image 傳遞 secret key

Jedis

Redis Client 端的 Java Library

AspectJ

在這用來做 redis 的連線控制，避免階層操作下佔用多組連線

TOTP Algorithm

基於 RFC6238：TOTP: Time-Based One-Time Password Algorithm 附錄程式並做了些簡化

Front-end

Spring MVC

試用 Servlet 3.0 免宣告 web.xml 的功能，但從 Tomcat 搬到 Jetty 遇到些問題，在 Jetty 的根目錄(/) 會被 DefaultServlet 覆蓋，導不到 spring mvc的DispatcherServlet，不想多加 Filter 只好換回 web.xml

Jackson Json
JQuery
Bootstap

Twitter 開發的前端工具庫