Category Archives: Test

[repost ]kafka Performance testing

original:https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing

It would be worthwhile to automate our performance testing to act as a generic integration test suite.

The goal of this would be to do basic performance analysis and correctness testing in a distributed environment.

Required Metrics

Client Side Measurements

  1. Throughput
  2. Response Time/Latency
  3. Timeouts
  4. Consumer lag

Common Stats

  1. Vmstat – Context Switch, User CPU utilization %, System CPU utilization %, Total CPU utilization %
  2. Iostat – Reads/sec, Writes/sec, KiloBytes read/sec, KiloBytes write/sec, Average number of transactions waiting, Average number of active transactions, Average response time of transactions, Percent of time waiting for service, Percent of time disk is busy
  3. Prstat – Virtual memory size of each java process, RSS size of each process, Total CPU utilization of each process

GC Log Analysis

  1. Footprint (Maximal amount of memory allocated)
  2. Freed Memory (Total amount of memory that has been freed)
  3. Freed Memory/min (Amount of memory that has been freed per minute)
  4. Total Time (Time data was collected for)
  5. Acc Pauses (Sum of all pauses due to GC)
  6. Throughput (Time percentage the application was NOT busy with GC)
  7. Full GC Performance (Performance of full GC collections. Full GC collections are marked so in the gc logs.)
  8. GC Performance (Performance of minor collections. These are collections that are not full according to the definition above.)
  9. CMS counts and frequency (Number of CMS collections and their frequency)
  10. CMS failure count and frequency (CMS failure metrics)

Server side metrics

  1. Throughput and response time breakdown for each request at the LogManager, RequestPurgatory level
  2. ISR membership churn aggregate and per partition
  3. Number of expirations in the request purgatory
  4. Leader election rate aggregate and per partition
  5. Leader election latency aggregate and per partition
  6. High watermark change aggregate and per partition
  7. Replica lag time and bytes aggregate and per partition
  8. Replica fetch throughput and response time aggregate and breakdown at the LogManager, RequestPurgatory level

Log analysis

  1. Exceptions in logs, their frequency and types of exception
  2. Warnings in logs, their frequency and types of warnings

Miscellaneous

  1. Capture all the server machine profiles before tests are being executed (Such as disk space, number of CPUS etc)
  2. Capture all configurations for each run

Phase I: Perf Tools

The goal of this phase is just to create tools to help run perf tests. We already have some of these so this will primarily just be about expanding and augmenting these.

  • kafka-producer-perf-test.sh – We will add a csv option to this to dump incremental statistics in csv format for consumption by automated tools.
  • kafka-consumer-perf-test.sh – Likewise we will add a csv option here.
  • jmx-dump.sh – This will just poll the kafka and zookeeper jmx stats every 30 seconds or so and output them as csv.
  • dstat – This is the existing perf tool to get read/write/fs stats
  • draw-performance-graphs.r – This will be an R script to draw relevant graphs given a bunch of csv files from the above tools. The output of this script will be a bunch of png files that can be combined with some html to act as a perf report.

Here are the graphs that would be good to see:

  • Latency histogram for producer
  • MB/sec and messages/sec produced
  • MB/sec and messages/sec consumed
  • Flush time
  • Errors (should not be any)
  • Consumer cache hit ratio (both the bytes and count, specifically 1 – #physical_reads / #requests and 1 – physical_bytes_read / bytes_read)
  • Write merge ratio (num_physical_writes/num_produce_requests and avg_request_size/avg_physical_write_size)
  • CPU, network, io, etc

Phase II: Automation

This phase is about automating the deployment and running of the performance tests. At the end of this phase we want to have a script that pulls from svn every night, runs a set of performance scenarios, and produces reporting on these.

We need the following helper scripts for this:

  • kafka-deploy-kafka.sh – This script will take a set of hosts and deploy kafka to each of them.
  • kafka-start-cluster.sh – This will start the kafka broker on the list of hosts
  • kafka-stop-cluster – Stops cluster

The tests will be divided up into scenarios. Each scenario is a directory which contains the following:

  • broker config
  • producer config
  • consumer config
  • producer perf test command
  • consumer perf test command
  • env file that contain # brokers, # producers, and # consumers

The output of the scenario will be a directory which contains the following:

  • Producer perf csvs
  • Consumer perf csvs
  • dstat csvs
  • jmx csvs
  • env file

Scenarios to test:

  • Producer throughput with no consumers. We should cover the following cases:
    • Vary the number of topics
    • Vary the async batch size
    • Vary the flush size
    • Vary the message size
  • Consumer throughput with no producer
    • Vary the message size
    • Vary the number of topics
  • Single producer/consumer pair
    • Cold consumption (i.e. not in cache)
    • Active consumption (i.e. consumer caught up to producer)
    • Vary the number of topics
  • Multiple consumers for one topic

We should add a script to take two scenarios and produce a summary/diff of them, i.e. what go worse and what got better. We can use this to track things over time. We can also rsync these up to a public location as a service to open source developers.

Phase III: Correctness

The correctness testing can be very straight-forward, all we want to validate is that every message produced gets consumed. This could be as simple as logging out a simple message id in the consumer and comparing it to the produced value.

Simplest idea is just to have each producer produce a set of known messages (say sequential integers in some unique range). Then have consumers validate that all integers were consumed (no gaps) and record the number of duplicates (if any).

Ideally we would repeat this scenario and script in broker failures (kills), server pauses (simulated), etc.

0.8 Performance testing

Producer throughput

Message size : ~1K Production Data

Throughput in MB/s

    Kafka Version 0.7   0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8   0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8
    Replication Factor n/a   1 1 1 1 1 1 1 1 1   2 2 2 2 2 2 2 2 2
    Acks n/a   -1 1 0 -1 1 0 -1 1 0   -1 1 0 -1 1 0 -1 1 0
    Compression Uncomp   Uncomp Uncomp Uncomp Gzip Gzip Gzip Snappy Snappy Snappy   Uncomp Uncomp Uncomp Gzip Gzip Gzip Snappy Snappy Snappy
Producer threads Batch size                                            
1 1   29.57   1.66 1.69 19.48 0.94 0.97 3.51 1.53 1.46 11.40   0.56 1.61 23.52 0.35 0.91 3.70 0.56 1.45 10.83
2 1   45.35   3.31 3.11 23.82 1.80 1.72 3.87 2.30 2.96 11.65   1.27 2.79 20.59 0.66 1.39 3.86 1.07 2.19 11.39
5 1   58.53   5.24 5.49 20.26 2.59 2.86 3.35 4.02 4.48 13.36   1.79 4.76 21.29 1.16 2.49 3.06 1.93 3.98 10.49
10 1   50.23   8.20 8.59 19.65 3.17 2.77 2.87 7.35 7.39 10.51   2.90 7.97 19.44 1.85 2.68 3.05 3.27 7.24 12.01
                                             
1 50   49.15   17.95 18.88 77.43 7.93 7.66 26.88 15.42 15.44 62.33   10.57 15.76 57.85 4.79 7.50 28.61 10.31 13.98 66.15
2 50   84.24   34.94 33.41 82.67 14.88 15.25 30.13 29.77 26.47 71.18   17.26 27.26 78.50 9.53 14.29 30.11 16.82 26.10 90.98
5 50   102.44   64.38 62.82 89.66 17.56 18.23 19.47 57.54 58.58 71.71   26.02 47.69 86.62 12.48 17.39 31.54 33.13 54.18 74.96
10 50   103.02   61.06 64.57 86.48 18.45 17.24 20.38 59.59 58.44 69.34   28.08 59.80 91.62 18.68 17.90 20.02 35.62 62.03 72.02
                                             
1 100   49.76   23.69 23.56 75.71 8.93 9.04 28.69 21.73 21.79 70.55   12.69 21.61 52.12 5.86 8.85 27.49 13.48 21.04 66.70
2 100   84.75   42.85 40.46 84.72 18.48 16.90 31.69 42.61 38.07 85.78   22.54 35.81 74.53 11.16 17.18 35.86 25.63 37.95 106.99
5 100   92.78   67.65 67.89 87.24 21.93 19.78 21.40 67.22 65.94 87.98   26.93 57.77 86.46 15.20 18.92 25.35 37.40 75.18 83.91
10 100   100.23   67.53 68.40 87.99 19.82 20.80 21.38 69.64 68.49 81.88   31.36 63.19 88.68 17.39 18.96 21.56 43.26 72.14 84.89

[repost ]异常模拟测试 — 场景抽象及解决方案

original:http://baidutech.blog.51cto.com/4114344/744429

1. 概述
本文主要是将之前调研的异常测试需求进行一个分类并抽象成不同的场景,然后针对每一个场景给出一些解决方案或者思路。目前大体分为4类:
 网络异常,网络相关的异常情况,比如连接超时、接收/发送失败等;
 内存异常,内存相关的异常情况,比如内存满、内存分配失败等;
 磁盘异常,磁盘相关的异常情况,比如磁盘频繁坏掉、磁盘满等;
 程序异常,程序逻辑相关的异常,这个根据不同的数据结构、设计和逻辑会有不同的需求,如函数的参数、返回值修改等。
异常测试的目的是为了测试到一些难以覆盖到的异常情况,如果把程序细分成逻辑单元的组合,我们的目的就是通过各种不同的途径(数据或者代码)来改变逻辑的走向以测试不同的异常情况。
2. 场景抽象及解决方案
2.1. 网络异常
2.1.1. 连接拒绝(connect refused)
在调用connect连接指定IP:PORT时被拒绝的情况。
2.1.1.1. 方案一 指定目的端口为一个没有进程监听的端口
将连接的目的IP指定为某一存在的IP,但是指定PORT为一个不处于listen状态的端口,此时connect会返回connect refused错误。

2.1.1.2. 方案二 Hook网络函数connect
连接被拒绝会返回ECONNREFUSED错误,可以通过hook connect函数,设置errno为ECONNREFUSED,然后编译成so通过LD_PRELOAD环境变量来达到hook的目的。示例步骤如下:
1. 编写hook.c实现connect函数:
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <unistd.h>
#include <dlfcn.h>
#include <errno.h>

extern int errno;

extern “C” {

int connect(int sockfd, const struct sockaddr *serv_addr, socklen_t addrlen){
srand(time(NULL));

int r = rand() % 100;

if (r < 50) {
//设置需要的errno
errno = ECONNREFUSED;
return -1;
} else {
int (*_connect)(int sockfd, const struct sockaddr *serv_addr, socklen_t addrlen) = NULL;
_connect = (int (*)(int sockfd, const struct sockaddr *serv_addr, socklen_t addrlen)) dlsym(RTLD_NEXT, “connect”);

return _connect(sockfd, serv_addr, addrlen);
}
}
}
2. 编译成so。
g++ -shared -rdynamic -o hook.so -fPIC hook.c -ldl
3. 在程序执行前设置LD_REPLOAD环境变量
export LD_PRELOAD=”./hook.so”

2.1.1.3. 方案三 iptables
拒绝发往目的地址的数据,并返回错误给发送方:
iptables -t filter -p tcp -A OUTPUT -d 目标IP –dport 目标端口 -j REJECT –reject-with tcp-reset
2.1.2. 连接超时
使用connect连接时返回timeout的情况。
2.1.3.1. 方案一 Hook网络函数connect
连接超时返回EINPROGRESS,可以采用LD_PRELOAD的hook方法实现connect函数,然后设置errno为EINPROGRESS并且返回-1,具体步骤参考2.1.1.2。
2.1.3.2. 方案二 iptables 丢包
Drop发往目的地址端口的数据(SYN)
iptables -t filter -p tcp -A OUTPUT -d 目标IP –dport 目标端口 -j DROP

或者Drop目的地址返回的数据(ACK)
iptables -t filter -p tcp -A INPUT -s 目标IP –sport 目标端口 -j DROP

2.1.3. 读超时
使用read/recv函数读取数据时超时的情况。
2.1.3.1. 方案一 Hook网络函数recv/read/select等
读超时需要分为blocking和non-blocking2种情况:
 socket设置为阻塞时,如果设置了SO_ RCVTIMEO,则recv或者read在超过设定时间未读取到数据后会返回-1,并且设置EAGAIN的错误号,可以使用LD_REPLOAD hook读函数recv/read函数,置errno为EAGAIN并返回-1,具体参考2.1.1.2。
 socket设置为非阻塞的情况,recv/read函数是立即返回的,超时的判断一般是通过select/poll/epoll等相关函数来查看,需要同时hook对应的函数,让其返回0,表示timeout。
非阻塞示例如下:
1. 编写hook.c实现recv和select的hook函数:
#include <sys/select.h>
#include <sys/time.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <unistd.h>
#include <dlfcn.h>
#include <errno.h>

ssize_t recv(int s, void *buf, size_t len, int flags)
{
errno = EAGAIN;
return -1;
}

int select(int n, fd_set *readfds, fd_set *writefds, fd_set *exceptfds, struct timeval *timeout){
if (NULL != readfds) {
return 0; //返回0表示超时
} else {
int (*_select)(int n, fd_set *readfds, fd_set *writefds, fd_set *exceptfds, struct timeval *timeout) = NULL;
_select = (int (*)(int n, fd_set *readfds, fd_set *writefds, fd_set *exceptfds, struct timeval *timeout))dlsym(RTLD_NEXT, “select”);

return _select(n, readfds, writefds, exceptfds, timeout);
}
}
2. 编译成so。
g++ -shared -rdynamic -o hook.so -fPIC hook.c -ldl
3. 在程序执行前设置LD_REPLOAD环境变量
export LD_PRELOAD=”./hook.so”

2.1.3.2. 方案二 iptables丢包
建立连接后,Drop目的地址返回的数据
iptables -t filter -p tcp -A INPUT -s 目标IP –sport 目标端口 -j DROP
2.1.3.3. 方案三 teeport限速
利用teeport的限速功能,在client-server类型的通信,如果要模拟client读取server端数据时超时,假设超时时间为5s,可以使用如下方法:
./teeport_2.py -l 50008 -r localhost:50007 –server=11 –stype=total –stime=5000

2.1.4. 写超时
使用write/send发送数据时超时的情况。
2.1.4.1. 方案一 Hook网络函数send/write
写超时也需要分为blocking和non-blocking2种情况:
 socket设置为阻塞时,如果设置了SO_ SNDTIMEO,则send/write在超过设定时间未读取到数据后会返回-1,并且设置EAGAIN的错误号,可以使用LD_REPLOAD hook读函数send/write函数,置errno为EAGAIN并返回-1。
 socket设置为非阻塞的情况,send/write函数是立即返回的,超时的判断一般是通过select/poll/epoll等相关函数来查看,需要同时hook对应的函数,让其返回0,表示timeout。
具体实现步骤参考2.1.3.1。
2.1.4.2. 方案二 iptables丢包
建立连接后,Drop源端口发出的数据包返回的数据
iptables -t filter -p tcp -A INPUT -s 目标IP –sport 目标端口 -j DROP

2.1.5. 读写过程中连接断开
在调用send/write/recv/read函数进行数据通信时,网络连接断开的情况。
2.1.5.1. 方案一Hook网络函数send/write/recv/read
读写过程中连接断开会返回ECONNRESET错误,可以通过Hook send/write/recv/read函数,返回-1,置errno为ECONNRESET,具体参考2.1.1.2的实现。
2.1.5.2. 方案二 iptables reject
建立连接后,使用iptables reject被测程序发往目标IP:Port的数据包。
iptables -t filter -p tcp -A OUTPUT -d 目标IP –dport 目标端口 -j REJECT –reject-with tcp-reset
2.1.5.3. 方案三 teeport中转断开
用teeport建立转发关系后,在通信过程中kill teeport。
2.1.6. 慢连接
限制网络数据的传输速度,模拟慢连接的情况。
2.1.6.1. 方案一 teeport限速
利用teeport的限速功能,在client-server类型的通信,如果要模拟client读取server端数据时超时,假设超时时间为5s,可以使用如下方法:
./teeport_2.py -l 50008 -r localhost:50007 –server=11 –stype=total –stime=5000 –client=11 –ctype=total –ctime=5000
2.1.6.2. 方案二 iptables限制数据包个数
通过iptables限制单位时间通过的数据包数,例如每分钟只能通过一个数据包:
iptables -A INPUT -p tcp –dport 目标端口 -m limit –limit 1/m –limit-burst 1 -j ACCEPT
iptables -A INPUT -p tcp –dport 目标端口 -j DROP

2.1.7. 异常数据包
模拟由于逻辑异常或者硬件异常导致的数据损坏和丢失。
2.1.7.1. 方案一 通过应用层fuzzing后转发
建立转发关系(类似teeport),在中间层对上游的数据进行修改或者丢弃后,转发给下游。
2.1.7.2. 方案二 Hook网络函数send/write
在上游发送方,使用LD_REPLOAD hook发送函数send或者write,对传进来的buffer中的数据进行修改或者丢弃后,再调用真实的send或write函数。
2.1.7.3. 方案二 Hook网络函数recv/read
在下游接收方,使用LD_REPLOAD hook接收函数recv或者read,先调用原始函数获取接收到的数据,然后修改其buffer参数的数据再返回。
2.2. 内存异常
2.2.1. 内存申请失败
OOM或者硬件异常导致的内存申请失败。
2.2.1.1. 方案一 Hook内存函数malloc
内存申请malloc或者new等都是通过调用malloc函数实现,可以直接hook malloc函数,然后让其return NULL表示申请失败。示例步骤如下:
1. 编写hook.c实现malloc函数,直接让其返回NULL:
#include <stdio.h>
#include <stdlib.h>
#include <dlfcn.h>

void *malloc(size_t size)
{
size_t *(*_malloc)(size_t size) = NULL;
_malloc = (size_t *(*)(size_t size)) dlsym(RTLD_NEXT, “malloc”);

return NULL;
}
2. 编译成so。
g++ -shared -rdynamic -o hook.so -fPIC hook.c -ldl
3. 在程序执行前设置LD_REPLOAD环境变量
export LD_PRELOAD=”./hook.so”

2.2.1.2. 方案二 使用GNU提供的Memory Allocation Hooks(适用于单测)
使用GNU提供的Memory Allocation Hooks,替换malloc,适用于单测(类似的,通过设置__free_hook变量,可以对free函数进行替换):
#include <malloc.h>

static void *my_malloc_hook (size_t size, const void *caller){
return NULL; //直接返回NULL
}

TEST_F(test_armor_create_suite, test_armor_create__param_metanum_1)
{
void *(*old_malloc_hook)(size_t size, const void *caller) = __malloc_hook; //保存旧的Hook
__malloc_hook = my_malloc_hook; //安装自定义Hook
void* mem = malloc(100);
__malloc_hook = old_malloc_hook; //恢复原Hook
ASSERT_EQ((void*)NULL, mem);
}
2.3. 磁盘异常
2.3.1. 磁盘频繁坏掉
磁盘频繁坏掉,表现为经常打开、读或者写失败。
2.3.1.1. 方案一 Hook文件系统函数
文件的打开、读和写操作实际上都是通过调用系统函数open/read/write(或fopen/fread/fwrite)来进行操作的,所以可以考虑自己实现open/read/write(或fopen/fread/fwrite)函数,随机返回错误,然后编译成so通过LD_PRELOAD环境变量来达到hook的目的。示例步骤如下:
1. 编写hook.c实现open/read/write(或fopen/fread/fwrite)函数,如下示例实现50%概率read失败,其它函数类似:
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <unistd.h>
#include <dlfcn.h>

ssize_t read(int fd, void *buf, size_t count)
{
srand(time(NULL));

int r = rand() % 100;

if (r > 50) {
return -1;
} else {
ssize_t (*_read)(int fd, void *buf, size_t count) = NULL;
_read = (ssize_t (*)(int fd, void *buf, size_t count)) dlsym(RTLD_NEXT, “read”);
//调用原函数
return _read(fd, buf, count);
}
}
2. 编译成so。
g++ -shared -rdynamic -o hook.so -fPIC hook.c -ldl
3. 在程序执行前设置LD_REPLOAD环境变量
export LD_PRELOAD=”./hook.so”

2.3.1.2. 方案二 直接链接so
该方案和上面的方案一基本类似,同样是需要实现系统read/write(或fread/fwrite)等函数,不同的是需要程序重新编译并且把hook.so链接进入,这样就不需要在执行的时候设置LD_PRELOAD环境变量了,如:
g++ -g -Wall -o test test.cpp -L. hook.so
注意,程序拷贝到其它地方执行时需要把hook.so也拷贝过去,否则不会有效果,可以通过ldd查看hook.so是否存在。
2.3.1.3. 方案三 GDB修改
该方案的思路是gdb在指定读写函数的地方下断点,可以是系统的read/write等函数,也可以是业务层的读写封装接口,在函数返回之后修改其返回值为错误,让后面的逻辑接收到磁盘读写错误的信息。不过这种方式在每次读写操作时都需要断点修改,比较麻烦,这里就不具体介绍了。
2.3.2. 磁盘满
磁盘满表现为文件无法写入内容,write函数返回-1(fwrite错误通过ferror判断返回非0),并且errno == ENOSPC错误。
2.3.2.1. 方案一 Hook系统写函数write或fwrite
实现write或fwrite函数,取决于程序所使用的文件操作接口,如果不确定,可以2个函数同时实现,然后编译成so采用LD_PRELOAD来加载。示例:
1. 编写hook.c实现fwrite函数,write函数类似:
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <unistd.h>
#include <dlfcn.h>
#include <errno.h>

size_t fwrite(const void *ptr, size_t size, size_t nmemb, FILE *stream)
{
srand(time(NULL));

int r = rand() % 100;

if (r > 50) {
//设置错误,用户可通过ferror判断
stream->_flags |= _IO_ERR_SEEN;
//设置错误码,perror会输出
errno = ENOSPC;
return 0;
} else {
size_t (*_fwrite)(const void *ptr, size_t size, size_t nmemb, FILE *stream) = NULL;
_fwrite = (size_t (*)(const void *ptr, size_t size, size_t nmemb, FILE *stream)) dlsym(RTLD_NEXT, “fwrite”);

return _fwrite(ptr, size, nmemb, stream);
}
}
2. 编译成so。
g++ -shared -rdynamic -o hook.so -fPIC hook.c -ldl
3. 在程序执行前设置LD_REPLOAD环境变量
export LD_PRELOAD=”./hook.so”

2.3.2.2. 方案二 直接链接so
参考2.3.1.2。
2.3.3. 文件损坏
文件损坏一般的表现是可以读成功,但是读取到的内容不正确。方案一以假乱真,采用的是hook系统读函数,修改其返回的buffer;方案二偷梁换柱,采用的是hook文件打开函数,修改其文件名参数指向自己修改过的数据文件,这样之后的读写操作都是在自己指定的数据文件中进行。
2.3.3.1. 方案一 Hook系统读函数read或fread
文件损坏后读取到的内容不正确,可以通过hook系统读函数read或fread,通过将实际读取到的buffer改写,或者随机写入一串数据进行fuzzing中,用户获取到的即是不正确的内容,但实际数据文件没有被破坏,仍可以下次正常读取。示例:
4. 编写hook.so实现fread函数,实现50%概率破坏文件,read函数类似:
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <unistd.h>
#include <dlfcn.h>

size_t fread(void *ptr, size_t size, size_t nmemb, FILE *stream)
{
size_t (*_fread)(void *ptr, size_t size, size_t nmemb, FILE *stream) = NULL;
_fread = (size_t (*)(void *ptr, size_t size, size_t nmemb, FILE *stream)) dlsym(RTLD_NEXT, “fread”);

size_t ret = _fread(ptr, size, nmemb, stream);

srand(time(NULL));
int r = rand() % 100;
if (r > 50) {
snprintf((char *)ptr, ret, “data is broken”, r);
}

return ret;
}
5. 编译成so。
g++ -shared -rdynamic -o hook.so -fPIC hook.c -ldl
6. 在程序执行前设置LD_REPLOAD环境变量
export LD_PRELOAD=”./hook.so”
2.3.3.2. 方案二 Hook文件打开函数open或fopen
方案一是直接在读的过程中修改数据,而方案二是直接“替换”了原文件,文件的内容可以随意的构造,对于一个全局把控更方便些,一般来说可以拷贝一份原文件对其修改。示例:
1. 编写hook.so实现fopen函数,如果文件名是data.txt则替换为hook.txt。
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <time.h>
#include <unistd.h>
#include <dlfcn.h>

FILE *fopen(const char *path, const char *mode)
{
FILE *(*_fopen)(const char *path, const char *mode) = NULL;
_fopen = (FILE *(*)(const char *path, const char *mode)) dlsym(RTLD_NEXT, “fopen”);

if(strcmp(path, “data.txt”) == 0) {
char *hook_filename = “hook.txt”;
return _fopen(hook_filename, mode);
}
return _fopen(path, mode);
}
2. 编译成so。
g++ -shared -rdynamic -o hook.so -fPIC hook.c -ldl
3. 在程序执行前设置LD_REPLOAD环境变量
export LD_PRELOAD=”./hook.so”

2.3.4. 数据fuzzing
数据fuzzing的目的是为了随机的改写数据文件来检测程序的稳定性,其解决方案和2.3.3节的“文件损坏”基本类似,可以直接参考。
2.4. 程序异常
2.4.1. 正常的逻辑错误返回
正常的逻辑错误返回,即本来是要返回正确的值,实际却返回了错误的值,改变了正常的逻辑走向,主要是为了让下游能走到异常的处理逻辑来测试不同的情况。示例图如下,正常的流程是A -> B -> C,D是异常处理逻辑,比较难构造数据走到,这时我们可以hook函数B,让其返回错误值,这样就可以走到D逻辑来测试。

17 #include <stdio.h>
18
19 void A()
20 {
21 printf(“this is A\n”);
22 }
23 int B()
24 {
25 printf(“this is B\n”);
26 return 0;
27 }
28 void C()
29 {
30 printf(“this is C\n”);
31 }
32 void D()
33 {
34 printf(“this is D\n”);
35 }
36 int main()
37 {
38 A();
39 int ret = B();
40 if(ret == 0) {
41 C();
42 } else {
43 D();
44 }
45 return 0;
46 }

2.4.3.1. 方案一 Hook关键静态函数
根据上面的代码示例,为了要走到D函数的逻辑,需要让B函数返回非0,而正常情况下B很难或者不会走到D函数,如上函数B始终return 0,这时可以通过testdbg hook函数B,让其返回非0值来测试D函数的逻辑。示例:
1. 编写hook.cpp实现B函数的hook版本,50%概率返回-1,50%概率调用原函数:
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include “hookmon.h”

int (* old_B)();

int B()
{
srand(time(NULL));

int r = rand() % 100;

if(r > 50) {
printf(“this is hook B\n”);
return -1;
} else {
return old_B();
}
}

void __attribute__ ((constructor)) hook_init(void)
{
attach_func(“B”, (void *)B, (void **)&old_B);
}
2. 编译成hook.so,需要依赖testdbg的include。
WORKROOT=../../../../../..
TESTDBG=$(WORKROOT)/svn/com-test/itest/tools/testdbg/output

hook.so : hook.cpp
g++ -shared -rdynamic -o $@ -fPIC $< -I$(TESTDBG)/include
3. 用testdbg进行启动执行,执行脚本如下:
#!/bin/sh
WORKROOT=../../../../../..
TESTDBG=$WORKROOT/svn/com-test/itest/tools/testdbg/output
$TESTDBG/bin/testdbg -l $TESTDBG/bin/hookmon.so -s ./hook.so ./test
2.4.3.2. 方案二 GDB修改
GDB修改有很多种方法,前期是程序需要用-g来编译,可以在函数B内(26行)下断点,然后让其return -1,也可以在40行下断点,if判断时修改ret的值为非0,不过这种方法需要每次都得修改,当然也可以将这些命令序列存成一个文本文件,然后通过gdb < cmds来执行。示例:
1. 将以下gdb命令保存到文本cmds中:
file test
b 40
r
set var ret=-1
c
2. gdb执行
gdb < cmds

2.4.2. 触发信号量处理函数
触发自定义的信号量处理函数一般来说可以直接产生相应的信号给程序。如果没有其它条件要求,只是希望触发函数可以采用方案一;如果需要在指定的时候发送信号量则需要在关键点hook进行逻辑判断再使用kill函数发送信号,参考方案二。
2.4.2.1. 方案一 命令行发送信号
命令行下直接通过kill –s signal pid来发送。示例:
1. 发送SIGIO信号给pid=22651的进程
kill -s SIGIO 22651
2.4.2.2. 方案二 程序发送信号
该方案是为了解决需要在指定的地方发送信号的问题,通过testdbg进行关键函数的hook,在函数开始前进行kill(0, signal)操作,然后直接调用原函数返回。示例:
1. 编写hook.cpp实现func函数的hook,先是kill发送SIGIO信号,再调用原函数:
#include <stdio.h>
#include <sys/types.h>
#include <signal.h>
#include “hookmon.h”

int (* old_func)();

int func()
{
kill(0, SIGIO);
return old_func();
}

void __attribute__ ((constructor)) hook_init(void)
{
attach_func(“func”, (void *)func, (void **)&old_func);
}
2. 编译成hook.so,需要依赖testdbg的include。
WORKROOT=../../../../../..
TESTDBG=$(WORKROOT)/svn/com-test/itest/tools/testdbg/output

hook.so : hook.cpp
g++ -shared -rdynamic -o $@ -fPIC $< -I$(TESTDBG)/include
3. 用testdbg进行启动执行,执行脚本如下:
#!/bin/sh
WORKROOT=../../../../../..
TESTDBG=$WORKROOT/svn/com-test/itest/tools/testdbg/output
$TESTDBG/bin/testdbg -l $TESTDBG/bin/hookmon.so -s ./hook.so ./test

2.4.3. 时序问题
时序问题一般在多模块或者多线程方面有较多存在,测试中往往希望程序按照某种特定的顺序来执行,这种情况需要细化到具体场景,然后在关键的地方采用hook的方式修改其逻辑,或者在hook函数前sleep之类的以达到所想要的执行顺序,hook的方法参考2.4.1。

[repost ]Netflix: Harden Systems Using A Barrel Of Problem Causing Monkeys – Latency, Conformity, Doctor, Janitor, Security, Internationalization, Chaos

original:http://highscalability.com/blog/2011/7/20/netflix-harden-systems-using-a-barrel-of-problem-causing-mon.html

With a new Planet of the Apes coming out, this may be a touchy subject with our new overlords, but Netflix is using a whole lot more trouble injecting monkeys to test and iteratively harden their systems. We learned previously how Netflix used Chaos Monkey, a tool to test failover handling by continuously failing EC2 nodes. That was just a start. More monkeys have been added to the barrel. Node failure is just one problem in a system. Imagine a problem and you can imagine creating a monkey to test if your system is handling that problem properly. Yury Izrailevsky talks about just this approach in this very interesting post: The Netflix Simian Army.

I know what you are thinking, if monkeys are so great then why has Netflix been down lately. Dmuino addressed this potential embarrassment, putting all fears of cloud inferiority to rest:

Unfortunately we’re not running 100% on the cloud today. We’re working on it, and we could use more help. The latest outage was caused by a component that still runs in our legacy infrastructure where we have no monkeys :)

To continuously test the resilience of Netflix’s system to failures, they’ve added a number of new monkeys, and even a gorilla:

  • Latency Monkey induces artificial delays in our RESTful client-server communication layer to simulate service degradation and measures if upstream services respond appropriately.
  • Conformity Monkey finds instances that don’t comply with best-practices and shuts them down.
  • Doctor Monkey taps into health checks that run on each instance as well as monitors other external signs of health (i.e. CPU load) to detect unhealthy instances.
  • Security Monkey searches for security violations or vulnerabilities, such as improperly configured AWS security groups, and terminates the offending instances. It also makes sure all our SSL and DRM certificates are valid and are not coming up for renewal.
  • 10-18 Monkey (short for Localization-Internationalization, or l10n-i18n) detects configuration problems in instances serving customers in multiple geographic regions, using different languages and character sets.
  • Chaos Gorilla is similar to Chaos Monkey, but simulates an outage of an entire Amazon availability zone.

Just as important as the test tools is that they’ve gone to the trouble to identify failure scenarios, implement code to deal with them, and then implement code that can verify that the problems have been dealt with. That’s just as important a process as the tools. Rarely do things just fail. They become intermittent, they become slow, they don’t failover cleanly, the don’t recover cleanly, they lie about what is really happening, they don’t collect the data you need to figure out what is going, they corrupt messages, they drop work randomly, they do crazy things that are failures but don’t fit in clean crisp failure definition boundaries. Figuring out what can be done in response to a complex set of interacting problems is the real magic here.

Layering Is Key

Without an architecture built around APIs, that is one with aggressive layering to isolate components, it would be impossible to test a system in this way. By creating insertion points for normal services and for testing, it becomes possible to expose the entire system to scripting and manipulation. Big ball of mud systems can’t do this, neither can monolithic 2-tier type systems.

When Do Monkeys Run? Protecting Against Cascading Failure

The obvious downside of letting monkeys into your system is that you can easily imagine the whole thing melting down through an innocent coding error. There’s a little more supervision than that.Dmuino explains:

Most monkeys only run when we have developers who could notice and fix problems. It also happens that our peak usage is while we’re home and our quiet time is while we’re at the office. That means, in general, the monkeys don’t run on a Sunday night.

 

More details in the original post.

[repost ]Tuning IBM HTTP Server to maximize the number of client connections to WebSphere Application Server

original: https://www-304.ibm.com/support/docview.wss?uid=swg21167658

Problem(Abstract)

Out of the box, IBM HTTP Server supports a maximum of 600 concurrent connections. Performance will suffer if load dictates more concurrent connections, as incoming requests will be queued up by the host operating system.

You can increase the number of maximum connections allowed by IBM HTTP Server by editing the httpd.conf file.

Resolving the problem

 

First and foremost, you must determine the maximum number of simultaneous connections required for this Web server. Using mod_status or mod_mpmstats (available with ihsdiag) to display the active number of threads throughout the day will provide some starting data.

There are 3 critical aspects to MPM (Multi-processing Module) tuning in IBM HTTP Server.

  1. Configuring the maximum number of simultaneous connections (MaxClientsdirective)
  2. Configuring the maximum number of IBM HTTP Server child processes (ThreadsPerChild directive)
  3. Less importantly, configuring the ramp-up and ramp-down of IBM HTTP Server child processes (MinSpareThreads, MaxSpareThreads, StartServers)

The first setting (MaxClients) has the largest immediate impact, but the latter 2 settings help tune IBM HTTP Server to accommodate per-process features in Apache modules, such as the WebSphere Application Server Web server plug-in.

Configuring the maximum number of simultaneous connections

If the number of concurrent connections required is smaller than the current value of MaxClients (default 600), then MaxClients should remain unchanged and further tuning is likely not required.

If the number of concurrent connections is larger than the current value of MaxClients, this will be a very serious inhibitor of performance. MaxClients represents the total number concurrent connections IBM HTTP Server can process.

  1. Determine a new value for MaxClients, which is the desired number of total simultaneous connections you require the server to support. This new value should remain a multiple of ThreadsPerChild.
  2. If the new value of MaxClients is beyond 2000, consider a more horizontal topology where the IBM HTTP Server workload is distributed amongst different physical servers.
  3. Update MaxClients in the httpd.conf file to reflect the desired capacity.
  4. In order to increase MaxClients,it is typically necessary to make a corresponding increase in the value of the ServerLimit directive. ServerLimit does not directly change the available capacity, but acts as a limit on the total number of active IBM HTTP Server child processes.

    Calculate a new ServerLimit by dividing MaxClients by ThreadsPerChild(this ratio is the required number of processes). If ServerLimitexceeds this prescribed ratio, MaxClientscan then be increased during a graceful restart of IBM HTTP Server.

  5. If MaxSpareThreads was previously equal to MaxClients, increase MaxSpareThreads to MaxClients.
Configuring the maximum number of HTTP Server child processes

The number of IBM HTTP Server child processes is controlled by the ratio of MaxClients divided by ThreadsPerChild. Since MaxClients is dictated exclusively by the amount of load the system will need to handle, ThreadsPerChild is the directive used to change this ratio.

This does not affect how much load IBM HTTP Server can handle, just the way that load is distributed among separate processes.

Some features of various Apache components might perform better when fewer child processes, each with more threads, are used. For example, the WebSphere Application Server Web server plug-in MaxConnections parameter and ESI cache are each per-process and are more effective with fewer child processes (higher ThreadsPerChild).

The number of IBM HTTP Server child processes should be independent of any WebSphere Application Server or front-end load balancer settings.

There are reasons to avoid a high value for ThreadsPerChild. Some operating system facilities might require per-process locks that more threads suffer contention on, such as access to the heap. A child process crash results in all threads terminating, so in environments with frequent child process crashes it is reasonable to use a lower ThreadsPerChild.

Third-party modules might have per-process caches or connection pools that might favor a larger ThreadsPerChild for efficiency purposes. However, their architectures could just as likely favor a smaller ThreadsPerChild if a per-process resource is protected by a mutex.

Generally, heavily loaded systems doing SSL favor a small ThreadsPerChild due to contention on the native heap.

If the ESI invalidation servlet is configured in the WebSphere Plugin, each additional process causes a dedicated web container thread to be permanently consumed.

If a change to the number of IBM HTTP Server child processes is required:

  1. Determine the number of child processes desired, and divide the current value of MaxClients by this number.
  2. Replace ThreadsPerChild with the value obtained from step 1.
  3. Replace ThreadLimit with the value obtained from step 1.
  4. If ThreadsPerChild has increased by a large amount, evaluate the current value of StartServers (the initial number of child processes created). Note that StartServers is not a very important tunable, because IBM HTTP Server re-evaluates the number of available threads every second.
Configuring the ramp-up and ramp-down of HTTP Server child processes

With the default httpd.conf file, IBM HTTP Server tries to maintain a pool of at least 25 spare processing threads and no more then 75 at any given time. These are specified with the MinSpareThreads and MaxSpareThreads directives respectively.

MaxSpareThreads
It is suggested that you disable the periodic termination of IBM HTTP Server child processes by setting MaxSpareThreads equal to MaxClients. Under this configuration, IBM HTTP Server child processes are created based on demand but not killed when the demand recedes. This is the recommended configuration because of complexities of graceful process termination and long-running requests; child processes are replaced as soon as they are scheduled for termination but might take considerable time to fully exit.

If periodic graceful child process termination is desired, MaxSpareThreads should be set to any value substantially smaller than MaxClients. If a module is maintaining a per-process cache or connection pool, like the WebSphere Application Server Web server plug-in, these will be discarded and created anew for the replacement child process.

MinSpareThreads
IBM HTTP Server compares the number of available processing threads to MinSpareThreads every second to decide whether or not to spawn additional child processes. Each additional child process contains ThreadsPerChild number of threads.

MinSpareThreads is usually safely set to the same value (or double) ThreadsPerChild. Since this value is checked every second, it is not an especially sensitive setting.

Example 1:
This configuration uses a single IBM HTTP Server child process with 2000 threads.

With this configuration, there will be only ONE child process with 2000 threads. Consequently, there will also only be ONE plug-in instance. This is much better than having many processes and many plug-in instances.

This also maximizes the effectiveness of the ESI cache as well as the shared information about marked-down Application Servers, which are both maintained by the WebSphere Application Server Web server plug-in on a per-process basis.

However, a crash in this child process takes out all 2000 threads. Heavy SSL usage, or heavy load under old operating systems, might consume extra CPU.

<IfModule worker.c>
ServerLimit 1
ThreadLimit 2000
StartServers 1
MaxClients 2000
MinSpareThreads 2000
MaxSpareThreads 2000
ThreadsPerChild 2000
MaxRequestsPerChild 0

# On linux, you may need to set ulimit -s 512
# in IHS/bin/envvars for high ThreadsPerChild

</IfModule>

Example 2:
This configuration uses multiple IBM HTTP Server child processes, with a fixed capacity of 2000 concurrent connections spread amongst 20 child processes.

If MaxConnections is used with the WebSphere Application Server Web server plug-in, MaxConnections will refer to the number of backend connections permitted in each IBM HTTP Server child process. While the number of processes is predictable in this case, the distribution of users over these processes is not. For this reason MaxConnections would be difficult to use in such an environment.

If the ESI cache is used, it is fragmented or duplicated between the child processes, which is ineffective.

When a backend application server is misbehaving, each IBM HTTP Server child process must detect the markdown and the ultimately retry — this information is not shared between child processes.

A crash of an individual child process affects between 1 and 100 threads only, depending on how busy the child process is.
<IfModule worker.c>
ServerLimit 20
# Upper limit on ThreadsPerChild
ThreadLimit 100

# Start with all 20 processes
StartServers 20

MaxClients 2000
MinSpareThreads 2000

# Don't kill child processes in response to load
MaxSpareThreads 2000

# Constant number of child processes, 2000/100 = 20.
ThreadsPerChild 100
MaxRequestsPerChild 0

# On linux, you may need to set ulimit -s 512
# in IHS/bin/envvars for high ThreadsPerChild
</IfModule>


Example 3:

This configuration uses multiple IBM HTTP Server child processes, with a variable capacity of 2000 concurrent connections spread amongst 1-20 child processes.

If the ESI cache is used, it is fragmented or duplicated between the child processes, which is ineffective.

When a backend application server is misbehaving, each IBM HTTP Server child process must detect the markdown and the ultimately retry — this information is not shared between child processes.

To mitigate the impact to the WebSphere Application Server Web server plug-in, you can configure IBM HTTP Server to conservatively create new child processes and aggressively kill off old ones. This ensures that the smallest working set of child processes will know about ESI, markdowns, and so on.

A crash of an individual child process affects between 1 and 100 threads only, depending on how busy the child process is.

Note: While MinSpareThreads and MaxSpareThreads do not have to be multiples of ThreadsPerChild, it is convenient to use multiples of ThreadsPerChild to ensure that there is no churn between creating a new child process and immediately killing it.

<IfModule worker.c>
ServerLimit 20
# Upper limit on ThreadsPerChild
ThreadLimit 100

# Start with only 2 processes
StartServers 2

MaxClients 2000

# Don't create a new process until we
# have one process worth free.
MinSpareThreads 100

# Kill a child when we've got an entire extra process
MaxSpareThreads 100

# Constant number of child processes, 2000/100 = 20.
ThreadsPerChild 100
MaxRequestsPerChild 0

# On linux, you may need to set ulimit -s 512
# in IHS/bin/envvars for high ThreadsPerChild

</IfModule>

Notes:

 

Cross Reference information
Segment Product Component Platform Version Edition
Application Servers WebSphere Application Server IBM HTTP Server AIX, HP-UX, Linux, Solaris, Windows 7.0, 6.1, 6.0.2, 6.0