Hub Performance

This article is meant to discuss the following:

1. How to write a hub that performs well.
2. How well existing hubs perform.

Threaded vs. Non-threaded

Threaded performance

Non-threaded performance

Importance of readiness-notification mechanism?

Poll vs. Select vs. /dev/poll vs. /dev/epoll vs. kqueue vs RT signals etc. Experiences?

Buffer management

Pooled buffers?

Does anyone have any performance points comparing between pooled buffers and static buffers?

Do not abuse malloc()/free(). This could make your system process choke. The best way is to cache mostly used commands such as

hub <-> hub $Up $UpToo

client <-> hub $MyINFO $UserIP

DConnect Daemon because of this speeded up and system process is not choked.

Note: If you use pooled buffers and reuse the most recently used buffer, this could make the kernel copy your data instead of using zero-copy. This is at least the case on FreeBSD. For a send to be zero-copy on FreeBSD, you need to:

1. Send at least one page of data, page-aligned. (Question is: does the end also have to be page-aligned/size be a multiple of the page size?)
2. Don't overwrite the data before the kernel has freed the buffer. There is no indication of when this happens. The best way to guarantee the data has been written is setting a sendbuffer size that is appropriate, and make sure you have sent out as twice as much data as the sendbuffer size before reusing a buffer.

I guess similar restrictions apply to Linux.

Zero-copy receive has much more complex requirements under FreeBSD, such as a MTU > pagesize, making most NICs useless for that. Linux might be better.

Send strategy

Coalescing small sends into larger ones should be beneficial for broadcast messages where only one memcpy per message is needed to merge the buffers, then the final buffer can be reused for every socket.

Zero-copy should be good for high-bandwidth/slow CPU hubs. FreeBSD 5.x is supposed to have it, I've got no info on Linux or Windows yet.

Profiling

Here are some statistics gathered by fusbar.

((from a 800-2200 user hub over 6 days))

Bandwidth Estimation

The running formula for estimating the amount of bandwidth you need given a user count:

(Number of users/20)^2 kilobits per second

This formula was originally given by Verliba, author of verlihub