cstream
cstream is a general-purpose stream-handling tool like UNIX dd, usually used in commandline-constructed pipes.
cstream-3.0.0 adds ipv6 support. Please try it out and give feedback.
Features:
- Sane commandline switch syntax.
- Exact throughput limiting, on the incoming side. Timing variance in previous reads are counterbalanced in the following reads.
- Precise throughput reporting. Either at the end of the transmission or everytime SIGUSR1 is received. Quite useful to ask lengthy operations how much data has been transferred yet, i.e. when writing tapes. Reports are done in bytes/sec and if appropriate in KB/sec or MB/sec, where 1K = 1024.
- SIGHUP causes a clean shutdown before EOF on input, timing information is displayed.
- Build-in support to write its PID to a file, for painless sending of these signals.
- Build-in support for fifos. Example usage is a 'pseudo-device', something that sinks or delivers data at an appropriate rate, but looks like a file, i.e. if you test soundcard software. See the manpage for examples.
- Built-in data creation and sink, no more redirection of
/dev/null
and/dev/zero
. These special devices speed varies greatly among operating systems, redirecting from it isn't appropriate benchmarking and a waste of resources anyway. - Accepts 'k', 'm' and 'g' character after number for "kilo, mega, giga" bytes for overall data size limit.
- "gcc -Wall" clean source code, serious effort taken to avoid
undefined behavior in ANSI C or POSIX, except
long long
is required. Limiting and reporting works on data amounts > 4 GB.
Short usage instructions
Changes in recent versions
Full manual page
Short usage instructions:
cstream by Martin Cracauer - version 4.0.0 -V = print version number to stdout and exit with 0 -v <n> = verbose [default: off] 0 = nothing 1 = report bytes transferred and throughput 2 = also throughput after first read/write 3 = also seperate throughput for read and write (unimplemented) 3 = verbose stats on every read/write -b <n> = blocksize [default: 8192] -B <n> = buffer (at most) <n> bytes [default: one block] -c <n> = Concurrency, writing done by a seperate process 0 = no concurrency, one one process 1 = read side buffers 2 = write side buffers 3 = both sides buffer, -B amount of data will be transferred at once -n <n> = overall size of data [default: unlimited] -t <n> = throughput in bytes/sec [default: unlimited] if positive, bandwith is average over whole session. if negative, every write is delayed to not excceed. -i <s> = name of input file, - = generate stream yourself to use stdin, use -i '' -o <s> = name of output file, - = just sink data to use stdout, -o '' -I <s> = Type of input file -O <s> = Type of ouput file 'f' = fifo (create it) 'F' = issue fsync(2) on output file before closing 'a' = set audio modes on file (i.e. CD quality) 'N' = don't use TCP even if filename has ':' 't' = tee - in addition to outfile, copy stream to fd 3 'D' = O_DIRECT 'S' = O_SYNC 'M' = use mmap for output file (do not use write(2)), for hugetlbfs [Multiple chars allowed] -p <s> = Write pid as ascii integer to file <s> -l include line count in statistics -w <n> = Set write block size (-c 5 only) -S Don't output statistic on SIGINFO -T <n> = Report throughput every <n> seconds SIGINFO causes statistics to be written to stderr SIGUSR1 causes statistics to be written to stderr SIGUSR2 causes loop end after next buffer transfer <file> if -i has not been used, specifies input file -6 <n> Use IPV6: -1 = don't, 1 = allow both, 2 = force v6 On some platforms server mode 1 forces ipv6, as they don't open both v4 and v6 ports from one bind call.
Changes in recent versions:
3.2.4: ------ Fix incorrect rejection of large block sizes. Fixes from Jimmy Olgeni. Fixes from Hans Ulrich Niedermann. 3.2.3: ------ Fix bug when invoking with no arguments. 3.2.2: ------ Move throughput measuring end to after close(2). 3.2.1: ------ print throughput after fsync, if requested. 3.2.0: ------ -O F option (do an fsync at the end of the output file). When closing files fails, exit with an error code. 3.1.1: ------ -nwas not clear to use numbers > 2 GB. I didn't notice since it worked fine if you used suffixes "K/M/G" as long as the number was < 2 G. Sorry about that. 3.1.0: ------ O_DIRECT supported for input. 3.0.0: ------ IPV6 support for IPV6 day 2011. The IPV6 support shouldn't break anything after hostname lookup succeeds. Issues might be building on older platforms if I screwed up the autoconf mechanism to not compile it in. Having said that, assorted little code cleanups are in this release, too. They shouldn't break anything but non-IPV6 things were touched. Aftermath for 2.7.4 - 2.7.6: ---------------------------- ATTENTION: I'm afraid that support for the '-B ' was clobbered in 2.7.4 or whereabouts. This one allowed you do have a reader do multiple reads before the writer could write or vice versa. Change reverted in 2.7.6. 2.7.4 and 2.7.5 do not have working -B. 2.8.0: ------ Support platforms that do not have open(2) with O_DIRECT. Such as MacOS X. 2.7.6: ------ Revert 2.7.3 which broke -B 2.7.5: ------ NetBSD and general pkgsrc compatibility. Should get rid of the only patch used in pkgsrc. 2.7.4: ------ Print the message that we switch to normal from O_DIRECT only when verbose > 0. SEE ABOVE, broke -B. CHANGE reverted 2.7.3: ------ More c flags changes for more portability. 2.7.2: ------ Fix compilation under Redhat-7.3. 2.7.1: ------ Support for $CSTREAM_AUDIO_BITRATE. 2.7.0: ------ Support for O_SYNC and O_DIRECT, but only on the output side for now. Please send mail if you want this for input. 2.6.1: ------ Timer-based throughput reports from -T will now also print what the throughput since the last report was. Useful when performance changes during runtime. 2.6.0: ------ Bandwidth limiting was not > 4 GB clean on platforms with 32 bit integers. I had lost an already existing fix here. 2.5.2: ------ Commandline options, options to -O and -I: -ON -IN which prevents cstream from interpreting filenames with ":" in them as TCP specifications. By default the presense of a colon means host:port. 2.5.1: ------ Add option -T which will print the statistics every seconds. 2.5: ---- There was a 4 GB limitation on 32 bit machines when you used bandwith limiting. Thanks to Sergey M. Serov for reporting! 2.4.1b: ------- Use SO_REUSEADDR on serving socket. 2.4: ---- Bandwidth limiting is not not only as a strict maximum at any point in time but now you can also limit for the session as a whole. Do so by giving a negative bandwidth limit. To make it clear: if the session was running below the limit for some time for other bottlenecks in the chain, then it will run above the limit to make good for it at other times when it can. 2.3: ---- Identical to 2.3b3, sigchld handling is assumed to be correct now and works with various versions of the Linux kernel. Version 2.3 has sound support, see e.g. `cstream -Oa -o /dev/dsp`. 2.3b3: ------ Further SIGCHLD fixes. 2.3b2: ------ Fix SIGCHLD handling so that stopping and continuing the child in a concurrent mode would not terminate. 2.3b1: ------ Version 2.3 has sound support, see e.g. `cstream -Oa -o /dev/dsp` version 2.2: ------------ TCP/IP socket support. Linux glibc 64 bit files. Version 2.1: ------------ - Add audio mode: -O a/-I a try to switch output file into CD-quality 16bit 44100Hz stereo mode. If input is just generated, a sine wave of 440 Hz will be played. - Add tee-fd mode. -O t will copy the stream to file descriptor 3 (which must already be opened by parent process). Version 2.0: ------------ - Add -c flag for concurrent mode. - Add -B flag (buffering of multiple input blocks before output blocks are written). - Add -l flag (statistics will include line count). - If input data is just 'generated', use a buffer that remotely resembles ASCII text with a line length of 76 chars and has a newline at the end of data. - Number of bytes transferred is also displayed in GB/MB/KB, if appriopriate. - Change throughput report to be shorter. - Add automatic tests `make check`. - Througput reporting on signals got the wrong time when cstream was not called with -v > 0. - Use SIGINFO for throughput report. - Use more values from configure script. - Throughput report is now asynchronous, not after next read() or write(). - Throughput report will report buffer fillage if -B is in use. Version 1.4: ------------ - Fix Linux compilation problems. - The -b and -t options now also recognize 'k', 'm', 'g' for kilo-, mega-, gigabyte like the -n option did. - The signal for shutdown is now SIGUSR2, not SIGHUP. Version 1.3: ------------ - Fix permissions of created files - You can now specify the input file without using a switch (if it is the last argument) - Improve messages Version 1.2: ------------ - Argument checking for -v was broken, fixed. Version 1.1: ------------ - Fix version number reporting, introduce -V switch. - Fix usage() typos. Version 1.0: ------------ - Initial version.
Full manual page:
cstream(1) FreeBSD General Commands Manual cstream(1) 1mNAME0m 1mcstream 22m-- direct data streams, with bandwidth limiting, FIFO, audio, du- plication and extended reporting support. 1mSYNOPSIS0m 1mcstream 22m[1m-b 4m22mnum24m] [1m-B 4m22mnum24m] [1m-i 4m22mfilename24m] [1m-I 4m22mstring24m] [1m-l22m] [1m-n 4m22mnum24m] [1m-o 4m22mfilename24m] [1m-O 4m22mstring24m] [1m-p 4m22mfilename24m] [1m-t 4m22mnum24m] [1m-T 4m22mnum24m] [1m-v 4m22mnum24m] [1m-V22m] [4mfilename24m] 1mDESCRIPTION0m 1mCstream 22mfilters data streams, much like the UNIX tool dd(1). It has a more traditional commandline syntax, support for precise bandwidth limit- ing and reporting and support for FIFOs. Data limits and throughput rate calculation will work for files > 4 GB. 1mCstream 22mreads from the standard input and writes to the standard output, if no filenames are given. It will also 'generate' or 'sink' data if de- sired. Options: 1m-b 4m22mnum24m Set the block size used for read/write to 4mnum24m. The default is 8192 bytes. 1m-B 4m22mnum24m Buffer input up to 4mnum24m bytes before writing. The default is the blocksize. It is an error to set this to anything below the blocksize. Useful when writing tapes and similar that prefer few large writes of many small. 1m-c 4m22mnum24m Concurrent operation. Use a separate process for output. This is especially useful in combination with the -B option. 0 = use one process only (default) 1 = read process will buffer 2 = write process will buffer 3 = both processes will buffer. In combination with a large buffer size this will often load your memory heavily, every time the reader transfers the buffer it collected to the writer. If you use -c 3 and have a buffer size of 128 Megabytes 256 MB of memory will be touched at once. 1m-i 4m22mnum0m 1m-o 4m22mnum24m Set the file names to use for input or output, respectively. If the output file name is "-", data will just be discarded. If the input file name is "-", data will be generated 'out of the void'. If these options aren't given, stdin/stout will be used. If you need to give 1m-o 22mor 1m-i 22moptions and want stdin/stdout, specify the empty string, like this: cstream -i'' If TCP support has been compiled in (default), hostname:port- number will try to connect to the specified host at the speci- fied port and :portnumber will open a TCP socket on the local machine and wait for a connection to arrive. SECURITY NOTE: cstream includes no mechanism to restrict the hosts that may connect to this port. Unless your machine has other network filters, anyone will be able to connect. 1m-I 4m22mstring0m 1m-O 4m22mstring0m Specify the type of input and output file, respectively. If string includes 'f', a fifo will be created. If string includes 'F', a rsync(2) will be issued before closing the output file. If string includes 'a', the file will be assumed to be a opensound- compatible audio device and will be switched to CD-like settings. If string includes 't', a copy of the stream will be sent to file de- scriptor 3. If string includes 'N', TCP will not be used for that file even if the name has a ":". 1m-l 22mInclude line count in statistics. 1m-n 4m22mnum24m Limit the total amount of data to 4mnum24m. If there is more input available, it will be discarded, 1mcstream 22mwill exit after the limit has been reached. If there is less input, the limit will not be reached and no error will be signaled. 4mnum24m may have a trailing 'k', 'm' or 'g' which means Kilobytes, Megabytes or Gigabytes (where Kilo = 1024). This applies to all numeric options. 1m-p 4m22mfilename0m Write the process id of cstream to 4mfilename24m. If cstream uses a separate writer process (option -c), this is the pid of the parent (reader) process. 1m-t 4m22mnum24m Limit the throughput of the data stream to 4mnum24m bytes/second. Limiting is done at the input side, you can rely on cstream not accepting more than this rate. If the number you give is posi- tive, cstream accumulates errors and tries to keep the overall rate at the specified value, for the whole session. If you give a negative number, it is an upper limit for each read/write system call pair. In other words: the negative number will never exceed that limit, the positive number will exceed it to make good for previous underutilization. 1m-T 4m22mnum24m Report throughput every num seconds. 1m-v 4m22mnum24m Set verbose level to 4mnum24m. By default, it is set to 0, which means no messages are displayed as long as no errors occur. A value of 1 means that total amount of data and throughput will be displayed at the end of program run. A value of 2 means the transfer rate since the end of the first read/write pair will also be reported (useful when there is an initial delay). A value of 3 means there will also be separate measurements for read and write. This option is resource-consuming and currently isn't implemented. A value of 4 means that notices about each single read/write will be displayed. High values include all message types of lower values. 1m-V 22mPrint version number to stdout and exit with 0. 4mfilename24m A single filename as the last argument without an option switch will be used as input file if -i has not been used. 1mSIGUSR10m 1mSIGINFO 22mSending SIGUSR1 (or SIGINFO, which is usually mapped to Con- trol-T on you keyboard) to cstream causes it to display throughput rates to stderr. The stream will continue as if nothing happened. 1mSIGUSR2 22mExit and report throughput rates, if requested. 1mSIGHUP 22mI found myself sending SIGHUP accidentally too often. But ig- noring or misusing SIGHUP is not an option for me. Thus, when 1mcstream 22mreceived SIGHUP, it will wait 5 seconds for another SIGHUP, to give users a chance to correct a possible mistake. If no additional SIGHUP is received, 1mcstream 22mkills itself with SIGHUP. 1mEXAMPLES0m 1mcstream -o tmpfile -v 1 -n 384m -i -0m Writes 384 Megabytes of unspecified data to file 4mtmpfile24m and dis- play verbose throughput rate. Makes a good benchmark, the speed of 4m/dev/null24m varies too much from system to system. 1mcstream -i tmpfile -v 1 -n 384m -o -0m Read the same file back in and discard data. 1mcstream -b 2000 -t 10000 /var/log/messages0m Will display the file in a more or less watchable speed. 1mdump 0sf 400000 - / | cstream -v 1 -b 32768 -o /dev/rst0 -p pidfile0m 1mkill -USR1 `cat pidfile`0m Write the output from dump(1) to tape. Each time the signal is sent, the throughput and data rate so far will be displayed. 1mcstream -t 176400 -i /dev/dsp0 -I f -o -0m Makes kind of a soundcard emulator which may be used to test au- dio applications that need something to write to that limits the data rate as a real soundcard does. This obviously doesn't work when the application tries to write data using mmap(2) and the application has to ignore errors when it tries to set soundcard parameters using ioctl(2). 1mcstream -t 176400 -i /dev/dsp0 -I f -o /dev/dsp1 -O f0m Similar soundcard emulator, except that it allows you to grab the data your applications sends to it from the other fifo, while still having precise timing. 1mcstream -Oa -o /dev/dsp0 myhost.mydomain.com:173240m Connects port 3333 on host myhost.mydomain.com and whatever data it finds there will be sent to the soundcard, with appropriate settings for CD quality stereo play. 1mcstream -i myaudiofile.raw -o :173240m This will open a TCP server on port 17324 and waits until someone connects (for example, the commandline from the previous exam- ple). Then it will send the contents of myaudiofile.raw down the TCP stream (for the previous audio example, typically a CD audio track like you get from the tosha or cdparanoia utilities). 1mcstream -OD -o myfile0m Write to file myfile with O_DIRECT. That usually means that the filesystem buffer cache will not try to cache this file. You can use that to prevent copying operations from eating up physical memory. Note that when cstream encounters a write error it will switch the output file from O_DIRECT to a normal file and write all further blocks without O_DIRECT if writes without O_DIRECT succeed. In practice that usually means that your last block, if not a multiple of the filesystem block size, will still be writ- ten into the file (the maximum amount of data written without O_DIRECT is your blocksize minus one). That way cstream ensures that the output file has the length of the input, however odd the length was and no matter what restrictions your OS places on O_DIRECT output. Again, cstream will *not* pad the output to the block size, you get the same file and file size as if not using O_DIRECT, at the cost of switching to non-O_DIRECT whenever a block is not the right size. 1mcstream -i :3333 22m| 1mdd obs=8192 22m| 1m./cstream -omyfile -v7 -OD0m This is what you need to do to buffer TCP input, so that the last cstream will not switch away from O_DIRECT prematurely because of short reads. If your input can do short reads (e.g. from TCP), and you want to ensure that O_DIRECT stays in effect, you need a buffer between the TCP stream and the O_DIRECT stream. Since cstream does not yet support different input and output block sizes, dd is suitable here. Note that this is only necessary if the OS requires multiples of the filesystem block size for O_DI- RECT. At the time of this writing this construct is needed on Linux for using TCP streams with O_DIRECT, but it is not needed on FreeBSD. 1mcstream -OS -o myfile0m Writes to file myfile with O_SYNC. This means by the time the system call returns the data is known to be on disk. This is not the same thing as O_DIRECT. O_DIRECT can do its own buffering, with O_SYNC there is no buffering at all. At the time of this writing, O_SYNC on both Linux and FreeBSD is very slow (1/5th to 1/10th of normal write) and O_DIRECT is reasonably fast (1/4th to 1/2 of normal write). You can combined O_SYNC and O_DIRECT. 1mERRORS0m Exit code 0 means success. Exit code 1 means a commandline syntax usage error. Exit code 2 means other errors, especially system errors. 1mBugs0m There should be an option to begin writing directly after the first read ended and then fill the buffer with reads in the background. Right now writing will not begin before the reader has filled the buffer completely for the first time. Not a bug: the code to do O_DIRECT is reasonably sophisticated. It will fall back to normal I/O on errors. But before doing that it knows about both filesystem blocksize requirements (will default I/O blocksize to whatever the filesystem of the output file is in) and page alignment re- quirements (I/O will happen from a page-aligned buffer). However, the combination of concurrent read/writes (-c options) and O_DIRECT has not been tested beyond basic verification that it gets some tests right. 1mSEE ALSO0m dd(1), mkfifo(2) 1mHISTORY0m 1mcstream 22mwas initially written by Martin Cracauer in 1998. For updates and more information see http://www.cons.org/cracauer/cstream.html FreeBSD 13.0 March, 30, 1999 FreeBSD 13.0