message throttling?

[Sivaram Neelakantan]

I've been tinkering around with kafka and other bits of streaming
software. In order to test them, is there any utility to send text or
binary data at a specific rate to a specific port?

Is it netcat or socat? Has anyone done anything of this sort? A few
examples would be appreciated.

sivaram
--

[Janis Papanagnou]

On 07.04.2017 06:01, Sivaram Neelakantan wrote:
>
> I've been tinkering around with kafka and other bits of streaming
> software. In order to test them, is there any utility to send text or
> binary data at a specific rate to a specific port?

Not quite sure about your actual requirements. For simple sending (or
receiving) data over TCP/UDP to/from some port on a script application
level I used GNU awk's network programming capabilities
https://www.gnu.org/software/gawk/manual/gawk.html#TCP_002fIP-Networking
which are high level thus convenient to use. But I don't know about any
way to control the [lower OSI-level] transmission rate. You may be able
to program rough delays on application level, though not sure that fits
in your case.

Janis

[Janis Papanagnou]

On 07.04.2017 06:59, Janis Papanagnou wrote:
> On 07.04.2017 06:01, Sivaram Neelakantan wrote:
>>
>> I've been tinkering around with kafka and other bits of streaming
>> software. In order to test them, is there any utility to send text or
>> binary data at a specific rate to a specific port?
>
> Not quite sure about your actual requirements. For simple sending (or
> receiving) data over TCP/UDP to/from some port on a script application
> level I used GNU awk's network programming capabilities
> https://www.gnu.org/software/gawk/manual/gawk.html#TCP_002fIP-Networking
> which are high level thus convenient to use. But I don't know about any
> way to control the [lower OSI-level] transmission rate. You may be able
> to program rough delays on application level, though not sure that fits
> in your case.

In addition, as I just recalled, there's a similar feature in Kornshell
available (not sure about such a feature in other shells); from the man
page:

"In each of the following redirections, if file is of the form
/dev/sctp/host/port, /dev/tcp/host/port, or /dev/udp/host/port, where
host is a hostname or host address, and port is a service given by name
or an integer port number, then the redirection attempts to make a tcp,
sctp or udp connection to the corresponding socket."

[Ian Zimmerman]

Maybe use the traffic shaping feature of your kernel (hoping there is
one, of course).

--
Please *no* private Cc: on mailing lists and newsgroups
Personal signed mail: please _encrypt_ and sign
Don't clear-text sign: http://cr.yp.to/smtp/8bitmime.html

[Thomas 'PointedEars' Lahn]

Sivaram Neelakantan wrote:

> I've been tinkering around with kafka and other bits of streaming
> software. In order to test them, is there any utility to send text or
> binary data at a specific rate to a specific port?

Probably yes.

> Is it netcat or socat?

As I did not know socat(1) [thanks], it would have been netcat (nc(1)).
Now that I know the former, I would consider it, too. Depends.

> Has anyone done anything of this sort?

Probably yes.

> A few examples would be appreciated.

You could write a shell script using sleep(1) to “read -n” from a stream
nchars characters at a time (see e.g. bash(1)) and pipe that into nc(1).
[I have learned of the read-sleep approach via
<https://superuser.com/q/609387/125462> to which I contributed
<https://superuser.com/a/1186711/125462>.]

# output one character of "foo" every second
echo foo |
# subshell avoids backing up and restoring $IFS
(
# read space and newline, too
IFS=''

while read -n 1 -r char
do
printf '%s' "$char"
sleep 1
done
)

With GNU sleep(1) you could have delays shorter than one second; with POSIX
sleep(1) you could compensate for the lack of this capability with reading
more characters at once.

buffer(1) can give you chunks, but apparently not specific throttling.

As I discovered via <https://www.howtogeek.com/50794/keep-rsync-from-using-all-your-bandwidth/> (Google: “rsync throttle”), rsync(1) can give you the
throttling with “--bwlimit”, but only for connections that rsync(1) can
make.

As Ian indicated, if userspace cannot do it, your kernel might. Linux
4.9.2, for example, appears to include *packet-based* traffic shaping
support, starting from the CONFIG_NET_SCHED kernel configuration option.
The key word to look for appears to be “QoS” (Quality of Service).
[I have never used it.]

How long have you STFW for this?

<http://catb.org/esr/faqs/smart-questions.html>

--
PointedEars

Twitter: @PointedEars2
Please do not cc me. / Bitte keine Kopien per E-Mail.

[Thomas 'PointedEars' Lahn]

Thomas 'PointedEars' Lahn wrote:

> You could write a shell script using sleep(1) to “read -n” from a stream
> nchars characters at a time (see e.g. bash(1)) and pipe that into nc(1).
> [I have learned of the read-sleep approach via
> <https://superuser.com/q/609387/125462> to which I contributed
> <https://superuser.com/a/1186711/125462>.]
>
> # output one character of "foo" every second
> echo foo |
> # subshell avoids backing up and restoring $IFS
> (
> # read space and newline, too
> IFS=''
>
> while read -n 1 -r char
> do
> printf '%s' "$char"
> sleep 1
> done
> )
>
> With GNU sleep(1) you could have delays shorter than one second; with
> POSIX sleep(1) you could compensate for the lack of this capability with
> reading more characters at once.

Note that POSIX read(1) does not have an “-n” option. Short of compiling
such a utility (it should should be too difficult to write even in C), I
currently have no idea for an adequate replacementment:

cut(1) with the “-b” or “-c” option would literally cut it in a standards-
compliant way, but you would have to store the data in a variable which is
different from processing a stream. A

variable=$(cat)

statement to obtain the data might never terminate or might truncate it.

Likewise, a POSIX-compliant awk program could split an input stream into
characters and output one (or more) at a time followed by system("sleep …"),
but I do not know how awk(1) would handle input streams of varying length.

echo foo |
awk -v RS='' '{
split($0, a, "");
for (key in a) {
printf("%s", a[key]);
system("sleep 1");
}
}'

<http://pubs.opengroup.org/onlinepubs/9699919799/utilities/read.html>
<http://pubs.opengroup.org/onlinepubs/9699919799/utilities/cut.html>
<http://pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html>

[Helmut Waitzmann]

Thomas 'PointedEars' Lahn <Point...@web.de>:

> Thomas 'PointedEars' Lahn wrote:
>
>> You could write a shell script using sleep(1) to “read -n” from a stream
>> nchars characters at a time (see e.g. bash(1)) and pipe that into nc(1).
>> [I have learned of the read-sleep approach via
>> <https://superuser.com/q/609387/125462> to which I contributed
>> <https://superuser.com/a/1186711/125462>.]
>>
>> # output one character of "foo" every second
>> echo foo |
>> # subshell avoids backing up and restoring $IFS
>> (
>> # read space and newline, too
>> IFS=''
>>
>> while read -n 1 -r char
>> do
>> printf '%s' "$char"
>> sleep 1
>> done
>> )
>>
>> With GNU sleep(1) you could have delays shorter than one second; with
>> POSIX sleep(1) you could compensate for the lack of this capability with
>> reading more characters at once.
>
> Note that POSIX read(1) does not have an “-n” option. Short of compiling
> such a utility (it should should be too difficult to write even in C), I
> currently have no idea for an adequate replacementment:

“dd” can read one character at a time.

echo foo |
{
until
LC_NUMERIC=POSIX LC_MESSAGES=POSIX dd bs=1 count=1 2>&1 1>&3 3>&- |
{
grep -F -q -x -e '0+0 records in' && cat
} > /dev/null 3>&-
do
sleep 1 3>&-
done 3>&1
}

[Sivaram Neelakantan]

On Fri, Apr 07 2017,Janis Papanagnou wrote:

> On 07.04.2017 06:59, Janis Papanagnou wrote:

[snipped 6 lines]

>> Not quite sure about your actual requirements. For simple sending (or
>> receiving) data over TCP/UDP to/from some port on a script application
>> level I used GNU awk's network programming capabilities
>> https://www.gnu.org/software/gawk/manual/gawk.html#TCP_002fIP-Networking
>> which are high level thus convenient to use. But I don't know about any
>> way to control the [lower OSI-level] transmission rate. You may be able
>> to program rough delays on application level, though not sure that fits
>> in your case.
>
> In addition, as I just recalled, there's a similar feature in Kornshell
> available (not sure about such a feature in other shells); from the man
> page:
>
> "In each of the following redirections, if file is of the form
> /dev/sctp/host/port, /dev/tcp/host/port, or /dev/udp/host/port, where
> host is a hostname or host address, and port is a service given by name
> or an integer port number, then the redirection attempts to make a tcp,
> sctp or udp connection to the corresponding socket."
>
>
> Janis
>

Thanks, will try them out.

[snipped 9 lines]



sivaram
--