How to implement this in golang?

[LetGo]

I have a proxy written in python with some logic that I would like to implement in a golang tool of mine, but I don't really know how to do it.

It splits the data sent via socket (stdout) in small chunks, with a delay between each other

I have a variable which gets a random number from a list:

....

 listbytes = [87, 88, 89, 90]

....

I have also a function which splits in small chunk of bytes(n) the data(lst):

............

def chunks(lst, n):

    "Yield successive chunks from lst, where n is a list of possible sizes"

    i = 0

    while i < len(lst):

        k = min(random.choice(n), len(lst) - i)

        yield lst[i:i + k]

        i += k

.............

Both things are executed this way:

...

# get the data

            data = s.recv(BUFFER_SIZE)

 

            if s == s_src:

                d = LOCAL_DATA_HANDLER(data)

                for chunk in chunks(d, listbytes): 

#sleep 1.6 seconds before sending the next chunk of data

                    time.sleep(1.6)

#send the chunked data

                    s_dst.send(bytes(chunk) )

... 

How do I implement the same exact logic illustrated here?

func (nObj NetObject) RunClient(cmd string) {

    // Try connection

    for {

        conn, err := net.Dial(nObj.Type, nObj.Service)

        fmt.Print("connect")

//        msg := "status"

        if err != nil {

            fmt.Println("fail")

        } 

        if err == nil {

            fmt.Println("ok")

//            defer conn.Close()

            defer conn.Close()

            log.Println("Connected to", conn.RemoteAddr())

            handleConn(conn, cmd)

            //handleConn(conn, cmd)

            fmt.Println("After handle")

                    } 

        fmt.Println("Before sleep")

        time.Sleep(5 * time.Second)

    }

}

Can you help me please?

[LetGo]

I forgot this:

// Manage connection for different behavior

func handleConn(conn net.Conn, binPath string) {

        msg := "Status?\n"

            if binPath != "" {

// Execute command and send Standard I/O net.Conn

cmd := exec.Command(binPath)

cmd.Stdin = conn

cmd.Stdout = conn

cmd.Stderr = conn

cmd.Run()

                fmt.Println("Disconnected!?")

                time.Sleep(5 * time.Second)

                pippo, err2 := conn.Write([]byte(msg))

                if err2 != nil {

                       fmt.Println(pippo)

                }

              

    } else {

// Copy Standard I/O in a net.Conn

go io.Copy(os.Stderr, conn)

go io.Copy(os.Stdout, conn)

io.Copy(conn, os.Stdin)

    }

}

[Brian Candler]

What sort of socket - TCP or UDP?  TCP is an infinite stream, and there is no reliable way to divide it into "chunks" (except possibly using the URG pointer, which is very rarely used).  There is no guarantee that waiting 1.6 seconds between sends will actually send in separate segments being sent, nor that the receiver will be able to determine that these were separate segments - they may well be merged.

To do it reliably, you'd want to add your own framing, e.g. send an 4-byte length followed by that number of bytes.

If you show a *complete* go program, together with a description of how it isn't working to your satisfaction, then maybe someone can help you fix it.  If you want to make it behave the same as some given python program, then maybe you should show the python program as well.

[LetGo]

Thanks for the answer! (:

In python it was straightforward to implement and it works like a charm. It sends small packets with delay between each other without even care if it is UDP or TCP:

The connection ( if  UDP/ TCP )  is handled independently, the sending part ( the one I pasted ) is equal for both of them, its just an "if" in the code.

The server ( which I control )  does riassembling by himself, so its fine.

I know that providing a solution is not an easy task, I know it for sure.

I'm just looking for a pro ( a saint ahaha ) with some patience that provides an example of code of how it should look like to work as I wish it does.

I do not expect do make it work in first place, its almost impossible, but we all know, with some errors you get the solution before or after(:

In this case I don't really know where to start and how to do it ( im a newbie in golang ), so I came here to ask help

[Robert Engels]

With Go you use a Go routine per connection and then you can read in chunks and delay in an imperative style. No need for generators or async. 

On Jun 29, 2021, at 10:25 AM, LetGo <non3...@gmail.com> wrote:

Thanks for the answer! (:

In python it was straightforward to implement and it works like a charm. It sends small packets with delay between each other without even care if it is UDP or TCP:

<prooff.png>

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f82a73ad-164e-4af5-b183-9d4bf0b54911n%40googlegroups.com.

<prooff.png>

<proof2.png>

[Marvin Renich]

[Please do not send images of text; copy and paste as text into the
message or into text attachments.]

* LetGo <non3...@gmail.com> [210629 11:25]:

> Thanks for the answer! (:
> In python it was straightforward to implement and it works like a charm. It
> sends small packets with delay between each other without even care if it
> is UDP or TCP:
>
> [image: prooff.png]
> [image: proof2.png]

[snip]

> Il giorno martedě 29 giugno 2021 alle 14:22:32 UTC+2 Brian Candler ha

> scritto:
>
> > What sort of socket - TCP or UDP? TCP is an infinite stream, and there is
> > no reliable way to divide it into "chunks" (except possibly using the URG
> > pointer, which is very rarely used). There is no guarantee that waiting
> > 1.6 seconds between sends will actually send in separate segments being
> > sent, nor that the receiver will be able to determine that these were
> > separate segments - they may well be merged.
> >
> > To do it reliably, you'd want to add your own framing, e.g. send an 4-byte
> > length followed by that number of bytes.

What Brian is saying is that TCP, by definition, is not required to use
the same packets across the wire that you send out. Packets in a TCP
stream _may_ be regrouped, both by being split into smaller packets and
by being joined into larger packets. They are reassembled at the
receiving end of the TCP stream as a stream, not as individual packets.

It is very feasible for the receiving end to see packets with exactly
the contents of the individual "send" function calls when under normal
load, but see something completely different when CPU, network, or
memory pressure is applied to the sending system.

...Marvin

[Jesper Louis Andersen]

On Tue, Jun 29, 2021 at 5:24 PM LetGo <non3...@gmail.com> wrote:

Thanks for the answer! (:

In python it was straightforward to implement and it works like a charm. It sends small packets with delay between each other without even care if it is UDP or TCP:

Beware! This is an assumption that will break at some point in time. Currently the delay and the OS makes things straightforward for you. But TCP doesn't behave like you expect, and you are very likely to run into trouble if the machine, the network, or the system starts taking additional load.

You need to frame the data. A good way is to use 4 bytes as a size (unsigned 32 bit integer), followed by a payload of that size. You can then avoid this becoming an uncontrolled explosion in your software at a later date. You can also close connections early if too large messages get sent, etc.

[LetGo]

Thank you guys for all your answers and suggestions!

I really appreciate!

Sorry about the screenshots, it was the only way to make the packets "human readable"

How could you code that kind of implementation based on your knowledge and skill?

I have noone of these in golang ahah as I said, im too newbie to do all this alone!

Also not working examples ( if they throw an error I don't care, based on my code are fine!

These examples could rapresent a great start from me!(:

[LetGo]

I think I made some progress.... I think. Is it right what I'm doing ?

................

cmd.Stdin = conn

// cmd.Stdout = conn

//         data := []byte(cmd.Stdout)

        data := cmd.Stdout

        var i int

    for {

    n := int(math.Min(float64(rand.Intn(len(data))), float64(len(data))))

    d := data[i : i+n]

    i += n

    time.Sleep(400 * time.Millisecond)

    d = conn

        if i >= len(data) {

    break

        }

            }

cmd.Stderr = conn

cmd.Run()

............................

But when I try to build I get these errors:

conn.go:75:46: invalid argument data (type io.Writer) for len

conn.go:76:16: cannot slice data (type io.Writer)

conn.go:83:22: invalid argument data (type io.Writer) for len

[Brian Candler]

You haven't shown which lines 75, 76 and 83 correspond to.  It's easier if you put the whole code on play.golang.org, and we'll be able to point to the error.  

But I'm guessing it's this:

 data := cmd.Stdout

...

n := int(math.Min(float64(rand.Intn(len(data))), float64(len(data))))  << line 75?

d := data[i : i+n]  << line 76?

...

        if i >= len(data) {   << line 83?

If I'm right, the compiler is saying: cmd.Stdout (which you assigned to 'data') is of type io.Writer.  It's not a string; you can't take len(...) of an io.Writer, nor can you slice it.

[LetGo]

Thanks for your answer!(:

You are right, sorry!

This is the code: https://play.golang.org/p/zEZ2HIUNffs

About the lines, wow! Yes, you got them! ahah

About the errors, I tried to convert ( cmd.Stdout ) io.Write to bytes/ strings, but.. I have then entered into a loop of errors...

[Brian Candler]

It makes no sense to convert an io.Writer to a string.

io.Writer is an interface: any type which has a Write() method.  So you can pass a string *to* a writer, to get it written somewhere, by calling the Write() method.  In general, you can't get a string *from* a writer.  If you google "go io.Writer" you'll get lots of tutorials and examples.

Depending on your application though, you might want to create a bytes.Buffer or strings.Builder object, both of which are an io.Writer.  The written data gets appended to a buffer that you can read later.

[LetGo]

I tried also both of them, but I got stuck into a loop of errors again.. probably I coded in the wrong way

[LetGo]

One of these is this:

...

           buf := new(bytes.Buffer)     

        foo := buf(cmd.Stdout) // this line is 87

        data := foo

        var i int

...

pkg/conn.go:87:20: cannot call non-function buf (type *bytes.Buffer)

[Levieux Michel]

[Sorry forgot to hit "Reply all"]

Are you trying to cast cmd.Stdout here?

What you can do is:

buf := new(bytes.Buffer)

cmd.Stdout = buf // buf is an io.Writer so this is fine

but I don't get the point of the data := foo?

Maybe, before trying to convert a whole complex program in Python to a whole working program in Go, you should take time to familiarize yourself with the language.

Go through the Go tour, read a little of the specs, have yourself code some small, simple programs that don't require using lots of std packages at once...

Once you are familiar with the language constructs, which I have to say are pretty different from Python's, you will have a better understanding of where to start and how to implement your program. Otherwise I think this will all only get you confused.

And understanding at least the important basics of Go will help you explain your pain points here, if any remains :)

Hope this helps,

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/466d016b-b90b-4505-b4a8-7e5fc62679b4n%40googlegroups.com.

[LetGo]

Thanks for your answer!(:

You are right, but I just wanted to have this one little tool in Go  and I have never thought that could be that hard... ahahah

By the way, it works as you said, it fixed the error! ( obviously.. ) the only thing left is to convert type *bytes.Buffer to []byte * I think* and then I will be almost done.

Im already searching how to do that.

Once it will work as I wish, I will add your names to my comments ( I think this is better than any "thank you" ) in the code, to remind me of your kind help(:

[Levieux Michel]

You cannot per se convert an interface or a struct to a builtin like []byte

You might wanna have a look at the Bytes method of *bytes.Buffer, which returns the internal buffer of the type as a slice of bytes. Normally that would have been a good exercise to let you find it yourself, and I don't know if it is really "help" to give it to you directly, but as I said, once you are done with your small tool, the next step for you will be to go back from the basic Go constructs :)

Glad I could help, and don't bother with the comments, the best "thank you" I can wish for is that we continue learning together ;)

Hope the following adventures of your Go journey are as interesting as they are for me!

Cheers

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0f12076e-f325-4583-ba85-857084aec83dn%40googlegroups.com.

[LetGo]

I thought that after getting the bytes, chunk the data, I need to have the io.writer back before send the chunked data

So far I came up with this:

...

cmd.Stdin = conn

                buf := new(bytes.Buffer)

                cmd.Stdout = buf

                bizz := buf.Bytes()

        var i int

    for {

    n := int(math.Min(float64(rand.Intn(len(bizz))), float64(len(bizz))))

    d := bizz[i : i+n]

    i += n

                    z := bytes.NewBuffer(d)

      

                    kk := io.Writer(z)                                         //this line creates the error

      time.Sleep(400 * time.Millisecond)

                    kk = conn       

        if i >= len(bizz) {

    break

        }

            }

cmd.Stderr = conn

cmd.Run()

.....

But know I get this error:

conn.go:95:21: kk declared but not used

[Kurtis Rader]

The error is self explanatory. You define variable "kk" but never use it. If you don't care about the return value of io.Writer() simply omit the assignment; just call the function. Or explicitly assign to the special "_" placeholder var to show that you know the function returns a value and you are deliberately ignoring it. Beyond that the statement doesn't make any sense since io.Writer is an interface type. Not a function you call. Also, bytes.NewBuffer() returns an object which implements the io.Writer interface. You don't need to do anything else to turn it intoan  io.Writer.

On Thu, Jul 8, 2021 at 10:54 AM LetGo <non3...@gmail.com> wrote:

I thought that after getting the bytes, chunk the data, I need to have the io.writer back before send the chunked data

So far I came up with this:

...

cmd.Stdin = conn

                buf := new(bytes.Buffer)

                cmd.Stdout = buf

                bizz := buf.Bytes()

        var i int

    for {

    n := int(math.Min(float64(rand.Intn(len(bizz))), float64(len(bizz))))

    d := bizz[i : i+n]

    i += n

                    z := bytes.NewBuffer(d)

      

                    kk := io.Writer(z)                                         //this line creates the error

      time.Sleep(400 * time.Millisecond)

                    kk = conn       

        if i >= len(bizz) {

    break

        }

            }

cmd.Stderr = conn

cmd.Run()

.....

But know I get this error:

conn.go:95:21: kk declared but not used

 

--

Kurtis Rader

Caretaker of the exceptional canines Junior and Hank

[Levieux Michel]

This error is pretty straightforward 😅

You declared a variable (i.e. kk) that is not used after its declaration. Note that assignment is not use. I have to admit I don't know what this variable is supposed to be used for but that does not matter, if you don't know, maybe just try dropping it and see?

This might get pretty "harsh" and I really, really don't wanna get disrespectful, but maybe going through the go tour and some experimenting before coming back to your program is, in my humble opinion, a good idea. If you are at ease with other languages already this should only take a few days. Please don't take that as an offense :)

Happy to help anyways,

Have a great night!

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/2aebe5af-a078-4165-bde3-e235845c3de1n%40googlegroups.com.

[Brian Candler]

Complete code on play.golang.org is much easier to understand. At the moment I don't think your code will do anything useful.  For example

               buf := new(bytes.Buffer)

               ...

               bizz := buf.Bytes()

will always set bizz to an empty slice; since you never modify bizz, len(bizz) will always be zero.  Also,

                kk := io.Writer(z)

creates an io.Writer variable with a particular value, but never uses that value, because 2 lines later you replace it with a different value.  I can't say for sure without seeing the whole program, but I don't think that's where the error is occurring; I think it's the later line ("kk = conn"), because you don't seem to use that value.  It's as if you wrote a program like this:

func main() {

    i := 1

    i = 2

}

i.e. you just return and never use the value of "i".

I think the previous advice given was good.  Do the go tour, several times if required; look at the tutorials at https://golang.org/doc/, look for interesting articles at https://blog.golang.org/index; after you pick up the core concepts, find some example sockets-based code, run it, modify it.  Stringing together statements, if you don't understand what each individual statement does, is unlikely to give a useful result.