Use kodo to send/receive coded packets via socket

[Lawrence Ho]

Hello everyone, very sorry to have bothered you, recently I would like to use KODO library to practice network coding implementation, but I don't familiar with C/C++ (I more familiar with JAVA), so I just write simple code to send/receive coded packets via socket, here is my code:

Server.cpp (Sender)

#include <sys/socket.h>

#include <netinet/in.h>

#include <arpa/inet.h>

#include <stdio.h>

#include <stdlib.h>

#include <unistd.h>

#include <errno.h>

#include <string.h>

#include <sys/types.h>

#include <kodo/file_encoder.hpp>

#include <kodo/object_decoder.hpp>

#include <kodo/rlnc/full_rlnc_codes.hpp>

#include <kodo/set_systematic_off.hpp>

/// Often we want to encode / decode data that exceed a single

/// encoding/decoding block. In this case we need to "chop" up

/// the data into manageable chunks and then encode and decode

/// each chuck separately. This examples shows how to use the

/// file encoder in Kodo. The file encoder operates directly on

/// a file in the file-system. For decoding we use an object decoder

/// which decodes data to memory, but which is compatible with

/// file encoder.

int main(){

    //Initial Server Socket

    int listenfd = 0,connfd = 0;

  

    struct sockaddr_in serv_addr;

 

    listenfd = socket(AF_INET, SOCK_STREAM, 0);

    printf("[socket retrieve success]\n");

    

    memset(&serv_addr, '0', sizeof(serv_addr));

    serv_addr.sin_family = AF_INET;    

    serv_addr.sin_addr.s_addr = htonl(INADDR_ANY); 

    serv_addr.sin_port = htons(5000);    

 

    bind(listenfd, (struct sockaddr*)&serv_addr,sizeof(serv_addr));

  

    if(listen(listenfd, 10) == -1){

        printf("Failed to listen\n");

        return -1;

    }

    // Set the number of symbols (i.e. the generation size in RLNC terminology) and the size of a symbol in bytes

    uint32_t max_symbols = 42;

    uint32_t max_symbol_size = 64;

    std::string encode_filename = "encodeFile.bin";

    // Create a test file for encoding.

    std::ofstream encode_file;

    encode_file.open (encode_filename, std::ios::binary);

    uint32_t file_size = 500000;

    std::vector<char> encode_data(file_size);

    std::vector<char> decode_data;

    // Just write some bytes to the file

    for(uint32_t i = 0; i < file_size; ++i){

        encode_data[i] = rand() % 255;

    }

    encode_file.write(&encode_data[0], file_size);

    encode_file.close();

    // Select the encoding and decoding algorithms

    typedef kodo::full_rlnc_encoder<fifi::binary> encoder_t;

    typedef kodo::full_rlnc_decoder<fifi::binary> decoder_t;

    // Now for the encoder we use a file_encoder with the chosen encoding algorithm

    typedef kodo::file_encoder<encoder_t> file_encoder_t;

    // For decoding we use an object_decoder with the chosen decoding algorithm

    typedef kodo::object_decoder<decoder_t> object_decoder_t;

    // Create the encoder factory - builds the individual encoders used

    file_encoder_t::factory encoder_factory(max_symbols, max_symbol_size);

    // Create the actual file encoder using the encoder factory and the filename of the file to be encoded

    file_encoder_t file_encoder(encoder_factory, encode_filename);

    // Create the decoder factory - build the individual decoders used

    object_decoder_t::factory decoder_factory(max_symbols, max_symbol_size);

    // Create the object decoder using the decoder factory and the size of the file to be decoded

    object_decoder_t object_decoder(decoder_factory, file_size);

    // Now in the following loop we go through all the encoders

    // needed to encode the entire file. We the build the corresponding

    // decoder and decode the chunk immediately. In practice where

    // encoders and decoders are on different devices e.g. connected

    // over a network, we would have to pass also the encoder and decoder

    // index between the source and sink to allow the correct data would

    // passed from encoder to corresponding decoder.

 

    //Send encode data via socket

    while(1){

        connfd = accept(listenfd, (struct sockaddr*)NULL ,NULL); // accept awaiting request

printf("[Start encode file]......\n");

   for(uint32_t i = 0; i < file_encoder.encoders(); ++i){

       auto encoder = file_encoder.build(i);

       auto decoder = object_decoder.build(i);

       // Set the encoder non-systematic

       if(kodo::has_systematic_encoder<encoder_t>::value)

           kodo::set_systematic_off(encoder);

       std::vector<uint8_t> payload(encoder->payload_size());

       while( !decoder->is_complete() ){

           // Encode a packet into the payload buffer

           encoder->encode( &payload[0] );

           // In practice send the payload over a network, save it to

           // a file etc. Then when needed build and pass it to the decoder

send(connfd,&payload[0],payload.size(),0);

           // Pass that packet to the decoder

           decoder->decode( &payload[0] );

}

   }

printf("[File encode already]\n");

        close(connfd);    

        sleep(1);

}

 

    return 0;

}

Client.cpp (Receiver)

#include <sys/socket.h>

#include <netinet/in.h>

#include <arpa/inet.h>

#include <stdio.h>

#include <stdlib.h>

#include <unistd.h>

#include <errno.h>

#include <string.h>

#include <sys/types.h>

#include <kodo/file_encoder.hpp>

#include <kodo/object_decoder.hpp>

#include <kodo/rlnc/full_rlnc_codes.hpp>

#include <kodo/set_systematic_off.hpp>

int main(){

//Initial Client Socket

    int sockfd = 0,n = 0;

    struct sockaddr_in serv_addr;

 

    if((sockfd = socket(AF_INET, SOCK_STREAM, 0))< 0){

        printf("\n Error : Could not create socket \n");

        return 1;

    }

 

    memset(&serv_addr, '0', sizeof(serv_addr)); 

    serv_addr.sin_family = AF_INET;

    serv_addr.sin_port = htons(5000);

    serv_addr.sin_addr.s_addr = inet_addr("127.0.0.1");

 

    if(connect(sockfd, (struct sockaddr *)&serv_addr, sizeof(serv_addr))<0){

        printf("\n Error : Connect Failed \n");

        return 1;

    }

    // Set the number of symbols (i.e. the generation size in RLNC terminology) and the size of a symbol in bytes

    uint32_t max_symbols = 42;

    uint32_t max_symbol_size = 64;

    uint32_t file_size = 500000;

std::vector<char> decode_data;

    typedef kodo::full_rlnc_decoder<fifi::binary> decoder_t;

    // For decoding we use an object_decoder with the chosen decoding algorithm

    typedef kodo::object_decoder<decoder_t> object_decoder_t;

    // Create the decoder factory - build the individual decoders used

    object_decoder_t::factory decoder_factory(max_symbols, max_symbol_size);

    // Create the object decoder using the decoder factory and the size of the file to be decoded

    object_decoder_t object_decoder(decoder_factory, file_size);

 

 

// Receive data and encode

std::vector<uint8_t> payload;

payload.resize(100);

n = recv(sockfd,&payload[0],payload.size(),0);

printf("[Receive encode file]......\n");

    while(n > 0){

auto decoder = object_decoder.build(n);//what's the parameter in build( )????

payload.resize(n);

recv(sockfd,&payload[0],payload.size(),0);

        // Pass that packet to the decoder

        decoder->decode( &payload[0] );

        std::vector<uint8_t> data_out(decoder->block_size());

        decoder->copy_symbols(sak::storage(data_out));

        data_out.resize(decoder->bytes_used());

        decode_data.insert(decode_data.end(),

                           data_out.begin(),

                           data_out.end());

}

    printf("[File decode already]\n");

    if( n < 0){

        printf("\n Read Error \n");

}

 

    return 0;

}

After executing Server.cpp/Client.cpp, I have some problems, "Client(Receiver)" cannot decode coded packet after receiving, here is message in server/client console:

Server:

[socket retrieve success]

[Start encode file]......

[File encode already]

Client:

[Receive encode file]......

Assertion failed: (symbol_index < SuperCoder::symbols()), function decode_symbol, file /Users/macbook/Documents/Network_Coding/kodo/src/kodo/bidirectional_linear_block_decoder.hpp, line 118.

Abort trap: 6

I search google and try to revise my code, but it doesn't work, would you mind give me some hints to let me finish my practice? 

I'm really so sorry to trouble you,  I would greatly appreciate it if you kindly give me some help.

[Morten V. Pedersen]

Hi Lawrence,
No problem we are here to help.

I took a look at your source code and I think you are close to having a working example.

I think the main bump you are facing is that you need to wrap the communication in a small protocol. When you encode a file that is larger than what fits into a single generation (a generation is the block of bytes coded by a single encoder). We need to make several generations.

When making several generations we need to make sure that data encoded from a specific generation goes into the corresponding decoder.

In this example you use the kodo::file_encoder and kodo::object_decoder.

Calling kodo::file_encoder::encoders() and kodo::object_decoder::decoders() will tell you how many generations will be created to encode a file with a certain size.

Using your for loop:

for(uint32_t i = 0; i < file_encoder.encoders(); ++i){

...

You iterate over all the encoders (i.e. the generations) created to encode the chosen file. Now, the data encoded by encoder i needs to be passed to decoder i on the receiver side.

So you probably need to perpend that information to the encoded payload buffer.

Also right now you are encoded until a local decoder (the decoder you create on the sender side is complete). This means that you would not take any packet loss into account. It would probably be better to encode some fixed amount of extra redundancy or similar at the sender side. You can ask the encoder, how many symbols it is supposed to encode by calling:

uint32_t symbols = encoder->symbols();

If you want to generate some extra redundancy i.e. one extra packet per generation you could change the while loop:

while( !decoder->is_complete() ){

to something like

uint32_t j = 0;
while( j < symbols + 1 ){
++j
...

Then you would be able to fix one packet loss per generation. In a real system you would probably implement something more generic by this could be a starting point.

Now assuming you have the information about which encoder produced a specific coded symbol. You just need to extract that information from the received packet and then build that corresponding decoder.

So this line:
auto decoder = object_decoder.build(n)

Means build the decoder corresponding to generation n. You need to keep receiving packets for decoder n until:

decoder->is_complete()

return true. This means that you need to send more redundancy over the network that what will be lost. With an actual protocol you could also implement a mechanism to request more redundancy etc.

I hope this helps a little bit - otherwise feel free ask more questions.

- M

--
You received this message because you are subscribed to the Google Groups "steinwurf-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to steinwurf-de...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nestor Hernandez]

Hi Lawrence,

Also complementing to Morten's answer, the error that you are seeing as the terminal output is because you are trying to access a symbol for which its index is equal or greater than number of symbols to encode (i.e. 42). In this case, the indexes in a single decoder should be between 0 and 41. For a 500 KB file and a generation size of 42 and symbol size of 64 (excluding the coefficients), you 'll need 19 generations (rounded up). So, you you should have 19 encoders and 19 decoders where each of them decodes a single generation. The objects that Morten described you at the beginning take this into account.

From what I can see from Morten's reply it might be possible that you are giving packets from one generation to a decoder that is trying to decode packets from other generation.

Check out what you are getting from recv(), because the payload reference is a buffer to contain one coded symbol and its coefficients at a single time. It is very likely that this is not the generation number you need to decode according and that is why you are getting the error ; )

Please, feel free to ask if you need something else. Hope this helps even further.

Best regards,

NH

--

Néstor Hernández
Mobile: +4551200349

[Ahmad Alhilal]

Hi Nestor, Morten.

A follow-up question. I'm trying to send a video frame by frame, and generation per frame. 
Everything is ok when there is no packet loss. When I apply packet loss using traffic control (tc), the decoder reports 'completion' but the data_out is corrupted to build the frame.
Using logger for debugging, encoder and decoder are compatible as I encapsulate each Kodo produced packet within another packet containing kodo encoder packet, frame number, and sequence number. 

Why skipping some packets at the decoder causes data corruption?


Settings:

Kodo-Python
Connection: Ethernet connection, 2 PCs on LAN.
Packet loss: 1% - 20% using tc.
max-redundancy: 300

field: binary8
Result: failure.

To unsubscribe from this group and stop receiving emails from it, send an email to steinw...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "steinwurf-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to steinw...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.