What is the possible reason for error messages like this?

[Wei Zhang]

I've got such repeating error messages when I was trying to use a single fd to perform both read and write tasks:

03-23-2013 15:29:14.150 INFO - (Writer.cc:1195) PW 1,46,14,dat.0 scheduling retry: 9 of 30 in 5 sec. op: allocate: fid: 46 offset: 201326592

03-23-2013 15:29:19.156 ERROR - (Writer.cc:1132) PW 1,46,14,dat.0 operation failure, seq: 1137091523519419392 status: -16 msg: valid read lease op: allocate: fid: 46 offset: 201326592 current chunk server:  -1 chunkserver: no data sent

Request:

ALLOCATE

Cseq: 1137091523519419392

Version: KFS/1.0

Client-Protocol-Version: 114

UserId: 505

GroupId: 505

User: bgtest

Client-host: dhcp-46-222.cs.ucsb.edu

Pathname: /root/VM-303B059F/append/data/dat.0

File-handle: 46

Chunk-offset: 201326592

Is there any guess on what could be the reason? Your help is greatly appreciated.

Wei

[Jeremy Fishman]

> --
> You received this message because you are subscribed to the Google
> Groups "QFS Development" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to qfs-devel+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

The error message is logged from the write operation as it is fails to
allocate space in the chunk due to an active read lease (likely held by
the same client).

I was previously under the impression that O_RDWR was not supported, but
the KfsClient.h claims it is. I will follow up on this to see if there
is an error here whereby old leases are not relinquished when executing
new operations on the same client.

Do you have a test case that can reproduce the error? I have managed to
cause read failures while holding the write lease but not vice versa.

- Jeremy

--
Jeremy Fishman | Senior Software Engineer
Quantcast Corporation | Big Data Platform
201 3rd Street FL 2, San Francisco, CA 94103

[Wei Zhang]

Hi Jeremy,

My application has one writer and one reader, they open the same file simultaneously, using O_WRONLY and O_RDONLY respectively.

The writer is keep writing to the end of the file, while the reader is doing random access to previous written data.

So I guess the error indicates that the reader get the lease of the chunk which the writer is trying to write, is that correct?

I couldn't provide a simple test case to repeat this error for now, because my application is quite complicated, but I will try to get a simpler one.

What is the proper way for my application to avoid such errors? Any suggestions?

Thanks

Wei

[Wei Zhang]

// test concurrent read/write to a single QFS file, this program reproduce the error messages in my previous email.

#include <iostream>

#include <string>

#include <fcntl.h>

#include "KfsClient.h"

#include "KfsAttr.h"

using namespace std;

#define DATA_BUF_SIZE (1024 * 1024)

int main(int argc, char** argv)

{

    // init

    KFS::KfsClient *pfs;

    pfs = KFS::Connect("127.0.0.1", 20000);

    if (!pfs) {

        cout << "unable to connect to QFS" << endl;

        return -1;

    }

    char* data = new char[DATA_BUF_SIZE];

    char* rdbuf = new char[DATA_BUF_SIZE];

    int rdfd = -1, wrfd = -1, res = 0;

    // 1. create a file and fill in 100MB data

    string write_file("/qfstest.write");

    cout << "QFS chunk size is " << pfs->GetChunkSize(write_file.c_str()) << endl;

    wrfd = pfs->Open(write_file.c_str(), O_WRONLY | O_CREAT | O_TRUNC);

    if (wrfd < 0) {

        cout << "writter open failed: " << KFS::ErrorCodeToStr(wrfd) << endl;

        return -1;

    }

    for (int i = 0; i < 100; i++) {

        res = pfs->Write(wrfd, data, DATA_BUF_SIZE);

        if (res < 0) {

            cout << "write failed: " << KFS::ErrorCodeToStr(res) << endl;

            return -1;

        }

        cout << "write number bytes: " << res << ", current position is: " << pfs->Tell(wrfd) << endl;

    }

    pfs->Close(wrfd);

    // 2. open file and read 100MB data

    rdfd = pfs->Open(write_file.c_str(), O_RDONLY);

    if (rdfd < 0) {

        cout << "reader open failed: " << KFS::ErrorCodeToStr(wrfd) << endl;

        return -1;

    }

    for (int i = 0; i < 100; ++i)

    {

        res = pfs->Read(rdfd, rdbuf, DATA_BUF_SIZE);

        if (res < 0) {

            cout << "read failed: " << KFS::ErrorCodeToStr(res) << endl;

            return -1;

        }

        cout << "read number bytes: " << res << ", current position is: " << pfs->Tell(rdfd) << endl;

    }

    // 3. continue write, but rdfd is still open

    wrfd = pfs->Open(write_file.c_str(), O_WRONLY);

    if (wrfd < 0) {

        cout << "writter open failed: " << KFS::ErrorCodeToStr(wrfd) << endl;

        return -1;

    }

    // seek to the end

    pfs->Seek(wrfd, 0, SEEK_END);

    cout << "writter seek to " << pfs->Tell(wrfd) << endl;

    for (int i = 0; i < 10; i++) {

        res = pfs->Write(wrfd, data, DATA_BUF_SIZE);

        if (res < 0) {

            cout << "write failed: " << KFS::ErrorCodeToStr(res) << endl;

            return -1;

        }

        cout << "write number bytes: " << res << ", current position is: " << pfs->Tell(wrfd) << endl;

    }

    pfs->Close(wrfd);

    pfs->Close(rdfd);

    delete[] rdbuf;

    delete[] data;

    return 0;

}

[Michael Ovsiannikov]

This is expected behavior. Concurrent read and write to the same *chunk* isn't supported. Moving pfs->Close(rdfd); before // 3. should make it work.