format difference for visual foxpro 9

[The Budiman]

Hello,

I'm new in this group, but I hope you don't mind I'm immediately asking questions.

In my recent project I have to integrate my software to a Visual FoxPro 9 application, so I just exploring the use of this library.

Although in the documentation it says that this library only support Visual FoxPro 6, I'm wondering what is the format difference between

6 and 9, and what is the potential problem if I use this library anyway.

And the second question is I come across this exception in one of the tables:

array('c', '\x07')

8

1 0

686 array('c', '                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             \x07')

('0', 685, 1, 686, 0, 5, <function none at 0xf69f50>, <function none at 0xf69f50>)

array('c', '\x07')

Traceback (most recent call last):

  File "reader.py", line 10, in <module>

    print record

  File "/home/mobile04/src/db/dbf.py", line 1796, in __str__

    result.append("%3d - %-10s: %r" % (seq, field, self[field]))

  File "/home/mobile04/src/db/dbf.py", line 1732, in __getitem__

    return self.__getattr__(item)

  File "/home/mobile04/src/db/dbf.py", line 1701, in __getattr__

    value = self._retrieve_field_value(index, name)

  File "/home/mobile04/src/db/dbf.py", line 1871, in _retrieve_field_value

    if ord(null_data[byte]) >> bit & 1:

IndexError: array index out of range

I'm using the latest version of the library (0.94.005), it happens when I'm trying to read a table using VfpTable and print the record.

I couldn't yet figured out what "if ord(null_data[byte]) >> bit & 1" does, so I need some clue here understand what's happening.

The code works for other tables in the same software though, so may be there's something specific in this table.

[Ethan Furman]

The Budiman wrote:
> Hello,
>
> I'm new in this group, but I hope you don't mind I'm immediately asking
> questions.

Not at all, and welcome!

> In my recent project I have to integrate my software to a Visual FoxPro
> 9 application, so I just exploring the use of this library.
> Although in the documentation it says that this library only support
> Visual FoxPro 6, I'm wondering what is the format difference between
> 6 and 9, and what is the potential problem if I use this library anyway.

9 has auto-increment fields and varchar fields -- if your tables don't use those features you should
be fine.

It's checking to see if nulls are allowed in that field, and if that record has a null in that field.

> The code works for other tables in the same software though, so may be
> there's something specific in this table.

Can you send me a copy of the table? Just a few records is fine as long as the problem record is
included (the whole thing is also fine, whatever is easy for you). Don't forget the .fpt file if
there are memos.

~Ethan~

[T. Budiman]

Thanks for the reply.

I'm sending you the problematic table.

I should mention also that I earlier I had another problem reading a datetime field from this table.

It's the one field that says:

6) tglupdate T binary

from the info function output

Since this particular field is not important to me, I'm currently kind of ignoring it by adding:

2661a2662

>     hours = hours % 24

2667a2669

>     possible = min(possible, datetime.date.max.toordinal())

to the library, pending further investigation. 

Another software I use to view the data couldn't display the date either, so maybe its just bad data.

T. Budiman

Bandung ID

[Ethan Furman]

I'm not sure what's going on here, but "T binary" is not right (at least, not for earlier versions
of vfp tables). That might be what's messing up the rest.

I haven't given up yet, though.

~Ethan~

[T. Budiman]

I'm not sure what's going on here, but "T binary" is not right (at least, not for earlier versions of vfp tables).  That might be what's messing up the rest.

I haven't given up yet, though.

~Ethan~

I don't think it's related to the "T binary" problem. I've found another table without "T binary" that has the same problem with the nullable field. I will have more time tomorrow to investigate this thing.

[The Budiman]

Hello,

Do you think it's right if we change in line 1871:

if ord(null_data[byte]) >> bit & 1:

to

if ord(null_data[0]) >> bit & 1:

Lets see the context:

        nullable = flags & NULLABLE

        binary = flags & BINARY

        if nullable:

            byte, bit = divmod(index, 8)

            null_def = self._meta['_nullflags']

            null_data = self._data[null_def[START]:null_def[END]]

            try:

                if ord(null_data[0]) >> bit & 1:

                    return Null

            except IndexError:

       print null_data

                print index

                print byte, bit

                print len(self._data), self._data

                print null_def

                print null_data

                raise

As far as I can tell, the enclosing function (_retrieve_field_value) will return a field value. This part of code are checking if the field is nullable and if the field  value is in fact null. I'm guessing that null_def[START] and null_def[END] should contain the location of the null data or flag. In my cases, the length of the resulting null_data is always 1, so null_data[byte] usually throws the array out of range exception. I'm not sure what the 'byte' and 'bit' variable supposed to mean, but maybe in this case we just need to read that 1 character?

T. Budiman

[Ethan Furman]

Each null byte stores the null state for up to eight fields; so if there are more than eight
nullable fields then null_data can be more than one byte long. bit is which bit in the byte
represents the field in question.

So to make a long story short, no, changing to null_data[0] will not work once there are more than
eight nullable fields.

One possible problem, though, is that (I think) the varchar fields also use the null_data area to
keep track of varchar info, and I don't know how that interaction works.

~Ethan~

[The Budiman]

Each null byte stores the null state for up to eight fields; so if there are more than eight
nullable fields then null_data can be more than one byte long.  bit is which bit in the byte
represents the field in question.

I see, now I get what 'ord(null_data[byte]) >> bit & 1' means. 

So in this case do you think that somehow we get wrong values in self._meta['_nullflags'] suggesting

that we only have 1 byte null_data while in fact we should have more than 1 bytes?

[The Budiman]

On Thursday, November 1, 2012 10:36:08 AM UTC+7, The Budiman wrote:

Each null byte stores the null state for up to eight fields; so if there are more than eight
nullable fields then null_data can be more than one byte long.  bit is which bit in the byte
represents the field in question.

I see, now I get what 'ord(null_data[byte]) >> bit & 1' means. 

So in this case do you think that somehow we get wrong values in self._meta['_nullflags'] suggesting

that we only have 1 byte null_data while in fact we should have more than 1 bytes?

A table with 10 fields (with 3 of them nullable) in it should have 2 bytes of null_data, right? Instead I get only 1 byte.

In the second table with 26 fields (with 13 of them nullable) should have 4 bytes of null_data, instead I get only 2 bytes.

Do you think it's possible that in this version they store only the nullable fields in this null data so it needs less bytes than before?

T. Budiman

 

[The Budiman]

A table with 10 fields (with 3 of them nullable) in it should have 2 bytes of null_data, right? Instead I get only 1 byte.

In the second table with 26 fields (with 13 of them nullable) should have 4 bytes of null_data, instead I get only 2 bytes.

Do you think it's possible that in this version they store only the nullable fields in this null data so it needs less bytes than before?

T. Budiman

 

I just found http://msdn.microsoft.com/en-US/library/st4a0s68(v=vs.80).aspx

My table's file descriptor is 0x30, so there won't be any autoincrement or varchar fields.

The page contains a note there that says:

"For each Varchar and Varbinary field, one bit, or "varlength" bit, is allocated in the last system field, which is a hidden field and stores the null status for all fields that can be null. If the Varchar or Varbinary field can be null, the null bit follows the "varlength" bit. If the "varlength" bit is set to 1, the length of the actual field value length is stored in the last byte of the field. Otherwise, if the bit is set to 0, length of the value is equal to the field size."

The null status is mentioned there, although its not clear if it meant it stores only the nullable fields or all fields.

But I think, its possible, right?

If that is so, we need to adjust the index value so it only count the nullable fields. Do you already have such function?

T. Budiman