error dbfntx1012 corruption detected

[Michael Green]

Has anyone encountered this error:

Error DFTNTX 1012 corruption detected <index name>

If so what was the cause? I'm running on FreeBSD 13.2 if that matters.

Sorry about the weak  description. Anyway thanks for looking.

[Daniele Campagna]

Under Linux, check file permissions for indexes,memo files.

Or check the RDD you are using (ntx? cdx?). Check the casing also. Undr Linux cdx is not the same as CDX (unLess you set SET_FILECASE).

HTH

Dan

--
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
Unsubscribe: harbour-user...@googlegroups.com
Web: https://groups.google.com/group/harbour-users
---
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to harbour-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/harbour-users/c3368df9-a468-49cb-8f96-9fbbc1145af3n%40googlegroups.com.

[Michael Green]

Hmmm. I thought it might be an issue with the number of files opened, so I checked; each instance (up to a maximum of 5) opens a total of 47 tables/indexes. I'm using FreeBSD and accessing the program over ssh, so no issues with Samba. 

[Michael Green]

I reduced the number of files opened and the problem seems cured. Odd. Has anyone seen anything similar. How would I research whatever is being used up?

[cod...@outlook.com]

My experience on Windows computers:

Error DFTNTX 1012 corruption detected <index name> is very rare but rough error.

I can not remember when I have it last time, but it was before almost 30 years. As I can remember this type of error is allways related to quality of computer hardware and software (power supply stability, operating system, quality of hard disk, quality of memory, etc.). In my case the cause of that error was never within PRGs.

 

I have aplication in which I can have up to 30 DBF files opened at same time, plus its .NTX index files. I can not remember if I ever had 1012 error.  I do not work with memo files.

I suggest you to have some backdoor function to reindex selected DBF to quickly solve the problem.  Then you can analyze situation on similar computer configurations.

Regards,

Simo.

[Michael Green]

thanks for the insight

[hmpaquito]

Hola Michael,

Yo le sugiero dos cosas:
- Primero, que haga operaciones por lotes y la ultima sea grabar
- Segunda, antes de empezar los procesos compruebe que los indices estan correctos: Yo tengo por ahí una funcion que lo que hace es añadir un registro en blanco al final de la .dbf y luego lo borra; Hace algunas cosas más para comprobar que los indices estan bien. . Lo que consigo con esto es que si el programa rompe con un run-time error lo haga antes del proceso. Esto me ha ahorrado mucho tiempo porque me ha evitado que los procesos quedaran a medio

Espero que le sirva.

[Michael Green]

I'm currently writing a test program to open a large number of files to see if I can replicate the error. Details soon.

[Michael Green]

Well, no luck in replicating the message when opening 100 tables/indexes. A mystery...

[Nenad Batoćanin]

Hello!

 

We have experience with a large number of computers (about 3000) with very demanding processing with a large number of files and indexes open. Corrupted indexes were always related to some hardware problem. Final proof: we transferred about half of the users to our servers with remote access (programs run directly on the server, there is no data transfer over the network). In 10 years we have not had a single case of corrupted indexes.

 

Regards, NB

--

You received this message because you are subscribed to the Google Groups "Harbour Users" group.
Unsubscribe: harbour-user...@googlegroups.com
Web: https://groups.google.com/group/harbour-users
---
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to harbour-user...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/harbour-users/6468d34d-996f-4dd8-963d-02cac9ec9757n%40googlegroups.com.

[Michael Green]

Thank you for the input. I'm not getting the error now. However all information is useful. Tar </northern English vernacular>

[danr...@gmail.com]

Yo después de lidiar años con los índices corruptos con los NTX nativos, me pasé a cliente servidor cambiado solo el RDD (Advantage Database Server) y el problema se solucionó con muy pocos cambios en el código

Saludos

Daniel Goldberg

La Reja

Bs As

--

You received this message because you are subscribed to the Google Groups "Harbour Users" group.
Unsubscribe: harbour-user...@googlegroups.com
Web: https://groups.google.com/group/harbour-users
---
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to harbour-user...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/harbour-users/23d673e7-a2c5-41ff-8f34-dc960660d74dn%40googlegroups.com.

[Michael Green]

As you were, I'm still getting the error. It's not a too many files open' issue, it's something else. Has anyone seen the error when the index wan't corrupted? I know the index is ok because a previous version works ok without any reindexing. All ideas welcome.

[Mario H. Sabado]

Hi Michael,

What's the create mask setting in your samba configuration?  Please test also the latency of your connection from workstation to server.  It should be consistent <1ms if your application is being accessed directly from the samba shared folder.  Fluctuating latency usually results in frequent index corruption in my experience.  Otherwise, you may try LetoDB/LetoDBf server to access your database and index files via TCP/IP that is more tolerant to higher network latency.

Best regards,

Mario

To view this discussion visit https://groups.google.com/d/msgid/harbour-users/6bffdefe-b00e-429d-83d3-f89ab7f7e54fn%40googlegroups.com.

[José M. C. Quintas]

1) Allways open files with indexes - all - on same order

2) After write/update, use SKIP 0, UNLOCK, Don't change this order ( or commit, unlock)

3) Old CLIPPER prolem, never try to do the same on harbour, to exists a memo field can cause index corruption message

4) Old CLIPPER problem: On W95 OSR2, W98 OSR2, W2000, wrong network driver cause problem, same with old Novell 3.11

5) Old CLIPPER problem: PACK can cause duplicate records

6) Old programmer error:  DO WHILE ! RLock(); ENDDO. Insert a wait time. DO WHILE ! RLock();Inkey(0.3);ENDDO (On CLIPPER, OL_Yield() inside DO WHILE)

For the option 2, I create RecLock() and RecUnlock(), to do allways the same, on same order.

FUNCTION RecLock()

   DO WHILE ! RLock()

        Inkey(0.3)

  ENDDO

   RETURN Nil

FUNCTION RecUnlock()

   SKIP 0

   UNLOCK

   RETURN Nil

This can be used for another alias:   alias->( RecLock() ),  alias->( RecUnlock() )

I think the best is to use CDX.

Note: 

On CLIPPER days, I was using SIXCDX on Clipper 5.2 and ADS local on Visual Basic with no problem with simultaneous access.

On that time, limit for ADS Local was 20 users, local or terminal service.

On a current client, reindex can be made one time at year, or less.

José M. C. Quintas

To view this discussion visit https://groups.google.com/d/msgid/harbour-users/6bffdefe-b00e-429d-83d3-f89ab7f7e54fn%40googlegroups.com.

[Michael Green]

So no Samba for me. I use ssh in terminals to access the application running on a FreeBSD server. 

[Michael Green]

I don't thinks there's any actual index corruption as an earlier version runs fine.

[Francesco Perillo]

Hi Michael

just to be sure I understand correctly:

you have a freebsd server that accepts incoming ssh connections. The ssh daemon spawns a harbour program that access files (dbf and ntx) as local files.

So your users are working and this error appears:

- to all users ?

- just opening the file ?

- doing a seek ?

What do you mean when you say "an earlier version runs fine"?

I sometimes get a "silent" index corruption. By silent I mean that I search a record and can't find it. After creating new indexes the record appears. Index is corrupted but all the internals are ok. Lately we have some problems with the docking stations losing connectivity writing records.

Never got something serious as your message in the last 30 years (clipper 87, clipper 5 and various harbour versions).

If I had to investigate I'd move in several directions:

- file system cache, mount options

- hard disk cache, is it an array, does it have a battery cache, can it be set to write-through (slower, i Know)

- search for zombie processes, halted, segfault, core dumps

- search for OOM killer

I'd write a producer/consumer software to test the full stack and in particular the RLOCK/FLOCK adherence under stress.

I'd look for programs running by cron that may be written with different harbour versions or different style of LOCK

I'd also look for backup software that may open the files in a different way.

Are the dbf/ntx reachable in some other way (samba, nfs,...)

I'm really curious since I'd like to port my program to linux to hide all the files from samba. I want to create a setup like your, but I'm afraid of different locking scheme so that I think to migrate to netio but if I migrate to netio I can hide the filesw anyway...

Sorry for the long message. Please check, as already suggested, that you SKIP 0/COMMIT as soon as possible after writing, so that the data can be pushed to the lower layers. And a fsck may help sometimes.... :-)

[Michael Green]

Yes, a FreeBSD system accessed via ssh. It isn't a load situation as it can appear when in use by a single user. The problem is intermittent and appears during an unlock command in the same program. An earlier version of the same system doesn't have the problem, even without a reindex of all files. The file system is ZFS and routine testing with the scrub command doesn't show any errors. I've never seen the error before or in any other circumstance. The problem only started to show, intermittently, after I made a number of modifications to the source and recompiled. I made a few changes to the source to the problem area and included an altd(), just before the unlock, to review the situation. I haven't had the problem again yet.  

[Michael Green]

Problem re-occurred just now, this time on a commit command, which I put in just before the unlock command that triggered the problem before.

[Francesco Perillo]

Michael,

this problem smells as a cache/locking or hardware error.

Cache can be also in the OS or the file system. How is it mounted ?

I can't believe that changes in harbour code can produce this kind of low level errors. It simply cannot be.

The error comes from internal data structures that are wrong, so the RDD low level code stops since it can't understand the situation. This means that one process writes the index and another process has a different view, as per a locking mismatch.

DBF files are really simple structure, really basic.

NTX/CDX files are really different beasts, adding/modifying one record can produce tens of page writes. If the writer process and another reader process have different values for the same data page.... booom !

How was your previous system configured? Did you upgraded Harbour compiler ?

Francesco

--

You received this message because you are subscribed to the Google Groups "Harbour Users" group.
Unsubscribe: harbour-user...@googlegroups.com
Web: https://groups.google.com/group/harbour-users
---
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to harbour-user...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/harbour-users/c2ce527d-40f2-4bc8-869e-da1f383ea0d0n%40googlegroups.com.

[Michael Green]

> How is it mounted?

The system runs FreeBSD 14.2. I has a ZFS filesytem which checks clean using the scrub command. I access the system using ssh from Linux desktops.

I haven't changed or updated FreeBSD or Harbour since before the problem began. I have an earlier version of the system which doesn't have the intermittent problem, and does not complain if I complete the same tasks, that cause the problem, without a reindex. Here's a section of the code. The problem area occurred at the point after the atld(). I have commented out the 'commit' command and the 'unlock' command and moved the 'unlock into the two part of the if/else/endif section:

      if storev = 'Y'

         if treatv[prapptrtv,1] > 0
            go treatv[prapptrtv,1]
            store locker('R') to nul
            replace date with date(), code with treatv[prapptrtv,2],;
               comment with treatv[prapptrtv,3], kiroku with kirokuv
            unlock && <==unlock moved here
         else
            store locker('F') to nul
            append blank
            replace client with cnov, date with date(),;
               code with treatv[prapptrtv,2],;
               comment with treatv[prapptrtv,3], kiroku with kirokuv
            unlock && <==unlock moved here
         endif
         altd()
&&         commit
&&         inkey(1)
&&         unlock

The function locker() is in a procedure file:

proc locker

parameters L_TYPEV

if len(trim(dbf())) = 0
   set color to i
   @ 0,0 say "No file open in area, press a key..."
   wait ""
   set color to
&&   return to master
else

   if L_TYPEV$"Rr"
      do while .not. rlock()
         @ 0,0 say "Attempting record lock..."
         store inkey(1) to NUL
         @ 0,0 say "                          "
         store inkey(1) to NUL
      enddo
   endif

   if L_TYPEV$"Ff"
      do while .not. flock()
         @ 0,0 say "Attempting file lock..."
         store inkey(1) to NUL
         @ 0,0 say "                       "
         store inkey(1) to NUL
      enddo
   endif
endif && (file not open)

return .t.

[hmpaquito]

Hola,

¿ Ha pensado vd. en comparar el codigo fuente de harbour de su version que si funciona con el codigo fuente de harbour que hace crash ?

Podria haber algo que está, no siendo causa del problema, pero si provocandolo

Salu2

[Michael Green]

Sí, revisé los cambios que hice en cuanto se produjo el error por primera vez, pero aún no logro identificar la causa del problema. Estoy seguro de que debe ser algo que hice, pero la nueva versión tiene muchas funciones adicionales. Por suerte, la versión anterior sigue funcionando.

[hmpaquito]

Hola Michael,

Me referia a los cambios producidos en Harbour

[Michael Green]

I've eliminated the index which seems at the center of the problem. The index expression is:

str(client,5,0)+dtos(date)

It seems innocuous enought. Any insights? Tar.

[Daniel Lopes Filho]

I had problems using fcreate() and managed to solve them using the TTxtFile() class, but unfortunately I encountered a very serious error when reading txt files. However, since I'm stuck on FiveWin 805 + gtwvw with xharbour 1.1.0 because of the GT change...

--

You received this message because you are subscribed to the Google Groups "Harbour Users" group.
Unsubscribe: harbour-user...@googlegroups.com
Web: https://groups.google.com/group/harbour-users
---
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to harbour-user...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/harbour-users/dc577b09-e9e4-4d38-a6ff-136508153061n%40googlegroups.com.

--

Lopes Informática

67-9-9202-9422 (tim)

67-9-9676-8637 (vivo/whatsapp)

[Itamar Lins]

Hi!
Do not use flock() to "append blank" command in files DBF.
The flock() function should not be used on a network; first, verify that the file is not in use by any process.
Avoid using flock() whenever possible.
The best way to copy, restructure, or index a DBF file is to open it in exclusive mode.

Function AddRec(cAlias)
LOCAL lRet := .T.
hb_default(@cAlias,Alias())  

(cAlias)->(DbAppend())
   If neterr()
      hwg_Msginfo('Attempting to Include in File, '+cAlias)
   Else
      exit
   EndIf

Return lRet

It is incorrect to use flock() to APPEND BLANK/DbAppend() to include a record.

Best regards,

Itamar M. Lins Jr.