Refactoring legacy code to OOP — class/object access 5–7× slower than array

[Atul]

Hello everyone,
I am refactoring a legacy Clipper/Harbour console application that processes DBF files and generates reports. The application uses multiple DBFs connected via primary fields. I am implementing hexagonal architecture to separate UI, core logic, and database adapter layers.

Environment:
  Harbour version : 3.0.0
  Compiler        : MinGW
  Platform        : Windows 11, 64-bit
  RDD             : DBFNTX

Previously the report processed ~4,000 records in 2 minutes using
direct DBF navigation. After refactoring to OOP class objects the
same processing now takes 14 minutes — a 7× slowdown.

The refactored code has two phases:
  Phase 1 — data collection: DBF SEEK/navigation, loads data into
             an array of class objects. All DB access is HERE only.
  Phase 2 — processing: pure in-memory loop over the object array.
             NO DBF access, NO SEEK, NO area switching.

The slowdown is entirely in Phase 2. My class has 140 DATA members.
The loop structure is Header → Detail → Sub-detail:

  FOR EACH oHdr IN aHeader       // ~1,000 records
      nn := AScan(aDetails, {|x| x:code+x:key = oHdr:code+oHdr:key })    // ~3,000 records total
      IF nn > 0
         oDtl := aDetails[nn]
         kk := AScan(aSubs, {|x| x:code+x:key = oDtl:code+oDtl:key })    // ~5,000 records total
         IF kk > 0
            oDtl := aSubs[kk]
            // read properties, compute, output
         ENDIF
      ENDIF
  NEXT

Any guidance from those who have done similar legacy refactoring
projects would be greatly appreciated.

Thank you,
Atul Parate

[Francesco Perillo]

Member access is slower, for sure.

Can you please post the array version?

Can you please add a member in your classes that holds code+key so that it shouldn't be calculated in the ascan?

Personally I tried to move to an OOP way of accessing dbf but I had the same problem as your, reports would be slower. One solution is to create a method to createvthe report that bypasses the oop to access data. Or reads the data in oop forms when it needs it, not everything in ram...

--
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
Unsubscribe: harbour-user...@googlegroups.com
Web: https://groups.google.com/group/harbour-users
---
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to harbour-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/harbour-users/947f4824-e06d-4d4e-aa9c-5ff689cb99f0n%40googlegroups.com.

[Francesco Perillo]

I was having a second look at your code and I still think that using objects is not the way to go.

 Also, you are not providing informations on the relationships of the records, one-to-one, one-to-many, many-to-many... the code seems to be a one-to-one

Without more informations it is difficult to give some clues, but I ask you, out of curiosity, to change the code in this way and report if there is a speed improvement

The new code explicitely uses local variables in the aScan codeblock; I don't know if Harbour optimizes oHdr:code+oHdr:key

LOCAL cKH, cKD

   FOR EACH oHdr IN aHeader       // ~1,000 records

      cKH := oHdr:code + oHdr:key

      nn := AScan(aDetails, {|x| x:code+x:key = cKH })    // ~3,000 records total

      IF nn > 0
         oDtl := aDetails[nn]

         cKD := oDtl:code + oDtl:key

         kk := AScan(aSubs, {|x| x:code+x:key = cKD })    // ~5,000 records total

[Nenad Batoćanin]

Hi!

I think AScan does a simple sequential search, so it can't be as fast as index search. Perhaps a replacement for AScan could be made that uses a memory table and a seek search?

 

Regards, NB

To view this discussion visit https://groups.google.com/d/msgid/harbour-users/CADPHLr9RmwyV7Gca1zGtUyudF0JCNuM%3DW04EvP2VvEX7Errwsw%40mail.gmail.com.

[Francesco Perillo]

Yes, of course. Perhaps HASH but I don't know how they are implemented internally.

But you are right, there should be a way for a better ram storage:

- one possibility is to create a new algorithm, in C of course, to create a sort of index, like a....

- NTX/CDX index... so it may be interesting to copy the 3 files in ram and create the indexes.

We lack important informations...

To view this discussion visit https://groups.google.com/d/msgid/harbour-users/008601dcb980%24ecd55a70%24c6800f50%24%40wings.rs.

[Nenad Batoćanin]

If anyone remembers the CA Visual Objects project... It had a very nice object-oriented approach to working with the database. It looks like this:

 

oDb := DBServer ("Test")

oDb:SetIndex ("Test_ID")

WHILE ! oDb:Eof()

   IF oDb:Id == 0

      oDb:Id := RecNo()

      oDb:Name := "None"

   END IF

   oDb:Skip()

END DO

 

I believe this code is clear to everyone :) The really good thing is that almost identical code could also work with SQL Server. Only the initialization was a little different:

 

oDb := SQLSelect ("SELECT * FROM Test ORDER BY Id")

To view this discussion visit https://groups.google.com/d/msgid/harbour-users/CADPHLr-OJcJniJq7XKDmy%3D7xfdW_GxBGP4JoYLa9WXG0co32PQ%40mail.gmail.com.

[Mario H. Sabado]

Hi Atul,

Not an OOP sample but I have come up with this approach when I have a need to export a DBF file with over ~21M records to CSV.  I need to process each line as an array because of some necessary data conversion before saving to the final CSV.  I also tried the FOR/While combination approach but it's over 10X slower than AEval() in my case scenario.

Below is my code snippet based on my understanding of your case.

Best regards,

Mario

*************************
function MasterDetailSubs
*************************
local xVal,yVal

local aHeader:={{"H1","H1 Header"},;
                {"H2","H2 Header"},;
                {"H3","H3 Header"} }

local aDetails:={{"H1","H1-Detail 1"},;
                 {"H1","H1-Detail 2"},;
                 {"H1","H1-Detail 3"},;
                 ;
                 {"H2","H2-Detail 1"},;
                 {"H2","H2-Detail 2"},;
                 {"H2","H2-Detail 3"},;
                 ;
                 {"H3","H3-Detail 1"},;
                 {"H3","H3-Detail 2"},;
                 {"H3","H3-Detail 3"} }

local aSubs  :={ {"H1","H1-Detail 1","H1 D1 Sub 1"},;
                 {"H1","H1-Detail 1","H1 D1 Sub 2"},;  
                 ;              
                 {"H1","H1-Detail 2","H1 D2 Sub 1"},;
                 {"H1","H1-Detail 2","H1 D2 Sub 2"},;
                 ;
                 {"H1","H1-Detail 3","H1 D3 Sub 1"},;
                 {"H1","H1-Detail 3","H1 D3 Sub 2"},;
                 ;
                 {"H2","H2-Detail 1","H2 D1 Sub 1"},;
                 {"H2","H2-Detail 1","H2 D1 Sub 2"},;
                 ;                
                 {"H2","H2-Detail 2","H2 D2 Sub 1"},;
                 {"H2","H2-Detail 2","H2 D2 Sub 2"},;
                 ;
                 {"H2","H2-Detail 3","H2 D3 Sub 1"},;
                 {"H2","H2-Detail 3","H2 D3 Sub 2"},;
                 ;  
                 {"H3","H3-Detail 1","H3 D1 Sub 1"},;
                 {"H3","H3-Detail 1","H3 D1 Sub 2"},;
                 ;                
                 {"H3","H3-Detail 2","H3 D2 Sub 1"},;
                 {"H3","H3-Detail 2","H3 D2 Sub 2"},;
                 ;
                 {"H3","H3-Detail 3","H3 D3 Sub 1"},;
                 {"H3","H3-Detail 3","H3 D3 Sub 2"} }  

AEval(aHeader,{|x|Qout( (xVal:=x[1])+"-"+x[2] ), ; //Header
             ,AEval(aDetails,{|y| iif(xVal==y[1],Qout(yVal:=y[1]+y[2]), ) ; //Detail
                            , AEval(aSubs,{|z| iif(xVal==z[1] .and. yVal==z[1]+z[2], Qout( yVal+"-"+z[3]) , ) ; //Sub-Detail
                                          }) ;
                              }) ;  
              })
return nil

/* SAMPLE OUTPUT
H1-Header 1
H1H1-Detail 1
H1H1-Detail 1-H1 D1 Sub 1
H1H1-Detail 1-H1 D1 Sub 2
H1H1-Detail 2
H1H1-Detail 2-H1 D2 Sub 1
H1H1-Detail 2-H1 D2 Sub 2
H1H1-Detail 3
H1H1-Detail 3-H1 D3 Sub 1
H1H1-Detail 3-H1 D3 Sub 2
H1H1-Detail 3-H1 D3 Sub 1
H1H1-Detail 3-H1 D3 Sub 2
H1H1-Detail 3-H1 D3 Sub 1
H1H1-Detail 3-H1 D3 Sub 2
H1H1-Detail 3-H1 D3 Sub 1
H1H1-Detail 3-H1 D3 Sub 2
H1H1-Detail 3-H1 D3 Sub 1
H1H1-Detail 3-H1 D3 Sub 2
H1H1-Detail 3-H1 D3 Sub 1
H1H1-Detail 3-H1 D3 Sub 2
H1H1-Detail 3-H1 D3 Sub 1
H1H1-Detail 3-H1 D3 Sub 2
H2-Header 2
H2H2-Detail 1
H2H2-Detail 1-H2 D1 Sub 1
H2H2-Detail 1-H2 D1 Sub 2
H2H2-Detail 2
H2H2-Detail 2-H2 D2 Sub 1
H2H2-Detail 2-H2 D2 Sub 2
H2H2-Detail 3
H2H2-Detail 3-H2 D3 Sub 1
H2H2-Detail 3-H2 D3 Sub 2
H2H2-Detail 3-H2 D3 Sub 1
H2H2-Detail 3-H2 D3 Sub 2
H2H2-Detail 3-H2 D3 Sub 1
H2H2-Detail 3-H2 D3 Sub 2
H2H2-Detail 3-H2 D3 Sub 1
H2H2-Detail 3-H2 D3 Sub 2
H3-Header 3
H3H3-Detail 1
H3H3-Detail 1-H3 D1 Sub 1
H3H3-Detail 1-H3 D1 Sub 2
H3H3-Detail 2
H3H3-Detail 2-H3 D2 Sub 1
H3H3-Detail 2-H3 D2 Sub 2
H3H3-Detail 3
H3H3-Detail 3-H3 D3 Sub 1
H3H3-Detail 3-H3 D3 Sub 2

*/    

To view this discussion visit https://groups.google.com/d/msgid/harbour-users/00e401dcba52%2417cc93c0%244765bb40%24%40wings.rs.