Hi Folks,
I really felt like it should somehow be possible to use gdb to directly inspect the 'data' array that is stashed inside the internal data structure used by the Vec implementation. I just spent waaaaay too much time figuring this out, BUT, I did determine that
there is a way to directly inspect the contents of a PETSc Vec inside PFLOTRAN using GDB. The way it has to be done is ugly (basically because Fortran has no concept of dealing with two levels of pointer indirection) but it works, and is nicer than having
to add VecGetArray() calls and recompile. Here is the short summary:
Given a PETSc Vec r in a Fortran code, to inspect the contents of the local portion of the array, which are stashed in an array pointed to by the 'data' field in PETSc's internal Vec implementation data structure, you need to do two steps, both employing the
gdb "examine" command ('x'), which lets you print values stored at an arbitrary memory location.
FIRST, Use the 'x' command to print (as a hex address) the value stored in the 'data' field in r:
(gdb) x/1xg (*((Vec) r))%data
0xaaaaab9d2550: 0x0000aaaaab9d1bd0
SECOND, Use the 'x' command to print as many entries starting at the above address as you want (10 in this example):
(gdb) x/10fg 0x0000aaaaab9d1bd0
0xaaaaab9d1bd0: 1.7507793726584433e-06 -4.4102733314192253e-09
0xaaaaab9d1be0: -4.3615950165322133e-09 -4.3130382801110155e-09
0xaaaaab9d1bf0: -4.2646017652029213e-09 -4.2162841220914224e-09
0xaaaaab9d1c00: -4.1680840119196081e-09 -4.1200000795427596e-09
0xaaaaab9d1c10: -4.0720309933405922e-09 -4.0241754108366118e-09
Below is a GDB session (with comments in it that explain what is going on) that demonstrates all of the gory details and gives an idea of how I arrived at this. (This is an instructive exercise in using GDB with Fortran!) If you want to try it yourself, go
into the 1D_Calcite regression problem directory, launch gdb on your PFLOTRAN executable, and follow along:
(gdb) set args -input_prefix calcite_flow_and_tran
(gdb) b richards.F90:1658
(gdb) run
Starting program: /home/rmills/proj/pflotran/regression_tests/shortcourse/1D_Calcite/pflotran -input_prefix calcite_flow_and_tran
[...]
Thread 1 "pflotran" hit Breakpoint 1, richards_module::richardsresidualinternalconn (r=...,
realization=..., skip_conn_type=1, ierr=0) at richards.F90:1658
(gdb) # Let's continue to the next time RichardsResidualInternalConn is called, so that we have nonzero things in r_p:
(gdb) c
[...]
Thread 1 "pflotran" hit Breakpoint 1, richards_module::richardsresidualinternalconn (r=...,
realization=..., skip_conn_type=1, ierr=0) at richards.F90:1658
1658 call VecRestoreArray(r,r_p,ierr);CHKERRQ(ierr)
(gdb) # We've already gotten, from VecGetArray(), a nice pointer to the contents of the Vec:
(gdb) p r_p
$2 = (1.7507793726584433e-06, -4.4102733314192253e-09, -4.3615950165322133e-09, -4.3130382801110155e-09, -4.2646017652029213e-09, -4.2162841220914224e-09, -4.1680840119196081e-09, -4.1200000795427596e-09, -4.0720309933405922e-09, -4.0241754108366118e-09, -3.9764320004098985e-09,
-3.9287994340576336e-09, -3.8812763765403726e-09, -3.8338615125254272e-09, -3.7865535140048967e-09, -3.739351060222541e-09, -3.6922528430740395e-09, -3.6452575472269158e-09, -3.5983638555354078e-09, -3.5515704707583925e-09, -3.5048760847976901e-09, -3.4582793931757632e-09,
-3.4117791004594807e-09, -3.3653739112171936e-09, -3.319062530017888e-09, -3.2728436614301268e-09, -3.226716029926053e-09, -3.180678334645807e-09, -3.1347293054893188e-09, -3.0888676488363194e-09, -3.0430921000141025e-09, -2.997401370830398e-09, -2.9517941966133467e-09,
-2.9062693000264643e-09, -2.8608254182086364e-09, -2.8154612756332764e-09, -2.7701756166748369e-09, -2.7249671730458986e-09, -2.679834690931447e-09, -2.634776903852478e-09, -2.5897925634221879e-09, -2.5448804104003161e-09, -2.5000391945914317e-09, -2.4552676694190521e-09,
-2.4105645828789075e-09, -2.365928690206107e-09, -2.3213587466338538e-09, -2.2768535092056723e-09, -2.232411740392874e-09, -2.1880322008587787e-09, -2.1437136476464873e-09, -2.099454854083309e-09, -2.0552545808342576e-09, -2.0111115976081177e-09, -1.9670246741151561e-09,
-1.9229925800656398e-09, -1.8790140905974109e-09, -1.835087975422642e-09, -1.7912130187240051e-09, -1.7473879865981127e-09, -1.7036116650352048e-09, -1.6598828309881035e-09, -1.6162002686405394e-09, -1.5725627531339547e-09, -1.5289690758908235e-09, -1.4854180210993233e-09,
-1.4419083675173034e-09, -1.3984389083796767e-09, -1.3550084351112467e-09, -1.3116157282801838e-09, -1.2682595847401375e-09, -1.2249387941062254e-09, -1.1816521496156896e-09, -1.1383984445013256e-09, -1.0951764701900544e-09, -1.0519850289635243e-09, -1.0088229068217163e-09,
-9.6568890966437985e-10, -9.2258183072663972e-10, -8.7950047229543813e-10, -8.3644362579451977e-10, -7.9341009893607294e-10, -7.5039868676639048e-10, -7.0740819880840537e-10, -6.6443742286966689e-10, -6.2148517389983635e-10, -5.7855024875743359e-10, -5.3563145334263167e-10,
-4.9272758451691505e-10, -4.4983745903518401e-10, -4.069598665208147e-10, -3.640936255434755e-10, -3.2123753115009453e-10, -2.7839039286466355e-10, -2.3555101840297074e-10, -1.9271821004984076e-10, -1.4989077190253702e-10, -1.0706751891305018e-10, -6.424724974979773e-11,
-1.5361492758900348e-06)
(gdb) # Now let's see what we can inspect about the original Vec r to get at the above numerical values.
(gdb) p r
$3 = ( tpetscobject = ( v = 187650000356032 ) )
(gdb) p &r
$4 = (PTR TO -> ( Type tvec )) 0xffffffffd198
(gdb) p (Vec) r
$5 = (Vec) 0xaaaaab9d02c0
(gdb) # We can dereference this to see all fields of the implementation's _p_Vec_ struct:
(gdb) p *((Vec) r)
$6 = ( hdr = ( bops = (( view = 0xfffff39a4580 <VecView>, destroy = 0xfffff39a15c8 <VecDestroy> )), classid = 1211235, comm = 0xaaaaab743b90, id = 51, refct = 3, non_cyclic_references = 0x0, cidx = 31, tag = 2147483604, qlist = 0x0, olist = 0xaaaaab9d3fc0,
class_name = 0xfffff79a36a8 'Vec\000', description = 0xfffff79a36b0 'Vector\000', mansec = 0xfffff79a36a8 'Vec\000', type_name = 0xaaaaab9d2bf0 'seq\000', name = 0x0, prefix = 0x0, tablevel = 0, cpp = 0x0, state = 11, int_idmax = 0, intstar_idmax = 0, intcomposedstate
= 0x0, intstarcomposedstate = 0x0, intcomposeddata = 0x0, intstarcomposeddata = 0x0, real_idmax = 10, realstar_idmax = 0, realcomposedstate = 0xaaaaab9d3910, realstarcomposedstate = 0x0, realcomposeddata = 0xaaaaab9d3260, realstarcomposeddata = 0x0, fortran_func_pointers
= 0x0, num_fortran_func_pointers = 0, fortrancallback = (0x0, 0x0), num_fortrancallback = (0, 0), python_context = 0x0, python_destroy = 0x0, noptionhandler = 0, optionhandler = (0x0, 0x0, 0x0, 0x0, 0x0), optiondestroy = (0x0, 0x0, 0x0, 0x0, 0x0), optionctx
= (0x0, 0x0, 0x0, 0x0, 0x0), options = 0x0, optionsprinted = .FALSE., donotPetscObjectPrintClassNamePrefixType = .FALSE. ), ops = (( duplicate = 0xfffff38fe9fc <VecDuplicate_Seq>, duplicatevecs = 0xfffff38ff130 <VecDuplicateVecs_Seq_GEMV>, destroyvecs = 0xfffff39aa5d4
<VecDestroyVecs_Default>, dot = 0xfffff38e9170 <VecDot_Seq>, mdot = 0xfffff390cc68 <VecMDot_Seq_GEMV>, norm = 0xfffff38f3404 <VecNorm_Seq>, tdot = 0xfffff38e9664 <VecTDot_Seq>, mtdot = 0xfffff390d160 <VecMTDot_Seq_GEMV>, scale = 0xfffff38e9b58 <VecScale_Seq>,
copy = 0xfffff38f1e10 <VecCopy_Seq>, set = 0xfffff390e738 <VecSet_Seq>, swap = 0xfffff38f26b0 <VecSwap_Seq>, axpy = 0xfffff38ea8ac <VecAXPY_Seq>, axpby = 0xfffff38eb6e8 <VecAXPBY_Seq>, maxpy = 0xfffff390ee48 <VecMAXPY_Seq>, aypx = 0xfffff391171c <VecAYPX_Seq>,
waxpy = 0xfffff3912348 <VecWAXPY_Seq>, axpbypcz = 0xfffff38ec404 <VecAXPBYPCZ_Seq>, pointwisemult = 0xfffff38efa48 <VecPointwiseMult_Seq>, pointwisedivide = 0xfffff38f0620 <VecPointwiseDivide_Seq>, setvalues = 0xfffff38fa420 <VecSetValues_Seq>, assemblybegin
= 0x0, assemblyend = 0x0, getarray = 0x0, getsize = 0xfffff38f11d4 <VecGetSize_Seq>, getlocalsize = 0xfffff38f11d4 <VecGetSize_Seq>, restorearray = 0x0, max = 0xfffff390dd18 <VecMax_Seq>, min = 0xfffff390e23c <VecMin_Seq>, setrandom = 0xfffff38f0b14 <VecSetRandom_Seq>,
setoption = 0xfffff38fdec8 <VecSetOption_Seq>, setvaluesblocked = 0xfffff38fac5c <VecSetValuesBlocked_Seq>, destroy = 0xfffff38fd510 <VecDestroy_Seq>, view = 0xfffff4f8cbdc <VecView_MPI_DA>, placearray = 0xfffff3913b04 <VecPlaceArray_Seq>, replacearray = 0xfffff3913fa0
<VecReplaceArray_Seq>, dot_local = 0xfffff38e9170 <VecDot_Seq>, tdot_local = 0xfffff38e9664 <VecTDot_Seq>, norm_local = 0xfffff38f3404 <VecNorm_Seq>, mdot_local = 0xfffff390cc68 <VecMDot_Seq_GEMV>, mtdot_local = 0xfffff390d160 <VecMTDot_Seq_GEMV>, load = 0xfffff4f95ff8
<VecLoad_Default_DA>, reciprocal = 0xfffff3a86d6c <VecReciprocal_Default>, conjugate = 0xfffff38f15ec <VecConjugate_Seq>, setlocaltoglobalmapping = 0x0, getlocaltoglobalmapping = 0x0, resetarray = 0xfffff38f19ec <VecResetArray_Seq>, setfromoptions = 0x0, maxpointwisedivide
= 0xfffff39131b8 <VecMaxPointwiseDivide_Seq>, pointwisemax = 0xfffff38eeae4 <VecPointwiseMax_Seq>, pointwisemaxabs = 0xfffff38ef554 <VecPointwiseMaxAbs_Seq>, pointwisemin = 0xfffff38ef01c <VecPointwiseMin_Seq>, getvalues = 0xfffff38f9c6c <VecGetValues_Seq>,
sqrt = 0x0, abs = 0x0, exp = 0x0, log = 0x0, shift = 0x0, create = 0x0, stridegather = 0xfffff3a81830 <VecStrideGather_Default>, stridescatter = 0xfffff3a82458 <VecStrideScatter_Default>, dotnorm2 = 0x0, getsubvector = 0x0, restoresubvector = 0x0, getarrayread
= 0x0, restorearrayread = 0x0, stridesubsetgather = 0xfffff3a83098 <VecStrideSubSetGather_Default>, stridesubsetscatter = 0xfffff3a84324 <VecStrideSubSetScatter_Default>, viewnative = 0xfffff38f8e30 <VecView_Seq>, loadnative = 0xfffff3a4cca8 <VecLoad_Default>,
createlocalvector = 0x0, getlocalvector = 0x0, restorelocalvector = 0x0, getlocalvectorread = 0x0, restorelocalvectorread = 0x0, bindtocpu = 0x0, getarraywrite = 0x0, restorearraywrite = 0x0, getarrayandmemtype = 0x0, restorearrayandmemtype = 0x0, getarrayreadandmemtype
= 0x0, restorearrayreadandmemtype = 0x0, getarraywriteandmemtype = 0x0, restorearraywriteandmemtype = 0x0, concatenate = 0x0, sum = 0x0, setpreallocationcoo = 0xfffff38fbb54 <VecSetPreallocationCOO_Seq>, setvaluescoo = 0xfffff38fcd50 <VecSetValuesCOO_Seq>,
errorwnorm = 0x0, maxpby = 0x0 )), map = 0xaaaaab9c3ac0, data = 0xaaaaab9d2550, array_gotten = .FALSE., stash = ( nmax = 0, umax = 0, oldnmax = 0, n = 0, bs = 0, reallocs = 0, idx = 0x0, array = 0x0, comm = 0x0, size = 0, rank = 0, tag1 = 0, tag2 = 0, send_waits
= 0x0, recv_waits = 0x0, send_status = 0x0, nsends = 0, nrecvs = 0, svalues = 0x0, rvalues = 0x0, sindices = 0x0, rindices = 0x0, rmax = 0, nprocs = 0x0, nprocessed = 0, donotstash = .FALSE., ignorenegidx = .FALSE., insertmode = NOT_SET_VALUES, bowners = 0x0
), bstash = ( nmax = 0, umax = 0, oldnmax = 0, n = 0, bs = 1, reallocs = 0, idx = 0x0, array = 0x0, comm = 0x0, size = 0, rank = 0, tag1 = 0, tag2 = 0, send_waits = 0x0, recv_waits = 0x0, send_status = 0x0, nsends = 0, nrecvs = 0, svalues = 0x0, rvalues =
0x0, sindices = 0x0, rindices = 0x0, rmax = 0, nprocs = 0x0, nprocessed = 0, donotstash = .FALSE., ignorenegidx = .FALSE., insertmode = NOT_SET_VALUES, bowners = 0x0 ), petscnative = .TRUE., lock = 0, lockstack = ( function = (0x0, <repeats 64 times>), file
= (0x0, <repeats 64 times>), line = (0, <repeats 64 times>), petscroutine = (0, <repeats 64 times>), currentsize = 0, hotdepth = 0, check = .FALSE. ), offloadmask = PETSC_OFFLOAD_CPU, spptr = 0x0, boundtocpu = .TRUE., bindingpropagates = .FALSE., minimum_bytes_pinned_memory
= 0, pinned_memory = .FALSE., defaultrandtype = 0xaaaaab9d1560 'rander48\000' )
(gdb) # Now let's try to get at the 'data' member:
(gdb) p (*((Vec) r))%data
$7 = (PTR TO -> ( void )) 0xaaaaab9d2550
(gdb) p *( (*((Vec) r))%data)
Attempt to dereference a generic pointer.
(gdb) # How do I cast the above to a pointer to a double/real*8 and then dereference?
(gdb) p *( (real_8 *) ((*((Vec) r))%data) )
$8 = 9.2711418621183019e-310
(gdb) # But the above is the wrong value! It should match r_p(1):
$9 = 1.7507793726584433e-06
(gdb) # The problem is that the value in %data isn't pointing to the memory address the array starts at;
(gdb) # It is pointing to another (void) pointer that points to the start of the array.
(gdb) # There are two levels of indirection, and Fortran just cannot deal with that!
(gdb) # The address where the array starts should be the address that r_p points to:
(gdb) p &r_p
$10 = (PTR TO -> ( real(kind=8) (100) )) 0xaaaaab9d1bd0
(gdb) # But, above, we already have seen that the address being pointed to by %data is 0xaaaaab9d2550.
(gdb) # Let's use the examine command ('x') to print the value of whatever resides at that address as a hex number:
(gdb) x/1xg 0xaaaaab9d2550
0xaaaaab9d2550: 0x0000aaaaab9d1bd0
(gdb) # AHA! That is the address that the array referenced by r_p starts at!
(gdb) # So now let's use the 'x' command to print 100 floating point numbers, starting at that address:
(gdb) x/100fg 0x0000aaaaab9d1bd0
0xaaaaab9d1bd0: 1.7507793726584433e-06 -4.4102733314192253e-09
0xaaaaab9d1be0: -4.3615950165322133e-09 -4.3130382801110155e-09
0xaaaaab9d1bf0: -4.2646017652029213e-09 -4.2162841220914224e-09
0xaaaaab9d1c00: -4.1680840119196081e-09 -4.1200000795427596e-09
0xaaaaab9d1c10: -4.0720309933405922e-09 -4.0241754108366118e-09
0xaaaaab9d1c20: -3.9764320004098985e-09 -3.9287994340576336e-09
0xaaaaab9d1c30: -3.8812763765403726e-09 -3.8338615125254272e-09
0xaaaaab9d1c40: -3.7865535140048967e-09 -3.739351060222541e-09
0xaaaaab9d1c50: -3.6922528430740395e-09 -3.6452575472269158e-09
0xaaaaab9d1c60: -3.5983638555354078e-09 -3.5515704707583925e-09
0xaaaaab9d1c70: -3.5048760847976901e-09 -3.4582793931757632e-09
0xaaaaab9d1c80: -3.4117791004594807e-09 -3.3653739112171936e-09
0xaaaaab9d1c90: -3.319062530017888e-09 -3.2728436614301268e-09
0xaaaaab9d1ca0: -3.226716029926053e-09 -3.180678334645807e-09
0xaaaaab9d1cb0: -3.1347293054893188e-09 -3.0888676488363194e-09
0xaaaaab9d1cc0: -3.0430921000141025e-09 -2.997401370830398e-09
0xaaaaab9d1cd0: -2.9517941966133467e-09 -2.9062693000264643e-09
0xaaaaab9d1ce0: -2.8608254182086364e-09 -2.8154612756332764e-09
0xaaaaab9d1cf0: -2.7701756166748369e-09 -2.7249671730458986e-09
0xaaaaab9d1d00: -2.679834690931447e-09 -2.634776903852478e-09
0xaaaaab9d1d10: -2.5897925634221879e-09 -2.5448804104003161e-09
0xaaaaab9d1d20: -2.5000391945914317e-09 -2.4552676694190521e-09
0xaaaaab9d1d30: -2.4105645828789075e-09 -2.365928690206107e-09
0xaaaaab9d1d40: -2.3213587466338538e-09 -2.2768535092056723e-09
0xaaaaab9d1d50: -2.232411740392874e-09 -2.1880322008587787e-09
0xaaaaab9d1d60: -2.1437136476464873e-09 -2.099454854083309e-09
0xaaaaab9d1d70: -2.0552545808342576e-09 -2.0111115976081177e-09
0xaaaaab9d1d80: -1.9670246741151561e-09 -1.9229925800656398e-09
0xaaaaab9d1d90: -1.8790140905974109e-09 -1.835087975422642e-09
0xaaaaab9d1da0: -1.7912130187240051e-09 -1.7473879865981127e-09
0xaaaaab9d1db0: -1.7036116650352048e-09 -1.6598828309881035e-09
0xaaaaab9d1dc0: -1.6162002686405394e-09 -1.5725627531339547e-09
0xaaaaab9d1dd0: -1.5289690758908235e-09 -1.4854180210993233e-09
0xaaaaab9d1de0: -1.4419083675173034e-09 -1.3984389083796767e-09
0xaaaaab9d1df0: -1.3550084351112467e-09 -1.3116157282801838e-09
0xaaaaab9d1e00: -1.2682595847401375e-09 -1.2249387941062254e-09
0xaaaaab9d1e10: -1.1816521496156896e-09 -1.1383984445013256e-09
0xaaaaab9d1e20: -1.0951764701900544e-09 -1.0519850289635243e-09
0xaaaaab9d1e30: -1.0088229068217163e-09 -9.6568890966437985e-10
0xaaaaab9d1e40: -9.2258183072663972e-10 -8.7950047229543813e-10
0xaaaaab9d1e50: -8.3644362579451977e-10 -7.9341009893607294e-10
0xaaaaab9d1e60: -7.5039868676639048e-10 -7.0740819880840537e-10
0xaaaaab9d1e70: -6.6443742286966689e-10 -6.2148517389983635e-10
0xaaaaab9d1e80: -5.7855024875743359e-10 -5.3563145334263167e-10
0xaaaaab9d1e90: -4.9272758451691505e-10 -4.4983745903518401e-10
0xaaaaab9d1ea0: -4.069598665208147e-10 -3.640936255434755e-10
0xaaaaab9d1eb0: -3.2123753115009453e-10 -2.7839039286466355e-10
0xaaaaab9d1ec0: -2.3555101840297074e-10 -1.9271821004984076e-10
0xaaaaab9d1ed0: -1.4989077190253702e-10 -1.0706751891305018e-10
0xaaaaab9d1ee0: -6.424724974979773e-11 -1.5361492758900348e-06
(gdb) # And this matches what gdb prints when we ask it to print r_p (command: "p r_p") above!
(gdb) # TLDR SUMMARY: to view the values of the array contained in r, we need two steps.
(gdb) # First, use the 'x' command to print (as a hex address) the value stored in the 'data' field in r:
(gdb) x/1xg (*((Vec) r))%data
0xaaaaab9d2550: 0x0000aaaaab9d1bd0
(gdb) # Then use the 'x' command to print as many entries starting at the above address as you want (10 in this example):
(gdb) x/10fg 0x0000aaaaab9d1bd0
0xaaaaab9d1bd0: 1.7507793726584433e-06 -4.4102733314192253e-09
0xaaaaab9d1be0: -4.3615950165322133e-09 -4.3130382801110155e-09
0xaaaaab9d1bf0: -4.2646017652029213e-09 -4.2162841220914224e-09
0xaaaaab9d1c00: -4.1680840119196081e-09 -4.1200000795427596e-09
0xaaaaab9d1c10: -4.0720309933405922e-09 -4.0241754108366118e-09