IBAMR (IBFE) restart instructions

[Alexander Hoover]

Hi all,

I'm trying to set up a run from a restart interval and am having a little trouble. I've installed a more recent version of IBAMR (libmesh 1.0, petsc 3.7.3) to IBFE, and everything seems to run ok. I first adjusted the restart dump parameters in the input file

// restart dump parameters

   restart_dump_interval       = 1     // just to test the restart

   restart_dump_dirname        = "restart_IB3d"

which outputted restore.1, restore.2,..., to the restart_IB3d folder in the initial run. I then set up tried to restart the run with the command 

./main3d input3d restart_IB3d 2

and the following error pops up:

IBFEMethod: using FIRST order LAGRANGE finite elements.

#0  0x0000003e0fa99e0f in ?? ()

#1  0xffffffffffff8e90 in ?? ()

#2  0x00007fffffffc28c in ?? ()

#3  0x0000000000000001 in ?? ()

#4  0x0000003e0fa3c4a1 in ?? ()

#5  0x0000000000000001 in ?? ()

#6  0x0000000000010000 in ?? ()

#7  0x0000000000000000 in ?? ()

[0] ../libmesh-1.0.0/src/utils/xdr_cxx.C, line 177, compiled Aug 22 2016 at 18:31:13

*********************************************************************

READING THE FILE "/libmesh_data_part_0.000000.xdr" FAILED.

It is possible this file contains infinite element information,

but the version string does not contain " with infinite elements"

Let's try this again, but looking for infinite element information...

*********************************************************************

#0  0x0000003e0fa99e0f in ?? ()

#1  0xffffffffffff8e90 in ?? ()

#2  0x00007fffffffc28c in ?? ()

#3  0x0000000000000001 in ?? ()

#4  0x0000003e0fa3c4a1 in ?? ()

#5  0x0000000000000001 in ?? ()

#6  0x0000000000010000 in ?? ()

#7  0x0000000000000000 in ?? ()

[0] ../libmesh-1.0.0/src/utils/xdr_cxx.C, line 177, compiled Aug 22 2016 at 18:31:13

*********************************************************************

Well, at least we tried!

Good Luck!!

*********************************************************************

--------------------------------------------------------------------------

MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD 

with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.

You may or may not see output from other processes, depending on

exactly when Open MPI kills them.

--------------------------------------------------------------------------

Any hints? This is my app initializer portion of main.C:

       Pointer<AppInitializer> app_initializer = new AppInitializer(argc, argv, "IB.log");

       Pointer<Database> input_db = app_initializer->getInputDatabase();

//  Pointer<Database> input_db = app_initializer->getRestartDatabase();

        

        // Get various standard options set in the input file.

        const bool dump_viz_data = app_initializer->dumpVizData();

        const int viz_dump_interval = app_initializer->getVizDumpInterval();

        const bool uses_visit = dump_viz_data && app_initializer->getVisItDataWriter();

        const bool uses_exodus = dump_viz_data && !app_initializer->getExodusIIFilename().empty();

        const string exodus_filename = app_initializer->getExodusIIFilename();

        

        const bool dump_restart_data = app_initializer->dumpRestartData();

        const int restart_dump_interval = app_initializer->getRestartDumpInterval();

        const string restart_dump_dirname = app_initializer->getRestartDumpDirectory();

        

        const bool dump_postproc_data = app_initializer->dumpPostProcessingData();

        const int postproc_data_dump_interval = app_initializer->getPostProcessingDataDumpInterval();

        const string postproc_data_dump_dirname = app_initializer->getPostProcessingDataDumpDirectory();

Best,

Alex

[Boyce Griffith]

On Aug 30, 2016, at 3:57 PM, Alexander Hoover <hoover.a...@gmail.com> wrote:

Hi all,

I'm trying to set up a run from a restart interval and am having a little trouble. I've installed a more recent version of IBAMR (libmesh 1.0, petsc 3.7.3) to IBFE, and everything seems to run ok. I first adjusted the restart dump parameters in the input file

// restart dump parameters

   restart_dump_interval       = 1     // just to test the restart

   restart_dump_dirname        = "restart_IB3d"

which outputted restore.1, restore.2,..., to the restart_IB3d folder in the initial run. I then set up tried to restart the run with the command 

./main3d input3d restart_IB3d 2

and the following error pops up:

IBFEMethod: using FIRST order LAGRANGE finite elements.

#0  0x0000003e0fa99e0f in ?? ()

#1  0xffffffffffff8e90 in ?? ()

#2  0x00007fffffffc28c in ?? ()

#3  0x0000000000000001 in ?? ()

#4  0x0000003e0fa3c4a1 in ?? ()

#5  0x0000000000000001 in ?? ()

#6  0x0000000000010000 in ?? ()

#7  0x0000000000000000 in ?? ()

[0] ../libmesh-1.0.0/src/utils/xdr_cxx.C, line 177, compiled Aug 22 2016 at 18:31:13

*********************************************************************

READING THE FILE "/libmesh_data_part_0.000000.xdr" FAILED.

This is not a super informative error message, but what is going on here is that libMesh cannot find the restart file that you are requesting. Notice that the file name is "/libmesh_data_part_ ..." --- it is looking for this file in the root directory, /, which is almost certainly not where it is actually located.

This is not very well documented in IBAMR (well ... maybe not documented at all is more accurate), but I think that the way that IBFE restart works is that you have to tell the IBFEMethod constructor where the restart directory is and, if you are restarting, what the restart number is. It might make more sense to put the directory name into the input database for the class --- usually we don't require people to set this kind of stuff programatically. Anyway, take a look at the IBFEMethod constructors and let me know if you can't figure out how to update them to "do the right thing".

-- Boyce

--
You received this message because you are subscribed to the Google Groups "IBAMR Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ibamr-users...@googlegroups.com.
To post to this group, send email to ibamr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexander Hoover]

So, I believe I've gotten past this issue by adjusting the IBFEMethod constructor.

Pointer<IBFEMethod> ib_method_ops = new IBFEMethod( "IBFEMethod", app_initializer->getComponentDatabase("IBFEMethod"), &mesh, app_initializer->getComponentDatabase("GriddingAlgorithm")->getInteger("max_levels"), is_from_restart, restart_directory, restart_restore_number);

where earlier in the code:

        const bool is_from_restart= app_initializer->isFromRestart();

        const int restart_restore_number = app_initializer->getRestartRestoreNumber();

        const string restart_directory = app_initializer->getRestartReadDirectory();

Now the issue is if it's actually restarting. I take it that we can use the bool from above to select the proper input_db, ala

        if(is_from_restart){

    Pointer<Database> input_db = app_initializer->getRestartDatabase();

}else{

       Pointer<Database> input_db = app_initializer->getInputDatabase();

        }

If one doesn't do that, it would just use the input database, correct? But now I'm having a bit of trouble find how to unload parameters from the restart database. I.e. with the input database we could call 

const double dx = input_db->getDouble("DX");

What's the equivalent command for restart database?

-Alex

[Boyce Griffith]

On Aug 31, 2016, at 1:18 PM, Alexander Hoover <hoover.a...@gmail.com> wrote:

So, I believe I've gotten past this issue by adjusting the IBFEMethod constructor.

Pointer<IBFEMethod> ib_method_ops = new IBFEMethod( "IBFEMethod", app_initializer->getComponentDatabase("IBFEMethod"), &mesh, app_initializer->getComponentDatabase("GriddingAlgorithm")->getInteger("max_levels"), is_from_restart, restart_directory, restart_restore_number);

where earlier in the code:

        const bool is_from_restart= app_initializer->isFromRestart();

        const int restart_restore_number = app_initializer->getRestartRestoreNumber();

        const string restart_directory = app_initializer->getRestartReadDirectory();

Now the issue is if it's actually restarting. I take it that we can use the bool from above to select the proper input_db, ala

        if(is_from_restart){

     Pointer<Database> input_db = app_initializer->getRestartDatabase();

}else{

         Pointer<Database> input_db = app_initializer->getInputDatabase();

        }

No that is not what you want to do --- you just use the regular input database as usual. The restart database has all of the data needed to reinitialize the state of the various classes in the application. Some objects may also re-initialize some state information based on the input database.

The only change you should need to make in the code is to add the extra lines on where the IBFE restart data are stored.

[Alexander Hoover]

> No that is not what you want to do --- you just use the regular input
> database as usual. The restart database has all of the data needed to
> reinitialize the state of the various classes in the application. Some
> objects may also re-initialize some state information based on the input
> database.

I tried this but it didn't seem to work based on some of the outputted
quantities (max/min Jacobian restarted to 1, simulation time set to
0). Should getRestartDatabase ever be called in main.C or is that done
somewhere else? Is there anywhere else in the code I should be careful
(like initial coordinate mappings, other non-solver specific things)?

> The only change you should need to make in the code is to add the extra
> lines on where the IBFE restart data are stored.
>

Is this already set up in the examples?

[Alexander Hoover]

> The only change you should need to make in the code is to add the extra
> lines on where the IBFE restart data are stored.
>

Is this already set up in the examples?

^Or is this what I already did with IBFEMethod?

[Boyce Griffith]

> On Aug 31, 2016, at 1:48 PM, Alexander Hoover <hoover.a...@gmail.com> wrote:
>
>> No that is not what you want to do --- you just use the regular input
>> database as usual. The restart database has all of the data needed to
>> reinitialize the state of the various classes in the application. Some
>> objects may also re-initialize some state information based on the input
>> database.
>
> I tried this but it didn't seem to work based on some of the outputted
> quantities (max/min Jacobian restarted to 1, simulation time set to
> 0).

These min/max values are not treated as state values and won't survive restarting. You need to register them for restart if you want them to be maintained across restarts. I don't think it is worth the trouble.

OTOH, the simulation time should be correct so long as you are reading the time from the time integrator and not tracking it yourself.

> Should getRestartDatabase ever be called in main.C or is that done
> somewhere else? Is there anywhere else in the code I should be careful
> (like initial coordinate mappings, other non-solver specific things)?

Initial conditions should not have any effect upon restart.

>> The only change you should need to make in the code is to add the extra
>> lines on where the IBFE restart data are stored.
>>
>
> Is this already set up in the examples?

Try out the standard IB examples.

[Alexander Hoover]

Previous issue related to not inputting the full restart directory and so ignored the restart. I set bool to output a confirmation we're using the restart db. I now get the issue below, which is in line with what you mentioned of having to write FEData to the restart.

Starting from a restart...


*** Warning, This code is deprecated, and likely to be removed in future library versions! ./include/libmesh/auto_ptr.h, line 270, compiled Aug 22 2016 at 18:24:08 ***
P=00000:Program abort called in file ``/nas02/home/h/o/hooverap/sfw/ibamr/IBAMR_082016/IBAMR/lib/../src/IB/IBFEMethod.cpp'' at line 2379
P=00000:ERROR MESSAGE:
P=00000:IBFEMethod:  Restart database corresponding to IBFEMethod not found in restart file.
P=00000:
[killdevil-login2:01972] *** Process received signal ***
[killdevil-login2:01972] Signal: Aborted (6)
[killdevil-login2:01972] Signal code:  (-6)
[killdevil-login2:01972] [ 0] /lib64/libpthread.so.0 [0x3e10a0eca0]
[killdevil-login2:01972] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x3e0fa2ffc5]
[killdevil-login2:01972] [ 2] /lib64/libc.so.6(abort+0x110) [0x3e0fa31a70]
[killdevil-login2:01972] [ 3] ./main3d(_ZN6SAMRAI4tbox10SAMRAI_MPI5abortEv+0x4e) [0xc3688e]
[killdevil-login2:01972] [ 4] ./main3d(_ZN5IBAMR10IBFEMethod14getFromRestartEv+0x5b5) [0x742ba5]
[killdevil-login2:01972] [ 5] ./main3d(_ZN5IBAMR10IBFEMethod17commonConstructorERKSsN6SAMRAI4tbox7PointerINS4_8DatabaseEEERKSt6vectorIPN7libMesh4MeshESaISB_EEibS2_j+0xa3a) [0x75adfa]
[killdevil-login2:01972] [ 6] ./main3d(_ZN5IBAMR10IBFEMethodC1ERKSsN6SAMRAI4tbox7PointerINS4_8DatabaseEEEPN7libMesh4MeshEibS2_j+0x467) [0x75d4a7]
[killdevil-login2:01972] [ 7] ./main3d(main+0x9f6) [0x607f86]
[killdevil-login2:01972] [ 8] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3e0fa1d9f4]
[killdevil-login2:01972] [ 9] ./main3d(_ZNK7libMesh10SerialMesh13pid_nodes_endEt+0x71) [0x525579]
[killdevil-login2:01972] *** End of error message ***

Aborted

To fix this I tried using

void writeFEDataToRestartFile(const std::string& restart_dump_dirname, unsigned int time_step_number);

during the restart file writing

            if (dump_restart_data && (iteration_num%restart_dump_interval == 0 || last_step))
            {
                pout << "\nWriting restart files...\n\n";
                RestartManager::getManager()->writeRestartFile(restart_dump_dirname, iteration_num);
                ib_method_ops::writeFEDataToRestartFile(restart_dump_dirname, iteration_num);
            }

but I still get the same errors. I couldn't find any other call of writeFEDataToRestartFile.

 The examples do not seem to be set up for restarts (I get my initial error) and their code doesn't seem to do anything differently than what I initially set up.

[Alexander Hoover]

Also, I noticed that files associated with the mesh restart file, like 'libmesh_data_part_0.000001.xdr', are being created.

[Boyce Griffith]

On Aug 31, 2016, at 2:50 PM, Alexander Hoover <hoover.a...@gmail.com> wrote:

Previous issue related to not inputting the full restart directory and so ignored the restart. I set bool to output a confirmation we're using the restart db. I now get the issue below, which is in line with what you mentioned of having to write FEData to the restart.

Starting from a restart...


*** Warning, This code is deprecated, and likely to be removed in future library versions! ./include/libmesh/auto_ptr.h, line 270, compiled Aug 22 2016 at 18:24:08 ***
P=00000:Program abort called in file ``/nas02/home/h/o/hooverap/sfw/ibamr/IBAMR_082016/IBAMR/lib/../src/IB/IBFEMethod.cpp'' at line 2379
P=00000:ERROR MESSAGE:
P=00000:IBFEMethod:  Restart database corresponding to IBFEMethod not found in restart file.

This probably means that IBFEMethod was not set up to write to restart. I'd suggest wiping the restart directory and starting de novo.

You are correct, the IBFE examples are not set up for restart currently. If you want to make a PR that adds in support, that would be great. :-)

[Alexander Hoover]

Still getting the same  error

./main3d input3d $PWD/restart_IB3d 4

 Starting from a restart...

*** Warning, This code is deprecated, and likely to be removed in future library versions! ./include/libmesh/auto_ptr.h, line 270, compiled Aug 22 2016 at 18:24:08 ***

P=00000:Program abort called in file ``/nas02/home/h/o/hooverap/sfw/ibamr/IBAMR_082016/IBAMR/lib/../src/IB/IBFEMethod.cpp'' at line 2379

P=00000:ERROR MESSAGE: 

P=00000:IBFEMethod:  Restart database corresponding to IBFEMethod not found in restart file.

P=00000:

[killdevil-login2:08851] *** Process received signal ***

[killdevil-login2:08851] Signal: Aborted (6)

[killdevil-login2:08851] Signal code:  (-6)

[killdevil-login2:08851] [ 0] /lib64/libpthread.so.0 [0x3e10a0eca0]

[killdevil-login2:08851] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x3e0fa2ffc5]

[killdevil-login2:08851] [ 2] /lib64/libc.so.6(abort+0x110) [0x3e0fa31a70]

[killdevil-login2:08851] [ 3] ./main3d(_ZN6SAMRAI4tbox10SAMRAI_MPI5abortEv+0x4e) [0xc3629e]

[killdevil-login2:08851] [ 4] ./main3d(_ZN5IBAMR10IBFEMethod14getFromRestartEv+0x5b5) [0x7425b5]

[killdevil-login2:08851] [ 5] ./main3d(_ZN5IBAMR10IBFEMethod17commonConstructorERKSsN6SAMRAI4tbox7PointerINS4_8DatabaseEEERKSt6vectorIPN7libMesh4MeshESaISB_EEibS2_j+0xa3a) [0x75a80a]

[killdevil-login2:08851] [ 6] ./main3d(_ZN5IBAMR10IBFEMethodC1ERKSsN6SAMRAI4tbox7PointerINS4_8DatabaseEEEPN7libMesh4MeshEibS2_j+0x467) [0x75ceb7]

[killdevil-login2:08851] [ 7] ./main3d(main+0x9ed) [0x60793d]

[killdevil-login2:08851] [ 8] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3e0fa1d9f4]

[killdevil-login2:08851] [ 9] ./main3d(_ZNK7libMesh10SerialMesh13pid_nodes_endEt+0x71) [0x525579]

[killdevil-login2:08851] *** End of error message ***

Aborted

Looking inside the restart directory I see

libmesh_data_part_0.000002.xdr libmesh_data_part_0.000004.xdr libmesh_data_part_0.000005.xdr restore.000002 restore.000004 restore.000005

Does that look right? The three differences between my code and the example's code is:

1. Setting the restart read variables

        const string restart_read_dirname = app_initializer->getRestartReadDirectory();

        const int restart_restore_num = app_initializer->getRestartRestoreNumber();

        const bool is_from_restart= app_initializer->isFromRestart();

if (is_from_restart)

        {

            pout << "\n\n Starting from a restart...\n\n";

            }

2. Setting up IBFE from restart

        Pointer<IBFEMethod> ib_method_ops = new IBFEMethod(

                                                           "IBFEMethod", app_initializer->getComponentDatabase("IBFEMethod"), &mesh, app_initializer->getComponentDatabase("GriddingAlgorithm")->getInteger("max_levels"), is_from_restart, restart_read_dirname, restart_restore_num);

3. Writing IBFE at the restart step

            if (dump_restart_data && (iteration_num%restart_dump_interval == 0 || last_step))

            {

                pout << "\nWriting restart files...\n\n";

                RestartManager::getManager()->writeRestartFile(restart_dump_dirname, iteration_num);

                ib_method_ops->writeFEDataToRestartFile(restart_dump_dirname, iteration_num);

            }

[Alexander Hoover]

You are correct, the IBFE examples are not set up for restart currently. If you want to make a PR that adds in support, that would be great. :-)

Happy to do so. Is there a proper head/compare branch selection (... I assume PR=Pull Request... I've never done this)?

--

You received this message because you are subscribed to the Google Groups "IBAMR Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ibamr-users+unsubscribe@googlegroups.com.

[Alexander Hoover]

Looking at the line and testing it out, it seems that it depends on what I set as the restart directory. If I do

./main3d input3d $PWD/restart_IB3d 4

I get the previous error. If I instead do

./main3d input3d /restart_IB3d 4

There's no error, but the code ignores the restart input (the boolean associated with it is 0), so everything is read from the input.

[Boyce Griffith]

If you haven't already done so, please try to replicate with an IBAMR example and I will take a look.

--

[Boyce Griffith]

OK, so the only thing that is different here is that the absolute path is required for the restart file location?

On Oct 21, 2018, at 9:51 AM, Yan <yansha...@gmail.com> wrote:

Dear Prof. Griffith,

The same error came to me when I was running a case based on an IBFE example.

[0] ../src/utils/xdr_cxx.C, line 177, compiled Apr 28 2018 at 19:55:12

*********************************************************************

READING THE FILE "/libmesh_data_part_0.000000.xdr" FAILED.

It is possible this file contains infinite element information,

but the version string does not contain " with infinite elements"

Let's try this again, but looking for infinite element information...

*********************************************************************

*********************************************************************

Well, at least we tried!

Good Luck!!

*********************************************************************

[0] ../src/utils/xdr_cxx.C, line 177, compiled Apr 28 2018 at 19:55:12.

A Solution works with me:

First, modify the IBFEMethod initiation part:

Pointer<IBFEMethod> ib_method_ops =

            new IBFEMethod("IBFEMethod",

                           app_initializer->getComponentDatabase("IBFEMethod"),

                           &mesh,

                           app_initializer->getComponentDatabase("GriddingAlgorithm")->getInteger("max_levels"),

                           dump_restart_data,// bool register_for_restart

                           restart_dump_dirname,//const std::string& restart_read_dirname

                           restart_dump_interval);//unsigned int restart_restore_number

Then tell it to write IBFEMethod file

            if (dump_restart_data && (iteration_num % restart_dump_interval == 0 || last_step))

            {

                pout << "\nWriting restart files...\n\n";

                RestartManager::getManager()->writeRestartFile(restart_dump_dirname, iteration_num);

                ib_method_ops->writeFEDataToRestartFile(restart_dump_dirname, iteration_num);//write restart file for IBFEMethod

            }

run the program with absolute path of the restart_file, eg

./main3D input3d $PWD/restart_IB3d 1

1 is the time_interval

[Yan Shang]

Sorry for the interruption. 

./main3D input3d restart_IB3d 1

works as well.

I developed my case from examples/IBFE/explicit/ex7 but it was not set for restarting. And I checked ex0, it was not either. So I thought it was not solved. But then I found it was fixed in ex1. So I deleted my post.

Yan

[Boyce Griffith]

Oh, I see.

So just to be clear, restart works in ex1 but not in the other IBFE examples; is that right?

[Yan Shang]

I have not tried every example in IBFE. But from the code I guess, ex1 and ex2 should work and the others do not.