Simulation crash after finish

[tanktoo]

Hello,

I have got a problem. My simulation works fine but if I enable the writing of some Simulation values with "recordScalar" the simulation crashes at the moment it reaches sim time limit and nothing is written to the "*.sca" files.

Maybe I have I have to consider something if using recordScalar.
I hope you got an idea.

With greetings

tanktoo

[tanktoo]

It seems that not a single finish method or destructor is called as I have printouts in there.

** Event #880161792   T=33808.40186516608   Elapsed: 2160.077s (36m 00s)  93% completed   ev/sec=431090

Simulation terminated with exit code: -1073741819

This is all at the crash...
I really dont know where to start. If I start a shorter simulation everything works fine and the long one is too long to debug...

[HAEFLINGER, PATRICK (PATRICK)]

Did you try to launch the « long » simulation with the debugger and observe where your simulation is crashing  ?

 

Patrick

 

De : omn...@googlegroups.com [mailto:omn...@googlegroups.com] De la part de tanktoo
Envoyé : jeudi 7 février 2013 10:49
À : omn...@googlegroups.com
Objet : [Omnetpp-l] Re: Simulation crash after finish

--
--
Sent from the OMNeT++ mailing list. To configure your membership,
visit http://groups.google.com/group/omnetpp
 
---
You received this message because you are subscribed to the Google Groups "omnetpp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to omnetpp+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[tanktoo]

I tried a day ago with graphical userinterface and debug. It worked fine...
I really dont know why this happens. I delete every message with cancelAndDelete now. So I think this can´t be the reason.

[tanktoo]

Again started with debug in graphics mode. This time resulting in segfault.
The printout:

Thread [1] (Suspended: Signal 'SIGSEGV' received. Description: Segmentation fault.)    
    58 ntdll!LdrQueryProcessModuleInformation()  0x776c15de    
    57 ntdll!LdrQueryProcessModuleInformation()  0x776c15de    
    56 ntdll!LdrFindResource_U()  0x776b014e    
    55 <symbol is not available> 0x0028bd18    
    54 MessageStore::deleteExpiredMessages() MessageStore.cc:43 0x00402b10    
    53 BundleUDPTCP::finish() BundleUDPTCP.cc:123 0x00407244    
    52 cModule::callFinish() cmodule.cc:1290 0x69a6d6ee    
    51 cModule::callFinish() cmodule.cc:1282 0x69a6d680    
    50 cModule::callFinish() cmodule.cc:1282 0x69a6d680    
    49 cModule::callFinish() cmodule.cc:1282 0x69a6d680    
    48 cSimulation::callFinish() csimulation.cc:442 0x69a8151b    
    47 Tkenv::finishSimulation() tkenv.cc:668 0x68243470    
    46 Tkenv::runSimulation() tkenv.cc:451 0x68242a8d    
    45 run_cmd() tkcmd.cc:447 0x68249a8b    
    44 tcl84!TclInvokeStringCommand()  0x00762340    
    43 tcl84!TclEvalObjvInternal()  0x0076372a    
    42 tcl84!TclCompEvalObj()  0x007929ab    
    41 tcl84!TclCompEvalObj()  0x00791968    
    40 tcl84!Tcl_EvalObjEx()  0x00764a22    
    39 tcl84!Tcl_IfObjCmd()  0x0076d411    
    38 tcl84!TclEvalObjvInternal()  0x0076372a    
    37 tcl84!TclCompEvalObj()  0x007929ab    
    36 tcl84!TclCompEvalObj()  0x00791968    
    35 tcl84!TclObjInterpProc()  0x007ce1dd    
    34 tcl84!TclEvalObjvInternal()  0x0076372a    
    33 tcl84!TclCompEvalObj()  0x007929ab    
    32 tcl84!TclCompEvalObj()  0x00791968    
    31 tcl84!TclObjInterpProc()  0x007ce1dd    
    30 tcl84!TclEvalObjvInternal()  0x0076372a    
    29 tcl84!TclCompEvalObj()  0x007929ab    
    28 tcl84!TclCompEvalObj()  0x00791968    
    27 tcl84!Tcl_EvalObjEx()  0x00764a22    
    26 tk84!TkInvokeButton()  0x1024cc68    
    25 tk84!Tk_RaiseObjCmd()  0x1024bbf9    
    24 tcl84!TclEvalObjvInternal()  0x0076372a    
    23 tcl84!Tcl_EvalObjv()  0x0076396b    
    22 tcl84!Tcl_EvalObjEx()  0x007648c0    
    21 tcl84!Tcl_UplevelObjCmd()  0x007cd8ce    
    20 tcl84!TclEvalObjvInternal()  0x0076372a    
    19 tcl84!TclCompEvalObj()  0x007929ab    
    18 tcl84!TclCompEvalObj()  0x00791968    
    17 tcl84!TclObjInterpProc()  0x007ce1dd    
    16 tcl84!TclEvalObjvInternal()  0x0076372a    
    15 tcl84!Tcl_EvalEx()  0x00764261    
    14 tcl84!Tcl_Eval()  0x00764707    
    13 tcl84!Tcl_GlobalEval()  0x007661ac    
    12 tk84!Tk_BindObjCmd()  0x102461f7    
    11 tk84!TkBindEventProc()  0x1026ad8e    
    10 tk84!Tk_IdToWindow()  0x10278788    
    9 tk84!TkQueueEventForAllChildren()  0x10278d28    
    8 tcl84!Tcl_ServiceEvent()  0x007c22ba    
    7 tcl84!Tcl_DoOneEvent()  0x007c2659    
    6 runTk() tklib.cc:111 0x68253a71    
    5 Tkenv::run() tkenv.cc:274 0x6824215d    
    4 EnvirBase::run() envirbase.cc:245 0x00c44495    
    3 setupUserInterface() startup.cc:235 0x00c42824    
    2 evMain() evmain.cc:37 0x00c42d64    
    1 main() main.cc:27 0x0041088c    

I think then the error should within "deleteExpiredMessages()".

int MessageStore::deleteExpiredMessages(){
    int counter = 0;
    IdBundleMessageMap::iterator it;
    for(it = storedMessages.begin(); it != storedMessages.end(); ++it){
        BundleMessage *msg = storedMessages.at(it->first);
        if(msg->getExpireTime() < simTime()){
            //TODO changed delete
            cancelAndDelete(msg);
//            delete msg;

            storedMessages.erase(it);
            counter++;
        }
    }
    savedMessagesVec.record(storedMessages.size());
    return counter;
}
Mabe someone sees what could be wrong here?

[Daniel Pfefferkorn]

It looks like the answer must lie at call level 54 @line 43 in
MessageStore.cc

Are you sure that all of the message pointers you store still point to
valid message objects? Maybe you could add a trap for this kind of error.

Without the code it is just wild guessing.

--Daniel

> BundleMessage*msg =storedMessages.at(it->first);

> if(msg->getExpireTime()<simTime()){
> //TODO changed delete
> cancelAndDelete(msg);
> // delete msg;
>
> storedMessages.erase(it);
> counter++;
> }
> }
> savedMessagesVec.record(storedMessages.size());
> returncounter;
> }
> Mabesomeone sees what could bewrong here?
> |
>
>

> --
> --
> Sent from the OMNeT++ mailing list. To configure your membership,
> visit http://groups.google.com/group/omnetpp
>
> ---
> You received this message because you are subscribed to the Google
> Groups "omnetpp" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to omnetpp+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--
M.Sc. Daniel Pfefferkorn

Institute of Microelectronic Systems

Leibniz University Hannover phone: +49 (511) 762 - 19609
Appelstr. 4 fax: +49 (511) 762 - 19601
30167 Hannover e-mail: pfef...@ims.uni-hannover.de
Germany

[tanktoo]

Hello again,

we spent the last hours and checked every finish method and destructors. There should be no problem.
I have another idea. May it be possible that the simulation crashes because I save a lot of generated messages in maps and evaluate them at the finish call?

[Rudolf Hornig]

Firs tip: generally you should NOT use the finish() method for something that affects the behavior of the simulation i.e. freeing objects, messages etc. The reason is that finish() is called ONLY if the simulation has finished successfully.

Second: A C++ issue. You are iterating over the storedMessages map, but the iterator is incremented at the END of the loop, while you are deleting from the map in the middle of the loop: storedMessages.erase(it);

So at the end of the loop you are calling it++ on an invalid item.

Rudolf

[tanktoo]

Hello,

thanks for the tips. The method was stupid I know. Fixed it tonight.

The simulation still seems to crash if there are too many messages.

[tanktoo]

I did another try with debug. The short simulation I testet says it finished and output files are written.
But the debugger says segmentation fault with the following stack:

Thread [1] (Suspended: Signal 'SIGSEGV' received. Description: Segmentation fault.)

   
    9 tk84!TkFinalize()  0x102d320a    
    8 tk84!TkWinGetDrawableDC()  0x10225748    
    7 tk84!Tk_ScaleObjCmd()  0x102d7315    
    6 ntdll!RtlpNtMakeTemporaryKey()  0x776d9950    
    5 <symbol is not available> 0x10220000    
    4 ntdll!LdrVerifyImageMatchesChecksum()  0x776ed6b2    
    3 tk84!Tk_ScaleObjCmd()  0x102d72c5    
    2 ntdll!TpSetPoolStackInformation()  0x776ed554    
    1 <symbol is not available> 0x00000000    

[tanktoo]

After a long time of debugging unter Linux I get the following segmentation fault at 78% of the simulation being completed:

Thread [1] (Suspended: Signal 'SIGSEGV' received. Description: Segmentation fault.)

   
    18 cObject::take() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/sim/cobject.cc:95 0x00007ffff5ec1167    
    17 cPacket::_detachEncapMsg() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/sim/cmessage.cc:570 0x00007ffff5eae179    
    16 cPacket::getEncapsulatedPacket() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/sim/cmessage.cc:649 0x00007ffff5eae76e    
    15 IPFragBuf::addFragment() /home/tanktoo/Arbeit/svn/omnetpp/bundle_implementation/src/inetmanet-inetmanet-7b54fe1/src/networklayer/ipv4/IPFragBuf.cc:80 0x00007ffff7529660    
    14 IP::reassembleAndDeliver() /home/tanktoo/Arbeit/svn/omnetpp/bundle_implementation/src/inetmanet-inetmanet-7b54fe1/src/networklayer/ipv4/IP.cc:594 0x00007ffff751bcba    
    13 IP::routePacket() /home/tanktoo/Arbeit/svn/omnetpp/bundle_implementation/src/inetmanet-inetmanet-7b54fe1/src/networklayer/ipv4/IP.cc:328 0x00007ffff751a30d    
    12 IP::handlePacketFromNetwork() /home/tanktoo/Arbeit/svn/omnetpp/bundle_implementation/src/inetmanet-inetmanet-7b54fe1/src/networklayer/ipv4/IP.cc:200 0x00007ffff751993f    
    11 IP::endService() /home/tanktoo/Arbeit/svn/omnetpp/bundle_implementation/src/inetmanet-inetmanet-7b54fe1/src/networklayer/ipv4/IP.cc:108 0x00007ffff75192df    
    10 AbstractQueue::doEndService() /home/tanktoo/Arbeit/svn/omnetpp/bundle_implementation/src/inetmanet-inetmanet-7b54fe1/src/base/AbstractQueue.cc:75 0x00007ffff73c227b    
    9 AbstractQueue::doStartService() /home/tanktoo/Arbeit/svn/omnetpp/bundle_implementation/src/inetmanet-inetmanet-7b54fe1/src/base/AbstractQueue.cc:70 0x00007ffff73c2240    
    8 AbstractQueue::handleMessage() /home/tanktoo/Arbeit/svn/omnetpp/bundle_implementation/src/inetmanet-inetmanet-7b54fe1/src/base/AbstractQueue.cc:54 0x00007ffff73c215a    
    7 cSimulation::doOneEvent() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/sim/csimulation.cc:648 0x00007ffff5edac35    
    6 Cmdenv::simulate() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/cmdenv/cmdenv.cc:408 0x00007ffff6209351    
    5 Cmdenv::run() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/cmdenv/cmdenv.cc:275 0x00007ffff6208d75    
    4 EnvirBase::run() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/envir/envirbase.cc:245 0x00007ffff6736b83    
    3 setupUserInterface() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/envir/startup.cc:235 0x00007ffff67340e4    
    2 evMain() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/envir/evmain.cc:37 0x00007ffff67350d9    
    1 main() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/envir/main.cc:27 0x000000000043babc    

As it seems not my modules are responsible. But what to do now?

[Alfonso Ariza Quintana]

I have my suspect that you are working with an old version of inetmanet,

--

[tanktoo]

How to find out the version number?

[tanktoo]

Changed to newest INET Framework. Seems to work fine. But the problem with bigger packetsize is still existing:

<!> Error in module (IPv4) realWorld.host[7].networkLayer.ip (id=162) at event #17941, t=20.034582583249: (IPv4Datagram)BundleMessage-frag: decapsulate(): packet length is smaller than encapsulated packet.

make: *** [r3] Segmentation fault

This happens with a 75000 Byte packet and DSDV as routing.

[tanktoo]

After another 8 hours of debugging I got the segmentation fault again:

Thread [1] (Suspended: Signal 'SIGSEGV' received. Description: Segmentation fault.)

   
    17 cObject::take() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/sim/cobject.cc:95 0x00007ffff5e10167    
    16 cPacket::_detachEncapMsg() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/sim/cmessage.cc:570 0x00007ffff5dfd179    
    15 cPacket::getEncapsulatedPacket() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/sim/cmessage.cc:649 0x00007ffff5dfd76e    
    14 IPv4FragBuf::addFragment() /home/tanktoo/Arbeit/svn/omnetpp/omnetpp/bundle_implementation/src/inet/src/networklayer/ipv4/IPv4FragBuf.cc:86 0x00007ffff74d4132    
    13 IPv4::reassembleAndDeliver() /home/tanktoo/Arbeit/svn/omnetpp/omnetpp/bundle_implementation/src/inet/src/networklayer/ipv4/IPv4.cc:563 0x00007ffff74cda52    
    12 IPv4::handlePacketFromNetwork() /home/tanktoo/Arbeit/svn/omnetpp/omnetpp/bundle_implementation/src/inet/src/networklayer/ipv4/IPv4.cc:198 0x00007ffff74cba22    
    11 IPv4::endService() /home/tanktoo/Arbeit/svn/omnetpp/omnetpp/bundle_implementation/src/inet/src/networklayer/ipv4/IPv4.cc:116 0x00007ffff74cb3ea    
    10 AbstractQueue::doEndService() /home/tanktoo/Arbeit/svn/omnetpp/omnetpp/bundle_implementation/src/inet/src/base/AbstractQueue.cc:76 0x00007ffff73b9757    
    9 AbstractQueue::doStartService() /home/tanktoo/Arbeit/svn/omnetpp/omnetpp/bundle_implementation/src/inet/src/base/AbstractQueue.cc:71 0x00007ffff73b971c    
    8 AbstractQueue::handleMessage() /home/tanktoo/Arbeit/svn/omnetpp/omnetpp/bundle_implementation/src/inet/src/base/AbstractQueue.cc:55 0x00007ffff73b9636    
    7 cSimulation::doOneEvent() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/sim/csimulation.cc:648 0x00007ffff5e29c35    
    6 Cmdenv::simulate() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/cmdenv/cmdenv.cc:408 0x00007ffff6158351    
    5 Cmdenv::run() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/cmdenv/cmdenv.cc:275 0x00007ffff6157d75    
    4 EnvirBase::run() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/envir/envirbase.cc:245 0x00007ffff6685b83    
    3 setupUserInterface() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/envir/startup.cc:235 0x00007ffff66830e4    
    2 evMain() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/envir/evmain.cc:37 0x00007ffff66840d9    
    1 main() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/envir/main.cc:27 0x0000000000440010    

But now I use the newest INET framework. So the error still seems to exist. Maybe anyone knows to fix it?

[Alfonso Ariza Quintana]

Ups I have seen in your previous messages that you are using a packet of 75000 bytes. This is forbidden, the maximum size in IP is 65000 (in reality 65528), and any application that tries to send packets bigger that this size can provoke a problem and I don’t know any applications that could create packets of these size. Usually the applications modules create packets with a maximum size of 1500 bytes to avoid fragmentation in the lower layers.

 

If there is an error in the IP the error is not delete packets with size bigger that the maximum.

--

[tanktoo]

Hello Alfonso,

ok good to know this. The last error with the segmentation fault I posted must have another reason. The maximum packet size there is 50.000bytes. So I don know where this error comes from. Somewhere if INET tries reassembling it crashes.

[Alfonso Ariza Quintana]

I can execute the simulations with frames of 30000B but the destination node won’t receive any frame, with 30000B the number of frames that are sent to the mac layer is bigger than the maximum queue size and some fragment will be lost and it will be impossible to rebuild the original packet, in the simulation the packet is deleted by the IP layer in the next node because it is impossible to rebuild.

--

[tanktoo]

My maximum packet size in simulation is 50000bytes. Thats works fine. And if there is too much data it should be deleted. Thats not the problem.
The problem is the segmentation fault. If he could not reassemble he should simply drop the fragments but not cause a segmentation fault. As I see the error is within


 cObject::take() /home/tanktoo/Arbeit/omnetpp-4.2.2/src/sim/cobject.cc:95 0x00007ffff5e10167

which is directly in Omnet and not in INET framework.