Increasing memory usage with Protobuf

[Øyvind Bakken]

Hi there

We have an issue with increasing memory usage of an application using Protobuf. The application has been tested with Valgrind in Ubuntu to find the reason for this increase. The results shown with Massif Visualiser is shown below:

Here is a snapshot of the part of code highlighted by Valgrind:

The figure is suggesting that the issue is related to Protobuf library functions, and further studies of the source code have strongened these suspicions.

Here are the functions in ProtobufReceiver:

void ProtobufReceiver::runClient()

{

	string receivedData;

	while(!terminationRequested)

	{

		string tmpData;

		*socket >> tmpData;

		receivedData += tmpData;

		while(parseContent(receivedData));

	}

}

bool ProtobufReceiver::parseContent(string &receivedData)
{
    ....
    ....
    ....

    if(dataSize >= packetSize)
    {
        DataList *dl = new DataList();
        string dataString = receivedData.substr(packetSize-messageSize, messageSize);

        dl->ParseFromString(dataString);

        
        ...
        ...
       
        if(dataSize > packetSize)
        {
           retval = true;
           receivedData = receivedData.substr(packetSize);
        }
        else
        {
           receivedData.clear();
        }
    }

}

The DataList is the protobuf message, and even though this is not seen deleted here, it has been verified that every object is correctly deleted later in a getMessage() function.

The memory usage has been shown to increase up to values of around 1GB, before flattening out (increases at an almost negligible rate). At this stage, the program behaviour still seems normal.
No crashes have been observed because of this, but it would still be nice to figure out the reason for this behaviour. 

Also, a strange thing is that when running the program on a newer version of the HP EliteBook series, the issue seems to not appear. The program has been runned on a few other computers 
(mine is an EliteBook 8740w), on both Windows 7 and Ubuntu, with the same memory issues. I guess this could imply that the issue is HW dependant.

We are using Protobuf version 2.4.1.

Does anybody have a clue what's going on here? Any feedback would be appreciated.

Regards

[Joel Holveck]

Many parts of Protobuf will cache empty objects, in the expectation that they'll be reused.  For example, if you have a Message object with a repeated message field, and there are five entries in that field, when you later call .clear on the object, the five entries will be marked as unused but the objects will still exist.  That way, when the application needs to allocate the field again, it will be able to reuse the old object instead of having to allocate more memory.

I also see that the main source of size increase is a RepeatedPtrField, but the legend in your chart truncates the element type.  If I were troubleshooting this, I'd be looking at that most closely.