system_data is used for storing compatibility information in Joomla 2.5.28. See
https://github.com/joomla/joomla-cms/pull/5015However that function hasn't been implemented in Joomla 3.
Other than that both fields aren't used currently.
I'd say you're fine to use the custom_data, but I would not count on system_data never being used for anything.