I think I figured it out. This actually seems to be a feature of Jenkins and not a bug in my code.
After looking trough Jenkins code I found in textbox.jelly the following.
<j:if test="${customizedFields != null and attrs.field != null and value != default}">
<j:mute>${customizedFields.add(name)}</j:mute>
</j:if>
So it seems that this is normal, and this is to show that the data has been changed and it's no longer the default value.
I was actually thinking otherwise that it's a bug in my code as I haven't seen this "modified" icon in any other plugin that has an "advanced" section, that I've tested.
If anyone can please confirm my above findings are true it would be great.
Thanks.