There are a few reasons we don’t use XML syntax. First, RSP is designed to handle any kind of text files, such as CSS. Second, it requires a XML parser, and, worse, HTML is not 100% compatible with XML (see html5plus – it is why I forked it from html5lib). Also, XML is too complicated but less flexible. For example, it is hard to specify an expression in XML attributes with another XML tags.
Regarding extension, I don’t think namespace is a must. For example, you can prefix the custom tags with a special prefix. XML namespace is overkill to me -- without tool, it is hard to remember these namespaces and not easy to link a prefix back to a namespace (unless all docs follows the same rule).
On the other hand, the inability of loading Dart code at run time is actually the limited factor for customization. I don't find a good way to import custom code without knowing its existence in advance.