Nested ordered lists DO work. The standard HTML default style for OL is to use decimal numbers at all levels.
However, in TiddlyWiki Classic, the StyleSheetLayout shadow tiddler defined the styles for ordered list this way:
ol {list-style-type:decimal;}
ol ol {list-style-type:lower-alpha;}
ol ol ol {list-style-type:lower-roman;}
ol ol ol ol {list-style-type:decimal;}
ol ol ol ol ol {list-style-type:lower-alpha;}
ol ol ol ol ol ol {list-style-type:lower-roman;}
ol ol ol ol ol ol ol {list-style-type:decimal;}
For TW5, paste the above into a tiddler tagged with $:/tags/Stylesheet, and you should then get the bullet styles you expect.
enjoy,
-e