The issue here is that the latex property passed into setExpression has to use LaTeX syntax, and in LaTeX, curly braces need to be escaped with backslashes in order to be printed (otherwise, they function as grouping operators and are not printed).
If you write expressions in the calculator, and then copy-paste them, they will paste as LaTeX. Doing this for the expression you asked about gives
There's one more wrinkle though: in Javascript strings, backslashes must be escaped by an additional backslash, so if I want to use this in a call to setExpression, it should look like
calc.setExpression({ latex: 'y=\\left\\{0<x<3:x\\right\\}' });
I wish that the system gave better feedback about LaTeX syntax errors, and it's something I'd like to improve in the future, but it wouldn't actually help in this case (and many other cases where this issue comes up), because the string you're using is valid LaTeX; it just means something different than what you intended.