Doctrine is very nice, but DQL error reporting is quite poor

38 views
Skip to first unread message

Cerulean

unread,
Jul 22, 2020, 12:43:43 PM7/22/20
to doctrine-user
Hello everyone,

I'm new to learning Doctrine, and so far it's been really great! I love how easy it is to work with entities and have Doctrine map table data to php classes. I'm looking forward to learning more and seeing how Doctrine can improve my productivity. That said, there has been one really noticeable pain point: DQL error messages. When everything works correctly DQL is nice and convenient. But when DQL fails (or rather when I fail at writing DQL) the error messages are terrible. 

For really obvious mistakes, like when I'm accidentally missing a keyword like WHERE or FROM, the error message is basically useless. Sure I see ->syntaxError('end of string') in the exception trace, but there's no information regarding the location of the error. Where exactly is the unexpected end of the string?

For non-obvious mistakes, where the DQL syntax is correct, but there's a problem with what I'm asking Doctrine to do, the error messages improve, but they are still poor. I might see something like semanticalError('line 0, col 61...') that at least gives me an idea as to the location of the problem. But it would be much better if the error included the problematic detail directly. Consider the dead simple example "SELECT s.wrong FROM Submission s", where the "wrong" property doesn't exist for the Submission entity. Why can't DQL quote that "s.wrong" is the unknown path expression (or that "wrong" is the unknown property?) instead of making me interpret Doctrine parser backtraces and look up character offsets?

Those are all simple examples. Right now I'm trying to get a LEFT JOIN working and I have no idea what I'm doing wrong. I'm certain that it's my fault (working with databases isn't my regular job) but the DQL error messages sure aren't helping. It's quoting me a line/col number that puts the error inside the JOIN keyword; the "col" index is literally between the "I" and "N" characters! I expect this is due to the fact that my DQL string is multi-line. I assume my string is collapsed to a single line before execution, since the error location is always "line 0". Most likely Doctrine did not adjust its character index to account for the removed newlines. I can take that into account manually when I count characters, but this is just another example of poor error reporting wasting my time. I can give many more examples, and I've only recently started using DQL.

As I mentioned earlier, working with databases isn't what I do most of the time. But when I've worked with regular SQL the error messages were pretty good at helping me figure out what I did wrong. An SQL error like "unknown column 'wrong'" points you right to your mistake. I'd love to see Doctrine do the same. I think this would go a long way towards making Doctrine more usable and friendly.

I don't know if this is the best venue for this feedback. If there's a better way to communicate this issue to the Doctrine team please let me know. I just want to see the project improved, because so far I see that it's a very nice tool with a lot of benefits.

Thanks for listening!
~Martin

guilher...@gmail.com

unread,
Jul 22, 2020, 3:14:34 PM7/22/20
to doctrine-user
Hi,

One of the DQL parser authors here. Comments inline.

On Wed, Jul 22, 2020 at 12:43 PM Cerulean <ceru...@gmail.com> wrote:
>
> Hello everyone,
>
> I'm new to learning Doctrine, and so far it's been really great! I love how easy it is to work with entities and have Doctrine map table data to php classes. I'm looking forward to learning more and seeing how Doctrine can improve my productivity. That said, there has been one really noticeable pain point: DQL error messages. When everything works correctly DQL is nice and convenient. But when DQL fails (or rather when I fail at writing DQL) the error messages are terrible.
>
> For really obvious mistakes, like when I'm accidentally missing a keyword like WHERE or FROM, the error message is basically useless. Sure I see ->syntaxError('end of string') in the exception trace, but there's no information regarding the location of the error. Where exactly is the unexpected end of the string?

You actually have the column information in this scenario. Assuming:
SELECT s.wrong FROM Submission s s.wrong = :check (missing where here)
Look at the source code:
https://github.com/doctrine/orm/blob/master/lib/Doctrine/ORM/Query/Parser.php#L422-L435
It should trigger a QueryException to you something like: [Syntax
Error] line 0, col 34: Error: Expected end of string, got 's'.
The specific place where you got this in your stack trace is:
https://github.com/doctrine/orm/blob/master/lib/Doctrine/ORM/Query/Parser.php#L836

>
> For non-obvious mistakes, where the DQL syntax is correct, but there's a problem with what I'm asking Doctrine to do, the error messages improve, but they are still poor. I might see something like semanticalError('line 0, col 61...') that at least gives me an idea as to the location of the problem. But it would be much better if the error included the problematic detail directly. Consider the dead simple example "SELECT s.wrong FROM Submission s", where the "wrong" property doesn't exist for the Submission entity. Why can't DQL quote that "s.wrong" is the unknown path expression (or that "wrong" is the unknown property?) instead of making me interpret Doctrine parser backtraces and look up character offsets?
>

Now this gets interesting.
Let's talk about Parsers. The previous item you mentioned is what we
call as a syntax error. Correlating this to a language, like English,
is the same as you formulating a sentence without a verb. These errors
are easy to catch when implementing a DSL parser.
The error you just mentioned, "s.wrong" is the unknown path expression
because wrong is an unknown property, let's review the query from a
Parser perspective.

Analize tokens in this order:
* SELECT
* s
* .
* wrong
....

Can you tell me what "s" and "wrong" are? Is "s" valid? Is "wrong"
valid? Is "wrong" within the context of "s" valid?
Well, that's when it enters semantical errors. There are several ways
to do it. We did NONE of them, but mixed a few different ways
together.
In Compiler's theory, you were supposed to create stubs for parsing,
and then do a second pass verifying, once all the "code" is
syntactically verified. This means we'd have to parse the DQL twice.
Well, that is a waste of time in an interpreted language like PHP,
right? Exactly... so we cheated. We effectively create the proper
values, but we also schedule them for validation once the "code" is
completed. Verification happens as part of "processDeferred*" methods.
So, what does that do? It makes sure all the "s"s and "wrong"s are validated.

Getting back to your question: Why can't DQL quote that "s.wrong" is
the unknown path expression (or that "wrong" is the unknown property?)
instead of making me interpret Doctrine parser backtraces and look up
character offsets?

It does, and the specific method that verifies this is here:
https://github.com/doctrine/orm/blob/master/lib/Doctrine/ORM/Query/Parser.php#L709-L783
Reporting of the error happens through this method:
https://github.com/doctrine/orm/blob/master/lib/Doctrine/ORM/Query/Parser.php#L445-L470
Your QueryException would be something like: [Semantical error] line
0, col 61 near 's.wrong FROM': Error: Class Submission has no field or
association named 'wrong'.

> Those are all simple examples. Right now I'm trying to get a LEFT JOIN working and I have no idea what I'm doing wrong. I'm certain that it's my fault (working with databases isn't my regular job) but the DQL error messages sure aren't helping. It's quoting me a line/col number that puts the error inside the JOIN keyword; the "col" index is literally between the "I" and "N" characters! I expect this is due to the fact that my DQL string is multi-line. I assume my string is collapsed to a single line before execution, since the error location is always "line 0". Most likely Doctrine did not adjust its character index to account for the removed newlines. I can take that into account manually when I count characters, but this is just another example of poor error reporting wasting my time. I can give many more examples, and I've only recently started using DQL.
>
> As I mentioned earlier, working with databases isn't what I do most of the time. But when I've worked with regular SQL the error messages were pretty good at helping me figure out what I did wrong. An SQL error like "unknown column 'wrong'" points you right to your mistake. I'd love to see Doctrine do the same. I think this would go a long way towards making Doctrine more usable and friendly.
>
> I don't know if this is the best venue for this feedback. If there's a better way to communicate this issue to the Doctrine team please let me know. I just want to see the project improved, because so far I see that it's a very nice tool with a lot of benefits.
>

We're always open for contributions! The pointers I gave are the
actual code it runs to show you these messages.
I do feel however that something in your application is swallowing the
actual exception, and you only have visibility to the stack trace.

My recommendation is to traverse through your code and make sure the
errors are actually not being swallowed, and then proceed to looking
into the Parser class for a contribution! =)


Regards,

> Thanks for listening!
> ~Martin
>
> --
> You received this message because you are subscribed to the Google Groups "doctrine-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to doctrine-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/doctrine-user/4fa5e98a-1247-47b4-8acb-d5012a97ae91o%40googlegroups.com.



--
Guilherme Blanco
SVP Technology at Statflo Inc.
Mobile: +1 647 232 5599

Cerulean

unread,
Jul 23, 2020, 12:17:19 PM7/23/20
to doctrine-user
Thank you very much for the detailed reply Guilherme. I really appreciate you taking the time to explain things and help me!


> We effectively create the proper values, but we also schedule them for validation once the "code" is completed. Verification happens as part of "processDeferred*" methods. 

That makes complete sense to me. I've written a few parsers and interpreters over the years– small potatoes stuff, but I understand the different stages of validation involved.

> Your QueryException would be something like: [Semantical error] line 0, col 61 near 's.wrong FROM': Error: Class Submission has no field or association named 'wrong'. 

This is the part I'm just not seeing over here. Here's a full line from the exception backtrace:

#1 /home/name/subfolder/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(771): Doctrine\ORM\Query\Parser->semanticalError('line 0, col 9 n...', Array)

You can see the error message is truncated with ellipses just at the word "near". That's what I see from the error dump on the loaded page, and in the php error log file.

> I do feel however that something in your application is swallowing the actual exception, and you only have visibility to the stack trace. 

You are absolutely correct. If I wrap the erroneous DQL in a try/catch block and print the exception's message myself, I do see the full information. It shows everything I expected it should. That's wonderful and will really help reduce my frustration.

Do you have any idea how to prevent Doctrine's error messages from being truncated? I assume it's php's fault. I don't see any relevant php setting, aside from "log_errors_max_len", but I don't think that affects the stack backtrace. It's great to know this try/catch trick, but it would be more convenient if php just showed the crucial part of Doctrine's errors automatically.

> We're always open for contributions! 

Perhaps one day! I don't have the time for it now, but it would be cool and gratifying to work on a project as useful and technically interesting as Doctrine.

Thanks again for your help :)

Best,
~Martin Wierschin 

Christophe Coevoet

unread,
Jul 23, 2020, 12:32:51 PM7/23/20
to doctri...@googlegroups.com
Why are you trying to rely on the arguments in the backtrace (which are
indeed dumped in a compact form) to see the Doctrine error message ? The
argument passed to Parser::semanticalError gets set as the message of
the exception. If your error reporting renders the backtrace but drops
the error message, there is nothing we can do for you.
>
> > We're always open for contributions!
>
> Perhaps one day! I don't have the time for it now, but it would be
> cool and gratifying to work on a project as useful and technically
> interesting as Doctrine.
>
> Thanks again for your help :)
>
> Best,
> ~Martin Wierschin

--
Christophe | Stof

Cerulean

unread,
Jul 23, 2020, 12:39:52 PM7/23/20
to doctrine-user
Why are you trying to rely on the arguments in the backtrace (which are indeed dumped in a compact form) to see the Doctrine error message ?

Because that is the only part of the logged error that is specific to the problem. The full error looks like this:

PHP Fatal error:  Uncaught Doctrine\ORM\Query\QueryException: SELECT s.wrong
FROM Submission s
 in /home/user/folder/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:43
Stack trace: ...

As you can see, the top part of the logged error is just my bad DQL. Only the stack trace shows any interesting specifics from Doctrine.
 
If your error reporting renders the backtrace but drops the error message, there is nothing we can do for you.

So far as I know I haven't customized php or Doctrine error reporting in any way, but I'd be thankful to be told otherwise. Are there any settings I should adjust?

~Martin

Christophe Coevoet

unread,
Jul 23, 2020, 12:45:02 PM7/23/20
to doctri...@googlegroups.com
--
You received this message because you are subscribed to the Google Groups "doctrine-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to doctrine-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/doctrine-user/5f958fb6-e6f2-442a-b448-2bfe644f444fo%40googlegroups.com.

The exception containing the full DQL query as the message is not the exception thrown by Doctrine. It is the previous exception attached in the thrown exception. So it looks like your error reporting system dropped one layer of exception to report the inner one.

-- 
Christophe | Stof

Cerulean

unread,
Jul 23, 2020, 1:04:14 PM7/23/20
to doctrine-user

it looks like your error reporting system dropped one layer of exception to report the inner one 

Thanks for the explanation. Unfortunately I have no idea how to fix it. I'm in a shared hosting environment running php 7.3, and so far as I know I haven't customized anything that should affect this. I certainly haven't fiddled with how php handles exceptions.

What you described sounds unusual, and like it wouldn't be something php itself would do. I would expect any code that is doing that kind of exception unwrapping to be the frameworks/packages involved. But I'm using Doctrine directly, not as part of some larger tool like Symfony. And I definitely have not customized its error reporting or exception handling.

~Martin

Cerulean

unread,
Jul 23, 2020, 8:31:49 PM7/23/20
to doctrine-user
I just stumbled on the kind of error message that's apparently expected from Doctrine. Consider the following reduced and contrived DQL:

SELECT COUNT(s.id) FROM Submission s JOIN AppVersion v HAVING (v.app = :app)

Doctrine throws an exception that's logged as:

Fatal error:  Uncaught PDOException: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'a1_.app_id' in 'having clause' in ...

Other DQL statements are still producing exceptions that don't include specifics like the unknown column name.

~Martin
Reply all
Reply to author
Forward
0 new messages