Can't run tests on new test class methods: Unexpected status code 403 returned

[Jon Wu]

Hi,

On one of my test classes, I can't seem to get new test methods I'm adding to run from IC. I'm clicking the arrow next to them in the UI to run them, but I've also verified that the run configs look right for them.

I tried updating IC (2.x) and IntelliJ (2018.1) to the latest, regenerating offline symbol tables and restarting, and invalidating caches and restarting.

I also tried manually recompiling all classes from Salesforce Setup web UI. I was able to run the new tests for this class manually, and even adding a test class like the following doesn't run:

@isTest static void newTestMethod() {}

I looked into idea.log and saw the following when I try to run the new tests:

2018-04-10 19:10:01,504 [13131723]   INFO - .ApexUnitTestRunProcessHandler - Running unit tests synchronously.

2018-04-10 19:10:02,910 [13133129]   WARN - .ApexUnitTestRunProcessHandler - Unexpected status code 403 returned.

Anything else I should try?

Thanks,

Jon

[Scott]

Jon, a 403 running tests generally means that the test you're trying to run hasn't been successfully deployed to the server. Can you verify that the method and/or enclosing test class are on the server?

Regards,

Scott

[Jon Wu]

Hi Scott,

That was my suspicion as well, which is why I pulled up the code in the UI and hit recompile. I also was able to run the (empty) test successfully from the web Developer Console, and the logs showed that it ran.

When I run the test in IC, the UI where it normally lists the tests, just shows:
- Terminated

  - TestClassName

It doesn't list the test I tried to run.

All arrows point to it not being there, but since I could run it from the Developer Console, it must be there. I'm stumped :(

Thanks,

Jon

[Scott]

Jon, a few other things to try:

1. Force the test to run using async by checking Illuminated Cloud>Configure Application>Unit Tests and Code Coverage>Always run unit tests asynchronously and see if you get a different result. As crazy as it sounds, sync vs. async can yield different results...sometimes behavioral and sometimes just more detailed messages.
2. If that doesn't help, enable debug logging for unit test execution as documented here, reproduce the issue, and send me the resulting idea.log.

Please let me know your findings either way.

Regards,

Scott

[Jon Wu]

Hi Scott,

I tried running the tests async and that didn't work either. I would definitely believe that with Salesforce 2 things that do the same thing, have different results!

Upon enabling more logging, I see the issue is that IC is just submitting nothing to the API. Here's an example:

2018-04-11 12:18:21,016 [ 455443]  DEBUG - .ApexUnitTestRunProcessHandler - Creating a trace flag.

2018-04-11 12:18:22,365 [ 456792]   INFO - .ApexUnitTestRunProcessHandler - Running unit tests synchronously.

2018-04-11 12:18:22,365 [ 456792]  DEBUG - .ApexUnitTestRunProcessHandler - Starting the test run.

2018-04-11 12:18:22,366 [ 456793]  DEBUG - .ApexUnitTestRunProcessHandler - Posting the following to runTestsSynchronous:

2018-04-11 12:18:22,366 [ 456793]  DEBUG - .ApexUnitTestRunProcessHandler -   {"tests":[]}

2018-04-11 12:18:22,837 [ 457264]   WARN - .ApexUnitTestRunProcessHandler - Unexpected status code 403 returned.

2018-04-11 12:18:22,838 [ 457265]  DEBUG - .ApexUnitTestRunProcessHandler - Finishing test run.

I force rebuilt the indices again and restarted to see if that would untangle anything, but no such luck here.

Thanks,

Jon

[Scott]

Well that's just strange. Is it doing that across the board or for very specific test classes/methods? Any other errors/stack traces in the log around that? When happens if you use Run>Edit configurations and open the properties of that run configuration? Do you see any classes/methods selected there?

Regards,

Scott

[Jon Wu]

I tried adding an empty test method to a few other classes I have, and I was surprised to see that it ran fine on some, but not on others.

It looks like there's something that finds the test methods in the test class, and it's not always finding the new method I'm adding. For example, given a test class called something like TestClass, I had one existing test method and then I added a method called zTesting: "@isTest static void zTesting() {}".

The problem seems to be class-specific. So far in a few tries, I've added tests with a few names to a few classes. Classes that work with the new dummy methods seem to work with all the new methods (and existing test methods). Classes that don't work with new dummy methods, consistently don't show them.

2018-04-11 16:19:57,147 [14951574]  DEBUG - .ApexUnitTestRunProcessHandler - Running tests for [ developer: { TestClass } ].

2018-04-11 16:19:57,212 [14951639]  DEBUG - .ApexUnitTestRunProcessHandler - Starting test run.

2018-04-11 16:19:57,212 [14951639]  DEBUG - .ApexUnitTestRunProcessHandler - Finding class declaration for TestClass.

2018-04-11 16:19:57,217 [14951644]  DEBUG - .ApexUnitTestRunProcessHandler - Finding test methods in TestClass.

2018-04-11 16:19:57,218 [14951645]  DEBUG - .ApexUnitTestRunProcessHandler -   existingTestMethod is a test method.

2018-04-11 16:19:57,218 [14951645]  DEBUG - .ApexUnitTestRunProcessHandler - Creating a trace flag.

2018-04-11 16:19:58,169 [14952596]   INFO - .ApexUnitTestRunProcessHandler - Running unit tests synchronously.

2018-04-11 16:19:58,169 [14952596]  DEBUG - .ApexUnitTestRunProcessHandler - Starting the test run.

2018-04-11 16:19:58,169 [14952596]  DEBUG - .ApexUnitTestRunProcessHandler - Posting the following to runTestsSynchronous:

2018-04-11 16:19:58,169 [14952596]  DEBUG - .ApexUnitTestRunProcessHandler -   {"tests":[]}

2018-04-11 16:19:58,583 [14953010]   WARN - .ApexUnitTestRunProcessHandler - Unexpected status code 403 returned.

2018-04-11 16:19:58,583 [14953010]  DEBUG - .ApexUnitTestRunProcessHandler - Finishing test run.

Above we see some logs for TestClass (renamed), which had an existing method called existingTestMethod (renamed). I added zTesting to it and tried to run only that test, but it doesn't show up under "Finding test methods in TestClass". Even when I run a single test, it seems like that piece of code is trying to enumerate all test methods, but it doesn't find my new methods in some classes for whatever reason.

When I go to edit the run configuration, everything looks normal and I see the one test zTesting checked. It seems like the bug is in the code that does "Finding test methods in ____".

Thanks,

Jon

[Scott]

Thanks for the research, Jon. Do you have a standalone reproducible example of this that you could share, ideally just a simple blank class and method that exhibit this behavior? That would be the quickest way for me to get to the bottom of this and produce a solution. If you can create one feel free to email it to me at sup...@illuminatedcloud.com.

Regards,

Scott

[Jon Wu]

Thanks for working with me on this. I'm still trying to figure out more details on this, as I'm not sure I can actually send you something that you can reproduce.

So far, I've tried a few more things with no luck:

- Uninstall the IC plugin, restart, reinstall

- Added empty test method to 10 or so other test classes to see if there was a pattern. I have mixed API versions from 33-42, but I wasn't able to find a correlation.

- Switch API versions of problematic test file

- Add multiple new tests with different names to a problematic test file

- Copy and paste a failing file into a new one. This works fine in the new test file with the exact same code (except that the name of the class is different of course).

My next step will be to test the same classes on a co-worker's machine to see if they have any issues or if the same classes fail to recognize added tests.

[Scott]

Oh, I'm definitely curious to know why this is happening as well. This is the first I've heard of it, but very rarely does a problem happen to just one person. I'd like to understand and resolve it. If you are able to characterize it, please let me know the characteristics and I'll hopefully be able to reproduce it on my side. Once I can do that, it should be as good as fixed.

Please keep me posted.

Regards,

Scott

[Jon Wu]

I found one other thing just now. Before I saw that the debug logging shows that you find all test methods in the class first, and the new methods weren't showing up. I tried deleting existing test methods and re-ran the new methods, and it still failed, but I noticed that the previously existing and working methods (now deleted) were still getting listed in the debug logs.

The code that logs "Finding test methods in [...]" seems to be looking at something stale. I would have expected that clearing cache and indices or rebuilding the offline symbol table would fix this, but is there some place I can manually nuke on the filesystem that would reset whatever this code is looking at? It seems that the state of these classes is just not updating for some reason.

[Scott]

Rebuilding indices and closing/reopening the project should be sufficient, but if you want to go for the full nuclear option, you can use File>Invalidate Caches / Restart which rebuilds ALL IDE caches/indices, not just IC's. I'll be very curious to know if that takes care of it.

[Jon Wu]

That's what I thought, and I've tried these steps in the past, but I've done it again just in case.

I regenerated offline symbol tables (full), then restarted, then I did File>Invalidate Caches / Restart.

Upon re-running the same file with the deleted previously existing tests, the logs still show that the code still sees the method that no longer exists:

2018-04-12 06:54:10,622 [ 111766]  DEBUG - .ApexUnitTestRunProcessHandler - Finding test methods in CaptureOrderCalloutTest.

2018-04-12 06:54:10,665 [ 111809]  DEBUG - .ApexUnitTestRunProcessHandler -   previouslyExistingButNowDeletedTest(renamed) is a test method.

It feels like there's some state that's outside the scope of all this nuking.

[Jon Wu]

Also FWIW my coworker had an issue that sounded a lot like this and maybe was the same, except the nuke worked for him a few weeks back.

[Scott]

I think you may have just cornered it...at least I hope so. Please go into Run>Edit Configurations and delete any existing unit test configurations that might have included this test method, then recreate the test config. That's the only place where a reference to the method would exist outside of the indices, and I bet there's a stale ref in there that's not being properly ignored/purged. Assuming that takes care of it, I know what I'll need to do on my end to keep this from happening to anyone else going forward.

[Jon Wu]

Unfortunately, that doesn't quite do the trick either. I had tried that earlier by deleting all the temporary / auto-created run configs, but I just deleted 100% of them and restarted and it still remembers my old test class name (captureOrdersTest), which no longer exists as you can see.

What's interesting is that in this view, your code does recognize the 2 new zTesting and zTesting1 methods.

[Scott]

Well darn. Okay, the only thing I can think of to do next is for me to instrument this area with some additional debug logging so we can see clearly where it's finding that zombie method. I'll see if I can add that today and post a test build you can use to help diagnose this further. So strange...

[Jon Wu]

Hi Scott,

Is there a build I can try out with extended logging to try and hunt this down?

Thanks,

Jon

[Scott]

Jon, I'm planning to release a new build tomorrow. I'll include the required debug logging additions in that build. I'll let you know when it's out there and hopefully it'll include more useful diagnostic info for us.

Regards,

Scott

[Scott]

Jon, I've just uploaded 2.0.1.7 with the additional debug logging. The existing debug logging config should be sufficient. Please just reproduce the issue and email me the resulting log. Hopefully it'll shed more light on the issue.

Regards,

Scott

[Szabolcs]

Hi,

I think I have the exact same issue as the one in this post. I have a test class with 2 existing methods, and when I added a third, it doesn't show up when running tests for the whole class. The class contains two other non-test helper methods. If I run just the test method itself, it shows up as terminated. Here is the log I got:

2020-02-14 09:56:35,255 [ 602814]  DEBUG - ij.coverage.ApexCoverageEngine - Creating a new coverage suite for name = '********************** Coverage Results' and connection name = '**********************'. 

2020-02-14 09:56:35,256 [ 602815]  DEBUG - lij.coverage.ApexCoverageSuite - Created an Apex coverage suite with project name = Procom and connection name = **********************(MDAPI). 

2020-02-14 09:56:35,301 [ 602860]  DEBUG - .ApexUnitTestRunProcessHandler - Starting test run. 

2020-02-14 09:56:35,301 [ 602860]  DEBUG - .ApexUnitTestRunProcessHandler - Finding class declaration for TestLeadTrigger. 

2020-02-14 09:56:35,302 [ 602861]  DEBUG - .ApexUnitTestRunProcessHandler - Finding test methods in TestLeadTrigger. 

2020-02-14 09:56:35,302 [ 602861]  DEBUG - .ApexUnitTestRunProcessHandler -   testConnectAccountInsert is a test method. 

2020-02-14 09:56:35,302 [ 602861]  DEBUG - .ApexUnitTestRunProcessHandler -   testConnectAccountUpdate is a test method. 

2020-02-14 09:56:35,302 [ 602861]  DEBUG - .ApexUnitTestRunProcessHandler -   getTestAccount is not a test method. 

2020-02-14 09:56:35,302 [ 602861]  DEBUG - .ApexUnitTestRunProcessHandler -   getTestLead is not a test method. 

2020-02-14 09:56:35,302 [ 602861]  DEBUG - .ApexUnitTestRunProcessHandler -   Found the following test methods for TestLeadTrigger: { TestLeadTrigger.testConnectAccountInsert()v, TestLeadTrigger.testConnectAccountUpdate()v }. 

2020-02-14 09:56:35,302 [ 602861]  DEBUG - .ApexUnitTestRunProcessHandler -   Filtering by the configured test methods based on simple class name TestLeadTrigger: { testOpportunityRecordTypeSet }. 

2020-02-14 09:56:35,302 [ 602861]  DEBUG - .ApexUnitTestRunProcessHandler -   testConnectAccountInsert is not a configured test method in test class TestLeadTrigger. 

2020-02-14 09:56:35,302 [ 602861]  DEBUG - .ApexUnitTestRunProcessHandler -   testConnectAccountUpdate is not a configured test method in test class TestLeadTrigger. 

2020-02-14 09:56:35,302 [ 602861]   WARN - .ApexUnitTestRunProcessHandler - All test methods for TestLeadTrigger were filtered. No tests will be run. 

2020-02-14 09:56:35,303 [ 602862]  DEBUG - .ApexUnitTestRunProcessHandler - Creating a trace flag. 

2020-02-14 09:56:35,995 [ 603554]   INFO - .ApexUnitTestRunProcessHandler - Running unit tests synchronously. 

2020-02-14 09:56:35,995 [ 603554]  DEBUG - .ApexUnitTestRunProcessHandler - Starting the test run skipping coverage. 

2020-02-14 09:56:35,995 [ 603554]  DEBUG - .ApexUnitTestRunProcessHandler - Posting the following to runTestsSynchronous: 

2020-02-14 09:56:35,995 [ 603554]  DEBUG - .ApexUnitTestRunProcessHandler -   {"tests":[],"skipCodeCoverage":true} 

2020-02-14 09:56:36,141 [ 603700]   WARN - .ApexUnitTestRunProcessHandler - Unexpected status code 403 returned. 

2020-02-14 09:56:36,141 [ 603700]  DEBUG - .ApexUnitTestRunProcessHandler - Finishing test run. 

2020-02-14 09:56:48,040 [ 615599]   INFO - rationStore.ComponentStoreImpl - Saving Module: 'dev'FacetManager took 17 ms 

2020-02-14 09:57:09,630 [ 637189]   INFO - j.ide.actions.RevealFileAction - 

Exit code 1 

Thank you in advance for your help.

Best regards,

Szabolcs

[Scott]

Do you happen to have a version of TestLeadTrigger in the offline symbol table? Perhaps another way of asking this is whether the problem is resolved if you regenerate your OST completely. Please let me know either way and I'll be happy to investigate further.

Regards,

Scott Wells