Retrieve Showing Server Files As 'Local Only'

[Ethan Miller]

Hello,

Our team uses Illuminated Cloud 2 daily and are in the processes of updating CI pipeline. For this, we change how we deploy our source. We used to deploy the source to scratch orgs by deploying the entire module using Illuminated Cloud 2. Now, we have a CI process that builds an Unlocked package with a namespace and then that package is installed into the scratch org.

I am still able to deploy classes and other metadata to the scratch org as usual using the illuminated cloud "Deploy Modified Metadata" button, but the retrieve command seems to think that a number metadata types are 'Local Only'. This includes all apex classes.

Even after making a change and deploying and hitting "update now", the files still do not show up as retrievable from the scratch org. Funny enough, I can use the "Compare with server" button which does a retrieve and that will succeed.

Doing a retrieve from a terminal works as well. As things stand, I can use Illuminate Cloud 2 to deploy, and CLI commands to retrieve which isn't ideal. This only became an issue after swapping to working with an installed unlocked namespaced package.

I tried turning on the "use deploy/retrieve/delete instead of push/pull" and all that seemed to do is make the "configure module" window show the package.xml file and make the subscription editable there. Even with the apex classes selected there, they do not show up in the retrieve. 

I hit the "update now" button to update the cached metadata and/or "rebuild caches and indices" between all of the things I did to try and fix this.

Info that might help someone help me...:
if I "show meta.xml" files they show up, so the project is metadata not source I would assume.
Project Structure:
 

I'll try and provide more info if needed.

Thanks!
Ethan

[Scott]

Hi, Ethan. I believe what you're seeing is due to a current artificial (and intentional as I'll explain below) limitation in IC2 around metadata from installed unlocked packages with a namespace where the retrieved metadata includes the namespace, but IC2 doesn't know anything about that namespace in the project which leads to any number of issues when working with that metadata.

We can confirm that to be the case one way or the other, and it's likely a good idea to do that before proceeding on any other assumptions. You'll need to add the following to Help > Diagnostic Tools > Debug Log Settings:

#com.illuminatedcloud.intellij.configCache.FilePropertiesCache

then refresh the list of server metadata using the Refresh toolbar button or the Update Now link. That is going to pretty much flood your idea.log file with information about the list of server metadata returned by the API and whether or not it's eligible to be managed in an IC2 project. What you're going to be looking for is a set of log entries in that file like the following:

2023-05-24 11:30:48,661 [ 101432]   FINE - #com.illuminatedcloud.intellij.configCache.FilePropertiesCache - listMetadata() returned 7 results.
2023-05-24 11:30:48,661 [ 101432]   FINE - #com.illuminatedcloud.intellij.configCache.FilePropertiesCache - Evaluating for inclusion:
{
  "createdById": "0058B000002KmXgQAK",
  "createdByName": "User User",
  "createdDate": 1684875552000,
  "fileName": "classes/DynamicLabelAccess.cls",
  "fullName": "DynamicLabelAccess",
  "id": "01p8B000003z30dQAA",
  "lastModifiedById": "0058B000002KmXgQAK",
  "lastModifiedByName": "User User",
  "lastModifiedDate": 1684875567000,
  "manageableState": "UNMANAGED",
  "type": "ApexClass"
}
2023-05-24 11:30:48,661 [ 101432]   FINE - #com.illuminatedcloud.intellij.configCache.FilePropertiesCache - Including DynamicLabelAccess.

...

Look for some of the specific metadata objects that you'd expect/hope to be there and see how they're reported, specifically the value for manageableState. My guess is that's going to show as INSTALLED_EDITABLE in which case you're definitely hitting the aforementioned limitation. To be perfectly candid, here's the comment in IC2 regarding that exact limitation:

// TODO: I think that technically anything that's INSTALLED_EDITABLE is...well...editable,
//  but that can create serious issues with, e.g., Apex source files because they come back
//  with namespaces in their filenames that must be interpreted properly. For now disallow

//  that and we'll see if anyone specifically asks for it.

and you would technically be the first "anyone specifically ask[ing] for it".

Let's start with confirmation of that assertion (one way or the other), and then we can discuss potential resolutions.

Regards,

Scott Wells

[Ethan Miller]

"manageableState": "INSTALLED_EDITABLE"

It is as you expected.

The Unlocked package is namespaced to keep the CI pipeline simple.
This package is actually a first generation managed package, but we work with it like it was 2nd gen and in scratch orgs and we version the code into bitbucket. When we release, we do a diff on the release and manually deploy the modified source to the 1st gen packaging/developer org using illuminated cloud 2 and package it there. I'm sharing this info to help you understand why I think we need an unmanaged package with a namespace.

Removing the namespace on the unmanaged package would fix this issue I think...but it would mean we'd have to run some sort of script to add/remove the namespace prefix from all or objects and apex to make it line up with the managed package when we go to release, which is not a great solution. Too bad the salesforce gen1 to gen2 migration feature is still in the developer preview stage.

I really appreciate the response and learned a bit about using the logs in idea too!

Thanks again!
Ethan

[Scott]

Okay, it's good to know that it's as-expected (if certainly not as-desired!).

I'm happy to take a look at making this work, but it's non-trivial (which is probably understatement) with an external namespace suddenly "leaking" into the local project and trying to make sure that things resolve properly, etc. Right now the only place that external namespaces come into play is in the offline symbol table, and that's structured explicitly for namespaced symbols. My guess is that a (safe) fix wouldn't be coming in any short timeframe.

Well, that's not entirely true...if you're using IntelliJ IDEA, you can set up a multi-module project that might solve this for you since each module can have its own namespace, and via inter-module dependencies, IC2 can resolve namespaced references across module boundaries as if they were package boundaries. If you set things up that way, all I'd need to be able to do is allow retrieval of INSTALLED_EDITABLE metadata into a module with the same namespace.

If that's something you're interested in investigating--and again, it pretty much depends on the use of IntelliJ IDEA...though there might be a workaround/hack for the other JetBrains IDEs...because of the use of multiple modules--let me know and I can explain how it might work further and provide a build of IC2 with the ability to retrieve INSTALLED_EDITABLE metadata into a module with a matching namespace.

Regards,

Scott Wells

[Ethan Miller]

I'd be more than willing to investigate this with you. The whole team exclusively uses IntelliJ IDEA. 

The project is currently set up with a module. Do you mean I'd need to break it out into multiple? Thanks again for the help! You rock!

Thanks!
Ethan

[Matthew Chevalier]

Any update on this feature?

I encountered this same issue.

Luckily the package I'm using has a non-namespaced version, so I'll pivot to that for the short term.

[Scott]

Matthew, I apologize but I'm not sure which aspect you're asking about as t his discussion ventured out into a few directions. I'll take a stab, though, and say that symbols from installed unlocked packages, even those with a namespace, should be rendered properly into the OST now so that they're part of the project's "API". The last known issue with should have been resolved in 2.2.7.5 such that @NamespaceAccessible symbols are also included in the OST stubs for installed packages with the same namespace as the current project:

https://illuminatedcloud.blogspot.com/2023/07/2275-release-notes.html

Again, apologies if that's not what you're asking, though. If not, can you please specify what you're looking for exactly?

Regards,

Scott Wells

[besrour walid]

is there an extension "Salesforce Package.xml Generator" in illuminated cloud 2 

[Scott]

There's not a separate extension for generating a package.xml file, but IC2 can use and maintain a package.xml file for your projects as a first-class feature as documented here:

https://bitbucket.org/RoseSilverSoftware/illuminatedcloud/wiki/User_Guide/Managing_Metadata.md

Regards,

Scott Wells