BadIndex

[Olly Witney]

Hi

I am getting problems with my Excel addin (using Office 2010), and it is essentially the same problem as https://groups.google.com/forum/#!msg/exceldna/FzkZz9giA4M/mTNkqLSPP1MJ 

Some users at my company are getting the issue, some are not. A reboot does fix it. I believe it is something to do with Windows Updates and patches and they are conflicting or causing issues with the registry for my addins. I am comparing 2 machines, one is loading fine, one isn't.

Working

EXCEL.EXE	5308	RegCreateKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	Desired Access: Maximum Allowed, Granted Access: None 0x0, Disposition: REG_CREATED_NEW_KEY
EXCEL.EXE	5308	RegSetInfoKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	KeySetInformationClass: KeySetHandleTagsInformation, Length: 0
EXCEL.EXE	5308	RegQueryKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	Query: HandleTags, HandleTags: 0x401
EXCEL.EXE	5308	RegCloseKey	HKCU\Software\Classes\ExcelAddin	SUCCESS
EXCEL.EXE	5308	RegOpenKey	HKCU\Software\Classes\ExcelAddin	REPARSE	Desired Access: Read/Write
EXCEL.EXE	5308	RegOpenKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	Desired Access: Read/Write
EXCEL.EXE	5308	RegSetInfoKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	KeySetInformationClass: KeySetHandleTagsInformation, Length: 0
EXCEL.EXE	5308	RegQueryKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	Query: Cached, SubKeys: 1, Values: 0
EXCEL.EXE	5308	RegQueryKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	Query: Cached, SubKeys: 1, Values: 0
EXCEL.EXE	5308	RegEnumKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	Index: 0, Name: CLSID
EXCEL.EXE	5308	RegQueryKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	Query: HandleTags, HandleTags: 0x401
EXCEL.EXE	5308	RegQueryKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	Query: HandleTags, HandleTags: 0x401
EXCEL.EXE	5308	RegCloseKey	HKCU\Software\Classes\ExcelAddin	SUCCESS
EXCEL.EXE	5308	RegOpenKey	HKCU\Software\Classes\ExcelAddin	REPARSE	Desired Access: Delete
EXCEL.EXE	5308	RegOpenKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	Desired Access: Delete
EXCEL.EXE	5308	RegSetInfoKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	KeySetInformationClass: KeySetHandleTagsInformation, Length: 0
EXCEL.EXE	5308	RegDeleteKey	HKCU\Software\Classes\ExcelAddin	SUCCESS
EXCEL.EXE	5308	RegCloseKey	HKCU\Software\Classes\ExcelAddin	SUCCESS

Non-Working

EXCEL.EXE	10984	RegCreateKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	Desired Access: Maximum Allowed, Granted Access: None 0x0, Disposition: REG_CREATED_NEW_KEY
EXCEL.EXE	10984	RegSetInfoKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	KeySetInformationClass: KeySetHandleTagsInformation, Length: 0
EXCEL.EXE	10984	RegQueryKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	Query: HandleTags, HandleTags: 0x401
EXCEL.EXE	10984	RegCloseKey	HKCU\Software\Classes\ExcelAddin	SUCCESS
EXCEL.EXE	10984	RegOpenKey	HKCU\Software\Classes\ExcelAddin	NAME NOT FOUND	Desired Access: Read
EXCEL.EXE	10984	RegOpenKey	HKCU\Software\Classes\ExcelAddin	NAME NOT FOUND	Desired Access: Read
EXCEL.EXE	10984	RegOpenKey	HKCU\Software\Classes\ExcelAddin	NAME NOT FOUND	Desired Access: Maximum Allowed
EXCEL.EXE	10984	RegOpenKey	HKCU\Software\Classes\ExcelAddin	NAME NOT FOUND	Desired Access: Maximum Allowed
EXCEL.EXE	10984	RegOpenKey	HKCU\Software\Classes\ExcelAddin	NAME NOT FOUND	Desired Access: Read
EXCEL.EXE	10984	RegOpenKey	HKCU\Software\Classes\ExcelAddin	NAME NOT FOUND	Desired Access: Read
EXCEL.EXE	10984	RegOpenKey	HKCU\Software\Classes\ExcelAddin	NAME NOT FOUND	Desired Access: Maximum Allowed
EXCEL.EXE	10984	RegOpenKey	HKCU\Software\Classes\ExcelAddin	NAME NOT FOUND	Desired Access: Maximum Allowed
EXCEL.EXE	10984	RegOpenKey	HKCU\Software\Classes\ExcelAddin	NAME NOT FOUND	Desired Access: Read
EXCEL.EXE	10984	RegOpenKey	HKCU\Software\Classes\ExcelAddin	NAME NOT FOUND	Desired Access: Read
EXCEL.EXE	10984	RegOpenKey	HKCU\Software\Classes\ExcelAddin	NAME NOT FOUND	Desired Access: Maximum Allowed
EXCEL.EXE	10984	RegOpenKey	HKCU\Software\Classes\ExcelAddin	NAME NOT FOUND	Desired Access: Maximum Allowed
EXCEL.EXE	10984	RegOpenKey	HKCU\Software\Classes\ExcelAddin	NAME NOT FOUND	Desired Access: Read
EXCEL.EXE	10984	RegOpenKey	HKCU\Software\Classes\ExcelAddin	NAME NOT FOUND	Desired Access: Read
EXCEL.EXE	10984	RegOpenKey	HKCU\Software\Classes\ExcelAddin	NAME NOT FOUND	Desired Access: Maximum Allowed
EXCEL.EXE	10984	RegOpenKey	HKCU\Software\Classes\ExcelAddin	NAME NOT FOUND	Desired Access: Maximum Allowed
EXCEL.EXE	10984	RegOpenKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	Desired Access: Read/Write
EXCEL.EXE	10984	RegSetInfoKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	KeySetInformationClass: KeySetHandleTagsInformation, Length: 0
EXCEL.EXE	10984	RegQueryKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	Query: Cached, SubKeys: 1, Values: 0
EXCEL.EXE	10984	RegQueryKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	Query: Cached, SubKeys: 1, Values: 0
EXCEL.EXE	10984	RegEnumKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	Index: 0, Name: CLSID
EXCEL.EXE	10984	RegQueryKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	Query: HandleTags, HandleTags: 0x401
EXCEL.EXE	10984	RegQueryKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	Query: HandleTags, HandleTags: 0x401
EXCEL.EXE	10984	RegCloseKey	HKCU\Software\Classes\ExcelAddin	SUCCESS
EXCEL.EXE	10984	RegOpenKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	Desired Access: Delete
EXCEL.EXE	10984	RegSetInfoKey	HKCU\Software\Classes\ExcelAddin	SUCCESS	KeySetInformationClass: KeySetHandleTagsInformation, Length: 0
EXCEL.EXE	10984	RegDeleteKey	HKCU\Software\Classes\ExcelAddin	SUCCESS
EXCEL.EXE	10984	RegCloseKey	HKCU\Software\Classes\ExcelAddin	SUCCESS

I have raised a request with our desktop team to see what is happening with Wnidows updates, as the issue only re-appears in the morning, suspecting something is running overnight. Has anyone managed to see what Windows process could be causing this?

The registry is showing REPARSE on the working machine.

TIA, Olly

[Olly Witney]

I also don't need to reboot. I just need to log off and back on again, refreshing my profile. I now get no errors and the registry is loading the key all OK.

[Randy Qian]

I'd love to know an answer to this question myself if you figure it out. I've opened up a thread at MSDN regarding this issue:

https://social.msdn.microsoft.com/Forums/vstudio/en-US/f3d5accf-d8dc-4407-9afb-9e4ba16e5a33/inconsistency-issue-when-writing-to-the-registry-using-c?forum=netfxbcl

There's one more thing I want to try which is to tell all my users to run Excel as Admin but that has its own issues. I suspect there's some bug with the consistency of the registry after windows updates. Hopefully we'll get to the bottom of this.

[Randy Qian]

P.S. The "REPARSE" result indicates that UAC virtualization has occurred:

http://blogs.msdn.com/b/patricka/archive/2010/11/30/troubleshooting-application-compatibility-issues-tools-tips-and-tricks.aspx

I'm going to try to replace all the string based registry writes done by ExcelDNA with more invariant code, I think there were a few places where I didn't do this the first time:

            RegistryKey rk = Registry.CurrentUser.CreateSubKey(@"Software\Microsoft\Office\Excel\Addins\" + progId, RegistryKeyPermissionCheck.ReadWriteSubTree);

            rk.SetValue("LoadBehavior", 0, RegistryValueKind.DWord);

            rk.SetValue("FriendlyName", friendlyName, RegistryValueKind.String);

            rk.SetValue("Description", description, RegistryValueKind.String);

I'll get back to you with results.

[Randy Qian]

I implemented registry writes using the Registry class method above. Instead of using a string registry path reference, it uses the static property of the class to write to the registry, and so far it's fixed things for all my users for 2 days. If it lasts a week, I'm willing to call it "case closed." I think either the default behavior of writing to a hive using a string reference or reading from it during invocation deviated from some default behavior for some reason.

[Olly Witney]

Hi

I downloaded the ExcelDNA source and went through step by step and found out in the ExcelRibbon.cs the using(ProgIdRegistration... is where it fails. I changed this to ProgIdUacRegistration and it seems to have resolved the issue. My only concern is that I have had to now implemented the ExcelDNA source within my solution directly instead of just using the built libraries.

I haven't compared what the differences between non-uac and uac methods use though to see what exactly it is thats failing. 

[Randy Qian]

That's the first move I tried, using "ProgIdUacRegistration" instead of "ProgIdRegistration" which worked for a few days but that wasn't a permanent fix for me. I don't believe the solution is to the "ClassesRoot" hive instead because I think the same virtualization inconsistency could still occur since there's quite a few places that the string "HKEY_CLASSES_ROOT\" could be referring to depending on what UAC elevation you have and whether you're a 32bit or 64bit app for instance; it seems like it's more important to ensure that reads and writes are properly virtualized (or unvirtualized) to the app that is trying to read or write.

Perhaps what's happening is COM elevations by default are changed and lowered by default for some reason maybe from a security patch. Maybe the writes done from the dll are getting higher security billing in some cases than what a default non-elevated COM object can see.

http://msdn.microsoft.com/en-us/library/bb530198.aspx#vistaregvirt_topic8

I think another avenue that could work would be to replace the COM objects with UAC elevated ones since they would presumably see the correct registry entries now:

http://msdn.microsoft.com/en-us/library/bb530198.aspx#vistaregvirt_topic8

but I hesitant to experiment since I'm extremely busy at the moment.

http://chrison.net/UACElevationInManagedCodeANETCOMComponentElevated.aspx

I think either of the fixes alone should make reads and writes consistent. I'm very tempted to just move everything to 64 bit so I can avoid this mess, however a critical component of my office's workflow, Bloomberg, hasn't made the shift yet.

[Randy Qian]

I misspoke, presumably moving to 64 bit would only solve the WOW aspect of registry virtualization. The UAC registry issues are separate.

[Randy Qian]

Just brainstorming (http://msdn.microsoft.com/en-us/library/bb530198.aspx#vistaregvirt_topic8):

Note that virtual value take precedence over global values when present. In the example above, even if the global store had value V3 under Key1, the value V3 from the virtual store will be returned to the caller. If the virtual value were to be deleted from the virtual store, then the reads would fallback to the global store. For example, if V3 were to be deleted from HKU\{SID}_Classes\VirtualStore\Machine\Software\Key1, and if HKLM\Software\Key1 had a value V3, then that value would be returned.

So perhaps for whatever reason, the registry writes are being done to the global store sometimes instead of the virtualized store which an unelevated COM object gets to see. That failure seems to occur with a "NAME NOT FOUND" error and I'm not sure if that's consistent with this theory.

[Govert van Drimmelen]

Hi Randy,

Thanks for keeping the thread updated with your investigations.

I'm surprised that the way the registry API is called might make a difference (changing it to the Registry.CurrentUser. style). I'll have to look through the .NET source to figure out how the calls to the WIN32 API differ.

I don't expect the .NET wrappers to be involved with (or even aware of) the virtualization.

One question I have is why so few Excel-DNA users have this problem. 

What are the setting on your computers / network that make this fail?

How would we configure a vanilla Windows 7 installation to exhibit the same behaviour?

In the last link you posted, I noticed this:

• By default Virtualization is disabled for certain Windows keys (and their subkeys) in the software hive like HKLM\Software\Classes; HKLM\Software\Microsoft\Windows;HKLM\Software\Microsoft\Windows NT

HKCR is as alias for HKLM\Software\Classes 

Perhaps you have enabled virtualization for HKLM\Software\Classes somehow. There are some instructions in the post for checking this.

Finally I can imagine some approaches to harden this part of Excel-DNA:

 - always read back from the registry after writing.

 - if the COM load fails when writing as admin, always try it again in the user hive (even though the admin write worked)

 - investigate the (extremely frustrating) ActivationContext APIs again.

So I'm watching the thread, I don't have much insight yet, and really appreciate your detective work and your posts here.

Cheers,

Govert

But please con

[Randy Qian]

Hi Govert,

Good to know you've found my investigations to be useful. There seems to be a useful system function to check whether virtualization is in effect from the token that was grabbed:

http://www.tech-archive.net/Archive/DotNet/microsoft.public.dotnet.languages.csharp/2007-06/msg00139.html

I suppose there isn't any firm guarantee any of those registry paths mentioned should stay virtualized despite how they are supposed to behave by default. The absence of virtualization for users who get this error would make me much more confident in my theory and the function could perhaps be useful in the coding of the hardening cases.

-Randy

[Randy Qian]

Here is another more detailed solution which might give more information:

http://stackoverflow.com/questions/14005081/how-to-detect-file-redirection-to-the-windows-virtualstore

I'm not sure if the virtualization status number in the prior post covers both UAC and WOW virtualization, sorry I don't have the time to do a truly deep dive.

These functions are the "blue-pill" for an app though. ;)

-Randy

[Randy Qian]

Hi Govert,

I still had a few users who got sporadic errors, so I'm continuing my investigations in my spare time. One possible lead is the impact of App-V on user registry settings:

https://madvirtualizer.wordpress.com/tag/hkey_classes_root/

But I don't have the time to initiate a long latency ping pong set of experiments with my IT services.

I've decided to  deploy a change which implements your second idea to write to try and write to the uac virtualized current user hive no matter what:

    internal class ComAddInRegistration : Registration

    {

...

        readonly string _sId;
...

            //First generate the user SID prefix:

            NTAccount acct = new NTAccount(SystemInformation.UserName);

            SecurityIdentifier sId = (SecurityIdentifier) acct.Translate(typeof(SecurityIdentifier));

            _sId = sId.ToString();

            string uacVirtualizedHiveKey = _sId + @"\Software\Microsoft\Office\Excel\Addins\" + _progId;

            //Commit value

            try

            {

                RegistryKey rk2 = Registry.Users.CreateSubKey(uacVirtualizedHiveKey, RegistryKeyPermissionCheck.ReadWriteSubTree);

                rk2.SetValue("LoadBehavior", 0, RegistryValueKind.DWord);

                rk2.SetValue("FriendlyName", friendlyName, RegistryValueKind.String);

                rk2.SetValue("Description", description, RegistryValueKind.String);

            }

            catch (Exception e)

            {

                Debug.Print(e.Message);

            }

               

        }

        protected override void Deregister()

        {

            // Remove Add-In registration from Excel

            Registry.CurrentUser.DeleteSubKey(@"Software\Microsoft\Office\Excel\Addins\" + _progId);

            try

            {

                Registry.Users.DeleteSubKey(_sId + @"\Software\Microsoft\Office\Excel\Addins\" + _progId);

            }

            catch (Exception e)

            {

                Debug.Print(e.Message);

            }

        }

    }

And:

    internal class ProgIdRegistration : Registration

    {

...

        readonly string _sId;

...

            //First generate the user SID prefix:

            NTAccount acct = new NTAccount(SystemInformation.UserName);

            SecurityIdentifier sId = (SecurityIdentifier)acct.Translate(typeof(SecurityIdentifier));

            _sId = sId.ToString();

            string uacVirtualizedHiveKey = _sId + @"\Software\Classes\" + _progId + @"CLSID";

            //Commit value

            try

            {

                RegistryKey rk2 = Registry.Users.CreateSubKey(uacVirtualizedHiveKey, RegistryKeyPermissionCheck.ReadWriteSubTree);

                rk2.SetValue(null, clsId.ToString("B").ToUpperInvariant(), RegistryValueKind.String);

            }

            catch (Exception e)

            {

                Debug.Print(e.Message);

            }

        protected override void Deregister()

        {

            // Deregister the ProgId for CLSIDFromProgID.

            Registry.CurrentUser.DeleteSubKeyTree(@"Software\Classes\" + _progId);

            try

            {

                Registry.Users.DeleteSubKeyTree(_sId + @"\Software\Classes\" + _progId);

            }

            catch (Exception e)

            {

                Debug.Print(e.Message);

            }

        }

   }

Good news is nothing breaks so far. :) I'll report back if things stay quiet for my remaining users who have issues.

-Randy

[Govert van Drimmelen]

Hi Randy,

Thank you for the update.

I am right to understand that you use the App-V platform? Is it used to deliver Excel or anything realted to Excel, or other applications only?

Please keep us updated on how the changes work out.

Regards,

Govert

---

From: exce...@googlegroups.com [exce...@googlegroups.com] on behalf of Randy Qian [rand...@gmail.com]
Sent: 13 February 2015 12:12 AM
To: exce...@googlegroups.com
Subject: [ExcelDna] Re: BadIndex

--
You received this message because you are subscribed to the Google Groups "Excel-DNA" group.
To unsubscribe from this group and stop receiving emails from it, send an email to exceldna+u...@googlegroups.com.
To post to this group, send email to exce...@googlegroups.com.
Visit this group at http://groups.google.com/group/exceldna.
For more options, visit https://groups.google.com/d/optout.

[Randy Qian]

I didn't install Excel myself, and I haven't asked my IT services yet about if App-V is used.

I stumbled upon that link in my search for how the uac virtualized current user hive is named, but it seems like it could potentially cause issues. It seems like other interactions with roaming profiles could be a possible source of such bugs too.

I'll keep you posted.

-Randy

To post to this group, send email to exc...@googlegroups.com.

[Randy Qian]

Here is an interesting link about some details of roaming profiles in managed environments:

http://blogs.msdn.com/b/oldnewthing/archive/2005/06/30/434209.aspx

The mechanism sounds complex as it's mentioned that caching is used.

I've even had the wrong profile load after I upgraded my home machine to an SSD, then logged into my user account TOO FAST. The machine is not with Active Directory and this seems to match others' experiences:

https://social.technet.microsoft.com/Forums/windows/en-US/5ec0b949-effa-4e30-ba09-dc948a4c7a8b/windows-7-starting-with-a-temporary-profile?forum=w7itprogeneral

The following article blames the specific error I got to mismatched registry values:

http://blogs.technet.com/b/deployment_field_notes/archive/2011/11/03/a-corrupted-profile-problem.aspx

My impression is that the login \ registry caching mechanism for Windows 7 is all sorts of fucked and that the registry has some genuine data hazards in certain exceptional circumstances.

My current theory is that if a user logs into their local machine and quickly opens Excel, reads and writes are just not referencing the same logical copy of the registry because of some synchronization issue with roaming profiles. The hope is that writing to the HKU\{SID}_Classes\... will address this because as mentioned here:

https://msdn.microsoft.com/en-us/library/bb530198.aspx#vistaregvirt_topic8

that hive is supposed to be non-roaming.

-Randy

[Randy Qian]

P.S.

In theory HKCU is just a pointer to the currently logged in user's {SID} subdirectory from the HKU directory.

It seems that a big difference is that the latter is non-roaming whereas the former is presumably roaming and subject to synchronization and caching mechanisms and policies.

[Randy Qian]

Here's another lead:

https://msdn.microsoft.com/en-us/library/windows/desktop/ms724894(v=vs.85).aspx

Remarks
The HKEY_CURRENT_USER key maps to the root of the current user's branch in theHKEY_USERS key. It is cached for all threads in a process. Therefore, this value does not change when another user's profile is loaded. RegOpenCurrentUser uses the thread's token to access the appropriate key, or the default if the profile is not loaded.

 

Could it be that in some cases either the COM invocation from the latebound Excel app or the key writing isn't taking place with the logged on user's profile because of some syncing issue at logon? If what they mean is that the symbolic association is cached, then writing to the HKU\{SID} paths should avoid such a hitch.

-Randy

[Govert van Drimmelen]

I've added an issue on GitHub to track this problem: https://github.com/Excel-DNA/ExcelDna/issues/31

I'm not planning to implement the suggested workaround for version 0.33.

-Govert

[Randy Qian]

Roger that, thanks for the update.

I'll just roll my own "exceldna.integration.dll" for now.

-Randy