<GUID>.tmp.compiled "in use" error under concurrency

[ygrig...@gmail.com]

I have discussed this with Oleg before as relating to the caching, but still running into this issue (much improved now, however).
Wondering if there is a fix or workaround that Oleg or users can recommend.
Under concurrent use of the LoadMethod scenario, there can be an error like this

The process cannot access the file 'C:\WINDOWS\TEMP\CSSCRIPT\Cache\949632885\4bb50f69-f946-45a0-86b5-0a05693163a6.tmp.compiled' because it is being used by another process.

This tends to happen when the process is fresh and has not executed/cached the scripts and significant degree of concurrency takes place.

I understand that it can be related to the non-transactional nature of file system or possibly imperfection in GUID generation.

I am ready to introduce retrying where it can help, but any other approaches are much appreciated.

[Oleg Shilo]

You summed it up very well. The problem can be easily caused by the "non-transactional nature of file system or possibly imperfection in GUID generation". 

The only practical approach I can think of is to take over the assembly (compiled script) file name allocation and do it manually. This way you can guarantee the uniqueness of the name because you can use something as simple (and yet ultimately reliable) as a global counter. 

[ygrig...@gmail.com]

Thank you for your comment. I have a mix of LoadMethod and LoadCode calls. I can see now the overloads allowing assembly name for both (did not pay attention to them before).
Looks like it is would be possible to do exactly what you are describing by channeling both through static wrappers with a locked global counter.

[ygrig...@gmail.com]

Multi-process situation is also possible and likely however. Static counter won't work there. Back to the drawing board!

[ygrig...@gmail.com]

We have incorporated retries, it fixed these problems.
We currently use this function which others may find useful for the cs-script and other retry scenarios with varying error conditions:

        public static T RetryingCall<T>(Func<T> func, int retryTimes, int retryErrorWaitMilliseconds, IEnumerable<String> exceptionMessageContentToRetryOn)
        {
            bool nonRetryException = false;
            int attemptedTimes = 0;
            Exception toRethrow = null;

            do
            {
                try
                {
                    return func();
                }
                catch (MemberAccessException)
                {
                    throw;
                }
                catch (TargetException)
                {
                    throw;
                }
                catch (Exception ex)
                {
                    toRethrow = ex;
                    String exText = ex.ToString();
                    bool shouldRetry = exceptionMessageContentToRetryOn.Any(containedInException => exText.Contains(containedInException));
                    if (shouldRetry)
                    {
                        attemptedTimes++;
                        Thread.Sleep(retryErrorWaitMilliseconds);
                    }
                    else
                    {
                        nonRetryException = true;
                    }
                }

            } while ((nonRetryException == false) && (attemptedTimes <= retryTimes));

            throw toRethrow;
        }

to call, for example:

return RetryingCall<Assembly>(() => CSScript.LoadMethod(ScriptText), 5, 5000, scriptCompileRecoverableErrorIndicators);

where scriptCompileRecoverableErrorIndicators is a list of strings that are substrings of exception text bodies commonly encountered during compilation, currently

            "because it is being used by another process",
            "Could not find file",
            "is denied"

of course this must be used with care not to mask unrelated permanent problems with drive access.

Oleg, there is another one that occasionally comes out from LoadMethod: "The wait completed due to an abandoned mutex"
We can probably retry on this too, but I wanted to ask you if this is potentially something you would be interested in examining. We are on 3.5 SP1.

[Oleg Shilo]

Thank you for letting me know. 

After analyzing the code related to the mutex release I found that "LoadWithConfig" (in contrast to "CompileWithConfig") did not release mutex explicitly while still disposing it properly. This in fact can cause unpredictable synchronization behavior. 

The problem is fixed now and the HotFix release is published: http://csscript.net/LatestBuild.html.

As a generic comment...you should be aware about other opportunities while ensuring concurrency of the compiled script assembly.  

• CS-Script can load the script assembly as a byte array instead of a file. You just to set CSScript.GlobalSettings.InMemoryAsssembly to true and the script engine will read all bites from the assembly file and load the data without locking the file. Though this loading model can trigger some unexpected assembly probing behavior by CLR. That is why InMemoryAsssembly false by default. 
  
  
• Starting from v3.5.0 CS-Script allows script execution under the Mono "compiler as service". Thus there is no need for the temporary AppDomain as well as there is no need to the script assembly (all compilation and loading is done in memory). The feature is described in details here: http://www.csscript.net/Help/evaluator.html   

Cheers,.

Oleg

[ygrig...@gmail.com]

Thank you for the fix.
We indeed had problems with InMemoryAssembly set to true. I will need to go back and recall the exact issues. I do remember that, contrary to expectations, there were many more intermittent issues.
I think that retry approach helps the compilation phase most. Compilation phase appears most vulnerable to concurrency file system issues. And, as we discussed, there is not a way around that since there must be an underlying file for compilation.
Loading the assembly is also helped by retries. That phase also has issues under concurrency. I do need to reexamine the reasons why InMemoryAssembly appeared more fragile.

[Oleg Shilo]

> I do need to reexamine the reasons why InMemoryAssembly appeared more fragile...

Arguably the following reason is the the most serious one:

• Assembly probing can be severely affected as there is no more assembly file.
  Though can be addressed (to a some degree) by AppDomain.Current.ResolveAssumbly += <custom probing routine>

--
You received this message because you are subscribed to the Google Groups "CS-Script" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cs-script+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[ygrig...@gmail.com]

Oleg, the hotfix introduced an issue: C# compiler compains about "duplicate public modifier". It's not in the script.

[ygrig...@gmail.com]

Double checked the script, positive that there is no duplicate modifier in the script, also runs correctly with pre-hotfix version.
Oleg, am I missing something or indeed there might be an error in the hotfix?

[Oleg Shilo]

It is possible CS-S injects extra access modifiers. 

The best way to address it to provide the test case so I can reproduce the problem. 

Can you compose the simple "hello world" style sample?

[ygrig...@gmail.com]

I am unable to recreate the exact error in a hello world-like form,
I think the problem starts from my side. I believe it happens on erroneously formed scripts containing "using" statements and static methods but no namespace or class declaration. Like this:

            CSScript.CacheEnabled = true;
            CSScript.GlobalSettings.InMemoryAsssembly = false;
            CSScript.GlobalSettings.TargetFramework = "v3.5";
            CSScript.GlobalSettings.OptimisticConcurrencyModel = true;

            String script = @"using System; public static String Test(String toShow) { return ""passed "" + toShow; }";
            Assembly assembly = CSScript.LoadMethod(script);
            AsmHelper method = new AsmHelper(assembly);
            Console.WriteLine((String)method.Invoke("*.Test", "some string"));
            Console.ReadKey();

in this case, however, I get the error CS1518: Expected class, delegate, enum, interface, or struct

I will keep trying to recreate the "duplicate public modifier" error. The pre-hotfix dll runs the more complex case that follows this pattern fine but I do get CS1518 on this example from pre-hotfix as well.
The pattern is same, with more using statements, a cs-ref directive and more method parameters.

[Oleg Shilo]

Great. Thank you. With your code sample I was able to reproduce the problem (including the "duplicate public modifier" error). 

Quick work around

Just ensure you have line breaks between any "using ...;" and the actual method code. Also you will need to remove both "public" and "static" as they will be auto-injected anyway. Thus the sample you sent will work if you modify it as follows:

        String script = "using System;\r\n static public String Test(String toShow) { return \"passed \" + toShow; }";
 
        var method = new AsmHelper(CSScript.LoadMethod(script));
        Console.WriteLine(method.Invoke("*.Test", "some string"));

Cause

The problem with "duplicate public modifier" was caused by the unfortunate mistake in the CSScript.WrapMethodToAutoClass implementation, which led to the failure to detect presence of the "public" in the original code and the injection of the additional "public" to the code to be compiled. 

Fix

The defect has been fixed and the fixed (v3.5.6) is published as HotFix (http://csscript.net/LatestBuild.html). Also the fix is available from NuGet.

I have also prepared for you the sample demonstrating the work around and some hosting code improvements: https://dl.dropboxusercontent.com/u/2192462/ygrigoryev/Script.cs.

Thus the sample you sent me can be "collapsed" into:

        var Test = CSScript.LoadMethod("string Test(String toShow) { return \"passed \" + toShow; }")
                           .GetStaticMethod();
 
        Console.WriteLine(Test("some string"));

[ygrig...@gmail.com]

Thank you. Quick testing with latest hotfix fixes the previous problems. I will keep at it and test a wide script surface area to make sure it is good.
Also appreciate the examples. I have to admit that permitted abbreviated method grammar was always a black box to myself and team. There was understanding that some formal c# can be omitted and compensated by CS-Script but we were reluctant to try it out. This discussion clarifies much.

[Thias]

Am Dienstag, 27. November 2012 00:13:50 UTC+1 schrieb Oleg Shilo:

You summed it up very well. The problem can be easily caused by the "non-transactional nature of file system or possibly imperfection in GUID generation". 

Hi Oleg,

this is not true for the problem mentioned here. I also ran into the "file is in use by another process" issue and after digging around a  bit, I found that this is due to the use of static variables in the CScript lib.

The problem with the locked files is as follows:

In CSExecutor.Compile the assembly file name is assigned to a static variable options.forceOutputAssembly. In a multithreaded environment, another thread may overwrite this variable before the actual thread has run. This can easily reproduced by doing something like

            Parallel.For(0, 100, i => { var asm = GetAssembly(source); });

(GetAssembly is a helper that calls LoadCode...

This means that in a multi-threaded environment the programmer has to make sure that only one thread is calling Load... at a time; otherwise, undesired behaviour may occur (even if the compilation succeeds, other configuration parameters may have been changed by another thread. This is why the proposed retry-solution is not helping in my opinion).

It would be great if this limitaion would be removed in a future version of this great lib.

Kind regards

M.

[Oleg Shilo]

I will have a look and try to reproduce the problem.

I am not convinced yet that the problem is caused by static vars (though I am convinced the problem exists).

The all LoadCode calls just redirect to a single method LoadWithConfig, which is system-wide synched with Mutex and thread-protected with Lock:

        static public Assembly LoadWithConfig(string scriptFile, ...)
        {
            using (Mutex fileLock = new Mutex(false, GetCompilerLockName(assemblyFile, CSScript.GlobalSettings)))
            {
                lock (CSExecutor.options)
                {

I will give it another try nevertheless as apparently the synch measures do not always work as expected. 

As a side note the problem is a difficult one as I even once witnessed csc.exe (MS C# compiler) returning the control after the assembly was built but the assembly file still being locked for the short amount of time. 

Thank you for the feedback.

Oleg

On Fri, Jan 17, 2014 at 12:48 AM, Thias <matthias....@googlemail.com> wrote:

Am Dienstag, 27. November 2012 00:13:50 UTC+1 schrieb Oleg Shilo:

You summed it up very well. The problem can be easily caused by the "non-transactional nature of file system or possibly imperfection in GUID generation". 

Hi Oleg,

this is not true for the problem mentioned here. I also ran into the "file is in use by another process" issue and after digging around a  bit, I found that this is due to the use of 

The problem with the locked files is as follows:

In CSExecutor.Compile the assembly file name is assigned to a static variable options.forceOutputAssembly. In a multithreaded environment, another thread may overwrite this variable before the actual thread has run. This can easily reproduced by doing something like

            Parallel.For(0, 100, i => { var asm = GetAssembly(source); });

(GetAssembly is a helper that calls LoadCode...

This means that in a multi-threaded environment the programmer has to make sure that only one thread is calling Load... at a time; otherwise, undesired behaviour may occur (even if the compilation succeeds, other configuration parameters may have been changed by another thread. This is why the proposed retry-solution is not helping in my opinion).

It would be great if this limitaion would be removed in a future version of this great lib.

Kind regards

M.

[Oleg Shilo]

Hi M.,

I was able to reproduce the problem reliably. Thanks to your tip about Parallel it became possible for me to catch the problem in the action. This is something that was very difficult with plain multi-threading. I guess the implementation differences of Threading and Parallel somehow make concurrency limitations are easier to reproduce with Parallel. This is the code that I used for testing:

CSScript.CacheEnabled = false;

var source = @"using System;
               class Script
               {
                   public void Main()
                   {
                       Console.WriteLine(""Hello World!"" );
                   }
               }";
            
Parallel.For(0, 500, i =>
{
    var asm = CSScript.LoadCode(source);
});

Anyway, after spending quite some time experimenting I identified one major flaw: the synchronization measures implemented for CSScript.Compile and CSScript.Load family of API calls did not have the some coverage. I have corrected the problem and made the release (v3.7.0). I have done the intensive testing and it seems that the problem now is eliminated completely.

Please download the release with the fix from this location: http://www.csscript.net/CurrentRelease.html. I would also appreciate your feedback on the fix.

Thank you for your help,

Oleg Shilo
--------------------------------------------------------------------------------------------
Internet: http://www.csscript.net
E-Mail: csscript...@gmail.com

[Thias]

Thank you Oleg for the quick help. It seems to work now; the unit tests I had to isolate the problem are no longer failing.

Many thanks for this great library!

Regards

Matthias

[Oleg Shilo]

It was your good pick (and my miss). Thank you for your feedback.

Oleg 

--