CSScript.LoadCode randomly generates FileNotFoundException?

[stephan...@gmail.com]

Hi,

I did some stress tests on CSScripLibrary.dll, and it appears that LoadCode randomly fails on a FileNotFoundException.
You need several thousands of executions to have the error.

Did anyone find such issue? Is there something I miss here?
Is the CSScript.LoadCode function asynchronous?

Regards,
Stéphane


Here is the code calling LoadCode (actually this code is wrapped inside a loop which I removed for the sake of clarity):

                CSScript.GlobalSettings.UseAlternativeCompiler = Assembly.GetExecutingAssembly().Location;
                var cacheDir = CSScript.GetScriptTempDir();
                CSScript.GlobalSettings.InMemoryAsssembly = true;
                string asm = Path.Combine(cacheDir,Path.GetRandomFileName());
                var ass = CSScript.LoadCode(codeToCompile, asm, false);
                hostType = ass.GetType(hostClassName);
                File.Delete(asm);

[Oleg Shilo]

Hi there,

>Is the CSScript.LoadCode function asynchronous?

No it is synchronous.

I will be doing some investigation during weekend as I have received another reports about synch problem (different problem but probably in the same area). I will let you know the outcome.

Cheers, 

Oleg 

--
You received this message because you are subscribed to the Google Groups "CS-Script" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cs-script+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Oleg Shilo]

Hi Stéphane,

The new release v3.7.0 addresses the problem. At least this is what the latest testing has indicated. :o)

Please download the release from here: http://www.csscript.net/CurrentRelease.html.

You can find some more detail on the problem in this thread:

https://groups.google.com/d/msg/cs-script/6JSX95dnXds/Us-37BDk8jIJ

Cheers,

Oleg

[stephan...@gmail.com]

Thanks, I will try that soon

S.

[stephan...@gmail.com]

Hi,

We tested v. 3.7.1, but the same problem reappeared after ~1000 runs.
The compiled code is small (only one line of compiled VB).
Below is the callstack.


Regards,
Stéphane


System.IO.FileNotFoundException: Could not find file 'C:\Users\myuser\AppData\Local\Temp\CSSCRIPT\vrkewmt5.pdb'.
File name: 'C:\Users\myuser\AppData\Local\Temp\CSSCRIPT\vrkewmt5.pdb'
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
   at System.IO.FileStream..ctor(String path, FileMode mode)
   at csscript.MetaDataItems.StampFile(String file)

Unhandled Exception: System.IO.FileNotFoundException: Could not find file 'C:\Users\myuser\AppData\Local\Temp\CSSCRIPT\vrkewmt5.pdb'.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize)
   at System.IO.File.OpenFile(String path, FileAccess access, SafeFileHandle& handle)
   at System.IO.File.SetLastWriteTimeUtc(String path, DateTime lastWriteTimeUtc)
   at System.IO.FileSystemInfo.set_LastWriteTimeUtc(DateTime value)
   at csscript.CSExecutor.ProcessCompilingResult(CompilerResults results, CompilerParameters compilerParams, ScriptParser parser, String scriptFileName, String assemblyFileName, String[] additionalDependencies)
   at csscript.CSExecutor.Compile(String scriptFileName)
   at csscript.CSExecutor.Compile(String scriptFile, String assemblyFile, Boolean debugBuild)
   at CSScriptLibrary.CSScript.LoadWithConfig(String scriptFile, String assemblyFile, Boolean debugBuild, Settings scriptSettings, String compilerOptions, String[] refAssemblies)
   at CSScriptLibrary.CSScript.LoadCode(String scriptText, String tempFileExtension, String assemblyFile, Boolean debugBuild, String[] refAssemblies)
   at CSScriptLibrary.CSScript.LoadCode(String scriptText, String assemblyFile, Boolean debugBuild, String[] refAssemblies)
   at Engine.CSScriptCompiler.TryCompile(Type& hostType, IList`1& compileErrors) in c:\Dev\Engine\CSScriptCompiler.cs:line 53
   at Engine.CCEngine.TryCompileCCs(IEnumerable`1 ccs, Type& hostType, IList`1& compileErrors) in c:\Dev\Engine\CCEngine.cs:line 138
   at Engine.CCEngine.TryProcess(IEnumerable`1 ccs, ISequence sequence, String constantFileName, List`1& processErrors) in c:\Dev\Engine\CCEngine.cs:line 51
   at UnattendedProcessingCC.Program.EvaluateFile(String fileName, ISequence sequence, Boolean dumpResults)
   at UnattendedProcessingCC.Program.Main(String[] args)

[Oleg Shilo]

Hi Stéphane,

I have again tested 1500 parallel script compilation of both C# and VB. All successful. 

Thus, first of all I want to ensure that our are testing the same things. Please find the test project here: https://dl.dropboxusercontent.com/u/2192462/Support/Stephane%20Thiers/CS-Script.Testpad.7z

Please use it to see if you can reproduce the problem. Keep in mind that if you test with LoadCode then it may take significant tome to complete the test. This is because the all optimization is disabled just to force compilers to compile and load the same code again and again. The purpose is to put the environment under stress and handling 1500 assemblies in the same AppDomain is not an easy task for CLR.   

-----------------

 

During numerous CS-Script concurrency experiments I have seen a few times strange behavior of .NET compilers (e.g. csc.exe) when they (or something else) kept the compiled files locked even after the compiler process exited but before CS-Script loaded the compiled file. But I have difficulties to reproduce it.And anyway it is outside of the CS-Script responsibilities.

What I am confident now is that CS-Script indeed respects concurrency. And the test code I just shared with you demonstrates this. Saying that, I would gladly have a look at the problem if it is possible to reproduce it (either share your test code or adjust mine).   

Regards,

Oleg

On Fri, Jan 31, 2014 at 8:26 PM, <stephan...@gmail.com> wrote:

Hi,

We tested v. 3.7.1, but the same problem reappeared after ~1000 runs.
The compiled code is small (only one line of compiled VB).
Below is the callstack.


Regards,

--

[stephan...@gmail.com]

Hi Oleg,

Sorry for late answer.
I will look at the problem more closely.
We still get issue, and now even in our automated test suite (2 occurences in several thousand runs).
I will keep you informed soon.

Regards,
Stéphane

[stephan...@gmail.com]

Hi,

I downloaded your script and modified it in order to reproduce the issue, which I got after less than 1000 runs.
In order to repriduce it, replace the code of "static void SyncTest()" by the following: (no other modification!)

            var source = @"using System;
                           class Script
                           {
                                public void Main()
                                {
                                   Console.WriteLine(""Hello World!"" );
                                }
                           }";

            for (int i = 0; i < 150000; i++)
            {

                var cacheDir = CSScript.GetScriptTempDir();
                CSScript.GlobalSettings.InMemoryAsssembly = true;

                string assembyPath = Path.Combine(cacheDir, Path.GetRandomFileName());
                var ass = CSScript.LoadCode(source, assembyPath, false);
                var hostType = ass.GetType("Script");
                if (i%500==0)
                    Console.WriteLine(hostType+" "+i);
                File.Delete(assembyPath);

            }

Regards,
Stéphane

[Oleg Shilo]

Thank you Stephane,

I will have a look at it. Though please give me a few days as I will be able to get back to this project only in a few days.

Regards,

Oleg

[stephan...@gmail.com]

Hi,

Some news:
- Strangely, as far as I remember, each time I reproduced the issue I think ( but not 100% sure) I always got the error on a filename ending with ".pdb". I saw in CSScript code that this extension is treated in a particular way. Just a guess: maybe there's a bug when dealing with assemblies generated in a file ending with .pdb? If I have time, I will check. But anyway: I will now look closely to filenames generating errors!

- I modified the prototype so that we have retry mechanism: it has no effect. If it fails once, then it will always fail.

Regards,
Stéphane

[Oleg Shilo]

Hi Stéphane,

I ran an overnight test and find no problems. The system behaved as expected. 

After reaching 26500+ loaded assemblies I got the exception OutOfMemoryException what is expected considering nature of your test changes, but no "FileLocked" exception. Thus unfortunately it may mean that the problem cannot be reproduce outside of your environment. But Will still will try to find another system to run it on, just in case.

Though I have to question the test changes you done. Now we are not testing concurrency at all. There are no concurrency in the single 'for' loop. And you are not even loading the assembly file but only its in-memory copy (InMemoryAsssembly=true). Meaning that effectively you are testing concurrency/transactions of OS FileSystem and true randomness of the 'GetRandomFileName'. The test still makes sense as it can expose some deficiencies in the subsystems I just named but the target of the testing has been dramatically changed.

It would be interesting to see how the outcome of the test is changed if you replace:

 var ass = CSScript.LoadCode(source, assembyPath, false);
 var hostType = ass.GetType("Script");

with

'substitution A'

 var assFile = CSScript.CompileCode(source, assembyPath, false);

 var ass = Assembly.LoadFrom(assFile);

 var hostType = ass.GetType("Script");

and eventually with

'substitution B'

 var codeFile = File.WriteAllText(code);
 var compilerParams = new compilerParams() { OutFile = assembyPath };                                                //pseudo code

 new Microsoft.CSharp.CSharpCodeProvider().CreateCompiler().Compile(codeFile, compilerParams );  //pseudo code

 var ass = Assembly.LoadFrom(assembyPath );

 var hostType = ass.GetType("Script");

>...maybe there's a bug when dealing with assemblies generated in a file ending with .pdb?

Assemblies are not generated in this file. This is a Program Database file containing debugging information. The error is extremely strange as in your test you are not deleting this file....
Though it may be a clue. This file is always generated by MS csc.exe and I always read it. May be it will help if I do it for debug builds only.

Please find the attached slightly modified version of where I do not do it for the release build.

If would be fantastic if you can test the new CSScriptLibrary.dll and if it fails the 'substitution A' test.

Thank you,

Oleg

[Oleg Shilo]

Hi Stéphane,

I decided not to wait and publish the change to the CSCSriptLibrary.dll now. Please find the binaries here: http://www.csscript.net/LatestBuild.html.

In this release I prevent loading .pdb files in ALL non-debug script execution. It is not a fix exactly as there is nothing wrong with the previous approach. However not loading PDB can decrease probability of locking these files (by whatever software component) on your PC.

Regards,

Oleg

On Tue, Feb 11, 2014 at 12:51 AM, <stephan...@gmail.com> wrote:

[stephan...@gmail.com]

Hi Oleg,

There's a way to reproduce the problem each time now:

Please replace the code inside the loop with the following. Now we force the use of extension .pdb.

var cacheDir = CSScript.GetScriptTempDir();
CSScript.GlobalSettings.InMemoryAsssembly = true;

var assemblyName = Path.GetRandomFileName();
assemblyName = Path.ChangeExtension(assemblyName, ".pdb");
string assembyPath = Path.Combine(cacheDir, assemblyName);

var ass = CSScript.LoadCode(source, assembyPath, false);
var hostType = ass.GetType("Script");
if (i%500==0)     Console.WriteLine(hostType+" "+i);
File.Delete(assembyPath);

Now it fails each time -at least on my system- on FileNotFoundException, which is exactly my initial problem (no link with concurrency).

Regards,
Stéphane

[stephan...@gmail.com]

BTW I tested new CScriptLibrary.dll with substitution 'A4', then it fails everytime with
System.UnauthorizedAccessException was unhandled
But my original error is FileNotFoundException. And I don't think it's linked to concurrency.

And I still obtain it when forcing usage of .pdb extension as in my message from 5 minutes ago.

Regards,
Stéphane

[Oleg Shilo]

Hi Stéphane,

Your proposed test is invalid. Any assembly name with the extension .pdb is illegal. The MS C# compiler reserves '.pdb' extension for the uses Program Database file name. Such a test is expected to fail every time you run it.

Regards,

Oleg

[Oleg Shilo]

With the 'substitution A' it was my mistake. Of course it should do loading from the byte array. This is the correct code:

'substitution A'

 var assFile = CSScript.CompileCode(source, assembyPath, false);

 var ass = Assembly.LoadFrom(File.ReadAllBytes(assFile));

 var hostType = ass.GetType("Script");

Thank you,

Oleg 

[stephan...@gmail.com]

Actually, the test does fail, but the FileNotFoundException that we get is really not obvious! I think I never got this issue on any other extensions.

Stéphane

[Oleg Shilo]

>I think I never got this issue on any other extensions.

Correct. With the .pdb extesnsion it should fail 100% 

>Actually, the test does fail...

Great, this means that the error is not caused by the CS-Script reading the compiled assembly (in the test we did it manually) but by something else. 

Now we have the last test to conduct:

I sincerely hope the test below fails on your system. 

static void SyncTest()
{
    var source = @"using System;
                           public class Script

                           {
                                public void Main()
                                {
                                   Console.WriteLine(""Hello World!"" );
                                }
                           }";
 
 
    for (int i = 0; i < 150000; i++)
    {
        var cacheDir = CSScript.GetScriptTempDir();
        CSScript.GlobalSettings.InMemoryAsssembly = true;
        string assembyPath = Path.Combine(cacheDir, Path

.GetRandomFileName());
 
        Compile(source, assembyPath);
        var ass = Assembly.Load(File.ReadAllBytes(assembyPath));
  
        var hostType = ass.GetType("Script");
        if (i % 500 == 0)
            Console.WriteLine(hostType + " " + i);
        File.Delete(assembyPath);
 
    }
}

static bool Compile(string code, string assembyPath)
{
    CompilerParameters parameters = new CompilerParameters();
    parameters.GenerateExecutable = false;
    parameters.OutputAssembly = assembyPath;
 
    CompilerResults results = new CSharpCodeProvider().CompileAssemblyFromSource(parameters, code);
    return results.Errors.Count == 0;
}

Oleg 

[stephan...@gmail.com]

Hi Oleg,

I think there is a misunderstanding: all issues I got from the beginning were caused by the random generation of filename ending with ".pdb"!

I never had any other issues. So from my point of view, the issue is closed. BTW I suggest that CSScript.LoadCode throws an ArgumentException when being passed a filename ending with ".pdb" as 2nd argument. It would make situation more obvious and less confusing than the FileNotFoundException we get.

Regards,

Stéphane

[Oleg Shilo]

Hi Stéphane,

Indeed I misunderstood you. 

I interpreted "Actually, the test does fail, " as that we still have the problem after you stopped using .pdf name as an input. Fortunately I was wrong :o)

Anyway. I am glad we sorted it out.  

I also agree that all Compile/Load should validate the input assembly name. The improvement will be available in the very next release.

Thank you for your help,

Oleg 

[stephan...@gmail.com]

Hi Oleg,

Finally I start understanding why there was a misundertanding :)

I finally got the same FileNotFoundException on a filename *not* ending with ".pdb": j3r4suzy.mkw

So my errors were due to filenames ending in ".pdb", but not only. Now I discover there might be another issue with completely different root cause, but with same effects.

So first question: does the ".mkw" extension has special meaning/handling?


Regards,
Stéphane

[Oleg Shilo]

Hi Stéphane

No as far as I know ".mkw" does not have any special purpose in C# compilation scenarios.

While we agree that we are no longer testing CS-Script concurrency model (we are practically not testing CS-Script at all) I still have my interest in the finding the cause of the problem on your system. This interest is very practical: even if CS-Script may not be an offender this exercise can help to improve the CS-Script UX. Thus we already identified that validating asm name for ".pdb" extension can really assist with the troubleshooting. Good.

Now I suggest that we use the following simple trick to eliminate any dependency on the file extension (this is what CS-Script does internally):     

string assembyPath = Path.GetRandomFileName() + ".tmp";

If after the change you still get the exceptions I suggest you change SyncTest (as per my prev email) to use CodeDom compilation:  

static bool Compile(string code, string assembyPath)
{
    CompilerParameters parameters = new CompilerParameters();
    parameters.GenerateExecutable = false;
    parameters.OutputAssembly = assembyPath;
 
    CompilerResults results = new CSharpCodeProvider().CompileAssemblyFromSource(parameters, code);
    return results.Errors.Count == 0;
}

It would be really interesting to see if you can fail this one.

Regards,

Oleg 

On Sat, Feb 15, 2014 at 2:45 AM, <stephan...@gmail.com> wrote:

Hi Oleg,

[stephan...@gmail.com]

Hi Oleg,

I applied your ".tmp" trick and still did *not* get the FileNotFoundException. But for the moment, it means not much. I keep you informed.
However, I got another exception.( Maybe should we start another discussion thread for that?)
The call stack is below.
To me, it's clear it's the antivirus. On our system, we always implement a retry mechanism when using functions such as File.Delete, because otherwise it always ends-up in such a bug.
What do you think?

System.UnauthorizedAccessException: Access to the path 'C:\Users\stephane\AppData\Local\Temp\CSSCRIPT\lbab1mna.tmp' is denied.

at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)

   at System.IO.File.InternalDelete(String path, Boolean checkHost)
   at System.IO.File.Delete(String path)

   at csscript.CSExecutor.Compile(String scriptFileName)
   at csscript.CSExecutor.Compile(String scriptFile, String assemblyFile, Boolean debugBuild)
   at CSScriptLibrary.CSScript.LoadWithConfig(String scriptFile, String assemblyFile, Boolean debugBuild, Settings scriptSettings, String compilerOptions, String[] refAssemblies)
   at CSScriptLibrary.CSScript.LoadCode(String scriptText, String tempFileExtension, String assemblyFile, Boolean debugBuild, String[] refAssemblies)
   at CSScriptLibrary.CSScript.LoadCode(String scriptText, String assemblyFile, Boolean debugBuild, String[] refAssemblies)

Regards,
Stéphane

[Oleg Shilo]

Sorry Stéphane I was on the business trip and missed your response...

>To me, it's clear it's the antivirus. On our system, we always implement a retry mechanism when using functions such as File.Delete, because otherwise it always ends-up in such a bug.

>What do you think?

Well, to be honest I do not like the idea of hosting the "work around" for the other products defects. No antivirus should lock any file unless it has been detected as infected and is being in process of "healing" by this antivirus.

Saying that I already use a simple retry for the low level script compilation call (effectively work around for the csc.exe concurrency flaws) and definitely can have the same for File.Delete.  

In fact I have already adjusted the code and do the File.Delete retry. The solution will be available with the very next release. Tough I really doe not like it. And only practical inconvenience of the CS-Script users forced my to go this way.

Regards,

Oleg