SQL Replication Error: Too many steps in script

[Robert Edin]

Hi,

I'm getting an exception while replicating to SQL. See below. Using version 2579.

The exception appears after about 500 replicateTo[tableNanme] calls.

Is it a limitation in how many SQL entries that can be generated from one Raven doc?

Our need is to replicate thousands of SQL entries from one document.

-- Robert

System.InvalidOperationException: Unable to execute JavaScript: 

replicateToStepDoc({ 

RavenID: documentId,

StepName: this.StepName, 

StepId: this.StepId,

DeviceId: this.DeviceId,

WaferId: this.WaferId,

Project: this.Project,

Batch: this.BatchNo,

Instruction: this.Instruction,

InstructionVersion: this.InstructionVersion,

Station: this.StationPos

});

output(documentId);

for ( var i = 0; i < this.Parameters.length  ; i++) { 

  var param = this.Parameters[i];

  var paramType = Object.prototype.toString.call(param.Value).match(/\s([a-zA-Z]+)/)[1].toLowerCase();

  

  try {

 if( paramType == "number") {

if( param.Value == "nan" ) {

param.Value = 0;

}

if( param.Value == "infinity" ) {

param.Value = 0;

}

if( param.Value == "-infinity" ) {

param.Value = 0;

}

replicateToParNum( { Name: param.Name, Value: param.Value, N: 0 } );

 }

 

if( paramType == "string") {

replicateToParStr( { Name: param.Name, Value:  param.Value.toString().substring(0, 250) , N: 0 } );

}

  }

  catch (err) {

      output( { Name: param.Name, Value: err } );

  }

  

  if( paramType == "array") { 

    for ( var j = 0; j < param.Value.length; j++) {

  try {

  var arrayType = Object.prototype.toString.call(param.Value[j]).match(/\s([a-zA-Z]+)/)[1].toLowerCase();

value = param.Value[j].toLowerCase();

if( arrayType == "number") {

if( value == "nan" ) {

param.Value[j] = 0;

}

if( value == "infinity" ) {

param.Value[j] = 0;

}

if( value == "-infinity" ) {

param.Value[j] = 0;

}

replicateToParNum( { Name: param.Name, Value: param.Value[j], N: j } );

 }

    

if( arrayType == "string") {

replicateToParStr( { Name: param.Name, Value: param.Value[j], N: j } );

} 

 }

 catch (err) {

output( { Name: param.Name, Value: err } );

 }

    }

  }

}

Error: 

Too many steps in script

Debug information: 

lasers/22711057/400.130.160.130

 ---> Jint.Native.JsException: Exception of type 'Jint.Native.JsException' was thrown. Error: Too many steps in script

   at Jint.ExecutionVisitor.EnsureSteps() in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\ExecutionVisitor.cs:line 162

   at Jint.ExecutionVisitor.Visit(BlockStatement statement) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\ExecutionVisitor.cs:line 301

   at Jint.Expressions.BlockStatement.Accept(IStatementVisitor visitor) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\Expressions\BlockStatement.cs:line 17

   at Jint.ExecutionVisitor.Visit(ForStatement statement) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\ExecutionVisitor.cs:line 444

   at Jint.Expressions.ForStatement.Accept(IStatementVisitor visitor) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\Expressions\ForStatement.cs:line 18

   at Jint.ExecutionVisitor.Visit(BlockStatement statement) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\ExecutionVisitor.cs:line 305

   at Jint.Expressions.BlockStatement.Accept(IStatementVisitor visitor) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\Expressions\BlockStatement.cs:line 17

   at Jint.ExecutionVisitor.Visit(BlockStatement statement) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\ExecutionVisitor.cs:line 305

   at Jint.Expressions.BlockStatement.Accept(IStatementVisitor visitor) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\Expressions\BlockStatement.cs:line 17

   at Jint.Native.JsFunction.Execute(IJintVisitor visitor, JsDictionaryObject that, JsInstance[] parameters) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\Native\JsFunction.cs:line 94

   at Jint.ExecutionVisitor.ExecuteFunction(JsFunction function, JsDictionaryObject that, JsInstance[] parameters, Type[] genericParameters) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\ExecutionVisitor.cs:line 1476

   at Jint.ExecutionVisitor.ExecuteFunction(JsFunction function, JsDictionaryObject that, JsInstance[] parameters) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\ExecutionVisitor.cs:line 1409

   at Jint.Native.JsApplyFunction.Execute(IJintVisitor visitor, JsDictionaryObject that, JsInstance[] parameters) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\Native\JsApplyFunction.cs:line 46

   at Jint.ExecutionVisitor.ExecuteFunction(JsFunction function, JsDictionaryObject that, JsInstance[] parameters, Type[] genericParameters) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\ExecutionVisitor.cs:line 1476

   at Jint.ExecutionVisitor.Visit(MethodCall methodCall) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\ExecutionVisitor.cs:line 1377

   at Jint.Expressions.MethodCall.Accept(IStatementVisitor visitor) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\Expressions\MethodCall.cs:line 22

   at Jint.ExecutionVisitor.Visit(MemberExpression expression) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\ExecutionVisitor.cs:line 1249

   at Jint.Expressions.MemberExpression.Accept(IStatementVisitor visitor) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\Expressions\MemberExpression.cs:line 21

   at Jint.ExecutionVisitor.Visit(ExpressionStatement statement) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\ExecutionVisitor.cs:line 353

   at Jint.Expressions.ExpressionStatement.Accept(IStatementVisitor visitor) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\Expressions\ExpressionStatement.cs:line 16

   at Jint.ExecutionVisitor.Visit(BlockStatement statement) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\ExecutionVisitor.cs:line 305

   at Jint.Expressions.BlockStatement.Accept(IStatementVisitor visitor) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\Expressions\BlockStatement.cs:line 17

   at Jint.ExecutionVisitor.Visit(BlockStatement statement) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\ExecutionVisitor.cs:line 305

   at Jint.Expressions.BlockStatement.Accept(IStatementVisitor visitor) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\Expressions\BlockStatement.cs:line 17

   at Jint.Native.JsFunction.Execute(IJintVisitor visitor, JsDictionaryObject that, JsInstance[] parameters) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\Native\JsFunction.cs:line 94

   at Jint.ExecutionVisitor.ExecuteFunction(JsFunction function, JsDictionaryObject that, JsInstance[] parameters, Type[] genericParameters) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\ExecutionVisitor.cs:line 1476

   at Jint.JintEngine.CallFunction(JsFunction function, Object[] args) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\JintEngine.cs:line 416

   at Jint.JintEngine.CallFunction(String name, Object[] args) in c:\Work\ravendb-2.0\SharedLibs\Sources\jint-22024d8a6e7a\Jint\JintEngine.cs:line 410

   at Raven.Database.Json.ScriptedJsonPatcher.ApplySingleScript(RavenJObject doc, ScriptedPatchRequest patch, Int32 size, String docId) in c:\Builds\RavenDB-Unstable-v2.5\Raven.Database\Json\ScriptedJsonPatcher.cs:line 123

   --- End of inner exception stack trace ---

   at Raven.Database.Json.ScriptedJsonPatcher.ApplySingleScript(RavenJObject doc, ScriptedPatchRequest patch, Int32 size, String docId) in c:\Builds\RavenDB-Unstable-v2.5\Raven.Database\Json\ScriptedJsonPatcher.cs:line 151

   at Raven.Database.Json.ScriptedJsonPatcher.Apply(RavenJObject document, ScriptedPatchRequest patch, Int32 size, String docId) in c:\Builds\RavenDB-Unstable-v2.5\Raven.Database\Json\ScriptedJsonPatcher.cs:line 70

   at Raven.Database.Bundles.SqlReplication.SqlReplicationTask.ApplyConversionScript(SqlReplicationConfig cfg, IEnumerable`1 docs) in c:\Builds\RavenDB-Unstable-v2.5\Raven.Database\Bundles\SqlReplication\SqlReplicationTask.cs:line 393

[Oren Eini (Ayende Rahien)]

Put simply, we have a budget on the number of operations that a script is allowed to make.

How many Parameters are there in this document?

--
You received this message because you are subscribed to the Google Groups "ravendb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Robert Edin]

The number of parameters are not fixed.

It is measurement data from production of lasers.

The database SW have no control in what is saved by measurement scripts.

It is common with a few hundred parameters and some of them are vectors that also may be in the 100+ range.

=>  several thousand SQL rows.

Is it possible to have something like the Take concept in queries when replicating?

[Oren Eini (Ayende Rahien)]

How much do you have _for that document_?

[Robert Edin]

The biggest document I can find in the DB is about 300 kB, and we are working on cutting down the size on that one for other reasons,

They usually are 2 - 100 kB. A 100k document will generate many SQL rows also...

[Oren Eini (Ayende Rahien)]

This is the document in question: lasers/22711057/400.130.160.130

How many items do you have there?

[Robert Edin]

Its will be about 1000 SQL replications and the document is 100 kB in Raven.

[Oren Eini (Ayende Rahien)]

Set Raven/MaxStepsForScript to a high value, then.

[Kijana Woodard]

This is starting to sound like a job for this guy: http://ayende.com/blog/162273/raven-xyz-trying-out-some-ideas

[Robert Edin]

Useful, but not for us. We care about each data for each item. 

[Robert Edin]

What is a "Step"? A replicateTo() or one line of executed JavaScript or ...?

In https://groups.google.com/forum/?fromgroups#!topic/ravendb/pEhGX1i-qhU

you state that it is only implemented in 2.5. We run 2.0 in production. Is it still only 2.5?

[Robert Edin]

Adding 

<add key="Raven/MaxStepsForScript" value="100000000"/>

to Raven.Server.exe.config did not help.

Logging out for weekend.

Den fredagen den 24:e maj 2013 kl. 17:54:56 UTC+2 skrev Oren Eini:

[Oren Eini (Ayende Rahien)]

Any op.

i++; is a step

Calling a method is a step, as well as anything it does.

And this is for 2.5, yes.

[Robert Edin]

Adding 

<add key="Raven/MaxStepsForScript" value="1000000000"/>

<add key="Raven/AdditionalStepsForScriptBasedOnDocumentSize" value="100"/>

to Raven.server.exe.Config does not give more replication before the "Error: Too many steps in script".

Is it a fixed ceiling independent of the config?

[Kijana Woodard]

I don't see a max being set in Raven's source code. 

Maybe there's something in JintEngine, but I had trouble finding the method in the jint source code on codeplex or github.

[Kijana Woodard]

Ok. No max here: 

https://github.com/ayende/ravendb/blob/2.5/SharedLibs/Sources/jint-22024d8a6e7a/Jint/JintEngine.cs#L295

or here:

https://github.com/ayende/ravendb/blob/2.5/SharedLibs/Sources/jint-22024d8a6e7a/Jint/ExecutionVisitor.cs#L160

Just to rule out the simple stuff, you've restarted Raven.Server, correct?

[Robert Edin]

Restarted, yes. Running these tests in console mode. Maybe this is the time to start to build from source to be able to debug....

[Oren Eini (Ayende Rahien)]

This is set here for 2.0 (no config):

https://github.com/ayende/ravendb/blob/master/Raven.Database/Json/ScriptedJsonPatcher.cs#L108

Here for 2.5 (has config):

https://github.com/ayende/ravendb/blob/2.5/Raven.Database/Json/ScriptedJsonPatcher.cs#L121

[Robert Edin]

I have tried hard to isolate this problem. As of now I doubt that the configuration parameter works.

replicateToStepDoc({

        RavenID: documentId,

        StepName: this.StepName,

        StepId: this.StepId,

        DeviceId: this.DeviceId,

        WaferId: this.WaferId,

        Project: this.Project,

        Batch: this.BatchNo,

        Instruction: this.Instruction,

        InstructionVersion: this.InstructionVersion,

        Station: this.StationPos

    });

for (var i=0; true; i++) output(i );

[Robert Edin]

Hmm, last post sent before ment...

However the forever loop 

  for (var i=0; true; i++) output(i );

produces the same result independent of how I change value for

  <add key="Raven/MaxStepsForScript" value="1000000000"/>

Can anyone confirm that this setting work?

[Oren Eini (Ayende Rahien)]

This never ends, of course it would error.

[Robert Edin]

Yes ofcourse it "fails". I used it to see if the number of output result varied due to the max steps check if I changed the "Raven/MaxStepsForScript". I did not.

[Oren Eini (Ayende Rahien)]

What build are you using?

--

[Robert Edin]

For these tests i use build 2579.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Oren Eini (Ayende Rahien)]

That should work, can you create a full failing test?

http://ravendb.net/docs/2.0/samples/raven-tests/createraventests

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Robert Edin]

I don't see how I can create a failing test for this matter using a console app?

I tried to build Raven from source to track it down, but it fails to build for me. 

I think I have followed the build guide. Any idea what I have missed?

PS C:\Users\robert.edin\Documents\RavenDB\ravendb-2.5\ravendb-2.5> .\build.cmd

All packages listed in packages.config are already installed.

psake version 4.2.0

Copyright (c) 2010 James Kovacs

Executing Stable

Executing Verify40

Executing Clean

Executing Init

fatal: Not a git repository (or any of the parent directories): .git

2013-05-30 13:40:22: An Error Occurred:

You cannot call a method on a null-valued expression.

At C:\Users\robert.edin\Documents\RavenDB\ravendb-2.5\ravendb-2.5\build_utils.ps1:15 char:23

+         return $gitLog.Split <<<< (' ')[0]

    + CategoryInfo          : InvalidOperation: (Split:String) [], RuntimeException

    + FullyQualifiedErrorId : InvokeMethodOnNull

Exec: Error executing command  invoke-psake "$scriptDir\default.ps1" $task .

At C:\Users\robert.edin\Documents\RavenDB\ravendb-2.5\ravendb-2.5\packages\psake.4.2.0.1\tools\psake.psm1:139 char:14

+         throw <<<<  ("Exec: " + $errorMessage)

    + CategoryInfo          : OperationStopped: (Exec: Error exe...lt.ps1" $task .:String) [], RuntimeException

    + FullyQualifiedErrorId : Exec: Error executing command  invoke-psake "$scriptDir\default.ps1" $task .

[Oren Eini (Ayende Rahien)]

You need to clone the repository from git for this.

Or use this:

http://ravendb.net/docs/2.0/samples/raven-tests/createraventests