low unit test performance after upgrading to 5.1.5

[Mario J.]

We recently upgraded from raven 4.2.101 to raven 5.1.5 and found a huge performance drop in our unit tests. They used to take around 3.5 min but they now take over 6 min. The CPU usage with 5.1.5 is around 50 % on the build runner whereas it was at always almost 100 % with 4.2.101.

Some further investigations with a reduced set of 8 in parallel executed unit test assemblies showed the following:
* Raven 4.2.101: 25 s / 16 used cores
* Raven 4.2.103: 27 s / 16 used cores
* Raven 4.2.104: 1 m / 3 used cores
* Raven 4.2.105: 1 m / 3 used cores

It seems like something was introduced in 4.2.104 which made it to 5.1.5 as well. Do you have any ideas?

[Egor Shamanaev]

Hi 

I am not aware of any issues, can you share the unit tests so we can check it locally ?

--
You received this message because you are subscribed to the Google Groups "RavenDB - an awesome database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/ad2b1b60-fed7-4229-b522-997bd20db34cn%40googlegroups.com.

--

Egor Developer   /   Hibernating Rhinos LTD Sales:  sa...@ravendb.net Support:  sup...@ravendb.net

[Mario J.]

Sure, please check the following link: https://drive.google.com/file/d/10J31AhXPb4eOa9hy32u4MUKDXv9kEThI/view

The problem arises if multiple test-dlls are run in parallel. You find some bat-files in the zip which simulates the script on our build runner. These are the numbers on my local machine. The most interesting fact is, that Raven5 only uses 3 of my 16 cores.

Raven 5.1.5

• 4 tests in parallel: 26 s
• 8 tests in parallel : 50 s
• 16 tests in parallel : 3.3 min (25 failed)

Raven 4.2.103

• 4 tests in parallel : 22 s
• 8 tests in parallel : 24 s
• 16 tests in parallel : 51 s

[Mario J.]

Further research showed that it might be a licensing issue. I wrote a little helper class which activates a developer license for the unit tests. With this fix in place, I get the following numbers for Raven 5.1.5:

• 4 tests in parallel: 18 s
• 8 tests in parallel: 27 s
• 16 tests in parallel: 52 s

public static class SlowRavenDB5TestHelper
{
    private static readonly SemaphoreSlim Semaphore = new SemaphoreSlim(1, 1);
    private static bool _isFixed = false;

    public static async Task FixLicenseAsync(IDocumentStore store)
    {
        if (!_isFixed)
        {
            await Semaphore.WaitAsync();
            try
            {
                if (!_isFixed)
                {
                    var license = GetLicense();
                    var url = store.Identifier.Split(' ')[0];
               
                    var request = WebRequest.Create(url + "/admin/license/activate");
                    request.Method = "POST";
                    request.ContentType = "application/json";
                    request.ContentLength = license.Length;

                    using (var stream = await request.GetRequestStreamAsync())
                    {
                        await stream.WriteAsync(license, 0, license.Length);
                        await request.GetResponseAsync();
                    }

                    _isFixed = true;
                }
            }
            finally
            {
                Semaphore.Release();
            }
        }
    }

    private static byte[] GetLicense() =>
        Encoding.UTF8.GetBytes(/* insert developer license string*/);
}

The usage is as simple as follows:

protected async Task TestTemplate(int items)
{
    var database = Guid.NewGuid().ToString();
    using (var store = GetDocumentStore(database: database))
    {
        await SlowRavenDB5TestHelper.FixLicenseAsync(store);
       
        [..]
    }
}

It's just a dirty workaround but it might help you in finding the root cause of this issue.

[Mario J.]

I'm now pretty confident that the RavenTestDriver is limited to 3 cores. This new behavior was introduced in version 4.2.104, in earlier versions no such limit was present. Can you confirm that? Is this a bug or is there any documentation regarding this behavior?

Thank you

[Egor Shamanaev]

Hi 

Did you register the license using the environment variable ? or using the command line argument ?

Here is an example on how to register it using the command line arg: https://github.com/ravendb/ravendb/issues/12012#issuecomment-823494180

To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/28196de2-39e6-4c55-aa18-730689519e6en%40googlegroups.com.

[Marko Lahma]

Instead of the need to fiddle with license files, I think it would be really valuable for test driver to be able to run in some "unlimited" mode but with license expiration of couple of hours or so after start. This would allow seamless CI story.

Just my $0.02,

-Marko

[Christoph Enzmann]

I agree, TestDriver should use all CPU cores even without providing a license file.

This was already the case with 4.0 and earlier 4.1 versions.

The documentation of TestDriver (https://ravendb.net/docs/article-page/5.1/csharp/start/test-driver) also doesn't mention that you need to pass a license file.

I guess most people will just assume that RavenDB has bad performance in unit tests.

Chris

[Ryan Heath]

But the other way could also happen:

good performance in unit tests, but because of chosen license, less performance in production.
I think you would want to mimic production (perf) in your unit tests.

// Ryan

To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/c1380442-ecee-467d-aff2-a3addf781906n%40googlegroups.com.

[Christoph Enzmann]

Still should be documented. Also most unit test aren't performance tests.

Licences have an expiry date. So you need to keep them up to date. This isn't the best experience in an automated CI system.

Chris

[Mario J.]

Egor, I switechd to a commandline argument in the meantime. However, I think that this is not relevant for my question.
Ryan, your point is valid regarding perf tests. But I'm talking about tests which I want to be as fast as possible (tight feedback loop). This has no implications for production.
Marko and Chris, I agree that RavenTestDriver shouldn't be limited in this scenario. It's the wrong place to deal with licensing issues.

Considering the lack of documentation, I still think this is a bug.

[Mario J.]

Hi

Is there a chance to get an statement wheter this is an excpected behavior or kind of a bug?

Just to figure out how we should proceed with this issue.

Thanks

[Igal Merhavia]

Hi,

This is expected behavior, yes.

Sorry for the inconvenience :)

Just config the test to use your license.

Best regards,

Igal

To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/23b9a67e-05f9-4d27-a42a-4eb0591dabc3n%40googlegroups.com.

[Mario J.]

Hi Igal, thank you for your response. Perhaps it would be appropriate to add a corresponding note to the documentation of the test driver. Next to the topic "Continuous Integration Servers" might be a good place. The following is a suggestion:

Be aware that RavenTestDriver is limited to 3 cores. If you run raven-tests in parallel, you should provide a (developer) license to get the maximum performance out of your buildrunners. You can use the TestServerOptions to provide such a license.

ConfigureServer(new TestServerOptions
{
    CommandLineArgs = new List<string>
    {
        "--License.Path=C:\\temp\\license.json"
    }
});

If you use a developer license, be aware that you have to update it every six months. You might want to add a corresponding check to your testsuite.

[Egor Shamanaev]

Hi

Thanks for the suggestion

https://issues.hibernatingrhinos.com/issue/RDoc-1930

To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/bbd9097c-3fa0-4478-83ff-5ae8a53214f1n%40googlegroups.com.

[Christoph Enzmann]

A way to test if the dev license is still valid:

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Newtonsoft.Json.Linq;
using NUnit.Framework;

[TestFixture]
    public class RavenTestSuiteTests
    {
        // Code from https://github.com/ravendb/ravendb/blob/45f33a7c78954bdb731df961ecf298bb49c2268a/src/Raven.Server/Commercial/LicenseValidator.cs#L119
        private const int TypeBitsToShift = 5;

        private static readonly string[] Terms =
        {
            "type", "version", "expiration", "memory", "cores", "redist", "encryption", "snmp",
            "distributedCluster", "maxClusterSize", "snapshotBackup", "cloudBackup", "dynamicNodesDistribution",
            "externalReplication", "delayedExternalReplication", "ravenEtl", "sqlEtl", "highlyAvailableTasks",
            "pullReplicationAsHub", "pullReplicationAsSink", "encryptedBackup", "letsEncryptAutoRenewal", "cloud",
            "documentsCompression", "timeSeriesRollupsAndRetention", "additionalAssembliesNuget"
        };

        private enum ValueType : byte
        {
            False = 0,
            True = 1,
            Int8 = 2,
            Date = 3,
            String = 4,
            Int32 = 5
        }

        private static DateTime FromDosDate(ushort number)
        {
            var year = (number >> 9) + 1980;
            var month = (number & 0x01e0) >> 5;
            var day = number & 0x1F;
            return new DateTime(year, month, day);
        }

        public static Dictionary<string, object> ParseLicense(string licenseJson)
        {
            var license = JObject.Parse(licenseJson);
            var keys = Convert.FromBase64String(String.Join(String.Empty, license["Keys"].Values<string>()));

            var result = new Dictionary<string, object>();
            using var ms = new MemoryStream();
            using var br = new BinaryReader(ms);
            var buffer = keys.Skip(128).Take(keys.Length - 128).ToArray();
            ms.Write(buffer, 0, buffer.Length);
            ms.Position = 0;

            while (ms.Position < buffer.Length)
            {
                var token = ms.ReadByte();
                var index = token & 0x1F;
                object val;
                var curr = (ValueType)(token >> TypeBitsToShift);
                switch (curr)
                {
                    case ValueType.False:
                        val = false;
                        break;
                    case ValueType.True:
                        val = true;
                        break;
                    case ValueType.Int8:
                        val = (int)br.ReadByte();
                        break;
                    case ValueType.Int32:
                        val = br.ReadInt32();
                        break;
                    case ValueType.Date:
                        val = FromDosDate(br.ReadUInt16());
                        break;
                    case ValueType.String:
                        var valLength = (int)br.ReadByte();
                        val = Encoding.UTF8.GetString(br.ReadBytes(valLength));
                        break;
                    default:
                        throw new Exception("unknown term");
                }

                if (index >= Terms.Length)
                {
                    continue; // new field, just skip
                }

                result[Terms[index]] = val;
            }

            return result;
        }

        [Test]
        public async Task RavenLicenseValid()
        {
            var pathToDevelopmentLicense = "path to your RavenDevelopmentLicense.json";
            var parsedLicense = ParseLicense(await File.ReadAllTextAsync(pathToDevelopmentLicense));

            var expirationDate = (DateTime)parsedLicense["expiration"];
            if (expirationDate < DateTime.Today)
            {
                throw new Exception("RavenDB Build Pipeline Developer license expired, renew 'RavenDevelopmentLicense.json'.");
            }
        }