Is FhirClient intended to be thread safe?

[Jon Conley]

We are trying to troubleshoot a problem where in certain circumstances, FhirClient appears to not be thread safe (or somehow there is a problem in our code, although we've been over it and so far don't see any thread safety issues).  We are testing by running up to 15 concurrent instances of our code with 20,000 or so "messages" in which a shared FhirClient, performing operations based on each incoming message such as searching for a Patient by Identifier, making updates to it, then using that reference to update other resources such as Encounter (and other properties of the Encounter such as Practitioner, etc.).  

We have debugged the issue to where in some cases we supply a Patient AsyncSearch operation with a particular Identifier, and we are returned back the completely wrong Patient.  This only happens when we run multiple concurrent instances of this code (again, with a shared FhirClient).  It also seems to be the case that when this issue occurs (not very often, but often enough that it is a concern), we are also having to retry due to timeouts/server-side issues (our own system issues, usually overcome with 1 or 2 retries).  Not sure if this has anything to do with the issue, but just something to mention, and that we have a retry mechanism that should retry a given operation in its entirety under certain circumstances when an exception occurs on any call to FhirClient.

So we just wanted to verify that FhirClient is intended to be thread safe?  And wondering if anyone else has run into similar issues when using it concurrently?  Unfortunately we don't have a smaller scale test harness to reproduce this.. the only way we currently have to do it is using our code which would be too difficult to describe here (although we should be using FhirClient correctly).

[Jon Conley]

I should also note that we are only seeing this happen when we try to run our code concurrently, leading to believe there is a thread safety issue somewhere.

[Jon Conley]

I am able to reproduce this issue pretty reliably, using both Hl7.Fhir.STU3 0.96.0.0 and 1.3.0 (When we run this, one of those Assert.IsTrue()'s eventually fail for us).  Here is my test code (uses NUnit 3.10.0).  There are some helper methods and whatnot that we've implemented (i.e. the "retryPolicyFactory" just supports the ability to retry any operation a few times if they fail for any reason, the _patientRootOid could be just any old OID, and the FhirClient here is actually a wrapper because we need to inject an OAUTH token header, but none of that should have a bearing on the results of the test).

Using _numParallelThreads = 15 for the purposes of this test.

Also, when we spin up multiple FhirClients (or possibly create a new one at the beginning of every iteration of the Parallel.ForEach), this issue does not occur.  So per my original question - is that how multithreading with FhirClient is supposed to work?  We were under the impression that a single FhirClient instance could be used by multiple concurrent Tasks, but per this test that does not appear to be the case.

If anyone is able to reproduce this issue, please comment back here as this is becoming a real operational headache for us.  Thanks!

 [Test]
    public async System.Threading.Tasks.Task TestFhirClientCallsInParallel()

    {
      // Generate patients
      var patients = new List<Patient>();
      for (int i = 1; i < 2000; i++)
      {
        patients.Add(new Patient
        {
          Name = new List<HumanName>
          {
            new HumanName
            {
              Family = $"Doe{i}",
              Given = new List<string>
              {
                $"John{i}"
              }
            }
          },
          Identifier = new List<Identifier>
          {
            new Identifier(_patientRootOid, i.ToString())
            {
              Use = Identifier.IdentifierUse.Official,
              Type = new CodeableConcept("http://hl7.org/fhir/v2/0203", "MR")
            }
          }
        });
      }


      await System.Threading.Tasks.Task.Factory.StartNew(() =>
        Parallel.ForEach(patients, new ParallelOptions
        {
          MaxDegreeOfParallelism = _numParallelThreads
        }, async patient =>
        {
          // Use single FhirClient (instantiated as a private var)
          var client =  _fhirCient;


          Console.WriteLine($"Processing Patient {patient.Identifier.FirstOrDefault()?.Value}");
          // create patient
          var createdPatient = await client.CreateAsync(patient, _retryPolicyFactory.CreateCatalystRetryPolicy());
          Assert.NotNull(createdPatient);
          Assert.IsTrue(patient.Identifier.IsExactly(createdPatient.Identifier));


          // query for patient
          var queriedPatientBundle = await client.SearchAsync<Patient>(new[] {$"identifier={patient.Identifier.FirstOrDefault()?.System}|{patient.Identifier.FirstOrDefault()?.Value}"}, _retryPolicyFactory.CreateCatalystRetryPolicy());
          var queriedPatient = queriedPatientBundle?.Entry?.ByResourceType<Patient>()?.FirstOrDefault();
          Assert.NotNull(queriedPatient);
          Assert.IsTrue(patient.Identifier.IsExactly(queriedPatient.Identifier), $"First search for identifier {JsonConvert.SerializeObject(patient.Identifier)} but returned {JsonConvert.SerializeObject(queriedPatient.Identifier)}");


          // update patient
          queriedPatient.Name.FirstOrDefault()?.GivenElement.Add(new FhirString("Q"));
          var updatedPatient = await client.UpdateAsync(queriedPatient, _retryPolicyFactory.CreateCatalystRetryPolicy());
          Assert.NotNull(updatedPatient);
          Assert.IsTrue(patient.Identifier.IsExactly(updatedPatient.Identifier));


          // query for patient (2nd)
          queriedPatientBundle = await client.SearchAsync<Patient>(new[] {$"identifier={patient.Identifier.FirstOrDefault()?.System}|{patient.Identifier.FirstOrDefault()?.Value}"}, _retryPolicyFactory.CreateCatalystRetryPolicy());
          queriedPatient = queriedPatientBundle?.Entry?.ByResourceType<Patient>()?.FirstOrDefault();
          Assert.NotNull(queriedPatient);
          Assert.IsTrue(patient.Identifier.IsExactly(queriedPatient.Identifier), $"Second search for identifier {JsonConvert.SerializeObject(patient.Identifier)} but returned {JsonConvert.SerializeObject(queriedPatient.Identifier)}");


          // update patient (2nd)
          queriedPatient.Gender = AdministrativeGender.Male;
          updatedPatient = await client.UpdateAsync(queriedPatient, _retryPolicyFactory.CreateCatalystRetryPolicy());
          Assert.NotNull(updatedPatient);
          Assert.IsTrue(patient.Identifier.IsExactly(updatedPatient.Identifier));


          // query for patient
          queriedPatientBundle = await client.SearchAsync<Patient>(new[] {$"identifier={patient.Identifier.FirstOrDefault()?.System}|{patient.Identifier.FirstOrDefault()?.Value}"}, _retryPolicyFactory.CreateCatalystRetryPolicy());
          queriedPatient = queriedPatientBundle?.Entry?.ByResourceType<Patient>()?.FirstOrDefault();
          Assert.NotNull(queriedPatient);
          Assert.IsTrue(patient.Identifier.IsExactly(queriedPatient.Identifier), $"Third search for identifier {JsonConvert.SerializeObject(patient.Identifier)} but returned {JsonConvert.SerializeObject(queriedPatient.Identifier)}");


          // delete patient
          await client.DeleteAsync(queriedPatient, _retryPolicyFactory.CreateCatalystRetryPolicy());
        })
      );
    }

[Jon Conley]

I've created a github issue for this: https://github.com/FirelyTeam/fhir-net-api/issues/1110