Nexus Nuget Proxy and search/browse?

[Ulf Axelsson]

Hi!

I have been testing Nexus OSS for Nuget management locally, both v2.12 and v3m7.

While M7 still lacks a few things that were configurable i v2 and has that Win2012 issue I find it much easier to get running than v2.

But both versions share a very odd behaviour when one sets up a Proxy to nuget.org and uses that to browse/search within Visual Studio.

My two best test-words to search for that triggers this is nlog and entityframework.

Search for nlog among stable packages I get lists that starts with the following three items:

VS2013:

NLog 4.4.0-beta3

NLog configuration 4.3.0-rc2

NLog Schema for Intelligence 4.3.0-rc2

VS2015:

Logentries.nlog 2.6.0

Castle.Core-NLog 3.3.3

ServiceStack.NLog.Logging 4.0.54

(_nowhere_ in this list does NLog itself turn up)

Against nuget.org directly the list is:

NLog 4.2.3

NLog configuration 4.2.3

in both VS versions but differs on the third result

NLog.Web vs NLog.InstallNLogConfig

NLog Schema for Intelligence 4.3.0-rc2

Try with entityframework and the results are just as odd.

I did manage to enable logging in v2 that gave me the "REST" Uri that Nexus sent to nuget.org, and if I try that Uri manually I get the expected list, but after processing by Nexus something different is returned.

Has anyone else seen this?

/Ulf

[Michael Prescott]

Ulf, we do know that Nexus can return different search results than nuget.org in some cases - what's happening under the covers is that Nexus is issuing the query to the remote server (as you saw).  Then, instead of passing those results directly on to the client, they're cached, and then Nexus re-queries this cached metadata.

This is done to support cases where the remote repository can't be reached, but also so that queries to groups can be made quickly (we query all repositories' cached metadata simultaneously, rather than merging result sets). Unfortunately, it does have this symptom, where the search results can differ - nuget.org uses a fuzzy 'relevance' sort order by default, which Nexus can't replicate exactly (as it has a different search index implementation).

What you're experiencing, however, appears to be a different issue, which we're investigating.

Michael

[Ulf Axelsson]

Ok, yes there is something fishy (or fuzzy) with nuget.org itself. You do get a VS2013/2015 difference, but that might just be the v2/v3 API:s having different fuzziness.

Wonder what is causing the mixup with stable/prerelease and the nlog going missing though, I am eager to hear about your findings :-)

/Ulf

On 1 April 2016 at 18:52, Michael Prescott <mpre...@sonatype.com> wrote:

Ulf, we do know that Nexus can return different search results than nuget.org in some cases - what's happening under the covers is that Nexus is issuing the query to the remote server (as you saw).  Then, instead of passing those results directly on to the client, they're cached, and then Nexus re-queries this cached metadata.

This is done to support cases where the remote repository can't be reached, but also so that queries to groups can be made quickly (we query all repositories' cached metadata simultaneously, rather than merging result sets). Unfortunately, it does have this symptom, where the search results can differ - nuget.org uses a fuzzy 'relevance' sort order by default, which Nexus can't replicate exactly (as it has a different search index implementation).

What you're experiencing, however, appears to be a different issue, which we're investigating.

Michael

--
You received this message because you are subscribed to a topic in the Google Groups "Nexus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/a/glists.sonatype.com/d/topic/nexus-users/VhtlT_L23Y8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nexus-users...@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/62ae8dd5-868e-4d08-8d29-b01ac0678c0f%40glists.sonatype.com.

For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.

[Michael Prescott]

Ulf,

All right. As of Visual Studio 2015/NuGet 3.4, NuGet has become much stricter with semantic versioning. Any suffix after the version number is taken to mean the package is a pre-release version.

c.f. http://docs.nuget.org/Create/Versioning#Normalized_Version_Numbers

Now, for whatever reason, the NLog project has started marking their last pre-release with 'release' metadata, despite having a pre-release version number.  Their last proper release version (going by version number alone) was 4.2.3, but they've declared 4.4.0-beta3 to be a release version in the package metadata.

(You can see this by hitting https://www.nuget.org/api/v2/Packages(Id='NLog',Version='4.4.0-beta3') and see the offending tag, <d:IsPrerelease m:type="Edm.Boolean">false</d:IsPrerelease>, which is inconsistent with a semver of 4.4.0-beta3.)

My guess is that there's more than a few projects that have been doing this, prompting the NuGet team to tighten the rules. Unfortunately, the nuget.org metadata hasn't caught up - if you query nuget.org for the latest released version of nlog, you get 4.4.0-beta3. Nexus accepts this as gospel, and happily passes it on in search results to Visual Studio.

Visual Studio 2015, however, enforces the new semver rules, so it prunes NLog out of the search results it gets from Nexus, making it impossible to see NLog unless you tell Visual Studio to show pre-release versions. We'll have to review this policy within Nexus.

Michael

[Michael Prescott]

So, while I was figuring this out, it seems that nuget.org has changed how they're handling these search queries. NLog 4.4.0-beta3 no longer shows up in its feed unless you ask for pre-release versions.

Unfortunately, if you do ever search for nlog with pre-release, the bad metadata gets into nexus and stops it returning NLog 4.2.3 on later searches for release versions, so that's something I'll put on our queue.

Michael

[Ulf Axelsson]

Hmmm, I wonder if you can actually trust that finding. I see the same thing right now when I try from a browser, but reading it jogged my mind. When I first installed M7 I did try this and it worked the way it should with a proxy and VS but then I went and talked to a colleague that uses Xamarin Studio so I could get him to test and he got the "bad" behaviour through the proxy and then I went and tested at my end with VS and got the same thing there and forgot the initial success. 

I wonder if nuget.org is unstable in its behaviour or if Xamarin queried for prereleases (without saying so) and that broke Nexus the way you describe....

BTW, can I clear away the metadata in M7 in some way other than manually selecting each asset and then deleting it? Perhaps just deleteing the repo and recreating would work?

One problem though is that this don't quite explain what happens to queries for the term entityframework, that one too gets lost in the proxy. I can't see anything obviously wrong with https://www.nuget.org/api/v2/Packages(Id='EntityFramework',Version='6.1.3') 

VS2015 through proxy:

WebBackgrounder.EntityFramework 0.1.0

EntityFramework.SqlServerCompact 6.1.3

EntityFramework.Extended 6.1.0.168

VS2015 direct to nuget.org:

EntityFramework 6.1.3

Microsoft.AspNet.Identity.EntityFramework 2.2.1

EntityFramework.SqlServerCompact 6.1.3

/Ulf

[Michael Prescott]

I notice that EntityFramework 7.0.0-beta4 has the same issue - it's got a pre-release semver, but it's not marked as a pre-release version in the metadata.

It could be that Xamarin asked for release packages only, nuget.org passed along 7.0.0-beta4 (which it was doing earlier today), Nexus cached that, and Visual Studio is pruning it.

A couple of things you can do is to look at Nexus' request.log to see what VS is asking it for, which will allow you to look at the XML Nexus is returning to VS.

You can also 'block' your Nexus nuget proxy to prevent it from talking to nuget.org, then use the NX3 M7 Search or Browse features (or even VS with pre-release versions turned on) to see what it's got.

Deleting and re-creating the proxy will definitely wipe away the cached metadata.

Michael

[Ulf Axelsson]

Yes, you are right on that, 7.0.0-beta4 is in the cache.

I dropped the Repo and recreated it and now the really strange things happen. 

I tried querying from VS2015 through the proxy with NLog and EntityFrame as search terms alternating with querying directly to nuget.org.

First try worked fine, second as well, but third time results started vanishing when coming through the proxy. First the top result, then the newly promoted top one on the fourth try until it seemingly stabilised with the same results I described earlier.

Now if I look in the the cache (with outgoing queries blocked for the Proxy repo) the beta:s/rc:s with the incorrect prelease tags are there again.

Unless there is something else wrong somewhere I guess the results from nuget.org are not stable, sometimes the badly tagged packages are in the feed and sometimes they are not... And probably nobody notices because at least VS2015 is filtering the results as well, hiding that issue.

/Ulf

[Ulf Axelsson]

I upgraded to 3.0.0 from m7 and the behaviour did not change.

(I did not expect it to, so just an observation)

/Ulf

[Michael Prescott]

Ulf, thanks for following up. The change was too late to squeak into Nexus 3.0.0 but it's currently queued up for our 3.1 release.

[Ulf Axelsson]

Hi,

Do we know anything more on this? 

https://issues.sonatype.org/browse/NEXUS-10030

Seems to be in some sort of limbo?

/Ulf

[Ulf Axelsson]

This one is still present in 3.7.1.

(https://issues.sonatype.org/browse/NEXUS-10030)

/Ulf