Nexus 3 REST API

[Fraser]

My company currently make extensive use of the REST API and the Atlassian Puppet module for v2.11.x but we are anxious to get more information regarding v3 so we can plan migration ASAP. Does anyone know whether any documentation exists (I have emailed Sonatype but haven't heard back from them).

Kind Regards

Fraser.

[Jason Dillon]

Could you explain specifically what you mean by “extensive use of the REST API”?  Or perhaps also explain what you are trying to do with the REST API?

NX3 does not as of *yet* have an official public REST API, but we will bring back a REST API, likely much simpler, focused… and *stable*.

—jason

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/847c993d-409d-4f01-8973-f2ba87d19655%40glists.sonatype.com.
For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.

[Fraser]

As an example, if you are familiar with the Atlassian Puppet module for Nexus, we use pretty much all of the high level resources that it provides. This includes creating repos, groups, assigning repo's to groups, updating the main config settings, things like proxy, email, realms, and so on. The Puppet module is simply a wrapper over the Nexus REST API and allows us to essentially import those abstractions into our Config Management tool of choice (Puppet in this case). There are things that are not currently present in this module (security for example) so that's one area of interest beyond the basic tasks that we would like to use via the API also.

Whilst these are straight-forward capabilities of the UI, keep in mind we are auto-provisioning Nexus servers on the Cloud following an immutable server pattern i.e we create, use and destroy servers at will (we don't expect servers to be long-lived or maintain any state). This gets us to a standard automated 'base build' for Nexus. In addition, we provide tools for our distributed DevOps groups to automate their own repo's, groups, network shares and so on, again mostly using the REST API performing similar operations described above. These also need to be automated since they form part of the ability to recover a server to a defined state quickly, easily and in a repeatable way that we can easily wrap with a Continuous Integration or Continuous Delivery pipeline as/when we want to trigger it.

So, hopefully you can start to appreciate how central the API capability is to us and how important it is that we are able to continue using it as we consider migration to v3.

HTHs

Fraser.

[Jason Dillon]

Thanks, we appreciate your willingness to explain.  

The importance of a REST API or ability to automate provisioning is known and understood.  How that actually ends up becoming reality we are still working out.  We want to ensure whatever we can do here will be stable, not not suffer problems NX2 had in terms of big bloated REST endpoints (which were mainly intended for the UI to consume).

Provisioning however, IMO, is a complex multi-entity/multi-service dance to wire up properly, consistency and sanely… which I’m not sure that REST is very well suited for, unless there was a single end-point to essentially take the full provisioning instructions to apply/commit at once...

One concept I’m considering, which we already use to some degree for development, is to use a script to execute on the server, where that script can lookup the right service abstractions, construct the right configuration and commit it all in one go, w/o a complex request client-driven process.

 * * *

Nothing here is set in stone, but I’m curious if you could imagine a provisioning script as a simpler option to setup your environment as a feasible alternative (in the short-term at least, but perhaps long-term as well) over needing to perform multiple REST requests to configure the instance for your requirements?

FTR this is a script that runs “inside the server” and has full access to all of the public service apis and thus can very quickly and efficiently configure all aspects of the server.  NX3 already has as ‘nexus:groovy’ shell command to facilitate this, refined in upcoming version as ‘nexus:script’ command which can support any javax.script engine available (which is groovy and then whatever your JRE comes with, usually just ‘javascript’, though groovy is our preferred means to script and likely will be the language recommended or used for examples FYI).

I’m 100% sure we’ll be building a REST API as well, but the above technique is nearly immediately useful for automated provisioning… sans the documentation of what the public apis are…. which is also something we are planning on ensuring is stable, documented and comprehensive.

Thanks,

—jason

[Fraser]

> Nothing here is set in stone, but I’m curious if you could imagine a provisioning script as a simpler option to setup your environment as a feasible alternative
> (in the short-term at least, but perhaps long-term as well) over needing to perform multiple REST requests to configure the instance for your requirements?

Absolutely, the smaller the number of moving parts and external dependencies the better.

Are you able to provide more details and some examples ?

A REST API is still a requirement for ad hoc GETs and PUTs though. All kinds of use cases for those.

Fraser.

[Jason Dillon]

On November 21, 2015 at 2:05:28 PM, Fraser (gof...@gmail.com) wrote:

> Nothing here is set in stone, but I’m curious if you could imagine a provisioning script as a simpler option to setup your environment as a feasible alternative 
> (in the short-term at least, but perhaps long-term as well) over needing to perform multiple REST requests to configure the instance for your requirements? 

Absolutely, the smaller the number of moving parts and external dependencies the better. 

Are you able to provide more details and some examples ? 

Nothing official yet, but here is a simple example showing repository creation, for latest 3.0.0-M6 release, to give you a basic idea of the concept:

First you have to enable the local shell (its disabled by default ATM), by editing:

bin/nexus.vmoptions and change the ‘karaf.startLocalConsole’ property to true:

<snip>

-Dkaraf.startLocalConsole=true

</snip>

Then start nexus with the ‘run’ option to get access to the local shell:

<snip>

bin/nexus run

</snip>

Once NX reports its started, you can just hit ENTER to redisplay the “karaf@root()>" prompt, which gets overwritten by boot logging.

Save this snip to a file:

<snip file="<path-to>/create_repo.groovy">

import org.sonatype.nexus.repository.config.Configuration
import org.sonatype.nexus.repository.manager.RepositoryManager

def manager = container.lookup(RepositoryManager.class.name)
manager.create(new Configuration(
    repositoryName: 'test-maven-hosted-release',
    recipeName: 'maven2-hosted',
    online: true,
    attributes: [
        maven: [
            versionPolicy: 'RELEASE',
            layoutPolicy: 'STRICT'
        ],
        storage: [
            blobStoreName: 'default',
            writePolicy: 'ALLOW_ONCE'
        ]
    ]
))

</snip>

And in the shell execute the ‘nexus:groovy’ command:

<snip>

nexus:groovy <path-to>/create_repo.groovy

</snip>

This may burp-up some logging about baseurl you can ignore, and then spits out something like:

<snip>

Name                 test-maven-hosted-release

Type                 hosted

Configuration        Configuration{repositoryName='test-maven-hosted-release', recipeName='maven2-hosted', attributes={maven={versionPolicy=RELEASE, layoutPolicy=STRICT}, storage={blobStoreName=default, writePolicy=ALLOW_ONCE}}}

Format               maven2

</snip>

^^^ above is the Karaf shell trying to interpret the return result of the script, which in this case is the reference to the created Repository object implicitly returned by groovy as the last statements return value.

This is just a simple example showing a single repository creation, but hopefully you could imagine how multiple repos, or other configuration can be applied within a more complex script providing the provisioning logic your environment requires.

We use similar technique to avoid having to go clicking around a ton to re-setup environment for development or testing.  It is as of yet not something we support, but this is along the lines I’m currently thinking for advanced provisioning systems (which we’d likely use ourselves to provision oss.sonatype.org once its updated, to NX3 etc).

Result is after this runs you can see a repository was created in the server, and its now online and ready to serve requests:

A REST API is still a requirement for ad hoc GETs and PUTs though. All kinds of use cases for those. 

Do you have actual use-cases for this?  Can you give me some real examples of what ad-hock GET or PUT you are referring to here please?

Thanks!

—jason

[Fraser]

Thanks for the example. Typically we would want to locate assets off the Nexus server on a network share such as AWS EFS, does the object model that you showed above support over-riding the location as in 2.x ?

As for other use cases beyond the base level install, ... come on Jason, there are any number activities that we might want to perform against Nexus in the context of another application, just like may people do already from 2.x. We may want to upload artefacts, query repos, groups and specific assets, assign metadata to assets, and so on. Some of those applications may have to do with software development, others are required to support security audit and all manner of other processes. Either way, they require integration with Nexus via an API and REST would be our preferred option especially internally where network latency is much less of an issue.

Keep in mind that whilst the most common repo type is likely Maven2, in my case we are a long way from being a Java shop, so all of the other repo types are definitely in play and will likely form the majority of artefacts that we hold. Indeed one of the reasons we are enthusiastic for the GA for v3 are the Docker and 'raw' repo types.

We also make a lot of use of Nuget repos both for in-house Nuget packages that support distributed software development and release, as well as Chocolatey installation packages. Similarly with 'raw' repos for assets that have no specialised type.

All of the above activities (and more) are part of everyday software engineering practice that spans application development to DevOps. Many are implemented via CI workflows and as many as possible are fully automated, so manual interaction with the Nexus application is minimal.

What's the anticipated GA for v3 ?

HTHs

Fraser.

[Jason Dillon]

On November 21, 2015 at 10:48:43 PM, Fraser (gof...@gmail.com) wrote:

Thanks for the example. Typically we would want to locate assets off the Nexus server on a network share such as AWS EFS, does the object model that you showed above support over-riding the location as in 2.x ? 

Yes, in NX3 this is part of the blobstore abstraction.  Example I showed was using the ‘default’ blobstore, to relocate content to another location you would need to configure a new blobstore (or use a previously configured blobstore) and instead of ‘default’ use the name of this blobstore to inform the repository where to store contents.

Indeed one of the reasons we are enthusiastic for the GA for v3 are the Docker and 'raw' repo types. 

Interesting, what about ‘raw’ makes you enthusiastic?  This is a bit of a surprise. 

—jason

[Fraser]

Raw is interesting because we often have assets which we currently host on a separate web server and I would like to bring under the same tech, process and management of other binary assets (i.e. Nexus) rather than look after two separate resources.

Fraser.

[Christoph Henrici]

Hi,

 

I think this a very valid discussion and Fraser is making very good points:

 

a. DevOps requirements for Nexus  - if to be taken seriously - shouldn't be restricted to the Java / Maven perspective. Specifically DevOps needs to be able to move efficiently in heterogeneous world, even if "mainstream" development is done in Java. There is always code you don't want to or can't do in Java.

 

b. DevOps has specific requirements in the supply chain from development to delivery, which necessarily need to be automated and adapted to company specific requirements. Also Nexus  is only one piece in a whole chain of tools, which need to collaborate in the process.  One way to do this would certainly be a Restful API of Nexus , which also would be my first "guess" .....

 

I' m a little astonished that both requirements seem to come as a surprise?

 

So asking the other way around: how does nexus3 envision to  support company specific requirements for automation of provisioning and delivery of artifacts ? The background of my question is that we are planning a major "upgrade" of our build & delivery processes.

 

 

Best regards

Christoph

[Jason Dillon]

On November 23, 2015 at 12:28:53 AM, Christoph Henrici (chhe...@gmail.com) wrote:

Hi,

 

I think this a very valid discussion and Fraser is making very good points:

 

a. DevOps requirements for Nexus  - if to be taken seriously - shouldn't be restricted to the Java / Maven perspective. Specifically DevOps needs to be able to move efficiently in heterogeneous world, even if "mainstream" development is done in Java. There is always code you don't want to or can't do in Java.

 

b. DevOps has specific requirements in the supply chain from development to delivery, which necessarily need to be automated and adapted to company specific requirements. Also Nexus  is only one piece in a whole chain of tools, which need to collaborate in the process.  One way to do this would certainly be a Restful API of Nexus , which also would be my first "guess" .....

 

I' m a little astonished that both requirements seem to come as a surprise?

Its not a surprise.

If you would like to contribute some specific use cases you have please feel free.

—jason

[Grierson, David]

We use the REST API to search for artifacts and retrieve their specific details for inclusion in constructed release notes.

 

I'm presuming that because tools such as Eclipse Maven integration will use those kind of search and retrieval features that this is something which won't be going away in v3.

 

Dg.

---

--
David Grierson - SDLC Tools Specialist

Sky Broadcasting - Customer Business Systems - SDLC Tools

Email: david.g...@sky.uk

Watermark Building, Alba Campus, Livingston, EH54 7HH

 

--

You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/etPan.5652def0.2d6b7c8d.5a1%40Lust.local.

For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.

Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trademarks of Sky plc and Sky International AG and are used under licence. Sky UK Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration No. 2067075) and Sky Subscribers Services Limited (Registration No. 2340150) are direct or indirect subsidiaries of Sky plc (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD.

[Christoph Henrici]

Just one of many usage szenarios: We have a standardized Installer Service on each of our target VM's. Currently the Service, when it get's a update request, pull's "aggregated" build output(Maven assembly) from the file system via ftp, mounted directories etc to install locally. Which is unsatisfactory is several ways: the granularity (Assembly) and the need to store the artifacts to a file system. Ideally we would be able to retrieve the artifacts to be "updated" directly from nexus.

[Christoph Henrici]

And naturally as you can imagine, we are not only installing java artefacts

[Fraser]

The lack of any published details for the HTTP/REST API for Nexus v3.x is making it very difficult for me to plan our migration to v3 (we are in the process of doing that now).

Is there *anything* that you can point me to that describes what services and resources are [likely to be] available and their respective endpoint URIs and payloads, so I can begin this work ?

I don't mind if it's still in alpha/beta whatever, it's just so central to the way that we provision Nexus in our Cloud and in containers (included user defined repos and groups) that its become a blocker issue to making any significant progress (I appreciate the insight into using groovy, but that doesn't really cover all of the use case we have).

At this stage, even just the basics would be enough to move forwards. For example, creating repos, groups, assigning repos to groups, and core configuration (such as setting proxies, work location etc ...).

Very happy to work with Sonatype and this forum in testing and providing feedback for what you have now on the HTTP/REST API and as this progresses through whatever milestone releases remain before GA.

Can you help me out here ?

Regards

Fraser.

[Jeffry Hesse]

Hello there Fraser,

We as of right now don't have any REST capabilities exposed for Nexus Repository Manager 3. As Jason has implied, we are revisiting them, and we are reaching out to customers and users to understand the needs they have. It would be neat to talk 1:1 with you about what problems you are solving with REST. Feel free to reach out directly. I'm loving your enthusiasm :)

As well it's worth stating, OSS GA will not include REST. We will be adding REST immediately thereafter. The reason for delaying is such that we can take a more comprehensive look at how customers are using REST such that we provide the right solution for the future.

Jeffry

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/bbc45d00-bfc7-42d1-88e1-5174703fcafd%40glists.sonatype.com.

[Fraser Goffin]

Hey Jeffry,

Thanks for the quick response. In this thread I have outlined the key
aspects of our current use of the REST API with v2.11. As you can see
this majors in two related areas. The first is auto-provisioning of
base build instances, and the second is auto-provisioning of custom
user repos and groups. As you can see there is a theme here
(auto-provisioning) which is critical to our the way that we
commission and manage immutable servers in our private Cloud (AWS) and
more recently Docker machine images/containers.

TBH I'm shocked that you are intended to delay release of the REST API
until after GA. That would certainly be a major issue for the company
that I work for. Surely while we are in the milestone releases its an
ideal time to collaborate on this requirement and look for an MVP for
GA (as I mentioned earlier, even if that is just the basic resources,
repo, group, config). I'm sure you guys follow Agile approaches to
feature development right ?

Can you give me a contact number so we can discuss the possibilities
for collaboration ?

Kind Regards

Fraser

[Jeffry Hesse]

REST is important without a doubt.  The question we asked ourselves was whether there was value in releasing early and often, even if a comprehensive REST API was not part of that.  Since there is substantial value that people who do not yet require this support can leverage, it makes sense to ship sooner rather than later.

Email sent separately with my contact information.

[Fraser]

Actually I think that question is better put to the Nexus community, although I do understand the position of Sonatype from the support perspective.

That said, I think most of us involved in Agile development would support the principle of release early/often and in some ways this is what the 6 (soon to be 7) milestone releases are about. I would have thought that this was (still is) an ideal opportunity. However from my perspective an equally important consideration, is the idea of a Minimum Viable Product (MVP) and what that should include/exclude. For my organisation the REST API definitely falls within that category even if that means a reduced scope to a core set of resources where there is probably little or no disagreement of there need/value (repository, group, config). Whilst there are alternate approaches, none really hit the sweet-spot for auto-provisioning a base build and support automated [re]-creation of user defined resources in a way that would allow continuation of service and immutability. So much so that we have decided to delay (possibly re-asses) our migration for now (most organisations can't really afford to implement solutions that will likely need to be refactored again in the near term).

I will continue to advocate the absolute requirement for the REST API and for its inclusion as early as possible, especially as we continue to migrate our applications into the Cloud where [perhaps] the need for this level of automation is more apparent. I accept of course that others have different priorities, including Sonatype, but for us it's also about when and where to apply our resources. It appears we are too early to this party !

kind regards

Fraser.

[Da Tong]

Our team maintains a service which support all the product teams to automating build, test, release. Nexus is an important part of our service to act as persistence layer. We are using REST API of NX2 to collect artefacts' metadata and provide statistic dashboard for report.

Without REST API, I have no idea of how to achieve this goal. We are evaluating NX3, while lacking REST API is currently a very critical concern of migrating to new version.

Hope REST API for NX3 will be published ASAP.

[Fraser Goffin]

Absolutely. We also only operate in full automated provisioning mode
for our Cloud and Docker platforms. Whilst Nexus is a core component,
providing external and [critically] internally hosted repos that
support provisioning artefacts, it also is provisioned and configured
in a fully automated fashion using various build tools such as Packer
and Puppet. The REST API is central to maintaining that approach. In
the Cloud, our base images (AMIs if we are talking AWS, although
that's not our only Cloud platform) are cycled regularly (to meet a
number of security, legal, audit and regulatory requirements) so no
instances (including Nexus) have a particularly long 'shelf life'.
Without total automation this would be impossible.

Sonatype have told me that they do not intend to publish the REST API
with the initial GA release. For us, that means we will delay
migration to v3, which causes us a major issue for at least two of the
new repository types that we were/are targeting (raw and docker). For
the latter, we have moved over to Dockers Trusted Registry and for
vanilla binaries we will continue to use Apache/Nginx for now. As time
goes on, the compelling case for unifying our management of assets
under Nexus diminishes.

Fraser.

>>> > On Mon, Dec 7, 2015 at 5:35 AM Fraser <gof...@gmail.com <javascript:>>

>>> >> email to nexus-users...@glists.sonatype.com <javascript:>.

>>> >> To post to this group, send email to nexus...@glists.sonatype.com

>>> <javascript:>.

[Da Tong]

The v2 REST API is not not only bad designed, but also slow. I respect Sonatype's decision of rethink of it carefully. If v3 is going to provide REST API, it should act better then v2.

Actually, to gain sufficient performance and some special function, we are planning to build a matadata database of Nexus.

To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/CAJ6jfGEOrcHOqBw6CbUyW5Fk%2BPVLoG7bchLksiynFBgJZGCuBw%40mail.gmail.com.

[Fraser Goffin]

I'm not suggesting that the v2 API is perfect, few designs in software
are. No my argument is that (a) for an initial MVP release its
shouldn't be too difficult to define the standard set of RESTful verbs
over the core Nexus resources. I would be surprised if that would be
contraversial. That I suspect will satisfy a good many use cases,
certainly nearly all of mine. The other edge cases can come later.

As for performance, that's an entirely separate concern (i.e. It's an
optimisation) that doesn't need to hold back the delivery of the API
which after all is simply an abstraction.

Most of us in our professional life would like a bit more time to
perfect our design and implementation, but most times we need to
deliver against the commercial pressures that exist and work in an
agile way. Abstractions (including APIs) help here. I haven't heard
any compelling argument that suggests such a delay is justified. The
API can't be an after-thought, rather one of the first architectural
constructs surely ?

[Brian Fox]

Hi Fraser, Da,

I appreciate the enthusiasm and discussion. A little more info below:

On Wed, Jan 13, 2016 at 3:23 PM, Fraser Goffin <gof...@gmail.com> wrote:

I'm not suggesting that the v2 API is perfect, few designs in software
are. No my argument is that (a) for an initial MVP release its
shouldn't be too difficult to define the standard set of RESTful verbs
over the core Nexus resources. I would be surprised if that would be
contraversial. That I suspect will satisfy a good many use cases,
certainly nearly all of mine. The other edge cases can come later.

The approach we took with Nexus 1/2 was generally aligned with what you described above. Every resource had a CRUD operation. It was nice that we could say effectively "sure, do anything you want".

The reality was a bit different. The feedback we received was that the REST was too complex for all but the most die hard to really use. By providing a bunch of individual resources, we put the onus on the users to construct cohesive sets of instructions. For many things like provisioning, you really want/need them all to be transactional.

When we build the IQ server as part of our Nexus Lifecycle product line, we intentionally took an alternate approach: We didn't build any public rest apis until we captured and clearly understood the desired use case. As you say, REST apis themselves aren't difficult to provide, but we want to do them in a thoughtful way that doesn't require each user to chain a large string of discrete operations together. The feedback on this approach has been generally well received. The most common request is just for us to add new resources, and we do this constantly as our users understand what they need to do.

This is exactly how we are approaching the api in Nexus Repository Manager v3. It should explain why we haven't just gone off and created a bunch of random apis in the hopes they solve someone's specific problem.

We are keenly aware that people need a way to automate the provisioning and configuration of Nexus. In fact, this is the most common use case of the Nexus v1/2 apis and so we don't need to sit back and let someone tell us about it. This will be the first API created and we are approaching this in a fashion that should allow you to do it all with a single call. (Jason outlined the thinking in previous mails on this list).  

As for performance, that's an entirely separate concern (i.e. It's an
optimisation) that doesn't need to hold back the delivery of the API
which after all is simply an abstraction.

Most of us in our professional life would like a bit more time to
perfect our design and implementation, but most times we need to
deliver against the commercial pressures that exist and work in an
agile way. Abstractions (including APIs) help here. I haven't heard
any compelling argument that suggests such a delay is justified. The
API can't be an after-thought, rather one of the first architectural
constructs surely ?

It's certainly not an after thought, it's very top of mind. It was an important architectural decision to build an API to solve specific use cases rather than blast out lots of bit parts.

 

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/CAJ6jfGFoq1KTiwmtvSROM%3D1QVCpsKRJLZLd5joE1%2BPpqSR-Hwg%40mail.gmail.com.

[Fraser Goffin]

Brian,,

I'm not suggesting that the v2 API is perfectly factored nor that
consideration shouldn't be given to accommodating specific use cases
(such as automated provisioning) in v3. What I would say though is
that to attempt to [over] anticipate user requirements is a fools
errand. The whole purpose of an API is to provide access to underlying
resources at some level of reasonable abstraction that allows not only
for expected use cases but for compositions that were not explicitly
catered for but are equally valid. As soon as you start to (over)
apply specifics you are just adding constraints where before there
were none. I can accept this kind of approach makes sense within an
organisation where consistency is valued over innovation, but since
you have such a diverse user base I would posit that it is less
appropriate. I think what I would suggest is that you provide base
level resources such as those I have previously referred to and, where
you have common use cases, pre-defined compositions that allow for
efficiency. You can think of the former as an interface for advanced
users, a common pattern in many products.

Aside from the technical discourse, there is the question of timing,
which, as we all know, is critical for whether various technologies
succeed or do not, regardless of the sophistication their eventual
capability. My fear is that if you spend too much time polishing up
the v3 GA release (with good intention), you will lose those who would
have been willing to follow the journey (albeit with some 'warts'),
but not prepared to wait for some grand announcement in the unknown
foreseeable future. This is the nature of IT in the modern age.

My 4 cents

Fraser.

[Brian Fox]

On Sun, Jan 17, 2016 at 5:32 PM, Fraser Goffin <gof...@gmail.com> wrote:

Aside from the technical discourse, there is the question of timing,
which, as we all know, is critical for whether various technologies
succeed or do not, regardless of the sophistication their eventual
capability. My fear is that if you spend too much time polishing up
the v3 GA release (with good intention), you will lose those who would
have been willing to follow the journey (albeit with some 'warts'),
but not prepared to wait for some grand announcement in the unknown
foreseeable future. This is the nature of IT in the modern age.

Hi Frasier,

This is exactly what we intend to do. We are not gold plating the 3.0 release, rather we are optimizing for value delivery. We have empirical evidence from many customers who have been going into various forms of production on the milestones, that a fully defined REST is not a minimal feature. 

It is also worth noting that the 3.0 release will not meet everyone's needs out of the box specifically because we are not waiting for some grand announcement of feature parity. 

[Fraser Goffin]

Brian,

As I expected, so how about a v3 MVP that includes a minimal API that
includes just the base resources then ? Surely that minimal set are
not controversial are they ?, and for those who prefer to wait for
more conveniently wrapped resources .. well, as you have suggested,
these will be along in due course, no need to make everyone wait ?

From other threads on this subject and your own comments, I am clearly
not the only one who views this as an essential prior to migration.

Fraser.

[Brian Fox]

On Sun, Jan 17, 2016 at 6:52 PM, Fraser Goffin <gof...@gmail.com> wrote:

Brian,

As I expected, so how about a v3 MVP that includes a minimal API that
includes just the base resources then ? Surely that minimal set are
not controversial are they ?, and for those who prefer to wait for
more conveniently wrapped resources .. well, as you have suggested,
these will be along in due course, no need to make everyone wait ?

I'm not sure why exactly you're so concerned about REST right now? We are still working on critical things like migration from 2 to 3, upgrades and other key features. Having an api for base resources when the rest of this is still being done doesn't really solve a problem. 

From other threads on this subject and your own comments, I am clearly
not the only one who views this as an essential prior to migration.

Yes we are fully aware. We know that the initial 3.0 won't solve everyone's use cases. In fact, it won't solve a lot of them and it's not an accident.

Most people are not unhappy with what Nexus does, they simply want it to do more. We've focused on a lot of stuff in the initial 3.x that Nexus 2.x simply wouldn't do like Docker, NPM v2, HA, etc. In that way, Nexus 3.x's early releases complement 2.x. That is to say: We are not anticipating heavy users of NX2 to upgrade immediately to NX 3.0. Rather what nearly everyone I've talked to about this intends to do is run them together to solve the disparate use cases. We are designing the migration to make this easy to manage. 

Let me state that a different way: We have heavily prioritized functionality in Nexus 3 based on two main things: 

1) making sure the core is capable of doing all the things we have planned. Some of them not yet fully announced but yet highly desirable based on hundreds of conversations with users. 

 - We chose to focus on new formats here for a few reasons: First it was the best way to ensure the core is capable of being truly generic. Second, it provides functionality for users that doesn't exist in Nexus 2 yet.

2) We have prioritized the remaining functionality in a way to deliver the most net new value to users in the most expedient way.

- This means detailed analysis of what functionality is used and required by users and ordering to cover the most ground in the least amount of time. We specifically did not go build every feature just because it was in Nexus 2...because this would have created the situation you talked about earlier where everyone waits for a full complement of functionality when maybe each person only needs 20% of it.

As I said before, automating the provisioning via a REST api is a key focus area for us. It's even possible that 3.0 will have a solution to this. It is also quite possible that 3.0 doesn't have some other key feature you need to fully upgrade your system. We are completely aware of this as well.

[Fraser Goffin]

Well, the word 'critical' depends on who you are asking. None of the
capability that you have mentioned fits into that category for me.

Personally I would have started from the point of view of not removing
existing functionality so as to allow users to migrate at the very
least like-for-like regardless of whether they want new features.
Clearly you have chosen not to, which I find disappointing since it
delays our migration. Whilst I do anticipate some period of dual
running, we certainly don't want to maintain v2 and v3 for an extended
period.

The REST API as I have said many times is absolutely central to our
automated provisioning model. We, as do many others, operate and
immutable server pattern which allows (actually *requires*) us to very
regularly rebuild our Cloud and container based application servers to
comply with a number of security and audit demands.

We too will want to leverage some of the new feed types (interestingly
'raw' is close to the top of our list, Docker less so now, that ship
has now sailed).

Anyway, I'll maintain an active watch on progress. At some point our
requirements may be fulfilled, I've clearly laboured my position long
enough.

Kind Regards

Fraser.

[Dave Cowden]

+1 for a REST api.

By way of listing specific requirements and/or why we need the REST api, here's our use case.

When we build products, like everyone else using maven, we use a X.Y.Z numbering scheme.

Most people use the Jenkins build number for the Z when they are using CD. Or, they use the maven release plugin. But we do neither.

We view nexus as the 'single source of truth' about what versions are there. So what we do is to run a small script before each build that asks nexus what the current, latest version of the artifact is in nexus, then, we increment that number to do the next build.

This provides a number of advantages:

   * we dont have a bunch of commits for each build [ maven release plugin]

   * if the build fails, we dont skip version numbers-- we'll only consume a version number when we actually deploy

   * multiple builds on multiple branches dont get screwed up

I should mention that the 2.X api was not ideal for us to accomplish this. We need to find the latest version of a particular GA coordinate.  We found that neither none of these endpoints worked for us, which would have been a single call:

/service/local/artifact/maven

/service/local/artifact/maven/content

/service/local/artifact/maven/resolve

These supported a v=latest value, but would fail when the project was a pom-only project or when i didnt know the jar file i wanted, etc. 

We ended up brute forcing it with this endpoint:

/service/local/lucene/search

with this endpoint, we can find all of the artifiacts for a given GA coordinate, and then find the latest. But the results of this endpoint didnt make it easy for us to do it at all:

  * the data structure did not contain a timestamp, 

  * the endpoint doesnt allow explicitly ordering the results in any particular order

We ended up just listing all of the artifacts, and ASSUMING that they were listed in newest-to-oldest order, which happens to work. 

In short, what would be great is if the NX3 rest api makes it easy to answer the question: 'what is the current, latest version of an artifact with a GA coordinate?

On Saturday, November 21, 2015 at 11:57:29 AM UTC-5, Fraser wrote:

My company currently make extensive use of the REST API and the Atlassian Puppet module for v2.11.x but we are anxious to get more information regarding v3 so we can plan migration ASAP. Does anyone know whether any documentation exists (I have emailed Sonatype but haven't heard back from them).

Kind Regards

Fraser.

[Mark]

+1 for the REST API.

We use the REST API with Jenkins to deploy to Tomcat.

Here's an overview of how that works.

1. Jenkins queries Nexus for a list of artifact versions for a particular artifact

2. Jenkins presents this in a list to the user

3. The user selects which version to deploy

4. Based on the version, this may restrict the target hosts - SNAPSHOTS don't go into production, etc.

5. The user deploys the artifact to Tomcat using the Tomcat Maven plugin, and the build number for a parallel deploy

6. Jenkins sends interested parties email

Without the API, getting an up to date list of artifact versions becomes problematic.

Without a mechanism like this, deployment, rollback, and notification becomes a manual process, subject to greater operator error.

Mark

/mde/

[David Karlsen]

This is interesting.
How do you do #2 in jenkins?

> --
> You received this message because you are subscribed to the Google Groups
> "Nexus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nexus-users...@glists.sonatype.com.
> To post to this group, send email to nexus...@glists.sonatype.com.
> To view this discussion on the web visit

> https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/c61f689c-586f-4524-b46a-aaf85d4f0e26%40glists.sonatype.com.

>
> For more options, visit
> https://groups.google.com/a/glists.sonatype.com/d/optout.



--

--
David J. M. Karlsen - http://www.linkedin.com/in/davidkarlsen

[Mark]

I use the Active Choices Plugin for Jenkins.

I then wrote some groovy code that grabs the current versions of a selected groupId:artifactID, sorts the results, and returns it to the user interface as a drop down list.

A snippet of the groovy code:

  // get all available SNAPSHOT versions

  try {

   def scontent = new XmlParser().parseText(new URL(locations).getText());

   for (contentNode in scontent.data.'content-item') {

     if (contentNode.leaf.text() != "true") {

       versions.add(contentNode.text.text())

     }

   }

  } catch (FileNotFoundException e) {

  }

locations is the URL of the SNAPSHOT repository plus the groupId (periods replaced with /) plus the artifactId.

I then filter based on what the user selects (TEST, UAT, PRODUCTION), and don't show snapshots for production machines. I finally do a reverse sort of the versions so that the newest is displayed first (and is the default) in the drop down list.

There's a lot more to the Jenkins job, but this is the only place where I access the Nexus REST API.

[Michael Süß]

+1 for the REST API.

As mentioned elsewhere, we provide a central platform hosting Jenkins, SonarQube and Nexus. For reducing administration efforts, we have written a small administration service that accesses the REST services of those tools.
The main use case, where the REST API is used, is to onboard projects. We use it to create the default set of repositories and configure their access.

At the moment we are using following api endpoints:
/privileges  -> GET
/privileges_target  -> POST
/privileges/{id}  -> DELETE
/roles  -> GET, POST
/roles/{id}  -> GET, PUT, DELETE
/repositories  -> GET, POST
/repositories/{id}  -> GET, PUT, DELETE
/repo_groups  -> GET, POST
/repo_groups/{id}  -> GET, PUT, DELETE

I hope that the new REST API will be better documented than the old one (especially regarding the accepted json structures).

Best regards,

Michael

[Manfred Moser]

A lot of this can be achieved with the new scripting API that is
coming out with 3.0 this month. Check out the docs for a preview.

http://books.sonatype.com/nexus-book/3.0/reference/scripting.html

Manfred

> --
> You received this message because you are subscribed to the Google Groups
> "Nexus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nexus-users...@glists.sonatype.com.
> To post to this group, send email to nexus...@glists.sonatype.com.
> To view this discussion on the web visit

> https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/9834f165-b453-42d8-8af9-e77d1c82b557%40glists.sonatype.com.

[Mark]

Manfred,

I've looked at the provision.sh script on github and scanned the link you provided. It appears that you can run Groovy scripts on the repository.

What's not clear from a cursory reading is the following:

1. Can I send parameters to the installed Groovy script?

2. Can the Groovy script return information to the caller?

Without these two, I don't see how this implementation will help me with my current automation task (deploy job on Jenkins querying Nexus for available artifact versions).

Mark

/mde/

[Manfred Moser]

This is just an initial simple scripting and not aimed at solving your
particular situation of querying for artifact versions.

However you can do a lot of the provisioning of repositories and users and such.

The scripts can receive parameters and thats documented in the chapter
and there is an example in the examples git repository.

Results are limited to success/failure at the moment.

Manfred

> https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/ba30175e-cc2e-4677-937e-7cce91d407ad%40glists.sonatype.com.

[sidd rc]

Hi,

   I am new to Nexus 3.0, however I have read the documentation and I understand the fact that groovy scripts will be needed to run so as to perform any task via application.

   I understand that the Nexus REST frameworks has exposed core,repository etc objects for the interaction my query is as below

   1. Is there a proper documentation page for the objects exposed so as to see the methods available.

   2. The reason I ask this I need to query Nexus to fetch me the artifacts present in a particular repository, I am unable to make as to which method would I have to call on the repository object to achieve this.

   3. I hope this is achievable :)

  Any help would be really appreciated, thank you.

-Siddharth

  

    

[Manfred Moser]

The best help is available directly in your IntelliJ IDEA. Check out
the videos with my recent blog post to see it in action.

http://www.sonatype.org/nexus/2016/06/08/integrating-nexus-repository-3/

The API is currently in a V1 state and we would love to hear from you
with regards to further enhancements to it.

In terms of fetching artifacts you would just download them with a
normal HTTP GET from the usual URLs. Finding them is another question.
Currently there is no search available as part of the API.

Manfred

> --
> You received this message because you are subscribed to the Google Groups
> "Nexus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nexus-users...@glists.sonatype.com.
> To post to this group, send email to nexus...@glists.sonatype.com.
> To view this discussion on the web visit

> https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/1909cd88-6ddc-4b19-9bb6-af96655a57ab%40glists.sonatype.com.

[sidd rc]

Thanks for the quick reply Manfred,

My use case here is I need to query Nexus and supply it with a repository name and artifact name and in return I expect to find out how many artifacts with that name are present in the repository and I need to store them either as file paths or as blob files in my other database and later control them as to direct them to some ABC deploy server box.

Here I am trying to put them in a seperate database so that I can see and control them via some other application or 

the other approach would be to tell my CI server to directly deploy after uploading in Nexus.

Because in either case I need my artifact to be deployed in any case.

What do you think would be the best approach.

Thanks

Siddharth

[Manfred Moser]

I would wire your deployment tool to get them directly from the Nexus
Repository Manager instead of pushing them over into yet another tool.

So ...

- build runs and deploys component to repository - location is recorded somehow
- your testing and whatever (automated or not) goes through the process
- if it all passes it kicks of a deployment tool feeding it the
location in the repo manager
- it download from the repo and deploys to wherever

Mnafred

> --
> You received this message because you are subscribed to the Google Groups
> "Nexus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nexus-users...@glists.sonatype.com.
> To post to this group, send email to nexus...@glists.sonatype.com.
> To view this discussion on the web visit

> https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/2b1099b3-bb94-48a1-906a-e35c37d43098%40glists.sonatype.com.

[sidd rc]

Hi Manfred,

   Any ETA on the next version of the API, and what would features would it be able to provide.

Thanks

Siddharth 

[Manfred Moser]

The upcoming 3.1 release focusses on migration from 2.x to 3.x and Nexus Pro features in 3.x. Currently the plan is for a following release later this year to introduce the new REST API. Feature wise we welcome any suggestions or reqirements from your end. We already have a pretty good idea what is needed from usage of the old API as well as the existing scripting API that is ready to be used now. You should check it out.

http://www.sonatype.org/nexus/2016/06/08/integrating-nexus-repository-3/

http://books.sonatype.com/nexus-book/3.0/reference/scripting.html

Manfred

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/aa8eaa11-3e68-4831-8856-d8da31b7b989%40glists.sonatype.com.

[sidd rc]

Hi Manfred,

   Thank you for your reply and I really appreciate the effort that has been put in making Nexus what it is today.

   Also I have visited the two links you have mentioned but for now they will not be able to help me, and that is why I am dialing down to Nexus 2.11 because the REST API is 

  1. Well documented

  2. Is like 90% close to what I want to achieve.

  I really wish all the best to the Nexus team for Nexus3 API and I really like the approach for making the client a 1st class consumer and querying the application via groovy scripts   but I also understand that doing this wont be easy a lot of thought,energy and time will have to be put in into understanding what to expose and its consequences and this will take time but none the less we will all be patient and wait Nexus 3 v1 API to come out :)

Thanks

Siddharth

[Fraser Goffin]

I have petitioned Sonatype for over a year to introduce the REST API
since this is the primary feature that prevent the company that I work
for from moving to v3. I have suggested many times a minimal feature
set that would allow 80%+ of basic automation and integration
requirements to be met as an MVP before the need to consider other
edge cases. I have represented those views with our account manager. I
recently watched a v3 webcast where time and again Sonatype repeated
the mantra of '.. Come and tell us what you want, we really want to
know'. Unfortunately the reality has been very different, Sonatype are
clearly following their own development path and have already
determined the priority order. TBH it just feels like ST have stopped
listening. Maybe next year.

> --
> You received this message because you are subscribed to the Google Groups
> "Nexus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nexus-users...@glists.sonatype.com.
> To post to this group, send email to nexus...@glists.sonatype.com.
> To view this discussion on the web visit

> https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/cd8c5f8b-4bf5-4604-9e53-4768f14d44d5%40glists.sonatype.com.

[Brian Fox]

Come on now Fraser, we've explained the theory here multiple times. This is a sequencing issue above all else.. We've been focused on providing migration mechanisms for the content, which defines the 3.1 release. The vast majority of users do not require a REST API, but 100% will require migration. After migration, the REST API becomes much closer to the top of the list. I've mentioned before that we don't intend to go off and create CRUD resources for every object, rather we are listening and collecting use cases that will inform which APIs get created.

Overwhelmingly, the people that are using REST, are doing so to automate provisioning. Yet everyone is doing it a little differently, which is exactly why we created the groovy provisioning api. Some users have even gone off and created and shared scripts showing that this is in fact effective.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/CAJ6jfGHaqXBwijT5cC-_CRQFW5HCX_KKBQDLFXar0F0Qoa4dYA%40mail.gmail.com.

[Fraser Goffin]

I whole-heartedly agree that migration is an absolute must have, and
frankly was shocked that it wasn't included with the 3.0 release,
although that would likely have required a higher level of feature
parity.

I'm all for expansion of the range of features but personally would
have liked to have seen a focus on like-for-like existing
functionality before moving on to other areas. I don't suppose 100% of
users needed a docker registry or support for some of the other new
feed types, but they made it in (and judging by the recent webcast
will continue to be added at a pace ?).Actually better Yum support
would be my first choice.

I guess I was looking for an upgrade that made it easy to move to
without having to maintain two versions for an indeterminate period,
unless that is, you are willing to share more specifically the roadmap
timetable ? It's clear from the questions that frequently come up,
that the community is still unsure when each of our items of interest
will land and 'xxx is nearing the top of the list' doesn't really work
for my bosses and nor I would guess, anyone else's.

I don't recall ever asking for CRUD for all objects and certainly
don't need that either. The groovy approach is OK up to a point. It's
all we have for now, but like many things it's hard to reason about
how much effort to expend on something without having much of an idea
when other (perhaps preferable) approaches might be available.

Perhaps if you could expand on how you view the primary use cases for
the functionality provided by Groovy scripting and the TBA REST API
are differentiated. Maybe that will help me understand better the
priorities that have been chosen ?

Regards

Fraser

[Brian Fox]

On Tue, Jun 21, 2016 at 4:07 PM, Fraser Goffin <gof...@gmail.com> wrote:

I whole-heartedly agree that migration is an absolute must have, and
frankly was shocked that it wasn't included with the 3.0 release,
although that would likely have required a higher level of feature
parity.

That was part of it..

 

I'm all for expansion of the range of features but personally would
have liked to have seen a focus on like-for-like existing
functionality before moving on to other areas. I don't suppose 100% of
users needed a docker registry or support for some of the other new
feed types, but they made it in (and judging by the recent webcast
will continue to be added at a pace ?).Actually better Yum support
would be my first choice.

Support for more formats was the single largest request, and one that was best satisfied with 3.x. It didn't make sense to hold back adding new capabilities while most people were largely content with what Nexus 2.x did. That is so say, people wanted more capabilities, but for the most part, the capabilities already present in 2.x were working just fine.

 

I guess I was looking for an upgrade that made it easy to move to
without having to maintain two versions for an indeterminate period,
unless that is, you are willing to share more specifically the roadmap
timetable ? It's  clear from the questions that frequently come up,
that the community is still unsure when each of our items of interest
will land and 'xxx is nearing the top of the list' doesn't really work
for my bosses and nor I would guess, anyone else's.

I don't recall ever asking for CRUD for all objects and certainly
don't need that either. The groovy approach is OK up to a point. It's
all we have for now, but like many things it's hard to reason about
how much effort to expend on something without having much of an idea
when other (perhaps preferable) approaches might be available.

Perhaps if you could expand on how you view the primary use cases for
the functionality provided by Groovy scripting and the TBA REST API
are differentiated. Maybe that will help me understand better the
priorities that have been chosen ?

The Groovy scripting is largely intended to allow people to roll their own form of integration api. A formal REST api will cover the most common use cases, but in a way that makes their usage easy and not cumbersome, which likely means the objects being manipulated are higher level than what was done in 2.x

[jhe...@sonatype.com]

To follow up, the REST API is now in BETA, and you can give us some GOOD FEEDBACK about it!

Go here to get some more info: http://blog.sonatype.com/nexus-repository-new-beta-rest-api-for-content

Thanks for all those who were patient while we worked on this, it's appreciated, totally. Writing this in 2017 when this email is from 2015 is making me feel a bit nauseous, so you know, I 100% mean what I said in the previous sentence.

Jeffry