Programmatically perform complex searches using BZ::Bugzilla::Bug::search

[Andrew DeFaria]

The documentation for BZ::Client::Bug::search says merely:

search

my $bugs = BZ::Client::Bug->search($client, $params);

Searches for bugs matching the given parameters.

But doesn't describe what the format for $params is. I've figured out that I can use a hash which name => value pairs and <value> can be an array. When I submit this search I get a list of bugs who's field names match up with the search parameters but everything is considered "equal". IOW status => qw(open, assigned) will find all bugs opened or assigned. But how do I formulate out more complicated queries like say "not open nor assigned". Or any of the other various kinds of conditionals I might wish to query like "contains" or "less than" or "regex"?

Additionally, is there any way to programmatically call a saved search? That would be quite handy in that it could allow the users to control the search that some programmatic process may use.

Thanks in advance...

[Thorsten Schöning]

Guten Tag Andrew DeFaria,
am Montag, 24. März 2014 um 22:04 schrieben Sie:

> The documentation for BZ::Client::Bug::search says merely:

> search

> my $bugs = BZ::Client::Bug->search($client, $params);

> Searches for bugs matching the given parameters.

> But doesn't describe what the format for $params is.

BZ::Client is no official Bugzilla project, therefore you may have
better chances for help contacting the current maintainer or the
projects own support channels. Besides that, BZ::Client can only give
logic the official Webservice-API of Bugzilla provides and those is
documented on it's own:

http://www.bugzilla.org/docs/4.4/en/html/api/Bugzilla/WebService/Bug.html#search

> But how do I formulate out more complicated
> queries like say "not open nor assigned". Or any of the other
> various kinds of conditionals I might wish to query like "contains" or "less than" or "regex"?

Doesn't seem to be possible using the Webservices currently.

Mit freundlichen Grüßen,

Thorsten Schöning

--
Thorsten Schöning E-Mail:Thorsten....@AM-SoFT.de
AM-SoFT IT-Systeme http://www.AM-SoFT.de/

Telefon...........05151- 9468- 55
Fax...............05151- 9468- 88
Mobil..............0178-8 9468- 04

AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow

[Gervase Markham]

On 25/03/14 08:31, Thorsten Schöning wrote:
> Doesn't seem to be possible using the Webservices currently.

Eventually, the WebServices will support fully-powerful search, but I
don't _think_ they do yet. Although that might have changed recently.

The BzAPI proxy does, of course:
https://wiki.mozilla.org/Bugzilla:REST_API

Gerv

[Andrew DeFaria]

On Tuesday, March 25, 2014 12:31:14 AM UTC-7, Thorsten Schöning wrote:

> BZ::Client is no official Bugzilla project, therefore you may have
> better chances for help contacting the current maintainer or the
> projects own support channels. Besides that, BZ::Client can only give
> logic the official Webservice-API of Bugzilla provides and those is
> documented on it's own:
> >http://www.bugzilla.org/docs/4.4/en/html/api/Bugzilla/WebService/Bug.html#search

You're right, it seems woefully inadequate. Still how does one get this Bugzilla::Webserves::Bug? Looking at https://wiki.mozilla.org/Bugzilla:Addons#Client_libraries_for_the_Bugzilla_Webservices.2FREST_API I see it telling me to use BZ::Client (and not even mentioning BZ::Client::Bug, which is much more to the point) for a client side solution. I guess I didn't mention I'm working client side. Actually I see little difference between client and server side API's here. I want the API that can get me the Bugzilla data from wherever I happen to run it. Oh and I'm working in Perl.

> > But how do I formulate out more complicated
> > queries like say "not open nor assigned". Or any of the other
> > various kinds of conditionals I might wish to query like "contains" or "less than" or "regex"?
>
> Doesn't seem to be possible using the Webservices currently.

Indeed. I've worked with 2 REST APIs so far and they both shared the characteristic of 1) being poorly documented and 2) not supporting full search. What's up with that?

[Andrew DeFaria]

From what I can tell from https://wiki.mozilla.org/Bugzilla:Addons#Client_libraries_for_the_Bugzilla_Webservices.2FREST_API BzAPI is also a server side utility. I'm client side or at least I do not wish to only work on the Bugzilla machine. Additionally it's not clear how you get this BzAPI thing. It talks about installing Catalyst and cloning Mecurial(?) repositories with hg? I don't have hg. This smells like a rat hole I don't think I should be going down...

[Thorsten Schöning]

Guten Tag Andrew DeFaria,
am Dienstag, 25. März 2014 um 20:00 schrieben Sie:

> Still how does one get this Bugzilla::Webserves::Bug?

There are different ways of course, one is BZ::Client, others
depending on your environment may exist or you need to create your own
client or whatever.

> Looking at
> https://wiki.mozilla.org/Bugzilla:Addons#Client_libraries_for_the_Bugzilla_Webservices.2FREST_API
> I see it telling me to use BZ::Client (and not even mentioning
> BZ::Client::Bug, which is much more to the point)

Because the wiki can't know your use case. :-)

> for a client side
> solution. I guess I didn't mention I'm working client side. Actually
> I see little difference between client and server side API's here.

That sounds like one of the design goals of BZ::Client, but I'm only
guessing.

> I
> want the API that can get me the Bugzilla data from wherever I
> happen to run it. Oh and I'm working in Perl.

I don't think I understand this paragraph, BZ::Client works wherever
your client is able to access your Bugzilla from. But your problem is
totally different, you want functionality that may or may not be
provided currently by the WebService-API. Check the docs and if it's
not sufficient check the code and maybe provide patches for the docs
for anyone else with a similar problem.

I for example simply don't use the WebServices and therefore can't
tell you if your needed behavior is implemented or not.

> Indeed. I've worked with 2 REST APIs so far and they both shared
> the characteristic of 1) being poorly documented

I don't find the WebServices API of Bugzilla documented that poorly,
if it is you are in a happy position because you just need to check
the implementation and provide patches to improve the docs, because
your behavior is already implemented. If it is not your have a bigger
problem because your functionality needs implementation. :-)

> and 2) not supporting full search. What's up with that?

As often it's a matter of available resources.

[Andrew DeFaria]

>> Still how does one get this Bugzilla::Webserves::Bug?
>
> There are different ways of course, one is BZ::Client, others
> depending on your environment may exist or you need to create your own
> client or whatever.

Name a way then. You see when I say BZ::Client I can do cpan BZ::Client and I will get downloaded actually code that I can run (It's pretty clear I'm workign in Perl no?). However when I say Bugzilla::Webservice::Bug I cannot do cpan Bugzilla::Webservice::Bug and get code because cpan says it can't find anything listed under Bugzilla::Webservice::Bug. Now I know that it's not part of CPAN so what is it part of and how do I get the code?

'Cause you just got through telling me that BZ::Client is not an official Bugzilla project and yet you pointed me to specifically to http://www.bugzilla.org/docs/4.4/en/html/api/Bugzilla/WebService/Bug.html#search which is in bugzilla.org. Was I not to think that that would be "an official Bugzilla project"? Yet there's nothing there that I can see about getting that code!

>> Looking at
>> https://wiki.mozilla.org/Bugzilla:Addons#Client_libraries_for_the_Bugzilla_Webservices.2FREST_API
>
>> I see it telling me to use BZ::Client (and not even mentioning
>> BZ::Client::Bug, which is much more to the point)
>
> Because the wiki can't know your use case. :-)

Looking at CPAN for BZ::Client (http://search.cpan.org/~jwied/BZ-Client-1.04/lib/BZ/Client.pm) we have the following methods:

new
url
user
password
login
api_call

Not exactly a bastion of functionality! The Wiki could anticipate that I might need a bit more functionality for any functional use case! Just my opinion of course :-).

Looking at BZ::Client::Bug (http://search.cpan.org/~jwied/BZ-Client-1.04/lib/BZ/Client/Bug.pm) I see:

CLASS METHODS
get
new
create
search
INSTANCE METHODS
id
alias
assigned_to
component
creation_time
dupe_of
is_open
last_change_time
priority
product
resolution
severity
status
summary

At the very least, couldn't the wiki mention both?

>> for a client side
>> solution. I guess I didn't mention I'm working client side. Actually
>> I see little difference between client and server side API's here.
>
> That sounds like one of the design goals of BZ::Client, but I'm only
> guessing.

You mean a design goal to work on more than one machine? I'd say it's basic functionality but perhaps that's just me.

>> I want the API that can get me the Bugzilla data from wherever I
>> happen to run it. Oh and I'm working in Perl.
>
> I don't think I understand this paragraph, BZ::Client works wherever
> your client is able to access your Bugzilla from. But your problem is
> totally different, you want functionality that may or may not be
> provided currently by the WebService-API. Check the docs and if it's
> not sufficient check the code and maybe provide patches for the docs
> for anyone else with a similar problem.

It looked like you were telling me that BZ::Client was not an official project and that I shouldn't be using it. I realize you were saying that I'd be better off contacting that module's owner, and I have (have not heard back yet). It's just I thought there would be some sort of official or at least full featured way to access the functionality of Bugzilla from scripts, specifically Perl scripts as from what I can tell Bugzilla is largely written in Perl. And as it is quite old in Internet terms I find it surprising there isn't a full featured Perl module that everybody's using already.

So far I've found WWW::Bugzilla and things like WWW::Bugzilla::Bugtree and WWW::Bugzilla::Search. These use some sort of WWW::Mechanize thingy and seems to be horribly inefficient. For example, the new method requires a bugid and takes a while to instantiate. If a script had to process a 1000 bugs and instantiate each one of them at like 4-5 second per instantiation it would take just way too long to get anything done. The Search module's no better. If you have a couple of thousand bugs qualifying then it gathers all of the data, returns a huge structure and takes tens of minutes to return from the search call. I emailed the author about this module but haven't received a response.

I then found BZ::Client and BZ::Client::Bug, which work fairly decently. It's quicker and pretty flexible but got stopped at the lack of search capability.

I see this Bugzilla::Webservice::Bug but have no idea on how to download the code.

I also see this BzAPI thing but it appears to be server side only.

One thing I'm trying to accomplish: We have a number of saved searches. One process we have here executes a number of these and saves the results to .csv files. This is done manually through the web UI. Another Perl script gathers a bunch of these .csv files and produces another .csv file in the form of a report that is useful to management. I'd like to change this Perl process to simply interrogate the Bugzilla database directly, possibly utilizing the saved searches directly and produce the final .csv file directly (or perhaps make a web page instead). There are also many other scripts I could imagine Perl doing by having full functioned access to Bugzilla.

Oh and I checked BZ::Client::Bug::search's code. It descends off in to a plethora of web related technologies that sufficiently obscures what's going on, what's expected and what works WRT search.

> I for example simply don't use the WebServices and therefore can't
> tell you if your needed behavior is implemented or not.

I don't particularly care if it's WebServices or not. I just want to interact with the functionality of Bugzilla programmatically through Perl. The backend can be WebServices, client/server socket stuff or a direct API.

> > Indeed. I've worked with 2 REST APIs so far and they both shared
> > the characteristic of 1) being poorly documented
>
> I don't find the WebServices API of Bugzilla documented that poorly,

Starting from http://www.bugzilla.org/docs/4.4/en/html/api/index.html it's not immediately clear that this is a server side only technology. If it's not a server side only technology then it's not at all clear how to get the client side code. And there's an appalling lack of real world examples.

> if it is you are in a happy position because you just need to check
> the implementation and provide patches to improve the docs, because
> your behavior is already implemented.

You make the incorrect assumption that 1) I understand enough about the code base to make contributions, 2) I have that time and 3) I wish to work for free. I'm a consultant and I need to get stuff done for my client. My client does not pay me to work on open source projects. I'm not totally opposed to working on open source projects nor contributing to them but I don't have the time nor inclination to do so here. Sorry.

> If it is not your have a bigger problem because your functionality needs
> implementation. :-)

Not necessarily. Somebody else could have already implemented this or there may be another solution. That's why I'm posting here. I'm shocked that such a mature technology has such a lack of an interface, supported or not...

>> and 2) not supporting full search. What's up with that?
>
> As often it's a matter of available resources.

I see it as more of a matter of professionalism. I know that I would not release code, open source or not, unless it was not at least reasonably feature complete.

REST APIs seem, in my experience, to use too much web technologies and vagueness, attempting to be language agnostic, that they read as if they say nothing and you need to go elsewhere for answers to simple questions. There's never any good examples of real code, they often don't describe what data is returned other than to say that it's of XML or JSON format - how are multiple records indicated? How empty fields handled? Different field data types? How are exceptions handled? These questions are often not answered. And usually the search portion is "not totally implemented". I think that this is because representing search conditions appears to be difficult to do in a REST scenario (largely because they seem to try to represent the search in terms of data structures like XML instead of simply strings that must be parsed - REST APIs seem to eschew conditional parsers).

[Thorsten Schöning]

Guten Tag Andrew DeFaria,
am Mittwoch, 26. März 2014 um 19:50 schrieben Sie:

> Name a way then.

I did and you already know some.

> However when I say
> Bugzilla::Webservice::Bug I cannot do cpan Bugzilla::Webservice::Bug
> and get code because cpan says it can't find anything listed under
> Bugzilla::Webservice::Bug. Now I know that it's not part of CPAN so
> what is it part of and how do I get the code?

It's a web service API implementing XML-RPC and JSON-RPC, use it
with which ever client you want. BZ::Client is one possible way, but
you can use others as well, XML::RPC for example.

http://www.bugzilla.org/docs/4.4/en/html/api/Bugzilla/WebService.html
http://search.cpan.org/~daan/XML-RPC-0.9/lib/XML/RPC.pm

> 'Cause you just got through telling me that BZ::Client is not an
> official Bugzilla project and yet you pointed me to specifically to
> http://www.bugzilla.org/docs/4.4/en/html/api/Bugzilla/WebService/Bug.html#search
> which is in bugzilla.org. Was I not to think that that would be "an
> official Bugzilla project"? Yet there's nothing there that I can see about getting that code!

Because it's a serve side API and doesn't provide any client side
code. I provided the link because I thought you are using it as an
entry point to read the docs for the API.

> Not exactly a bastion of functionality! The Wiki could anticipate
> that I might need a bit more functionality for any functional use
> case! Just my opinion of course :-).

The wiki'S purpose is to provide pointers to tools or solutions, no
documentation on how to use those.

> It looked like you were telling me that BZ::Client was not an
> official project and that I shouldn't be using it.

I didn't tell you not to use it, I just gave a hint that this is the
support mailing list for Bugzilla and not BZ::Client, therefore it's
less likely you will get help for very specific questions like your
regarding BZ::Client. Of course there are some devs around Bugzilla
reading and answering on this list and some may use BZ::Client as
well, but that doesn't change anything regarding that it's more likely
to get your kind of support from the original project.

> It's just I thought there
> would be some sort of official or at least full featured way to
> access the functionality of Bugzilla from scripts,

There's a documented web service API describing the functionality
available for remote clients, which is not everything Bugzilla is able
to do, but the most important and most frequently requested parts.
Patches welcome of course! ;-)

Besides that and depending on your use case, if you don't need remote
client access you have some additional choices like accessing Bugzilla
natively using it's native "full featured" API and even this can be
done with different ways.

> specifically Perl
> scripts as from what I can tell Bugzilla is largely written in Perl.
> And as it is quite old in Internet terms I find it surprising there
> isn't a full featured Perl module that everybody's using already.

But that's the way it is, simply a matter of resources and other
peoples use cases.

> I see this Bugzilla::Webservice::Bug but have no idea on how to download the code.

There's nothing to download, you need to implement or use a web
service client with the help of a lib of your choice in a language of
your choice.

> I also see this BzAPI thing but it appears to be server side only.

BzAPI is a server side REST API to be able to access Bugzilla remotely
by clients using a REST API instead of XML-RPC or JSON-RPC, which is
what Bugzilla only supports oob. So yes, BzAPI is server side in terms
of installation/administration, but can be used remotely by clients
because it provides a REST service. An no, you can't download code for
BzAPI clients as well, you need to choose how and with what etc. you
want to implement your own REST client capable of using BzAPI.

> One thing I'm trying to accomplish: We have a number of saved
> searches. One process we have here executes a number of these and
> saves the results to .csv files. This is done manually through the
> web UI.

Which is not uncommon, there are quite some tutorials and stuff out
there just automating the web ui for specific tasks.

> Another Perl script gathers a bunch of these .csv files and
> produces another .csv file in the form of a report that is useful to
> management. I'd like to change this Perl process to simply
> interrogate the Bugzilla database directly, possibly utilizing the
> saved searches directly and produce the final .csv file directly (or
> perhaps make a web page instead). There are also many other scripts
> I could imagine Perl doing by having full functioned access to Bugzilla.

None of this sound like you need remote client access, but the logic
might as well run on the same server as Bugzilla does. If this is the
case you might want to access the native Bugzilla API, not the web
service API.

> I don't particularly care if it's WebServices or not. I just want
> to interact with the functionality of Bugzilla programmatically
> through Perl. The backend can be WebServices, client/server socket stuff or a direct API.

You need to care because it makes a huge difference if you are a
remote client or not and have access to the local Bugzilla
installation.

> Starting from
> http://www.bugzilla.org/docs/4.4/en/html/api/index.html it's not
> immediately clear that this is a server side only technology.

Web services are always something a server provides and a client
consumes, which doesn't necessarily say anything about where server
and client are situated, how they communicate etc.

> You make the incorrect assumption that 1) I understand enough about
> the code base to make contributions,

You should contribute to the docs, not the code, because the docs are
what you seem to think needs improvement.

> 2) I have that time and 3) I wish to work for free.

Others created Bugzilla or answer your mails for free. ;-)

> Not necessarily. Somebody else could have already implemented this
> or there may be another solution. That's why I'm posting here. I'm
> shocked that such a mature technology has such a lack of an interface, supported or not...

As I already said, not everyone has your use case.

> I see it as more of a matter of professionalism.

Calm down, you're throwing nonsense like "server side technology"
regarding the web service API, want to download code for or from it or
whatever and don't care how you can access Bugzilla, which is
essentially in knowing which interfaces are available to use. You
really shouldn't tell people here about professionalism if you don't
understand the technologies you want to use and ask for.

> I know that I
> would not release code, open source or not, unless it was not at
> least reasonably feature complete.

Feature complete for whom?

> There's never any good examples of real code, they
> often don't describe what data is returned other than to say that
> it's of XML or JSON format - how are multiple records indicated? How
> empty fields handled? Different field data types? How are exceptions
> handled? These questions are often not answered.

Bugzilla's docs answer most of those, if not all.

[Andrew DeFaria]

On Wednesday, March 26, 2014 3:03:53 PM UTC-7, Thorsten Schöning wrote:

> It's a web service API implementing XML-RPC and JSON-RPC, use it
> with which ever client you want. BZ::Client is one possible way, but
> you can use others as well, XML::RPC for example.

I see, so then there is no full featured binding to the functionality and you have to build your own. Why didn't you say so in the first place.

[Thorsten Schöning]

Guten Tag Andrew DeFaria,
am Freitag, 28. März 2014 um 09:17 schrieben Sie:

> I see,

No you don't.

> so then there is no full featured binding to the
> functionality and you have to build your own. Why didn't you say so
> in the first place.

Complex searches itself are obviously implemented because one can use
them using the Bugzilla Web UI. I never said that those are available
through the web services, instead I advised you to have a look at the
official docs for the web services and their implementation to decide
on your own if the implementation you need is available as a web
service or not. Additionally you have already been told that BzAPI
seems to expose the complex searches as a REST web service.

And besides all of that I never told you to use web services at all or
with which technology, but tried to explain that you maybe don't even
need those, depending on your use case and your use case sounds like
you don't necessarily need them, which would increase your chance to
get your complex searches e.g. using native API or automating the Web
UI or whatever.

HTH?! :-)

[Andrew DeFaria]

On Friday, March 28, 2014 2:11:14 AM UTC-7, Thorsten Schöning wrote:
> Guten Tag Andrew DeFaria,
>
> am Freitag, 28. März 2014 um 09:17 schrieben Sie:

>
> > I see,
>
> No you don't.

Actually no, you don't what I'm saying - and I don't have the time, nor interest in explaining it to you here. Nor do I think it's appropriate for this forum. But if you'd like to hash it out privately then by all means email me.