Please make "wajig search" search the package name AND summary

[Reuben Thomas]

Intuitively, it's a bit odd when you say "wajig search foo", and it fails to display a package that has "foo" in its summary line, because one expects "wajig search" to find anything in the information that is shown, and not only the package names, but also the summaries are shown.

Also, just as a matter of utility. For example, if I'm trying to find a pager program, then "wajig search pager" should turn up "less", "more" and "most".

I understand, on the other hand, why "wajig search" does not match against the entire package description, as that information is not shown as part of the result.

(One could argue from the above that "wajig search -v" should show entire package descriptions, but I am not going to argue that.)

Thinking further, perhaps it might be good to provide a command to search tags, perhaps using axi-cache search --all.

--
http://rrt.sc3d.org

[Tshepang Lekhonkhobe]

On Sun, May 12, 2013 at 1:38 AM, Reuben Thomas <r...@sc3d.org> wrote:
> Intuitively, it's a bit odd when you say "wajig search foo", and it fails to
> display a package that has "foo" in its summary line, because one expects
> "wajig search" to find anything in the information that is shown, and not
> only the package names, but also the summaries are shown.

I agree, but all wajig does is call "apt-cache search" and "apt-cache
search --names-only". Is there an "apt-cache" command that would only
search packages names and their short descriptions? We can do this
ourselves of course, I would rather we wrap existing tools. Also, if
we add this feature, do we kill the older one of searching through the
long description, or do we keep providing it with, for example, "-vv"?

> Also, just as a matter of utility. For example, if I'm trying to find a
> pager program, then "wajig search pager" should turn up "less", "more" and
> "most".

How would one implement such a search? It would be easier if "less"
and "most" provided "pager" as a virtual package. Also, note that
"more" is not a separate package, but part of util-linux.

> Thinking further, perhaps it might be good to provide a command to search
> tags, perhaps using axi-cache search --all.

Sounds like a good idea. How would you prefer the interface? "wajig
search --tag"?

[Reuben Thomas]

On 12 May 2013 23:38, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:

I agree, but all wajig does is call "apt-cache search" and "apt-cache
search --names-only". Is there an  "apt-cache" command that would only
search packages names and their short descriptions?

I can't see such a thing. It's a pity. I wonder if the apt-cache maintainers would be amenable to such a thing.

We can do this
ourselves of course, I would rather we wrap existing tools. Also, if
we add this feature, do we kill the older one of searching through the
long description, or do we keep providing it with, for example, "-vv"?

I would say the most "natural" search is the one that searches names and package descriptions, and that should be the default; -v should mean what it still means, and there should be an option to search names only.

An alternative would simply be to make searching the package descriptions the default, as it is for apt-cache search.

 

How would one implement such a search?

This is easy for the "search" function: less and most mention "pager" in their short description.

It would be easier if "less"
and "most" provided "pager" as a virtual package. Also, note that
"more" is not a separate package, but part of util-linux.

This is a different, and already-solved problem: use a tags search. If any package is not properly tagged, that's easily fixed, as the tags are not held in the packages.

 

Sounds like a good idea. How would you prefer the interface? "wajig
search --tag"?

Actually, I guess it should use debtags; I'd forgotten what the standard interface to tags was, sorry.

As for the interface, it would be nice if by default it would use "debtags" search for a "taggy" looking term, i.e. one that contains double colons, and a command-line flag were added to specifically do an apt-cache search. I'm trying for a design where "wajig search" is reasonably "DWIM".

[Tshepang Lekhonkhobe]

On Mon, May 13, 2013 at 12:53 AM, Reuben Thomas <r...@sc3d.org> wrote:
> An alternative would simply be to make searching the package descriptions
> the default, as it is for apt-cache search.

I don't like it (too many results).

>> How would one implement such a search?
>
>
> This is easy for the "search" function: less and most mention "pager" in
> their short description.

Yeah, that would require changing default to "apt-cache search" (i.e.
current "wajig search --verbose") by default.

>> It would be easier if "less"
>> and "most" provided "pager" as a virtual package. Also, note that
>> "more" is not a separate package, but part of util-linux.
>
>
> This is a different, and already-solved problem: use a tags search. If any
> package is not properly tagged, that's easily fixed, as the tags are not
> held in the packages.

Oh, ok.

>> Sounds like a good idea. How would you prefer the interface? "wajig
>> search --tag"?
>
>
> Actually, I guess it should use debtags; I'd forgotten what the standard
> interface to tags was, sorry.
>
> As for the interface, it would be nice if by default it would use "debtags"
> search for a "taggy" looking term, i.e. one that contains double colons, and
> a command-line flag were added to specifically do an apt-cache search. I'm
> trying for a design where "wajig search" is reasonably "DWIM".

I'm wary of introducing yet-another subcommand. Why don't you want us
just add another command line switch to "search" subcommand? Or do you
mean "wajig search --debtags"?

[Reuben Thomas]

On 13 May 2013 13:40, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:

On Mon, May 13, 2013 at 12:53 AM, Reuben Thomas <r...@sc3d.org> wrote:
> An alternative would simply be to make searching the package descriptions
> the default, as it is for apt-cache search.

I don't like it (too many results).

I agree it's too many, but I would argue that's better than too few (the current default). But let's concentrate on ways to do better.

 

>> How would one implement such a search?
>
>
> This is easy for the "search" function: less and most mention "pager" in
> their short description.

Yeah, that would require changing default to "apt-cache search" (i.e.
current "wajig search --verbose") by default.

But ideally only searching the short description and name, which there appears to be unimplemented. However, it does not take much doing: simply

apt-cache search SEARCH_TERM | grep SEARCH_TERM

 

I'm wary of introducing yet-another subcommand. Why don't you want us
just add another command line switch to "search" subcommand? Or do you
mean "wajig search --debtags"?

Exactly, I don't want any more sub-commands. I am suggesting that by default, wajig search does one of two things:

1. If the pattern contains colons (or some more sensitive test), it does a debtags search.

2. Otherwise, it does a search as above (apt-cache search TERM | grep TERM).

Additionally it has three flags:

--debtags  specify a tags search

--names-only  specify an apt-cache --names-only search

--cache  specify an apt-cache search

--
http://rrt.sc3d.org

[Tshepang Lekhonkhobe]

On Tue, May 14, 2013 at 1:15 AM, Reuben Thomas <r...@sc3d.org> wrote:
> On 13 May 2013 13:40, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:
>>
>> On Mon, May 13, 2013 at 12:53 AM, Reuben Thomas <r...@sc3d.org> wrote:
>> > An alternative would simply be to make searching the package
>> > descriptions
>> > the default, as it is for apt-cache search.
>>
>> I don't like it (too many results).
>
>
> I agree it's too many, but I would argue that's better than too few (the
> current default). But let's concentrate on ways to do better.

Writers of aptitude didn't think so when they made the search only
search package names. I am not sure what the larger number of people
would prefer as default. My intuition tells me that they would go for
lesser results (what you define as "too few"), vs. too many (which is
already available via "-v" option). Obviously your intuition tells you
the opposite. But I agree, searching both package name and short
description is an excellent compromise.

>> I'm wary of introducing yet-another subcommand. Why don't you want us
>> just add another command line switch to "search" subcommand? Or do you
>> mean "wajig search --debtags"?
>
>
> Exactly, I don't want any more sub-commands. I am suggesting that by
> default, wajig search does one of two things:
>
> 1. If the pattern contains colons (or some more sensitive test), it does a
> debtags search.

I don't understand this. Do you have an example?

> 2. Otherwise, it does a search as above (apt-cache search TERM | grep TERM).

Great solution. I will implement (< week).

> Additionally it has three flags:
>
> --debtags specify a tags search

I will implement (< week).

> --names-only specify an apt-cache --names-only search
> --cache specify an apt-cache search

I am actually very tempted to chuck these 2 away. Will anyone miss them?

[Reuben Thomas]

On 14 May 2013 00:48, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:

>

> 1. If the pattern contains colons (or some more sensitive test), it does a
> debtags search.

I don't understand this. Do you have an example?

I am saying that if for example I say

wajig search implemented-in::c

then wajig should act as if I supplied --debtags.

 

> --names-only  specify an apt-cache --names-only search

> --cache  specify an apt-cache search

I am actually very tempted to chuck these 2 away. Will anyone miss them?

The point of --cache is to specify that a pattern which would by default trigger --debtags does not.

I think sometimes one does want to search just names, hence --names-only.

By the way, one thing that I've omitted so far is the possibility to search everything INCLUDING tags. This can make sense for tags which talk about a function, e.g. when searching for programs that do something with mail, "wajig search -v mail" should also find programs tagged works-with::mail. (In an ideal future in which every package is fully tagged, one would just use a tags search, but since we'll never reach that ideal future, we might as well try to work with what we have.) My suggestion is to make -v by default run both apt-cache search and debtags search. If you want to search long descriptions, but not tags, you can use -v --cache.

--
http://rrt.sc3d.org

[Tshepang Lekhonkhobe]

On Tue, May 14, 2013 at 1:15 AM, Reuben Thomas <r...@sc3d.org> wrote:
> But ideally only searching the short description and name, which there
> appears to be unimplemented. However, it does not take much doing: simply
>
> apt-cache search SEARCH_TERM | grep SEARCH_TERM

apt-cache search accepts multiple TERMs, so me wonders if we should
continue supporting that. If so, what would the grep arguments be (my
regex skillz aren't impressive).

[Reuben Thomas]

On 14 May 2013 19:48, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:

apt-cache search accepts multiple TERMs, so me wonders if we should
continue supporting that. If so, what would the grep arguments be (my
regex skillz aren't impressive).

The obvious dumb way would just be to make multiple invocations of the commands. Otherwise, you'd just do

apt-cache search TERM1 TERM2 TERM3 | grep "TERM1\|TERM2\|TERM3"

And I guess some escaping is needed on the regex, but if you do that bit in python, by matching against the output of a pipe, then there will also be a Python escaping routine.

-- 

http://rrt.sc3d.org

[Tshepang Lekhonkhobe]

On Tue, May 14, 2013 at 8:57 PM, Reuben Thomas <r...@sc3d.org> wrote:
> On 14 May 2013 19:48, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:
>>
>>
>> apt-cache search accepts multiple TERMs, so me wonders if we should
>> continue supporting that. If so, what would the grep arguments be (my
>> regex skillz aren't impressive).
>
>
> The obvious dumb way would just be to make multiple invocations of the
> commands. Otherwise, you'd just do
>
> apt-cache search TERM1 TERM2 TERM3 | grep "TERM1\|TERM2\|TERM3"

I went for this grep solution, and pushed it. Can you check if my
implementation is alright.

[Reuben Thomas]

It works fine, but of course I can make it fail by introducing metacharacters into the patterns, e.g.:

$ wajig search "pager|"
/bin/sh: 1: Syntax error: "|" unexpected

It would be better to use subprocess's Popen, but that doesn't fit with perform.execute, so instead, given that you're using Python 3, and that 3.3 will be in the next Debian (and is already in Ubuntu), use shlex.quote (which is new in 3.3). Here's a patch which does that:

diff -r 65106d8dcc60 src/commands.py
--- a/src/commands.py    Tue May 14 22:16:11 2013 +0200
+++ b/src/commands.py    Tue May 14 21:51:24 2013 +0100
@@ -11,6 +11,7 @@
 import subprocess
 import urllib.request
 import webbrowser
+import shlex
 
 import apt
 
@@ -777,12 +778,13 @@
 
 def search(args):
     """Search for package names containing the given pattern"""
+    pats = map(shlex.quote, args.patterns)
     if args.verbose:
-        command = "apt-cache search " + " ".join(args.patterns)
+        command = "apt-cache search " + " ".join(pats)
     else:
         command = "apt-cache search {} | /bin/grep --ignore-case '{}'"
-        command = command.format(" ".join(args.patterns),
-                                 "\|".join(args.patterns))
+        command = command.format(" ".join(pats),
+                                 "\|".join(pats))
     perform.execute(command)
 
 

--- cut here ---

Unfortunately, it doesn't protect against regex errors, but they're much less harmful. For example, in the case above, the empty alternative thus created simply results in every single package being returned; but at least it doesn't make it possible to run malicious shell commands.

--
http://rrt.sc3d.org

[Tshepang Lekhonkhobe]

A better way of doing this is to use a list comprehension, first
because it's more readable, and 2nd because it returns a list, which
you can re-iterate through, as we'll need to in the ''.joins() below.

> if args.verbose:
> - command = "apt-cache search " + " ".join(args.patterns)
> + command = "apt-cache search " + " ".join(pats)
> else:
> command = "apt-cache search {} | /bin/grep --ignore-case '{}'"
> - command = command.format(" ".join(args.patterns),
> - "\|".join(args.patterns))
> + command = command.format(" ".join(pats),
> + "\|".join(pats))

By the time we've reached the 2nd join, pats is already exhausted,
which would result in a grep error... try "grep --ignore-case".

> perform.execute(command)
>
>
> --- cut here ---
>
> Unfortunately, it doesn't protect against regex errors, but they're much
> less harmful. For example, in the case above, the empty alternative thus
> created simply results in every single package being returned; but at least
> it doesn't make it possible to run malicious shell commands.

So, I don't know how you got your code to work. I could have missed
something. Also, can you attach patch files next time (Gmail doesn't
format these things nicely).

[Reuben Thomas]

On 17 May 2013 00:41, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:

A better way of doing this is to use a list comprehension, first
because it's more readable, and 2nd because it returns a list, which
you can re-iterate through, as we'll need to in the ''.joins() below.

>      if args.verbose:
> -        command = "apt-cache search " + " ".join(args.patterns)
> +        command = "apt-cache search " + " ".join(pats)
>      else:
>          command = "apt-cache search {} | /bin/grep --ignore-case '{}'"
> -        command = command.format(" ".join(args.patterns),
> -                                 "\|".join(args.patterns))
> +        command = command.format(" ".join(pats),
> +                                 "\|".join(pats))

Note, map also returns a list, which explains why my code works.

 

Also, can you attach patch files next time (Gmail doesn't
format these things nicely).

Sure, sorry. I get conflicting requests!

--
http://rrt.sc3d.org

[Tshepang Lekhonkhobe]

On Fri, May 17, 2013 at 2:03 AM, Reuben Thomas <r...@sc3d.org> wrote:
> On 17 May 2013 00:41, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:
>>
>>
>> A better way of doing this is to use a list comprehension, first
>> because it's more readable, and 2nd because it returns a list, which
>> you can re-iterate through, as we'll need to in the ''.joins() below.
>>
>> > if args.verbose:
>> > - command = "apt-cache search " + " ".join(args.patterns)
>> > + command = "apt-cache search " + " ".join(pats)
>> > else:
>> > command = "apt-cache search {} | /bin/grep --ignore-case '{}'"
>> > - command = command.format(" ".join(args.patterns),
>> > - "\|".join(args.patterns))
>> > + command = command.format(" ".join(pats),
>> > + "\|".join(pats))
>>
>
> Note, map also returns a list, which explains why my code works.

Not on Python 3 surely?

>> Also, can you attach patch files next time (Gmail doesn't
>> format these things nicely).
>
>
> Sure, sorry. I get conflicting requests!

What requests?

[Reuben Thomas]

On 17 May 2013 17:17, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:

On Fri, May 17, 2013 at 2:03 AM, Reuben Thomas <r...@sc3d.org> wrote:
> On 17 May 2013 00:41, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:
>>
>>
>> A better way of doing this is to use a list comprehension, first
>> because it's more readable, and 2nd because it returns a list, which
>> you can re-iterate through, as we'll need to in the ''.joins() below.
>>
>> >      if args.verbose:
>> > -        command = "apt-cache search " + " ".join(args.patterns)
>> > +        command = "apt-cache search " + " ".join(pats)
>> >      else:
>> >          command = "apt-cache search {} | /bin/grep --ignore-case '{}'"
>> > -        command = command.format(" ".join(args.patterns),
>> > -                                 "\|".join(args.patterns))
>> > +        command = command.format(" ".join(pats),
>> > +                                 "\|".join(pats))
>>
>
> Note, map also returns a list, which explains why my code works.

Not on Python 3 surely?

No indeed. But it seems to work. Have you tested it?

 

>> Also, can you attach patch files next time (Gmail doesn't
>> format these things nicely).
>
>
> Sure, sorry. I get conflicting requests!

What requests?

To send patches as/not as attachments.

[Tshepang Lekhonkhobe]

On Fri, May 17, 2013 at 7:00 PM, Reuben Thomas <r...@sc3d.org> wrote:
> On 17 May 2013 17:17, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:
>> > Note, map also returns a list, which explains why my code works.
>>
>> Not on Python 3 surely?
>
>
> No indeed. But it seems to work. Have you tested it?

I have. Try it in a Python shell:

>>> m = map(str.upper, 'a b c'.split())
>>> list(m)
['A', 'B', 'C']
>>> list(m)
[]

But it's different for Python 2:

>>> m = map(str.upper, 'a b c'.split())
>>> list(m)
['A', 'B', 'C']
>>> list(m)
['A', 'B', 'C']
>>> list(m)
['A', 'B', 'C']

[Reuben Thomas]

On 17 May 2013 23:05, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:

On Fri, May 17, 2013 at 7:00 PM, Reuben Thomas <r...@sc3d.org> wrote:
> On 17 May 2013 17:17, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:

>> > Note, map also returns a list, which explains why my code works.
>>
>> Not on Python 3 surely?
>
>
> No indeed. But it seems to work. Have you tested it?

No, I mean, have you tried running wajig with my patch? (I have read the documentation for Pythons 2 & 3, so I quite understand your point!)

[Tshepang Lekhonkhobe]

I get the grep error I mentioned before, due to the 2nd join()
producing an empty string. I don't know how you got it to work.

[Reuben Thomas]

On 17 May 2013 23:34, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:

I get the grep error I mentioned before, due to the 2nd join()
producing an empty string. I don't know how you got it to work.

It turns out I didn't: "wajig search pager" returns exactly the same result as "apt-cache search pager" with my patch applied. I still don't know why I get no error, though.

By the way, grep seems to be called as both /bin/grep and plain grep (and sometimes egrep, but there's an obvious reason for that!). Is that justified?

--

http://rrt.sc3d.org

[Tshepang Lekhonkhobe]

On Sat, May 18, 2013 at 12:42 AM, Reuben Thomas <r...@sc3d.org> wrote:
> On 17 May 2013 23:34, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:
>>
>> I get the grep error I mentioned before, due to the 2nd join()
>> producing an empty string. I don't know how you got it to work.
>
>
> It turns out I didn't: "wajig search pager" returns exactly the same result
> as "apt-cache search pager" with my patch applied. I still don't know why I
> get no error, though.

Forgive me. It was my fault. I had another change that caused the grep
error. Silly.

Anyways, what is the output of your "wajig search pager --simulate"?

> By the way, grep seems to be called as both /bin/grep and plain grep (and
> sometimes egrep, but there's an obvious reason for that!). Is that
> justified?

It's just historical. You wanna fix the mess?

[Reuben Thomas]

On 18 May 2013 00:14, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:

On Sat, May 18, 2013 at 12:42 AM, Reuben Thomas <r...@sc3d.org> wrote:
> On 17 May 2013 23:34, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:
>>
>> I get the grep error I mentioned before, due to the 2nd join()
>> producing an empty string. I don't know how you got it to work.
>
>
> It turns out I didn't: "wajig search pager" returns exactly the same result
> as "apt-cache search pager" with my patch applied. I still don't know why I
> get no error, though.

Forgive me. It was my fault. I had another change that caused the grep
error. Silly.

Anyways, what is the output of your "wajig search pager --simulate"?

apt-cache search pager | /bin/grep --ignore-case ''

It's just historical. You wanna fix the mess?

Sure, commit the patch for search and I'll then do another patch for this.

--
http://rrt.sc3d.org

[Tshepang Lekhonkhobe]

On Sat, May 18, 2013 at 1:21 AM, Reuben Thomas <r...@sc3d.org> wrote:
> On 18 May 2013 00:14, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:
>>
>> On Sat, May 18, 2013 at 12:42 AM, Reuben Thomas <r...@sc3d.org> wrote:
>> > On 17 May 2013 23:34, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:
>> >>
>> >> I get the grep error I mentioned before, due to the 2nd join()
>> >> producing an empty string. I don't know how you got it to work.
>> >
>> >
>> > It turns out I didn't: "wajig search pager" returns exactly the same
>> > result
>> > as "apt-cache search pager" with my patch applied. I still don't know
>> > why I
>> > get no error, though.
>>
>> Forgive me. It was my fault. I had another change that caused the grep
>> error. Silly.
>>
>> Anyways, what is the output of your "wajig search pager --simulate"?
>
>
> apt-cache search pager | /bin/grep --ignore-case ''

Do you notice the bug? That should be "/bin/grep --ignore-case 'pager'.

>> It's just historical. You wanna fix the mess?
>
>
> Sure, commit the patch for search and I'll then do another patch for this.

Why wait? There wouldn't be any conflict, or would there?

[Reuben Thomas]

On 18 May 2013 00:46, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:

On Sat, May 18, 2013 at 1:21 AM, Reuben Thomas <r...@sc3d.org> wrote:
> On 18 May 2013 00:14, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:
>>
>> On Sat, May 18, 2013 at 12:42 AM, Reuben Thomas <r...@sc3d.org> wrote:
>> > On 17 May 2013 23:34, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:
>> >>
>> >> I get the grep error I mentioned before, due to the 2nd join()
>> >> producing an empty string. I don't know how you got it to work.
>> >
>> >
>> > It turns out I didn't: "wajig search pager" returns exactly the same
>> > result
>> > as "apt-cache search pager" with my patch applied. I still don't know
>> > why I
>> > get no error, though.
>>
>> Forgive me. It was my fault. I had another change that caused the grep
>> error. Silly.
>>
>> Anyways, what is the output of your "wajig search pager --simulate"?
>
>
> apt-cache search pager | /bin/grep --ignore-case ''

Do you notice the bug? That should be "/bin/grep --ignore-case 'pager'.

Sure!

 

>> It's just historical. You wanna fix the mess?
>
>
> Sure, commit the patch for search and I'll then do another patch for this.

Why wait? There wouldn't be any conflict, or would there?

No conflict, but just makes it easier to keep track of things for me. I'll see the update, I'll do the next thing!

[Tshepang Lekhonkhobe]

On Sat, May 18, 2013 at 1:57 AM, Reuben Thomas <r...@sc3d.org> wrote:
> On 18 May 2013 00:46, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:
>> >> Anyways, what is the output of your "wajig search pager --simulate"?
>> >
>> >
>> > apt-cache search pager | /bin/grep --ignore-case ''
>>
>> Do you notice the bug? That should be "/bin/grep --ignore-case 'pager'.
>
>
> Sure!

Not sure what you mean here. Do you accept that using map() is wrong?

[Reuben Thomas]

Yes of course, I didn't need a demo, the fact that it's an iterable in Python 3 is sufficient to convince me!

--
http://rrt.sc3d.org

[Tshepang Lekhonkhobe]

Ok, it wasn't clear to me. Wanted to make sure.

[Tshepang Lekhonkhobe]

I thought a bit about this (making the default search also look in
short package description), and I'm thinking it's too arbitrary a
change, so will revert it. If you want such a match, you can do so on
the CLI. I also disagree with you that it's unintuitive for it to not
match short package description. Forgive me.

[Reuben Thomas]

OK, so how do I search just the titles and short package descriptions? Do I have to do "wajig search -v PATTERN | grep PATTERN"?

--

http://rrt.sc3d.org

[Tshepang Lekhonkhobe]

Indeed.

[Reuben Thomas]

Why not add searching titles & short package descriptions under a flag, and simply not make it the default?

--
http://rrt.sc3d.org

[Tshepang Lekhonkhobe]

Ok. It will be -v, and the current -v be -vv. Will do so soonest.

You are quite persistent :)

I wonder what else should be in the next release too. I'm thinking of
cleaning up the packaging, and upload wajig to PyPI (FWIW).

[Reuben Thomas]

On 18 May 2013 13:43, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:

Ok. It will be -v, and the current -v be -vv. Will do so soonest.

Great!

 

I wonder what else should be in the next release too. I'm thinking of
cleaning up the packaging, and upload wajig to PyPI (FWIW).

 

Sounds good; I'll look at making command invocation consistent. Let me know when you've committed so I can test.

--
http://rrt.sc3d.org

[Tshepang Lekhonkhobe]

'tis done

[Reuben Thomas]

OK, here's a patch that a) regularizes the use of grep (always egrep), and removes all absolute paths from binary invocations.

[Reuben Thomas]

On 20 May 2013 13:30, Reuben Thomas <r...@sc3d.org> wrote:

OK, here's a patch that a) regularizes the use of grep (always egrep), and removes all absolute paths from binary invocations.

Hopefully obviously, I meant "and b) removes…" (insert "b)").

--
http://rrt.sc3d.org

[Tshepang Lekhonkhobe]

Apparently, in code, you only want to use absolute paths when invoking
binaries. It has something to do with security.

Also, the manpage sez egrep is deprecated, in favor of grep -E.

[Reuben Thomas]

On 20 May 2013 15:16, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:

Apparently, in code, you only want to use absolute paths when invoking
binaries. It has something to do with security.

Citation? I find it a pain, because if I want to install my own version of some binary (e.g. get a bug fix in grep) then I don't get the benefit when it is executed by such programs.

I would suggest anything run as root ought to set PATH to a sensible default (e.g. /bin:/usr/bin), and anything else ought to respect the user's settings.

 

Also, the manpage sez egrep is deprecated, in favor of grep -E.

Fine, I'll redo that part.

--
http://rrt.sc3d.org

[Tshepang Lekhonkhobe]

On Mon, May 20, 2013 at 4:29 PM, Reuben Thomas <r...@sc3d.org> wrote:
> On 20 May 2013 15:16, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:
>>
>>
>> Apparently, in code, you only want to use absolute paths when invoking
>> binaries. It has something to do with security.
>
>
> Citation? I find it a pain, because if I want to install my own version of
> some binary (e.g. get a bug fix in grep) then I don't get the benefit when
> it is executed by such programs.
>
> I would suggest anything run as root ought to set PATH to a sensible default
> (e.g. /bin:/usr/bin), and anything else ought to respect the user's
> settings.

I am not so conversant with security topics (Graham can explain
further for he's the one who alerted me to the issue), but how I
understand it is that someone can place an evil executable in your
PATH, and providing an absolute path avoids that risk.

[Reuben Thomas]

On 20 May 2013 23:17, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:

I am not so conversant with security topics (Graham can explain
further for he's the one who alerted me to the issue), but how I
understand it is that someone can place an evil executable in your
PATH, and providing an absolute path avoids that risk.

This argument makes sense for things executed as root. It doesn't make sense for things executed as the user, because the evil executable could be "wajig".

--
http://rrt.sc3d.org

[Tshepang Lekhonkhobe]

I guess you would be doing it interactively right? I was referring to
stuff done in code.

[Reuben Thomas]

If a malicious wajig binary has been placed on my PATH, then any precautions the normal wajig program takes are useless.

If I have installed an improved "grep" on my PATH, I expect it to be used by every program that calls "grep" as a normal user.

[Reuben Thomas]

On the other hand, if you're claiming that calling "grep" and "ls" is an implementation detail of wajig, not something it "promises" the user to do, then hard-wiring the path is acceptable; however, you had better then be certain that those paths actually exist (which seems OK, as both coreutils and grep are Essential: yes/Priority: required).

However, I hope you see the principle here: hard-wiring the paths does not protect against attacks on the wajig binary; and at the same time it removes a valid user configuration mechanism. There is a trade-off. Also, if a bogus grep or ls is installed, the user is likely to have a big problem anyway, since those are both commands frequently used interactively.

Grepping /usr/bin, I can find a handful of instances of "/bin/grep", and many of plain "grep" or "egrep".

I cannot find any advice of this sort in the Debian Policy manual, e.g. http://www.debian.org/doc/debian-policy/ch-files.html#s-scripts

On the contrary, some policy specifically mentions the use of PATH: "Before installation is started, the package management system checks to see if the programs ldconfig, start-stop-daemon, install-info, and update-rc.d can be found via the PATH environment variable."

dpkg-buildpackage explicitly searches PATH for gpg or pgp.

As I said before, I understand hard-wiring paths for commands executed as root, but I'm unconvinced this gains much for commands executed as the user.

One further argument against hardwiring paths: it reduces portability. This is perhaps not so important for wajig, but recall that apt is not only used on Debian any more.

--
http://rrt.sc3d.org

[Graham Williams]

I think the issue is that if grep or ls get replaced in your PATH by a malicious executable - a common use case was when users had "." in their path and they cd into random folders that might have a trojan "grep" that then replaces the grep in wajig if absolute paths are not used.

Graham Williams

http://togaware.com

--
You received this message because you are subscribed to the Google Groups "wajig" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wajig+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at http://groups.google.com/group/wajig?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Reuben Thomas]

On 21 May 2013 02:19, Graham Williams <Graham....@togaware.com> wrote:

I think the issue is that if grep or ls get replaced in your PATH by a malicious executable - a common use case was when users had "." in their path and they cd into random folders that might have a trojan "grep" that then replaces the grep in wajig if absolute paths are not used.

Sure, so wajig should not use the user's PATH to execute commands as root. Standard advice is not to have "." in your PATH, and since Debian doesn't put it there by default, I see no reason to prevent users from shooting themselves in the foot in this way if they wish.

[Tshepang Lekhonkhobe]

I went to look into CPython repo and it appears even there[example],
full paths aren't used, so will have to agree with Graham here. Again,
I don't fully grasp security matters, so could be missing something.
Users putting '.' in their PATH variable should hamper anyone (as it's
not hampering Python core developers).

[example]: http://hg.python.org/cpython/rev/050c94f5f72c

[Tshepang Lekhonkhobe]

s/should/should not

[Reuben Thomas]

On 26 May 2013 12:11, Tshepang Lekhonkhobe <tshe...@gmail.com> wrote:

I went to look into CPython repo and it appears even there[example],
full paths aren't used, so will have to agree with Graham here.

I'm at a bit of a loss here. As you say, CPython doesn't use full paths. But Graham suggested full paths should be used. So I'm not sure what either your conclusion or your reasoning is!

--
http://rrt.sc3d.org

[Tshepang Lekhonkhobe]

Forgive me... s/Graham/Reuben

[Tshepang Lekhonkhobe]

Forgive me... s/Graham/Reuben

[Tshepang Lekhonkhobe]

You bring the patch, I commit.

[Tshepang Lekhonkhobe]

On Sat, May 18, 2013 at 2:43 PM, Tshepang Lekhonkhobe
<tshe...@gmail.com> wrote:
> I wonder what else should be in the next release too. I'm thinking of
> cleaning up the packaging, and upload wajig to PyPI (FWIW).

Ok, uploading this to PyPI feels like a waste of time (and space)
especially given that not even a recent python-apt is there (and the
version there doesn't look official).

[Reuben Thomas]

See my patch earlier in the thread.

--
http://rrt.sc3d.org/

[Tshepang Lekhonkhobe]

On Wed, May 29, 2013 at 11:12 PM, Reuben Thomas <r...@sc3d.org> wrote:
> See my patch earlier in the thread.

You are still to update it.

[Reuben Thomas]

Ah yes. Attached, a first patch, which removes all absolute paths (I believe they were all unnecessary), and changes egrep to grep -E.

Once this is installed, I'll supply another patch which adds absolute paths for root commands (most are currently missing).

[Tshepang Lekhonkhobe]

Pushed, and thanks.

[Reuben Thomas]

Attached, a patch to add full paths to all root commands. Also, remove root=True from a couple of commands that don't need it, and remove an instance of root=False.

--
You received this message because you are subscribed to the Google Groups "wajig" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wajig+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at http://groups.google.com/group/wajig?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

--
http://rrt.sc3d.org

[Tshepang Lekhonkhobe]

On Thu, May 30, 2013 at 12:49 AM, Reuben Thomas <r...@sc3d.org> wrote:
> Attached, a patch to add full paths to all root commands. Also, remove
> root=True from a couple of commands that don't need it, and remove an
> instance of root=False.

Excellent stuff, however I could not find anywhere where you removed
'root=False'.

[Tshepang Lekhonkhobe]

On Thu, May 30, 2013 at 12:49 AM, Reuben Thomas <r...@sc3d.org> wrote:
> Attached, a patch to add full paths to all root commands. Also, remove
> root=True from a couple of commands that don't need it, and remove an
> instance of root=False.

There was one instance you missed where root=True was not needed. I
fixed and pushed, and thanks again.

[Reuben Thomas]

Indeed, that didn't seem to make the final patch. It comes under "fakeroot"; perhaps you can remove it, since it's a very simple patch?

[Tshepang Lekhonkhobe]

> --
> You received this message because you are subscribed to the Google Groups
> "wajig" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to wajig+un...@googlegroups.com.
> To post to this group, send email to wa...@googlegroups.com.
> Visit this group at http://groups.google.com/group/wajig?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.

I is lost. I would not if you simply sent a patch :)

[Reuben Thomas]

Sorry, I thought it was obvious. Attached.

--
http://rrt.sc3d.org

[Tshepang Lekhonkhobe]

On Fri, May 31, 2013 at 12:13 AM, Reuben Thomas <r...@sc3d.org> wrote:
> Sorry, I thought it was obvious. Attached.

I actually thought I searched for root=False, and could not find it.
Maybe I just needed sleep :)

PS: what's up with the top-posting?

[Reuben Thomas]

Sorry, lazy.