What am I doing wrong?

[Aron Griffis]

I'm testing against http://libproxy.googlecode.com/svn/trunk
revision 508

$ cmake .
-- checking for module 'xulrunner-js'
-- package 'xulrunner-js' not found
-- checking for module 'firefox-js'
-- package 'firefox-js' not found
-- checking for module 'mozilla-js'
-- found mozilla-js, version 1.9.1.7
-- checking for module 'seamonkey-js'
-- package 'seamonkey-js' not found
MODULES TO BUILD:
m config_wpad
+ config_envvar
+ config_file
m config_gnome
config_kde4
m network_networkmanager
pacrunner_mozjs
m pacrunner_webkit
+ ignore_domain
+ ignore_ip
+ wpad_dns_alias

-- Configuring done
-- Generating done
-- Build files have been written to: /home/agriffis/proj/libproxy-read-only

First thing I notice above is that it found mozilla-js but
apparently isn't building against it. No problem, it found
webkit, and that should work.

$ make
[snipped expected output]

$ cd utils

$ ldd proxy
linux-vdso.so.1 => (0x00007fff931ff000)
libproxy.so.0.0.0 => /home/agriffis/proj/libproxy-read-only/libproxy/libproxy.so.0.0.0 (0x00007ff767a81000)
libm.so.6 => /lib/libm.so.6 (0x00007ff7677fd000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007ff7675e1000)
libdl.so.2 => /lib/libdl.so.2 (0x00007ff7673dd000)
libc.so.6 => /lib/libc.so.6 (0x00007ff76706e000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ff766d5e000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007ff766b47000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff767ce2000)

All good so far. But nothing I try actually works right. Here's
one example, rather than bore you with the long list of things
I tried.

$ cat test.pac
function FindProxyForURL(url, host) { return "PROXY yayitworked:8080"; }

$ PX_CONFIG_ORDER=config_envvar \
http_proxy="pac+file://$PWD/test.pac" ./proxy http://google.com
direct://

What am I doing wrong?

Thanks,
Aron

[Aron Griffis]

(Updated to rev 509)

Not even this is working:

$ PX_CONFIG_ORDER=config_envvar http_proxy=http://foo:8080 \

[Nathaniel McCallum]

Trunk is going through a lot of flux right now. I'll see if I can track
down this issue tonight.

One of the things that I am working on is a mechanism to automatically
detect which javascript engine an app linking against libproxy is using
and loading the correct pacrunner. So the core module code is going to
be changing some in the coming days. The end result is that an
application will no longer have to specify a preference for javascript
engine.

Another coming change is that we will check for "loadability" without
sucking in dependencies into the process space. This should save memory
and load time.

Nathaniel

[Nathaniel McCallum]

On 01/28/2010 10:18 AM, Aron Griffis wrote:

This is fixed now in r511.

Nathaniel

[Aron Griffis]

Nathaniel McCallum wrote: [Fri Jan 29 2010, 12:51:18AM EST]

I tried r511 and then r518, neither changes this behavior for me.
I'm getting the example same results I described.

Thanks,
Aron

[Nathaniel McCallum]

$ http_proxy=http://foo.com:8080 LD_LIBRARY_PATH=/usr/local/lib64/
/usr/local/bin/proxy http://www.google.com
http://foo.com:8080

That is from r518.

However, there is a big caveat here. As of r513 PX_CONFIG_ORDER is gone.
config_file and config_wpad are gone as well. I meant to write up the
reasoning here in an email, but didn't get around to it. So here it goes.

Config modules were a source of code complexity. When I originally
designed the code I wanted the system admin to be able to lock libproxy
to a certain configuration. Thus, I made provisions for PX_CONFIG_ORDER
and /etc/proxy.conf which would allow admins to hard-code the
configuration resolution. To summarize, I traded code complexity for
lock-down support.

Enter multi-platform libproxy (it now more or less runs fully supported
on UNIX/Linux/Mac/Windows). We currently have the following config modules:
1. config_envvar - provides backwards compatibility on UNIX/Linux
2. config_gnome - GNOME/gconf support
3. config_kde4 - KDE4/Kconfig support
4. config_macosx - Mac OS X support
5. config_w32reg - Windows support

gconf, kconfig, macosx and windows registry all already provide some
form of lock-down support. So lockdown is supported in all the major
desktop environments. That left envvar. The answer for envvar is found
in read-only environment variables. Thus, lock-down is possible without
the need for libproxy to implement it, but requires libproxy to enforce
which config plugin is chosen. Thus PX_CONFIG_ORDER must be removed.

The end result is no feature loss, less code, less blocking-io (a good
thing!) and smaller binary size. Pretty much a win all around.

I still have yet to figure out how to re-create the test suite. The
actual module loading code is now split out into a separate library and
contains a basic test suite. I need to create roughly the same thing
for libproxy proper.

In short, r518 should be working (and is for me), but you won't be able
to specify a config module manually (without removing/not building the
module for your DE).

One last thought. The envvar for BLACKLIST/WHITELIST is gone too. In
its place we now autodetect if the linking application has already
linked to webkit/mozjs and load the correct pacrunner. Our end goal is
to always just do the right thing automatically.

If you have any questions, let me know.

Nathaniel

[Aron Griffis]

Nathaniel McCallum wrote: [Fri Feb 05 2010, 08:13:17PM EST]

>Enter multi-platform libproxy (it now more or less runs fully
>supported on UNIX/Linux/Mac/Windows). We currently have the
>following config modules:
> 1. config_envvar - provides backwards compatibility on UNIX/Linux
> 2. config_gnome - GNOME/gconf support
> 3. config_kde4 - KDE4/Kconfig support
> 4. config_macosx - Mac OS X support
> 5. config_w32reg - Windows support
>
>gconf, kconfig, macosx and windows registry all already provide some
>form of lock-down support. So lockdown is supported in all the major
>desktop environments.

Makes sense. Leave it to the DE to lock down rather than
needing to provide that ability in libproxy.

>That left envvar. The answer for envvar is
>found in read-only environment variables.

What do you mean by read-only env vars?

>Thus, lock-down is
>possible without the need for libproxy to implement it, but requires
>libproxy to enforce which config plugin is chosen. Thus
>PX_CONFIG_ORDER must be removed.
>
>The end result is no feature loss, less code, less blocking-io (a
>good thing!) and smaller binary size. Pretty much a win all around.
>
>I still have yet to figure out how to re-create the test suite. The
>actual module loading code is now split out into a separate library
>and contains a basic test suite. I need to create roughly the same
>thing for libproxy proper.
>
>In short, r518 should be working (and is for me), but you won't be
>able to specify a config module manually (without removing/not
>building the module for your DE).
>
>One last thought. The envvar for BLACKLIST/WHITELIST is gone too.
>In its place we now autodetect if the linking application has already
>linked to webkit/mozjs and load the correct pacrunner. Our end goal
>is to always just do the right thing automatically.
>
>If you have any questions, let me know.

Thanks, this is all helpful to know. Unfortunately the "proxy"
utility program continues to return "direct://" no matter what
I do. Based on what you said above, I've tried various settings
in gnome-network-preferences, in case it was overriding the
environment variable, but nothing changes.

Here's my current cmake output, for review:

$ cmake .
-- checking for module 'xulrunner-js'
-- package 'xulrunner-js' not found
-- checking for module 'firefox-js'
-- package 'firefox-js' not found
-- checking for module 'mozilla-js'
-- found mozilla-js, version 1.9.1.7
-- checking for module 'seamonkey-js'
-- package 'seamonkey-js' not found
MODULES TO BUILD:

* config_envvar
* config_gnome
config_kde4
config_macosx
config_w32reg
* ignore_domain
* ignore_hostname
* ignore_ip
* network_networkmanager
pacrunner_mozjs
* pacrunner_webkit
* wpad_dns_alias

Thanks,
Aron

[Nathaniel McCallum]

On 02/05/2010 09:07 PM, Aron Griffis wrote:
> Nathaniel McCallum wrote: [Fri Feb 05 2010, 08:13:17PM EST]

>> That left envvar. The answer for envvar is
>> found in read-only environment variables.
>
> What do you mean by read-only env vars?

$ readonly NAME=nathaniel
$ NAME=aron
bash: NAME: readonly variable
$ unset NAME
bash: unset: NAME: cannot unset: readonly variable

Thus, as root you can do:
echo "readonly http_proxy=http://proxy:8080" >> /etc/profile

This locks down the proxy for all users. This works on most (all?
posix?) shells.

> Thanks, this is all helpful to know. Unfortunately the "proxy"
> utility program continues to return "direct://" no matter what
> I do. Based on what you said above, I've tried various settings
> in gnome-network-preferences, in case it was overriding the
> environment variable, but nothing changes.

Ok we might have a problem then.

A common problem I've run into is that Fedora (which I'm developing on)
has libproxy already installed. Not a problem, I just build in another
directory. However, occasionally I forget to put that directory in
LD_LIBRARY_PATH before I execute proxy. Thus, it uses the old library
(since they are binary compatible) and I get results I'm not expecting.
This might not be your problem, but it was for me.

Probably the most helpful next step is to add a printf() line in
proxy.cpp. See my note here (you'll have to scroll down):
http://code.google.com/p/libproxy/source/browse/trunk/libproxy/proxy.cpp?r=518

If neither of those work, let me know and I'll put a bunch of debug
printf()s in activated by an envvar (I'll do this anyway before the next
release, but I'll bump up the priority if you need it).

Nathaniel

[Aron Griffis]

>>What do you mean by read-only env vars?
>
>$ readonly NAME=nathaniel
>$ NAME=aron
>bash: NAME: readonly variable
>$ unset NAME
>bash: unset: NAME: cannot unset: readonly variable
>
>Thus, as root you can do:
>echo "readonly http_proxy=http://proxy:8080" >> /etc/profile
>
>This locks down the proxy for all users. This works on most (all?
>posix?) shells.

These are only readonly in the context of the current shell. For
example,

$ FOO=bar
$ readonly FOO
$ export FOO # otherwise it's just a shell variable
$ FOO=baz
bash: FOO: readonly variable

So far so good, but it's trivial to override, because the shell
can't assert any real control over the environment:

$ env FOO=baz bash -c 'echo $FOO'
baz

In short, there's no such thing in UNIX as a readonly environment
variable.

>>Thanks, this is all helpful to know. Unfortunately the "proxy"
>>utility program continues to return "direct://" no matter what
>>I do. Based on what you said above, I've tried various settings
>>in gnome-network-preferences, in case it was overriding the
>>environment variable, but nothing changes.
>
>Ok we might have a problem then.
>
>A common problem I've run into is that Fedora (which I'm developing
>on) has libproxy already installed. Not a problem, I just build in
>another directory. However, occasionally I forget to put that
>directory in LD_LIBRARY_PATH before I execute proxy. Thus, it uses
>the old library (since they are binary compatible) and I get results
>I'm not expecting. This might not be your problem, but it was for
>me.

I don't think that's my problem, since it's explicitly linked
with -rpath. ldd shows it using the correct libs, and I tested
with LD_LIBRARY_PATH to double-check.

>Probably the most helpful next step is to add a printf() line in
>proxy.cpp. See my note here (you'll have to scroll down): http://code.google.com/p/libproxy/source/browse/trunk/libproxy/proxy.cpp?r=518

I don't see a note there, what am I missing?

>If neither of those work, let me know and I'll put a bunch of debug
>printf()s in activated by an envvar (I'll do this anyway before the
>next release, but I'll bump up the priority if you need it).

I'll take a swipe at debugging this tomorrow unless you get
a chance to insert those printfs first. Thanks for the response.

Aron

[Aron Griffis]

Nathaniel McCallum wrote: [Fri Feb 05 2010, 11:28:53PM EST]

>A common problem I've run into is that Fedora (which I'm developing
>on) has libproxy already installed. Not a problem, I just build in
>another directory. However, occasionally I forget to put that
>directory in LD_LIBRARY_PATH before I execute proxy. Thus, it uses
>the old library (since they are binary compatible) and I get results
>I'm not expecting. This might not be your problem, but it was for
>me.

I haven't tracked this down entirely, but I think proxy might not
be able to find the config modules in the build tree. At least
ltrace/strace aren't showing any file operations pertaining to
the config modules.

[Nathaniel McCallum]

Yes, this is as designed. If the user can configure the module dir, it
is a security risk, since then anyone could add a config module which
redirects traffic and grabs passwords.

Thus, modules are only read from a dir that only root can modify.

Nathaniel

[Aron Griffis]

Nathaniel McCallum wrote: [Mon Feb 08 2010, 08:29:59AM EST]

So this was at least part of the problem...

Is this something you'd like to "fix", i.e. make it possible to
test libproxy from the context of the source tree, or do you
consider it sufficient to require installation for testing?

However I've now installed libproxy and the proxy test program
still isn't working:

$ sudo make install
$ sudo ldconfig /usr/local/lib
$ export http_proxy=http://jabberwock:8080
$ strace -e trace=file /usr/local/bin/proxy http://google.com 2>&1 | grep config_
stat("/usr/local/lib/libproxy/0.4.0/modules//config_envvar.so", {st_mode=S_IFREG|0644, st_size=49122, ...}) = 0
open("/usr/local/lib/libproxy/0.4.0/modules//config_envvar.so", O_RDONLY) = 3
stat("/usr/local/lib/libproxy/0.4.0/modules//config_gnome.so", {st_mode=S_IFREG|0644, st_size=170163, ...}) = 0
open("/usr/local/lib/libproxy/0.4.0/modules//config_gnome.so", O_RDONLY) = 3

...so we're using the config modules now, but...

$ /usr/local/bin/proxy http://google.com
direct://

[Nathaniel McCallum]

On 02/08/2010 09:53 AM, Aron Griffis wrote:
> Nathaniel McCallum wrote: [Mon Feb 08 2010, 08:29:59AM EST]
>> On Feb 7, 2010, at 11:31 PM, Aron Griffis <agri...@gmail.com> wrote:
>>
>>> Nathaniel McCallum wrote: [Fri Feb 05 2010, 11:28:53PM EST]
>>>> A common problem I've run into is that Fedora (which I'm developing
>>>> on) has libproxy already installed. Not a problem, I just build in
>>>> another directory. However, occasionally I forget to put that
>>>> directory in LD_LIBRARY_PATH before I execute proxy. Thus, it uses
>>>> the old library (since they are binary compatible) and I get results
>>>> I'm not expecting. This might not be your problem, but it was for me.
>>>
>>> I haven't tracked this down entirely, but I think proxy might not
>>> be able to find the config modules in the build tree. At least
>>> ltrace/strace aren't showing any file operations pertaining to
>>> the config modules.
>>
>> Yes, this is as designed. If the user can configure the module dir, it
>> is a security risk, since then anyone could add a config module which
>> redirects traffic and grabs passwords.
>>
>> Thus, modules are only read from a dir that only root can modify.
>
> So this was at least part of the problem...
>
> Is this something you'd like to "fix", i.e. make it possible to
> test libproxy from the context of the source tree, or do you
> consider it sufficient to require installation for testing?

I'd like the ability to test from the tree, but I'm not sure how it
would be possible without opening a security hole. I'm open to suggestions.

I can probably at least do some unit testing by testing each class.
However, the ProxyFactory class is going to be the hardest (due to the
problem above) since this is where the MODULEDIR is hardcoded.

> However I've now installed libproxy and the proxy test program
> still isn't working:
>
> $ sudo make install
> $ sudo ldconfig /usr/local/lib
> $ export http_proxy=http://jabberwock:8080
> $ strace -e trace=file /usr/local/bin/proxy http://google.com 2>&1 |
> grep config_
> stat("/usr/local/lib/libproxy/0.4.0/modules//config_envvar.so",
> {st_mode=S_IFREG|0644, st_size=49122, ...}) = 0
> open("/usr/local/lib/libproxy/0.4.0/modules//config_envvar.so",
> O_RDONLY) = 3
> stat("/usr/local/lib/libproxy/0.4.0/modules//config_gnome.so",
> {st_mode=S_IFREG|0644, st_size=170163, ...}) = 0
> open("/usr/local/lib/libproxy/0.4.0/modules//config_gnome.so", O_RDONLY)
> = 3
>
> ...so we're using the config modules now, but...
>
> $ /usr/local/bin/proxy http://google.com
> direct://

My guess is that it is reading gconf instead of envvar. But debug
printf()s would tell you for sure.

Try the latest commit. It fixes an export problem as well as adding in
debug messages for libmodman. You can activate debug messages at
runtime by defining _MM_DEBUG in your environment.

You should keep in mind that some modules will fail to load by design.
For instance, if you are running KDE, config_gnome should fail.

Nathaniel