Firefox doesn't correctly invoke duckduckgo under certain conditions

[Karl Winzig]

What is the recommended way to get Firefox sans Javascript
to work with duckduckgo without having to hit multiple pages
at the beginning of every single search?

Illustration: https://i.imgur.com/QNhW7Wx.gif

DETAILS:
1. I have Noscript so javascript is turned off (for privacy),
and I have my search engine set to "duckduckgo" (again
for privacy).
2. When I type "a b c" in the Firefox address bar, I always
get the warning that Duckduckgo requires javascript:
https://i.imgur.com/QNhW7Wx.gif
This page requires JavaScript. Get the non-JS version here.
3. When I click on "here", it takes me to this HTML link:
https://duckduckgo.com/html/?q=a%20b%20c

I didn't do anything to "install" duckduckgo except go to the
Firefox search-engine GUI to select duckduckgo as my search
engine.

Why doesn't Firefox install Duckduckgo correctly such that
duckduckgo just goes to the link it *already* knows it has
to go to?

My problem is that it's supremely wasteful for Firefox/duckduckgo
to tell me to go to a link when it knows very well that javascript
is turned off so that it has to go to that link anyway.

Why not just go there?

Anyway, to have to constantly click on the "here" is just insane.
Duckduckgo should *know* that it should just go there instead.

What is the correct method for going *directly* to the non-JS
version of duckduckgo when running a common search in Firefox?

Searching, I found both search.json and duckduckgo.xml solutions,
but which one is the right way to go (the duckduckgo.xml solution
didn't work for me but that's the solution most often mentioned).

What is the recommended way to get Firefox sans Javascript
to work with duckduckgo without having to hit multiple pages
at the beginning of every single search?

[»Q«]

In
<news:mailman.544.1461121348...@lists.mozilla.org>,

Karl Winzig <kwi...@notgmail.com> wrote:

> What is the correct method for going *directly* to the non-JS
> version of duckduckgo when running a common search in Firefox?

Use <https://addons.mozilla.org/firefox/addon/duckduckgo-lite/> or
<https://addons.mozilla.org/firefox/addon/duckduckgo-html/>.

[Ralph Fox]

On Tue, 19 Apr 2016 22:01:56 -0500, Karl Winzig wrote:

> What is the recommended way to get Firefox sans Javascript
> to work with duckduckgo without having to hit multiple pages
> at the beginning of every single search?
>
> Illustration: https://i.imgur.com/QNhW7Wx.gif
>
> DETAILS:
> 1. I have Noscript so javascript is turned off (for privacy),
> and I have my search engine set to "duckduckgo" (again
> for privacy).
> 2. When I type "a b c" in the Firefox address bar, I always
> get the warning that Duckduckgo requires javascript:
> https://i.imgur.com/QNhW7Wx.gif
> This page requires JavaScript. Get the non-JS version here.
> 3. When I click on "here", it takes me to this HTML link:
> https://duckduckgo.com/html/?q=a%20b%20c
>
> What is the recommended way to get Firefox sans Javascript
> to work with duckduckgo without having to hit multiple pages
> at the beginning of every single search?

When you have been taken to https://duckduckgo.com/html/?q=a%20b%20c
you will see a green "+" on top of the magnifying class in Firefox's
toolbar search box.

Click on it, and in the drop-down click on ‘Add "DuckDuckGo (HTML)"’.

Now go to Firefox's search settings and change your default search
engine from "DuckDuckGo" to "DuckDuckGo HTML".


Here is a 5-frame screen-shot of this ---> http://imgur.com/guwGm7g



--
Kind regards
Ralph

[VanguardLH]

How is a *site* wanting to use Javascript that you don't want to allow a
fault with the web browser? The search URL for the DuckDuckGo search in
Firefox points at DuckDuckGo's home page. They aren't going to point at
some place drilled down into that web site because you don't want to use
the home page for the search engine.

Why can't you whitelist duckduckgo.com in NoScript? Been awhile since I
last used NoScript but, as I recall, you could have it disable only
off-domain scripts, not scripts at the domain that you chose to visit.

Another option is to use bookmark keywords. Create a bookmark whose
properties are:

Name = DuckDuckGo non-JS
Location = https://duckduckgo.com/html/?q=%s
Keyword = dnj

Then use the dnj keyword in the address bar, like "dnj a b c". I have
24 of such keyworded bookmarks - a keyword (used as a prefix in the
address bar) and a URL with %s for where the search criteria goes, like
g for Google, gn for Google News, wiki for Wikipedia, dict for
dictionary.com, mss for Microsoft Support, sp for Ixquick's StartPage
(rather than their Google-only startpage), etc. The default search
engine isn't what I always want to use for a search from the address
bar. You don't want the default search, either, because you instead
want to use the non-Javascript page for DuckDuckGo.

While I tend to create a bookmark and then define it for special cases,
like this, an easy way is to just visit the web page, right-click in the
search box in the web page, and select "Add a keyword for this Search"
from the context menu. You get a keyworded bookmark (which you can move
to wherever you want it stored in your Bookmarks store). I would check
after creating the keyworded bookmark that the %s got positioned at the
correct spot in the URL (location). The %s might get omitted (because
you have not yet done a search to land on the page). You may have to
add it to the URL in the keyworded bookmark.

The standard search providers you add to Firefox go to the standard page
for those search providers. You don't want to use DuckDuckGo's standard
search page. You want a different URL for a different search page. You
don't want to use the standard search provider's page in Firefox.

[bernd]

Is this the new firefox policy to not allow people to edit their search
engines anymore?

I did not believe that would happen but I am near to consider other
browsers that may have the advantages firefox *had* that made all the
sluggishness worth it.

Congratulations Mozilla

[bernd]

Am 20.04.2016 um 08:29 schrieb »Q«:

[bernd]

Am 20.04.2016 um 08:29 schrieb »Q«:

[bernd]

Am 20.04.2016 um 08:29 schrieb »Q«:

[bernd]

Am 20.04.2016 um 08:29 schrieb »Q«:

Is this the new firefox policy, to not allow people to alter their
search engines so they cannot see what is stored in the profiles and
won't be able to change it if they are not happy with that either?

I didn't think that I'd consider other options but I'm doing so now.
Something like firefox *was*, which made all it's sluggishness worth in
the first place. Over years.

Thank you Mozilla

[bernd]

Am 20.04.2016 um 08:29 schrieb »Q«:

[bernd]

Is this the new firefox policy, to not allow people to alter their
search engines so they cannot see what is stored in the profiles and
won't be able to change it if they are not happy with that either?

I didn't think that I'd consider other options but here I'm doing so
now. Wanted: some browser like firefox used to be respecting peoples
freedom, which made all it's sluggishness worth in the first place. Over
years.

Thank you Mozilla

_________________

[bernd]

Is this the new firefox policy, to not allow people to alter their
search engines so they cannot see what is stored in the profiles and
won't be able to change it if they are not happy with that either?

I didn't think that I'd consider other options but here I'm doing so
now. Wanted: some browser like firefox used to be respecting peoples
freedom, which made all it's sluggishness worth in the first place. Over
years.

Thank you Mozilla

_________________

[Mayayana]

| How is a *site* wanting to use Javascript that you don't want to allow a
| fault with the web browser?

I don't understand it, but it does seem to be related
to using built-in search. I don't enable script. My
bookmark to DDG is simply this:
https://duckduckgo.com/

It works fine. I have seen the JS error, though I
don't remember where/when. I'm guessing that FF,
in sending search parameters in the URL, is activating
DDG differently. I don't know why that would be
necessary. Maybe it's just that DDG can test for
script when their page is loaded, but can't in the
case of built-in search, resulting in a default script
version being returned.

Of course the obvious solution would be to stop
thinking of a browser as a search engine
and just go to DDG. That habit may also prevent
tracking in the future. I see no reason to trust the
privacy of search built into a browser. It's a bad
design and a bad habit from the start, designed for
datamining and search market control. In fact, it
accounts for virtually all of Mozilla's income. They
get something like $1 per install from Google (now
Yahoo) in exchange for setting the default search
engine, knowing full well that the vast majority of
people will just be lazy and use the search bar, never
even thinking about how it works or what search
engine they might be using.

But far be it from me to suggest that people should
click one thing more than they absolutely have to. :)

[EE]

You could bookmark the page you want and use that, or perhaps find a
DuckDuckGo search plugin which will start there. The mycroftproject.com
site is a good place to find search plugins.

[Karl Winzig]

EE <nu...@bees.wax> wrote on Wed, 20 Apr 2016 08:43:29 -0700

> You could bookmark the page you want and use that, or perhaps find a
> DuckDuckGo search plugin which will start there. The mycroftproject.com
> site is a good place to find search plugins.

Bookmarking would be trivial; but it's the wrong approach.
A separate plugin may work; but it's also the wrong approach.

The right approach (IMO) is for the default Firefox search plugin GUI
to work correctly, even when Javascript is turned off.

The default search plugin that comes with Firefox has the smarts
to do that; it just doesn't do that.

I'm just looking for the best way to make the default plugin
smarter. I will try the other suggested approaches, but dumb
bookmarks are the wrong approach to the solution of default
plugins not being smart enough on their own to work.

[Karl Winzig]

VanguardLH <V...@nguard.LH> wrote on Tue, 19 Apr 2016 23:48:58 -0500

> How is a *site* wanting to use Javascript that you don't want to allow a
> fault with the web browser? The search URL for the DuckDuckGo search in
> Firefox points at DuckDuckGo's home page. They aren't going to point at
> some place drilled down into that web site because you don't want to use
> the home page for the search engine.

I may be wrong, but it seems to me that all I want to do is use Firefox'
default search engine plugin without Javascript being enabled.

It's certainly obvious that anyone wanting to use Duckduckgo wants to
do so for privacy reasons, and that same person wouldn't want javascript
for the exact same privacy reasons.

Firefox developers should make it easier (not harder) for people to
set a privacy switch (e.g., the Tor Browser Bundle proves it can be done,
but I don't need the 3-hop encryption that comes with Tor and which
slows it down immensely).

All I would want (in the future) is for the search plugins for Firefox
to be smart enough to still work if someone has Javascript disabled.

It's a pretty simple algorithm for a *programmer* to implement:
- If javascript is on, and the person selects duckduckgo, then
point them to the duckduckgo javascript site.
- If javascript is off, and the person selects duckduckgo, then
point them to the duckduckgo non-javascript site.

I'll try out the other suggestions (not bookmarks though, which
is a dead-wrong approach to patching a Firefox bug), and will
let you know what I find out.

[Karl Winzig]

VanguardLH <V...@nguard.LH> wrote on Tue, 19 Apr 2016 23:48:58 -0500

> Why can't you whitelist duckduckgo.com in NoScript? Been awhile since I
> last used NoScript but, as I recall, you could have it disable only
> off-domain scripts, not scripts at the domain that you chose to visit.

Whitelists are like back doors. If the whole point is to disable
Javascript for privacy, then enabling Javascript (especially when it's
not even needed) is contrary to basic privacy policy.

There's no need for duckduckgo to have a back door to javascript,
so, whitelists are (IMO) the wrong approach.

> Another option is to use bookmark keywords. Create a bookmark whose
> properties are:
>
> Name = DuckDuckGo non-JS
> Location = https://duckduckgo.com/html/?q=%s
> Keyword = dnj

Bookmarks are the duct tape of browsers. They fix things that the
browser broke.

As duct tape, bookmarks are fine. But as a real solution, to ask
everyone who cares about privacy to duct tape their browser just
to make it go, by default, to the correct search plugin web site
seems like the wrong approach to me.

The right solution is to tell Firefox to go to the correct site
when the searchplugin duckduckgo is selected (under conditions
of javascript being disabled).

Duckduckgo doesn't *need* javascript to work, and therefore the
default Firefox plugin is pointing people to the wrong location.

Certainly the default plugin is pointing me to the wrong location,
even though it *knows* the correct location.

> I have 24 of such keyworded bookmarks - a keyword (used as a prefix in the
> address bar) and a URL with %s for where the search criteria goes, like
> g for Google, gn for Google News, wiki for Wikipedia, dict for
> dictionary.com, mss for Microsoft Support, sp for Ixquick's StartPage
> (rather than their Google-only startpage), etc.

That's a lot of duct tape! :)

> The default search engine isn't what I always want to use for a search
> from the address bar.

There are only two things I want from (any) upper bar in Firefox:
1. If I type a URI, then go to that URI.
2. If I type something else, then run a default search with the given keywords.

That default search should be intelligent and should forsee that
duckduckgo users, as a rule, would deprecate javascript.

> You don't want the default search, either, because you instead
> want to use the non-Javascript page for DuckDuckGo.

Duckduckgo works perfectly fine without javascript, so, why is a privacy
search engine defaulting to requiring javascript in the first place?

If there must be a default, it should be the other way around.

> While I tend to create a bookmark and then define it for special cases,
> like this, an easy way is to just visit the web page, right-click in the
> search box in the web page, and select "Add a keyword for this Search"
> from the context menu. You get a keyworded bookmark (which you can move
> to wherever you want it stored in your Bookmarks store). I would check
> after creating the keyworded bookmark that the %s got positioned at the
> correct spot in the URL (location). The %s might get omitted (because
> you have not yet done a search to land on the page). You may have to
> add it to the URL in the keyworded bookmark.

Bookmarks are duct tape.
They work.
But they're not a viable long-term solution for everyone.

> The standard search providers you add to Firefox go to the standard page
> for those search providers. You don't want to use DuckDuckGo's standard
> search page. You want a different URL for a different search page. You
> don't want to use the standard search provider's page in Firefox.

In that context, I could argue that the *reason* for using Duckduckgo
is *privacy* and that the reason for turning off javascript is the same
reason.

In addition, I could argue there is no added value in the duckduckgo
results when using javascript versus when not using javascript.

Therfore, I could easily argue, why isn't the default duckduckgo
page installed by Firefox the more private non-javascript page?

[Caver1]

The default plugin points to DuckDuckGo's Javascript site
not the HTML site.
Being that's the case you need something that points to
DuckDuckGo's HTML site.
As stated above mycroftproject.com has- DuckDuckGo (no
filter, World)
search plugin that I have never had a problem with no
javascript.

--
Caver1

[Karl Winzig]

»Q« <box...@gmx.net> wrote on Wed, 20 Apr 2016 01:29:18 -0500

>> What is the correct method for going *directly* to the non-JS
>> version of duckduckgo when running a common search in Firefox?
>
> Use <https://addons.mozilla.org/firefox/addon/duckduckgo-lite/> or
> <https://addons.mozilla.org/firefox/addon/duckduckgo-html/>.

Installing duckduckgo_html_ssl-20140120.xml worked perfectly!

That's how things *should* have worked all along!

There's no reason for a search engine to *need* Javascript
anyway.

What does a search engine need Javascript for anyway?

So (IMO), https://addons.mozilla.org/firefox/addon/duckduckgo-html
should be the default search engine.

Anyway, it is now, for me!

That solves all the problems I was having editing the searchplugins
duckduckgo.xml file and the search.json file, neither of which worked
for me.

PS: What could a search engine possibly be doing with javascript anyway?

[Karl Winzig]

Karl Winzig <kwi...@notgmail.com> wrote on Wed, 20 Apr 2016 14:25:30
-0500

> Installing duckduckgo_html_ssl-20140120.xml worked perfectly!

Well, I spoke too soon.

Installing https://addons.mozilla.org/firefox/addon/duckduckgo-html works,
but now I have to figure out *how* it worked, because I copy my profile
and always use the copy (for security reasons).

And, I had deleted *all* search engines other than Duckduckgo, so,
I only had one search engine before (and now I have two).

But what got installed and where?

I see the installed HTML duckduckgo is not a plugin (thank God).
http://s31.postimg.org/6cw8itrwa/plugins.jpg

And it's not an extension either.
http://s31.postimg.org/e71fhyp2y/extensions.jpg

Looking at what installing the HTML version of duckduckgo did, I see
only two obvious changes to my copied profile:
1. It created $HOME/defaultprofile/searchplugins/duckduckgo-html.xml
2. It added a second engine choice to Firefox
http://s31.postimg.org/k1gty7d6i/search.jpg

Since I have all search engines removed, the only search engine I had
was Duckduckgo, but now I see two search engines, one for each version
of Duckduckgo.

So, while it works, at the moment, I have to reproduce *what* it did
into my default profile.
http://s31.postimg.org/tgwpbev7e/search_html.jpg

Just leaving the engine set to duckduckgo (java version) and renaming
the $HOME/defaultprofile/searchplugins/duckduckgo-html.xml file to
$HOME/defaultprofile/searchplugins/duckduckgo.xml should have worked,
but it didn't accomplish anything.

Likewise, this didn't work either (but it should of):
1. Start Firefox (which starts FF in a copied profile)
2. Install Duckduckgo.html
http://s31.postimg.org/k1gty7d6i/search.jpg
3. Save the pref.js to a safe place
4. Delete the Duckduckgo engine (so that Duckduckgo.html is the setting)
http://s31.postimg.org/tgwpbev7e/search_html.jpg
5. Test that we're using duckduckgo.html in the copied profile
6. Kill Firefox
8. Diff the resulting prefs.js with the saved prefs.js

Unfortunately, the duckduckgo.html addition doesn't seem to add
*anything* to the prefs.js file.

So, I'm at a loss (at the moment) as to how to reproduce whatever the
duckduckgo html addon did to the profile.

All I *know* it did was add a duckduckgo-html.xml file, but now I have
to figure out how firefox knows to *use* that xml file.

[Karl Winzig]

Karl Winzig <kwi...@notgmail.com> wrote on Wed, 20 Apr 2016 15:00:30
-0500

> Installing https://addons.mozilla.org/firefox/addon/duckduckgo-html works,
> but now I have to figure out *how* it worked, because I copy my profile
> and always use the copy (for security reasons).

I think this is the solution.

1. Edit the static profile user.js file:
$ gedit $HOME/firefox_default_profile/user.js

2. Comment out these two lines:
//user_pref("browser.search.defaultenginename.US", "DuckDuckGo");// Use DuckDuckGo JS engine
//user_pref("browser.search.hiddenOneOffs", "DuckDuckGo");// Use DuckDuckGo JS engine

3. Add these two lines:
user_pref("browser.search.defaultenginename.US", "DuckDuckGo HTML"); Use DuckDuckGo HTML engine
user_pref("browser.search.hiddenOneOffs", "DuckDuckGo"); Use DuckDuckGo HTML engine

I am testing it more thoroughly, but it seems to work in the abc test previously noted.

[VanguardLH]

Karl Winzig wrote:

> VanguardLH wrote:
>
>> How is a *site* wanting to use Javascript that you don't want to allow a
>> fault with the web browser? The search URL for the DuckDuckGo search in
>> Firefox points at DuckDuckGo's home page. They aren't going to point at
>> some place drilled down into that web site because you don't want to use
>> the home page for the search engine.
>
> I may be wrong, but it seems to me that all I want to do is use Firefox'
> default search engine plugin without Javascript being enabled.
>
> It's certainly obvious that anyone wanting to use Duckduckgo wants to
> do so for privacy reasons, and that same person wouldn't want javascript
> for the exact same privacy reasons.
>
> Firefox developers should make it easier (not harder) for people to
> set a privacy switch (e.g., the Tor Browser Bundle proves it can be done,
> but I don't need the 3-hop encryption that comes with Tor and which
> slows it down immensely).
>
> All I would want (in the future) is for the search plugins for Firefox
> to be smart enough to still work if someone has Javascript disabled.
>
> It's a pretty simple algorithm for a *programmer* to implement:
> - If javascript is on, and the person selects duckduckgo, then
> point them to the duckduckgo javascript site.
> - If javascript is off, and the person selects duckduckgo, then
> point them to the duckduckgo non-javascript site.

What if Javascript is on for the current page you are visiting but off
when you visit a different page (the search engine)? Remember that you
are using an *extension* to do Javascript enable/disable, not the web
browser to do that. You want Mozilla to adapt their search provider
definitions based on what some extension is doing. Not sure how the web
browser is going to know what an extension is doing.

> I'll try out the other suggestions (not bookmarks though, which
> is a dead-wrong approach to patching a Firefox bug), and will
> let you know what I find out.

I tried hunting around in my Firefox profile folder for something with
the "duckduckgo" string to see if I could find a file that I could edit.
Then I would've tried editing the URL within that file to change to the
one where you add "/html/" to the end of the URL to go to their
non-Javascript search page. Alas, didn't find such a file to let me
edit. I thought there used to be a searchplugin subfolder that had the
XML or whatever that defined the search provider pseudo-extension but I
didn't find one under my Firefox profile folder. Maybe that was to
prevent malware from altering that file to take you to their malicious
forged site. That's why I suggested using a keyworded bookmark.

I did find a <profile>\search.json file that had a "duckduckgo" string
within it; however, I don't know how to edit json files. I can open
them with a text editor. You could copy the file (as a backup) and try
editing the "https://duckduckgo.com/" string to go to a different URL,
like "https://duckduckgo.com/html/". You may also have to change
"https://ac.duckduckgo.com/ac/" to whatever would be DuckDuckGo's
non-Javascript search page. As I recall, some searches there end up
using the "ac" hostname and also in the path. There are multiple
locations in the .json file where the search provider's home or URL
search page are listed. If editing the .json file didn't work, delete
it and copy back the original file.

I don't know that a JSON file called as an external function is going to
be able to look back into the web browser to see under what conditions
it is currently running. https://en.wikipedia.org/wiki/JSON

Search engine sites can have lots of pages. I doubt a "search provider"
election is going to list all of them or anything other than the
standard page for the search provider. You'd end up with DuckDuckGo
(Javascript - standard home page), DuckDuckGo (non-Javascript),
DuckDuckGo (Images), DuckDuckGo (Videos), DuckDuckGo (pick a country),
DuckDuckGo (without page breaks), DuckDuckGo (time frame), and so on.
So instead of listing one search provider just for DuckDuckGo, there
would be half a dozen or more. Then multiple the same for every search
provider that had multiple search criteria scenarios. You'd end up with
a huge list of search providers that would be longer than the height of
your screen.

You use DuckDuckGo to avoid having Google, Microsoft, Yahoo, or some
other biggie track to where you navigate. Not you personally but you
don't want to let them compile a database of sites to rank them to
provide relevant results. Okay, but then why don't you trust
DuckDuckGo? Their Javascript ensures that an HTTP Referer header does
*not* get sent to the target site when you click on a link to go there.
With just their HTML-only (non-Javascript) page, all DuckDuckGo can give
you is the direct URL link which means the visited site will get the
HTTP Referer header to know from whence you came. You could use an
add-on to block sending the Referer header but some sites refuse to let
you navigate between their web pages without it (they use it as security
to ensure you get to a web page from another of their own rather than
let you directly drill into their site). Some add-ons will allow
Referer within the same domain but block it across domains (when you
visit one domain but navigate to another); however, sites nowadays are
often spread across multiple domains so blocking Referer from one of
their domains when you click a link to visit another of their domains
can still result in breaking their check that you got there from an
allowed page.

They also let you somehow add !proxy into the URL so when you click on a
search result link that you anonymously visit the target site. Never
used it so not sure how it works. Saw it mentioned in their privacy
page (https://duckduckgo.com/privacy). Rather than have to putz with
the URL (edit it), I use Ixquick's StartPage which has a Proxy link.
Rather than use Startpage (operated by Ixquick) which only searches
Google, I use ixquick.com (also called StartPage) that searches multiple
providers. When I do a search there (and, yes, I do permit Javascript),
there is a Proxy link that can hide me when I visit the target site.
That's a lot easier than trying to figure out how to modify the URL
before using it.

[»Q«]

In
<news:mailman.612.1461174415...@lists.mozilla.org>,

Karl Winzig <kwi...@notgmail.com> wrote:

> VanguardLH <V...@nguard.LH> wrote on Tue, 19 Apr 2016 23:48:58 -0500

> > I have 24 of such keyworded bookmarks - a keyword (used as a prefix
> > in the address bar) and a URL with %s for where the search criteria
> > goes, like g for Google, gn for Google News, wiki for Wikipedia,
> > dict for dictionary.com, mss for Microsoft Support, sp for
> > Ixquick's StartPage (rather than their Google-only startpage),
> > etc.
>
> That's a lot of duct tape! :)

It's not duct tape at all, and it makes searching from the address bar
infinitely more powerful and flexible than just relying on
keyword.enabled searching. (And anyone obsessed with "privacy" should
certainly use keyworded bookmarks instead of relying on keyword.enabled
searching, because typos.)

[Karl Winzig]

»Q« <box...@gmx.net> wrote on Wed, 20 Apr 2016 18:49:33 -0500

> anyone obsessed with "privacy" should
> certainly use keyworded bookmarks instead of relying on keyword.enabled
> searching, because typos.)

We could discuss what anyone obsessed with privacy would use, but
I, for one, certainly have all keyword.enabled suggestions turned off.

user_pref("browser.search.suggest.enabled", false);
user_pref("browser.urlbar.suggest.searches", false);
user_pref("browser.urlbar.userMadeSearchSuggestionsChoice", true);
etc.

[Karl Winzig]

VanguardLH <V...@nguard.LH> wrote on Wed, 20 Apr 2016 14:03:43 -0500

>> It's a pretty simple algorithm for a *programmer* to implement:
>> - If javascript is on, and the person selects duckduckgo, then
>> point them to the duckduckgo javascript site.
>> - If javascript is off, and the person selects duckduckgo, then
>> point them to the duckduckgo non-javascript site.
>
> What if Javascript is on for the current page you are visiting but off
> when you visit a different page (the search engine)? Remember that you
> are using an *extension* to do Javascript enable/disable, not the web
> browser to do that. You want Mozilla to adapt their search provider
> definitions based on what some extension is doing. Not sure how the web
> browser is going to know what an extension is doing.

I would argue that it's a bug that Firefox installs the "wrong"
duckduckgo searchplugin, by default.

Let's assume "Alice" makes the decision for Firefox, and that I make
a case to her why Firefox should default to the HTML version instead
of to the javascript version.

My arguments might include:
1. Why people use DuckDuckGo?
2. If it's privacy, then why do people turn off Javascript?
3. The answer to both questions is the same. Privacy.

Then I might include the argument:
A. What does DuckDuckGo need Javascript for anyway?
B. What added value does the Javascript version have for the user?
C. Isn't the HTML version faster, simpler, and more secure?

So, "my" argument that it's a bug that Firefox defaults to the 'wrong'
duckduckgo implementation would hinge on the answers to those
questions.

Does anyone here actually *know* the answer to those questions?
(I, for one, don't know the answer to A, B, and C above.)

[Karl Winzig]

VanguardLH <V...@nguard.LH> wrote on Wed, 20 Apr 2016 14:03:43 -0500

> I tried hunting around in my Firefox profile folder for something with
> the "duckduckgo" string to see if I could find a file that I could edit.
> Then I would've tried editing the URL within that file to change to the
> one where you add "/html/" to the end of the URL to go to their
> non-Javascript search page. Alas, didn't find such a file to let me
> edit. I thought there used to be a searchplugin subfolder that had the
> XML or whatever that defined the search provider pseudo-extension but I
> didn't find one under my Firefox profile folder. Maybe that was to
> prevent malware from altering that file to take you to their malicious
> forged site. That's why I suggested using a keyworded bookmark.

As mentioned in the OP, I already tried to edit the file as per Google
result suggestions.
$ gedit $defaultprofile/searchplugins/duckduckgo.xml

I failed.

Only after I failed, did I ask the question here.

I have since SOLVED the problem, and will write up a complete solution
to give back to the newsgroup.

[Karl Winzig]

VanguardLH <V...@nguard.LH> wrote on Wed, 20 Apr 2016 14:03:43 -0500

> I did find a <profile>\search.json file that had a "duckduckgo" string
> within it; however, I don't know how to edit json files. I can open
> them with a text editor. You could copy the file (as a backup) and try
> editing the "https://duckduckgo.com/" string to go to a different URL,
> like "https://duckduckgo.com/html/". You may also have to change
> "https://ac.duckduckgo.com/ac/" to whatever would be DuckDuckGo's
> non-Javascript search page. As I recall, some searches there end up
> using the "ac" hostname and also in the path. There are multiple
> locations in the .json file where the search provider's home or URL
> search page are listed. If editing the .json file didn't work, delete
> it and copy back the original file.
>
> I don't know that a JSON file called as an external function is going to
> be able to look back into the web browser to see under what conditions
> it is currently running. https://en.wikipedia.org/wiki/JSON

When I searched for solutions to this common default issue with
Firefox' choice of which duckduckgo search plugin to install by
default, I found answers that included editing the xml file
and the json file.

I tried both, as explained in the OP, and failed.

Luckily, the solution to install the HTML version and to delete
the Javascript version is working perfectly fine now, and I will
write up how I did that so everyone can reproduce my results.

[Karl Winzig]

VanguardLH <V...@nguard.LH> wrote on Wed, 20 Apr 2016 14:03:43 -0500

> Search engine sites can have lots of pages. I doubt a "search provider"
> election is going to list all of them or anything other than the
> standard page for the search provider. You'd end up with DuckDuckGo
> (Javascript - standard home page), DuckDuckGo (non-Javascript),
> DuckDuckGo (Images), DuckDuckGo (Videos), DuckDuckGo (pick a country),
> DuckDuckGo (without page breaks), DuckDuckGo (time frame), and so on.
> So instead of listing one search provider just for DuckDuckGo, there
> would be half a dozen or more. Then multiple the same for every search
> provider that had multiple search criteria scenarios. You'd end up with
> a huge list of search providers that would be longer than the height of
> your screen.

It is my humble opinion that it's a bug that Firefox defaults to
the most complicated duckduckgo search engine (i.e., Firefox defaults
to the javascript search engine).

It is my humble opinion that Firefox should default to the safest
or fastest or simplest or most obvious search engine, which, IMHO,
is the html version.

However, what I do not know is what *value* is added by the default
javascript search engine over the elective html search engine.

Do you know what the javascript search engine buys the user?

[Karl Winzig]

VanguardLH <V...@nguard.LH> wrote on Wed, 20 Apr 2016 14:03:43 -0500

> You use DuckDuckGo to avoid having Google, Microsoft, Yahoo, or some
> other biggie track to where you navigate. Not you personally but you
> don't want to let them compile a database of sites to rank them to
> provide relevant results. Okay, but then why don't you trust
> DuckDuckGo?

Yes. I use DuckDuckGo for only one purpose.
To stay out of Google's, Yahoo's, Microsoft's, etc., database.

You ask why I don't trust Duckduckgo?
I do trust duckduckgo.

In fact, duckduckgo is my *only* search engine!
http://s31.postimg.org/tgwpbev7e/search_html.jpg

Separately, I have javascript turned off for privacy reasons.

I think it's a bug that the Firefox developers chose to implement the
javascript version of duckduckgo by default, so that is my only "issue".

However, I am wholly unaware of what the difference is, from the standpoint
of results or privacy, for the HTML version versus the Javascript version.

That is important information that I need to make a better decision.

[Karl Winzig]

VanguardLH <V...@nguard.LH> wrote on Wed, 20 Apr 2016 14:03:43 -0500

> Their Javascript ensures that an HTTP Referer header does
> *not* get sent to the target site when you click on a link to go there.
> With just their HTML-only (non-Javascript) page, all DuckDuckGo can give
> you is the direct URL link which means the visited site will get the
> HTTP Referer header to know from whence you came. You could use an
> add-on to block sending the Referer header but some sites refuse to let
> you navigate between their web pages without it (they use it as security
> to ensure you get to a web page from another of their own rather than
> let you directly drill into their site). Some add-ons will allow
> Referer within the same domain but block it across domains (when you
> visit one domain but navigate to another); however, sites nowadays are
> often spread across multiple domains so blocking Referer from one of
> their domains when you click a link to visit another of their domains
> can still result in breaking their check that you got there from an
> allowed page.

This is the FIRST indication that I have of the difference between the
javascript and html versions of duckduckgo!

As you note, nobody wants their "HTTP Referer header" to give away
whence they came. I think it can even give away their search string.

Is there a good test to see if our setup blocks referrer headers?

If I look for the word "refer" in my user.js, I find this:
$ grep -i refer user.js
user_pref("network.http.sendSecureXSiteReferrer", false);// Turn off spying
user_pref("network.http.sendSecureXSiteReferrer", false);// Disable referer from an SSL Website
user_pref("network.http.sendRefererHeader", 1);
user_pref("network.http.referer.spoofSource", true);

But I don't know if that's actually working to block all referrer
headers.

Is there an easy way to test if my referrer headers are blocked?

[Karl Winzig]

Karl Winzig <kwi...@notgmail.com> wrote on Wed, 20 Apr 2016 22:15:52
-0500

> This is the FIRST indication that I have of the difference between the
> javascript and html versions of duckduckgo!

I made a boo boo in the user_pref list so I repeat the post without the mistake:

[Karl Winzig]

Karl Winzig <kwi...@notgmail.com> wrote on Tue, 19 Apr 2016 22:01:56
-0500

> What is the recommended way to get Firefox sans Javascript
> to work with duckduckgo without having to hit multiple pages
> at the beginning of every single search?

Thanks to the help here, below is a writeup I made to give back
to the newsgroup, of how "I" installed duckduckgo.html on a
system which is *designed* to be as private as "I" can make it.

Suggestions for improvement are always welcome.

****************************************************************************
How to set up Firefox to use DuckDuckGo.html by default.
****************************************************************************
NOTE: The javascript DDG will block "HTTP Referer header" headers.

0. The problem with the Duckduckgo which is installed, by default,
in Firefox is that the Firefox default version expects javascript.
https://i.imgur.com/QNhW7Wx.gif

Anyone who cares about privacy probably would be concerned about that.

This howto assumes you have a pristine Firefox profile which is
copied whenever Firefox is started. The howto works even if you
do not have a pristine profile, but it would be many fewer steps.

The pristine profile is created by starting Firefox and setting
up Firefox with your preferences, extensions, user.js changes, etc.
$ /usr/lib/firefox/firefox -new-instance -profile $HOME/firefox/default_profile/ &

Then a script is used to *copy* that pristine read-only profile whenever
Firefox is invoked and to use the working copy of the profile instead.

The key lines of the script are as follows:
cp $readonlyProfile $workingProfile
/usr/lib/firefox/firefox -new-instance -profile $workingProfile

1. Back up your profile (just in case):
$ cp -r $HOME/firefox/default_profile $HOME/firefox/default_profile.bak

2. Start Firefox such that it edits the pristine profile:
$ /usr/lib/firefox/firefox -new-instance -profile $HOME/firefox/default_profile/ &
$ /usr/lib/firefox/firefox -new-instance -profile $HOME/firefox/default_profile/ &

3. Note what is in the default profile before you add the DDG HTML search engine:
$ gedit $HOME/defaultprofile/searchplugins/duckduckgo.xml

$ grep -i duckduckgo user.js
user_pref("browser.search.defaultenginename.US", "DuckDuckGo");// Use DDG JS engine
user_pref("browser.search.hiddenOneOffs", "DuckDuckGo");// Use DDG JS engine

4. Install the duckduckgo-xml searchplugin
Load: https://addons.mozilla.org/firefox/addon/duckduckgo-html
https://i.imgur.com/kMUBGCl.gif

Note that installing the DDG HTML engine adds a 2nd engine choice to Firefox:
http://s31.postimg.org/k1gty7d6i/search.jpg

See also the untested suggestion:
https://addons.mozilla.org/firefox/addon/duckduckgo-lite

5. Remove the original DDG JS engine so you're back to a single engine:
http://s31.postimg.org/tgwpbev7e/search_html.jpg

6. Note that installing the HTML version of duckduckgo added this file:
$HOME/defaultprofile/searchplugins/duckduckgo-html.xml

But you still need to edit the user.js file to make it stick.

7. Edit the static profile user.js file:
$ cp $HOME/firefox/default_profile/user.js user.js.old
$ gedit $HOME/firefox/default_profile/user.js

Comment out these two lines:

//user_pref("browser.search.defaultenginename.US", "DuckDuckGo");// Use DDG JS engine
//user_pref("browser.search.hiddenOneOffs", "DuckDuckGo");// Use DDG JS engine

Add these two lines instead:
user_pref("browser.search.defaultenginename.US", "DuckDuckGo HTML"); Use DDG HTML engine
user_pref("browser.search.hiddenOneOffs", "DuckDuckGo"); Use DDG HTML engine

8. Test and make a note of any changes:
Note the installed HTML duckduckgo is not a plugin:

http://s31.postimg.org/6cw8itrwa/plugins.jpg

And it's not an extension either.
http://s31.postimg.org/e71fhyp2y/extensions.jpg

9. Make a backup of all installation files as normally done with all software:
$ mkdir /data/software/browser/firefox/searchplugins/duckduckgo-html

$ ffox https://addons.mozilla.org/firefox/addon/duckduckgo-html

Note that re-installing will fail with the message:
Firefox could not install the search plugin from
"https://addons.mozilla.org/firefox/downloads/latest/252586/addon-252586-latest.xml?
src=dp-btn-primary" because an engine with the same name already exists.

As shown in this screenshot:
https://i.imgur.com/kMUBGCl.gif

Right click and select "Save As" "XML Document"
/data/software/browser/firefox/searchplugins/duckduckgo-html/duckduckgo_html_ssl-20140120.xml
https://i.imgur.com/Fzxc9Za.gif

10. The problem is that loading this xml file into a new Firefox session
fails to load because Firefox *displays* the XML file instead of executing it:
https://i.imgur.com/Acc28bp.gif

So, I think my only question at this point is what do I save to archive this
installer; and how do I reload it (without going back to the original download
page, of course)?
============================================================================

[Mayayana]

| But I don't know if that's actually working to block all referrer
| headers.
|

If you disable anything in about: config with "refer" then
it works. I always forget the exact settings. The Mozilla
people have changed it at least once. I haven't tested the
https version. I guess you could try that. This site seems to
work:

https://referer.rustybrick.com/

Most search no longer sends usable data in the referer.
I find at my own site that nearly all searches are from
Google. Of those, many have the referer stripped out,
except for the Google URL. Some have code that tells
me the page rank of the link they clicked and also sends
a number of encoded values that are mostly of use only
to Google. A rare few will still send the search terms.
Apparently it depends on where Google was accessed.
The Googlites are not known for respecting privacy, so my
theory is that they block referers to make their own
Google Analytics service more attractive to web masters.
But I don't know that for sure.

Personally I've blocked referers for many years, but I
miss them at my own website. They told me a lot about
what people wanted and whether my pages were being
catalogued accurately. (If one owns an apple orchard but
gets a steady stream of visitors looking for iPads then
one's site needs work. Without referers the orchard owner
will likely just think their website is a success. With referers
they can figure out what combination of words is triggering
the search engine to send the wrong people to their site.)

I also don't normally enable script (or cookies, iframes,
3rd-party cookies, or a number of options that Secret
Agent can block). If you care so much about privacy
then why are you doing search from the Firefox search
bar in the first place? Do you know for a fact that FF
doesn't call home with that? If it doesn't, how do you
know it won't start? Searching from the browser (or from
the OS) is a privacy intrusion waiting to happen, at best.
Search terms are marketable data. And all the search
bar serves is to save you from clicking a bookmark. Yet
you've clearly spent some hours trying to perfect that
tiny convenience.

[Karl Winzig]

"Mayayana" <maya...@invalid.nospam> wrote on Thu, 21 Apr 2016 10:20:09
-0400

> If you disable anything in about: config with "refer" then
> it works. I always forget the exact settings. The Mozilla people have
> changed it at least once. I haven't tested the https version. I guess
> you could try that. This site seems to work:
>
> https://referer.rustybrick.com/

Thank you for the information.

I did some research and it's diabolically confusing because if you
turn referrer headers off, you break some sites but if you turn them
on you have to spoof them.

Anyway, here's the best I can come up with so far, but assume it is
posted here as a request for review for accuracy.

To test whether DDG-HTML allows refer headers:
http://www.ghacks.net/2015/01/22/improve-online-privacy-by-controlling-
referrer-information/

$ egrep -i "referer|referrer" user.js
user_pref("network.http.sendRefererHeader", 1);//0=Do not send the
Referer header or set document.referrer, 1=send
user_pref("network.http.sendSecureXSiteReferrer", false);//false=do not
send the Referer header when going from one https site to another https
site, true=send
user_pref("network.http.referer.spoofSource", true);//false=real
referer, true=spoof referer (use target URI as referer)
user_pref("network.http.referer.trimmingPolicy", 2);//0=send full URI,
1=scheme+host+port+path, 2=scheme+host+port
user_pref("network.http.referer.XOriginPolicy", 2);//0=always send,
1=send if base domains match, 2=send if hosts match

1 Point Firefox to https://www.grc.com
2 Press on the "ShieldsUp!" graphic.
3 Under the gray "Services" tab at the top, select "ShieldsUp!" from the
pulldown.
4 Click on the "Proceed" button.
5 Locate the white-on-blue 'ShieldsUP!! Services' line.
6 Below that is a black-on-gray 'Browser Headers' button.
7 Click on that 'Browser Headers' button.
8 That takes you to the "Exploring Your Browser's Web Server Requests"
page.
9 My results were the following browser headers:
Accept-Language: en-US,en;q=0.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Connection: keep-alive
Content-Length: 31
Content-Type: application/x-www-form-urlencoded
DNT: 1
FirstParty: https://www.grc.com
Host: www.grc.com
Nonsecure: http://www.grc.com
Referer: https://www.grc.com/x/ne.dll?rh1dkyd2
Secure: https://www.grc.com
Session: qpbzej2wdcm4q
ThirdParty: https://www.grctech.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:44.0) Gecko/20100101
Firefox/44.0

10 Note the referer line above!
It's diabolically difficult to get rid of!
https://blog.mozilla.org/security/2015/01/21/meta-referrer/

[Mayayana]

| I did some research and it's diabolically confusing because if you
| turn referrer headers off, you break some sites but if you turn them
| on you have to spoof them.
|

I don't find it confusing. If you shut them off in
about:config that turns them off.

// one of these is outdated, but I've forgotten which one:
network.http.sendRefererHeader 0
network.sendRefererHeader false
network.http.sendSecureXSiteReferrer false

There are occasional sites that require them. I
don't run into those very often. Typically it's a
site checking to make sure you're downloading a
file from their link. But I actually use 2 browsers:

Pale Moon with nothing enabled.

Firefox with NoScript. Session cookies, frames,
referers enabled.

So if I have trouble I just switch to Firefox.

When I went to GRC I saw no referer listed. But
on my first visit I didn't disable Secret Agent,
which spoofs headers. GRC warned me that I
was proxying through the US Dept. of Defense. :)
I then went to a check-my-IP site, which told
me I was coming from JP Morgan Chase. That's
when I realized I needed to disable Secret Agent.

[»Q«]

In
<news:mailman.601.1461207276...@lists.mozilla.org>,

Karl Winzig <kwi...@notgmail.com> wrote:

> I would argue that it's a bug that Firefox installs the "wrong"
> duckduckgo searchplugin, by default.

Arguing here about what Mozilla should or shouldn't do just adds noise
to the group. Filing a bug is the way to start the process of possibly
changing anything in Firefox.

[»Q«]

In
<news:mailman.600.1461206938...@lists.mozilla.org>,

Karl Winzig <kwi...@notgmail.com> wrote:

> »Q« <box...@gmx.net> wrote on Wed, 20 Apr 2016 18:49:33 -0500
>
> > anyone obsessed with "privacy" should
> > certainly use keyworded bookmarks instead of relying on
> > keyword.enabled searching, because typos.)
>
> We could discuss what anyone obsessed with privacy would use, but
> I, for one, certainly have all keyword.enabled suggestions turned off.

Hans Donitz recently wrote

I changed that line to:
user_pref("keyword.enabled", true)

Are you not the morpher who recently used the nym 'Hans Donitz'?

[Mayayana]

| > This is the FIRST indication that I have of the difference between the
| > javascript and html versions of duckduckgo!
|
| I made a boo boo in the user_pref list so I repeat the post without the
mistake:
|
| This is the FIRST indication that I have of the difference between the
| javascript and html versions of duckduckgo!
|
| As you note, nobody wants their "HTTP Referer header" to give away
| whence they came. I think it can even give away their search string.
|

A bit of info that may be of interest, which was
clarified for me by this discussion:

The official doc on referers is here:

https://tools.ietf.org/html/rfc7231#section-5.5.2

It says the following:

"A user agent MUST NOT send a Referer header field in an
unsecured HTTP request if the referring page was received with a
secure protocol."
[ That is, no referer from https to http.]

"A user
agent MUST NOT include the fragment and userinfo components of the
URI reference [RFC3986], if any, when generating the Referer field
value."
[Fragment is technically the #xxx portion of a link, but
though the "spirit of the law" would also dicate that
details like page rank and search terms should be
considered private if the selected portion of a webpage
is considered private.]

Google and Bing both cheat a bit on this. Over http
they both send search terms and link page rank, which
I'd call fragment. Over https Google strips out the search
terms but still, often, sends unique IDs and page rank
of the link clicked. Yet according to the official rules they
should not be sending any referer at all. I'm guessing
that Google is very invested in letting webmasters
know just how important they are to traffic, but that
they strip out search terms over https as a concession
to avoid confrontations over their flouting of the rules.

So, moral of the story? Aside from the issue of blocking
referers, there is a real difference between http and
https.... or at least there should be and often is.