FireFox Search provider XML description sought

[R.Wieser]

Hello all,

Over time I've been looking for how to add a (dummy) "search provider" to
FireFox 52, and today I found enough information to be able to do so :

https://askubuntu.com/questions/173822/how-to-make-thunderbird-not-use-bing-as-search-engine

Funny though: That page is neither for FireFox nor Windows, but it gave me
everything I needed nonetheless ...

The entry by "belacqua" at May 28 '13 at 20:44 shows a "google.xml" file, in
which I simply replaced the "https://...." parts to see if it would work.

It does.

The only question I now have is why the XML file has got pretty-much the
same URL three times (and not just once).

Or, differently put, when is each of the below entries used (what action in
FF needs which entry) :

<Url type="application/x-suggestions+json" method="GET"
template="http...."/>

<Url type="text/html" method="GET" template="http....">

<SearchForm>http...</SearchForm>

I've also found
https://developer.mozilla.org/en-US/docs/Mozilla/Creating_MozSearch_plugins

but alas, it doesn't describe anything, and the "OpenSearch format" link
there just drops me in a thick forrest, leaving me on my own in search for a
needle in a haystack (if that much, as I have no idea what I'm actually
looking for :-\ )

tl;dr:
Does anyone know of a "search provider" XML file content description.

Regards,
Rudy Wieser

[Ralph Fox]

On Sat, 10 Jul 2021 16:28:37 +0200, R.Wieser wrote:

> Hello all,
>
> Over time I've been looking for how to add a (dummy) "search provider" to
> FireFox 52, and today I found enough information to be able to do so :
>
> https://askubuntu.com/questions/173822/how-to-make-thunderbird-not-use-bing-as-search-engine
>
> Funny though: That page is neither for FireFox nor Windows, but it gave me
> everything I needed nonetheless ...
>
> The entry by "belacqua" at May 28 '13 at 20:44 shows a "google.xml" file, in
> which I simply replaced the "https://...." parts to see if it would work.
>
> It does.
>
> The only question I now have is why the XML file has got pretty-much the
> same URL three times (and not just once).
>
> Or, differently put, when is each of the below entries used (what action in
> FF needs which entry) :
>
> <Url type="application/x-suggestions+json" method="GET"
> template="http...."/>
>
> <Url type="text/html" method="GET" template="http....">
>
> <SearchForm>http...</SearchForm>

<https://developer.mozilla.org/en-US/docs/Web/OpenSearch>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ QUOTE ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Firefox supports three URL types:

* type="text/html" specifies the URL for the actual search query.
* type="application/x-suggestions+json" specifies the URL for
fetching search suggestions. In Firefox 63 onwards,
type="application/json" is accepted as an alias of this.
* type="application/x-moz-keywordsearch" specifies the URL used
when a keyword search is entered in the location bar. This is
supported only in Firefox.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ QUOTE ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


You might possibly get information more aligned to your Firefox 52 by
reading archived versions of pages from 2017.
<https://web.archive.org/web/20170629015436/https://developer.mozilla.org/en-US/Add-ons/Creating_OpenSearch_plugins_for_Firefox>

> I've also found
> https://developer.mozilla.org/en-US/docs/Mozilla/Creating_MozSearch_plugins
>
> but alas, it doesn't describe anything, and the "OpenSearch format" link
> there just drops me in a thick forrest, leaving me on my own in search for a
> needle in a haystack (if that much, as I have no idea what I'm actually
> looking for :-\ )
>
> tl;dr:
> Does anyone know of a "search provider" XML file content description.
>
> Regards,
> Rudy Wieser

--
Kind regards
Ralph
🦊

[JJ]

On Sat, 10 Jul 2021 16:28:37 +0200, R.Wieser wrote:

Here are the rules which Firefox consider a valid OpenSearch.

1. A valid OpenSearch XML file is present and be served in a web server
using the correct MIME type.

2. An OpenSearch LINK element to the XML is present on a HTML document which
is served in a web server (can be in a different server).

3. User must manually add the search engine by interacting with the web
browser. This can no longer be programmatically automated in current Firefox
version.

For #3... In Firefox, click the Page Action (dots) icon on the address bar,
then click the Add Search Engine from the popup menu - which was made
possible by #2.

When adding the search engine, Firefox sneakily sends a non loggable network
request using the URL from the OpenSearch LINK element where it will be
validated then added into Firefox's Search Engine list which users can't
directly modify existing or create a new Search Engine.

Chromium's OpenSearch processing rules might be different.

[R.Wieser]

JJ,

> Here are the rules which Firefox consider a valid OpenSearch.

I'm sorry, but its not about creating a valid search provider XML file (the
one I have works) or how to add it to the list of available search
providers, but really about the meaning and usage in the browser of the
entries I posted.

Also, the question is posted from the POV of a user (not a website owner),
just wanting to add a new (dummy) search provider - which also means that I
can do some nasty stuff like deleting the "search.json.mozlz4" file to force
the browser to re-initialize the "available search providers" list (there
might be a better, cleaner way, but for now just deleting that file works).
:-)

> When adding the search engine, Firefox sneakily sends a non loggable
> network request using the URL from the OpenSearch LINK element
> where it will be validated then added into Firefox's Search Engine list
> which users can't directly modify existing or create a new Search Engine.

Well, than I'm lucky that my FF is ancient enough that it accepts search
provider XML files without that LINK element. :-)

Regards,
Rudy Wieser

[R.Wieser]

Ralph,

> <https://developer.mozilla.org/en-US/docs/Web/OpenSearch>

Thanks for the link, its stored for further reference.

One problem though : it doesn't seem to mention anything in regard to
when/where each of the entries is used.

The thing is that I can make an educated guess to the usage of each of the
posted entries, but I rather have a solid description (which might also
contain usages I'm not at all aware of).

Regards,
Rudy Wieser

[JJ]

On Sun, 11 Jul 2021 11:43:20 +0200, R.Wieser wrote:
>
> Also, the question is posted from the POV of a user (not a website owner),
> just wanting to add a new (dummy) search provider - which also means that I
> can do some nasty stuff like deleting the "search.json.mozlz4" file to force
> the browser to re-initialize the "available search providers" list (there
> might be a better, cleaner way, but for now just deleting that file works).
> :-)

If by "search provider", you meant the search engine list (instead of a
search engine), then yes. However, we can't add another "search provider",
because there can only be one search engine list in Firefox. We can only
replace the existing.

[R.Wieser]

JJ,

> If by "search provider", you meant the search engine list (instead
> of a search engine), then yes.

With "search provider" I ment what shows up as a single entry in that
"search engine list", not the list itself. So, no.

In what you quoted I referred to what you call a "search engine list" with
the phrase " 'available search providers' list".

Though I'm now curious to why you thought I ment otherwise ...

Regards,
Rudy Wieser

[JJ]

On Mon, 12 Jul 2021 09:22:59 +0200, R.Wieser wrote:
>
> Though I'm now curious to why you thought I ment otherwise ...
>
> Regards,
> Rudy Wieser

Because you think deleting "search.json.mozlz4" file is enough.

[R.Wieser]

JJ,

>> Though I'm now curious to why you thought I ment otherwise ...
>

> Because you think deleting "search.json.mozlz4" file is enough.

???? I'm missing the part where you explain why it wouldn't be. Mind you,
I already wrote that deleting that file seems to work fine for me.

It seems more than likely to me that you are thinking/talking about a
different version of FF than I specified in my initial message (v52). It
certainly would explain the confusion.

Also, somehow I can't wrap my head around how you could have jumped to me
wanting to create a new, second "search engine list" from that 'delete the
file to reinitialize' remark. You stil absolutily lost me there.

Regards,
Rudy Wieser

[JJ]

On Wed, 14 Jul 2021 09:51:37 +0200, R.Wieser wrote:
>
> It seems more than likely to me that you are thinking/talking about a
> different version of FF than I specified in my initial message (v52). It
> certainly would explain the confusion.

Ah, you're right. I wasn't paying attention to the version number.

For older version, use the `window.external.AddSearchProvider(url)` where
`url` is the URL pointing to a HTTP/S served OpenSearch XML resource with
the correct OpenSearch MIME type.

[R.Wieser]

JJ,

> Ah, you're right. I wasn't paying attention to the version number.

For further reference (though I think you already know) : FF v52 is the very
last one that will run on XP (the newsgroup we are in).

> For older version, use the `window.external.AddSearchProvider(url)`

Two things: First that would need some coding, likely in the form of yet
another plugin. I think that for "one off" stuff like this the crude
method works as well and takes a lot less work. Second is that it doesn't
allow me to load my own, local (dummy) "search engine" XML file.

You know, it /almost/ feels like the FF guys really didn't want me to be
able to add my own. :-D

Regards,
Rudy Wieser

[JJ]

On Wed, 14 Jul 2021 15:09:34 +0200, R.Wieser wrote:
>
> You know, it /almost/ feels like the FF guys really didn't want me to be
> able to add my own. :-D

You're not wrong. They never want us to be able to do so.

First of all, they never adds the ability to specifically add a new, or edit
existing search engine (and not just delete).

Then they make ridiculously strict OpenSearch requirements.

And finally, they "lock" the search engine list file with a compression
which no way for a user to specifically uncompress/recompress it. They might
argue that it's for protection against malwares, but the fact that users
can't still able to specifically add a new, or edit existing search engine,
only gives hints that they simply don't want us to be able to do those.

[R.Wieser]

JJ,

> You're not wrong. They never want us to be able to do so.

The question is: why ? The "search engine" XML file effectivily just
contains a three references to the same website, only differing in usage.
Whats there to be denied access to - or dangerous if a user edits it himself
?

> And finally, they "lock" the search engine list file with a
> compression which no way for a user to specifically
> uncompress/recompress it.

:-) I've been looking at that too - if only to see whats actually in that
"search.json.mozlz4" file. Presumably the same as in the "search.json"
file, but you never know. The compression seems to be LZ4, and I did manage
to find a Windows DLL for it (on GitHub). I also found some references to
programs (in sourcefile format) to at least unpack it.

[quote]

When adding the search engine, Firefox sneakily sends a non loggable
network request using the URL from the OpenSearch LINK element
where it will be validated then added into Firefox's Search Engine list
which users can't directly modify existing or create a new Search Engine.

[/quote]

A convoluted approach no doubt build upon legacy requirements.

If the targetted, supposed to be a search engine, website must be asked to
verify the XML file it could as well just return the (microscopic in size)
file itself (which is than always up-to-date). Which means that the
initial, to-be-verified copy of the XML file could be replaced by a simple
URL.

Regards,
Rudy Wieser

[Paul]

R.Wieser wrote:

> :-) I've been looking at that too - if only to see whats actually in that
> "search.json.mozlz4" file. Presumably the same as in the "search.json"
> file, but you never know. The compression seems to be LZ4, and I did manage
> to find a Windows DLL for it (on GitHub). I also found some references to
> programs (in sourcefile format) to at least unpack it.

I wrote such a program myself. A short piece of C code to call "lz4.dll",
so not exactly rocket science :-) I never try to work too hard when
I write code - I am a copy/pasta programmer by nature.

You should open "search.json.mozlz4" and "search.json" in
a text editor and compare them. The LZ4 compressor is little
better than RLE. You can read a lot of the text strings
under mozlz4, or guess at their formulation. It's not
a compressor, and not even a good obfuscater. It's
a nuisance-ifier.

It does make you wonder what they were thinking when they
applied LZ4 :-)

Writing the compressor call, would be good exercise for you :-)

And why wouldn't LZ4.c be in the Firefox tarball ? You should
not necessarily have to go to another site to get it.

gcc -c lz4.c
gcc -c unmoz.c
gcc -o unmoz.exe lz4.o unmoz.o

See, it's hard hard work. You have to check the label on
the thing, to see whether they've changed it or not. This
is a small snippet from the program.

/* mozLz40\0 */
/* 109, 111, 122, 76, 122, 52, 48, 0 */

if ( bufin[0] != 109 ||
bufin[1] != 111 ||
bufin[2] != 122 ||
bufin[3] != 76 ||
bufin[4] != 122 ||
bufin[5] != 52 ||
bufin[6] != 48 ||
bufin[7] != 0 ) {

perror("\n Version magic does not match mozLz40\n");
return 0;
}

I wrote this because I wanted my damn output, not because
the source code is "pretty". If you want the compression
function, copy it from the tarball. There would have to
be a pre-format call before lz4() is called, to put the
version string on it. Since they write out mozlz4 files,
using Firefox, the compressor code has to be in there. And
with the correct header, a portion of which is demoed above.

You could try a text search on "mozLz40" and see what pops up
in the unpacked tarball.

361MB (unpack with 7ZIP or whatever)

http://releases.mozilla.org/pub/firefox/releases/90.0/source/firefox-90.0.source.tar.xz

Paul

[R.Wieser]

Paul,

> I wrote such a program myself.

As will I. If I would have been lazy I would have just grabbed me a
pre-compiled extractor from somewhere, and be done with it. But no, I want
to do it myself. :-)

> A short piece of C code to call "lz4.dll",

In my case it will be Assembly, but same thing.

> You should open "search.json.mozlz4" and "search.json" in a text editor
> and compare them.

That was pretty-much my idea. I'm currently not sure about the value of
that comparision anymore though, as deleting the "search.json" file does not
seem to bother FF, nor does it get re-generated.

> The LZ4 compressor is little better than RLE. You can read a lot of the
> text strings under mozlz4

I noticed (the first thing I do is throwing it it in a hex viewer).

> It does make you wonder what they were thinking when they applied LZ4 :-)

Well, you already said it yourself : to make anuissance of themselves. :-p

> Writing the compressor call, would be good exercise for you :-)

Most of the work will go into figuring out (by looking at the example code)
which functions need to be called, and which arguments they need.

.. wait. The "example" directory /doesn't/ contain example pack and unpack
progams, just a "fullbench" and "xxhash". Oh well, its going to take a bit
more.

> gcc -c lz4.c
> gcc -c unmoz.c
> gcc -o unmoz.exe lz4.o unmoz.o

Two problems: I'm not using C, nor am I using Linux ...

Although I was able to find the "lz4.c" file in a sourcefile pack that I
have backed up (you never know), it did not contain anything by the name
"unmoz"

> You could try a text search on "mozLz40" and see what pops up
> in the unpacked tarball.
>
> 361MB (unpack with 7ZIP or whatever)
>
> http://releases.mozilla.org/pub/firefox/releases/90.0/source/firefox-90.0.source.tar.xz

:-) You're in a XP newsgroup. The highest version of FF that XP can run
is 52.

Thanks for the suggestions though, I'll see what I can do with them.

Regards,
Rudy Wieser

[R.Wieser]

Paul,

> You should open "search.json.mozlz4" and "search.json" in a text editor
> and compare them.

I just wrote the code to unpack the file (which takes one call with four,
rather obvious arguments), and was ready to compare the files - until I
realized that "search.json.mozlz4" was about twice as large as
"search.json". IOW, no chance in hell that the uncompressed (and thus
larger) file would even come near to having the same contents as
"search.json". :-)

But I died find something interresting : references to the build-in "search
engine" XML files. If I ever want to make sure that a profile has my
preferred engine on top than I know where to look.

Regards,
Rudy Wieser