[fossil-users] Fossil is Awesome
Caleb Gray
1) Compress the releases found on http://www.fossil-scm.org/download.html using UPX ( http://upx.sourceforge.net/).
While deploying Fossil to my home computers, work machines, and servers, I found the extra step of unzipping the "self-contained" executable unnecessary.
Speaking from personal experience: UPX does its job flawlessly every time. There are entire Linux distributions compressed using UPX proving that it has no effect on executable integrity. UPX even does a better job of compressing the binary than ZIP does:
linux-x86-20111021125253 (ZIP: 477K) (UPX: 383K)
macosx-x86-20111021125253 (ZIP: 550K) (UPX: 473K)
openbsd-x86-20111021125253 (ZIP: 537K) (UPX: 416K)
w32-20111021125253.exe (ZIP: 498K) (UPX: 402K)
Although, one caveat to this is that many Windows users have Anti-virus solutions installed that claim UPX compressed executables are either viruses, or "possible threats." Perhaps it would be best to leave the Windows release as a ZIP.
2) Add (or replace zlib with) LZMA.
Some of my repositories are very large; the compression difference between the DEFLATE algorithm and the LZMA algorithm is not negligible, and can even be substantial.
Recently, even the Linux kernel has officially begun using lzmalib to compress their releases: http://www.kernel.org/pub/linux/kernel/v3.0/ (even the difference between BZ2 and XZ is about 13M, or 31M smaller than GZ).
A quick implementation of LZMA (by editing auto.def and blob.c's blob_compress, blob_compress2, and blob_uncompress) resulted in the following repository sizes for the latest trunk of SQLite:
sqlite-zlib.fossil (4.8M) (~5.0K per file)
sqlite-lzma.fossil (3.5M) (~3.6K per file)
How's that for bandwidth efficient enough for " dial-up"?
3) The web interface could use a face lift, as well as some HTML5 functionality.
I've got a lot of web development experience and would love to contribute in this area, also.
All of the work on the JSON APIs is a great step toward making the entire interface XHR compatible. What are the community's feelings on jQuery? I get the gist that externals are trying to be avoided, so that's why I'm asking, I would love to write a library that turns the current site in to a highly interactive version without touching the HTML or CSS at all.
Thanks for your time! I look forward to your replies. :D
Sincerely,
Caleb
Stephan Beal
3) The web interface could use a face lift, as well as some HTML5 functionality.
I've got a lot of web development experience and would love to contribute in this area, also.
All of the work on the JSON APIs is a great step toward making the entire interface XHR compatible. What are the community's feelings on jQuery? I get the gist that externals are trying to be avoided, so that's why I'm asking,
I would love to write a library that turns the current site in to a highly interactive version without touching the HTML or CSS at all.
----- stephan beal
http://wanderinghorse.net/home/stephan/
Konstantin Khomoutov
Caleb Gray <ca...@calebgray.com> wrote:
[...]
> 3) The web interface could use a face lift, as well as some HTML5
> functionality.
>
> I've got a lot of web development experience and would love to
> contribute in this area, also.
>
> All of the work on the JSON APIs is a great step toward making the
> entire interface XHR compatible. What are the community's feelings on
> jQuery? I get the gist that externals are trying to be avoided, so
> that's why I'm asking, I would love to write a library that turns the
> current site in to a highly interactive version without touching the
> HTML or CSS at all.
I strongly disagree.
First, please don't fix what's not broken.
Web UI does what it's intended to do, and does it well. JS usage is
constrained to what can't be done without it (those arrows in the
timeline view), and it's even usable with JS turned off.
P.S.
I'm one of those crazy folks who usually has NoScript turned on except
for the intranet sites, so yes, I'm biased.
_______________________________________________
fossil-users mailing list
fossil...@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Ron Aaron
> I strongly disagree.
> First, please don't fix what's not broken.
> P.S.
> I'm one of those crazy folks who usually has NoScript turned on except
> for the intranet sites, so yes, I'm biased.
>
Yes, so am I ...
Lluís Batlle i Rossell
>
> [...]
> I'm one of those crazy folks who usually has NoScript turned on except
> for the intranet sites, so yes, I'm biased.
Same here. I don't think we should require c89 and
html5-browsers-with-javascript at once.
Regards,
Lluís.
Stephan Beal
On 10/26/2011 10:59 AM, Konstantin Khomoutov wrote:Agree 100%
> I strongly disagree.
> First, please don't fix what's not broken.
Ron Aaron
>
>
> imagine what we could do for version, file/dir, and diff browsing with
> something like:
>
> http://mbostock.github.com/d3/talk/20111018/#8
>
I think that things of that nature would be computationally intensive,
and better suited to a separate utility using e.g. your JSON interface
to query data to present.
Michael Barrow
Michael Barrow
michael at barrow dot me
+1 (408) 782-4249
Steve Havelka
>> jQuery? I get the gist that externals are trying to be avoided, so
>> that's why I'm asking, I would love to write a library that turns the
>> current site in to a highly interactive version without touching the
>> HTML or CSS at all.
I think a well-made js-enabled UI could be a wonderful improvement over
the current static one. I'd love to see what you have in mind. (in
other words, I'm not one of those noscript people..;) )
Dmitry Chestnykh
> 2) Add (or replace zlib with) LZMA.
>
> Some of my repositories are very large; the compression difference between
> the DEFLATE algorithm and the LZMA algorithm is not negligible, and can even
> be substantial.
>
> Recently, even the Linux kernel has officially begun using lzmalib to
> compress their releases: http://www.kernel.org/pub/linux/kernel/v3.0/ (even
> the difference between BZ2 and XZ is about 13M, or 31M smaller than GZ).
>
> A quick implementation of LZMA (by editing auto.def and blob.c's
> blob_compress, blob_compress2, and blob_uncompress) resulted in the
> following repository sizes for the latest trunk of SQLite:
> sqlite-zlib.fossil (4.8M) (~5.0K per file)
> sqlite-lzma.fossil (3.5M) (~3.6K per file)
>
> How's that for bandwidth efficient enough for " dial-up"?
LZMA is good for release tarballs, but has unacceptable speed and
memory requirements for version control systems.
Note that due to self-checks, Fossil compresses and then extracts and
verifies content before committing it:
http://www.fossil-scm.org/index.html/doc/trunk/www/selfcheck.wiki
I'd say DEFLATE is a good compromise between LZO/SNAPPY and LZMA for our use.
--
Dmitry Chestnykh
Dmitry Chestnykh
floppies and hard drives. There's no reason to use it nowadays. It
slows down execution, takes more memory, may introduce
vulnerabilities, etc. Just read about all the disadvantages of UPX
here http://linux.die.net/man/1/upx.
Note that Fossil in CGI mode is executed for each request, so with UPX
it would have to decompress itself on each request. The first time you
run it, it may be faster (due to smaller disk reads compared to
uncompressed version), but for each subsequent request, once Fossil is
in RAM, it would be slower and require more CPU.
Lluís Batlle i Rossell
> On Wed, Oct 26, 2011 at 3:51 AM, Caleb Gray <ca...@calebgray.com> wrote:
> > 1) Compress the releases found on http://www.fossil-scm.org/download.html> using UPX ( http://upx.sourceforge.net/).
> it would have to decompress itself on each request. The first time you
> run it, it may be faster (due to smaller disk reads compared to
> uncompressed version), but for each subsequent request, once Fossil is
> in RAM, it would be slower and require more CPU.
Forking and executing is not the same thing. I don't know in Windows, but in
unix, a fork of a upx program should be the same as a non-upx program.
Dmitry Chestnykh
executing is not the same thing. I don't know in Windows, but in>
unix, a fork of a upx program should be the same as a non-upx program.
run by webserver with fork/exec.
Also, Windows doesn't have fork.
--
Dmitry Chestnykh
Lluís Batlle i Rossell
> 2011/10/26 Lluís Batlle i Rossell <viri...@gmail.com>:> Forking and
> executing is not the same thing. I don't know in Windows, but in>
> unix, a fork of a upx program should be the same as a non-upx program.
> Fork (on *nix) applies to "fossil server". Fossil's CGI is, obviously,
> run by webserver with fork/exec.
> Also, Windows doesn't have fork.
Ah right, for the CGI yes. My fault :)
Remigiusz Modrzejewski
On Oct 26, 2011, at 4:39 PM, Steve Havelka wrote:
> On Tue, 25 Oct 2011 18:51:05 -0700 Caleb Gray <ca...@calebgray.com> wrote:
>>> [trimmed ...] What are the community's feelings on
>>> jQuery? I get the gist that externals are trying to be avoided, so
>>> that's why I'm asking, I would love to write a library that turns the
>>> current site in to a highly interactive version without touching the
>>> HTML or CSS at all.
>
> I think a well-made js-enabled UI could be a wonderful improvement over the current static one. I'd love to see what you have in mind. (in other words, I'm not one of those noscript people..;) )
I'm one of the noscript people, but I'm also for a nice html5 ui. But it'd better be standalone using the json api.
Kind regards,
Remigiusz Modrzejewski
Stephan Beal
I'm one of the noscript people, but I'm also for a nice html5 ui. But it'd better be standalone using the json api.
Stephan Beal
API we'll almost certainly create a separate interface (quite possibly as a separate
project)
Nolan Darilek
I like the JSON work, but I hope we'll eventually see a more dynamic means of creating internal HTML pages. After about a year of using Fossil, there are two features I'd dearly like to see: hooks, and the ability to query the internal database and output the results into a page. So, for instance, I'm currently storing one of my projects' sets of release notes as events. I'd like to pull the most recent 5, or 10, or whatever and display them on the main page such that http://spielproject.info doesn't look incredibly bare. :)
Anyhow, I mention the "year" bit because I'm sensitive to not wanting to bloat Fossil, and to establish that I don't have ten other feature requests waiting in the wings. The new JSON support is awesome, but I also hope that we see some way to make the current HTML interface a bit more dynamic, if only a little.
Thanks.
Stephan Beal
like to see: hooks, and the ability to query the internal database and output the results into a page.
Anyhow, I mention the "year" bit because I'm sensitive to not wanting to bloat Fossil, and to establish that I don't have ten other feature requests waiting in the wings. The new JSON support is awesome, but I also hope that we see some way to make the current HTML interface a bit more dynamic, if only a little.
Remigiusz Modrzejewski
On Oct 26, 2011, at 11:50 PM, Stephan Beal wrote:
> The topic of "real" hooks has come up many times, and the main reason it
> hasn't been added so far is platform portability. (Sorry, i don't mean to
> start another dead-horse-beating thread.)
Nope, this has already been resolved. The reason now is "there is not enough programmer throughput in the project". Or however it was worded.
Kind regards,
Remigiusz Modrzejewski
Nolan Darilek
On Wed, Oct 26, 2011 at 11:37 PM, Nolan Darilek <no...@thewordnerd.info> wrote:
like to see: hooks, and the ability to query the internal database and output the results into a page.The JSON API provides a "query" command[1] which allows you to run arbitrary queries and get the results as JSON, but it requires admin access (because it allows one to do anything with the db).
The topic of "real" hooks has come up many times, and the main reason it hasn't been added so far is platform portability. (Sorry, i don't mean to start another dead-horse-beating thread.)
Caleb Gray
I see the appeal in creating a separate HTML application. I will take this approach, and will see how everyone feels about having "installable skins" in Fossil: shipping it with only the "Default" skin.
Awesome, I didn't find either the JSON demo or "The Doc" while reading through everything. Thanks for the links!
I've already mailed my copyright release last week sometime, so it should be arriving any day now. :D
You are exactly on track with what I was envisioning for the JS enhancements with: http://mbostock.github.com/d3/talk/20111018/#8
@Konstantin Khomoutov, @Ron Aaron, @Lluís Batlle i Rossell:
You're absolutely right, the web interface is fine. What I'm saying that it could be enhanced with complimentary JavaScript, leaving all of the previous CSS/HTML alone. Disabled JS wouldn't be a problem at all; you just wouldn't see the enhancements.
@Ron Aaron:
I definitely agree that features like syntax highlighting can be very expensive. If the Fossil server is running a JS enhanced skin, the end-user will be presented with a way to disable features they feel aren't worth their computational time.
@Michael Barrow:
I'm hoping that we can abstract all of Fossil's hard-coded pages and skins, and have "installable skins" to dodge bloating Fossil with any and all unnecessary data.
@Dmitry Chestnykh:
I just wrote a script for testing the speed and size difference between the different compressions available, find the results here: http://uploads.calebgray.com/contributions/compression/index.html
As far as I know, memory usage depends entirely on dictionary size... so shouldn't DEFLATE and LZMA use the same amount of memory if configured correctly?
I'm used to 500KiB/s download speed, but my only choice at home is Clearwire (which is true, I'm sure, for more than just myself). Unfortunately, it's not rare for me to get 20KiB/s download speeds on it, if the Fossil releases were UPX compressed, that would have saved me ~5 seconds over the ZIP. Obviously, this isn't doesn't seem like a big deal, but keep in mind that the people in Australia/New Zealand have to pay for their bandwidth. It's not just time we're saving people, it's money too, in the long run.
UPX has zero effect on memory usage, and would probably add a millisecond or two to each request, leaving CPU as the only truly impacted factor... I suppose that if a Pentium 133 uncompressed at ~10MB/s as they claim on their homepage... then if you're getting 10 requests a second on a 1MB executable... your server would begin seeing the performance impact of Fossil being compressed using UPX. Anyway, I'm not a huge proponent to the idea, it was just a thought.
Dmitry Chestnykh
wrote:
> @Dmitry Chestnykh:
> I just wrote a script for testing the speed and size difference
> between the different compressions available, find the results here:
> http://uploads.calebgray.com/contributions/compression/index.html
Thanks! I grepped a bit, and found out that Fossil uses 9 level for
deflate, which seem to me _not_ a good balance; and LZMA 1
level outperforms it in both compression ratio and speed.
Plus I've heard that xz is for some reason slower than 7zip:
http://ck-hack.blogspot.com/2011/04/quick-lrzip-comparison.html
BTW, how LZMA performs for tiny files? I ask because most blobs in
Fossil in normal operation (source code) are tiny due to
delta compression.
> As far as I know, memory usage depends entirely on dictionary size...
> so shouldn't DEFLATE and LZMA use the same amount of memory if
> configured correctly?
Not sure, but I thought a good part of why LZMA is so good is that it
has a large dictionary size. Though, maybe low compression levels don't
require a lot of memory.
> I'm used to 500KiB/s download speed, but my only choice at home is
> Clearwire (which is true, I'm sure, for more than just myself).
> Unfortunately, it's not rare for me to get 20KiB/s download speeds on
> it, if the Fossil releases were UPX compressed, that would have saved
> me ~5 seconds over the ZIP. Obviously, this isn't doesn't seem like a
> big deal, but keep in mind that the people in Australia/New Zealand
> have to pay for their bandwidth. It's not just time we're saving
> people, it's money too, in the long run.
Sure, I understand. When I lived in Russia, the first Ethernet
connection I got was pretty expensive: $0.15 per megabyte or so (with
average income in Moscow ~$500 at that time). Today a lot of places
in the world have expensive mobile internet.
Still, 500 KB difference in a world of torrents, 500 MB text
editors, and 2 GB software updates? Come on! :-)
> UPX has zero effect on memory usage, and would probably add a
> millisecond or two to each request, leaving CPU as the only truly
> impacted factor... I suppose that if a Pentium 133 uncompressed at
> ~10MB/s as they claim on their homepage... then if you're getting 10
> requests a second on a 1MB executable... your server would begin
> seeing the performance impact of Fossil being compressed using UPX.
> Anyway, I'm not a huge proponent to the idea, it was just a thought.
UPX works by allocating a buffer of original executable size, then
uncompressing the file to it, and executing this buffer. The statically
compiled fossil running on my server (amd64) is ~5 MB. This means that
on each request, if I packed it with UPX, I'd get 5 MB allocation --
not good for my little VPS :-)
--
Dmitry Chestnykh
http://www.codingrobots.com
Stephan Beal
Awesome, I didn't find either the JSON demo or "The Doc" while reading through everything. Thanks for the links!
You are exactly on track with what I was envisioning for the JS enhancements with: http://mbostock.github.com/d3/talk/20111018/#8
Alek Paunov
> @ Stephan Beal:
> I see the appeal in creating a separate HTML application. I will take
> this approach, and will see how everyone feels about having "installable
> skins" in Fossil: shipping it with only the "Default" skin.
DB) skins with JSON datasources, I am sure that you also follow the
progress of ace.ajax.org and the other JS projects, which are good
candidates for employment ;-).
I think that the only missing part is feature for stored (parametrized)
SQL queries, which can be called with lower privileges via JSON interface.
> You are exactly on track with what I was envisioning for the JS
> enhancements with: http://mbostock.github.com/d3/talk/20111018/#8
+1: d3 is the best for the SVG part in my eyes too, it seems even more
powerful than his famous predecessor - protovis.org (and can be seen
also as smart XSLT like engine - thus, can have more applications than
just SVG drawing for fossil UX applications - i.e. "live" JSON->DOM
templates for non-SVG enhanced pages also)
Kind Regards,
Alek
Steve Bennett
On 10/26/2011 04:50 PM, Stephan Beal wrote:On Wed, Oct 26, 2011 at 11:37 PM, Nolan Darilek <no...@thewordnerd.info> wrote:
like to see: hooks, and the ability to query the internal database and output the results into a page.The JSON API provides a "query" command[1] which allows you to run arbitrary queries and get the results as JSON, but it requires admin access (because it allows one to do anything with the db).
Right, I understand that. I was hoping for an extension of TH1 that would allow for something like selecting the most X Y (where Y == commits, events, tickets, etc.) and displaying properties of those objects, such that the default generated page automatically included the most recent news items or whatever. This would eliminate the need for client-side rendering, and would make those pages searchable via Google, etc. I know that I've mentioned this before, and there has seemed to be some interest, but not enough to take it up and I unfortunately lack the skill/time. So it isn't a complaint, more a "here's something I wish I could have."