[Plugin] Presenting TW5-Uglify! Plugin compression the TiddlyWIki way!

[Flibbles]

Some people want a minified version of the TW5 core, but it'd be too much work to maintain.
Some people think it should remain uncompressed for the sake of editors, but TW5 continues to grow by the day.
Some developers compress their plugins, making efficient code, but it's a pain for anyone debugging it.

Some developers leave their plugins uncompressed, but that means it's larger than necessary for 90% of all end-users.

Now there's a solution for everybody: TW5-uglify.

This is a plugin primarily designed to be used by a Node.JS server. Your server has access to all your uncompressed and unpacked plugins, which is fine, since servers have the speed and memory for it. But what it sends to browsers are fully compressed plugins. Your little smart phones and tablets get the efficient TW5 they deserve.

Or maybe you're a plugin developer who wants to minify their code. Now you no longer have to resort to using npm-gulp like some kind of chump. Your build process can remain an entirely TiddlyWiki process.

Or maybe you want to miniify your standalone TiddlyWiki file. You can, but... think hard about this before you do it. Because unlike with TW5-Uglified server, compressing your standalone files is a one-way process. You'll forever have compressed code unless you download a new Tiddlywiki, redownload all your plugins, and transfer all your tiddlers from your old file. But hey! Maybe you're running TiddlyWiki on your 2000 era Nokia phone in an wifi wasteland. Then go ahead.

Check it out.

Here's the demo site.

And here's the github page.

Let me know what you all think.

[Saq Imtiaz]

@Flibbles hurrah! This has immediate implications and uses for work that my colleagues are doing and potentially for myself, where being able to reduce the file size of a standalone TW file is essential.

I will give it a whirl tomorrow.

Thank you.

Saq

[coda coder]

> Let me know what you all think.

Comprehensive, is what I think. Kudos.

[Joshua Fontany]

Wow, what an impressive accomplishment. Checking out with interest. :)

Best,
Joshua F

[Mohammad Rahmani]

Hi Flibbles

This is amazing! I am one of those who request for a minified TW5 and you did it!

The empty.html for TW5.1.24 is 2371 kB and the Uglified TW 5.1.24 is 1297 kB  EXCELLENT!! around 50% smaller!

We have a big pain distributing a TW with required plugins (empty.html ~ 4500 kB)

Love it, I will test it with much of interest.

One question: The doc says it compresses the JS code, how about plugins developed in pure wikitext? Commander is around 112kB, but I think it should be 25-40kB.

What do you think?

Best wishes

Mohammad

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/2ce6fad2-8600-4c73-940c-515548e49c1en%40googlegroups.com.

[Flibbles]

Hey Mohammad,

Currently, the bulk of the compression occurs in the javascript, but it does optimize all plugins somewhat by improving the way they're packed. Commander will come down a bit in size.

I'd love to have wikitext compression too; there are some things I can do. But it's super tricky. Unlike javascript, wikitext is dependent on white space. Definitely not sure if I could reduce them by over 60% like you're suggesting though, unless you're thinking of stream compression, but that'd be outside the scope of this plugin.

-Flibbles

[Mohammad Rahmani]

Flibbles, thanks for clarification. 

Best wishes

Mohammad

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/e3868d68-0328-4fe3-88c8-067116fcbeean%40googlegroups.com.

[TiddlyTweeter]

Re: Some people want a minified version of the TW5 core ... Now there's a solution for everybody: TW5-uglify.

Flibbles wrote ...

Check it out.

Here's the demo site.

And here's the github page.

 Flibbles asked ...

Let me know what you all think.

Whoah!

Could I have your brain---it is marvelous? :-)

I did a couple of initial tests (single file wikis with a lot of JS-heavy plugins) and its impressive! 

And no glitches after as far as I can see ... Wiki seems to behave exactly the  same, no problem.

This is a keeper!

One early point ...

• I do not think it is a problem it is a one-way-ticket for creating wikis you put online. It is just a step in a process.

Best wishes

TT

 

[Nicolas Petton]

Flibbles <challenge...@gmail.com> writes:

Hi Flibbles,

> Now there's a solution for everybody: TW5-uglify.

This is great! I really like the approach you took with NodeJS.

Thanks!
Nico

[TiddlyTweeter]

Flibbles wrote:

Let me know what you all think.

I did a further test (single file). I tried minifying the  Offical TW Full Edition (i.e. all plugins).

It went from 10.8 to 7Mb. Good!

FWIW along the way the browser froze on removal of Uglify but recovered in time, no problem.

Though one error message for one plugin in passing ...

 Failed to compress $:/plugins/tiddlywiki/innerwiki/innerwiki.js.

• message: Unexpected token: name «browser», expected: punc «,»
• line: 374
• col: 31
• pos: 11458

That plugin is "recursive" so it maybe is a special case?

Anyway, the compression is impressive!

Best wishes

TT

[TiddlyTweeter]

Side thoughts on text/wikitext uglification :-)

Flibbles wrote:

I'd love to have wikitext compression too; there are some things I can do. But it's super tricky.

 

Right. 

Maybe that would a later thought for other situations than we normally face? 

I'm thinking particularly of e-pubs where you can have huge texts (e.g. novels) with vast excess space use. But I have NO idea how you could get a minify of that working in a TW. 

Best wishes

TT

[Flibbles]

Hey @TiddlyTweeter

I checked out innerwiki.js. It's less about it being recursive, and more about innerwiki using async functions which are >=ECMA 6 features, and aren't supported by UglifyJS. No worries. It just means that single file wasn't compressed.

The freezing on modification is one issue I'm not sure what to do with. Apparently TW5 completely rerenders itself after every change, and without modifying the core, I don't think there's a way for Uglify to recognize the difference between a rendering from a change, and rendering because TW is saving to disk. Fortunately this problem only happens once per browser session, since results are cached, and it doesn't occur with NodeJS implementations.

Otherwise, glad you like it!

-Flibbles

[TiddlyTweeter]

 Flibbles wrote:

The freezing on modification is one issue I'm not sure what to do with. Apparently TW5 completely rerenders itself after every change, and without modifying the core, I don't think there's a way for Uglify to recognize the difference between a rendering from a change, and rendering because TW is saving to disk. 

""Freezing" is NOT a problem I think! So long as the user knows they have to wait they won't conclude it is a crash. And I WAS aware I might need to wait from the docs you provided already, so, no sweat. Though one thing that users maybe need fully grasp is its use is a one-way-ticket. It only needs to go to work once and you are done!

Great piece of work!

TT

[Ste]

Uglified TW 5.1.24 is 1297 kB.... 

Time to get out the floppy disks... 

[Mark S.]

Added to the TiddlyWiki Toolmap under Developer/Advanced Tools.

[okido]

Hi All,

If you require ES6 try Terser.js, it can be included like a browser script, no node.js required.

Have a nice day, Okido

Op woensdag 20 januari 2021 om 20:16:22 UTC+1 schreef Mark S.:

[PMario]

Hi,

Very interesting. .. Did you try to use the source.map function?

-mario

[Flibbles]

I don't think so. I'm not sure what that is, and I can't find it in the core. Uglify works by hijacking the View Widget, which is used for all plugin writing during saves. It's not great, but it's what I found works. If there's a better way, I'd gladly change it over.

[PMario]

On Wednesday, January 20, 2021 at 9:18:38 PM UTC+1 Flibbles wrote:

I don't think so. I'm not sure what that is,

I didn't have a look at your code and I also need to test the plugin in detail. ... It seems the uglify.js library is capable of creating source-map files. source-maps contain information for browsers, to connect "uglified code" with the original source code. So it's possible to debug ugly code, as if it is source code.

The system is designed to be used with js code, that is loaded from js-files that are served by servers and is activated by the html <script> tag.

TW js-code is activated by the TW core, using common-js functions ... So the mapping may be trickier, than normal. ... I just wanted to know, if you did explore this mechanism.

 

and I can't find it in the core.

That's right. TW core has no info about that atm.

 

Uglify works by hijacking the View Widget, which is used for all plugin writing during saves. It's not great, but it's what I found works.

I think that's a nice way to do it for TW node-server. Because it's transparent and seems to "just work"!

 

If there's a better way, I'd gladly change it over.

... As I wrote ... Just a question

-mario

[Jan Johannpeter]

Hi Flibbles,

great idea! My wikis tended to grow recently this seems to be a great remedy.

Some Questions: Could it also remove 

- overridden core Tiddlers

- temporarily deactivated plugins and inactive themes 

from the version which is served?

Best wishes Jan

[Flibbles]

Hey Jan,

I'm glad Uglify works well for you. To answer your questions

1. Removing overridden core Tiddlers could reduce the code size somewhat, but I'm not sure how often that use case would come up. Even in my beastly TiddlyWiki projects, my overridden tiddlers don't make up very much space, particularly since I (and most people I'd think) only override smallish tiddlers. If it's something larger, a monkey-patch injection is used more often. Meanwhile, removing those overridden shadows could lead to a whole slew of nasty bugs if someone A) Removed the override down the line, or B) Tried to export their project into any form other than exactly what it is.
2. Temporarily removing deactivated plugins seems out of scope for a content compression plugin (But just by a little. It compresses plugins. Removing them is a different job), and it's probably best done by the user. If they are serving versions of a tiddler where they want certain plugins excluded, they'd could use the publishFilter. A slightly elaborate tiddly-filter could exclude all deactivated plugins.

Hope this helps!

-Flibbles

[Jan Johannpeter]

Hi Flibbles, 

I think a publish filter is a good idea, I wanted to do that for years with Mat's publish-plugin but lost it out of sight. What is the best tool to do this right now?

It would be interesting to combine this with your plugin to have the maximum reduction.

Best wishes Jan

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/f7283bcf-1f16-426e-bfd7-402dfe712a45n%40googlegroups.com.

[Flibbles]

I'm not sure how you're "servering versions" or even what that means, but I assume it's about generating standalone tiddlywiki files from a NodeJS tiddlywiki. If that's the case, then a publish-filter already exists. It's part of the "rendertiddler" command. You can see the uglify tiddlywiki.info file for an example of how to generate a tiddlywiki file without certain plugins or files. In Uglify's case, it creates one without the jasmine plugin, or any tiddlers with the "Test/' prefix.

-Flibbles

[Siniy-Kit]

Thank you   Flibbles ! I use your plugin for TW 5.2.0 + replace svg icons by fontawesome (I use it my projects) TW5 become 1.2 Mb

Is it possible to compress it more an make 1 mb TW 5.2.0?

вторник, 19 января 2021 г. в 23:26:32 UTC+3, Flibbles:

[Flibbles]

Hey All!

You know what TW5-Uglify was missing? A nice, easy-to-use uglify wizard.

Before, if you wanted to compress your standalone tiddlywiki file, you had to follow a whole bunch of steps like some kind of recovery program. Now, just visit the wizard and drag your files on. It's build from a retrofitted tiddlywiki updated, but it works pretty well. Also, I've revamped the demo site's presentation in order to be much more concise and straight forward. Hopefully this'll help this plugin get the adoption it deserves.

Siniy-Kit: <1mb is super ambitious. But as it turns out, I've been considering compressing WikiText. I've thought a lot about it, and there are ways to do it, though I don't think we'll get nearly the trimming we get by compressing javascript.

[Flibbles]

And now, because I know you've all been waiting for it, CSS compression!

Uglify handles both your javascript and your static css files now. The gains aren't much, but it's a necessary step on the path to wikitext uglifying.