removing whitespace once and for all

Brian FitzGerald

unread,

Jan 14, 2016, 10:25:17 PM1/14/16

to framework-one

Hey guys, sorry I feel like this on has been discussed before, but I didn't see a clear resolution for me. We all know ColdFusion loves generating whitespace. I have output="false" all over the place and yet I'm still getting a lot of whitespace. Rather than try tracking every bit down, I would rather kill it right before rendering.

I have solved this for my layout by doing something like so:

// layouts/app.cfm

<cfsavecontent variable="local.layoutMarkup">
  <cfoutput>
    <!-- all my layout markup -->
  </cfoutput>
</cfsavecontent>
<cfoutput>#myCompressionUtility.compres(local.layoutMarkup)#</cfoutput>

Being a layout, I can do that in one place. Perhaps not ideal if you had a bunch of layouts, but it works for me here. However, I'm having trouble figuring out the best way to handle this for views and renderData() ajax responses. Any tips?

I was looking in setupView(), hoping to do something like this:

function setupView(rc) {
  rc.markup = myCompressionUtil.compres(rc.markup);
}

Thank you

Nando Breiter

unread,

Jan 15, 2016, 2:48:08 AM1/15/16

to framew...@googlegroups.com

Can you show us your compress() function?

Aria Media Sagl

+41 91 600 9601

Skype: ariamedia

--
FW/1 documentation: http://framework-one.github.io
FW/1 source code: http://github.com/framework-one/fw1
FW/1 chat / support: https://gitter.im/framework-one/fw1
FW/1 mailing list: http://groups.google.com/group/framework-one
---
You received this message because you are subscribed to the Google Groups "framework-one" group.
To unsubscribe from this group and stop receiving emails from it, send an email to framework-on...@googlegroups.com.
Visit this group at https://groups.google.com/group/framework-one.
For more options, visit https://groups.google.com/d/optout.

Sean Corfield

unread,

Jan 15, 2016, 2:52:17 AM1/15/16

to framework-one

If all your CFCs are script, you won’t get whitespace.

If you CFCs are tag-based, you’ll need output=”false” on all cffunction tags AND all cfcomponent tags – including your Application.cfc file. You certainly should not need to do any explicit whitespace “compression” in your code.

Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org

--

Sean Corfield

unread,

Jan 15, 2016, 2:52:19 AM1/15/16

to framework-one

Weird, I swear I replied to this but I don’t see it in my sent email or on the list…

If you’re using tag-based components, you need output=”false” on both cffunction AND cfcomponent tags.

If you’re using script-based components, you will get no whitespace.

You certainly should NOT be trying to programmatically compress / suppress whitespace in your code. You’re just doing it wrong at that point 😊

Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org

From: Brian FitzGerald
Sent: Thursday, January 14, 2016 7:25 PM
To: framework-one
Subject: [framework-one] removing whitespace once and for all

Hey guys, sorry I feel like this on has been discussed before, but I didn't see a clear resolution for me. We all know ColdFusion loves generating whitespace. I have output="false" all over the place and yet I'm still getting a lot of whitespace. Rather than try tracking every bit down, I would rather kill it right before rendering.

--

Sean Corfield

unread,

Jan 15, 2016, 3:03:00 AM1/15/16

to framework-one

Oh gee, thanks Google Mail, now I have two near-identical responses in this thread… ☹

Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org

Brian FitzGerald

unread,

Jan 15, 2016, 8:56:27 AM1/15/16

to framework-one

Jeez Sean, you're really passionate about not needing to remove whitespace! .... jk (about the multiple replies). Wait, jokes are bad when you have to explain them, right?

| If all your CFCs are script, you won’t get whitespace.

I am dealing w/ a large mix of script and tag based components here. I have output="false" (will double check) but even with that, it doesn't get me to my desired result (below).

To be clear, I'm trying to smash ALL whitespace and deliver my entire page to the browser in one compressed string, so instead of:

... potentially lots of whitespace missed in the calling code

<html>

  <head>
    ... somehow some more stubborn whitespace
  </head>

  <body>
    .... some awesome code and whitespace
    <p>hello world</p>
  </body>

</html>

I end up with something like this:

<html><head></head><body>Awesome code</body></html>

Using this technique on my main layout dropped the size of my home page (according to chrome dev tools network tab) from 14 kb to 10.4 kb.

| Can you show us your compress() function? - Nando

Sure thing. Spot anything problematic?

        <!--- compress input --->
	<cffunction name="compress" output="false">
		<cfargument name="input" required="true">
		<cfargument name="tabs" default="true">
		<cfargument name="newlines" default="true">
		<cfargument name="whitespace" default="true">
		<cfargument name="reset" default="false">
		
		<cfset var output = trim(arguments.input)>

		<cfif arguments.tabs>
			<cfset output = output.replaceAll("\t", " ")>
		</cfif>

		<cfif arguments.newlines>
			<cfset output = output.replaceAll("\r", " ")>
			<cfset output = output.replaceAll("\n", " ")>
		</cfif>

		<cfif arguments.whitespace>
			<cfset output = output.replaceAll("(\s)\s+", "$1")>
		</cfif>

		<cfif arguments.reset><cfcontent reset="true"></cfif>
		<cfreturn replace(output, "> <", "><", "all")>
	</cffunction>

Thanks for your insight.

Brian

Nando Breiter

unread,

Jan 16, 2016, 7:42:57 AM1/16/16

to framew...@googlegroups.com

Brian,

Here's a regex that may come in handy:

content = REReplace(content, "[[:space:]]{2,}","","ALL");

What that does is strip all whitespace characters in the string content, including tabs, spaces and new line chars, when there are 2 or more of them adjacent to each other.

What I have done in the past is simply wrap the entire layout in a cfsavecontent tag, including the #body# output from the views served by that layout, and use that regex to get a single compressed line of html. I'm told it is relatively expensive, but my idea was to compress requests that might be viewed over a mobile device on a slow connection.

That said, I've run into various issues, particularly with it breaking javascript. For example, if you have comments in your inline js, particularly those denoted by //, everything after the first comment is interpreted by the js parser as part of that comment after all whitespace is stripped. I've run into other issues as well with inline js.

So in js blocks, I've needed to take a progressive approach, as follows:

strip // comments

jsBlock = REReplace(jsBlock, "\/\*.*?\*\/","","ALL");

strip new line chars

jsBlock = REReplace(jsBlock, "\/\/(.*?)\n","","ALL");

strip any remaining consecutive space chars

jsBlock = REReplace(jsBlock, "[[:space:]]{2,}","","ALL");

Except that I found the above breaks the javascript if any urls are present, as it strips the rest of the line after http://, interpreting it as a comment.

At this point, I realized that there are better tools available to compress JS and CSS than what I could come up with with a few rereplace statements. Probably the best for JS is Google's Closure Compiler: https://developers.google.com/speed/articles/compressing-javascript. There is also Grunt and Gulp. Why compress js and css for every request when they remain the same for every request?

... except when they don't remain the same for every request. I sometimes generate inline javascript dynamically.

There's another potential problem I've run into: compressing the content within textarea tags. If a user has saved text with line breaks in it, when that text is opened for editing in a textarea tag, compression that removes line breaks renders their nicely formatted text in a single line, with not even a space between what used to be paragraphs. And when they save it again without going to the trouble of reformatting their text (which they may find useless after a couple of tries), it is persisted to the database that way.

What I am left with is a very piecemeal approach to compression of dynamic content if I am not to break functionality, one that targets cases where lots of indented data is output and leaves most other dynamic content alone. So while I could recommend that you simply wrap your entire app in a default.cfm layout, if you haven't already, wrap the markup in that layout with a cfsavecontent tag and use content = REReplace(content, "[[:space:]]{2,}","","ALL"); to nuke all whitespace, I don't think it's a good idea. Even if it works today, some tomorrow will likely bring a modification to your app that will not work the way it should in some strange way. And you might be left digging for hours or days trying to figure it out until it hits you - "Ah,it'sthewhitespacecompression.Damn!!!Ishouldhavethoughtofthatbefore!!!"

Aria Media Sagl
Via Rompada 40
6987 Caslano
Switzerland

+41 (0)91 600 9601
+41 (0)76 303 4477 cell
skype: ariamedia

Sean Corfield

unread,

Jan 16, 2016, 2:21:40 PM1/16/16

to framew...@googlegroups.com

I am dealing w/ a large mix of script and tag based components here. I have output="false" (will double check) but even with that, it doesn't get me to my desired result (below).

In addition to cfcomponent and cffunction in CFCs (which both need output="false"), tag-based code in views can generate a lot of whitespace too since every single tag generates a newline and indented whitespace and if you have a loop, you’ll get a lot of that. Consider:

Some html

Other html

<cfif>

</cfloop>

That generate five blank lines for each time around the loop, so the less logic you have in views, the better off you are. If you can move (nearly) all your logic to a cfsilent block at the top of each view, you’ll do better.

To be clear, I'm trying to smash ALL whitespace and deliver my entire page to the browser in one compressed string, so instead of:

Isn’t there an CF Admin level whitespace management feature that can help you here?

Seriously, doing this in your own CFML code is just a bad idea: you’re adding more logic / processing time to every request and any custom compression code introduces the likelihood of breakage, as Nando noted. And turning 14k into 10.4k is a tiny improvement in real terms that just isn’t worth the effort.

Sean

Julian Halliwell

unread,

Jan 16, 2016, 3:18:25 PM1/16/16

to framew...@googlegroups.com

My experience is that the built-in compression tools in both ACF and
Lucee aren't smart enough to deal with the anomalies Nando mentions,
so I've been using my own custom compression UDF on my outer layout
for years with little effect on CPU resources. Again in my experience
these are rarely in short supply in web applications. It's memory,
disk and DB/external resource calls that are expensive. As Richard
Campbell (IIS performance guy) puts it: most of the time processors on
modern web servers are just "sitting around smoking and playing
poker".

If you're just serving a few pages then of course a 4KB saving per
page doesn't matter. But for a busy site that can add up to
significant bandwidth, which you may be paying for by the MB. Many end
users will also be paying by the MB if they're on limited mobile data
plans and those connections will often be very slow, so the smaller
the HTML that needs to be delivered the better for speed and cost.

I think that doing what you can to minimise rendered page size is a
very good use of today's very fast CPUs.

One practical tip if you're trying to hunt down the cause of generated
whitespace: check for any .cfm includes you might have in your
components: Are they wrapped appropriately in <cfsilent>?

Cheers
Julian.

Nolan Erck

unread,

Jan 16, 2016, 4:29:31 PM1/16/16

to framew...@googlegroups.com

Aren't there various web server plugins (Apache mod_minify, for example) that will strip the extra whitespace out as the pages are send down to the browser? Then you're not dealing with adding output="false" all over your code, or worrying about CF Admin settings (tho the "minimize whitespace" option is easy to turn on and will certainly help).

Why not handle this at the web serer level instead?

--Nolan

Sean Corfield

unread,

Jan 16, 2016, 4:54:22 PM1/16/16

to framew...@googlegroups.com

Nolan Erck wrote on Saturday, January 16, 2016 at 1:29 PM:

Why not handle this at the web serer level instead?

A good suggestion. And enabling web server level gzip (as well as whitespace suppression) would have an even bigger impact on your download sizes.

If you’re concerned about bandwidth costs, consider a CDN (fastly, cloudflare, etc) since that will cache all the static assets so your servers won’t even be delivering that bandwidth (this is particularly good for JS-heavy sites as long as you separate your status JS from your dynamically generated JS — the latter is probably a bad idea anyway these days).

Sean Corfield -- (904) 302-SEAN

An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

Julian Halliwell

unread,

Jan 17, 2016, 4:42:11 AM1/17/16

to framew...@googlegroups.com

On 16 January 2016 at 21:54, Sean Corfield <se...@corfield.org> wrote:
> Nolan Erck wrote on Saturday, January 16, 2016 at 1:29 PM:
> Why not handle this at the web serer level instead?
>
> A good suggestion. And enabling web server level gzip (as well as whitespace
> suppression) would have an even bigger impact on your download sizes.
>
> If you’re concerned about bandwidth costs, consider a CDN

Both good suggestions. Do everything you can to minimise what travels
over the network. And that starts with making sure that what your app
spits out is as small as possible regardless of processing that may or
may not happen downstream.

Code level compression allows you to intelligently control whitespace
and costs very little in CPU terms these day. It's easier to implement
than a CDN, which won't address potential mobile bandwidth costs at
the client end.

Brian FitzGerald

unread,

Jan 17, 2016, 10:27:23 AM1/17/16

to framework-one

Thanks so much to everyone for the thoughtful replies. For what it's worth, I already have things like gzip enabled.

There are certainly solid points on both sides of the discussion, though I tend to side with Julian's viewpoint on the topic that the processing time is a non-factor in order to save precious bytes being sent over the wire (granted, one still needs to be careful, as Nando point out so well).

At the end of the day, my point is that it should be the choice of the developer/architect how they want to handle it in there app. It is my opinion then, is that there should be a convenient way within a fw/1 application to "get at" the output (for both full page and for ajax requests) that is going to be sent back to the client just before it is.

To me, this is how it should be done: the framework does whatever it needs to do to prepare the final output and again, this final output should be available to the developer in something like setupView() or some other application wide tie in point which would run just before the content is sent back.

I have just tested setupView() and setupResponse() but after inspecting both rc and the request scope (bad practice I know, but just checking), I don't see the markup anywhere that I could modify it before delivery. setupResponse() appears to run too late for this, but setupView() seems to run at the right time (for both full page loads and ajax requests) to perform this type of application level compression.

That said, if I'm just spinning my wheels and no one else thinks the ability to access the markup before final delivery should be part of the framework, then I will ask you Julian, where have you put your compression code to handle this task? I don't like the idea of having to wrap each view in a cfsavecontent like I did for my main layout, and I'm not even sure what I would need to do to implement such functionality for ajax responses delivered via renderData().

Thanks again for the discussion, you guys are awesome.

Brian

Nando Breiter

unread,

Jan 17, 2016, 12:09:43 PM1/17/16

to framew...@googlegroups.com

Brian,

Copy and paste the following into a default.cfm file in your layouts directory, on the first line at the top:

<cfoutput>#REReplace(body, "[[:space:]]{2,}","","ALL")#</cfoutput>

I'm assuming you currently do not have a default.cfm file in there. Otherwise you can wrap the whole thing in a cfsavecontent tag and apply that regex to the variable the tag produces. All whitespace chars will be stripped from the output from this single point. Not sure how Sean feels about this, but I'd suggest experimenting with this method for some time before any mods are made to the framework to allow a single point of access for such a thing.

And to anyone else following along, I do not recommend doing this. If you have only a single textarea tag in your entire app to collect data from users, you will nuke any line returns they place in their text, destroying user data. And as stated before, you will very likely break any inline js in your views or layouts. Note also that any future modifications that are made to the app by you or someone else can be broken using this approach, and debugging these may be very difficult, because you will likely forget that the aggressive stripping of whitespace is a potential source for bugs.

As a point of reference, even Twitter does not strip whitespace wholesale like this.

Disclaimers made, have at it Brian!

Aria Media Sagl
Via Rompada 40
6987 Caslano
Switzerland

+41 (0)91 600 9601
+41 (0)76 303 4477 cell
skype: ariamedia

--

Julian Halliwell

unread,

Jan 17, 2016, 3:43:06 PM1/17/16

to framew...@googlegroups.com

On 17 January 2016 at 15:27, Brian FitzGerald <bmfitz...@gmail.com> wrote:
> Julian, where have you put your compression code to handle this task?

Same place as you Brian: I wrap the final layout in a savecontent and
apply the compression to that variable.

But as you say that won't work for ajax calls. We don't tend to use
much ajax on public sites so I normally leave it at just making sure
there's no leading whitespace by using script components and
<cfsilent> (FW/1 definitely won't add any so if you're seeing leading
white space it'll be your code that's generating it).

Have you thought about simply rendering your own ajax content from
your controllers rather than leaving it to FW/1? That would give you
full control. I actually have a custom "outputContent" method which I
prefer using instead of renderData() because it's slightly more
flexible. I've not thought of applying the compression here but I
can't think of any reason why it wouldn't work:

void function outputContent( required string content,required
myCompresionUtil,string type="text/html",string charset="utf-8" ){
content type="#type#; charset=#charset#" reset="yes";
WriteOutput( myCompressionUtil.compress( content ) );
abort;
}

Cheers
Julian.

Sean Corfield

unread,

Jan 17, 2016, 7:04:33 PM1/17/16

to framew...@googlegroups.com

On 1/17/16, 12:42 PM, "Julian Halliwell" <framew...@googlegroups.com on behalf of julianh...@gmail.com> wrote:

>Have you thought about simply rendering your own ajax content from
>your controllers rather than leaving it to FW/1? That would give you
>full control. I actually have a custom "outputContent" method which I
>prefer using instead of renderData() because it's slightly more

>Flexible.

Check out the enhancements to renderData() in 4.0 — provide custom renderers is supported completely within the framework now.

Sean

Julian Halliwell

unread,

Jan 18, 2016, 3:43:55 AM1/18/16

to framew...@googlegroups.com

Excellent! Look forward to giving that a go.

Brian FitzGerald

unread,

Jan 18, 2016, 7:56:54 AM1/18/16

to framework-one

@Nando, twitter just needs to stop slacking if you ask me!

Jk, thanks so much for your additional reply. With your insights regarding the default layout technique (and shared regex option) along with Sean and Julian's comments regarding renderData() or alternative approaches to dealing with whitespace in ajaxed content, I'm sure I have plenty of information to make some informed decisions at this point.

Much appreciated all.

Reply all

Reply to author

Forward