lucee unstable under assault?

[Tim Clarke]

We have a new centos 6.7 install with the Lucee packaged installer (Lucee 4.5.1.024 on Tomcat 8.0.24). Its running our ex-Railo app quite happily under normal load but when we point an external PCI compliance tester at it, Lucee becomes unresponsive. There's nothing in catalina.out except possibly this:

Exception in thread "http-nio-8888-exec-10" java.lang.IllegalMonitorStateException
        at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:151)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
        at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:449)
        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)

The process is still there when it becomes unresponsive:

"ps wauxf" gives us:

lucee    22865  3.7  3.8 2529420 312460 pts/0  Sl   15:40   0:30 /opt/lucee/jdk/jre/bin/java -Djava.util.logging.config.file=/opt/lucee/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms256m -Xmx2048m -XX:MaxPermSize=512m -javaagent:lib/lucee-inst.jar -Djava.endorsed.dirs=/opt/lucee/tomcat/endorsed -classpath /opt/lucee/tomcat/bin/bootstrap.jar:/opt/lucee/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/lucee/tomcat -Dcatalina.home=/opt/lucee/tomcat -Djava.io.tmpdir=/opt/lucee/tomcat/temp org.apache.catalina.startup.Bootstrap start

not spectacularly large. We've set extra memory in the Java startup as you can see, together with extra threads in lucee/tomcat/conf/server.xml:

    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
        maxThreads="1500" minSpareThreads="10"/>
.......
    <Connector port="8888" protocol="HTTP/1.1"
               maxThreads="1500"
               connectionTimeout="20000"
               redirectPort="8443"
               server="Jetty" />
........
<Connector port="8009" protocol="AJP/1.3" maxThreads="1500" redirectPort="8443" />

but tbh I'm shooting in the dark here. Any ideas please?

[Andrew Dixon]

Hi Tim,

Which:

1) web server and version

2) Java and version

3) Which OS and version

Kind regards,

Andrew

about.me - mso - Lucee Association Member

--
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/2588f0d7-358d-4fda-a7ce-fb6f4eec7763%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Clarke]

Centos 6.7 as I said plus java version "1.8.0_65" and  apache httpd Apache/2.2.15

[Mark Drew]

More questions… are you connecting via AJP or HTTP? and how are you doing it? What’s your apache config? is it just apache AJP connector or using mod_cfml?

Regards

MD

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/ab946851-7d85-4bb6-84c7-ef17cc783d6e%40googlegroups.com.

[Andrew Dixon]

Hi Tim

Firstly I would remove -XX:MaxPermSize=512m as this is no longer valid in Java 1.8 and not required, don't think this will help but it's good to tidy up. 

Also I would change the -Xms and -Xmx to be the same value, so -Xms2048m -Xmx2048m, this saves it from doing too much garbage collection, which might be the problem once you are under load. 

I would then install FusionReactor and see what that shows you when you are under load, as it will probably help point you in the direction of the issue.

Also see Mark's questions.

Kind regards,

Andrew

about.me - mso - Lucee Association Member

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/ab946851-7d85-4bb6-84c7-ef17cc783d6e%40googlegroups.com.

[Tim Clarke]

On Monday, 9 November 2015 16:28:17 UTC, Mark Drew wrote:

More questions… are you connecting via AJP or HTTP? and how are you doing it? What’s your apache config? is it just apache AJP connector or using mod_cfml?

Regards

MD

Questions are good:)

.. AJP or HTTP? tbh I'm not sure how they are doing it - as I said its an external PCI compliance test.
...apache config attached

[Tim Clarke]

On Monday, 9 November 2015 16:33:07 UTC, Andrew Dixon wrote:

Hi Tim

Firstly I would remove -XX:MaxPermSize=512m as this is no longer valid in Java 1.8 and not required, don't think this will help but it's good to tidy up. 

Also I would change the -Xms and -Xmx to be the same value, so -Xms2048m -Xmx2048m, this saves it from doing too much garbage collection, which might be the problem once you are under load. 

I would then install FusionReactor and see what that shows you when you are under load, as it will probably help point you in the direction of the issue.

Also see Mark's questions.

Kind regards,

Andrew

I've put those suggestions in place and am re-running, thanks Andrew.

 

[Tim Clarke]

The re-run has stopped with the process not responding again. The last section of catalina.out is:

Exception in thread "http-nio-8888-exec-6" Exception in thread "http-nio-8888-exec-7" java.lang.IllegalMonitorStateException

[Jordan Michaels]

If you're using a default install of Lucee done via the installer, then you're connecting Apache to Tomcat over HTTP using mod_proxy_http. This is further evident due to the fact that you're hitting an error on Tomcat's 8888 port - which is the default HTTP port in an installer build.

In your stack trace, no Lucee classes are being hit, so the problem is directly related to something Tomcat itself is doing.

I did a couple quick reads on the "IllegalMonitorStateException" error, and it looks like you're hitting some kind of runtime error with relation to threading:
http://examples.javacodegeeks.com/java-basics/exceptions/java-lang-illegalmonitorstateexception-how-to-solve-illegalmonitorstateexception/

Does your server limit threading in any way? For example, are you on a Virtual Machine or some other kind of service that can impact threading? There may also be connector settings that can be adjusted to correct for this error, but we won't know what settings to adjust until we know why this error is being thrown in the first place.

If I were you, I'd start my search there.

Kind regards,
Jordan Michaels

--
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/7c50cf20-7fc7-45b0-a377-0b879c4afe42%40googlegroups.com.

[Tim Clarke]

On Monday, 9 November 2015 20:09:08 UTC, Jordan Michaels wrote:

If you're using a default install of Lucee done via the installer, then you're connecting Apache to Tomcat over HTTP using mod_proxy_http. This is further evident due to the fact that you're hitting an error on Tomcat's 8888 port - which is the default HTTP port in an installer build.

In your stack trace, no Lucee classes are being hit, so the problem is directly related to something Tomcat itself is doing.

I did a couple quick reads on the "IllegalMonitorStateException" error, and it looks like you're hitting some kind of runtime error with relation to threading:
http://examples.javacodegeeks.com/java-basics/exceptions/java-lang-illegalmonitorstateexception-how-to-solve-illegalmonitorstateexception/

Does your server limit threading in any way? For example, are you on a Virtual Machine or some other kind of service that can impact threading? There may also be connector settings that can be adjusted to  correct for this error, but we won't know what settings to adjust until we know why this error is being thrown in the first place.

If I were you, I'd start my search there.

Kind regards,
Jordan Michaels

Exactly right, Jordan, and I found Mr Maneas' excellent tutorial earlier. Unfortunately it doesn't help me since the problem is occurring inside Tomcat. This Lucee/Tomcat install is running directly on a new Centos 6.7 install, not a VM. I have no idea where one might "limit server threading" inside that set-up, we have *increased* the Tomcat maxThreads settings in server.xml as already detailed above.

tbh I have no idea where to search since debugging Tomcat itself is quite out of my league.

[Jordan Michaels]

> directly on a new Centos 6.7 install, not a VM

So, this is dedicated hardware?

The only reason I mention it is because kernel virtualization software
can restrict the threading of the host OS. I was simply shooting for
ideas of what might be causing issues with threading. It's quite
possible that it's not relevant to your situation. I wouldn't know
without doing more research.

Warm Regards,
Jordan Michaels

> --
> Love Lucee? Become a supporter and be part of the Lucee project today! -
> http://lucee.org/supporters/become-a-supporter.html
> ---
> You received this message because you are subscribed to the Google
> Groups "Lucee" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to lucee+un...@googlegroups.com

> <mailto:lucee+un...@googlegroups.com>.

> To post to this group, send email to lu...@googlegroups.com

> <mailto:lu...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/lucee/fab7461d-96a4-4400-b2b5-6168e1746b0c%40googlegroups.com
> <https://groups.google.com/d/msgid/lucee/fab7461d-96a4-4400-b2b5-6168e1746b0c%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Tim Clarke]

On Monday, 9 November 2015 23:54:10 UTC, Jordan Michaels wrote:

> directly on a new Centos 6.7 install, not a VM

So, this is dedicated hardware?

The only reason I mention it is because kernel virtualization software
can restrict the threading of the host OS. I was simply shooting for
ideas of what might be causing issues with threading. It's quite
possible that it's not relevant to your situation. I wouldn't know
without doing more research.

Warm Regards,
Jordan Michaels

It is dedicated, Jordan,  thanks for trying to help. I hope someone else chimes in.

[Mark Drew]

I would second the use of FusionReactor in this instance by the way. You will be able to monitor what is going on a lot better and see what requests are making the server inactive. It could be buffer overflow tests that the PCI company is running for example. 

Without this you are just taking stabs I the dark. 

If not. Try Newrelic, although the FR is much better. 

I am not at my desk (but will be later to copy the setting) but you also want to do  heap dump on out of memory errors (google it) setting in your setenv.sh to see what was in memory. It might help if you are actually crashing the server. 

Mark Drew

- Sent by typing with my thumbs. 

--

Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---
You received this message because you are subscribed to the Google Groups "Lucee" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/c94ca271-b87a-4e41-b5c2-915f7556b9dc%40googlegroups.com.

[Tim Clarke]

On Tuesday, 10 November 2015 09:39:17 UTC, Mark Drew wrote:

I would second the use of FusionReactor in this instance by the way. You will be able to monitor what is going on a lot better and see what requests are making the server inactive. It could be buffer overflow tests that the PCI company is running for example. 

Without this you are just taking stabs I the dark. 

If not. Try Newrelic, although the FR is much better. 

I am not at my desk (but will be later to copy the setting) but you also want to do  heap dump on out of memory errors (google it) setting in your setenv.sh to see what was in memory. It might help if you are actually crashing the server. 

Mark Drew

fusionreactor installed, server restarted and a new PCI scan initiated.

Picture "Screenshot from 2015-11-10 18-13-50.png" was caught as the freeze started, the server in fact did respond, just very slowly. A page of our application without CSS could be loaded to a browser. "Screenshot from 2015-11-10 18-14-06.png" contains the detailed lower stats from the page. Now I hope I'm dumping something informative here lol

After about 10 minutes only this can be loaded:

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /.

Reason: Error reading from remote server

---

Apache/2.2.15 (CentOS) Server at myserver.com Port 443



fusionreactor still continues to update, however, and I can see memory usage (top right graph) slowly increasing then dropping, presumably as garbage collection is performed. The PCI scan fails still but only on one item: "CGI Generic Cross-Site Scripting" (grrr why can't I turn off italics?)

[Tim Clarke]

On Tuesday, 10 November 2015 18:48:23 UTC, Tim Clarke wrote:

fusionreactor installed, server restarted and a new PCI scan initiated.

Picture "Screenshot from 2015-11-10 18-13-50.png" was caught as the freeze started, the server in fact did respond, just very slowly. A page of our application without CSS could be loaded to a browser. "Screenshot from 2015-11-10 18-14-06.png" contains the detailed lower stats from the page. Now I hope I'm dumping something informative here lol

After about 10 minutes only this can be loaded:

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /.

Reason: Error reading from remote server

---

Apache/2.2.15 (CentOS) Server at myserver.com Port 443



fusionreactor still continues to update, however, and I can see memory usage (top right graph) slowly increasing then dropping, presumably as garbage collection is performed. The PCI scan fails still but only on one item: "CGI Generic Cross-Site Scripting" (grrr why can't I turn off italics?)

A clue possibly in this thread graph?

[Jon Clausen]

Those that have chimed in already are much better with the JVM, Tomcat and Lucee server than I am, but the screenshots that I’m seeing (and what’s described in the rest of this thread) suggest that it’s not a JVM or Tomcat issue, but is rather something in the application, itself, that’s causing a race condition.  Are there any writes to the application scope or locks which take place every request or I/O actions such as logging that fire on errors which might have permissions issues which trigger additional errors, additional loging attempts, etc?  Those can produce these kinds of lock-ups, in my experience. If you have a high request timeout setting, conditions causing the race will not resolve themselves and the server will eventually stop responding. 

In many early CFML libraries, before OOP practices were widely implemented in CFML, locks were used quite a bit, so it’s possible that, if this has a legacy code base, there’s a old third party library that’s the culprit.

Jon 

--
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/c08e9dc9-c9b5-4dbe-825b-439cc7e755d0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jon Clausen]

An observation to illustrate my point:  you’ve got over 18,000 404’s (which PCI testing will produce).  Do you have a site wide error handler in place for missing templates that might be the culprit?

[Tim Clarke]

On 10 November 2015 at 18:59, Jon Clausen <jon_c...@silowebworks.com> wrote:

Those that have chimed in already are much better with the JVM, Tomcat and Lucee server than I am, but the screenshots that I’m seeing (and what’s described in the rest of this thread) suggest that it’s not a JVM or Tomcat issue, but is rather something in the application, itself, that’s causing a race condition.  Are there any writes to the application scope or locks which take place every request or I/O actions such as logging that fire on errors which might have permissions issues which trigger additional errors, additional loging attempts, etc?  Those can produce these kinds of lock-ups, in my experience. If you have a high request timeout setting, conditions causing the race will not resolve themselves and the server will eventually stop responding. 

In many early CFML libraries, before OOP practices were widely implemented in CFML, locks were used quite a bit, so it’s possible that, if this has a legacy code base, there’s a old third party library that’s the culprit.

Jon 

Ty for the thought, Jon, but since the application begins with a login page, the PCI scan is unable to access anything of our code except that. I've scanned the errors caused by the scan in our application log and you can see attempt-by-attempt these categories:

page not found

...has no remote function with name....
The parameter returnTo to function logIn is required but was not passed in
invalid url [empty string] for attribute url
component [controllers.LocalizationController] has no remote function with name

I'm sure that's not exhaustive but it certainly covers that last 200+ error lines. Looks like they can't get in to our app?

[Jon Clausen]

Tim,

That’s a start,then.  First off,  I would set a default value for “returnTo”, so that login redirects don’t throw 500’s.  Secondly, a lot of those PCI URLS being passed are for other language file extensions.  Do you have those being handled by the web server, or is Lucee forced to handle those as well?  You can very easily handle those in your rewrite rules, if you have them. 

Since your exception log i/o actions are firing on 500’s, rather than the web server’s log appender, it’s best to deal with those 404’s at the web server level and save yourself the expense on the app server.   With 18,000 404’s in a short period of time, that will take an instant load off of your application server.

Jon

On November 10, 2015 at 2:07:05 PM, Tim Clarke (hey...@gmail.com) wrote:

Ty for the thought, Jon, but since the application begins with a login page, the PCI scan is unable to access anything of our code except that. I've scanned the errors caused by the scan in our application log and you can see attempt-by-attempt these categories:

page not found

...has no remote function with name....
The parameter returnTo to function logIn is required but was not passed in
invalid url [empty string] for attribute url
component [controllers.LocalizationController] has no remote function with name

I'm sure that's not exhaustive but it certainly covers that last 200+ error lines. Looks like they can't get in to our app?

--
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/CAGC6_%2BoC2vsGkZY19iA0G0stsUrhdV9%3DiUyBhJqRPOxT%2BahhOw%40mail.gmail.com.

[Tim Clarke]

On Tuesday, 10 November 2015 19:16:14 UTC, Jon Clausen wrote:

Tim,

That’s a start,then.  First off,  I would set a default value for “returnTo”, so that login redirects don’t throw 500’s.  Secondly, a lot of those PCI URLS being passed are for other language file extensions.  Do you have those being handled by the web server, or is Lucee forced to handle those as well?  You can very easily handle those in your rewrite rules, if you have them. 

Amended returnTo parameter.

apache blocks in place (I'm really unhappy with this kind of blacklisting - I've just stabbed at a broad range that leaves our service happily doing its thing but blocks a high number of intrustions):

    RewriteEngine On
    RewriteRule ^/amazon - [F]
    RewriteRule ^/biz - [F]
    RewriteRule ^/certsrv - [F]
    RewriteRule ^/CFIDE - [F]
    RewriteRule ^/cgi - [F]
    RewriteRule ^/dev - [F]
    RewriteRule ^/fusetalk - [F]
    RewriteRule ^/flatnuke - [F]
    RewriteRule ^/flex - [F]
    RewriteRule ^/forum - [F]
    RewriteRule ^/guestbook - [F]
    RewriteRule ^/gbook - [F]
    RewriteRule ^/gb - [F]
    RewriteRule ^/idealbb - [F]
    RewriteRule ^/ifx - [F]
    RewriteRule ^/insertorder.cfm - [F]
    RewriteRule ^/jffnms - [F]
    RewriteRule ^/officescan - [F]
    RewriteRule ^/php - [F]
    RewriteRule ^/railo - [F]
    RewriteRule ^/texis.exe - [F]
    RewriteRule ^/script - [F]
    RewriteRule ^/shop - [F]
    RewriteRule ^/sitebuilder - [F]
    RewriteRule ^/store - [F]
    RewriteRule ^/viewcvs.cgi - [F]
    RewriteRule ^/w - [F]
    RewriteRule ^/x - [F]
    RewriteRule ^/y - [F]
    RewriteRule ^/z - [F]
 

Since your exception log i/o actions are firing on 500’s, rather than the web server’s log appender, it’s best to deal with those 404’s at the web server level and save yourself the expense on the app server.   With 18,000 404’s in a short period of time, that will take an instant load off of your application server.

Jon

Am running PCI test again now.

[Gert Franz]

Hey Tim,

 

Any update on this? Did it work now?

 

Sincerely
Gert Franz

 

RASIA GmbH

Spittelgasse 7

5103 Moeriken-Wildegg

Email: ge...@rasia.ch
Skype: gert.franz

Phone Switzerland: +41 76 5680 231

--

Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/5c674160-87ad-40b0-9834-a8d5096050d0%40googlegroups.com.

[Tim Clarke]

On Thursday, 12 November 2015 16:09:09 UTC, Gert Franz wrote:

Hey Tim,

 

Any update on this? Did it work now?

 

Sincerely
Gert Franz

Yesyes! The service now stays up under the PCI test loading.

Sincerest props to Jon Clausen. On to the PCI test results. Oo-er.... :/

[Sean Daniels]

Tim, I'll just throw this out there as someone who was also quite annoyed with the burden of PCI...

Not sure if it's feasible in your situation or application, but there are tons of payment gateways now that offer account number vaulting and hosted payment pages - these two combined services can relieve your organization sometimes entirely of it's PCI compliance burden (the gateway processor becomes the responsible organization).

Something to think about. I am fortunate to be able to utilize these services and it's been a nice relief to not have to deal with that stuff anymore (logistically and emotionally!).

> --
> Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
> ---
> You received this message because you are subscribed to the Google Groups "Lucee" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
> To post to this group, send email to lu...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/1b562ac8-b581-43be-8024-d00c63588d30%40googlegroups.com.

[Andrew Dixon]

+1 to what Sean said... De-scope from PCI, it's not worth it.

Kind regards,

Andrew

about.me - mso - Lucee Association Member

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/CFA0E2D0-B68D-4C47-9624-4E17C3BAD4C9%40gmail.com.

[Tim Clarke]

On Thursday, 12 November 2015 17:01:27 UTC, Andrew Dixon wrote:

+1 to what Sean said... De-scope from PCI, it's not worth it.

Kind regards,

Andrew

about.me - mso - Lucee Association Member

On 12 November 2015 at 16:51, Sean Daniels <daniel...@gmail.com> wrote:

Tim, I'll just throw this out there as someone who was also quite annoyed with the burden of PCI...

Not sure if it's feasible in your situation or application, but there are tons of payment gateways now that offer account number vaulting and hosted payment pages - these two combined services can relieve your organization sometimes entirely of it's PCI compliance burden (the gateway processor becomes the responsible organization).

Something to think about. I am fortunate to be able to utilize these services and it's been a nice relief to not have to deal with that stuff anymore (logistically and emotionally!).

I hope you've found some solace now, Daniel, but alas I cannot - we are compelled to do this.

[Jon Clausen]

Sweet!  Glad I could help.  I think 99.9% of all issues I’ve ever had with an ACF, Railo, or Lucee server becoming unresponsive have had something to do with exception handling and/or log files. :)

Yesyes! The service now stays up under the PCI test loading.

Sincerest props to Jon Clausen. On to the PCI test results. Oo-er.... :/

--
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html
---
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/1b562ac8-b581-43be-8024-d00c63588d30%40googlegroups.com.