CDF Issues

[Dusan Djuric]

Hi people (cc Pedro Alves )

I would love to become Committer-member to pentaho-cdf project.
Just for start, I have some issues to post

Here is one (Pedro can copy-Past to project issues):

Component timePlot not works properly when MDX query references any
dimension column
or nameColumn of values with Croatian characters (diacritics) symbols
ć Ć, č Č, đ Đ, š Š, and ž Ž

This is an pentaho.log section with error details:
___________________________________________________
mondrian.olap.MondrianException: Mondrian Error:Failed to parse query
' SELECT NON EMPTY {[Measures].[BrojOsobaUZSJ]} ON COLUMNS,
{[Vrijeme.Datumi].[20080101]:[Vrijeme.Datumi].[20081231]} ON ROWS
FROM [Statistika] where ([MSegmenti.VrsteMS].[Sve],[Zemlje.Grupe].
[Sve].[ISTOÄŒNA AZIJA])'
at mondrian.resource.MondrianResource$_Def0.ex(MondrianResource.java:
785)
at mondrian.olap.ConnectionBase.parseQuery(ConnectionBase.java:134)
at mondrian.olap.ConnectionBase.parseQuery(ConnectionBase.java:59)
at
org.pentaho.platform.plugin.services.connections.mondrian.MDXConnection.executeQuery
(MDXConnection.java:238)
at org.pentaho.platform.plugin.action.mdx.MDXBaseComponent.runQuery
(MDXBaseComponent.java:327)
at
org.pentaho.platform.plugin.action.mdx.MDXBaseComponent.executeAction
(MDXBaseComponent.java:189)
at org.pentaho.platform.engine.services.solution.ComponentBase.execute
(ComponentBase.java:461)
at
org.pentaho.platform.engine.services.runtime.RuntimeContext.executeComponent
(RuntimeContext.java:1322)
at
org.pentaho.platform.engine.services.runtime.RuntimeContext.executeAction
(RuntimeContext.java:1289)
at
org.pentaho.platform.engine.services.runtime.RuntimeContext.performActions
(RuntimeContext.java:1207)
at
org.pentaho.platform.engine.services.runtime.RuntimeContext.executeLoop
(RuntimeContext.java:1156)
at
org.pentaho.platform.engine.services.runtime.RuntimeContext.executeSequence
(RuntimeContext.java:1036)
at
org.pentaho.platform.engine.services.runtime.RuntimeContext.performActions
(RuntimeContext.java:1198)
at
org.pentaho.platform.engine.services.runtime.RuntimeContext.executeLoop
(RuntimeContext.java:1156)
at
org.pentaho.platform.engine.services.runtime.RuntimeContext.executeSequence
(RuntimeContext.java:1036)
at
org.pentaho.platform.engine.services.runtime.RuntimeContext.executeSequence
(RuntimeContext.java:929)
at
org.pentaho.platform.engine.services.solution.SolutionEngine.executeInternal
(SolutionEngine.java:413)
at
org.pentaho.platform.engine.services.solution.SolutionEngine.execute
(SolutionEngine.java:316)
at
org.pentaho.platform.engine.services.solution.SolutionEngine.execute
(SolutionEngine.java:192)
at
org.pentaho.platform.engine.services.BaseRequestHandler.handleActionRequest
(BaseRequestHandler.java:159)
at org.pentaho.platform.web.servlet.ViewAction.handleActionRequest
(ViewAction.java:148)
at org.pentaho.platform.web.servlet.ViewAction.doGet(ViewAction.java:
275)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter
(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter
(ApplicationFilterChain.java:188)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter
(FilterChainProxy.java:265)
at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke
(FilterSecurityInterceptor.java:107)
at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter
(FilterSecurityInterceptor.java:72)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter
(FilterChainProxy.java:275)
at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter
(ExceptionTranslationFilter.java:124)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter
(FilterChainProxy.java:275)
at org.acegisecurity.ui.switchuser.SwitchUserProcessingFilter.doFilter
(SwitchUserProcessingFilter.java:341)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter
(FilterChainProxy.java:275)
at
org.pentaho.platform.web.http.security.SecurityStartupFilter.doFilter
(SecurityStartupFilter.java:85)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter
(FilterChainProxy.java:275)
at
org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter
(AnonymousProcessingFilter.java:125)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter
(FilterChainProxy.java:275)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter
(RememberMeProcessingFilter.java:142)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter
(FilterChainProxy.java:275)
at
org.pentaho.platform.web.http.security.RequestParameterAuthenticationFilter.doFilter
(RequestParameterAuthenticationFilter.java:169)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter
(FilterChainProxy.java:275)
at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter
(BasicProcessingFilter.java:174)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter
(FilterChainProxy.java:275)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter
(AbstractProcessingFilter.java:271)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter
(FilterChainProxy.java:275)
at org.acegisecurity.ui.logout.LogoutFilter.doFilter
(LogoutFilter.java:110)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter
(FilterChainProxy.java:275)
at
org.pentaho.platform.web.http.security.HttpSessionReuseDetectionFilter.doFilter
(HttpSessionReuseDetectionFilter.java:134)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter
(FilterChainProxy.java:275)
at
org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter
(HttpSessionContextIntegrationFilter.java:249)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter
(FilterChainProxy.java:275)
at
org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter
(SecurityContextHolderAwareRequestFilter.java:81)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter
(FilterChainProxy.java:275)
at org.acegisecurity.util.FilterChainProxy.doFilter
(FilterChainProxy.java:149)
at org.acegisecurity.util.FilterToBeanProxy.doFilter
(FilterToBeanProxy.java:98)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter
(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter
(ApplicationFilterChain.java:188)
at org.pentaho.platform.web.http.filters.SystemStatusFilter.doFilter
(SystemStatusFilter.java:60)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter
(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter
(ApplicationFilterChain.java:188)
at
org.pentaho.platform.web.http.filters.SetCharacterEncodingFilter.doFilter
(SetCharacterEncodingFilter.java:113)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter
(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter
(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke
(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke
(StandardContextValve.java:174)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke
(AuthenticatorBase.java:433)
at org.apache.catalina.core.StandardHostValve.invoke
(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke
(ErrorReportValve.java:117)
at org.apache.catalina.core.StandardEngineValve.invoke
(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service
(CoyoteAdapter.java:174)
at org.apache.coyote.http11.Http11Processor.process
(Http11Processor.java:874)
at org.apache.coyote.http11.Http11BaseProtocol
$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:
665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket
(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt
(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run
(ThreadPool.java:689)
at java.lang.Thread.run(Unknown Source)
Caused by: mondrian.olap.MondrianException: Mondrian Error:Error while
parsing MDX statement ' SELECT NON EMPTY {[Measures].[BrojOsobaUZSJ]}
ON COLUMNS, {[Vrijeme.Datumi].[20080101]:[Vrijeme.Datumi].[20081231]}
ON ROWS FROM [Statistika] where ([MSegmenti.VrsteMS].[Sve],
[Zemlje.Grupe].[Sve].[ISTOÄŒNA AZIJA])'
at mondrian.resource.MondrianResource$_Def0.ex(MondrianResource.java:
785)
at mondrian.olap.Parser.parseInternal(Parser.java:772)
at mondrian.olap.ConnectionBase.parseQuery(ConnectionBase.java:129)
... 75 more
Caused by: mondrian.olap.MondrianException: Mondrian Error:MDX object
'[Zemlje.Grupe].[Sve].[ISTOÄŒNA AZIJA]' not found in cube 'Statistika'

___________________________________________________
The dimension [Zemlje] (Countries) and Hierarchy [Grupe] (Groups)
has member [ISTOČNA AZIJA] (EASTERN ASIA)
but timePlot component passes [ISTOÄŒNA AZIJA] to Mondrina

At the same time, some other component (JFreeChart) works correctly
with the same MDX Query
Although, JfreeChart components has some issues forcing '+' sign when
allMemberName is for example
'Sve vrste maketinškig segmenata' (All types of marketing segments).
In this case ChartClicked function
returns value 'Sve+vrste+maketinškig+segmenata'. I suppose this is
because of 'š' diacritic.
This is also issue for project posting.

All together, parsing or passing MDX Query to Mondrian, an in reverse,
results to components
is inconsistent. I made many test with those queries in Mondrian
Workbench and they are all
worked well.

Have anyone of you know the solution for this issue?

[Pedro Alves]

Hello Dusan

The first one seems to be a bug regarding encodings; Sometimes adding a
different system encoding to tomcat helps.


The second issue seems related. The problem in there is the + sign. Pass
it through the encode_prepare() function.


-pedro

+> at org.apache.catalina.core.ApplicationFilterChain.doFilter

[Dusan Djuric]

Hi Perdo

First of all, thanks for your answer.

Tomcat encoding:
--------------------------
Frankly, changing Tomcat encoding will almost certainly produce
problems on spots
which are now working correctly (reports or something else).

As JPivot, all object should communicate with Mondrian at same level
and principles.

Main problem is that components 'query: function()' implementation
works in different manner.
This is the direct consequence of using diverse core modules
(JFreeChars, Simile, JQuery....)
without deeper testing. Don't get me wrong , but CDF should work for
all implemented components
hawing in mind not all but at least major specifics like query
function is.

This looks like using fork for soup.

encode_prepare():
--------------------------
On the first look, encode_prepare() helps. But only if there is no '+'
sign in original value
(source data as DB or Mondrian, whatever it is). In that case, we have
more serious trouble,
specially if we base any logical/mathematical processing on signs like
- + * / (and so on).

For example, we have some constructions like ID->operation-
>next_ID ... (101->+->102->=->103)
witch tell as that we should add 101 to 102 to get calculated value of
103. Our user is granted to
change these constructions in DB and by that hi can change things 'on
flay'.

Ah, with encode_prepare() we can still have division,
subtraction ....hm! maybe!

Conclusion.
------------------
It seems to me that all java/js/jsp .... webapp concepts for that
matter, have rudiment oversight
of language and localization. I spend many hours to force
dateRangePicker and timePlot components
to work and cooperate with each other properly using one of the
standard date formats (dd.mm.yy).

Forgive me.
I'm an 50 years old IT project manager and programmer with 25 yeas
experience.
It must be that I'm ready for junkyard because I'm dealing here with
'bricks' instead of
building the 'BI house'.

Thank anyway

[Pedro Alves]

On 11/12/2009 02:54 PM, Dusan Djuric wrote:
>
> Tomcat encoding:

>
> This looks like using fork for soup.
>

Absolutely. It's a hack. On our roadmap we want to change all data
access mechanism to prevent stuff like this

> encode_prepare():
> --------------------------
> On the first look, encode_prepare() helps. But only if there is no '+'
> sign in original value

Known bug; related to the previous one in some point

> Conclusion.
> ------------------
> It seems to me that all java/js/jsp .... webapp concepts for that
> matter, have rudiment oversight
> of language and localization. I spend many hours to force
> dateRangePicker and timePlot components
> to work and cooperate with each other properly using one of the
> standard date formats (dd.mm.yy).
>

That is, indeed, on the list.

> Forgive me.
> I'm an 50 years old IT project manager and programmer with 25 yeas
> experience.
> It must be that I'm ready for junkyard because I'm dealing here with
> 'bricks' instead of
> building the 'BI house'.
>

I'm not sure what you mean by this; CDF is a complete solution? Not at
all; If we accept help? Absolutely. What exactly is your point?


-pedro

[Dusan Djuric]

My point is that bricks should be out of DW&BI project scope
(except in meaning of ingredient).
By bricks I mean, date format, encoding, decoding,
components usability and flexibility, way query function works,
...et cetera.

Try to imagine an architect. Hi deals with standard specifications,
materials, rules of physics and laws regulations.
At end, hi even bear legal liability, unlike us :)
Some of those thing still exist in our case too.
Specifications and standards for developing core modules
respecting languages, object specifications, methods, properties
and so.

Few riles
1. Talking to data providers (Mondrian) should be processed
throughout ONE-AND-ONLY-ONE process.

All CDF components must meet the conditions and satisfy rules
of that one process.

2. Source data (Mondrian) must be respected as NO-CHANGE,

WYGIWII (What You Get Is What It Is)

3. All code (program) parameters, variables and inputs passed
to that process, from programmers point, over component,
must stay unchangeable

If I type 'Premium+' it must stay that, not 'Premium'.
'JUŽNA EUROPA' (which is 'SOUTHERN EUROPE') must stay that,
not 'JUZNA EUROPA' or 'JU\u017dNA EUROPA'
Date of 01.05.2009 must not suddenly become 05.01.2009

All formatting and transformations must be outside that process
(except if that formatting is core of data provider language,
like SQL TO_CHARE or TO_DATE in Oracle for example)

In other words, between component calling moment and return point
there should be no changing of any parameter inputs or provider
outputs.
Leave the components free of that.

I can think of more principles and not even one of them is my
invention.

[Pedro Alves]

My opinion is exactly the same. If you're willing to help we'll all
appreciate.

> --
>
> You received this message because you are subscribed to the Google Groups "CDF Development List" group.
> To post to this group, send email to cdf-...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/cdf-devel?hl=.
>
>

[Dusan Djuric]

OK, my frustration is on the peek level now!
I found out some things that is beyond my skills to overcome.
Not all CDF component are processed via URL, but some do (passing to
ViewActin throe URL)
Fore instance, timePlot is triggered in peace of code
var timePlotEventSourceUrl = webAppPath + "/ViewAction?
solution=cdf&path=components&action=timelinefeeder.xaction&" +
parameters.join('&');
and somehow encoded afterwards, so final result is malformed MDX query
and MDXBaseComponet (or MDXLookupRule) ends up with error
because there is no query string decoding before hi gets it.

Why URL??? Every component should be processed internal like JfreChart
is fore instance.

[Dusan Djuric]

(I'm not sure that I pressed the right button, so I'm sending it
again)

Hi Pedro

First, sorry for my (may be) bitter tone in previous messages.
It was caused by mischievous 'bricks' and my lack of knowledge.
I'm not an Java but Oracles SQL, PL/SQL and Data Warehouse
project manager/ programmer.

To remind you, problem appears when you process any Pentaho xaction
over URL
which contains parameters/values with non ISO-8859-1 characters.

After few days of sweating in anger I finally find out solution.

Thanks for your tip on changing Tomcat system encoding,
I tried that but unfortunately not on the accurate place,
so this is the peace of change that should be done:

in ..\biserver-ce\tomcat\conf\server.xml
Original (preconfigured)
--------------------------------------------------------------------
<Connector port="8080" maxHttpHeaderSize="8192"
maxThreads="150" minSpareThreads="25"
maxSpareThreads="75"
enableLookups="false" redirectPort="8443"
acceptCount="100"
connectionTimeout="20000" disableUploadTimeout="true" /
>
--------------------------------------------------------------------
Changed
--------------------------------------------------------------------
<Connector port="8080" URIEncoding="UTF-8"
useBodyEncodingForURI="true" maxHttpHeaderSize="8192"
maxThreads="150" minSpareThreads="25"
maxSpareThreads="75"
enableLookups="false" redirectPort="8443"
acceptCount="100"
connectionTimeout="20000" disableUploadTimeout="true" /
>
--------------------------------------------------------------------
so, URIEncoding="UTF-8" useBodyEncodingForURI="true" instruct Tomcat
to
do encoding as I wont and have Croatian diacritics (šđč枊ĐČĆŽ)
correctly encoded/decoded.

I'm not sure where to post this tip so anybody can have benefit from
it.
I think that this on help not only Croatian developers but others too.

[Pedro Alves]

Thanks for telling us. There are, occasionally, some questions regarding
encoding and it's good to know.


This community "product" grew from projects. It's normal that you find
several things that - as a whole - don't make much sense. It's a
consequence of incremental development. Once in a while we get the
chance of revising some part of the code, and we do that in a proper
way. In the end of the day we guarantee it works for us - and we try our
best to make it work of others too.


-pedro