Combining 2 DList's using ++ Failing With ClassCastException

[Raghu]

Hi,

I am trying to combine 2 DLists of type A(This is an Avro type) in my code and it is failing with the below error. Interesting thing is that when I write these 2 DLists to disk, read them and then combine them using ++ it works fine. Can anyone tell me what the issue is.

Exception in thread "main" java.lang.ClassCastException: com.nicta.scoobi.impl.plan.AST$GbkReducer cannot be cast to com.nicta.scoobi.impl.exec.MapperLike

This is how my code looks like. list1 and list2 are both DList's of type A

val list1 = ....

val list2 = ...

val combinedList = list1++list2

Thanks,

Raghu

[Eric Torreborre]

Hi Raghu,

Can you please tell me which Scoobi version you are using and upgrade to 0.7.0-SNAPSHOT?

The exception you are showing indicates that you are probably using 0.6.x. and this code has been completely refactored in 0.7.0 enabling a few other new features in passing.

Thanks,

Eric.

[Raghu]

Hi Eric,

Thanks for responding. I tried to add the 0.7.0 snapshot dependency to my project. When I build my project I am getting the below error. Can you please tell me what's wrong? My repository and dependency settings in pom.xml are below.

[ERROR] Failed to execute goal on project ep-reports-model: Could not resolve dependencies for project com.ebay.experimentation.reporting:ep-reports-model:jar:0.0.1-SNAPSHOT: Could not transfer artifact com.nicta:scoobi_2.9.2:jar:0.7.0-cdh3-SNAPSHOT from/to cloudera (https://repository.cloudera.com/content/repositories/releases/): Failed to transfer https://repository.cloudera.com/content/repositories/releases/com/nicta/scoobi_2.9.2/0.7.0-cdh3-SNAPSHOT/scoobi_2.9.2-0.7.0-cdh3-SNAPSHOT.jar. Error code 409, The repository 'libs-release-local' rejected the artifact 'libs-release-local:com/nicta/scoobi_2.9.2/0.7.0-cdh3-SNAPSHOT/scoobi_2.9.2-0.7.0-cdh3-SNAPSHOT.jar' due to its snapshot/release handling policy. -> [Help 1]

<repository>
              <id>nicta's avro</id>
              <url>https://oss.sonatype.org/content/repositories/snapshots</url>
              <snapshots>
                <enabled>true</enabled>
            </snapshots>
            <releases>
                <enabled>false</enabled>
            </releases>
        </repository>

<dependency>
                  <groupId>com.nicta</groupId>
                  <artifactId>scoobi_2.9.2</artifactId>
                  <version>0.7.0-cdh3-SNAPSHOT</version>
            </dependency>

Thanks,
Raghu

[Eric Springer]

Scoobi master (aka 0.7-snapshot) isn't built against 2.9.2 afaik .. you'll need to use 2.10 >: D

--
You received this message because you are subscribed to the Google Groups "scoobi-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scoobi-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Bill Cauchois]

Hello,

Is there any workaround for those of us still using Scoobi 0.6.x?

I think I'm running into the same problem. In particular, I'm trying to prepend a constant DList (created with DList.apply) to a DList that is generated as a result of a computation. Maybe this isn't legal?

The exception I'm getting is exactly the same as in the OP.

Would appreciate any light into this issue or possible workaround!

Thanks,

Bill

[shil...@gmail.com]

This worked for me:

Just follow up the concatenation with groupBy and flatMap as shown in the code in blue. This workaround needs to be done during concatenation ONLY if atleast one of the dlists to the ++ method was the result of some operation(such as a join etc.).

 val dlistConcat = (joinresultDlist ++ dlist2).groupBy{ x => (x.getGuid(), x.getSessionKey().toLong) }.flatMap{ case (k, v) => v }