Kartik Mathur
unread,Jun 5, 2015, 5:57:36 PM6/5/15Sign in to reply to author
Sign in to forward
You do not have permission to delete messages in this group
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to spark-conn...@lists.datastax.com
Hello geeks ,
I am stuck with this problem , which is quite bizarre
when i try to use data.saveToCassandra like below , I get followng exception -
"Some primary key columns are missing in RDD or not selected"
data is RDD of rows like this :
RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
I verified multiple times that number of primary columns and data type matches
between RDD and cassandra table , unable to understand the problem !!!
data.saveToCassandra("report", "t_heat_map",
SomeColumns("advId".toLowerCase(),
//"impDayAdvZ".toLowerCase(),
"campId".toLowerCase(),
"agId".toLowerCase(),
"faId".toLowerCase(),
"pubId".toLowerCase(),
"brnm".toLowerCase(),
"brver".toLowerCase(),
"elid".toLowerCase(),
"elnm".toLowerCase(),
"eltyp".toLowerCase(),
"evttyp".toLowerCase(),
"evttypno".toLowerCase(),
"iswl".toLowerCase(),
"osnm".toLowerCase(),
"osver".toLowerCase(),
"pgid".toLowerCase(),
"pgord".toLowerCase(),
"ppid".toLowerCase(),
"x".toLowerCase(),
"y".toLowerCase()))
This is the table structure in cassandra -
CREATE TABLE t_heat_map (
advid text,
campid text,
agid text,
faid text,
pubid text,
brnm text,
brver text,
elid text,
elnm text,
eltyp text,
evttyp text,
evttypno bigint,
iswl boolean,
osnm text,
osver text,
pgid text,
pgord text,
ppid text,
x int,
y int,
PRIMARY KEY ((advid), campid, agid, faid, pubid)
)
Full RDD printed -
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,false,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,false,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,false,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,false,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,false,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,false,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,false,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,false,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,false,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,false,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,false,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,false,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,false,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,false,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,false,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,false,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,false,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)
15/06/05 14:51:52 INFO sparkjobs.HeatMapProcessor: RDD row:(1,2,3,4,5,6,7,8,9,10,11,1,true,12,13,14,15,16,10,10)