mapValues error: (String, Array[String]) does not take parameters

1,077 views
Skip to first unread message

Sam

unread,
Jul 22, 2016, 5:01:12 PM7/22/16
to scala-user
Input dataset:
1,2008-10-23 16:05:05.0,\N,Donald,Becton,2275 Washburn Street,Oakland,CA,94660,5100032418,2014-03-18 13:29:47.0,2014-03-18 13:29:47.0
2,2008-11-12 03:00:01.0,\N,Donna,Jones,3885 Elliott Street,San Francisco,CA,94171,4150835799,2014-03-18 13:29:47.0,2014-03-18 13:29:47.0
3,2008-12-21 09:19:50.0,\N,Dorthy,Chalmers,4073 Whaley Lane,San Mateo,CA,94479,6506877757,2014-03-18 13:29:47.0,2014-03-18 13:29:47.0

Using keyBy I have to create an RDD with the postal code(9th field) as the key which I did like below

val keydbycode = rdd.keyBy(line => line.split(',')(8))

res3: Array[(String, String)] = Array((94660,1,2008-10-23 16:05:05.0,\N,Donald,Becton,2275 Washburn Street,Oakland,CA,94660,5100032418,2014-03-18 13:29:47.0,2014-03-18 13:29:47.0), (94171,2,2008-11-12 03:00:01.0,\N,Donna,Jones,3885 Elliott Street,San Francisco,CA,94171,4150835799,2014-03-18 13:29:47.0,2014-03-18 13:29:47.0), (94479,3,2008-12-21 09:19:50.0,\N,Dorthy,Chalmers,4073 Whaley Lane,San Mateo,CA,94479,6506877757,2014-03-18 13:29:47.0,2014-03-18 13:29:47.0))

Now using mapValues I have to create a pair RDD with postal code as the key and a list of names (Last Name,First Name) in that postal code as the value
I triedbelow which results in  error: (String, Array[String]) does not take parameters

keydbycode.mapValues(line => line.split(',')).map(fields => (fields(3), fields(4)))

scala> keydbycode.mapValues(line => line.split(',')).map(fields => (fields(3), fields(4)))
<console>:28: error: (String, Array[String]) does not take parameters
              keydbycode.mapValues(line => line.split(',')).map(fields => (fields(3), fields(4)))

                                                                                 ^
<console>:28: error: (String, Array[String]) does not take parameters
              keydbycode.mapValues(line => line.split(',')).map(fields => (fields(3), fields(4)))

                                                                                            ^

Could someone explain me how I can get this done using mapValues ?

Thanks in advance !!


Sam

unread,
Jul 22, 2016, 10:56:01 PM7/22/16
to scala-user
I got this, instead of having Array[(String, String)], I converted it to Array[string, Array[String]] and then extracted the fields. Code below

val keydbycode = accountsfile.keyBy(line => line.split(',') (8)).mapValues(line => line.split(","))
val namesByPCode
= keydbycode.mapValues(fields => (fields(3), fields(4)))

Thanks
Reply all
Reply to author
Forward
0 new messages