List serialization/deserialization can overflow the stack in 2.10

211 views
Skip to first unread message

Chris Marshall

unread,
Jan 11, 2013, 4:36:32 AM1/11/13
to scala-i...@googlegroups.com
This is a big problem:

object ListSer extends App {

  val largeList = (1 to 1000000).toList

  val oos = new ObjectOutputStream(new FileOutputStream("C:/tmp/largeList.ser"))
  oos.writeObject(largeList)
  oos.flush()
  oos.close()

  val ois = new ObjectInputStream(new FileInputStream("C:/tmp/largeList.ser"))
  val read = ois.readObject().asInstanceOf[List[Int]]
  println(read.size)

}


Running the above program will overflow the stack in 2.10; it works fine in 2.9 - I'm sure you guys are working on emergency fixes - this looks like something which has to be included. I've reported it here: https://issues.scala-lang.org/browse/SI-6961

Chris

Eugene Burmako

unread,
Jan 11, 2013, 4:39:36 AM1/11/13
to <scala-internals@googlegroups.com>
Could you please also provide a stacktrace?

Chris Marshall

unread,
Jan 11, 2013, 4:40:18 AM1/11/13
to scala-i...@googlegroups.com
The only workaround I can find is increasing the stack size - if anyone has any other suggestions, please can you let me know. The issue is that the old serialization code in List has been changed (because of https://issues.scala-lang.org/browse/SI-5374), from this:

  private def writeObject(out: ObjectOutputStream) {
    var xs: List[B] = this
    while (!xs.isEmpty) { out.writeObject(xs.head); xs = xs.tail }
    out.writeObject(ListSerializeEnd)
  }

  private def readObject(in: ObjectInputStream) {
    hd = in.readObject.asInstanceOf[B]
    assert(hd != ListSerializeEnd)
    var current: ::[B] = this
    while (true) in.readObject match {
      case ListSerializeEnd =>
        current.tl = Nil
        return
      case a : Any =>
        val list : ::[B] = new ::(a.asInstanceOf[B], Nil)
        current.tl = list
        current = list
    }
  }

to this:

  private def writeObject(out: ObjectOutputStream) {
    out.writeObject(ListSerializeStart) // needed to differentiate with the legacy `::` serialization
    out.writeObject(this.hd)
    out.writeObject(this.tl)
  }

  private def readObject(in: ObjectInputStream) {
    val obj = in.readObject()
    if (obj == ListSerializeStart) {
      this.hd = in.readObject().asInstanceOf[B]
      this.tl = in.readObject().asInstanceOf[List[B]]
    } else oldReadObject(in, obj)
  }

  /* The oldReadObject method exists here for compatibility reasons.
   * :: objects used to be serialized by serializing all the elements to
   * the output stream directly, but this was broken (see SI-5374).
   */
  private def oldReadObject(in: ObjectInputStream, firstObject: AnyRef) {
    hd = firstObject.asInstanceOf[B]
    assert(hd != ListSerializeEnd)
    var current: ::[B] = this
    while (true) in.readObject match {
      case ListSerializeEnd =>
        current.tl = Nil
        return
      case a : Any =>
        val list : ::[B] = new ::(a.asInstanceOf[B], Nil)
        current.tl = list
        current = list
    }
  }

  private def oldWriteObject(out: ObjectOutputStream) {
    var xs: List[B] = this
    while (!xs.isEmpty) { out.writeObject(xs.head); xs = xs.tail }
    out.writeObject(ListSerializeEnd)
  }

Chris


From: oxbow...@hotmail.com
To: scala-i...@googlegroups.com
Subject: [scala-internals] List serialization/deserialization can overflow the stack in 2.10
Date: Fri, 11 Jan 2013 09:36:32 +0000

Chris Marshall

unread,
Jan 11, 2013, 4:42:59 AM1/11/13
to scala-i...@googlegroups.com
Sorry - thought it was obvious:

        at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
        at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
        at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
        at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
        at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
        at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
        at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
        at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)


Date: Fri, 11 Jan 2013 10:39:36 +0100
Subject: Re: [scala-internals] List serialization/deserialization can overflow the stack in 2.10
From: eugene....@epfl.ch
To: scala-i...@googlegroups.com

Jan Vanek

unread,
Jan 11, 2013, 5:51:14 AM1/11/13
to scala-i...@googlegroups.com
Hi Chris, I think the list could be serialized from behind.

A->B->C->D
when serializing A the order of writeObject calls should be D, C, B, A.

I didn't yet finish it, and have to leave now for 2 hours.
Regards,
Jan

Jan Vanek

unread,
Jan 11, 2013, 12:16:52 PM1/11/13
to scala-i...@googlegroups.com
Here is a gist, please have a look. It requires 2 Booleans in thread-local storage. Supports long lists including sub-structure sharing. The oldReadObject() is not incorporated.

https://gist.github.com/4512368

Regards,
Jan
Reply all
Reply to author
Forward
0 new messages