recurring error: "Unexpected HTTP Status code: 400 Bad Request"

Tyler Moore

unread,

Oct 18, 2018, 4:23:30 PM10/18/18

to H2O Open Source Scalable Machine Learning - h2ostream

1) 'h2o' cluster version = 3.20.0.8; Java version = 1.8.0_181
2) OS = Windows 10.
3) Working in R version 3.5.1

Hi all, I've been getting the following error pretty frequently - seems to always happen if I leave h2o.automl() running long enough:

ERROR: Unexpected HTTP Status code: 400 Bad Request (url = http://localhost:54321/3/Parse)

java.lang.IllegalArgumentException

[1] "java.lang.IllegalArgumentException: Total input file size of 37.6 KB is much larger than total cluster memory of Zero , please use either a larger cluster or smaller data."

[2] " water.parser.ParseDataset.forkParseDataset(ParseDataset.java:160)"

[3] " water.parser.ParseDataset.parse(ParseDataset.java:43)"

[4] " water.api.ParseHandler.parse(ParseHandler.java:36)"

[5] " sun.reflect.GeneratedMethodAccessor99.invoke(Unknown Source)"

[6] " sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)"

[7] " java.lang.reflect.Method.invoke(Unknown Source)"

[8] " water.api.Handler.handle(Handler.java:63)"

[9] " water.api.RequestServer.serve(RequestServer.java:451)"

[10] " water.api.RequestServer.doGeneric(RequestServer.java:296)"

[11] " water.api.RequestServer.doPost(RequestServer.java:222)"

[12] " javax.servlet.http.HttpServlet.service(HttpServlet.java:755)"

[13] " javax.servlet.http.HttpServlet.service(HttpServlet.java:848)"

[14] " org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)"

[15] " org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)"

[16] " org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)"

[17] " org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)"

[18] " org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)"

[19] " org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)"

[20] " org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)"

[21] " org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)"

[22] " water.JettyHTTPD$LoginHandler.handle(JettyHTTPD.java:197)"

[23] " org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)"

[24] " org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)"

[25] " org.eclipse.jetty.server.Server.handle(Server.java:370)"

[26] " org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)"

[27] " org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)"

[28] " org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)"

[29] " org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)"

[30] " org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)"

[31] " org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)"

[32] " org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)"

[33] " org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)"

[34] " org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)"

[35] " org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)"

[36] " java.lang.Thread.run(Unknown Source)"

When I restart R, I can run the script again, but the above error inevitably occurs at some point (sometimes minutes later, sometime hours later). It mentions memory, but my machine has plenty of memory to run this. The call just looks like this:

aclin <- h2o.automl(x = xclin, y = y, training_frame = train, max_runtime_secs = 10000, project_name="clinical")

But I get the error on other h2o.automl() calls too. Any idea what's happening?

Thanks!

Tyler

Phillip Burger

unread,

Oct 18, 2018, 6:12:09 PM10/18/18

to H2O Open Source Scalable Machine Learning - h2ostream

I was receiving this message as well. In my case, I looked at the processes that were running and found that there was a Java process running that was using 25GM of RAM on my 64GB RAM CentOS server. I killed the process. Re-ran my job, and didn't receive the error.

I have no idea if my experience will translate to the error you are encountering.

Darren Cook

unread,

Oct 19, 2018, 3:27:13 AM10/19/18

to h2os...@googlegroups.com

> 1) 'h2o' cluster version = 3.20.0.8; Java version = 1.8.0_181
> 2) OS = Windows 10.
> 3) Working in R version 3.5.1
>
> Hi all, I've been getting the following error pretty frequently - seems to
> always happen if I leave h2o.automl() running long enough:

> ...

> [1] "java.lang.IllegalArgumentException: Total input file size of 37.6 KB is
> much larger than total cluster memory of Zero , please use either a larger
> cluster or smaller data."

> ...

> It mentions memory, but my machine has plenty of memory to run this.

How much of the machine's memory are you giving to H2O? That is the
important thing. E.g. if you have 16GB of memory, and are not using
anything else on the machine, you could give 15GB to h2o, by doing:

h2o.init(max_mem_size="15G")

The default value will mean less memory is being given to the H2O process.

Darren

Tyler Moore

unread,

Oct 19, 2018, 4:23:01 PM10/19/18

to dar...@dcook.org, h2os...@googlegroups.com

Hi Darren and Phillip, thanks very much for your input. I tried Darren's suggestion (increase max memory size), and I think that worked. At least, h2o.automl() has now been running for longer than I've been able to run it before - will re-post here if the error creeps back up. Thanks again!

Tyler

--
You received this message because you are subscribed to the Google Groups "H2O Open Source Scalable Machine Learning - h2ostream" group.
To unsubscribe from this group and stop receiving emails from it, send an email to h2ostream+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Wiki Wang

unread,

Aug 13, 2022, 8:39:32 PM8/13/22

to H2O Open Source Scalable Machine Learning - h2ostream

maybe you should redefine h2o objects. the h2o objects you defined last time could not be used again when h2o was shutdown and init.

Hakeem Ojulari

unread,

Dec 15, 2022, 5:02:00 AM12/15/22

to H2O Open Source Scalable Machine Learning - h2ostream

I am experiencing the same thing. I restarted the RStudio and reloaded H2o. The same error persists. How would I resolve this.

The error is shown below:

ERROR: Unexpected HTTP Status code: 400 Bad Request (url = http://localhost:54321/3/ModelBuilders/deeplearning)

java.lang.IllegalArgumentException
[1] "java.lang.IllegalArgumentException: Object with key='RTMP_sid_a5ff_13' doesn't exist in DKV."
[2] " hex.ObjectConsistencyChecker.<init>(ObjectConsistencyChecker.java:14)"
[3] " hex.ModelBuilder.init(ModelBuilder.java:1407)"
[4] " hex.deeplearning.DeepLearning.init(DeepLearning.java:71)"
[5] " water.api.ModelBuilderHandler.handle(ModelBuilderHandler.java:47)"
[6] " water.api.ModelBuilderHandler.handle(ModelBuilderHandler.java:16)"
[7] " water.api.RequestServer.serve(RequestServer.java:472)"
[8] " water.api.RequestServer.doGeneric(RequestServer.java:303)"
[9] " water.api.RequestServer.doPost(RequestServer.java:227)"
[10] " javax.servlet.http.HttpServlet.service(HttpServlet.java:707)"
[11] " javax.servlet.http.HttpServlet.service(HttpServlet.java:790)"
[12] " org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)"
[13] " org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535)"
[14] " org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)"
[15] " org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)"
[16] " org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)"
[17] " org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)"
[18] " org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)"
[19] " org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)"
[20] " org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)"
[21] " org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)"
[22] " org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)"
[23] " water.webserver.jetty9.Jetty9ServerAdapter$LoginHandler.handle(Jetty9ServerAdapter.java:130)"
[24] " org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)"
[25] " org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)"
[26] " org.eclipse.jetty.server.Server.handle(Server.java:531)"
[27] " org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)"
[28] " org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)"
[29] " org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)"
[30] " org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)"
[31] " org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)"
[32] " org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)"
[33] " org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)"
[34] " org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)"
[35] " org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)"
[36] " org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)"
[37] " org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)"
[38] " org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)"
[39] " java.base/java.lang.Thread.run(Thread.java:1589)"

Error in .h2o.doSafeREST(h2oRestApiVersion = h2oRestApiVersion, urlSuffix = page, :

ERROR MESSAGE:

Object with key='RTMP_sid_a5ff_13' doesn't exist in DKV.

Reply all

Reply to author

Forward