Adding a new application is slow

[juha.s...@gmail.com]

Hi,

I am using ready-made AWS image from http://actionml.com/docs/awssetupguide with r3.large instance in eu-central-1 region.

I followed installation instructions to the letter. Integration test passes but takes many minutes. Probably because creating a new a new app takes so long.

Then I tried creating a new application and that takes about 10 minutes. Is this normal? 

What this command is actually doing? I had assumed that it would just create few records to the databases, which should take just a dozen seconds or so.

Here is a log for creating a new application. Most of the time is spent lines 'HBLEvents' and '[App$] Initialized Event...'

$ time pio app new foobar123

[INFO] [HBLEvents] The table pio_event:events_3 doesn't exist yet. Creating now...

[INFO] [App$] Initialized Event Store for this app ID: 3.

[INFO] [App$] Created new app:

[INFO] [App$]       Name: foobar123

[INFO] [App$]         ID: 3

[INFO] [App$] Access Key: ....

real 9m58.891s

user 0m9.632s

sys 0m0.352s

-- 

Juha Syrjälä

[Pat Ferrel]

Yes, this is sometimes true and it could be a couple things. Some operations in HDFS are slow to complete since they require replication. Even starting a cluster can take minutes. Is it a problem? Is it possible HBase is getting full (check remaining data in HDFS). or needs to be scaled? The all-in-one instance on AWS only uses vertical scaling by number of cores, amount of memory, and disk space since all services run on one machine. You can “stop” the instance and change the type to a larger instance then “start” it to get it scaled. You have to snapshot the disk to scale disk by copying to a new one and attaching the new volume.

I hope it was clear that the AWS AMI was meant for dev, not production.

-- 
You received this message because you are subscribed to the Google Groups "actionml-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to actionml-use...@googlegroups.com.
To post to this group, send email to action...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/6a38952f-6f23-445f-bea8-e97fbc04aa4b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[juha.s...@gmail.com]

Hi,

Yes, it is clear that the AWS image is for dev only. I was just wondering the slowness of the a newly created instance when it has no data. 

However can't be that the HDFS is out of space. I ran those commands right after creating a new AWS instance. I don't have any engines running or configured, and I have not added any data yet to the system. The machine seems to be completely idle while adding the new application.

Current HDFS usage:

$ ./hadoop fs -df -h

Filesystem              Size   Used  Available  Use%

hdfs://127.0.0.1:9000  7.7 G  220 K      1.9 G    0%

Also system has plenty of free memory

$ free -h

              total        used        free      shared  buff/cache   available

Mem:            14G        2.0G         12G        8.6M        404M         12G

Swap:            0B          0B          0B

[juha.s...@gmail.com]

Ah, I just found the problem: a fresh AWS instance takes a long time to generate enough entropy so that a random access key can be generated. 

Related github issue: https://github.com/apache/incubator-predictionio/issues/253

More discussion about the a similar problem: https://groups.google.com/forum/#!searchin/actionml-user/aws%7Csort:relevance/actionml-user/N9o2jBkyF_w/bUYr5U8zFQAJ

[Pat Ferrel]

Hmm, interesting. I have heard of this being a problem in PredictionIO but haven’t heard this explanation before IIRC.

To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/76ce9fa7-c6b0-4695-8b2f-724dee38653a%40googlegroups.com.

[juha.s...@gmail.com]

I'll try make a patch for this to PredicitionIO during the weekend if I have the time.

[juha.s...@gmail.com]

The patch is here https://github.com/apache/incubator-predictionio/pull/367