Cannot write after 40672 items.

[Airam Rguez]

Hi there!

I'm experiencing a weird problem with Appscale. It's been working since more than 2 years without any problems until now. For some reason, it is not writing records on the database. I use Go and I catch and log every error, but no error is thrown (at least apparently). I have several collections and it can read from all of them but it can write to any of them. I have enough CPU(), RAM (8Gb) and disk space(110Gb).

I've tried restarting the machine and restarting Appscale without luck.

I've been looking through the logs and the only ERROR I've found is this (inside the controller.log):

I, [2017-08-08T08:45:53.342888 #2330] INFO -- : Starting go app inviapub on XXX.XX.XX.XX:20001

I, [2017-08-08T08:45:55.759471 #2330] INFO -- : Done adding AppServer for inviapub.

I, [2017-08-08T08:46:12.902232 #2330] INFO -- : Saw negative request rate for app inviapub, so resetting our haproxy stats for this app.

I, [2017-08-08T08:46:12.902546 #2330] INFO -- : No AppServer available to scale inviapub

I, [2017-08-08T08:46:12.926067 #2330] INFO -- : Need to spawn 1 new AppServers.

I, [2017-08-08T08:46:12.926241 #2330] INFO -- : Received a request to acquire nodes with roles memcache, taskqueue_slave, appengine, with instance type for new nodes

E, [2017-08-08T08:46:12.932772 #2330] ERROR -- : Still need 1 more nodes, but we aren't in a cloud environment, so we can't acquire more nodes - failing the caller's request.

E, [2017-08-08T08:46:12.933529 #2330] ERROR -- : Was not able to add 1 new nodes because: {"success":false,"message":"not enough open nodes"}

Is this error related to the database problem?

Thanks in advance.

[Chris Donati]

The error from the controller log is normal for cluster deployments. It means that the deployment would like to add more AppServer instances based on the load, but it can't because the existing machines are already full of running AppServer instances and the deployment can't start more machines because it's not a cloud environment.

Are you using the datastore API to store entities? If so, what do the datastore logs look like when you try to write an entity? When you say collections, do you mean entities that are in the same entity group (share the same parent key), or do you mean something else?

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.
To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

[Airam Rguez]

For collections I mean entities/tables. I'm using the datastore (appscale/datastore). Inside the datastore_server-XXXX.log All lines are like this one: 

       2017-08-08 09:28:45,083 INFO web.py:1908 200 POST / (XXX.XX.XX.XX) 29.91ms 

Everything seems to be ok. No errors in the datastore_server-XXXX.log.

When the row is inserted no error is thrown, but the registry is not inserted. A subsequent query does not return the previously inserted record. Should I look into other log file?

[Chris Donati]

Is it just a problem for queries, or are the inserted entities also invisible to gets? In other words, if you insert an entity and then try to retrieve it directly by its entity key, do you get a result?

[Airam Rguez]

I can query records. Queries are ok. The problem is only with insertions. If I insert a new entity I cannot query it.

[Chris Donati]

I understand. I'm trying to determine if the problem is related to writing the index or the entity data itself. A query will look up the inserted entity using the index, but fetching the inserted entity by key will not use the index.

When you insert an entity, you should get a key back. If you try to look up that entity by key (not with a query by property value), do you get a result?

On Tue, Aug 8, 2017 at 12:43 AM Airam Rguez <ai...@digitalvalue.es> wrote:

I can query records. Queries are ok. The problem is only with insertions. If I insert a new entity I cannot query it.

--

[Airam Rguez]

I'm not using the key returned by the `put` method. I'm doing a query after the insertion using filters and I don't get back any result.

[Airam Rguez]

Chris, I'm using Appscale 2.7.0

[Chris Donati]

I was asking you to try using the key in order to figure out what the issue is. There are a few possible causes for the issue you are seeing:

1. The entity data is not being inserted at all.
   
2. The entity data is being inserted, but none of the indexes are being written.
3. The entity data is being inserted and the single-property indexes are being written, but the composite indexes are not being written.

#3 is the most likely, but I was trying to rule out the other options first. I need to sign off for the night, but I will follow up with you tomorrow. In the meantime, one thing that might be worth looking into is checking if your index.yaml file has changed recently.

On Tue, Aug 8, 2017 at 1:15 AM, Airam Rguez <ai...@digitalvalue.es> wrote:

Chris, I'm using Appscale 2.7.0

--

You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.
To post to this group, send email to appscale_community@googlegroups.com.

[Airam Rguez]

index.yaml and all source code has been intact for 2 years. Will nodetool rebuild_index help?

El martes, 8 de agosto de 2017, 10:22:09 (UTC+2), Chris Donati escribió:

I was asking you to try using the key in order to figure out what the issue is. There are a few possible causes for the issue you are seeing:

1. The entity data is not being inserted at all.
   
2. The entity data is being inserted, but none of the indexes are being written.
3. The entity data is being inserted and the single-property indexes are being written, but the composite indexes are not being written.

#3 is the most likely, but I was trying to rule out the other options first. I need to sign off for the night, but I will follow up with you tomorrow. In the meantime, one thing that might be worth looking into is checking if your index.yaml file has changed recently.

On Tue, Aug 8, 2017 at 1:15 AM, Airam Rguez <ai...@digitalvalue.es> wrote:

Chris, I'm using Appscale 2.7.0

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.

[Airam Rguez]

I've made this test:

1. Under App console, check the amount of entities of a certain type. (39 items)

2. Insert a new entity of that type. (The operation returns success).

3. Check again the amount of entities of that type under App console: still 39 items.

I suppose the information under App console is reliable so we are on the first case.

1. The entity data is not being inserted at all.

Am I wrong? What causes no entities being inserted at all? 

[Chris Donati]

By the App console, are you referring to the datastore viewer? Getting an entity count for a certain type from the datastore viewer uses a kind index, so it is not the same thing as fetching an entity by its key.

`nodetool rebuild_index` will not help because AppScale does not use Cassandra indexes.

If you want to email me privately, I can take a look at your deployment to try to figure out what the issue is. Otherwise, I would recommend adding more logging to the datastore server for both the write path (dynamic_put) and the read path (kind_query, if you use the test you described).

--

You received this message because you are subscribed to the Google Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

To post to this group, send email to appscale_community@googlegroups.com.