Commcare Production goes well but errors of cloudcare

[Danish Fayaz]

Hello,


I am doing a research demo for my university and I actually need some of your assistance. 

I am currently trying to deploy the commcare production app from github on a server. I've followed the production instructions carefully and able to run the app. But when I make an application and try to edit it I get the following errors (screenshot attached). As if formplayer isn't running, also I see an ajax request to my domain gives an error of "Disallowed".

Can you please guide me if there's a proper documentation available or where I need to look for. It means a lot to me. Thank you for your time.

[William Pride]

Can you confirm that you have Formplayer running per the instructions here?

If so can you provide the exception you're seeing in your browser (and in Formplayer/HQ if such exists)

Thanks,

Will

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Danish Fayaz]

Thank you so much for your reply, yes william I've cloned Formplayer in the commcarehq root directory and updated submodules as instructed.

Further I think the supervisor runs the formplayer so I changed the deployment config of supervisor in the services/supervisor/changeme_supervisor_formplayer_spring.conf 

Changed the name of dir from formplayer_build to formplayer (as per cloned). 

Still when I go to the application page I get the same error and when I click the start Icon in the App (in screenshot) I get the same error as in the screenshot. 

I checked the console, following errors occur: 

OPTIONS http://dev.commcare:8080/delete_application_dbs net::ERR_CONNECTION_REFUSED

Disallowed origin http://dev.commcare

Also when I click on Users (screenshot) I get the following in the console:

GET http://dev.commcare/a/test.org/cloudcare/api/login_as/users/?query=&limit=10&page=1 500 (Internal Server Error)

I'd appreciate it if someone can guide me on this. Thank you so much for the forum and the community's help.

[Danish Fayaz]

Just now, I've also tried running formplayer seperately, it builds and runs all fine. But still I am getting the disallowed origin error. Options (pre flight requests failing).

[William Pride]

Hi Danish,

Looks like a CORS issue - Web Apps and App Builder make cross domain requests in order to communicate directly with Formplayer. Thus we configure Ajax and Spring to accept cross domain requests. I'm not sure whether this configuration would be sufficient in all browsers/servers (for example if these intentionally disallowed cross-domain requests). Does Formplayer log the requests?

Cheers,

Will

[Danish Fayaz]

Following is the formplayer-spring.log. It seems to be starting fine I think.. still getting the disallowed. I've added access origin headers everywhere just to check if it works (nginx, django server settings) .

  .   ____          _            __ _ _

 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \

( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \

 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )

  '  |____| .__|_| |_|_| |_\__, | / / / /

 =========|_|==============|___/=/_/_/_/

 :: Spring Boot ::        (v1.4.3.RELEASE)

OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0

  .   ____          _            __ _ _

 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \

( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \

 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )

  '  |____| .__|_| |_|_| |_\__, | / / / /

 =========|_|==============|___/=/_/_/_/

 :: Spring Boot ::        (v1.4.3.RELEASE)

2017-08-17 21:29:40.307  WARN 7427 --- [           main] figurationPropertiesBindingPostProcessor : Multiple PropertySourcesPlaceholderConfigurer beans registered [swaggerProperties, propertiesResolver], falling back to Environment

2017-08-17 21:29:51.395  INFO 7427 --- [ost-startStop-1] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 36700 ms

OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0

  .   ____          _            __ _ _

 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \

( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \

 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )

  '  |____| .__|_| |_|_| |_\__, | / / / /

 =========|_|==============|___/=/_/_/_/

 :: Spring Boot ::        (v1.4.3.RELEASE)

2017-08-17 21:30:31.664  WARN 8457 --- [           main] figurationPropertiesBindingPostProcessor : Multiple PropertySourcesPlaceholderConfigurer beans registered [swaggerProperties, propertiesResolver], falling back to Environment

OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0

  .   ____          _            __ _ _

 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \

( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \

 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )

  '  |____| .__|_| |_|_| |_\__, | / / / /

 =========|_|==============|___/=/_/_/_/

 :: Spring Boot ::        (v1.4.3.RELEASE)

2017-08-17 21:31:23.456  WARN 8768 --- [           main] figurationPropertiesBindingPostProcessor : Multiple PropertySourcesPlaceholderConfigurer beans registered [swaggerProperties, propertiesResolver], falling back to Environment

2017-08-17 21:31:29.574  INFO 8768 --- [ost-startStop-1] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 29139 ms

OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0

  .   ____          _            __ _ _

 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \

( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \

 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )

  '  |____| .__|_| |_|_| |_\__, | / / / /

 =========|_|==============|___/=/_/_/_/

 :: Spring Boot ::        (v1.4.3.RELEASE)

2017-08-17 21:31:56.286  WARN 9152 --- [           main] figurationPropertiesBindingPostProcessor : Multiple PropertySourcesPlaceholderConfigurer beans registered [swaggerProperties, propertiesResolver], falling back to Environment

2017-08-17 21:32:05.293  INFO 9152 --- [ost-startStop-1] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 19518 ms

OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0

  .   ____          _            __ _ _

 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \

( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \

 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )

  '  |____| .__|_| |_|_| |_\__, | / / / /

 =========|_|==============|___/=/_/_/_/

 :: Spring Boot ::        (v1.4.3.RELEASE)

2017-08-17 21:32:28.030  WARN 9503 --- [           main] figurationPropertiesBindingPostProcessor : Multiple PropertySourcesPlaceholderConfigurer beans registered [swaggerProperties, propertiesResolver], falling back to Environment

[Danish Fayaz]

I was able to fix the formplayer-spring error. It was due to the run command (heap size was to be set). Now its running fine. But error is there, I checked forms-player (touchform server) it was giving similar error, so I set heap size and updated the jython.jar path in the command to load the right classpath. But I am getting following error when I run: 

Note: I am using the recommended jython 2.5.3

Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/collect/MapMaker

at org.python.core.PyType.fromClass(PyType.java:1280)

at org.python.core.PyObject.<init>(PyObject.java:66)

at org.python.core.PyBuiltinCallable.<init>(PyBuiltinCallable.java:21)

at org.python.core.PyBuiltinMethod.<init>(PyBuiltinMethod.java:16)

at org.python.core.PyBuiltinMethodNarrow.<init>(PyBuiltinMethodNarrow.java:24)

at org.python.core.PyType$type___subclasses___exposer.<init>(Unknown Source)

at org.python.core.PyType$PyExposer.<init>(Unknown Source)

at org.python.core.PyType.<clinit>(PyType.java)

at org.python.core.PyObject.<init>(PyObject.java:66)

at org.python.core.PyBuiltinCallable.<init>(PyBuiltinCallable.java:21)

at org.python.core.PyBuiltinMethod.<init>(PyBuiltinMethod.java:16)

at org.python.core.PyBuiltinMethodNarrow.<init>(PyBuiltinMethodNarrow.java:24)

at org.python.core.PyDataDescr$getset_descriptor___get___exposer.<init>(Unknown Source)

at org.python.core.PyDataDescr$PyExposer.<init>(Unknown Source)

at org.python.core.PyDataDescr.<clinit>(PyDataDescr.java)

at org.python.core.PyBuiltinCallable$PyExposer.<init>(Unknown Source)

at org.python.core.PyBuiltinCallable.<clinit>(PyBuiltinCallable.java)

at org.python.core.PyObject$PyExposer.<init>(Unknown Source)

at org.python.core.PyObject.<clinit>(PyObject.java)

at org.python.util.jython.run(jython.java:150)

at org.python.util.jython.main(jython.java:129)

Caused by: java.lang.ClassNotFoundException: com.google.common.collect.MapMaker

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

... 21 more

On Friday, 18 August 2017 00:36:42 UTC+4, William Pride wrote:

[William Pride]

Hi Danish,.

If you are running Formplayer then you shouldn't need to run Jython. The Formplayer code replaced the now-deprecated Jython code.

Cheers,

Will

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Danish Fayaz]

Hey william,

You've been great help. Can I ask if you know why the web app (web form filling) menu link isn't appearing on top. Also the data sync via the mobile app in a survey isn't also appearing on the web app. Also my data exports are not working at all giving a "Server maintenance in progress. Please try again later. If the problem persists, please Report an Issue." warning after a few mins.

Appreciate the assist. Thanks

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[William Pride]

Hi Danish,

Could you send pictures of what you mean about the "menu link" and "data sync" not appearing? Apologies but I'm not sure what's being referred to here.

The data exports issue sounds like a problem with Elastic Search - is ES running and error free?

Cheers,

Will

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Danish Fayaz]

William,

I really appreciate you taking time out and getting back at me. Attached are screenshots of commcare official and my production setup. I've marked the missing features.

The data sync is basically from mobile. Like I downloaded the commcare android app from playstore, used the barcodes when publishing my application to download them on the android app. Then after I filled a survey / case I sync it to the web to push the data. But I don't see it in the web platform. Although I see a message on mobile that the data was synced perfectly.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[William Pride]

Hi Danish,

In order for those features to appear you'll need your web user to have the correct subscription level and permissions.

As to your data - does this data appear in your databases (Couch or PostGres)? If yes and it doesn't appear in your reports, this could be an Elastic issue. If not then the issue could be with your host configuration - you'll want to make sure that your CommCare application's PostURL points correctly to your instance and that you're able to receive POST requests at this URL.

Cheers,

Will

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Danish Fayaz]

Hello william,

I've looked at postgres and couch, I do have a lot of tables in postgres and some in couchdb but I am not sure where to look for the data I submited from mobile. Also Following is the shard health of my elastic:

{"cluster_name":"elasticsearch","status":"yellow","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":52,"active_shards":52,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":22,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"indices":{"xforms_2016-07-07":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"case_search_2016-03-15":{"status":"green","number_of_shards":5,"number_of_replicas":0,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"hqgroups_2017-05-29":{"status":"green","number_of_shards":5,"number_of_replicas":0,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"report_cases_czei39du507m9mmpqk3y01x72a3ux4p0":{"status":"green","number_of_shards":5,"number_of_replicas":0,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"smslogs_2017-02-09":{"status":"green","number_of_shards":5,"number_of_replicas":0,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"hqapps_2017-05-22_1426":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"hqdomains_2016-08-08":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"ledgers_2016-03-15":{"status":"green","number_of_shards":5,"number_of_replicas":0,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"report_xforms_20160824_1708":{"status":"green","number_of_shards":5,"number_of_replicas":0,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"shards":{"0":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"1":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"2":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"3":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"4":{"status":"green","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}},"hqcases_2016-03-04":{"status":"yellow","number_of_shards":5,"number_of_replicas":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"2":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"3":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"4":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"hqusers_2017-05-30":{"status":"yellow","number_of_shards":2,"number_of_replicas":1,"active_primary_shards":2,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2,"shards":{"0":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1},"1":{"status":"yellow","primary_active":true,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}}}}

It seems that most of them are yellow. Can you guide me if that's the way it should be? Also I've ran connection tests from the commcare mobile app to check if the connection is fine, also the forms are being submitted to the server but I am unable to see the data on the web. 

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Simon Kelly]

Hi Danish

I'd suggest you use ES HEAD or some other ES management tool to see what shards are causing the issue. Based on the quick look at the shard status I think you have a number of replica shards which are unassigned. If you're running ES with only one node then you should just remove the replica shards.

If you're unable to see data on the web it may mean that you're not running 'pillowtop' using the 'run_ptop' command: https://github.com/dimagi/commcare-hq/blob/master/README.md#running-commcare-hq

Simon Kelly

Director of Server Engineer | Dimagi

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Danish Fayaz]

Hello Simon,

Sorry for replying late, I was held up by a few other things. I've gone through the elastic shards and also deleted the unassigned ones. But the actually problem is with pillow top (You're right). I ran the following ptop commands as per Production Instructions:

/home/commcarehq/bin/python manage.py ptop_preindex
/home/commcarehq/bin/python manage.py ptop_es_manage --flip_all_aliases




These are running all fine. But the ptop_all command fails with the following errors which I can't make anything of :( can you take a look at them and advise what's causing it? I've even searched the forum but found nothing relevant.

Process Process-33:

Traceback (most recent call last):

  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap

    self.run()

  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run

    self._target(*self._args, **self._kwargs)

  File "./corehq/ex-submodules/pillowtop/pillow/interface.py", line 96, in run

    self.process_changes(since=self.get_last_checkpoint_sequence(), forever=True)

  File "./corehq/ex-submodules/pillowtop/pillow/interface.py", line 80, in get_last_checkpoint_sequence

    return self.checkpoint.get_or_create_wrapped().wrapped_sequence

  File "./corehq/ex-submodules/pillowtop/checkpoints/manager.py", line 180, in get_or_create_wrapped

    checkpoints = self._get_checkpoints()

  File "./corehq/ex-submodules/pillowtop/checkpoints/manager.py", line 177, in _get_checkpoints

    return KafkaCheckpoint.get_or_create_for_checkpoint_id(self.checkpoint_id, self.topics)

  File "./corehq/ex-submodules/pillowtop/models.py", line 91, in get_or_create_for_checkpoint_id

    .values_list('topic', 'partition')

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/query.py", line 256, in __iter__

    self._fetch_all()

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/query.py", line 1087, in _fetch_all

    self._result_cache = list(self.iterator())

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/query.py", line 125, in __iter__

    for row in compiler.results_iter():

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 789, in results_iter

    results = self.execute_sql(MULTI)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql

    cursor.execute(sql, params)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 79, in execute

    return super(CursorDebugWrapper, self).execute(sql, params)

  File "/home/commcarehq/local/lib/python2.7/site-packages/raven/contrib/django/client.py", line 114, in execute

    return real_execute(self, sql, params)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute

    return self.cursor.execute(sql, params)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__

    six.reraise(dj_exc_type, dj_exc_value, traceback)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute

    return self.cursor.execute(sql, params)

OperationalError: SSL SYSCALL error: EOF detected

Process Process-36:

Traceback (most recent call last):

  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap

    self.run()

  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run

    self._target(*self._args, **self._kwargs)

  File "./corehq/ex-submodules/pillowtop/pillow/interface.py", line 96, in run

    self.process_changes(since=self.get_last_checkpoint_sequence(), forever=True)

  File "./corehq/ex-submodules/pillowtop/pillow/interface.py", line 80, in get_last_checkpoint_sequence

    return self.checkpoint.get_or_create_wrapped().wrapped_sequence

  File "./corehq/ex-submodules/pillowtop/checkpoints/manager.py", line 180, in get_or_create_wrapped

    checkpoints = self._get_checkpoints()

  File "./corehq/ex-submodules/pillowtop/checkpoints/manager.py", line 177, in _get_checkpoints

    return KafkaCheckpoint.get_or_create_for_checkpoint_id(self.checkpoint_id, self.topics)

  File "./corehq/ex-submodules/pillowtop/models.py", line 91, in get_or_create_for_checkpoint_id

    .values_list('topic', 'partition')

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/query.py", line 256, in __iter__

    self._fetch_all()

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/query.py", line 1087, in _fetch_all

    self._result_cache = list(self.iterator())

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/query.py", line 125, in __iter__

    for row in compiler.results_iter():

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 789, in results_iter

    results = self.execute_sql(MULTI)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql

    cursor.execute(sql, params)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 79, in execute

    return super(CursorDebugWrapper, self).execute(sql, params)

  File "/home/commcarehq/local/lib/python2.7/site-packages/raven/contrib/django/client.py", line 114, in execute

    return real_execute(self, sql, params)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute

    return self.cursor.execute(sql, params)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__

2017-09-16 17:24:06,225 INFO interface Starting pillow <class 'pillowtop.pillow.interface.ConstructedPillow'>

    six.reraise(dj_exc_type, dj_exc_value, traceback)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute

    return self.cursor.execute(sql, params)

OperationalError: SSL SYSCALL error: EOF detected

Process Process-32:

Traceback (most recent call last):

  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap

    self.run()

  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run

    self._target(*self._args, **self._kwargs)

  File "./corehq/ex-submodules/pillowtop/pillow/interface.py", line 96, in run

    self.process_changes(since=self.get_last_checkpoint_sequence(), forever=True)

  File "./corehq/ex-submodules/pillowtop/pillow/interface.py", line 80, in get_last_checkpoint_sequence

    return self.checkpoint.get_or_create_wrapped().wrapped_sequence

  File "./corehq/ex-submodules/pillowtop/checkpoints/manager.py", line 180, in get_or_create_wrapped

    checkpoints = self._get_checkpoints()

  File "./corehq/ex-submodules/pillowtop/checkpoints/manager.py", line 177, in _get_checkpoints

    return KafkaCheckpoint.get_or_create_for_checkpoint_id(self.checkpoint_id, self.topics)

  File "./corehq/ex-submodules/pillowtop/models.py", line 91, in get_or_create_for_checkpoint_id

    .values_list('topic', 'partition')

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/query.py", line 256, in __iter__

    self._fetch_all()

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/query.py", line 1087, in _fetch_all

    self._result_cache = list(self.iterator())

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/query.py", line 125, in __iter__

    for row in compiler.results_iter():

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 789, in results_iter

    results = self.execute_sql(MULTI)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql

    cursor.execute(sql, params)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 79, in execute

    return super(CursorDebugWrapper, self).execute(sql, params)

  File "/home/commcarehq/local/lib/python2.7/site-packages/raven/contrib/django/client.py", line 114, in execute

    return real_execute(self, sql, params)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute

    return self.cursor.execute(sql, params)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__

    six.reraise(dj_exc_type, dj_exc_value, traceback)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute

    return self.cursor.execute(sql, params)

OperationalError: SSL SYSCALL error: EOF detected

Process Process-34:

Traceback (most recent call last):

  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap

    self.run()

  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run

    self._target(*self._args, **self._kwargs)

  File "./corehq/ex-submodules/pillowtop/pillow/interface.py", line 96, in run

    self.process_changes(since=self.get_last_checkpoint_sequence(), forever=True)

  File "./corehq/ex-submodules/pillowtop/pillow/interface.py", line 80, in get_last_checkpoint_sequence

    return self.checkpoint.get_or_create_wrapped().wrapped_sequence

  File "./corehq/ex-submodules/pillowtop/checkpoints/manager.py", line 180, in get_or_create_wrapped

    checkpoints = self._get_checkpoints()

  File "./corehq/ex-submodules/pillowtop/checkpoints/manager.py", line 177, in _get_checkpoints

    return KafkaCheckpoint.get_or_create_for_checkpoint_id(self.checkpoint_id, self.topics)

  File "./corehq/ex-submodules/pillowtop/models.py", line 91, in get_or_create_for_checkpoint_id

    .values_list('topic', 'partition')

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/query.py", line 256, in __iter__

    self._fetch_all()

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/query.py", line 1087, in _fetch_all

    self._result_cache = list(self.iterator())

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/query.py", line 125, in __iter__

    for row in compiler.results_iter():

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 789, in results_iter

    results = self.execute_sql(MULTI)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql

    cursor.execute(sql, params)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 79, in execute

    return super(CursorDebugWrapper, self).execute(sql, params)

  File "/home/commcarehq/local/lib/python2.7/site-packages/raven/contrib/django/client.py", line 114, in execute

    return real_execute(self, sql, params)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute

    return self.cursor.execute(sql, params)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__

    six.reraise(dj_exc_type, dj_exc_value, traceback)

  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute

    return self.cursor.execute(sql, params)

OperationalError: SSL SYSCALL error: EOF detected

[Simon Kelly]

Hi

I've never seen that error before but a quick google search suggests it may be related to an underpowered database machine: https://stackoverflow.com/questions/24130305/postgres-ssl-syscall-error-eof-detected-with-python-and-psycopg

Simon Kelly

Director of Server Engineer | Dimagi

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Martín Ferrari]

I am having the same issue. I can't run ptop_all, it always fails with hundreds of lines of tracebacks. Although the first one might offer more information:

Process Process-2:

Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)

  File "./corehq/ex-submodules/pillowtop/pillow/interface.py", line 95, in run
    self.process_changes(since=self.get_last_checkpoint_sequence(), forever=True)
  File "./corehq/ex-submodules/pillowtop/pillow/interface.py", line 79, in get_last_checkpoint_sequence
    return self.checkpoint.get_or_create_wrapped().wrapped_sequence
  File "./corehq/ex-submodules/pillowtop/checkpoints/manager.py", line 175, in get_or_create_wrapped
    checkpoints = self._get_checkpoints()
  File "./corehq/ex-submodules/pillowtop/checkpoints/manager.py", line 172, in _get_checkpoints

    return KafkaCheckpoint.get_or_create_for_checkpoint_id(self.checkpoint_id, self.topics)
  File "./corehq/ex-submodules/pillowtop/models.py", line 91, in get_or_create_for_checkpoint_id
    .values_list('topic', 'partition')

  File "/home/commcare/local/lib/python2.7/site-packages/django/db/models/query.py", line 256, in __iter__
    self._fetch_all()
  File "/home/commcare/local/lib/python2.7/site-packages/django/db/models/query.py", line 1087, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/home/commcare/local/lib/python2.7/site-packages/django/db/models/query.py", line 125, in __iter__
    for row in compiler.results_iter():
  File "/home/commcare/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 789, in results_iter
    results = self.execute_sql(MULTI)
  File "/home/commcare/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql
    cursor.execute(sql, params)
  File "/home/commcare/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/commcare/local/lib/python2.7/site-packages/raven/contrib/django/client.py", line 114, in execute
    return real_execute(self, sql, params)
  File "/home/commcare/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/commcare/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/commcare/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
OperationalError: SSL error: decryption failed or bad record mac

[Martín Ferrari]

Some threads I've found googling around, seem to imply this could be related to sharing DB connections across forks, but I can't find a way to change that either.

[Simon Kelly]

Are you using an SSL connection to your PostgreSQL database?

Simon Kelly

Director of Server Engineer | Dimagi

[Simon Kelly]

OK great.

Simon Kelly

Director of Server Engineer | Dimagi

On 27 October 2017 at 01:18, Martín Ferrari <martin....@gmail.com> wrote:

On 26/10/17 08:55, Simon Kelly wrote:
> Are you using an SSL connection to your PostgreSQL database?

Actually, you gave me the key to fix this!

For some reason, the ptop_all command was trying to connect with SSL,
even if the other apps were not doing that. But I added this to the
database config and now it can connect:

        'OPTIONS': {
            'sslmode': 'disable',
        },

Thanks! :)

--
Martín Ferrari (Tincho)

[Martín Ferrari]

On 26/10/17 08:55, Simon Kelly wrote:

> Are you using an SSL connection to your PostgreSQL database?