Empty reports/no data in reports

[Eucalyptus D]

Thanks everyone. I have a running install of Commcare. Pillowtop/Elasticsearch and other components appear to be running normally.

 I am able to complete and successfully submit a form to the server from my test phones. Case sharing works and another user can see cases and update/close and sync with the server.

However, reports show 0 or are empty. I have run manage.py run_ptop --all/core/core_ext and received this error:

OperationalError: SSL SYSCALL error: EOF detected

Kindly provide leads/solutions, thanks.

Euc

[Cory Zue]

can you send a larger stack trace?

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eucalyptus D]

Thanks Cory,

I have attached three files for your attention. EOF Error has the trace when I run  /home/commcarehq/commcare-hq/bin/python manage.py run_ptop --pillow-key=core command. 

I noticed that /tmp has some xml files (I have attached a screenshot of tmp directory listing in tmp folder.jpg file) which has the data submitted by the phone. I have attached one of the xml files. I am happy that the server receives the data, just worrying that the data does not show up in reports.

Kindly help resolve this. I have learnt and I am continuing to learn with your guidance. Thanks for the push.

regards,

Euc

[Cory Zue]

Hi Eucalyptus,

That error is actually coming from the error logging/retry framework (so there is the logging error and then whatever real error is causing the error).

I would try to see what the query it is trying to run is, and then paste that output here. 

You could also try reraising the exception here instead of logging it and see if that gives you more information about the root cause.

Cory

Euc

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Eucalyptus D]

Dear Cory, Dear all,

Thanks for the guidance. I modified logging for pillowtop in  setting.py to:

PILLOWTOP_LOG_FILE = "%s/%s" % (FILEPATH, "pillowtop.log")

   'handlers': {
        'pillowtop': {
            'level': 'INFO',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': PILLOWTOP_LOG_FILE,
            'maxBytes': 10 * 1024 * 1024,  # 10 MB
            'backupCount': 20, # Backup 200 MB of logs
            'formatter': 'pillowtop'

ran:
 /home/commcarehq/commcare-hq/bin/python manage.py run_ptop  --all

and had these errors:

OperationalError: SSL error: decryption failed or bad record mac

OperationalError: SSL SYSCALL error: EOF detected

Log files are attached. I saw some posts about concurrency and Copy-on-write issues with uwsgi and postgresql and used --lazy-apps option with uwsgi but no luck. Also read about using CELERYD_FORCE_EXECV = True in celery.

Not tried this yet. Please assist. thanks.

Euc

Euc

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eucalyptus Deglupta]

Seems to be something with multiprocessing and database connections - process -1 is successful whoever, subsequent processes fail. here is the log -

Process Process-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    checkpoint = get_or_create_checkpoint(self.checkpoint_id)
  File "corehq/ex-submodules/pillowtop/checkpoints/manager.py", line 17, in get_or_create_checkpoint
    'timestamp': datetime.utcnow()
  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "corehq/ex-submodules/pillowtop/pillow/interface.py", line 82, in run
    self.process_changes(since=self.get_last_checkpoint_sequence(), forever=True)
  File "corehq/ex-submodules/pillowtop/pillow/interface.py", line 66, in get_last_checkpoint_sequence
    return self.checkpoint.get_or_create_wrapped().wrapped_sequence
  File "corehq/ex-submodules/pillowtop/checkpoints/manager.py", line 37, in get_or_create_wrapped
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", line 405, in get_or_create
    return self.get(**lookup), False
  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", line 328, in get
    num = len(clone)
  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", line 144, in __len__
    self._fetch_all()
  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", line 965, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", line 238, in iterator
    results = compiler.execute_sql()
  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
    cursor.execute(sql, params)
  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
DatabaseError: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

Euc

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to a topic in the Google Groups "CommCare Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to commcare-developers+unsub...@googlegroups.com.

[Eucalyptus D]

The issue is somewhere within run_pillowtop.py having issues with multiprocessing and database connection. Seems a single database connection is being reused by spawned "pillows".

Someone help a brother .... self hosted Commcare is not recommended. I need a proof of concept box to show my colleagues. thanks.

Euc

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Eucalyptus Deglupta]

Update: psycopg 2.6.2 addresses inconsistent states in externally closed connections bug in 2.6.1.

I will try this tomorrow. 

>>>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to commcare-develo...@googlegroups.com.

>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>>
>>>>> --
>>>>>
>>>>> ---
>>>>> You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to commcare-develo...@googlegroups.com.

>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to a topic in the Google Groups "CommCare Developers" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe.

>>> To unsubscribe from this group and all its topics, send an email to commcare-develo...@googlegroups.com.

>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
> --
>
> ---
> You received this message because you are subscribed to a topic in the Google Groups "CommCare Developers" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to commcare-develo...@googlegroups.com.

[Cory Zue]

Hey,

It's pretty weird to me that your DB connection is working in other places but not in pillowtop. I would also have a look in your postgres logs if the psycopg thing doesn't work.

Cory

>>>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.

>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>>
>>>>> --
>>>>>
>>>>> ---
>>>>> You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.

>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to a topic in the Google Groups "CommCare Developers" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe.

>>> To unsubscribe from this group and all its topics, send an email to commcare-developers+unsub...@googlegroups.com.

>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
> --
>
> ---
> You received this message because you are subscribed to a topic in the Google Groups "CommCare Developers" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to commcare-developers+unsub...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.

[Eucalyptus D]

Thanks Cory for the support and encouragement. Seems to be something around multiprocessing and database:

ERROR 2016-11-11 07:51:53,787 logging 17579 140672615249664 Notify Exception: processor error in pillow CaseToElasticsearchPillow connection already closed

Traceback (most recent call last):

  File "corehq/ex-submodules/pillowtop/pillow/interface.py", line 94, in process_changes

    self.process_with_error_handling(change)

  File "corehq/ex-submodules/pillowtop/pillow/interface.py", line 111, in process_with_error_handling

    handle_pillow_error(self, change, ex)

  File "corehq/ex-submodules/pillowtop/pillow/interface.py", line 179, in handle_pillow_error

    error = PillowError.get_or_create(change, pillow)

  File "corehq/ex-submodules/pillow_retry/models.py", line 88, in get_or_create

    error = cls.objects.get(doc_id=doc_id, pillow=pillow.pillow_id)

  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method

    return getattr(self.get_queryset(), name)(*args, **kwargs)

  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", line 328, in get

    num = len(clone)

  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", line 144, in __len__

    self._fetch_all()

  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", line 965, in _fetch_all

    self._result_cache = list(self.iterator())

  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/query.py", line 238, in iterator

    results = compiler.execute_sql()

  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 838, in execute_sql

    cursor = self.connection.cursor()

  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 162, in cursor

    cursor = self.make_debug_cursor(self._cursor())

  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 137, in _cursor

    return self.create_cursor()

  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/utils.py", line 98, in __exit__

    six.reraise(dj_exc_type, dj_exc_value, traceback)

  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 137, in _cursor

    return self.create_cursor()

  File "/home/commcarehq/commcare-hq/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 212, in create_cursor

    cursor = self.connection.cursor()

InterfaceError: connection already closed

I will keep looking.

Euc

[Cory Zue]

"connection already closed" might also imply that something went wrong earlier in the process leaving the connection in a broken state. so another avenue to pursue would be looking in the logs for different traces. though maybe the first error you sent is the root problem.

>>>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>>
>>>>> --
>>>>>
>>>>> ---
>>>>> You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to a topic in the Google Groups "CommCare Developers" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe.

>>> To unsubscribe from this group and all its topics, send an email to commcare-developers+unsubscribe...@googlegroups.com.

>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
> --
>
> ---
> You received this message because you are subscribed to a topic in the Google Groups "CommCare Developers" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to commcare-developers+unsubscribe...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Eucalyptus Deglupta]

Dear Cory,  please which  first error are you reffering to?  Thnks again.

>>>> >>>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to commcare-develo...@googlegroups.com.

>>>> >>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> --
>>>> >>>>>
>>>> >>>>> ---
>>>> >>>>> You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

>>>> >>>>> To unsubscribe from this group and stop receiving emails from it, send an email to commcare-develo...@googlegroups.com.

>>>> >>>>> For more options, visit https://groups.google.com/d/optout.
>>>> >>>>
>>>> >>>>
>>>> >>> --
>>>> >>>
>>>> >>> ---
>>>> >>> You received this message because you are subscribed to a topic in the Google Groups "CommCare Developers" group.
>>>> >>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe.

>>>> >>> To unsubscribe from this group and all its topics, send an email to commcare-develo...@googlegroups.com.

>>>> >>> For more options, visit https://groups.google.com/d/optout.
>>>> >>
>>>> >>
>>>> > --
>>>> >
>>>> > ---
>>>> > You received this message because you are subscribed to a topic in the Google Groups "CommCare Developers" group.
>>>> > To unsubscribe from this topic, visit https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe.

>>>> > To unsubscribe from this group and all its topics, send an email to commcare-develo...@googlegroups.com.

>>>> > For more options, visit https://groups.google.com/d/optout.
>>>>
>>>> --
>>>>
>>>> ---
>>>> You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

>>>> To unsubscribe from this group and stop receiving emails from it, send an email to commcare-develo...@googlegroups.com.

>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

>> To unsubscribe from this group and stop receiving emails from it, send an email to commcare-develo...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the Google Groups "CommCare Developers" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to commcare-develo...@googlegroups.com.

[Cory Zue]

i just meant the other trace you sent on get_or_create_checkpoint

>>>> >>>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.

>>>> >>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> --
>>>> >>>>>
>>>> >>>>> ---
>>>> >>>>> You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

>>>> >>>>> To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.

>>>> >>>>> For more options, visit https://groups.google.com/d/optout.
>>>> >>>>
>>>> >>>>
>>>> >>> --
>>>> >>>
>>>> >>> ---
>>>> >>> You received this message because you are subscribed to a topic in the Google Groups "CommCare Developers" group.
>>>> >>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe.

>>>> >>> To unsubscribe from this group and all its topics, send an email to commcare-developers+unsub...@googlegroups.com.

>>>> >>> For more options, visit https://groups.google.com/d/optout.
>>>> >>
>>>> >>
>>>> > --
>>>> >
>>>> > ---
>>>> > You received this message because you are subscribed to a topic in the Google Groups "CommCare Developers" group.
>>>> > To unsubscribe from this topic, visit https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe.

>>>> > To unsubscribe from this group and all its topics, send an email to commcare-developers+unsub...@googlegroups.com.

>>>> > For more options, visit https://groups.google.com/d/optout.
>>>>
>>>> --
>>>>
>>>> ---
>>>> You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

>>>> To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.

>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

>> To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the Google Groups "CommCare Developers" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/commcare-developers/EWLcDCLU4VY/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to commcare-developers+unsub...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "CommCare Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-developers+unsub...@googlegroups.com.