Bash script that flushes every possible kazoo cache

[Joe]

Learning kazoo by brute force often times involves making changes and seeing what happens while hoping that the changes match your current hopes and dreams...

I've lately had this nagging worry that sometimes I may not be invalidating the cache for something (for instance a change to the jonny5 document) and that changes are not reflecting the way I am expecting.

There sure are quite a few commands for reloading different caches.  If there were one that triggered all of them, that would great!  I do not know of any such magical command though.

I've started writing a simple bash script that calls the following commands...

note: ERLANG_NODE is passed in as $1, these are actually called from a script that determines the node and then calls these two scripts with the proper nodes, that bit is a quite specific to our setup though.

Whapps:

#!/bin/bash

ERLANG_NODE=whistle_apps

ERLANG_HOST=$1

ERLANG_COOKIE=$(cat /etc/secrets/erlang/cookie)

echo "couch_mgr flush_cache_docs:"

sup -n $ERLANG_NODE -h $ERLANG_HOST -c "$ERLANG_COOKIE" couch_mgr flush_cache_docs

echo "whistle_couch_maintenance flush"

sup -n $ERLANG_NODE -h $ERLANG_HOST -c "$ERLANG_COOKIE" whistle_couch_maintenance flush

echo "jonny5_maintenance flush"

sup -n $ERLANG_NODE -h $ERLANG_HOST -c "$ERLANG_COOKIE" jonny5_maintenance flush

echo "stepswitch_maintenance reload_resources:"

sup -n $ERLANG_NODE -h $ERLANG_HOST -c "$ERLANG_COOKIE" stepswitch_maintenance reload_resources

echo "whapps_config flush"

sup -n $ERLANG_NODE -h $ERLANG_HOST -c "$ERLANG_COOKIE" whapps_config flush

echo "whapps_maintenance blocking_refresh"

sup -n $ERLANG_NODE -h $ERLANG_HOST -c "$ERLANG_COOKIE" whapps_maintenance blocking_refresh

echo "whapps_controller restart_app crossbar"

sup -n $ERLANG_NODE -h $ERLANG_HOST -c "$ERLANG_COOKIE" whapps_controller restart_app crossbar

echo "whistle_media_maintenance refresh"

sup -n $ERLANG_NODE -h $ERLANG_HOST -c "$ERLANG_COOKIE" whistle_media_maintenance refresh

Ecallmgr:

#!/bin/bash

ERLANG_NODE=ecallmgr

ERLANG_HOST=$1

ERLANG_COOKIE=$(cat /etc/secrets/erlang/cookie)

echo "ecallmgr_maintenance flush_acls"

sup -n $ERLANG_NODE -h $ERLANG_HOST -c "$ERLANG_COOKIE" ecallmgr_maintenance flush_acls

echo "ecallmgr_maintenance flush_authn"

sup -n $ERLANG_NODE -h $ERLANG_HOST -c "$ERLANG_COOKIE" ecallmgr_maintenance flush_authn

echo "ecallmgr_maintenance flush_util"

sup -n $ERLANG_NODE -h $ERLANG_HOST -c "$ERLANG_COOKIE" ecallmgr_maintenance flush_util

echo "ecallmgr_config flush"

sup -n $ERLANG_NODE -h $ERLANG_HOST -c "$ERLANG_COOKIE" ecallmgr_config flush

Please let me know if there is any cache i might be missing here.  This could be quite handy to others when learning kazoo.

Thanks

[James Aimonetti]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256


Is there a reason you're making the changes directly in the database and
not via APIs or SUP commands? Can you share what you find yourself
manually changing in the database so we can either direct you to an
appropriate API or SUP command, or add tickets to expose that
functionality?

All caches are linked to AMQP and should be listening for internal
events that will programmatically flush those caches. We could
potentially have a nuclear event that they all bind for to force a flush
on all connected caches.

But I'd be more interested in improving the surgical, precise flushing
needed.

- --
James Aimonetti

Lead Systems Architect
"If Dialyzer don't care, I don't care"
2600HzPDX | http://2600hz.com
sip:ja...@2600hz.com
tel:415.886.7905
irc:mc_ @ freenode
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXLSRzAAoJENTKa+JPXCVgzgAH/2iBgMLcKABsZdNCaRGsI4gc
5nHa3pRRsARpg3QbfNmZIEqd+j5fWaypmKwg6XPVgfB0w3DwrELEVOj4TkCHZgCU
ZJ5cuo2VgUOwWsmpKz4dXrUp8mstgXFgjsPX39bSi+YR4sEWdrTTz0oDEIWvwmCC
LYLGAyn2G5vjyVDwyQ5+gFKN8tigLtrUN3/BoHiuPGSL2RzagZwVNwpBbLxitYt3
h5ZZIz7xF/4qCRNL9gDSS005HZ25nv78TincHyVlR9EMFJgSRPfz0HEIg82JCS5P
OKEyX1oBbyxIdHeIYmctIWxxOebusTSqFspg5TH0iWN0OblpRLWCq05UvkG4t8M=
=TLF5
-----END PGP SIGNATURE-----

[Darren Schreiber]

Caching regularly defeats the point of the cache.

As your system grows, this will backfire on you. The cache is actually really important when you start getting into the thousands-of-phones realm. What you're doing here would cause one change to the DB, followed by the flushes, to inherently cause thousands of hits to the DB for no reason.

I agree with James. Let's fix the problem of WHY the caches aren't invalidated when you need them to be invalidated. If that's because you're going direct to the DB, then let's learn why that is as well and fix / improve it.

>--
>You received this message because you are subscribed to the Google Groups "2600hz-dev" group.
>To unsubscribe from this group and stop receiving emails from it, send an email to 2600hz-dev+...@googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.

[Darren Schreiber]

Each of these should be filed as a ticket. It should be done via the API.

From: Joe <j...@valuphone.com>
Date: Monday, May 9, 2016 at 3:17 PM
To: 2600hz-dev <2600h...@googlegroups.com>
Cc: Computer User <dschr...@2600hz.com>
Subject: Re: Bash script that flushes every possible kazoo cache

There are plenty of things that can't be done from the api alone.

One would be changing the jonny5 doc, what is the best command to run after doing this?

Another would be changing the regex in a `resource` document for a carrier.  i think for this one i run a stepswitch flush or something like that. is that correct?

Another would be adding a crossbar module to be autoloaded, or a whapp to be autoloaded in whapp_controller.  Is there a better way to do this?

Another would be sometimes i am told to change the value of a key in an account document, what is the best way to invalidate a single account's cache?  It seems to be 3 different commands for this and the difference is hard to ascertain.

For completeness I think changing service plans and adding new service plans is another thing that can't be done from the api (overriding can however).  But I have never had a problem with this kind of thing being cached so far, so it's not something I'm completely worried about.

>sip:...@2600hz.com

[Joe]

There are plenty of things that can't be done from the api alone.

One would be changing the jonny5 doc, what is the best command to run after doing this?

Another would be changing the regex in a `resource` document for a carrier.  i think for this one i run a stepswitch flush or something like that. is that correct?

Another would be adding a crossbar module to be autoloaded, or a whapp to be autoloaded in whapp_controller.  Is there a better way to do this?

Another would be sometimes i am told to change the value of a key in an account document, what is the best way to invalidate a single account's cache?  It seems to be 3 different commands for this and the difference is hard to ascertain.

For completeness I think changing service plans and adding new service plans is another thing that can't be done from the api (overriding can however).  But I have never had a problem with this kind of thing being cached so far, so it's not something I'm completely worried about.

On Friday, May 6, 2016 at 7:25:20 PM UTC-4, Darren Schreiber wrote:

>sip:...@2600hz.com

[Joe Black]

Should I create a ticket for each of these things @ tickets.2600hz.org?

Thanks,

 

Joe Black

From: Joe <j...@valuphone.com>
Reply: 2600h...@googlegroups.com <2600h...@googlegroups.com>>
Date: May 9, 2016 at 6:23:45 PM
To: 2600hz-dev <2600h...@googlegroups.com>>
Cc: dschr...@2600hz.com <dschr...@2600hz.com>>
Subject:  Re: Bash script that flushes every possible kazoo cache

You received this message because you are subscribed to a topic in the Google Groups "2600hz-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/2600hz-dev/tzOJN6znTXo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to 2600hz-dev+...@googlegroups.com.

[James Aimonetti]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Joe,

Thanks for taking the time to enumerate some examples. I'm going to
parse this later when I have more time and see if I can get you some
pointers. Some are do-able via the APIs/SUP and some are straightforward
enough to add. Would be great to save you some time here!

James

>> On 5/6/16, 4:10 PM, "2600h...@googlegroups.com <javascript:> on behalf of
>> James Aimonetti" <2600h...@googlegroups.com <javascript:> on behalf of

>> >sip:...@2600hz.com <javascript:>

>> >tel:415.886.7905
>> >irc:mc_ @ freenode
>> >-----BEGIN PGP SIGNATURE-----
>> >Version: GnuPG v2
>> >
>> >iQEcBAEBCAAGBQJXLSRzAAoJENTKa+JPXCVgzgAH/2iBgMLcKABsZdNCaRGsI4gc
>> >5nHa3pRRsARpg3QbfNmZIEqd+j5fWaypmKwg6XPVgfB0w3DwrELEVOj4TkCHZgCU
>> >ZJ5cuo2VgUOwWsmpKz4dXrUp8mstgXFgjsPX39bSi+YR4sEWdrTTz0oDEIWvwmCC
>> >LYLGAyn2G5vjyVDwyQ5+gFKN8tigLtrUN3/BoHiuPGSL2RzagZwVNwpBbLxitYt3
>> >h5ZZIz7xF/4qCRNL9gDSS005HZ25nv78TincHyVlR9EMFJgSRPfz0HEIg82JCS5P
>> >OKEyX1oBbyxIdHeIYmctIWxxOebusTSqFspg5TH0iWN0OblpRLWCq05UvkG4t8M=
>> >=TLF5
>> >-----END PGP SIGNATURE-----
>> >
>> >--
>> >You received this message because you are subscribed to the Google Groups
>> "2600hz-dev" group.
>> >To unsubscribe from this group and stop receiving emails from it, send an

>> email to 2600hz-dev+...@googlegroups.com <javascript:>.

>> >For more options, visit https://groups.google.com/d/optout.
>>

- --
James Aimonetti

Lead Systems Architect
"If Dialyzer don't care, I don't care"
2600HzPDX | http://2600hz.com

sip:ja...@2600hz.com

tel:415.886.7905
irc:mc_ @ freenode
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXMRpkAAoJENTKa+JPXCVgxT8H/RSJi9VoEM91Cdw9VKZ+abQg
U2d6mqzU0Exi9Bau4EanBh5XT1RMMaR3iy1C70CE8BE0mCpHmudbesEUi7LRi+cy
X5OQV12Fd84GtaBM2/S3yt3xq776/OzRFaB2sceJqd/90l5FcADZigrdcC/ggzxG
9q0RUs0KZxtJ5KtRJ+X8Sn/emBXyi88z96YSx8QmH5Ypu2Lf6u4FLDqM18ly22E5
Slun4KzjuT7dCNPsn5V6/Puabgv/e9pl9Uv8YrHWxJF9rSTCnAdb369tZ3aG8xAh
bzGgshVulfA6Rfb9Qu+U9EUyoqrC6Mabm40QNSwuXnUutMjm2og1K6GaiXSF/zY=
=SaOl
-----END PGP SIGNATURE-----

[James Aimonetti]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Joe writes:

> There are plenty of things that can't be done from the api alone.
>
> One would be changing the jonny5 doc, what is the best command to run after
> doing this?

Most system config settings can be set via SUP:

sup whapps_config set jonny5 this_key this_value

There isn't support for setting nested keys or values other than scalars
like numbers, boolean, and string. These would make good functions to
add to jonny5_maintenance so you can set the specific values. Something
like:

sup jonny5_maintenance set_some_nested_key new_value

which would call whapps_config:set(<<"jonny5">>, [<<"some">>,
<<"nested">>, <<"key">>], NewValue) in the module.

>
> Another would be changing the regex in a `resource` document for a carrier.
> i think for this one i run a stepswitch flush or something like that. is
> that correct?

You should be able to POST the new resource doc (and we should add PATCH
support). If a global carrier, as superduper admin:

curl -x POST /v2/resources/{RESOURCE_ID} \
-d '{"data":{resource_doc_with_new_rules}}'

Once PATCH support is added:

curl -x PATCH /v2/resources/{RESOURCE_ID} \
-d '{"data":{"rules":["new_regex"]}}'

This should flush stepswitch's cache and reload the resource
automatically.

>
> Another would be adding a crossbar module to be autoloaded, or a whapp to
> be autoloaded in whapp_controller. Is there a better way to do this?

sup crossbar_maintenance start_module cb_whatever
This should persist the cb_whatever module to be auto-started on the
next boot. If its not, please file a ticket.

For whapps_controller, as I said, it isn't often that people start/stop
apps in an ad-hoc manner like this. But, we could easily add a second
argument to flag whether to persist the change. Something like:

sup whapps_controller start_app crossbar true

would persist the starting of crossbar on the local node.

>
> Another would be sometimes i am told to change the value of a key in an
> account document, what is the best way to invalidate a single account's
> cache? It seems to be 3 different commands for this and the difference is
> hard to ascertain.

curl -x PATCH /v2/accounts/{ACCOUNT_ID} \
-d '{"data":{"key":"value"}}'

This will patch the account doc with "key":"value" and flush any
relevant entries in the caches.

>
> For completeness I think changing service plans and adding new service
> plans is another thing that can't be done from the api (overriding can
> however). But I have never had a problem with this kind of thing being
> cached so far, so it's not something I'm completely worried about.

It does appear changing system-level service plans isn't feasible via
APIs. This should be a feature request ticket.

>
> On Friday, May 6, 2016 at 7:25:20 PM UTC-4, Darren Schreiber wrote:
>>
>> Caching regularly defeats the point of the cache.
>>
>> As your system grows, this will backfire on you. The cache is actually
>> really important when you start getting into the thousands-of-phones realm.
>> What you're doing here would cause one change to the DB, followed by the
>> flushes, to inherently cause thousands of hits to the DB for no reason.
>>
>> I agree with James. Let's fix the problem of WHY the caches aren't
>> invalidated when you need them to be invalidated. If that's because you're
>> going direct to the DB, then let's learn why that is as well and fix /
>> improve it.
>>
>>
>>
>>

>> On 5/6/16, 4:10 PM, "2600h...@googlegroups.com <javascript:> on behalf of
>> James Aimonetti" <2600h...@googlegroups.com <javascript:> on behalf of

>> >sip:...@2600hz.com <javascript:>

>> >tel:415.886.7905
>> >irc:mc_ @ freenode
>> >-----BEGIN PGP SIGNATURE-----
>> >Version: GnuPG v2
>> >
>> >iQEcBAEBCAAGBQJXLSRzAAoJENTKa+JPXCVgzgAH/2iBgMLcKABsZdNCaRGsI4gc
>> >5nHa3pRRsARpg3QbfNmZIEqd+j5fWaypmKwg6XPVgfB0w3DwrELEVOj4TkCHZgCU
>> >ZJ5cuo2VgUOwWsmpKz4dXrUp8mstgXFgjsPX39bSi+YR4sEWdrTTz0oDEIWvwmCC
>> >LYLGAyn2G5vjyVDwyQ5+gFKN8tigLtrUN3/BoHiuPGSL2RzagZwVNwpBbLxitYt3
>> >h5ZZIz7xF/4qCRNL9gDSS005HZ25nv78TincHyVlR9EMFJgSRPfz0HEIg82JCS5P
>> >OKEyX1oBbyxIdHeIYmctIWxxOebusTSqFspg5TH0iWN0OblpRLWCq05UvkG4t8M=
>> >=TLF5
>> >-----END PGP SIGNATURE-----
>> >
>> >--
>> >You received this message because you are subscribed to the Google Groups
>> "2600hz-dev" group.
>> >To unsubscribe from this group and stop receiving emails from it, send an

>> email to 2600hz-dev+...@googlegroups.com <javascript:>.

>> >For more options, visit https://groups.google.com/d/optout.
>>

- --
James Aimonetti

Lead Systems Architect
"If Dialyzer don't care, I don't care"
2600HzPDX | http://2600hz.com

sip:ja...@2600hz.com

tel:415.886.7905
irc:mc_ @ freenode
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXMkjzAAoJENTKa+JPXCVgPU4IAJlRetSWRfkJuYcbPhkM45vU
m0p5gSUnZ9Y6No8ctnsxBpkSl1e1aLkAr70pnXhrc/8VI/A/R983bE1pVNrws9a9
0OehJFgCxvMPGh1PXyTh5J3SnZJmMrp9nHXc0B45gPlO4IpDQeTEQqWm4SonXRCH
/QPTWFixO4gyP/I/k/i65navDzgiKhAoSTYnSxByRhwEmqFIIoMNv1nazOOMr9Z9
Khry4EWKISgmQlhtKaBsGa/9OvOAHpmrUw53iwhcho3wWH1p1IF5w4P0lOuiTCag
59I9Ge1rSfwsG0Nge99SPOuqMJTOnpuzakQoEgIL6ck3823SxvWD1voYxKMxTd8=
=Z3Fd
-----END PGP SIGNATURE-----