Kazoo Internals and Architecture - short questions

[A E G]

Hey guys,

This is sort-of a follow-on from the thread Zhisun started "internals of ecallmgr"

With all credit to James for taking the time and effort in explaining all of that, it's a bit hard to follow for people like me who aren't quite that advanced in our understanding of the whole architecture. Will have to bookmark and go back to it over and over again for it to all sink-in. 

But while it's fresh in my mind, I wanted to take this opportunity to clarify a few things to re-surface a few questions I'd asked in my previous emails that got left unanswered. After having read that thread, some of them got answered but then it raised more questions...

So, re: the architecture, to make it easy and quick for you guys, I'm just going to ask direct questions or make statements and maybe you can simply respond to them with a yes/no/kinda (and a brief explanation, if possible) would be greatly appreciated.

1. The communication between ecallmgr and freeswitch nodes happen through one of the ecallmgr process(es) e.g. ecallmgr_call_events -> freeswitch.erl wrapper -> mod_erlang_event -> make FS sing and dance?
2. Should the callmgr_call_events in #1 be ecallmgr_call_control as ecallmgr_call_events is a listener of incoming calls?
3. In one of the previous responses to something I'd asked, Darren had said that you (i.e.  provide a "mod_event_socket"-esque hook to directly send messages to FS via the AMQP bus.  If #1 and #2 are true, then did what he was referring to was ecallmgr and making FS sing and dance meant programming/interacting directly with ecallmgr module/process(es)?
4. Is the Kazoo API layer then basically an abstraction of the ecallmgr and its various process(es) and modules but offers a REST interface to the user to build GUI/Webapps, so that for common cases, one only has to write web-apps to build their product and not mess around with ecallmgr at all?
5. Does all communication within the Kazoo platform happen over the AMQP bus? By "all" I mean, ecallmgr <-> FS (as per #1 above), ecallmgr <-> Whapps layer, ecallmgr <-> BigCouch layer? 
6. What about communication between WhApps? Do they communicate with each other over AMQP? or do they not talk to each other directly but do so through ecallmgr, which is the orchestrator of the whole Kazoo show?
7. Is it possible to talk to the Whapps layer directly? or is that done through the Kazoo API or the ecallmgr?
8. There was another question I had which I forgot, but I thought of another one instead...when the call flow is built using the Kazoo UI with dragging and dropping the Whapps, is the configuration pushed down as XML into the FS nodes to create dialplan entries or is this controlled dynamically by the ecallmgr_call_control by following this "recipe" on a per call basis?

Wish I could remember what that last question was, but if you guys can answer the ones above, it will be of GREAT help. 

Sorry about so many questions but I couldn't find this information in the Wiki but if it's there somewhere then please feel free to tell me to go RTFM with a pointer to where this is. Also, I've been trying to find a schematic / architecture blue-print of the Kazoo platform but don't think it's in the Wiki, is that correct?

[James Aimonetti]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Speedy answer for the moment, but I created a system overview wiki doc
(not sure where the last one went):
https://2600hz.atlassian.net/wiki/display/docs/System+Overview

That should hopefully show how the components are related to each
other. Know that each piece can be on one or more actual servers
(meaning you can run ecallmgr and whapps on the same box, or put every
service on its own separate box, or put everything on one box as our
Chef Solo install does).

I'll try to create some flow charts about how a call progresses from
INVITE -> AMQP APIs -> DB Lookups -> AMQP APIs -> FS commands -> 200
OK soon.

Thanks for the questions; if I don't get anything up this weekend,
could you bump this mail at the start of next week?

James

On 09/26/2012 06:37 PM, A E G wrote:
> Hey guys,
>
> This is sort-of a follow-on from the thread Zhisun started
> "internals of ecallmgr"
>
> With all credit to James for taking the time and effort in
> explaining all of that, it's a bit hard to follow for people like
> me who aren't quite that advanced in our understanding of the whole
> architecture. Will have to bookmark and go back to it over and over
> again for it to all sink-in.
>
> But while it's fresh in my mind, I wanted to take this opportunity
> to clarify a few things to re-surface a few questions I'd asked in
> my previous emails that got left unanswered. After having read that
> thread, some of them got answered but then it raised more
> questions...
>
> So, re: the architecture, to make it easy and quick for you guys,
> I'm just going to ask direct questions or make statements and maybe
> you can simply respond to them with a yes/no/kinda (and a brief
> explanation, if possible) would be greatly appreciated.
>

> 1. The communication between ecallmgr and freeswitch nodes happen

> through one of the ecallmgr process(es) e.g. ecallmgr_call_events
> -> freeswitch.erl wrapper -> mod_erlang_event -> make FS sing and

> dance? 2. Should the callmgr_call_events in #1 be

> ecallmgr_call_control as ecallmgr_call_events is a listener of

> incoming calls? 3. In one of the previous responses to something

> I'd asked, Darren had said that you (i.e. provide a
> "mod_event_socket"-esque hook to directly send messages to FS via
> the AMQP bus. If #1 and #2 are true, then did what he was
> referring to was ecallmgr and making FS sing and dance meant
> programming/interacting directly with ecallmgr module/process(es)?

> 4. Is the Kazoo API layer then basically an abstraction of the

> ecallmgr and its various process(es) and modules but offers a REST
> interface to the user to build GUI/Webapps, so that for common
> cases, one only has to write web-apps to build their product and

> not mess around with ecallmgr at all? 5. Does all communication

> within the Kazoo platform happen over the AMQP bus? By "all" I
> mean, ecallmgr <-> FS (as per #1 above), ecallmgr <-> Whapps layer,

> ecallmgr <-> BigCouch layer? 6. What about communication between

> WhApps? Do they communicate with each other over AMQP? or do they
> not talk to each other directly but do so through ecallmgr, which

> is the orchestrator of the whole Kazoo show? 7. Is it possible to

> talk to the Whapps layer directly? or is that done through the

> Kazoo API or the ecallmgr? 8. There was another question I had

> which I forgot, but I thought of another one instead...when the
> call flow is built using the Kazoo UI with dragging and dropping
> the Whapps, is the configuration pushed down as XML into the FS
> nodes to create dialplan entries or is this controlled dynamically
> by the ecallmgr_call_control by following this "recipe" on a per
> call basis?
>
> Wish I could remember what that last question was, but if you guys
> can answer the ones above, it will be of GREAT help.
>
> Sorry about so many questions but I couldn't find this information
> in the Wiki but if it's there somewhere then please feel free to
> tell me to go RTFM with a pointer to where this is. Also, I've been
> trying to find a schematic / architecture blue-print of the Kazoo
> platform but don't think it's in the Wiki, is that correct?
>

- --
James Aimonetti
Distributed Systems Engineer / DJ MC_

2600hz | http://2600hz.com
sip:ja...@2600hz.com
tel: 415.886.7905
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQZIUDAAoJENc77s1OYoGgAHEIAKObPDORcRczsxAv157knMUk
bK4MgNgoahkapt/KDKL8VzoXYpwVVz6bBtCq5EsO65XAEWgCH3e6fDB9hu0gLtPv
KDv26oTBBkkHls46mlbYkU0LHW71/2PXr21Wbgj+JxlDQV+sMkggxxPY0YxtFjtu
6vU2KR4IXODWVhIH0IGM6o5EUqB2p/Nh6azsm9oWObSF3sB2bjcmQ6HjoNLU1AKa
yukY4Z20IhbjPFMOy1sJ9Jn0UI5A30kg2gNfj3KRodYmBHn30JSGEAiUjkRVcNen
M96e40flXsH5MJn8O4p+P5hZ7cwNiKwB49SOnqLFlA0tmLS43/Q8vfIqgFzOKBI=
=Kbvz
-----END PGP SIGNATURE-----

[A E G]

Hey James,

thanks so much. That system overview block diagram is great and helped a lot. And while it seems like I was close in my understanding of a bunch of things, that diagram and the info on the Wiki has now raised a couple of more questions. Should I send them through as well so you can maybe cover them all in one fell swoop?

Would most certainly appreciate your pointed responses to the questions below, if you manage to get around to them this weekend or over the next few days.

Thanks an advance
\a

[James Aimonetti]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Please continue to ask questions on the list. It's great for us to
start to open the internals up to others and get some feedback.

Why not hold off, though, on asking the next batch until I have a
moment to answer the current batch below? I'll try to allot some time
tomorrow to answer them.

Seriously though, really glad others are interested in how the system
does its thing; this is driving us to get more documented and,
hopefully, get some pull requests coming our way! :)

James

iQEcBAEBAgAGBQJQaSL5AAoJENc77s1OYoGgb/sH/A3Dxe0ZyhBRDob+wNVQzx3b
2OkR6DnpXr/Q4PyqzPiasAmqijfYC6oTtssEVzlOILgpAEhGMCQ7Z0uUhf0mrX1W
IetnOpIinfhtXTFWpWGBfMFAVcJkVw7vBeGKxfmWuY5xiO2ESy/WSK6wlejom6DL
NCbC9ca33QWIYBI0+aztCH44+h3h9W3C3HAo7yT/x4cA92Xh7VlPFG9Zl8aDq1i5
IdyKKhColu7pIkBHnb0AbcK0/HSWk+y+3F7Yqp4Ymg6yGicu3vWgi4B1xLtLhrOd
xIW+WlN1pQSPl0uyzJBkEd6m9Z2L8DfJCSMqLcvYEOMI8aUnh16KV6mL155wj7o=
=/SAu
-----END PGP SIGNATURE-----

[James Aimonetti]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

See inline, but also, I'm going to start documenting on the wiki:

Main page will start here:
https://2600hz.atlassian.net/wiki/display/docs/Internals+Development

ecallmgr:
https://2600hz.atlassian.net/wiki/display/docs/Overview+of+ecallmgr
https://2600hz.atlassian.net/wiki/display/docs/ecallmgr+and+FreeSWITCH

Will need to put the answers below in the wiki at some point, but hope
this helps for now. If you don't have access to edit yet, please email
Darren if you'd like to help create/edit the pages related to this.

Thanks,

James

For the most part. You have to remember that FreeSWITCH appears to
ecallmgr as an Erlang node, so ecallmgr communicates with FreeSWITCH
as such. freeswitch.erl is a convenience wrapper over that Erlang
message passing.

Some processes register themselves (their PID) with the FreeSWITCH
side for certain call events, handling calls, or xml requests (think
mod_xml_curl, but done via native Erlang).

2. Should the callmgr_call_events in #1 be
>>>> ecallmgr_call_control as ecallmgr_call_events is a listener
>>>> of incoming calls?

The incoming calls listener is actually ecallmgr_fs_route. When a call
comes in, ecallmgr_fs_route takes the dialplan fetch request from
FreeSWITCH and publishes a route_req to AMQP. It receives route_resp
messages and converts them to FreeSWITCH dialplan XML (generally
speaking, we just send back the park dialplan action). If FreeSWITCH
replies to our dialplan response with an 'ok', it means our dialplan
"won". Now we know which ecallmgr node is controlling the call, so
that ecallmgr will spin up two new processes, ecallmgr_call_control
and ecallmgr_call_events.

ecallmgr_call_control creates an AMQP queue for receiving dialplan
commands from the AMQP bus (see wapi_dialplan.erl), and also binds its
queue to call events related to its callid. It maintains a queue of
commands to send to FreeSWITCH.

ecallmgr_call_events asks FreeSWITCH to send it events related to the
callid. Things like CHANNEL_EXECUTE, CHANNEL_EXECUTE_COMPLETE,
CHANNEL_PROGRESS, CHANNEL_ANSWER, DTMF, CHANNEL_DESTROY, etc.
FreeSWITCH sends the ecallmgr_call_events process these event
proplists. We filter the fields and do some renaming, and publish the
resulting JSON to our AMQP bus (see wapi_call:publish_event/2).

Obviously I'm glossing over some of the minutia here, but hopefully
this helps some?

3. In one of the previous responses to something
>>>> I'd asked, Darren had said that you (i.e. provide a
>>>> "mod_event_socket"-esque hook to directly send messages to FS
>>>> via the AMQP bus. If #1 and #2 are true, then did what he
>>>> was referring to was ecallmgr and making FS sing and dance
>>>> meant programming/interacting directly with ecallmgr
>>>> module/process(es)?

This is likely ecallmgr_call_events. We publish call events with the
routing key call.event.{UUID} to the callevt exchange.

ecallmgr_fs_node publishes some events as well, one is a successful
registration (we bind to the CUSTOM_EVENT sofia::register for that
one). There are also some custom events that we publish/listen for, to
move calls between FreeSWITCH boxes and some other reasons.

>>>> 4. Is the Kazoo API layer then basically an abstraction of
>>>> the ecallmgr and its various process(es) and modules but
>>>> offers a REST interface to the user to build GUI/Webapps, so
>>>> that for common cases, one only has to write web-apps to
>>>> build their product and not mess around with ecallmgr at
>>>> all?

ecallmgr is an abstraction over FreeSWITCH/VoIP, and we have created a
set of JSON APIs, delivered via AMQP, for interacting with ecallmgr
(and the underlying switches as a result).

We have also created a whapp called Crossbar that exposes JSON APIs
delivered via HTTP/REST, and these focus more on higher level concepts
such as voicemail boxes, users, accounts, billing, etc. We hope these
provide building blocks to then control calls without having to muck
with VoIP concepts (which is what our callflow builder is designed for
- - you can route a call for a DID/extension to a user, and then a
voicemail box if the user doesn't answer in time, all without
understanding what a bridge is, how to send/record audio, how to
upload the stored recording to BigCouch, etc).

5. Does all communication
>>>> within the Kazoo platform happen over the AMQP bus? By "all"
>>>> I mean, ecallmgr <-> FS (as per #1 above), ecallmgr <->
>>>> Whapps layer, ecallmgr <-> BigCouch layer?

Hopefully the overview picture explained this clearly, but the primary
AMQP interaction is between whapps and ecallmgr.

6. What about communication between
>>>> WhApps? Do they communicate with each other over AMQP? or do
>>>> they not talk to each other directly but do so through
>>>> ecallmgr, which is the orchestrator of the whole Kazoo show?

Whapps definitely talk to each other via AMQP. For instance, Crossbar
(REST API whapp) has a mechanism such that every time a document is
created/edited/deleted, we publish that event to the bus. This allows
other whapps that may cache data in-memory (such as the heavily used
callflow whapp) to proactively clear their cache and fetch the new
data on the next request.

Whapp instances talk to each other as well. In the upcoming call
center application (acdc), the queue process(es) use AMQP to publish
when a customer call is available. Agent processes respond via AMQP of
their availability. This allows you to run two or more acdc whapps on
different servers. If one server goes down, the whole acdc doesn't
stop functioning. So AMQP is heavily utilized

7. Is it possible to
>>>> talk to the Whapps layer directly? or is that done through
>>>> the Kazoo API or the ecallmgr?

Not sure what you mean here? Each whapp has at least one AMQP listener
(callflow has a listener for route_req messages published by
ecallmgr_fs_route, for instance). Crossbar has an HTTP interface.
media_mgr has both and AMQP interface and an HTTP interface (AMQP for
asking for the URL for a media file, HTTP for delivering the binary data).

8. There was another question I had
>>>> which I forgot, but I thought of another one instead...when
>>>> the call flow is built using the Kazoo UI with dragging and
>>>> dropping the Whapps, is the configuration pushed down as XML
>>>> into the FS nodes to create dialplan entries or is this
>>>> controlled dynamically by the ecallmgr_call_control by
>>>> following this "recipe" on a per call basis?

NO! The callflow remains in the JSON format the Callflow builder
creates. This is stored in the database. When a call comes in that
will "execute" that callflow, the callflow whapp loads the callflow
JSON tree from the DB and starts sending dialplan commands based on
that tree and the caller's interactions (DMTF, voice, etc).

The only FreeSWITCH XML we generate is for authn_resp and route_resp
messages. Everything else is done via events/dialplan commands
(mod_dptools).

>>>>
>>>> Wish I could remember what that last question was, but if you
>>>> guys can answer the ones above, it will be of GREAT help.
>>>>
>>>> Sorry about so many questions but I couldn't find this
>>>> information in the Wiki but if it's there somewhere then
>>>> please feel free to tell me to go RTFM with a pointer to
>>>> where this is. Also, I've been trying to find a schematic /
>>>> architecture blue-print of the Kazoo platform but don't think
>>>> it's in the Wiki, is that correct?
>>>>
>
>
>>
>

- --
James Aimonetti
Distributed Systems Engineer / DJ MC_

2600hz | http://2600hz.com
sip:ja...@2600hz.com
tel: 415.886.7905
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQadP+AAoJENc77s1OYoGgiF4IAIT29PV9a8cOxuQ3UoESx5+f
iIO4nwmb+LFxK2XItpy+JA4HQMgx52LV6mc2ve7NQLm0179ZnF0BnrQLPtBW3sNj
+02GDCMnFoh1WqJdGCaUmlUkfdLeAneQS2SjEkVrLizKfr4K8u07+Oq3RDQ/v3Nz
j8/bt7In9Vxr5jVcoybjcToxjJRxpU92wMSWYD3qKmISoogsqYcsrg/i3W/QjnYy
kiTTS82WboXSa6AEOz3mQg10IrKG8rxSLSdBD9EVgQFcGrOjFAtxJDkSzwYsVkwn
QEgLEIleXlaqW6XVoj1c+igmgO+PDuMdbnFV7mp58o1WAxDkWygcgFkd22qIeNE=
=2iIm
-----END PGP SIGNATURE-----

[Darren Schreiber]

Hint: HUGE BONUS POINTS for helping us populate the Wiki.

[William Suffill]

Free Swag?

[Joshua Goldbard]

I'll hook you up with some sort of free swag if you help populate the wiki. At the very least, I'll buy you a beer if you're ever in SF :).

-Josh (the marketing guy at 2600hz)

Sent from my iPad

On Oct 1, 2012, at 1:35 PM, "William Suffill" <william...@gmail.com> wrote:

> Free Swag?

[zhi sun]

Hey James,

Thank you for sharing great infomation with us.

After going through the conference related source code, I have some questions that may need your help:

1. when the ecallmgr publishes route_req to AMQP, which component is reponsible for listening and receiving the route_req from AMQP,either conf_participant (of conference) or cf_route_req (of callflow)? i guesses the conf_particpant should be, since it has hook the handle_route_req to rout_req as below, by follwing the gen_listener mode. but it looks like the callflow app also monitor the route_req, right? how does the route_req know which one is the correct receiver?

-define(RESPONDERS, [{{?MODULE, relay_amqp}, [{<<"call_event">>, <<"*">>}]}
                     ,{{?MODULE, handle_participants_resp}, [{<<"conference">>, <<"participants_resp">>}]}
                     ,{{?MODULE, handle_conference_error}, [{<<"conference">>, <<"error">>}]}
                     ,{{?MODULE, handle_authn_req}, [{<<"directory">>, <<"authn_req">>}]}
                     ,{{?MODULE, handle_route_req}, [{<<"dialplan">>, <<"route_req">>}]}
                     ,{{?MODULE, handle_route_win}, [{<<"dialplan">>, <<"route_win">>}]}
                    ]).


2. in the conference app of whapps, where is the discovery_conference_req from (in conf_discovery.erl)? i have found the follwing code snippet in callflow app

cf_conference:handle(Data, Call)->
...
wapi_conference:publish_discovery_req(Command).

it seems that the callfow:cf_conference invokes the discovery_req. right? if so, what is the detailed workflow to trigger the conference discovery_req?


3. if i understanding correctly, in conf_descovery:handle_discovery_req, it start a participant child, and search_for_conference (by sending request to ecallmgr_conference_listener). After getting conference search result from ecallmgr side, the descovery tries to call conf_participant:join_local (in handle_search_resp), then the participant send out "participant" command to ecallmgr (via amqp).
 QUESTION: where is participants command going to finally? i can only see the ecallmgr_conference_listener receives and sends out the conference participants command to related FS via freeswitch:api. however, on FS side, i cannot find the match conference command like "participants".

4. on the FS side, after hitting the diaplan, the conference is started from the conference_function in mod_conference. what is the latest action/command/event happens on kazoo side, just before entering into the conference_function on the FS? will the user who make a conference call is parked? i just want to understand how the FS and kazoo interacts to start a conference.


5. on the whapps side, the conf_descovery creates the conf_participant per descovery request, does that mean each participant record logically represents one actual user on the FS? the participant gen_listener maintains the user case that one user may do with the conference, right? does each whapps_conference represent one actual conference on FS?

best regards,
-zhisun

2012/10/2 James Aimonetti <ja...@2600hz.com>

[James Aimonetti]

Hi Zhisun,

I'll defer most of the conference questions to Karl, should he deign to make an
appearance, as he wrote 99% of it and can point you in the right direction.

I can answer #1 though. The secret sauce is in the "topic" exchange that we
publish the route_req message to in AMQP land.

There are actually many listeners for route_req: callflows, trunkstore,
stepswitch, conference to name a few. Each of these apps will receive a
route_req and determine if they know how to handle the route. As soon as an app
knows it can't handle the route, the process it spawned to look into the matter
dies.

When an app knows it can handle a call, it crafts a route_resp and sends it
back to ecallmgr.

James