Foxx Microservice : use of a queue system (ie AWS SQS)

[Florian]

Hi everyone,

I would like to use an AWS SQS (Simple Queue Service) to push some events ( update_entity, create_entity, ...) from my Foxx microservices, in order to process those events in another service ( which could be another Foxx service or not ).

But as ArangoDB Foxx doesn't manage node packages like a NodeJS application, the aws-sdk package doesn't work (TypeError: util.crypto.lib.createHash is not a function)

Is there any workaround to push events from a Foxx microservice to a queue system (AWS or others) ?

I was wondering if I had to put a NodeJS app layer in front of my Foxx service but I think I'll loose the advantages of using Foxx (and I already have a API Gateway in front of the Foxx services, I think so 3 app layers is too much)

Thanks in advance !

Florian

[Claudius Weinberger]

Hi Florian,

as far as I know SQS have a HTTP interface. You can use HTTP (TLS 1 only atm) from FOXX and it has its own queue. Maybe that would be helpful.

Claudius

[Florian BOJDA]

Hi Claudius !

Thanks for your reply, that's what I actually did and I embed it in a Foxx service to reuse it in others Foxx.

I was stuck with the request's encryption but that's my feedback about it (maybe it would be helpful, ref : https://github.com/arangodb/arangodb/issues/1986 ) :

Signing the AWS request requires the createHmac function from crypto node's package :

 crypto.createHmac('sha256', 'a secret').update(stringToSign).digest('base64');

ArangoDB implements :

crypto.hmac('a secret', 'stringToSign', 'sha256')

which return a String in hexadecimal encoding .

So all we need is call the function like this to make a AWS-compliant encryption :

new Buffer(crypto.hmac('a secret', 'stringToSign', 'sha256'), 'hex').toString('base64');

Florian

--
You received this message because you are subscribed to a topic in the Google Groups "ArangoDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/arangodb/Ptw8hX9kbLs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to arangodb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Florian]

I have another question about HTTP request in ArangoDB :

ArangoDB uses synchronous connections so when we send an HTTP request in a Foxx route, is ArangoDB waiting for the HTTP response before sending its own response ?

An example :

I have a route in a Foxx service (/createEntity) which :

• save user data into ArangoDB
• send an HTTP request to the SQS (the queue service).

Is the Foxx route waiting for the HTTP request to be sent or return the response immediately after calling request() function ?

If it is, is it interesting to create a Foxx queue which handles these HTTP requests ? Is there another way to not wait ?

[Claudius Weinberger]

Am 13. September 2016 um 11:07:41, Florian (floria...@gmail.com) schrieb:

I have another question about HTTP request in ArangoDB :

ArangoDB uses synchronous connections so when we send an HTTP request in a Foxx route, is ArangoDB waiting for the HTTP response before sending its own response ?

Yes, it is. That's why we introduce Foxx Queue. Did you have a look at the Foxx Queues already?

An example :

I have a route in a Foxx service (/createEntity) which :

• save user data into ArangoDB
• send an HTTP request to the SQS (the queue service).

Is the Foxx route waiting for the HTTP request to be sent or return the response immediately after calling request() function ?

If it is, is it interesting to create a Foxx queue which handles these HTTP requests ? Is there another way to not wait ?

Thanks in advance !

Florian

--
You received this message because you are subscribed to the Google Groups "ArangoDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to arangodb+u...@googlegroups.com.

[Florian]

Sorry for my late answer ;)

Yes, I already use Foxx queues for some work, but I'm a bit confused about error handling

In the documentation, it says :

Any errors raised by the script will be handled depending on how the script was invoked:

• if the script was invoked from the HTTP API (e.g. using the web interface), it will return an error response using the exception's statusCode property if specified or 500.
• if the script was invoked from a Foxx job queue, the job's failure counter will be incremented and the job will be rescheduled or marked as failed if no attempts remain.

Since I use it from a Foxx Service, when I throw errors, the job remains in progress status and blocks every next jobs. How can I handle this problem ?

My code is :

if(response.statusCode >= 200 && response.statusCode < 300) {

    module.exports = true;

} else {

    throw new errors.EventFailedError("event_failed", eventData);

}

Thanks in advance !

Florian

[Claudius Weinberger]

Can you send us a skeleton of your use-case, so that we better understand your workflow and how to solve the problem you have encountered?

[Thilo Bangert]

Hi Claudius,

Den 16-10-2016 kl. 17:27 skrev Claudius Weinberger:
> Can you send us a skeleton of your use-case, so that we better
> understand your workflow and how to solve the problem you have encountered?
>

Why would you ask this? As Florian points out, there is a discrepancy
between the documentation and the actual implementation.

https://github.com/arangodb/arangodb/issues/2042

The documentation describes a very useful feature in Foxx ;-)

kind regards
Thilo

>
>> Am 06.10.2016 um 11:53 schrieb Florian <floria...@gmail.com

>> <mailto:floria...@gmail.com>>:

>>
>> Sorry for my late answer ;)
>>
>> Yes, I already use Foxx queues for some work, but I'm a bit confused
>> about error handling
>>
>> In the documentation, it says :
>>
>> Any errors raised by the script will be handled depending on how the
>> script was invoked:
>>

>> * if the script was invoked from the HTTP API (e.g. using the web

>> interface), it will return an error response using the
>> exception's |statusCode| property if specified or 500.

>> * if the script was invoked from a Foxx job queue, the job's failure

>> counter will be incremented and the job will be rescheduled or
>> marked as failed if no attempts remain.
>>
>> Since I use it from a Foxx Service, when I throw errors, the job
>> remains in progress status and blocks every next jobs. How can I
>> handle this problem ?
>>
>> My code is :
>>
>> if(response.statusCode >= 200 && response.statusCode < 300) {
>> module.exports = true;
>> } else {
>> throw new errors.EventFailedError("event_failed", eventData);
>> }
>>
>> Thanks in advance !
>> Florian
>>
>> Le mardi 13 septembre 2016 12:06:25 UTC+2, Claudius Weinberger a écrit :
>>
>>
>> Am 13. September 2016 um 11:07:41, Florian (floria...@gmail.com

>> <javascript:>) schrieb:

>>
>>> I have another question about HTTP request in ArangoDB :
>>>
>>>
>>> ArangoDB uses synchronous connections so when we send an HTTP
>>> request in a Foxx route, is ArangoDB waiting for the HTTP
>>> response before sending its own response ?
>>
>> Yes, it is. That's why we introduce Foxx Queue. Did you have a
>> look at the Foxx Queues already?
>>
>>
>>>
>>>

>>> _*An example :*_

>>> I have a route in a Foxx service (/createEntity) which :
>>>

>>> * save user data into ArangoDB
>>> * send an HTTP request to the SQS (the queue service).

>>>
>>> Is the Foxx route waiting for the HTTP request to be sent or
>>> return the response immediately after calling request() function ?
>>>
>>>
>>> If it is, is it interesting to create a Foxx queue which handles
>>> these HTTP requests ? Is there another way to not wait ?
>>>
>>> Thanks in advance !
>>>
>>> Florian
>>> --
>>> You received this message because you are subscribed to the
>>> Google Groups "ArangoDB" group.
>>> To unsubscribe from this group and stop receiving emails from it,

>>> send an email to arangodb+u...@googlegroups.com <javascript:>.

>>> For more options, visit https://groups.google.com/d/optout

>>> <https://groups.google.com/d/optout>.

>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "ArangoDB" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to arangodb+u...@googlegroups.com

>> <mailto:arangodb+u...@googlegroups.com>.

>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "ArangoDB" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to arangodb+u...@googlegroups.com

> <mailto:arangodb+u...@googlegroups.com>.

[Alan Plum]

Hi Florian,

did you check out ArangoDB 3.1? There was a bug in the queue logic that resulted in failing jobs getting stuck in pending state. Maybe this is what you encountered?

Cheers,

Alan

[Florian]

Hi Alan,

Yes I checked it out and it's okay for me, job queues are working as expected ;)

Thanks for you answer !

Florian