[ANN] lua-resty-random, lua-resty-scrypt, and lua-scrypt 0.0.1

[Aapo Talvensaari]

Hi Everyone,

I started to build a few modules for OpenResty. Currently I have done these:

• https://github.com/bungle/lua-resty-random (LuaJIT based random library)
  
• https://github.com/bungle/lua-resty-scrypt (LuaJIT based scrypt library)
  
• https://github.com/bungle/lua-scrypt (C-module for lua-resty-scrypt)
  

I attached a patch file to OpenResty configure-file for these (in this mail). These modules are the first ever that I have done for OpenResty, and they are considered version 0.0.1 quality.

Current the lua-resty-random library has a naming conflict with lua-resty-string library (both have random.lua -> --without-lua-resty-string) , so maybe they could be merged, or I will rename mine.

I think this was interesting task to do, and I'm looking forward to make more libraries. Currently I'm thinking about implementing lua-resty-session (both stateless, and statefull), lua-resty-cookie (cookie helper lib), lua-resty-auth (usename/password, and maybe oauth, openid, facebook login etc. support), and some others (libxl, and maybe some pdf-lib too). But as this is rather new to me, I would like to get some feedback about those that I have already done.

One more question, though. Right now I have this line in scrypt.lua:

local scrypt = ffi.load("/usr/local/openresty/lualib/scrypt.so")

This was the only way I figured out how to load that scrypt.so on my Mac (inside OpenResty). I know that there must be a better way to do this, but I cannot figure it out. I would love to get feedback on this.

Also, how should we approach OpenResty to include (reviewed, of course) libraries in main OpenResty bundle? I'm happy to hear if my libs are too low quality to be included, but I think that this platform could grow really fast if patches/pull requests start flowing to OpenResty. Forgive me if I'm asking something stupid.

Kind Regards

Aapo

[ch...@colman.io]

Hello,

I'm the original contributor for the random portion of lua-resty-string.

Overall, I think that lua-resty-random could be merged into lua-resty-string, but I do have two questions first:

For your bytes function, I noticed that it only uses OpenSSL's RAND_psuedo_bytes if native pseudo-random sources are unable to be read. Is this due to a noticeable performance hit when using OpenSSL's method?

What is the purpose of the token function? It seems fairly context-specific, as it seems to generate a string of varying length with characters matching [0-9A-Za-z]. Perhaps making it accept character ranges would be more suitable for the purposes of others.

As for loading scrypt and making it available to OpenResty's Lua, there are two alternatives that I know of:

1. You can make a compiled Lua module that uses Lua's API and distribute it as a LuaRock. This allows access by normal Lua and LuaJIT, but doesn't use FFI, so it will likely be slower.
   
2. Statically compile the necessary functions with Nginx and ensure they are accessible by LuaJIT's FFI. This requires recompiling Nginx every time there is a change to your code, but will likely be a bit faster. This would be similar to what is done in lua-resty-core

Regards,

Chase

[Aapo Talvensaari]

One more question, though. Right now I have this line in scrypt.lua:

local scrypt = ffi.load("/usr/local/openresty/lualib/scrypt.so")

Okay, I think I understand this better now. I could reduce this line to simply this:

local scrypt = ffi.load("scrypt")

If I put this in nginx config:

lua_package_cpath "/usr/local/openresty/lualib/?.so;;";

Then I started to think this a little bit more. I looked at the bundled lua_cjson. I checked that it is not an nginx module (it's not builded static to nginx executable). Then I looked this further, and found out that this does the magic:

int luaopen_cjson(lua_State *l) {

...

luaL_register(l, "cjson", reg);

...

}

Is this called automatically by Lua environment or something, so that you can right away use it with:

local json = require"cjson"

Now the question moves to Lua C-bindings vs. LuaJIT bindings. I could change my lua-scrypt from ordinary C-lib to Lua enabled C-lib (there is already lscrypt that does this on Github). But don't we loose some of LuaJIT magic here (easy to bring, and call whatever C-lib there is, and ability to JIT code effectively)? I know that agentzh is building lua-resty-core where he has replaced some of ngx_lua functions with pure Lua, and where he uses LuaJIT FFI to call C-functions that are statically linked to nginx executable. I could do that too, and build scrypt C-lib statically to nginx, and remove ffi.load alltogether, but that doesn't feel right. I think that external libs should be external, and maybe loaded automatically (ffi.load) from some path where you put them when OpenResty (nginx) is started. And then just use FFI to access them (without ffi.load, of course). Does this make any sense?

Regards

Aapo

[Yichun Zhang (agentzh)]

Hello!

On Tue, Dec 17, 2013 at 6:33 AM, Aapo Talvensaari wrote:
> I started to build a few modules for OpenResty. Currently I have done these:
>
> https://github.com/bungle/lua-resty-random (LuaJIT based random library)
> https://github.com/bungle/lua-resty-scrypt (LuaJIT based scrypt library)
> https://github.com/bungle/lua-scrypt (C-module for lua-resty-scrypt)
>

Great! Thank you for your contributions!

> I attached a patch file to OpenResty configure-file for these (in this
> mail). These modules are the first ever that I have done for OpenResty, and
> they are considered version 0.0.1 quality.
>

Libraries bundled in OpenResty requires careful code review and
testing on my side, which means long delay :P

Actually I'm thinking about setting out a centralized package
management web site and related toolchain for 3rd-party libraries for
ngx_lua/openresty. Kinda like CPAN and LuaRocks. So that everyone can
publish their libraries and everyone can install the libraries with a
single command, like

$ lresty resty.scrypt

> Current the lua-resty-random library has a naming conflict with
> lua-resty-string library (both have random.lua ->
> --without-lua-resty-string) , so maybe they could be merged, or I will
> rename mine.
>

Yeah, module name conflicting is bad. Maybe we can collaborate here?

> I think this was interesting task to do, and I'm looking forward to make
> more libraries. Currently I'm thinking about implementing lua-resty-session
> (both stateless, and statefull), lua-resty-cookie (cookie helper lib),
> lua-resty-auth (usename/password, and maybe oauth, openid, facebook login
> etc. support), and some others (libxl, and maybe some pdf-lib too).

Awesome! Keep them coming!

> But as
> this is rather new to me, I would like to get some feedback about those that
> I have already done.
>

I hope you can create a test suite for your libraries based on my
Test::Nginx test scaffold. You can take a look at my existing
lua-resty-* libraries for examples, like lua-resty-redis.

Also, please use "local" to declare your local Lua variables in
functions wherever possible. For example, in your "token" function in
your resty.random module, the variables "n", "count", "token" are all
global by accident. BTW, you can use the lua-releng tool to analyze
your Lua source for such issues. See

https://github.com/chaoslawful/lua-nginx-module#lua-variable-scope

and

https://github.com/agentzh/nginx-devel-utils/blob/master/lua-releng

> One more question, though. Right now I have this line in scrypt.lua:
> local scrypt = ffi.load("/usr/local/openresty/lualib/scrypt.so")
>
> This was the only way I figured out how to load that scrypt.so on my Mac
> (inside OpenResty).

I think you can just search the paths in package.cpath and build the
absolute path to the scrypt.so file yourself in Lua. And then feed the
path to ffi.load().

BTW, /usr/local/openresty/lualib/?.so should already in package.cpath
if you're using the OpenResty bundle, and you don't have to specify
that explicitly via the lua_package_cpath directive. You can dump out
the existing cpath by print out the package.cpath thing in Lua.

There're some other places in your resty.random module that deserve
optimizing or tweaking:

1. "local p = ffi_new("int[?]", 1)" in a Lua function is expensive. I
think you can create such buffers on the toplevel of your module and
reuse it everytime you need it.

2. Don't call ffi_load in your function bytes everytime it is gets
called.You should load it only once.

3. instead of opening and closing the file /dev/urandom, maybe you can
just cache the file handle on the toplevel of your Lua module?

4. I think you'd better local'ize things like math.randomseed() and
table.concat() on the toplevel of your Lua module to save the table
gets in your Lua function calls.

5. in your function token, maybe you should put the alphabet table
{'A', 'B', 'C', 'D', ... } onto your module's toplevel to prevent
duplicating it everytime your "token" function gets called?

6. For the coding style, I hope you can eliminate source code lines
longer than 80 columns ;)

Best regards,
-agentzh

[Yichun Zhang (agentzh)]

Hello!

On Wed, Dec 18, 2013 at 9:59 AM, Aapo Talvensaari wrote:
> Now the question moves to Lua C-bindings vs. LuaJIT bindings. I could change
> my lua-scrypt from ordinary C-lib to Lua enabled C-lib (there is already
> lscrypt that does this on Github). But don't we loose some of LuaJIT magic
> here (easy to bring, and call whatever C-lib there is, and ability to JIT
> code effectively)?

Create FFI-based modules wherever possible, otherwise the user code
paths using your stuff will never be JIT compiled in LuaJIT.

You can search the .so file in system library paths and in
package.cpath. Maybe we can create a dedicated Lua library for this?
;) The path search can just be done once (in the toplevel of the user
modules) so speed is not really an issue here.

> build scrypt C-lib statically to nginx,
> and remove ffi.load alltogether, but that doesn't feel right.

No, don't do that. That'll be impossible to use your stuff with
existing binary builds of OpenResty.

Regards,
-agentzh

[Aapo Talvensaari]

Overall, I think that lua-resty-random could be merged into lua-resty-string, but I do have two questions first:

For your bytes function, I noticed that it only uses OpenSSL's RAND_psuedo_bytes if native pseudo-random

sources are unable to be read. Is this due to a noticeable performance hit when using OpenSSL's method?

To others, we talked this on email, because I accidentally did "Reply to Author".

I studied this a little bit, and found out that RAND_pseudo_bytes just calls RAND_bytes. The only difference seems to be that RAND_pseudo_bytes clears errors stack, which is fine here. Both return 1 if they are able to acquire cryptographically secure bytes. And further, it seems that OpenSSL RAND_* uses Operating System normal random sources (like /dev/urandom). So, both RAND_* functions are non-blocking.

Today I revised my random.lua code. Now it works with RAND_pseudo_bytes by default if it is available, and if not it fall backs to /dev/urandom (*nix), and SystemFunction036 (= 

RtlGenRandom) (Windows). I also modified bytes function to return 2 values:

local bytes,is_crypto_secure = random.bytes(1)

So if you really need to get crypto-random bytes, you need to call this function until is_crypto_secure == true. Windows native implementation does set is_crypto_secure = true always (althought I'm not sure if it should). RAND_pseudo_bytes sets is_crypto_secure = true if it was able to get strong bytes, otherwise false. /dev/urandom sets is_crypto_secure = false always as there is no easy way to know if it succeed (you have to use /dev/random to always get strong bytes, but that is a blocking function).

What is the purpose of the token function? It seems fairly context-specific, as it seems to generate a string of varying length with characters matching [0-9A-Za-z]. Perhaps making it accept character ranges would be more suitable for the purposes of others.

This is now implemented, as you suggested. Supports strings, and arrays (arrays can contain strings and unicode or whatever chars too).

I have tested the code on mac, windows, and linux, and it seems to work alright, and fallbacks do seem to work in case OpenSSL is missing.

Regards

Aapo

[Aapo Talvensaari]

Actually I'm thinking about setting out a centralized package
management web site and related toolchain for 3rd-party libraries for
ngx_lua/openresty. Kinda like CPAN and LuaRocks. So that everyone can
publish their libraries and everyone can install the libraries with a
single command, like 

 

$ lresty resty.scrypt

 

That's a really good idea. I'm all in. Kinda like a npm for node, except that resty is

more like a web framework than a general purpose system. But yes, I

think package management is a good idea since many packages will depend on ngx.*

anyway, and they are not really usable elsewhere.

 

I think I made all the optimizations, and fixes to my code that you suggested.

I hope you can create a test suite for your libraries based on my
Test::Nginx test scaffold. You can take a look at my existing
lua-resty-* libraries for examples, like lua-resty-redis.

 

I will look at Test:Nginx. Is there anything special that I should look

at when setting it up? I just run make test or something (in a mean

time I will read this: https://github.com/agentzh/test-nginx)?

One thing did come to my mind. When caching variables, doesn't it

make problems with threading? So you have to be carefull in what you cache?

For example, if you look at this:

https://github.com/bungle/lua-resty-scrypt/blob/master/lib/resty/scrypt.lua#L36-L39

I know that you didn't suggest these. But please clarify me if you see the same problem here.

I.e. two threads calling scrypt.crypt at the same time. Isn't there a potential problem? So should

I move these back to function locals?

Regards

Aapo

[Yichun Zhang (agentzh)]

Hello!

On Thu, Dec 19, 2013 at 5:12 AM, Aapo Talvensaari wrote:
> That's a really good idea. I'm all in. Kinda like a npm for node, except
> that resty is
> more like a web framework than a general purpose system. But yes, I
> think package management is a good idea since many packages will depend on
> ngx.*
> anyway, and they are not really usable elsewhere.
>

I'll try to set up something here soon :)

> I think I made all the optimizations, and fixes to my code that you
> suggested.
>

Cool :)

> I will look at Test:Nginx. Is there anything special that I should look
> at when setting it up? I just run make test or something (in a mean
> time I will read this: https://github.com/agentzh/test-nginx)?
>

Yeah the documentation of Test::Nginx on CPAN is worth a look:

http://search.cpan.org/perldoc?Test::Nginx::Socket

though the CPAN version is older than the version on github, for
obvious reasons :)

Also, ngx_lua's test suite has some documentation:

https://github.com/chaoslawful/lua-nginx-module#test-suite

For your lua-resty-* libraries' test suite, you don't need most of the
extra dependencies there though.

> One thing did come to my mind. When caching variables, doesn't it
> make problems with threading? So you have to be carefull in what you cache?
>

Nginx's worker processes are single-threaded, which means that your
Lua code can not be preempted at all. You only need to be careful when
there's nonblocking operations (like ngx.sleep or cosocket operations)
between your Lua code path, which is be the case in your case.

> For example, if you look at this:
>
> https://github.com/bungle/lua-resty-scrypt/blob/master/lib/resty/scrypt.lua#L36-L39
>

They're fine because you use these cdata objects only in a tight
sequence of pure CPU-bound operations.

This is the beauty of cooperative (userland) threading based on single
OS threading :)

Regards,
-agentzh

[Aapo Talvensaari]

> On Thu, Dec 19, 2013 at 5:12 AM, Aapo Talvensaari wrote:
> That's a really good idea. I'm all in. Kinda like a npm for node, except

t> hat resty is more like a web framework than a general purpose system.

I'll try to set up something here soon :)

Great. I started to think about this naming conflict, and I think that we should have a convention in place for packages. Even without package management. I think that we should follow the convention you have used in lua-resty-core.

E.g.

/usr/local/openresty/lualib/resty/core.lua

/usr/local/openresty/lualib/restycore/* (all other lib files, if any)

So I propose that lua-resty-string should follow this too (this is a breaking change, I know, but better to to it now than later).

Eg.

/usr/local/openresty/lualib/resty/string.lua

/usr/local/openresty/lualib/resty/string/aes.lua

/usr/local/openresty/lualib/resty/string/md5.lua

/usr/local/openresty/lualib/resty/string/random.lua

/usr/local/openresty/lualib/resty/string/sha1.lua

/usr/local/openresty/lualib/resty/string/sha224.lua

/usr/local/openresty/lualib/resty/string/sha256.lua

/usr/local/openresty/lualib/resty/string/sha384.lua

/usr/local/openresty/lualib/resty/string/sha512.lua

What do you think? There might be a more elegant solution for that, but this looks rather simple. Name conflicts could still happen with external (non bundled libs) but that is responsibility of the user (package tool could report these).

Regards

Aapo

[Brian Akins]

The package management is one reason it's been so hard for me to get others at $DAYJOB to look at openresty.  I'm not a big fan of luarocks.  I really, really like npm, so something like that would be awesome - package.json in the app root, etc.

[Justin Cormack]

On 20 Dec 2013 17:31, "Brian Akins" <br...@akins.org> wrote:
>
> The package management is one reason it's been so hard for me to get others at $DAYJOB to look at openresty.  I'm not a big fan of luarocks.  I really, really like npm, so something like that would be awesome - package.json in the app root, etc.

The advantage of luarocks is that it already exists and it should not require a lot of work to just tweak and set up a new repo... I would hope that there might be enough citation that per app packages might not be necessary.

> --
> You received this message because you are subscribed to the Google Groups "openresty-en" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to openresty-en...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Jerome Lafon]

Hello
Is lua-resty-scrypt considered production ready?

[Aapo Talvensaari]

Hi Jeremy,

It is a small wrapper for scrypt c-lib, so I think it is ready for a production use (mainly for password hashing).

If you need more crypto functionality (say pbkdf2), I have also been creating LuaJIT bindings for Nettle:
https://github.com/bungle/lua-resty-nettle/tree/master/lib/resty/nettle

I will document and release the first version of it in following weeks. Almost all the bindings are done, but public key crypto is still work in progress (pubkey crypto adds dependency to gmp - others work without it).

Regards
Aapo

19.7.2014 16.15 kirjoitti "Jerome Lafon" <gg.l...@gmail.com>:

Hello
Is lua-resty-scrypt considered production ready?

--
You received this message because you are subscribed to a topic in the Google Groups "openresty-en" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openresty-en/7wrsSHiP72A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openresty-en...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stefan Parvu]

I think this was interesting task to do, and I'm looking forward to make more libraries. Currently I'm thinking about implementing lua-resty-session (both stateless, and statefull), lua-resty-cookie (cookie helper lib), lua-resty-auth (usename/password, and maybe oauth, openid, facebook login etc. support), and some others (libxl, and maybe some pdf-lib too). But as this is rather new to me, I would like to get some feedback about those that I have already done.

Many thanks indeed. We are using lua-resty-session and lua-resty-template , would be nice to see them bundled with OpenResty
or somehow easy to be plug as external packages.

Stefan

[Stefan Parvu]

Actually I'm thinking about setting out a centralized package
management web site and related toolchain for 3rd-party libraries for
ngx_lua/openresty. Kinda like CPAN and LuaRocks. So that everyone can
publish their libraries and everyone can install the libraries with a
single command, like

    $ lresty resty.scrypt

That would be nice.

Talking about std libraries for OpenResty - can you please consider adding by default to standard OpenResty libraries
Aapo's lua-resty-template and lua-resty-session libraries ? During our project it came vital to have a template engine
with OpenResty (we didnt want to use Lapis here) and have session management.

Thanks,
Stefan

[Stefan Parvu]

 

$ lresty resty.scrypt

 

That's a really good idea. I'm all in. Kinda like a npm for node, except that resty is

more like a web framework than a general purpose system. But yes, I

think package management is a good idea since many packages will depend on ngx.*

anyway, and they are not really usable elsewhere.

I hope this will be easy to use on any OS: MacOSX, Linux or *BSD ... Luarocks is a bit trouble to get
it compiled under FreeBSD.

Would make sense that OpenResty ships with a ready tool, like lresty (could be a Perl5 script) which can easily
add/remove OpenResty modules from a central repo. lresty should as well support adding a module from
local directory, rather than a network link since people sometimes might have a local repo locally.

What you guys think ?

stefan
 

[Yichun Zhang (agentzh)]

Hello!

On Sun, Jul 20, 2014 at 8:01 AM, Stefan Parvu wrote:
> I hope this will be easy to use on any OS: MacOSX, Linux or *BSD ...
> Luarocks is a bit trouble to get
> it compiled under FreeBSD.
>
> Would make sense that OpenResty ships with a ready tool, like lresty (could
> be a Perl5 script) which can easily
> add/remove OpenResty modules from a central repo. lresty should as well
> support adding a module from
> local directory, rather than a network link since people sometimes might
> have a local repo locally.
>
> What you guys think ?
>

Yes, that's exactly the plan :) This item is of a relatively high
priority on my TODO list :)

Regards,
-agentzh

[Jerome Lafon]

Thanks a lot!

Can I use it easily with the default parameters:

local scrypt = require"scrypt"
local hash   = scrypt.crypt("My Secret") 

Or should I configure some parameters before using it?

Jérôme

[Stefan Parvu]

>
> Yes, that's exactly the plan :) This item is of a relatively high
> priority on my TODO list :)

super. Let us know about its progress that we could help with testing and reporting
bugs.

Cheers,
--
Stefan Parvu <spa...@systemdatarecorder.org>

[Yichun Zhang (agentzh)]

Hello!

On Sun, Jul 20, 2014 at 2:38 PM, Stefan Parvu wrote:
> super. Let us know about its progress that we could help with testing and reporting
> bugs.
>

Yeah, sure. Thank you for offering help! :)

Regards,
-agentzh

[Aapo Talvensaari]

On Sunday, July 20, 2014 9:25:52 PM UTC+3, Jerome Lafon wrote:

Thanks a lot!

Can I use it easily with the default parameters:

local scrypt = require"scrypt"
local hash   = scrypt.crypt("My Secret") 

Or should I configure some parameters before using it?

By default it uses these defaults:

N (general work factor, iteration count): 32768

p (parallization factor): 1

r (block size for hash): 8

See: http://stackoverflow.com/a/12581268 (for more information)

And by default the scrypt.lua uses random salt each time using OpenSSL's RAND_pseudo_bytes. So the defaults are rather good for now. You may want to try to find better values for N, p, and r for your hardware using scrypt.calibrate() (which by default uses 104875 as maxmem, 0.5 as maxmemfrac, and 0.2 as maxtime - pretty okay defaults for password hashing).

You may call scrypt.crypt with parameters like this scrypt.crypt{ secret = "My Secret", n = 1048576, p = 1, r = 8, salt = "MySalt02", keysize = 512 }.

On my macbook the last example takes several seconds to count, so that might not be a good for user experience in logins, or heavy traffic sites.

Regards

Aapo

[Jerome Lafon]

Thanks for the details.

I will do some tests.

Regards

Jérôme