OCAWS.jl - Amazon Web Services library

[samoconnor]

I've ported (most of) my Tcl AWS library to Julia. See - https://github.com/samoconnor/OCAWS.jl

My reasons for doing this were: to learn Julia, to evaluate Julia's suitability for use in one of my client's production systems.

What I've found is that while Julia is doing lots of things right, and has lots of potential its not quite ready for real-world use yet in my situation.

The main show-stopper for me is start-up time. To be a viable scripting language replacement, startup time has to be much faster.

In order to implement web-services "Hello World' I need to pull in: JSON.jl, Zlib.jl, URIParser.jl, Requests.jl, HttpCommon.jl, Nettle.jl, Dates.jl and LightXml.jl. That means waiting almost 10 seconds for startup.

In the architecture I'm working with, worker processes are spawned by a queue poller and they need to start up fast. Many of the  existing scripts (written in Tcl) start up, do their work and shut down all within < 200ms.

I understand from reading the lists and github that people are working on improving this. I look forward to the results.

When Julia's startup time is down to the order of 100ms, I'll be back to continue work on the AWS library and start implementing some parts of the system I'm working on in Julia. At this point I'll have to put Julia on the back-burner for a while...

Features of OCAWS.jl compared to Amit's existing AWS.jl library -- https://github.com/amitmurthy/AWS.jl

Generic request processing mechanism that works across S3, SQS, SNS, EC2, IAM, STS, SDB, SimpleDB, DynamoDB, etc... 

AWS Signature Version 4 request signing.

Automatic HTTP request retry with exponential back-off.

Parsing of XML and JSON API error messages to AWSException type.

Automatic loading and refresh of EC2 Instance Credentials.

Automatic API Request retry in case of ExpiredToken or HTTP Redirect.

[John Myles White]

Hi Sam,

Have you tried using userimg.jl?

-- John

[samoconnor]

On Thursday, September 11, 2014 1:24:49 PM UTC+10, John Myles White wrote:

Hi Sam,

Have you tried using userimg.jl?

 -- John

No, mostly because what I read about userimg.jl on this list included stuff like "Not all packages support pre-compilation, and it can cause a few other problems".

It seems to be non-officialy-documents and a bit too bleeding edge.

Given that I was finding seg-fault bugs in the official parts of Julia, https://github.com/JuliaLang/julia/issues/7802, and basic RFC compliance bugs in the URI library, https://github.com/JuliaWeb/URIParser.jl/pull/13, I figured that it was best to stay away from the unofficial parts. Having too many potential sources of unexpected behaviour makes debugging too hard.

Sam

[John Myles White]

That seems like a fair assessment. Hope you come back as Julia matures.

 -- John

[Jake Bolewski]

Thanks for the report, constructive criticism is always useful.  That said, I don't know if 200 < ms startup time is obtainable.  A cold start REPL takes 0.4 s on my system.  Introducing more packages into the system image only makes this worse.  I guess a "slimmed down" base Julia would be the only way to hit those numbers, even with full precompilation.

[samoconnor]

On Thursday, September 11, 2014 1:51:22 PM UTC+10, Jake Bolewski wrote:

Thanks for the report, constructive criticism is always useful.  That said, I don't know if 200 < ms startup time is obtainable.  A cold start REPL takes 0.4 s on my system.  Introducing more packages into the system image only makes this worse.  I guess a "slimmed down" base Julia would be the only way to hit those numbers, even with full precompilation.

Back in July, when I was getting started I wrote a simple .jl script to compute an AWSV4 request signature (~8000ms). I also built a .tcl script to do exactly the same (~100ms).

It is worth noting that Tcl startup includes a whole bunch of dynamic module loading and dependency resolution. It seems reasonable to me to expect that a compiled language like Julia ought to be able to do this stuff way faster than an interpreted language like Tcl. The way I understand it the problem is that Julia compiles the whole system every time it is run. I don't see why a pre-compiled Julia system should not be able to just start executing.

If julia is going to be a viable scripting language replacement at some point it shouldn't be any slower than this:

time bash -c "echo hello"
hello


real 0m0.005s
user 0m0.002s
sys 0m0.003s

time (echo "puts hello" | tclsh -)

hello

real 0m0.019s

user 0m0.009s

sys 0m0.006s

The following is a message I sent to Amit and Viral back in July sharing my results...

As an exercise in trying out Julia, I've implemented a function to compute AWS Version 4 Authentication headers (attached).

The Julia code runs in 7887ms seconds vs 124ms for the Tcl code.

Google tells me that there is a heavy start-up penalty in Julia for importing modules.

Some other observations:

 - It took quite a bit of digging to select from the multiple available packages for SHA256, Dates, HTTP URIs, etc... If would be nice if things as basic as this were included in an "official" library of some sort.

 - Many of the package on http://pkg.julialang.org seem to be of far lower quality than Core and Base. It is great to have a place to share quick hacks and starting points for collaborative development, but as an engineer trying to get things done to a deadline, it would be great if there was a clear distinction between solid packages and works-in-progress.

 - Tcl's dicts preserve insertion order for iteration. This has very little overhead and is really nice in many situations.

 - I got confused about why I could not use the Base string() function without using the full name Base.string(). It turned out that I had a local variable lower down in the function called "string". The error message did not help me find this problem. It took quite a bit of trial and error to figure it out. In Tcl there is no name-clash between function names and variable names. I understand why there is a clash in Julia, and I like that functions are first-class data objects, I guess I just have to think more about name clashes when I choose variable names.

 - I was surprised that I could not do sort(keys(dict)). So I added: sort(i::Base.KeyIterator) = sort([k for k in i]). It is a really nice feature than I 

can do this without editing some system library source code (Tcl handles this nicely too).

 - It is a little disappointing that I can't implement @set(var_name, value) for local variables. This is a pretty big hole in the claim that meta-programming is supported. However, I can work around it. (A common pattern in my Tcl code is to unpack the contents of a dict into local variables.) It seems pretty arbitrary to allow assignment by name in the global scope, but not in the local scope. Smells like "harder to implement".

 - I don't understand why I have to provide a default value for named arguments. I understand why only positional arguments take part in dispatch decisions. I understand why there are no defaults for positional parameters (just add another method). But it seems that there is no benefit in insisting that named arguments have defaults. If there is a missing parameter, I want a "missing parameter error".

I've been using Tcl for production work since 1995.

I have taken time out to learn a few new-fangled high-level languages along the way (perl, lua, python, eiffel, java, javascript etc). However, I keep going back to Tcl. Julia is the first language I've tried where I'm not finding lots to dislike. (I assume the startup performance issues can be fixed easily by caching compiled code).

Regards,

Sam O'Connor

Sam-OConnors-MacBook-Pro-17:scratch sam$ time ./aws4_test.jl
Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/iam/aws4_request, SignedHeaders=content-md5;content-type;host;x-amz-content-sha256;x-amz-date, Signature=1a6db936024345449ef4507f890c5161bbfa2ff2490866653bb8b58b7ba1554a
Content-MD5: r2d9jRneykOuUqFWSFXKCg==
Content-type: application/x-www-form-urlencoded; charset=utf-8
host: iam.amazonaws.com
x-amz-content-sha256: b6359072c78d70ebee1e81adcbab4f01bf2c23245fa365ef83fe8f1f955085e2
x-amz-date: 20110909T233600Z

real 0m7.687s
user 0m7.720s
sys 0m0.324s
Sam-OConnors-MacBook-Pro-17:scratch sam$ time ./aws4_test.tcl
Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/iam/aws4_request, SignedHeaders=content-md5;content-type;host;x-amz-content-sha256;x-amz-date, Signature=1a6db936024345449ef4507f890c5161bbfa2ff2490866653bb8b58b7ba1554a
Content-MD5: r2d9jRneykOuUqFWSFXKCg==
Content-type: application/x-www-form-urlencoded; charset=utf-8
host: iam.amazonaws.com
x-amz-content-sha256: b6359072c78d70ebee1e81adcbab4f01bf2c23245fa365ef83fe8f1f955085e2
x-amz-date: 20110909T233600Z

real 0m0.124s
user 0m0.098s
sys 0m0.022s

[Tony Fong]

To chime in my 2 cents, I think Julia's computational model may favor a design that is "start once, offer different/repeated services". Like REPL. The incremental cost seems reasonable as long as the session is stable and is reused.

[samoconnor]

On Thursday, September 11, 2014 4:07:42 PM UTC+10, Tony Fong wrote:

To chime in my 2 cents, I think Julia's computational model may favor a design that is "start once, offer different/repeated services". Like REPL. The incremental cost seems reasonable as long as the session is stable and is reused.

One of the first places I read about Julia was here: http://graydon2.dreamwidth.org/189377.html

The author says "[Julia] is trying to span the entire spectrum of its target users' needs, from numerical inner loops to glue-language scripting to dynamic code generation and reflection. And it's doing a very credible job at it."

The prospect of a single language to replace a stack of  "scripting-language & C/C++" is what make Julia interesting to me.

I'm not sure to what extent the core Julia devs are interested in making Julia a viable scripting language. Maybe graydon2's assertion that Julia is trying to be good at "glue-language scripting" is not aligned with the Julia dev's priorities. If so, so-be-it. Language design is hard. Language implementation is hard too. Lots of trade-offs and compromises have to be made.

However, one thing is for sure. There are many scripting (and other) applications where startup time is important. Examples that come to mind include: security critical code where the risk of information leakage from one job to another means that each job must run in a new process; Safety critical code for much the same reasons; cloud/distributed code, where a job may have to shut down and start up on a different node with minimal delay; CGI scripts; boot-time initialisation scripts; command-line tools (what if git-commit was written in Julia, don't want to wait 8 seconds for commit)...

[YB Israel]

Hi Sam

just took a look at your package, and will look forward to testing it extensively.  I develop/data crunch for an online media company handling 8 billion plus impressions per day through the AppNexus exchange, and it took me less than a week to port our system to julia. Thus far have relied on Murphy's AWS.jl + tweaks/hacks to manage our cluster.  Management here is terrified, but heck, it works almost flawlessly, my productivity is just soaring now and the costs have dropped to almost nothing. AWS.jl is worrisome, there are issues with its dependency on Calendar.jl, and it's slow.  I rarely ask questions, and when I do they are often "naive".... so if there is anything you are looking for in particular don't hesitate to ask. 

[Viral Shah]

We really should try and create one AWS.jl package that works. I believe that Amit created it a long time ago and it has recently started gaining traction. We really should try to pool our resources and have one amazing package.

-viral

[Viral Shah]

On the specific issue of startup time, static compilation is one of the high priority items. Compared to TCL which is interpreted, Julia compiles just-in-time, and that is expensive the first time around. However, with static compilation, we should be able to precompile the packages and this should no longer be an issue.

Until then, we only have hacks such as userimg.jl.

-viral

[Leah Hanson]

For what it's worth, I've heard basically exactly this complaint (start-up time too slow for scripting) from other people, and I'm happy to see it discussed on this list. Hopefully, static compilation will just fix this, but I think it's great to remind us that this is a real problem, especially since the Julia community has self-selected down to people who can tolerate the current start up time (and who mostly could tolerate the multisecond no-sysimg start up time that made/makes the current start up feel fast).

-- Leah

[Tim Holy]

Sam, I agree that decreasing load times is a critical priority. But just to
make sure your expectations are reasonable (at least, for how hard of a
problem this is, and perhaps what final result we'll achieve): it's actually
_much_ easier to achieve short load times in an interpreted language than in
one that gets compiled. The kind of "compiling" (lowering) done in an
interpreted language is much simpler than what julia has to do to generate
machine code. Of course the advantage of generating machine code is that, once
generated, it runs a lot faster. In a sense, Julia has gone for fast
execution, but the inevitable consequence is that it takes much more analysis.

Here's an analogy: you'd say C is a fast language, right? But compare the
amount of time needed to _compile_ a C program to the time needed simply for
just the I/O (to read each file)---you're talking orders-of-magnitude
difference. tcl's job is a little closer to just reading the files, with only
minimal parsing at load time.

Not saying that we don't need to do a lot better. But personally, I'll be
reasonably happy if a package the size of Images loads in < 1 second.

--Tim

[Mike Innes]

 - It is a little disappointing that I can't implement @set(var_name, value) for local variables. This is a pretty big hole in the claim that meta-programming is supported. However, I can work around it. (A common pattern in my Tcl code is to unpack the contents of a dict into local variables.) It seems pretty arbitrary to allow assignment by name in the global scope, but not in the local scope. Smells like "harder to implement".

 Can you expand on this? Something like @set should be perfectly possible. So should unpacking dicts into the local scope, so long as you are explicit what you want to unpack. What do you mean by "assignment by name"?

 - I don't understand why I have to provide a default value for named arguments. I understand why only positional arguments take part in dispatch decisions. I understand why there are no defaults for positional parameters (just add another method). But it seems that there is no benefit in insisting that named arguments have defaults. If there is a missing parameter, I want a "missing parameter error".

 Actually, defaults for positional arguments work perfectly fine. As for errors by default you can do that with:

foo(a = 1; b = error("b required")) =

  a + b

You could make a MissingParameterError type to cut down the boilerplate, of course.

[Iain Dunning]

The need for faster package loading is well talked about at this point, I certainly don't have anything to add of value. My contribution to this thread:

 - Many of the package on http://pkg.julialang.org seem to be of far lower quality than Core and Base. It is great to have a place to share quick hacks and starting points for collaborative development, but as an engineer trying to get things done to a deadline, it would be great if there was a clear distinction between solid packages and works-in-progress.

This is an accurate statement. I've been trying to provide signals, including

- Nightly testing of the release version of package against Julia stable and Julia nightly, and displaying that test result

- Number of Github stars

- Time since last commit/version number

- Deprecated warning (i.e. package is declared to be deprecated)

Things I'm hoping to add:

- How many packages depend on this package?

- Estimate of number of installs (from Github)

- Maybe more quality indicators, e.g. do the tests trigger any deprecation warnings?

As a user: do you have any other signals you'd like to see?

[samoconnor]

Hi Tim,

Hi Tim, I mostly agree.

Tcl has a JIT and a byte code machine, but it does not do any deep analysis in its JIT. It is basically just a pre-parser.

A lot of Tcl's speed comes from lazy evaluation, copy-on-write etc.

Your basic point is that an interpreted language starts up faster than a compiled language compiles. That's fine.

But if Julia is a compiled language, it should be possible to compile once and then start the binary in near zero time.

It sounds like this kind of capability is coming.

However, there is a risk in the way some of Julia's current libraries are designed.

The problem is that Julia has nice dynamic language features, so it is tempting to do things dynamically at startup.

e.g. 

• Nettle.jl dynamically enumerates the hash algorithms provided by libnettle.so and dynamically generates Julia functions every time it starts up.
  https://github.com/staticfloat/Nettle.jl/blob/master/src/hash.jl#L31
  
• CRC.jl does not provide a CRC32 function, instead you do  crc32 = crc(CRC_32) which generates a CRC32 lookup table and a generates a function that knows how to use the table to compute CRC32.
  https://github.com/andrewcooke/CRC.jl
  

I see two possible solutions to this problem:

1. Ban the use of expensive initialisation code in generic libraries.
2. Have a compiler that is smart enough to execute all the initialisation code at compile time and serialise the results.

I don't know enough about LLVM internals to know if 2 is feasible, but I imagine it should be possible for the pre-compiler to run the system as usually up to the start of "main()" and then dump code+heap into a binary that can be executed fast as many times as needed.

[samoconnor]

On Friday, September 12, 2014 12:49:26 AM UTC+10, Mike Innes wrote:

 - It is a little disappointing that I can't implement @set(var_name, value) for local variables. This is a pretty big hole in the claim that meta-programming is supported. However, I can work around it. (A common pattern in my Tcl code is to unpack the contents of a dict into local variables.) It seems pretty arbitrary to allow assignment by name in the global scope, but not in the local scope. Smells like "harder to implement".

 Can you expand on this? Something like @set should be perfectly possible. So should unpacking dicts into the local scope, so long as you are explicit what you want to unpack. What do you mean by "assignment by name"?

My understanding (based on https://github.com/JuliaLang/julia/issues/2386) is that you can't eval assignment to a local variable name. You can only eval assignment to a global. I think this is a deliberate trade-off aimed at allowing the comelier to reason about and optimise the behaviour of a function body. i.e. you can't eval assignment to a local variable name because, e.g. it might have been completely optimised away.

Maybe I'm misunderstanding something. See also http://stackoverflow.com/questions/21267962/how-can-i-eval-a-local-variable-in-julia

 

 - I don't understand why I have to provide a default value for named arguments. I understand why only positional arguments take part in dispatch decisions. I understand why there are no defaults for positional parameters (just add another method). But it seems that there is no benefit in insisting that named arguments have defaults. If there is a missing parameter, I want a "missing parameter error".

 Actually, defaults for positional arguments work perfectly fine. As for errors by default you can do that with:

foo(a = 1; b = error("b required")) =

  a + b

You could make a MissingParameterError type to cut down the boilerplate, of course.

Nice one. You could do  foo(a = 1; b = @NotEmpty)...

[samoconnor]

Hi Iain,

All the automatic metrics are fine, but nothing beats a stamp that says "Endorsed by the Julia Core Team".

I think there should a Julia Standard Library that is part of the core distribution. This should include all the packages that are production ready and no packages that are not.

There should only be one way to do basic stuff like Dates, Zlib, SQL, XML, JSON, URIs, TLS, HTTP, CRC32, SHA256, MD5, CGI, etc...

This requires someone's time to curate the available packages. It can be especially difficult for things like platform specific interface libraries for platforms that none of the core team use.

However, I think a batteries-included core distribution is an essential part of a viable general-purpose language. When I realise that I need to compute a CRC32, I should just be able to look up the function in the standard library. I should not have to spend an hour evaluating three possible libraries, then finding that the one I chose is really slow. (Hint - use the crc32() from Zlib.jl)

Just to be clear, I don't want any of this to be interpreted as complaining. Julia is a new language. The Julia folks are doing great work for which I am grateful. I just hope that my perspective helps in some way.

[samoconnor]

Hi YB Israel,

Be warned that the package is incomplete and has not been tested in real-world use.
That being said, please let me know if you find any problems.

[Iain Dunning]

Hi Sam,

Python tried to do something like that, whereby popular packages get incorporated into the core. However, that led to the stagnation of those packages, to the point where people end up using third-party libraries anyway. I also think the "picking winners" thing is kind of at odds with the open-source mentality - especially if its "The Julia Project" (whoever that is) is the one to do it.

However, I think there is answer: I'm not sure if you are aware of Julia Organizations, e.g. https://github.com/JuliaWeb is one that was recently formed. These organizations are the way we have been doing curation - they avoid duplication of effort, and encourage interoperability. You may have noticed that there is duplication in HTTP requests/CURL/etc. - JuliaWeb has already cut through some of that duplication, and picked winners. 

Another example is JuliaOpt (http://juliaopt.org) makes it a requirement that our packages work on all 3 major platforms.

Of your examples:

- Dates - there was only one package, but is been merged into Base for Julia 0.4

- Zlib - only one package

- SQL - no idea

- XML - Only LightXML.jl

- JSON - only the excellent JSON.jl, part of the JuliaLang org (a sign of quality, I should add)

- URIs - was more than one, but only one is in JuliaWeb

- TLS, HTTP - ditto

- The CRC32 thing is funny, because I'd agree thats possibly the worst duplication of functionality out there right now!

What I don't agree is that it should be part of the standard library, at least for the current definition of it. What we've found is that people have very different use cases for Julia, and what some consider essential and completely redundant to others. For example, I don't ever use CRC32 (not in Base) or FFTs (is in Base)... This issue https://github.com/JuliaLang/julia/issues/5155 captures a lot of the discussion that has taken place on this topic.

In summary, I think the problem is communication and aiding discovery, not the need for "official" packages. In general my plans to improve the current situation is 

- to lean on organizations to be curators in their own personal domains

- I provide some level of curation on top of that, where I can, to funnel people to those packages.

Thanks for your feedback, its nice to have people be engaged about these dryer topics!

- Iain

[samoconnor]

On Friday, September 12, 2014 11:49:48 AM UTC+10, Iain Dunning wrote:

Hi Sam,

Python tried to do something like that, whereby popular packages get incorporated into the core. However, that led to the stagnation of those packages, to the point where people end up using third-party libraries anyway. I also think the "picking winners" thing is kind of at odds with the open-source mentality - especially if its "The Julia Project" (whoever that is) is the one to do it.

Picking winners is one way to see it. However, picking something and sticking with it can be valuable even if "it" turns out "not to be the winner". Stability and certainty lets people get on with writing their own code rather than thinking about choosing libraries. Every so often an "official" package can be replaced by a "winning" package if the benefit of the winning features out-weigh the upgrade pain for the installed base.

 

However, I think there is answer: I'm not sure if you are aware of Julia Organizations, e.g. https://github.com/JuliaWeb is one that was recently formed. These organizations are the way we have been doing curation - they avoid duplication of effort, and encourage interoperability. You may have noticed that there is duplication in HTTP requests/CURL/etc. - JuliaWeb has already cut through some of that duplication, and picked winners. 

That looks interesting, but it looks like JuliaWeb includes both LibCURL.jl and Requests.jl, which as far as I can see are different approaches to the same problem (e.g. AWS.jl uses LibCURL.jl whereas OCAWS.jl uses Requests.jl)

 

Another example is JuliaOpt (http://juliaopt.org) makes it a requirement that our packages work on all 3 major platforms.

Of your examples:

- Dates - there was only one package, but is been merged into Base for Julia 0.4

- Zlib - only one package

- SQL - no idea

- XML - Only LightXML.jl

- JSON - only the excellent JSON.jl, part of the JuliaLang org (a sign of quality, I should add)

- URIs - was more than one, but only one is in JuliaWeb

- TLS, HTTP - ditto

- The CRC32 thing is funny, because I'd agree thats possibly the worst duplication of functionality out there right now!

It looks like things are improving, when I started out I had to choose between

- Dates.jl vs Calendar.jl,

- LightXML.jl vs LibExpat.jl

- Codecs.jl vs Zlib.jl (and base64)

etc

 

What I don't agree is that it should be part of the standard library, at least for the current definition of it. What we've found is that people have very different use cases for Julia, and what some consider essential and completely redundant to others. For example, I don't ever use CRC32 (not in Base) or FFTs (is in Base)... This issue https://github.com/JuliaLang/julia/issues/5155 captures a lot of the discussion that has taken place on this topic.

I would agree that Base should be really small.

But I think there should be another official standard lib that includes things like FFTs and CRCs (and all the other TLAs I listed before).

 

In summary, I think the problem is communication and aiding discovery, not the need for "official" packages. In general my plans to improve the current situation is 

- to lean on organizations to be curators in their own personal domains

- I provide some level of curation on top of that, where I can, to funnel people to those packages.

Thanks for your feedback, its nice to have people be engaged about these dryer topics!

Keep up the good work!

[Viral Shah]

Almost always, we work towards collaboration among package authors rather than having multiple packages. Many packages have merged over time, and that will continue to happen. Sometimes, there are different approaches and you end up with different packages for a variety of reasons.

-viral