core_kernel in js_of_ocaml

[Andy Ray]

Hi,

I have been debugging my way through trying to get core_kernel up and running in js_of_ocaml.  My current progress is documented here;

http://nbviewer.ipython.org/gist/andrewray/9980135

In short there are problems in the module std_internal from typerep_lib.  I am just away to have a closer look at the module, but I am wondering how closely integrated typerep is into core_kernel.  I not sure it is going to work with js_of_ocaml due to the extensive use of the Obj module.

-Andy

[mba...@janestreet.com]

Hi Andy. I'd be more than happy to try looking with you into what specific treat of the language used in the library is causing problems in js_of_ocaml, and is there is some way of reformulating the code to make your process smoother. It seems based on your message that you're a little step ahead of the progress documentation already, right ? Looking there seems to show an issue with a recursive module, but I don't see any mention of the Obj module.

I will mp you if you'd like while sorting out the details.
Thank you,

Mathieu

[Andy Ray]

Upstream js_of_ocaml fix and typerep loads.

Add dummy unix_gethostname and caml_hash_string stubs and Core_kernel loads too.  Woohoo!

Now need to more carefully look at these stubs and figure out what is important to implement properly.

[Anil Madhavapeddy]

Wow, that's incredible progress!

unix_gethostname seems like it really shouldn't be in Core_kernel (which is supposed to be Unix-free). It's from the Unix module in the core distribution, so must be pulled in by one of the libraries.

-anil

--
You received this message because you are subscribed to the Google Groups "ocaml-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ocaml-core+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andy Ray]

Here's the complete list of unimplemented primitives

bigstring_alloc
bigstring_blit_bigstring_string_stub
bigstring_blit_string_bigstring_stub
bigstring_blit_stub
bigstring_find
bigstring_length
bigstring_marshal_blit_stub
bigstring_marshal_data_size_stub
bigstring_marshal_stub
bigstring_unmarshal_stub
bin_prot_blit_buf_float_array_stub
bin_prot_blit_buf_string_stub
bin_prot_blit_buf_stub
bin_prot_blit_float_array_buf_stub
bin_prot_blit_string_buf_stub
caml_ba_map_file_bytecode
caml_gc_counters
caml_gc_get
caml_gc_quick_stat
caml_gc_set
caml_gc_stat
caml_get_current_environment
caml_hash_double
caml_hash_string
caml_install_signal_handler
caml_int32_bits_of_float
caml_int32_float_of_bits
caml_invoke_traced_function
caml_register_code_fragment
caml_sys_system_command
caml_terminfo_backup
caml_terminfo_resume
caml_terminfo_standout
clear_caml_backtrace_pos
core_array_unsafe_float_blit
core_array_unsafe_int_blit
core_heap_block_is_heap_block
core_kernel_gc_compactions
core_kernel_gc_heap_chunks
core_kernel_gc_heap_words
core_kernel_gc_major_collections
core_kernel_gc_major_words
core_kernel_gc_minor_collections
core_kernel_gc_minor_words
core_kernel_gc_promoted_words
core_kernel_gc_top_heap_words
fixed_close_channel
unix_close
unix_environment
unix_fork
unix_kill
unix_pipe
unix_read
unix_select
unix_set_close_on_exec
unix_set_nonblock
unix_waitpid
unix_gethostname
unsafe_read_int16_t
unsafe_read_int16_t_swap
unsafe_read_int32
unsafe_read_int32_swap
unsafe_read_int32_t
unsafe_read_int32_t_swap
unsafe_read_int64
unsafe_read_int64_swap
unsafe_read_int64_t
unsafe_read_int64_t_swap
unsafe_read_uint16_t
unsafe_read_uint16_t_swap
unsafe_write_int16_t
unsafe_write_int16_t_swap
unsafe_write_int32
unsafe_write_int32_swap
unsafe_write_int32_t
unsafe_write_int32_t_swap
unsafe_write_int64
unsafe_write_int64_swap
unsafe_write_int64_t
unsafe_write_int64_t_swap
unsafe_write_uint16_t
unsafe_write_uint16_t_swap


unix_gethostname and caml_hash_string are the ones implemented at
module initialization time.


My first rough guess (without having actually done anything with core
before...) would be that the following are the most important stubs to
implement;

caml_hash_double
caml_hash_string
core_array_unsafe_float_blit
core_array_unsafe_int_blit
caml_int32_bits_of_float
caml_int32_float_of_bits
unsafe_read/write_*

> You received this message because you are subscribed to a topic in the
> Google Groups "ocaml-core" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/ocaml-core/sY3Lix1pTE0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Anil Madhavapeddy]

On 5 Apr 2014, at 14:49, Andy Ray <evilk...@gmail.com> wrote:

> core_kernel_gc_compactions
> core_kernel_gc_heap_chunks
> core_kernel_gc_heap_words
> core_kernel_gc_major_collections
> core_kernel_gc_major_words
> core_kernel_gc_minor_collections
> core_kernel_gc_minor_words
> core_kernel_gc_promoted_words
> core_kernel_gc_top_heap_words

These all used to be required at module initialization time, but
are not anymore since Core no longer seems to alter the default GC
settings when linked in. CCing the list just to make sure that
this is intended behaviour.

(incidentally, having a consolidated Core changelog web page
would make searching for all this much easier, since right now
I have to grep through GitHub pull requests or e-mail to go
back through versions).

-anil

[Andy Ray]

http://andrewray.github.io/iocamljs/core.html

It's quite a big download (~10MB after compression) so I'm not going
to advertise this generally for now.

[Anil Madhavapeddy]

A minor thing I noticed is that Clflags.real_paths isn't set to false in the toplevel, so you get full types being printed out:

List.map ;;
- :'a Core_kernel.Std.List.t -> f:('a -> 'b) -> 'b Core_kernel.Std.List.t

If you set that in the js toplevel, you should see the much more agreeable:

List.map ;;
- : ('a -> 'b) -> 'a list -> 'b list = <fun>

(this is the default in utop).

I'm not quite sure where to set Clflags in iocamljs though -- it should be picked up by Toploop I think.

-anil

> You received this message because you are subscribed to the Google Groups "ocaml-core" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to ocaml-core+...@googlegroups.com.

[Yaron Minsky]

We've actually had an internal discussion about exactly this issue.
We don't love having our own GC tuning, but when we removed it, we
found many performance regressions. We really do think that they're
better for the vast majority of applications.

Is it a problem if those GC tunings come back?

y

[Yaron Minsky]

Yeah, this is really great.

We should figure out where the gethostname links in. From looking at
the build rules briefly, it looks like core_kernel doesn't require
them directly, but maybe sexplib or bin-io or one of those libraries
brings it in. We'll figure it out.

y

[Andy Ray]

From a js_of_ocaml point of view if this will be called at module init time then so long as the stubs exist and return a 'status ok' code where necessary it's fine.

-Andy

[Yaron Minsky]

Yes, it will be called at module initialization time. So maybe
there's no issue here.

y

[Anil Madhavapeddy]

It does pull in a significant number of hooks though, and it all adds
up to bloat that we need to avoid if possible.

Wouldn't it make more sense to modify the GC settings within something
like the Async scheduler? I imagine that the settings for an Async
application are quite different from something doing Incremental
computation.

-anil

[Andy Ray]

I've spent a bit of time looking through the core_kernel sources and
so far as I can tell these seem to be the functions required for
core_kernel.

> caml_hash_double
> caml_hash_string
> core_array_unsafe_float_blit
> core_array_unsafe_int_blit
> fixed_close_channel

I will see what I can do to provide appropriate implementations and
then update the iocamljs toplevel but please do let me know if you
think I've missed some. Note that I don't see a problem with also
supporting bigstring and bin_prot as well but they seem to be fairly
self contained (and bigger) projects.

A couple of further questions;

For some reason that seemed to make sense at the time I only exposed
the module path 'Core_kernel.Std'. Should I actually just expose
everything under 'Core_kernel'?

Secondly, at the moment the build process for this is a bit
complicated requiring core be built into bytecode with a 32 bit
compiler then processed with a 64 bit js_of_ocaml. Building core as
64 bit leads to an exception in pool.ml:110 where checks are made
about the size of an int.

Are you interested in trying to properly support a build from 64 bit
bytecode? At least in the core_kernel code there don't seem to be too
many ARCH_SIXTYFOUR defines which are probably the main places where
something will need to be done.

Cheers,
Andy

[Yaron Minsky]

I have no idea why we don't support 64-bit bytecode, but we surely should.

Jeremie, any idea what that failure is about?

y

[Andy Ray]

So we have ocaml with 31 and 63 bit ints and js_of_ocaml 32 bit ints.

In a way js_of_ocaml is a new platform which doesn't exactly fit
either ocaml model.

32 bit ocaml generally maps easily to js_of_ocaml. 64 bit ocaml
sometimes needs a little massaging.

My only concern was to how you felt about (hopefully very slightly)
polluting your code with some code which specifically helped target
js_of_ocaml.

-Andy

> You received this message because you are subscribed to a topic in the Google Groups "ocaml-core" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/ocaml-core/sY3Lix1pTE0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to ocaml-core+...@googlegroups.com.

[Yaron Minsky]

Oh, dear. That's what you mean by 64 bit.

I don't see any fundamental problem with some small IFDEFs here or
there to better capture that case. I have no idea where they would
go, offhand. If you think you know, we'd surely appreciate the patch,
but we can start with some pointers to where the issues are.

y

[Andy Ray]

I've just sent a pull request for a minimal patch to core_kernel to
get this working.

With the latest js_of_ocaml, my proposed bigarray patch for
js_of_ocaml, and this patch core_kernel builds with js_of_ocaml from
both 32 and 64 bit ocaml compilers.

Note that I expect there to be runtime issues but with this working we
can at least find them!

-Andy

[Jeremie Dimino]

One issue I see with this approach is that there are several places where we use these kind of IFDEF without an assert and they will go unnoticed. For instance in the same file (pool.ml).

It seems to me a better solution would be to add a flag to configure to force 32/64bits.

Jeremie

[Andy Ray]

That would much less invasive from the Core_kernel point of view and
probably easier to support going forward. Perhaps opam would install
an ocamlfind subpackage with a 32 bit build if js_of_ocaml was
installed (or it could be a separate package).

On 32 bit arch this package is exactly the same as core_kernel. Or 64
bit arch it's a recompilation of core_kernel with 32 bit defines.

[hugo]

For the long term,
What about patching stdlib Sys.ml with
```
external int_size : unit -> int = "%int_size"
let int_size = int_size()
```
Libs could depend on this information instead of `Sys.word_size - 1`

It seems to me that limitation due to runtime representation should
always be computed in the runtime (c stubs)
The same thing for `Sys.max_array_length`

[Andy Ray]

The compilation issue with typerep went away with js_of_ocaml fix
#134. I don't know how typerep will behave at runtime. It would be
helpful if you could give it a try at;

http://andrewray.github.io/iocamljs/core.html

and report if anything/everything works.

(I can recompile if required with any syntax extensions you might need enabled).

For the remaining '64 bit libraries on 32 bit js_of_ocaml platform'
issue there is either the runtime solution I have provided on github
or the recompilation option suggested by Jeremie.

-Andy

[Sebastien Mondet]

On Mon, Apr 7, 2014 at 8:56 AM, Yaron Minsky <ymi...@janestreet.com> wrote:

We've actually had an internal discussion about exactly this issue.
We don't love having our own GC tuning, but when we removed it, we
found many performance regressions.  We really do think that they're
better for the vast majority of applications.

Is it a problem if those GC tunings come back?

Could you tell us which versions of Core(_kernel) have those GC tunnings? (Or at least where they are/were to check git history)

(we've had some GC-problems in Biocaml & related projects, and, to investigate, I'd like to know who's touching the GC parameters and when)

Thanks

[Jeremie Dimino]

On Fri, Apr 18, 2014 at 3:19 PM, Sebastien Mondet <sebastie...@gmail.com> wrote:

Could you tell us which versions of Core(_kernel) have those GC tunnings? (Or at least where they are/were to check git history)

All versions up to 109.60.00 have these settings. There were in the Core_gc module, which was in core up to 109.55.00 and in core_kernel since 109.58.00.

--
Jeremie