memcpy and sizeof

[Cian O'Connor]

Is there an equivalent of sizeof in xtlang?

So I'm creating a dynamic array (e.g. Vector, or ArrayList), where the data lives on the heap.

I have a datatype:

(bind-type ArrayL <!a*,i64,i64> (printer? . #f))

I create the type in the following fashion (the other elements in the tuple are length and capacity):

(bind-func ArrayL:[ArrayL{!a}*,i64]*
  (lambda (cap)
    (let ((data:!a* (halloc cap)))
      (ArrayI_z data 0 cap))))

I now want to create the function that resizes the data array:

(bind-func resize_data:[void,ArrayL{!a}*,i64]*

  (lambda (arr ncap)

    (let ((nloc:!a* (halloc ncap)))

      (memcpy (cast nloc) (cast (tref arr 0)) ncap)

      (free (tref arr 0))

      (tset! arr 0 nloc)

      (tset! arr 2 ncap)

      void)))

But this won't work because I need to pass the number of bytes to copy. What's the best way to achieve this? Obviously in C I'd use sizeof on the type, but I don't think that's available to me here.

Thanks,

Cian

[Andrew Sorensen]

Hi Cian,

In theory you can do this as xtlang includes a size_of macro.  However, in practice the generic case is not currently implemented.  Off the top of my head having size_of support generic types *should* be an easy change.  Please raise a bug against size_of not supporting generic types.

Let me show you how your example *does work* when you provide a specialization - i.e. non generic type.  The 'generic problem' being that where (size_of TYPE), TYPE must currently be fully specialized.

Assuming generic ArrayL as defined in your example ...  then we need to explicitly provide specialized variants of resize_data.  Clearly you want a generic option, but I'm showing you this by way of 'documentation' :) 

(bind-func resize_data:[void,ArrayL{i64}*,i64]*

  (lambda (arr ncap)

    (let ((nloc:i8* (halloc (* ncap (size_of ArrayL{i64})))))

      (memcpy (cast nloc) (cast (tref arr 0)) ncap)

      (free (tref arr 0))

      (tset! arr 0 (cast nloc))

      (tset! arr 2 ncap)

      void)))

(bind-func resize_data:[void,ArrayL{float}*,i64]*

  (lambda (arr ncap)

    (let ((nloc:i8* (halloc (* ncap (size_of ArrayL{float})))))

      (memcpy (cast nloc) (cast (tref arr 0)) ncap)

      (free (tref arr 0))

      (tset! arr 0 (cast nloc))

      (tset! arr 2 ncap)

      void)))

(bind-type MyType <i16,i64,|4,float|,double>)

(bind-func resize_data:[void,ArrayL{MyType}*,i64]*

  (lambda (arr ncap)

    (let ((nloc:i8* (halloc (* ncap (size_of ArrayL{MyType})))))

      (memcpy (cast nloc) (cast (tref arr 0)) ncap)

      (free (tref arr 0))

      (tset! arr 0 (cast nloc))

      (tset! arr 2 ncap)

      void)))

Cheers,

Andrew.

p.s. incidentally how did you get on with zone cleanup as per Toby's email?

--
You received this message because you are subscribed to the Google Groups "Extempore" group.
To unsubscribe from this group and stop receiving emails from it, send an email to extemporelang+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Cian O'Connor]

Hi Andrew,

I raised a bug for it. And the example was super helpful, thanks. I'll make sure that ends up in the new documentation.

zone_cleanup seems to be working for me so far. For zones I think it's great as it feels way more lispy than something like destructors. Plus you can use them as destructors if necessary. For the Dynamic Array code I'm working on at the moment zone_cleanup allows me to allocate on the heap, but scope using the zone in which the DynamicArray was created (please excuse prototype code):

;; <pointer to memory, length of array, capacity of storage>
(bind-type ArrayI <i16*,i64,i64> (printer? . #f))


;; Destructor for the Dynamic Array. Cleans up the heap memory.
(bind-func ArrayI_zhook:[void,ArrayI*]*
  (lambda (arr)
    (free (tref arr 0))))


(bind-func ArrayI:[ArrayI*,i64]*
  "
Create an ArrayList.
@param initial capacity for the ArrayList.
"
  (lambda (cap)
    (let ((data:i16* (halloc cap))
          (zone (peek_zone))
          (arr (ArrayI_z data 0 cap)))
      (zone_cleanup (ArrayI_zhook arr))
      arr)))

This feels like a useful pattern for managing heap allocated data.

It would be nice to have something similar to zone_cleanup for stack allocated objects (e.g. for closing sockets, file refs, etc). For heap objects that wouldn't make a lot of sense, but maybe if there was a way to overload the 'free' function then that would be sufficient? zone_cleanup does cover 90% of the cases though.

To unsubscribe from this group and stop receiving emails from it, send an email to extemporelan...@googlegroups.com.

[Cian O'Connor]

Incidentally is there any reason why this doesn't work?

(bind-type TestT <i64,mzone*>)

I get the following error:

Compiler Error cannot find symbol hcopy:[mzone*,mzone*]

[Toby Gifford]

That's how I ended up doing things too, i.e. explicitly code destructors for custom types, and call them in zone_cleanup from the zone they are allocated in.

With stack allocated objects, can you not just explicitly free resources in the function itself?  The reason I wanted to have a zone_cleanup was that there didn't seem to be any other way of freeing file resources etc allocated inside a closure when that closure was recompiled (which automatically free's the closure's top zone, but otherwise leaves any other resources hanging without handles to close/free/squelch). But I can't think of a circumstance where you would need to do this for something allocated on the stack.

+1 for your question on mzone. I often get that "cannot find symbol hcopy..." error, and not sure why. Actually, a great addition to the documentation would be expanding the memory management explanation with some API references for the memory management functions. Off the top of my head I think there are alloc, salloc, zalloc, halloc, memzone which are covered in the existing docs, but also let, lets, letz, leth, and then a bunch of memzone management functions for pushing and popping the zone stack, destroying zones etc, whose names I can't remember.  Does memzone return a handle to the new zone, which can be passed into these functions? I'm happy to update the docs if someone can point me to the appropriate functions. I've been thinking for a while of some way to visualise what zone a symbol is pointing to, to try to reign in the segfault bonanzas I often have.

 

To unsubscribe from this group and stop receiving emails from it, send an email to extemporelang+unsubscribe@googlegroups.com.

[Cian O'Connor]

With stack allocated objects, can you not just explicitly free resources in the function itself?  The reason I wanted to have a zone_cleanup was that there didn't seem to be any other way of freeing file resources etc allocated inside a closure when that closure was recompiled (which automatically free's the closure's top zone, but otherwise leaves any other resources hanging without handles to close/free/squelch). But I can't think of a circumstance where you would need to do this for something allocated on the stack.

It's certainly not necessary. It's more a useful convenience/abstraction if the data object that owns the file resources is responsible for deleting them before going out of scope. If you're writing a library its less 'knowledge' required by the users of that library and in general means that your abstraction is 'less leaky'. There's not a lot that I like in C++, but I think destructors are a useful thing. Very useful if you have exceptions of course.

Similarly for heap allocated objects you have to write a function 'cleanup' and users of your api have to know to do the following (begin (cleanup obj) (free obj)) - you can certainly work around it, but it lacks elegance and probably means more bugs over the long run.

Actually, a great addition to the documentation would be expanding the memory management explanation with some API references for the memory management functions. 

Look here:

https://github.com/extemporelang/extemporelang.github.io/tree/master/_docs/reference/new

It's very pre-alpha, but a lot of what you're looking for is documented and is probably accurate... Incidentally feel free to raise an issue/pull request if there's something missing.

Off the top of my head I think there are alloc, salloc, zalloc, halloc, memzone which are covered in the existing docs, but also let, lets, letz, leth, and then a bunch of memzone management functions for pushing and popping the zone stack, destroying zones etc, whose names I can't remember.  

most of these I've documented. lets and leth are new to me. So if anyone wants to tell me what they do I'll add some documentation on them. I'm guessing something to do with the stack and heap?

Does memzone return a handle to the new zone, which can be passed into these functions? I'm happy to update the docs if someone can point me to the appropriate functions. I've been thinking for a while of some way to visualise what zone a symbol is pointing to, to try to reign in the segfault bonanzas I often have.

It does. I'm actually in the very early stages of writing some documentation on memory management practices (which currently is documenting stuff I've learnt the hard way while writing prelude functions). Being able to reference a memory zone in a data-type would help reduce segfaults I think, as then the ownership/scope is easy to control. But I'm trying to think of other ways to easily enforce ownership/scope rules for memory zones. Letz is also useful as it creates a new zone which is deleted once you leave it's scope, and handles copying the returned object to the scope one level above you.

I also think once we have some basic data types for arrays, hashtables, linked lists (or more specifically VLists), etc that should help as well. I'm working on it...