Mini-Project: C++11-ifying the Codebase

[florian wagner]

Hello,

I would like to help with this mini project. I am used to writing C++ since I specialise in Computer Graphics and our department really likes C++11. As such I would like to take on the challenge of C++11-ifying Extempores Code.
Are there any good (isolated) pieces of code to get started in? I have already forked the code at https://github.com/Beta-Alf/extempore and will dig into it over the next few days.

Greetings,
Florian

[Andrew Sorensen]

Hi Florian,

That sounds great!

A significant contribution would be to remove our current dependence on boost (for windows builds). Particularly threading and networking.

I would suggest you start by looking at SchemeProcess.cpp.

Let us know how you get on, and give us a shout if you get stuck. There is some fuggly code in there but it's also battle hardened.

Cheers
Andrew.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[florian wagner]

Hi Andrew,

I poked around a little in the code, it might take me some time to get behind some things, but it should be okay.
The boost::thread could probably be replaced with vanilla c++11 threads now that they are in.
Networking might be more complex, since C++ does not bring anything out of the box for that, but I will have a look at some other libraries.

Since I am not all that well versed in both topics I will start with the threads, since they seem the easier thing to replace and I want to get used to the codebase.

Greetings
Florian

[Ben Swift]

Hi florian

Yeah, std::thread should be pretty much a drop-in replacement for the
boost threading stuff in most cases. In terms of the networking stuff,
there's another cross-platform networking library that I'm eyeing off at
the moment, so we should let each other know if we have any
breakthroughs.

Cheers,
Ben

[florian wagner]

Hey,

I just went through the codebase in search of the usages of boost thread and somebody seems to have done this work already :)

#ifdef EXT_BOOST
    // std::function<void*(void*)> fn = static_cast<std::function<void*(void*)> >(start_routine);
    std::function<void*()> fn = [start_routine,arg]()->void* { return start_routine(arg); };
    bthread = std::thread(fn);
    result = 0;
#else

I could not locate a single instance for boost::thread or even an include pointing to it, so all the better?
What other Library are you eyeballing off of and what are the basic demands that a library needs to meet?
I think the SFML has a really slim implementation of TCP and UDP, but I am not sure, whether it will provide all needed features.

Greetings
Florian

[Andrew Sorensen]

Thanks for the update Florian,

Yes, Ben made a start on swapping out Boost a while back, but there is still a lot to do.  

For threading, I was particularly thinking of EXTLLVM which has a bunch of WIN32 defines that are currently not implemented (thread_create, thread_fork, thread_join etc.).  I apologise for the misdirect, I had forgotten that EXTLLVM was using a WIN32 instead of a BOOST directive.  Ideally we would extend EXTThread where necessary and use it in EXTLLVM for all targets - which would get rid of some unnecessary code duplication.  The same is true for EXTMutex and EXTMonitor.

A *small* standalone networking library with a liberal licence is what we are after.  Something that we could potentially inline into the project.  Let me know if you find any likely candidates. 

Ben are you talking about nanomsg or did you have something else in mind?

 

Cheers,

Andrew.

--
You received this message because you are subscribed to the Google Groups "Extempore" group.
To unsubscribe from this group and stop receiving emails from it, send an email to extemporelan...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ross Bencina]

On 22/11/2015 11:38 AM, Andrew Sorensen wrote:
> A *small* standalone networking library with a liberal licence is what
> we are after. Something that we could potentially inline into the
> project. Let me know if you find any likely candidates.

Aside from "small", what are the requirements? UDP, TCP? event loop?

I've used libuv and will use it again.

oscpack has lightweight UDP networking that's widely used and hence
pretty well tested (e.g. in open frameworks).

Ross.

[Andrew Sorensen]

TCP and UDP.  I think nanomsg is probably the way we'll go, although I did just take a look at Iibuv which looks nice. Thanks for the heads up, I'll have a play with libuv sometime later this week.

[Ben Swift]

Yeah, I was talking about nanomsg (sorry, my reply to florian was
off-list), but I'm still not 100% sure it's what we're after. Need to
investigate more.

Cheers,
Ben

[florian wagner]

Hey,

so after the new pointers from Andrew I plan to use EXTThread in EXTLLVM.
To remove some more #ifdefs the EXTThread could solely depend on std::thread, since that encapsulates pthread on unix platforms.
Since the branches should be independent, I will cook up two branches you can decide whether you like it.
However I am currently facing two Problems:
1) In EXTLLVM there is a thread_cancel function. This is not provided by std::threads since nothing about the state can be guaranteed. Is this function actually called from xtlang or is it just there as a provision?
2) Under Windows the LLVM Download seems to have a strange hash. I get told:

1>      C:/Users/Flonk/Projekte/extempore/build/llvm/src/extempore-llvm-3.7.0.src.zip
1>  
1>    does not match expected value
1>  
1>      expected: 4a2eafb937becec468ba985fca47532b
1>        actual: a2fb39e154f62c1731d32f0c789d8f99

I can work around that of course, but it might be worth looking into.

Greetings
Florian

[Andrew Sorensen]

Hi Florian,

If you could work around thread_kill for the moment (leave the existing code), that would be good.  Will take a look at the zip hash.

Cheers,

Andrew.

[Ben Swift]

Hi guys

The zip hash is for an old zipball (which didn't have all the required
stuff in it). I fixed the problem a while ago but forgot to change the
md5 over - my bad. Will fix it shortly.

Cheers,
Ben

Andrew Sorensen <dig...@gmail.com> writes:

[florian wagner]

Hello everyone,

I finally got extempore to compile on my windows machine and god bless you for the plan to get rid of boost.
Compiling however is only half the battle. I run into a hard crash when trying to run the program. It seems to be an error in the compilation of some early Scheme things, I get all the way up to

------------- Extempore --------------
Andrew Sorensen (c) 2010-2015
andrew@moso.com.au, @digego

ARCH           : i686-pc-windows-msvc-elf
LLVM           : 3.7.0 MCJIT
Output Device  : Primõrer Soundtreiber
Input Device   :
SampleRate     : 44100
Channels Out   : 2
Channels In    : 0
Frames         : 128
Latency        : 0.24 sec
---------------------------------------
Allocated: 1 Cell Segements For A Total Of 0,  Free Cells = 0

Starting utility process
Trying to connect to 'localhost' on port 7098
New client connection
Successfully connected to remote process
Allocated: 1 Cell Segements For A Total Of 0,  Free Cells = 0

Starting primary process
Trying to connect to 'localhost' on port 7099
New client connection
Successfully connected to remote process
before
SetValue:  NaNf >>> float
SetValue:  NaNd >>> double

And my call stack when the error occurs looks like this

>    extempore.exe!std::basic_string<char,std::char_traits<char>,std::allocator<char> >::_Tidy(bool _Built, unsigned int _Newsize) Line 2275    C++
     extempore.exe!std::basic_string<char,std::char_traits<char>,std::allocator<char> >::operator=(std::basic_string<char,std::char_traits<char>,std::allocator<char> > && _Right) Line 908    C++
     extempore.exe!llvm::Module::setTargetTriple(llvm::StringRef T) Line 294    C++
     extempore.exe!extemp::SchemeFFI::jitCompileIRString(scheme * _sc, cell * args) Line 1713    C++
     extempore.exe!opexe_0(scheme * sc, scheme_opcodes op) Line 4049    C++
     extempore.exe!Eval_Cycle(scheme * sc, scheme_opcodes op) Line 6209    C++
     extempore.exe!scheme_load_string(scheme * sc, const char * cmd, unsigned __int64 start_time, unsigned __int64 end_time) Line 6676    C++
     extempore.exe!extemp::SchemeProcess::impromptu_task_executer(void * obj_p) Line 648    C++
     extempore.exe!extemp::EXTThread::create::__l6::<lambda>() Line 77    C++
     [External Code]    

I am using MSVC 2013 Ultimate and had to fiddle a little to get this to compile. Please tell me I don't have to wait through the hours it takes to install 2015.

Greetings
Florian

[Ben Swift]

Hi Florian

Thanks for your work on this. Glad you got it to build.

Unfortunately, I have some bad news: Extempore (inc. LLVM) is 64-bit
only at the moment[0]. From your screencap below, it looks like you've
compiled LLVM at least as 32-bit (ARCH : i686-pc-windows-msvc-elf). That
would certainly account for the strange crash you're seeing. Are you on
a 32-bit or 64-bit Windows? If the latter, it might just be a matter of
rebuilding Extempore (and LLVM) and making sure you're targeting x64 in
all cases (INSTALL.md contains the necessary CMake invocations to make
this happen). If you're on 32-bit Windows, then there's no easy fix (see
[0]).

Sorry to be the bearer of bad tidings.

Cheers,
Ben

[0] it is possible, although probably nontrivial to get it to run on
32-bit platforms. Mostly it would involve changing a few long ints to
size_t and other things like that, but it would be fiddly. Maybe there
are some compiler warnings which could help.

florian wagner <floria...@googlemail.com> writes:

> Hello everyone,
>
> I finally got extempore to compile on my windows machine and god bless
> you for the plan to get rid of boost.
> Compiling however is only half the battle. I run into a hard crash
> when trying to run the program. It seems to be an error in the
> compilation of some early Scheme things, I get all the way up to
> ------------- Extempore --------------
> Andrew Sorensen (c) 2010-2015

> and...@moso.com.au, @digego

[florian wagner]

Hi Ben,

since I am on a 64bit machine, I only needed to recompile everything, including LLVM.
But now I will be able to dig into the code.
Thanks to you and Andrew for walking me through this and updating the checksum.

Greetings
Florian

[florian wagner]

Hello again ;)

I have a first draft of using EXTThread in EXTLLVM. It can be found under https://github.com/Beta-Alf/extempore/tree/EXTThread_EXTLLVM
It builds under windows, so yeah, not tested at all. Tomorrow I will build under Linux and if somebody could give me a bit of xtlang to test threads then I could do that, too.

Greetings
Florian

[Andrew Sorensen]

Hi Florian,

Great, thanks Florian.  An xtlang test case to try is examples/core/xthread.xtm

Let us know when Linux is up and running and I'll take a look at the code.

Cheers,

Andrew.

​

[Ben Swift]

Yeah, thanks Florian, happy to help out.

I'm super pumped that you're trying things on Windows - the more testing
we do there the better.

Cheers,
Ben

Andrew Sorensen <dig...@gmail.com> writes:

[florian wagner]

Hey,

so I tried doing a clean build (of my branch) under linux and something with building LLVM failed. I will look into that once I get the time to do so.
My weekend is quite full, so new results will probably be in at the end of next week or whenever I find time to fix whatever I broke (probably not code but something with my machine).

Greetings
Florian

[florian wagner]

Hey,

so I finally got around to testing and from my end it seems to be working under Windows and Linux (at least all the examples in xthread.xtm do).
Since I am quite new to the codebase I am not quite sure what else I could have broken in the process, but I think that I did only small and almost equivalent transformations to a more readable and maintainable state.
Time for you guys to try it out: I created a pull request so you can test, whether it works for you.

Greetings
Florian

[Ben Swift]

Great, thanks mate. I'll have a look at it as soon as I can - probably
next week.

Cheers,
Ben

florian wagner <floria...@googlemail.com> writes: