attempt tom make vim threadsafe - concept - what's wrong with it?
Marc Weber
http://www.mail-archive.com/vim...@googlegroups.com/msg02355.html
My tovl lib also contains a client/server workaround to get the same result but that works only when X is availible.
The idea is simple: find the event loops within vim. It should be safe to execute any code there.
release a global lock at those points which python, ruby threads can acquire to
run vim.command(..) commands.
Source the test_case.vim file, has('python') is required. vim should crash instantly.
If it doesn't try again or type some characters.
But there is something I don't understand: In main I vim get's the lock However
the select timeout should only occur when running vim. Even when never
releasing the lock everything works fine. So I conclude that the vim.command
runs within the same thread as vim itself? What causes the crash in standard
vim then?
To make playing around with this change easier I uploaded all the code
to git://mawercer.de/vim (branch t/locking) (You may push your ideas and
modifications to this repo as well)
In the python case it's getting more interesting:
The code calling back into vim looks like this:
Py_BEGIN_ALLOW_THREADS
Python_Lock_Vim();
our_tv = eval_expr((char_u *)expr, NULL);
Python_Release_Vim();
Py_END_ALLOW_THREADS
Py_BEGIN_ALLOW_THREADS tells python that no python objects are used, thus
python will start true multithreading only here because python will always
run only one thread at a time because python isn't thread safe. If it run
multiple threads even python object reference counting may fail?
A simple python app only running two python threads shows that the sum of both
cores is approx 100%. So this seems to be true.
I got this from here http://www.python.org/doc/1.5.2/api/threads.html
Again: Why does vim crash at all then when not implementing Python_{Lock,Release}_Vim
as I've done?
Same seems to apply to ruby. It can't utilize two cores as either.
I din't compile vim to see whether it crashes yet though.
Of course this patch should be extended and cleaned up before distributing it.
A windows implementation should be straight forward. However I'd be glad you
shedding some light into the dark and telling my why vim crashes at all without
patch but doesn't have to get the lock to run the vim.command(..) commands?
Anyway: How do you think about it? By adding approximately 100 lines of code we
can make vim thread safe.. or safe enough to use those features.
We'll gain the ability to run processes in background.
That's fine for make, grep, mkid, ctags and so on. You can continue coding
while running those apps out of vim.
Also additional processes are required to talk to foreign processes such as gdb, pydb,
xdebug and so on. I can now even think about replacing irssi by vim or writing a
new Pidgin frontend .. Why ? Because editing and browsing text in vim is what I
like most.
Marc Weber
========================= test_case.vim ===============================================
" source this file
" g:init_count will report number of total started threads
" g:counter holds the amount of running threads
" this value is written to g:list for debugging purposes (to verify that
" multiple threads are running at the same time)
" without the patch vim should instantly crash !
fun! Test()
let g:counter = 0
let g:init_count = 0
let g:list = []
py << EOF
from subprocess import Popen, PIPE
import threading
import os
import vim
print "python start"
from time import sleep
class MyThread ( threading.Thread ):
def __init__(self, start, command):
vim.command("let g:init_count = g:init_count +1")
threading.Thread.__init__(self)
self.command=command
def run ( self ):
sleep(3)
vim.command("let g:run_at_least_once=1")
vim.command("let g:counter = g:counter +1")
for i in range(1,200):
vim.command(self.command)
vim.command("let g:counter = g:counter -1")
vim.command("let g:started=0")
print "python thread started"
threads = []
EOF
for i in range(1,200)
py << EOF
threads.append(MyThread("let g:dummy=10","call add(g:list, g:counter)"))
threads[-1].start()
vim.command("let g:started = g:started+1")
EOF
echo g:counter
endfor
endfun
call Test()
========================= PATH against git://repo.or.cz/vim_mainline.git 24bc3a663
diff --git a/src/config.h.in b/src/config.h.in
index b603c23..a500e93 100644
--- a/src/config.h.in
+++ b/src/config.h.in
@@ -311,6 +311,10 @@
/* Define if you want to include the Sniff interface. */
#undef FEAT_SNIFF
+/* Define if you want to add additional locking so that you
+ * can use threads in ruby, python, perl safely */
+#undef FEAT_SCRIPTTHREADING
+
/* Define if you want to add support for ACL */
#undef HAVE_POSIX_ACL
#undef HAVE_SOLARIS_ACL
diff --git a/src/configure.in b/src/configure.in
index eb7db76..4c74c66 100644
--- a/src/configure.in
+++ b/src/configure.in
@@ -1098,6 +1098,15 @@ if test "$enable_multibyte" = "yes"; then
AC_DEFINE(FEAT_MBYTE)
fi
+AC_MSG_CHECKING(--enable-scriptthreading argument)
+AC_ARG_ENABLE(scriptthreading,
+ [ --enable-scriptthreading add locking allowing threads in scripts.], ,
+ [enable_scriptthreading="no"])
+AC_MSG_RESULT($enable_scriptthreading)
+if test "$enable_scriptthreading" = "yes"; then
+ AC_DEFINE(FEAT_SCRIPTTHREADING)
+fi
+
AC_MSG_CHECKING(--enable-hangulinput argument)
AC_ARG_ENABLE(hangulinput,
[ --enable-hangulinput Include Hangul input support.], ,
diff --git a/src/globals.h b/src/globals.h
index 8f373f6..05c4c63 100644
--- a/src/globals.h
+++ b/src/globals.h
@@ -1587,3 +1587,23 @@ EXTERN char *ignoredp;
#ifdef FEAT_ARABIC
# include "arabic.h"
#endif
+
+
+#ifdef FEAT_SCRIPTTHREADING
+
+#include "pthread.h"
+// TODO: move this into its own file (its still a proof of concept)
+
+EXTERN pthread_mutex_t global_mutex INIT(= PTHREAD_MUTEX_INITIALIZER);
+
+ /* give a thread a chance to aquire the lock
+ * This function is called when select times out (vim) and
+ * in XtAppAddWorkProc idle funcction (gvim)
+ * TODO: nicer description
+ */
+
+#define GLOBAL_LOCK pthread_mutex_lock(&global_mutex);
+#define GLOBAL_UNLOCK pthread_mutex_unlock(&global_mutex);
+#define RELEASE_GLOBAL_LOCK pthread_mutex_unlock(&global_mutex); sleep(0); pthread_mutex_lock(&global_mutex);
+
+#endif
diff --git a/src/if_python.c b/src/if_python.c
index e483bfc..ae9caa4 100644
--- a/src/if_python.c
+++ b/src/if_python.c
@@ -462,6 +462,9 @@ Python_RestoreThread(void)
*/
static void Python_Lock_Vim(void)
{
+#ifdef FEAT_SCRIPTTHREADING
+ GLOBAL_LOCK
+#endif
}
/*
@@ -469,6 +472,9 @@ static void Python_Lock_Vim(void)
*/
static void Python_Release_Vim(void)
{
+#ifdef FEAT_SCRIPTTHREADING
+ GLOBAL_UNLOCK
+#endif
}
void
diff --git a/src/main.c b/src/main.c
index 84aa146..8a78fb6 100644
--- a/src/main.c
+++ b/src/main.c
@@ -170,6 +170,12 @@ main
int argc;
char **argv;
{
+
+
+#ifdef FEAT_SCRIPTTHREADING
+ GLOBAL_LOCK // I even couldn't crash vim without taking this global lock here after adding locking to if_pyhton.c ..
+#endif
+
char_u *fname = NULL; /* file name from command line */
mparm_T params; /* various parameters passed between
* main() and other functions. */
diff --git a/src/os_unix.c b/src/os_unix.c
index 3aa397b..bf63537 100644
--- a/src/os_unix.c
+++ b/src/os_unix.c
@@ -152,6 +152,7 @@ static char_u *oldicon = NULL;
static int did_set_icon = FALSE;
#endif
+
static void may_core_dump __ARGS((void));
static int WaitForChar __ARGS((long));
@@ -5010,7 +5011,24 @@ RealWaitForChar(fd, msec, check_for_gpm)
* required. Should not be used */
ret = 0;
# else
+#ifdef FEAT_SCRIPTTHREADING
+ if (tvp == NULL) {
+ // only let threads execute some code when there is no timeout
+ // is the timeout used for multi key mappings ?
+ tv.tv_sec = 0;
+ tv.tv_usec = (1) * (1000000/1000); // 1 ms
+ tvp = &tv;
+ while (1){
+ ret = select(maxfd + 1, &rfds, NULL, &efds, tvp);
+ if (ret) break; // no timeout
+ RELEASE_GLOBAL_LOCK
+ }
+ } else {
+ ret = select(maxfd + 1, &rfds, NULL, &efds, tvp);
+ }
+#else
ret = select(maxfd + 1, &rfds, NULL, &efds, tvp);
+#endif
# endif
# ifdef __TANDEM
if (ret == -1 && errno == ENOTSUP)
diff --git a/src/vimrun.c b/src/vimrun.c
index c423e6c..da7f22e 100644
--- a/src/vimrun.c
+++ b/src/vimrun.c
@@ -65,6 +65,11 @@ main(void)
p = _acmdln;
# endif
#endif
+#ifdef FEAT_SCRIPTTHREADING
+ global_mutex = PTHREAD_MUTEX_INITIALIZER;
+ GLOBAL_LOCK
+#endif
+
/*
* Skip the executable name, which might be in "".
*/
Marc Weber
Each time the same pid is printed! So I do no longer wonder why I don't
have to release the lock. However I still wonder why vim does crash
without the patch because the code is run from within the same thread
anyway?
Any ideas ?
Marc Weber
Tony Mechelynck
IIUC, that doesn't mean anything: the PID is a process ID, not a thread
ID. Even if you run several threads in parallel, if it's the same
instance of the executable it's the same process.
This said, IIUC Vim is basically a single-threaded application. Threads
can (I think) only happen due to interfaces with threaded interpreters.
The src/Makefile includes the following comment, which seems to be
(partially) outdated:
> # PYTHON
> # Uncomment this when you want to include the Python interface.
> # NOTE: This may cause threading to be enabled, which has side effects (such
> # as using different libraries and debugging becomes more difficult).
> # NOTE: Using this together with Perl may cause a crash in initialization.
> #CONF_OPT_PYTHON = --enable-pythoninterp
I suppose you looked them up too too, but uses of symbols _REENTRANT
(which is defined on my gcc command-line as shown by :version) and
_THREAD_SAFE (both in os_unix.c and once together in
/usr/include/features.h) look like they might be interesting.
Best regards,
Tony.
--
Help! I'm trapped in a PDP 11/70!
Marc Weber
> This said, IIUC Vim is basically a single-threaded application. Threads
> can (I think) only happen due to interfaces with threaded interpreters.
This explains a lot.
Yes. I'd like to add threading support. for Python it does already work
very well (At least vim doesn't crash)..
I also checked ruby. However I don't know why ruby doesn't call back
into vim betwenn :ruby calls. So there is no problem at all.
However I'd like ruby being able to tell vim sth. without waiting for a
:ruby command.
Marc Weber
Tony Mechelynck
There are also :rubydo and :rubyfile. How are the various functions,
objects and variables of the VIM module implemented? (see ":help
ruby-vim" and below)
Best regards,
Tony.
--
hundred-and-one symptoms of being an internet addict:
81. At social functions you introduce your husband as "my domain server."
Marc Weber
>
> On 18/07/09 17:21, Marc Weber wrote:
> >
> > Excerpts from Tony Mechelynck's message of Sat Jul 18 16:01:20 +0200 2009:
> >> This said, IIUC Vim is basically a single-threaded application. Threads
> >> can (I think) only happen due to interfaces with threaded interpreters.
> > Ah thanks, I'll try to add pthread_getthreadid_np to verify this.
I used pthread_self() to get some more debugging information.
current updated patch:
http://www.mawercer.de/~marc/threadsafe.patch
(or get git://mawercer.de/vim branch t/locking)
Status:
Console: no crash
Gtk: my simple counter [1] works fine. However running my heavy test [2]
causes some infinite locking somewhere.
Athena: Seems to work as fine as console (?)
*: I didn't test nor payed attention
I'm sure I've missed many race conditions etc.. Implementing this
properly would take much more time . So I reached my goal: Being able
to run grep or make like commands in background using console vim.
Maybe I would have had more sucess adding kind of messaging queue. So that
vim.command or vim.eval adds a request to a list which is run from
within the main thread and event loop? Then there would have been no
threading problems. If you say you're interested in this topic I may
implement and try it. Also I'd have more success if someone guides me here.
Thank you for paying attention.
Marc Weber
[1] counter test:
py << EOF
from subprocess import Popen, PIPE
import threading
import os
import vim
print "python start"
from time import sleep
class MyThread ( threading.Thread ):
def run ( self ):
counter = 0
while 1:
counter += 1
vim.command("echo %d" % counter)
sleep(1)
thread = MyThread()
thread.start()
EOF
[2] heavy test:
fun! Test()
py << EOF
from subprocess import Popen, PIPE
import threading
import os
import vim
print "python start"
from time import sleep
class MyThread ( threading.Thread ):
def __init__(self, start, command):
vim.command(start)
threading.Thread.__init__(self)
self.command=command
def run ( self ):
sleep(3)
for i in range(1,200):
vim.command(self.command)
vim.command("let g:started=0")
for i in range(1,200):
thread=MyThread("let g:start=1 | echo \"starting\"", "let g:done=1| echo \"succ\"")
thread.start()
vim.command("let g:started = g:started+1")
print "python thread started"
EOF
endfun
call Test()