midi package on windows

[et4]

Has anyone been able to build the midi package for windows, found at,

http://www.uttis.de/midi/index.htm

This was built was for tcl 8.3 and so the binary package no longer works. The wiki page

https://wiki.tcl-lang.org/page/MIDI

mentions the problem but the solutions aren't working for me. The source zip is a vis c++ 6.0 project.

I tried to build it using the free 2022 visual studio with no success. I installed an old vis c++ 6.0 but it failed to build a usable .dll plus it's only 32 bit.

I have a spare laptop where I've installed Ashok's Magicsplat distro but it seems it must be 64 bit to work on 64 bit windows. This might be a nice addition to a future Magicsplat, which would likely build correctly under Visual Studio 2017.

-------


Midi controllers are fast being adapted for use beyond just music DAWs. Just about every controller now also works over usb.

Midi would be a great interface, perhaps built into Tk 9.0. Tk bindings to use midi knobs and faders could operate spinboxes, scrollbars, and notebook tabs. Drum pads could be bound like gui buttons plus they generally support RGB as output.

Perhaps 3d graphics programs might use the knobs for rotate, zoom, and other transformations. Since Tk is used in several other languages, this could be a rather nice addition to Tk in general. I'm considering writing up a TIP.

Here's a relatively low cost controller I've considered, by Behringer, the X-Touch Mini

https://www.amazon.com/gp/product/B012CSKTYY

[Harald Oehlmann]

Hi !
I have quickly slipped over the code. Internal TCL headers version 8.3
are used. I suppose, due to the use of the channels.
I saw no reason for it. The code should be changed a bit.

I don't want to dive in this adventure. I already crashed tdbc::MySQL by
trying to help.

Take care,
Harald

[et4]

Thanks for looking into it. I don't think I can get any further here either. It likely will need to be changed and built as a new 64 bit project using a modern version of visual c.

Had it worked, I was looking to use it along with tcl and twapi to inject commands into a text editor. It'd be like having 8 independent mousewheel's. No end of fun applications.

thanks again.

[Ralf Fassel]

* Harald Oehlmann <wort...@yahoo.com>

| Am 17.12.2022 um 23:46 schrieb et4:
| > Has anyone been able to build the midi package for windows, found at,
| > http://www.uttis.de/midi/index.htm

--<snip-snip>--

| I have quickly slipped over the code. Internal TCL headers version 8.3
| are used. I suppose, due to the use of the channels.
| I saw no reason for it. The code should be changed a bit.

I have downloaded the src-package, the only file you need is midi.c.
Apply these changes to make it compile with tcl 8.6 (and probably 8.7 too):

diff -u midi.c\~ midi.c
--- midi.c~ 2022-12-19 15:43:46.054717000 +0100
+++ midi.c 2022-12-19 15:51:07.066234640 +0100
@@ -48,7 +48,7 @@

//#define USE_TCL_STUBS
#include "tcl.h"
-#include "tclerrno.h"
+// #include "tclerrno.h"

EXTERN void TclWinConvertError _ANSI_ARGS_((DWORD errCode));

@@ -813,7 +813,8 @@

DebugLog( "Rpmidi_InOpen #8\n" );
// Return channel name
- strcpy( interp->result, channelName );
+ // strcpy( interp->result, channelName );
+ Tcl_SetObjResult(interp, Tcl_NewStringObj(channelName, -1));
return TCL_OK;
}

@@ -1050,7 +1051,8 @@
DebugLog("Rpmidi_OutOpen #4\n");

// Return channel name
- strcpy( interp->result, channelName );
+ Tcl_SetObjResult(interp, Tcl_NewStringObj(channelName, -1));
+ // strcpy( interp->result, channelName );
return TCL_OK;
}


Diff finished. Mon Dec 19 16:09:42 2022

After that, compile it via

cl -o midi.dll midi.c -IPATH/TO/YOUR/TCL/INCLUDES /link -dll tcl86.lib Winmm.lib

I can load the resulting DLL just fine, but since I have no midi
interface attached, I can't test it.

HTH
R'

[Ralf Fassel]

* Ralf Fassel <ral...@gmx.de>

| After that, compile it via
>
| cl -o midi.dll midi.c -IPATH/TO/YOUR/TCL/INCLUDES /link -dll tcl86.lib Winmm.lib
>

Additional comment: there are *tons* of warnings relating to DWORD casts
on x64, which should at least be checked:

midi.c(666): warning C4312: "Typumwandlung": Konvertierung von "DWORD" in größeren Typ "Rpmidi_ChannelInstance *"
midi.c(667): warning C4312: "Typumwandlung": Konvertierung von "DWORD" in größeren Typ "MIDIHDR *"
midi.c(753): warning C4311: "Typumwandlung": Zeigerverkürzung von "void (__cdecl *)(HMIDIIN,UINT,DWORD,DWORD,DWORD)" zu "DWORD"
midi.c(753): warning C4311: "Typumwandlung": Zeigerverkürzung von "Rpmidi_ChannelInstance *" zu "DWORD"
midi.c(788): warning C4090: "Funktion": Unterschiedliche "const"-Qualifizierer
midi.c(788): warning C4028: Formaler Parameter "2" unterscheidet sich von der Deklaration
midi.c(1020): warning C4311: "Typumwandlung": Zeigerverkürzung von "void (__cdecl *)(HMIDIOUT,UINT,DWORD,DWORD,DWORD)" zu "DWORD"
midi.c(1020): warning C4311: "Typumwandlung": Zeigerverkürzung von "Rpmidi_ChannelInstance *" zu "DWORD"
midi.c(1031): warning C4090: "Funktion": Unterschiedliche "const"-Qualifizierer
midi.c(1031): warning C4028: Formaler Parameter "2" unterscheidet sich von der Deklaration
midi.c(1222): warning C4090: "Funktion": Unterschiedliche "const"-Qualifizierer
midi.c(1222): warning C4028: Formaler Parameter "4" unterscheidet sich von der Deklaration
[...]

Especially the truncating pointer-casts ("Zeigerverkürzung") will probably not work on x64.

HTH
R'

[et4]

Thanks for helping on this. I don't have cl or diff on windows, however I do have a linux VM where I created the files midi.c~ and patch.txt with \n line endings.

I am getting an error trying to apply the patch. Forgive my ignorance, as I just did a crash course on diff/patch on linux.

$ patch midi.c\~ <patch.txt
(Patch is indented 4 spaces.)
patching file midi.c~
Hunk #2 FAILED at 813.
Hunk #3 FAILED at 1050.
2 out of 3 hunks FAILED -- saving rejects to file midi.c~.rej


The reject file is this:

--- midi.c~ 2022-12-19 15:43:46.054717000 +0100
+++ midi.c 2022-12-19 15:51:07.066234640 +0100

@@ -813,7 +813,8 @@

DebugLog( "Rpmidi_InOpen #8\n" );
// Return channel name
- strcpy( interp->result, channelName );
+ // strcpy( interp->result, channelName );
+ Tcl_SetObjResult(interp, Tcl_NewStringObj(channelName, -1));
return TCL_OK;
}

@@ -1050,7 +1051,8 @@
DebugLog("Rpmidi_OutOpen #4\n");

// Return channel name
- strcpy( interp->result, channelName );
+ Tcl_SetObjResult(interp, Tcl_NewStringObj(channelName, -1));
+ // strcpy( interp->result, channelName );
return TCL_OK;
}

When I did the copy/paste of the patch (I use thunderbird here), I noticed that there are some tab chars that end up in the patch file. Could that be a problem here? These 2 rejected sections have tabs whereas the first section, that it didn't complain about doesn't have tabs.

Also, what C compiler are you using. I don't have one on my windows test system other than the 32 bit ms vis c++ 6.0. at present.

I do have a 64 bit magicsplat distro installed however, and I see where it has the include files.

[et4]

On 12/19/2022 2:32 PM, et4 wrote:
> On 12/19/2022 7:14 AM, Ralf Fassel wrote:

.. snip..

>
> Also, what C compiler are you using. I don't have one on my windows test system other than the 32 bit ms vis c++ 6.0. at present.
>
> I do have a 64 bit magicsplat distro installed however, and I see where it has the include files.
>

I have just installed the visual studio c++ community version on my test machine. I see it has a /link option when typing just cl to get a synopsis, so I'm guessing this is the right compiler.

I'll try to hand apply the patch if I can't figure out how to use patch.

Also, I do have a midi device (got it for my daughter who's the music person here) so I can test it out if I can get it built. And if I do get it working, I will put the .dll in a zip file on dropbox and post a link in the wiki page. Or is there a way to post binary files to the wiki?

As to the patch, I assumed that the file midi.c~ was the original, renamed from midi.c as a text editor backup. Does patch know to create the new version from the info in the patch file, or does it normally come out to stdout which one would redirect with say, >midi.c

thanks again.

[Christian Gollwitzer]

Am 20.12.22 um 04:31 schrieb et4:

> On 12/19/2022 2:32 PM, et4 wrote:
> I have just installed the visual studio c++ community version on my test
> machine. I see it has a /link option when typing just cl to get a
> synopsis, so I'm guessing this is the right compiler.
>
> I'll try to hand apply the patch if I can't figure out how to use patch.

Shouldn't be too difficult, Ralf basically changed only the returning of
a result back into Tcl, which uses an outdated mechanism, and removed
the include of tclerrno.h. patch would, in general, edit the file and
overwrite it with the new version. It may well be that there are
copy/paste errors with whitespace. I think it is simpler to apply the 3
small changes manually.

> Also, I do have a midi device (got it for my daughter who's the music
> person here) so I can test it out if I can get it built. And if I do get
> it working, I will put the .dll in a zip file on dropbox and post a link
> in the wiki page. Or is there a way to post binary files to the wiki?

The wiki is not a good place for binary data, better only provide links.
Nowadays one would typically upload software to github (sources and
binaries on the corresponding "release" page)

But be prepared that there are some refactoring changes necessary. The
code is not 64bit-safe. I've quickly looked into the warnings that Ralf
got during the compilation. The code passes pointers around as DWORDs
between the windows API and callbacks, which is 32 bit only. YOu may be
lucky that replacing some DWORD with DWORD_PTR is enough to get it
working, but there is no guarantee. In the worst case you need to adjust
the parameter passing.

Christian

[et4]

On 12/19/2022 11:10 PM, Christian Gollwitzer wrote:
> Am 20.12.22 um 04:31 schrieb et4:

...
> Shouldn't be too difficult, Ralf basically changed only the returning of a result back into Tcl, which uses an outdated mechanism, and removed the include of tclerrno.h. ....

Ok, think I got it figured out, -u includes 3 lines of context by default, and that was confusing me.

In the first "hunk" it is simply commenting out the #include "tclerrno.h"

And the 2nd and 3rd are commenting out a strcpy and adding a Tcl_SetObjResult

>
> The wiki is not a good place for binary data, better only provide links. Nowadays one would typically upload software to github (sources and binaries on the corresponding "release" page)

Ok, if I do get it working, I can then just put the code on the wiki with instructions for compiling to create the .dll since the comments say the code is free to use.

>
> But be prepared that there are some refactoring changes necessary. The code is not 64bit-safe. I've quickly looked into the warnings that Ralf got during the compilation. The code passes pointers around as DWORDs between the windows API and callbacks, which is 32 bit only. YOu may be lucky that replacing some DWORD with DWORD_PTR is enough to get it working, but there is no guarantee. In the worst case you need to adjust the parameter passing.

Hmmm, I guess I'll have to look at each error message and try to figure out what's wrong. If I'm lucky, the C compiler will complain until I fix them all.

But with the first error at line 666, this could be a sign :)

However, I do have a midi controller to try it out with.


>
> Christian
>


Thanks

et

[Ralf Fassel]

* et4 <tcl...@rocketship1.me>

| On 12/19/2022 11:10 PM, Christian Gollwitzer wrote:
| > Am 20.12.22 um 04:31 schrieb et4:
| ...
| > Shouldn't be too difficult, Ralf basically changed only the
| > returning of a result back into Tcl, which uses an outdated
| > mechanism, and removed the include of tclerrno.h. ....
>
| Ok, think I got it figured out, -u includes 3 lines of context by
| default, and that was confusing me.

Ah, I see. I figured that the 'diff' output might be easier to read
than trying to explain manually what needs to be changed, even if
someone applied the changes manually.

'cl' is the command line compiler provided by any Visual Studio release
I have worked with so far (starting approx in 2003). You should get it
if you open one of the "Visual Studio Command Prompt" entries from the
VisualStudio submenu in 'Programs'.

| > But be prepared that there are some refactoring changes
| > necessary. The code is not 64bit-safe. I've quickly looked into the
| > warnings that Ralf got during the compilation. The code passes
| > pointers around as DWORDs between the windows API and callbacks,
| > which is 32 bit only. YOu may be lucky that replacing some DWORD
| > with DWORD_PTR is enough to get it working, but there is no
| > guarantee. In the worst case you need to adjust the parameter
| > passing.

It might be possible to just remove the DWORD casts, but I haven't
looked at the MIDI function signatures whether this would be enough.

| Hmmm, I guess I'll have to look at each error message and try to
| figure out what's wrong. If I'm lucky, the C compiler will complain
| until I fix them all.
>
| But with the first error at line 666, this could be a sign :)

:-) Good luck!

R'

[et4]

Thanks. I will need a little more help with the command line and the tcl86.lib file, for my first try got this:

LINK : fatal error LNK1181: cannot open input file 'tcl86.lib'

So, I searched for the file but only found these 2 files

tcl86t.lib
tclstub86.lib

in Magicsplat's lib directory, so I tried both and also I gave it the full path

The last linker warning seems to suggest I'm not building a 64 bit version and the unresolve's cause a fatal error - wondering if I installed the right version of vis studio (the community 2022 version), do they have 32 and 64 bit versions ? They appear to think I'm building a 32 bit program.


Here's the run using the 't' library, using the stub version produced the same results


> cl -o midi.dll midi.c -IC:\Users\ET\AppData\Local\Apps\Tcl86\include /link -dll C:\Users\ET\AppData\Local\Apps\Tcl86\lib\tcl86t.lib Winmm.lib

Microsoft (R) C/C++ Optimizing Compiler Version 19.34.31937 for x86 <<<<<<<<<<<<<< wondering about this <<<<<<<<<<<<
Copyright (C) Microsoft Corporation. All rights reserved.

cl : Command line warning D9035 : option 'o' has been deprecated and will be removed in a future release
midi.c
midi.c(788): warning C4090: 'function': different 'const' qualifiers
midi.c(788): warning C4113: 'int (__cdecl *)(ClientData,char *,int,int *)' differs in parameter lists from 'Tcl_DriverOutputProc (__cdecl *)'
midi.c(1031): warning C4090: 'function': different 'const' qualifiers
midi.c(1031): warning C4113: 'int (__cdecl *)(ClientData,char *,int,int *)' differs in parameter lists from 'Tcl_DriverOutputProc (__cdecl *)'
midi.c(1222): warning C4090: 'function': different 'const' qualifiers
midi.c(1222): warning C4113: 'int (__cdecl *)(ClientData,Tcl_Interp *,int,char **)' differs in parameter lists from 'Tcl_CmdProc (__cdecl *)'
midi.c(1223): warning C4090: 'function': different 'const' qualifiers
midi.c(1223): warning C4113: 'int (__cdecl *)(ClientData,Tcl_Interp *,int,char **)' differs in parameter lists from 'Tcl_CmdProc (__cdecl *)'
midi.c(1224): warning C4090: 'function': different 'const' qualifiers
midi.c(1224): warning C4113: 'int (__cdecl *)(ClientData,Tcl_Interp *,int,char **)' differs in parameter lists from 'Tcl_CmdProc (__cdecl *)'
midi.c(1225): warning C4090: 'function': different 'const' qualifiers
midi.c(1225): warning C4113: 'int (__cdecl *)(ClientData,Tcl_Interp *,int,char **)' differs in parameter lists from 'Tcl_CmdProc (__cdecl *)'
midi.c(1226): warning C4090: 'function': different 'const' qualifiers
midi.c(1226): warning C4113: 'int (__cdecl *)(ClientData,Tcl_Interp *,int,char **)' differs in parameter lists from 'Tcl_CmdProc (__cdecl *)'
midi.c(1227): warning C4090: 'function': different 'const' qualifiers
midi.c(1227): warning C4113: 'int (__cdecl *)(ClientData,Tcl_Interp *,int,char **)' differs in parameter lists from 'Tcl_CmdProc (__cdecl *)'
midi.c(1228): warning C4090: 'function': different 'const' qualifiers
midi.c(1228): warning C4113: 'int (__cdecl *)(ClientData,Tcl_Interp *,int,char **)' differs in parameter lists from 'Tcl_CmdProc (__cdecl *)'
Microsoft (R) Incremental Linker Version 14.34.31937.0
Copyright (C) Microsoft Corporation. All rights reserved.

/out:midi.exe
/out:midi.dll
-dll
C:\Users\ET\AppData\Local\Apps\Tcl86\lib\tcl86t.lib
Winmm.lib
midi.obj
Creating library midi.lib and object midi.exp
midi.obj : error LNK2019: unresolved external symbol __imp__Tcl_PkgProvideEx referenced in function _Midi_Init
midi.obj : error LNK2019: unresolved external symbol __imp__Tcl_Alloc referenced in function _GetMidiErrorMsg
midi.obj : error LNK2019: unresolved external symbol __imp__Tcl_Free referenced in function _Rpmidi_InClose
midi.obj : error LNK2019: unresolved external symbol __imp__Tcl_GetInt referenced in function _OpenOut_Cmd
midi.obj : error LNK2019: unresolved external symbol __imp__Tcl_NewIntObj referenced in function _Rpmidi_Error
midi.obj : error LNK2019: unresolved external symbol __imp__Tcl_NewStringObj referenced in function _Rpmidi_InOpen
midi.obj : error LNK2019: unresolved external symbol __imp__Tcl_AppendElement referenced in function _GetOutDevs_Cmd
midi.obj : error LNK2019: unresolved external symbol __imp__Tcl_CreateChannel referenced in function _Rpmidi_InOpen
midi.obj : error LNK2019: unresolved external symbol __imp__Tcl_CreateCommand referenced in function _Midi_Init
midi.obj : error LNK2019: unresolved external symbol __imp__Tcl_GetErrno referenced in function _Rpmidi_OutOutput
midi.obj : error LNK2019: unresolved external symbol __imp__Tcl_RegisterChannel referenced in function _Rpmidi_InOpen
midi.obj : error LNK2019: unresolved external symbol __imp__Tcl_SetResult referenced in function _OpenOut_Cmd
midi.obj : error LNK2019: unresolved external symbol __imp__Tcl_SetObjResult referenced in function _Rpmidi_Error
midi.obj : error LNK2019: unresolved external symbol __imp__Tcl_UnregisterChannel referenced in function _Rpmidi_InClose
midi.obj : error LNK2019: unresolved external symbol __imp__TclWinConvertError referenced in function _Rpmidi_OutOutput

C:\Users\ET\AppData\Local\Apps\Tcl86\lib\tcl86t.lib : warning LNK4272: library machine type 'x64' conflicts with target machine type 'x86'

midi.dll : fatal error LNK1120: 15 unresolved externals

---------------


Here's some [info] running the "wish" console:


(bin) 2 % parray tcl_platform
tcl_platform(byteOrder) = littleEndian
tcl_platform(engine) = Tcl
tcl_platform(machine) = amd64
tcl_platform(os) = Windows NT
tcl_platform(osVersion) = 10.0
tcl_platform(pathSeparator) = ;
tcl_platform(platform) = windows
tcl_platform(pointerSize) = 8
tcl_platform(threaded) = 1
tcl_platform(user) = et
tcl_platform(wordSize) = 4

(bin) 3 % info pa
8.6.11

(bin) 4 % info nameof
C:/Users/ET/AppData/Local/Apps/Tcl86/bin/wish.exe

[Christian Gollwitzer]

Am 20.12.22 um 22:33 schrieb et4:

> The last linker warning seems to suggest I'm not building a 64 bit
> version
>

> C:\Users\ET\AppData\Local\Apps\Tcl86\lib\tcl86t.lib : warning LNK4272:
> library machine type 'x64' conflicts with target machine type 'x86'
>

Yes, that's correct. You run cl for 32 bit and the Tcl you link it with
is 64 bit. For Visual C++, the bitness depends on how you run the
command line. You should have different entries in the start menu,
labelled "Command prompt x86" and "Command prompt x64" or similar, maybe
also "cross compile x64 bit" (can't check it now, I'm not on Windows at
the moment). You might also look for the path where the start menu entry
points to, this thing is a batch file and in the same dir you should
find both versions.


There are two things to consider
a) 32 bit builds do not make much sense these days anymore
b) Your Tcl is 64 bit
c) your midi.c file needs to be updated for 64 bit

So: The easiest way to get it going would be to get a 32 bit Tcl, then
you don't need to do more changes to midi.c. However, for long-term use
it is much better to adapt midi.c for 64 bit and switch to the 64 bit
compiler....

Christian

[et4]

ahhhh, thanks, found it, yup was using the wrong one.

after switched to the 64 bit one, I now get the same results as Ralf, and it linked ok with the nonstub version of Magicsplat's tcl86t.lib

I did consider going 32bit, but magicsplat 32 won't install on a 64 bit system.

And the 64bit midi.dll I just built, while it will load into magicsplat's 64bit wish console, it won't load into a 64 bit tclkit-with-gui-and-twapi I use on my 64 bit system. It gets the below error.

% load midi.dll
couldn't load library "midi.dll": this library or a dependent library could not be found in library path

And so I suspect it would get the same with a 32 bit dll and a 32 bit tclkit - the only way I can currently run 32 bit on my 64 bit system. I would probably have to find another distro, like an old activestate 32 to run on my 64 bit system.

So, I might as well just go 64 bit now. It's been a few decades since I did C programming (tcl has spoiled me) but I might be able to figure it out.

Thanks for the info, I was worried I had installed the wrong compiler.

et

[et4]

On 12/20/2022 3:09 PM, et4 wrote:
...snip...

I eliminated all the compiler warnings, by

1. change DWORD to DWORD_PTR
2. add const in code like: const char *argv[] - and a few char *buf, that fixed most warnings

but some of the DWORDs might need to stay, since midi mostly has a fixed 3 bytes command structure and there's comments about fitting a midi command into a DWORD using bit fiddling. So, I left those as DWORD, and got no warnings. There were 6 of those.

It actually runs a bit, but has some troubling effects

it can detect all the midi devices of input and output
it can *sometimes* play a sound

there should be a default midi out device: "Microsoft GS Wavetable Synth" that can be used with no actual midi controller, it will show up in the output by itself. In the below code it will work with a 0 as the midi id.

load midi.dll

set ::f [midi::openout 0]
midi::sendshort 0x90 60 112 ;# 0x90 = send note, 60=note (a C) with velocity (loudness) 112
after 1000
midi::sendshort 0x90 70 30 ;# higher note, softer
close $::f

This code will work about 80% when I enter the above, by hand (the after not then needed), into a windows console. I then hear a piano playing 2 different notes with the second one quieter.

BUT --- if I place that in a source script either at script level or inside a proc, I get an error dialog:

"channel type midiout must define seekProc if defining wideSeekProc"

I even get the error sometimes by just pasting the above more than once (load only the first time).

I've noticed that there is a struct Tcl_ChannelType that has changed since 8.3 but the docs say it's ok to use the old struct: "It is still possible to create channel with the above structure. The internal channel code will determine the version."

Here's the code setting up the structure, the midiin is the essentialy the same, and wideSeekProc is NOT defined in either case.

instancePtr->tcl_ChannelType.typeName = "midiout";
instancePtr->tcl_ChannelType.blockModeProc = NULL;
instancePtr->tcl_ChannelType.closeProc = Rpmidi_OutClose;
instancePtr->tcl_ChannelType.inputProc = Rpmidi_OutInput;
instancePtr->tcl_ChannelType.outputProc = Rpmidi_OutOutput;
instancePtr->tcl_ChannelType.seekProc = NULL;
instancePtr->tcl_ChannelType.setOptionProc = NULL;
instancePtr->tcl_ChannelType.getOptionProc = NULL;
instancePtr->tcl_ChannelType.watchProc = Rpmidi_OutWatchChannel;
instancePtr->tcl_ChannelType.getHandleProc = Rpmidi_OutGetHandleProc;


This leads me to suspect something is not the right size and/or something is getting clobbered. By being sensitive to interactive vs. in a script or proc, makes me think the stack is getting corrupted by mis-sized elements somewhere. This is just a hunch however. After dismissing the dialog, it thinks for a few seconds and then exits.

This code is 20 years old, and I can't tell if it ever fully worked, so it could be more than just a 32 to 64 bit conversion.

I think I'm going to have to call it quits. Shame.


Just for laughs, here's a pic of a late 1970-ish e&s picture system which I worked with that has dials and was my inspiration here :)

https://www.museumwaalsdorp.nl/en/history/comphistory/computer-history-the-period-1978-1983/comp782e/

et

[et4]

On 12/21/2022 1:01 PM, et4 wrote:
> On 12/20/2022 3:09 PM, et4 wrote:

Interesting finding...

>
> I've noticed that there is a struct Tcl_ChannelType that has changed since 8.3 but the docs say it's ok to use the old struct: "It is still possible to create channel with the above structure. The internal channel code will determine the version."

In looking at the code for this, in Tcl_CreateChannel, I don't see anything that would determine the version for an old struct setup.

...

>     instancePtr->tcl_ChannelType.outputProc = Rpmidi_OutOutput;
>     instancePtr->tcl_ChannelType.seekProc = NULL;
>     instancePtr->tcl_ChannelType.setOptionProc = NULL;

...

The code for Tcl_CreateChannel does have 2 asserts, and then there are 5 or 6 checks, and one of them checks for a wideSeekProc:

if ((NULL!=typePtr->wideSeekProc) && (NULL == typePtr->seekProc)) {
Tcl_Panic("channel type %s must define seekProc if defining wideSeekProc", typePtr->typeName);
}

It now comes down to what Tcl_Alloc does? Does it zero the allocated memory? If not, then an undefined wideSeekProc might contain leftover garbage. This could explain why this panic would occur more on the second try.

So, I added a set to NULL of the wideSeekProc in the two cases and the code for writing the notes works now, whether typed in or in a script or a proc.

So... I can play notes now. But my real desire is to read them. For that, I'm not sure how to code it, since there are no docs and only some comments that say:

* NOTE: midi::openout and midi::openin returns a channel id, where
* the usual channel commands, read, gets, puts, close can then be used.


There's a free program called, MIDIKey2Key that can send a global shortcut key when a specific midi input occurs. It has a log window, which can be used to see all midi commands coming in for a selected input device. When I hit a C4 with force (velocity 127), it shows 0x90307f and it seems that these inputs are global, since both this program and my DAW received the note.

I'll have to think about how to try to get 3 bytes in. And I don't know how this channel would deal with input that is ignored, does it stall waiting or just overwrite a buffer? Time to read Ashok's tcl book on channels :)

So, I'm back in the game, for now...

et

[Ralf Fassel]

* et4 <tcl...@rocketship1.me>

| I've noticed that there is a struct Tcl_ChannelType that has changed
| since 8.3 but the docs say it's ok to use the old struct: "It is still
| possible to create channel with the above structure. The internal
| channel code will determine the version."
>
| Here's the code setting up the structure, the midiin is the essentialy
| the same, and wideSeekProc is NOT defined in either case.
>
| instancePtr->tcl_ChannelType.typeName = "midiout";
| instancePtr->tcl_ChannelType.blockModeProc = NULL;
| instancePtr->tcl_ChannelType.closeProc = Rpmidi_OutClose;
| instancePtr->tcl_ChannelType.inputProc = Rpmidi_OutInput;
| instancePtr->tcl_ChannelType.outputProc = Rpmidi_OutOutput;
| instancePtr->tcl_ChannelType.seekProc = NULL;
| instancePtr->tcl_ChannelType.setOptionProc = NULL;
| instancePtr->tcl_ChannelType.getOptionProc = NULL;
| instancePtr->tcl_ChannelType.watchProc = Rpmidi_OutWatchChannel;
| instancePtr->tcl_ChannelType.getHandleProc = Rpmidi_OutGetHandleProc;
>
| This leads me to suspect something is not the right size and/or
| something is getting clobbered. By being sensitive to interactive
| vs. in a script or proc, makes me think the stack is getting corrupted
| by mis-sized elements somewhere. This is just a hunch however. After
| dismissing the dialog, it thinks for a few seconds and then exits.

It might be a good idea to memset() the struct to 0 to clear all unused
fields. I think the code uses Tcl_Alloc() to allocate the memory, but
this just gives you random bytes. If the struct now contains additional
fields compared to 20 years ago, this would give you random bytes in the
additional field which could explain the wideSeekProc being set to
something.

| This code is 20 years old, and I can't tell if it ever fully worked,
| so it could be more than just a 32 to 64 bit conversion.
>
| I think I'm going to have to call it quits. Shame.

Can you show the changes you've made (diffs)? Maybe I can spare some
time today or tomrrow to have a glance at it.

R'

[Ralf Fassel]

* et4 <tcl...@rocketship1.me>

| Here's the code setting up the structure, the midiin is the essentialy
| the same, and wideSeekProc is NOT defined in either case.
>
| instancePtr->tcl_ChannelType.typeName = "midiout";
| instancePtr->tcl_ChannelType.blockModeProc = NULL;
| instancePtr->tcl_ChannelType.closeProc = Rpmidi_OutClose;
| instancePtr->tcl_ChannelType.inputProc = Rpmidi_OutInput;
| instancePtr->tcl_ChannelType.outputProc = Rpmidi_OutOutput;
| instancePtr->tcl_ChannelType.seekProc = NULL;
| instancePtr->tcl_ChannelType.setOptionProc = NULL;
| instancePtr->tcl_ChannelType.getOptionProc = NULL;
| instancePtr->tcl_ChannelType.watchProc = Rpmidi_OutWatchChannel;
| instancePtr->tcl_ChannelType.getHandleProc = Rpmidi_OutGetHandleProc;

In addition to clearing the memory after alloc, it might be necessary to
set the .version member of the struct so TCL knows it's not an 'old'
binary version (8.3 or earlier). However, the docs are not clear about
what to set it to, I would just use the highest number available
(TCL_CHANNEL_VERSION_5) if everything is zeroed properly.

R'

[et4]

On 12/22/2022 1:08 AM, Ralf Fassel wrote:

>
> Can you show the changes you've made (diffs)? Maybe I can spare some
> time today or tomrrow to have a glance at it.
>
> R'

I put it up on dropbox. If it nags you to get an account, just hit the close box.

https://www.dropbox.com/s/bohccd1a28ha72e/midi-c.zip?dl=0

I added a DebugLogv (printf like var args) logger and right indented a bunch of debug code so I could see the real code. Same with some if's around todo like comments. It's got debug turned on and I changed it to write debug to the a: drive (my ramdisk). Their debug logger is an open, append, close setup. If you use it, change the a: to your preference.


I was kinda done with the output part which works ok (by just adding that missing seek struct member), and began to look at input, which is what I'm really interested in.

AND THEN ....

I found a rather nasty problem. There are 2 kinds of midi data, fixed size 3 byte normal stuff (notes, dials, buttons etc.) and long sysex messages, used for sending instrument data to synths and things like that.

Turns out they only implemented the sysex stuff, so reading notes and dials isn't implemented. That's why there's no examples of how to do input.

The midi code has just one callback from windows with an extra 2 args. When it's normal data, the 2 args are just the data, but when it's the longer data messages, the first arg is an address to some header, since they need somewhere to store the variable length message.

I found that out when I was trying to clone the sysex code for use with the regular data.

Here's from the specs found with google:

MIM_LONGDATA
dwParam1 = (DWORD) lpMidiHdr
dwParam2 = dwTimestamp


MIM_DATA
dwParam1 = dwMidiMessage
dwParam2 = dwTimestamp


When I tried to implement the short DATA case, I crashed because there's no pointer coming in. Here's the top of that callback code:

void CALLBACK Rpmidi_InFunc(
HMIDIIN hMidiIn,
UINT wMsg,
DWORD_PTR dwInstance,
DWORD_PTR dwParam1,
DWORD_PTR dwParam2)

{


Here, dwParam1 is not an address with DATA, only with LONGDATA so I don't know how to return the data to a tcl call.

But the DATA coming into the callback, is correct, although the byte ordering is low to high in those DWORD_PTR's. I was able to do debug logging of the two params and it matches what I see when I use the midikey2key program. I think the 2nd param is a time.

And with my midi controller, there's one LONGDATA command it can send, and I actually was able to get that using a [read] call, like so, since I couldn't figure out how to get the data w/o polling.

proc bin2hex {args} { regexp -inline -all .. [binary encode hex [join $args ""]] } ;# from Ashok's book
proc doit {args} {
set f [midi::openin 3] ;# on input, 3 is my
fconfigure $f -translation binary -buffersize 1

while { 1 } {
set foo [read $f]
set foo2 [bin2hex $foo ]
if { [llength $foo2] < 1 } {
wait 5
# puts stderr "no data"
continue
}
puts "[incr i] foo2=|$foo2| slen= [string length $foo] - llen= [llength $foo2] foo=|$foo|"
wait 1000
}

}



Even w/o any midi hardware, you should still see 1 output device:

load midi.dll
set i -1
foreach device [midi::getoutdevs] {
puts "out [incr i]= $device"
}


it should return just the one if you don't have any midi hardware, mine has 4 devs as well as MS's

out 0= Microsoft GS Wavetable Synth
out 1= Minilab3 MIDI
out 2= Minilab3 DIN THRU
out 3= Minilab3 MCU
out 4= Minilab3 ALV


Here's some tcl code I used for playing 2 notes. The 0 is for the MS Synth.

set f [midi::openout 0]

midi::sendshort 144 60 112
after 1000
midi::sendshort 144 70 80
after 1000
close $f




et

[Ralf Fassel]

* et4 <tcl...@rocketship1.me>

| On 12/22/2022 1:08 AM, Ralf Fassel wrote:
>
| > Can you show the changes you've made (diffs)? Maybe I can spare
| > some
| > time today or tomrrow to have a glance at it.
| > R'
>
| I put it up on dropbox. If it nags you to get an account, just hit the close box.
>
| https://www.dropbox.com/s/bohccd1a28ha72e/midi-c.zip?dl=0

Ok, got it. Maybe I have time to look into it later today.

Right now I'd say you definitely:

- need to memset() the allocated memory for Rpmidi_ChannelInstance in
Rpmidi_InOpen() and Rpmidi_OutOpen() to 0 in order to clean out all
fields

- need to set the .version member of instancePtr->tcl_ChannelType to
TCL_CHANNEL_VERSION_5, since TCL looks at that field in order to
determine the binary structure of the data.
If that field by chance contains something between
1 and 4, than that is used for access to certain fields in the struct
which might explain why your program works only "most of the times".

| AND THEN ....
>
| I found a rather nasty problem. There are 2 kinds of midi data, fixed
| size 3 byte normal stuff (notes, dials, buttons etc.) and long sysex
| messages, used for sending instrument data to synths and things like
| that.
>
| Turns out they only implemented the sysex stuff, so reading notes and
| dials isn't implemented. That's why there's no examples of how to do
| input.
>
| The midi code has just one callback from windows with an extra 2
| args. When it's normal data, the 2 args are just the data, but when
| it's the longer data messages, the first arg is an address to some
| header, since they need somewhere to store the variable length
| message.
>
| I found that out when I was trying to clone the sysex code for use
| with the regular data.
>
| Here's from the specs found with google:
>
| MIM_LONGDATA
| dwParam1 = (DWORD) lpMidiHdr
| dwParam2 = dwTimestamp
>
>
| MIM_DATA
| dwParam1 = dwMidiMessage
| dwParam2 = dwTimestamp

https://learn.microsoft.com/de-de/windows/win32/multimedia/midi-functions

should be the reference used here.

| When I tried to implement the short DATA case, I crashed because
| there's no pointer coming in. Here's the top of that callback code:
>
| void CALLBACK Rpmidi_InFunc(
| HMIDIIN hMidiIn,
| UINT wMsg,
| DWORD_PTR dwInstance,
| DWORD_PTR dwParam1,
| DWORD_PTR dwParam2)
>
| {
>
>
| Here, dwParam1 is not an address with DATA, only with LONGDATA so I
| don't know how to return the data to a tcl call.

| But the DATA coming into the callback, is correct, although the byte
| ordering is low to high in those DWORD_PTR's. I was able to do debug
| logging of the two params and it matches what I see when I use the
| midikey2key program. I think the 2nd param is a time.

https://learn.microsoft.com/de-de/windows/win32/multimedia/mim-data

says:

IM_DATA
dwParam1 = dwMidiMessage
dwParam2 = dwTimestamp

Parameters

dwMidiMessage

MIDI message that was received. The message is packed into a
doubleword value as follows:

Requirement Value Description
High word High-order byte Not used.
Low-order byte Contains a second byte of MIDI data (when needed).
Low word High-order byte Contains the first byte of MIDI data (when needed).
Low-order byte Contains the MIDI status.

So that's how these data need to be parsed...

| Even w/o any midi hardware, you should still see 1 output device:
>
| load midi.dll
| set i -1
| foreach device [midi::getoutdevs] {
| puts "out [incr i]= $device"
| }

Well, I don't :-/ [midi::getoutdevs] has empty return on my Windows 10.
But I have a real USB MIDI interface at home, connected to a keyboard which
I didn't turn on for 6 years now... so maybe, just maybe... but don't
hold your breath :-)

HTH
R'

[et4]

On 12/22/2022 6:30 AM, Ralf Fassel wrote:
..snip..

>
> | Even w/o any midi hardware, you should still see 1 output device:
>>
> | load midi.dll
> | set i -1
> | foreach device [midi::getoutdevs] {
> | puts "out [incr i]= $device"
> | }
>
> Well, I don't :-/ [midi::getoutdevs] has empty return on my Windows 10.
> But I have a real USB MIDI interface at home, connected to a keyboard which
> I didn't turn on for 6 years now... so maybe, just maybe... but don't
> hold your breath :-)
>
> HTH
> R'
>

You did change the debug output yes? When I tried it out on another computer I had forgot that the .dll as built had debug on and I had no a:/files directory to write to, so tcl just silently crashed.

But after creating the a:/files dir, it worked and I get in the magicsplat tk console:

% load midi.dll
% midi::getindevs
% midi::getoutdevs
{Microsoft GS Wavetable Synth}
% set f [midi::openout 0]
midiout0
% midi::sendshort 0x90 70 127 ;# loudest value 127
% midi::sendshort 0xc0 16 0 ;# change instrument to an organ
% midi::sendshort 0x90 70 100
% midi::sendshort 0x90 74 100 ;# 2 part harmony
% close $f ;# emergency turn off


I haven't installed anything special on this machine, so Dunno.

btw, if that 3rd value is missing it crashes, it's only checking for argc > 4

Even if I can't get the midi DATA in via the channel, I should be able to set a global variable from the C code. Maybe I'll add a parameter to the midi::openin to send in the name of a global variable to be set to the results, probably just in text formatted as xxxxxx yyyyyy in hex.

Then either poll or vwait. Maybe also include a sequence number.

Anyway, I've gone and ordered that Behringer X-TOUCH MINI. That device has a ring of leds around each control knob and according to one review on amazon, I can write to the midi and turn them on/off:

MIDI commands accepted by Behringer X-TOUCH MINI:

0x90 [0..15] n Set button LED 0=off, 1=on, 2=blink
0xB0 [1..8] n Set LED ring mode 0=single, 1=pan, 2=fan, 3=spread, 4=trim
0xB0 [9..16] n Set LED ring illumination 0=off [1..13]=on, [14..26]=blink, 26=all on, 27=all blink
0xB0 127 n Set mode 0=standard (default), 1=MC
0xBA [1..8] n Set knob position to n
0xC0 n Select layer 0=Layer A (default), 1=Layer B (ONLY IF NOT IN MC MODE)


So, it could be a rather fun device to play with.

e

[et4]

I've implemented all of these and found a few more arg checks that used > instead of == so that's done.

Next, I tried to see if I could return the data by calling Tcl_SetVar since I don't know how to get the data back through the channel. But that doesn't work either, I can get a crash.

For what it's worth:

Looks like it's not safe to call

Tcl_SetVar(interp_global, "midi_data_input", data, TCL_GLOBAL_ONLY);


in the middle of a midi callback, I got an error that says,

alloc: invalid block: 000001A244E22800: 58 44

which comes from the panic below given the message and number of args. It's in tclThreadAlloc.c

I wonder if a midi callback actually can interrupt tcl scripts. I can produce a crash in 10-30 seconds but usually it's silent. Some times my script code would crash because the global variable is incorrect, e.g. blank or missing some list elements. Maybe there's a shimmer going on at the time?

In the above Tcl_SetVar call, I also had an issue with the interp arg, as it's not actually present in the callback, so I have to stash one away on the first call to a midi function. After that I never change it.

Can the main interp pointer value ever change over time?

I suspect that this isn't really the issue however, since it can work 10,000 times or more just fine, so I think this crash is something else. It really feels like an interrupt type of problem. Ordinary, I assume that if I don't enter the event loop from the script, nothing can interrupt me. But with a windows callback, I'm not so sure.

static Block *
Ptr2Block(
char *ptr)
{
Block *blockPtr;

blockPtr = (((Block *) ptr) - 1);
if (blockPtr->magicNum1 != MAGIC || blockPtr->magicNum2 != MAGIC) {
Tcl_Panic("alloc: invalid block: %p: %x %x",
blockPtr, blockPtr->magicNum1, blockPtr->magicNum2);
}
...

return blockPtr;
}


So, stuck for the moment...

et

[Ralf Fassel]

* et4 <tcl...@rocketship1.me>

| Looks like it's not safe to call
>
| Tcl_SetVar(interp_global, "midi_data_input", data, TCL_GLOBAL_ONLY);
>
| in the middle of a midi callback, I got an error that says,
>
| alloc: invalid block: 000001A244E22800: 58 44
>
| which comes from the panic below given the message and number of
| args. It's in tclThreadAlloc.c

I haven't verified it by looking at the thread-ids, but the only way
what I see (and what you describe here) makes sense is that the
TCL-program and the windows-MIDI-callbacks run in separate threads.
tclsh does not enter the event-loop, yet I see the windows callbacks
arriving.

The midi.c code does not handle this situation at all, it simply
modifies the data buffers in both the callback and the TCL-read
function. This cannot work reliably without proper locking mechanisms.

Looking at the poor quality of rest of the midi.c code (error checks
missing, 32/64bit issues, logic errors in some places), I myself would
discard that project and start from scratch (maybe using some code
blocks from midi.c).

It's not too complicated:

- you just need interfacing to a few midi-functions (open, close, and
the callbacks, probably the info functions, too)

- get rid of the channel idea, and just specify a TCL callback in the
'open' call for incoming data. Outgoing data is straight forward
anyway.

- in the input-callbacks, store the data in some data structure and
notify TCL by some inter-thread-calls
https://www.tcl.tk/man/tcl8.6/TclLib/Notifier.html
(eg Tcl_ThreadQueueEvent or the like)
Take care of proper locking as required.

- then in the TCL main thread, call the callback function (event loop
required for this) and handle the data at script level.

With my Roland UM-ONE connected to a Pioneer keyboard, there are data
coming in at a very moderate rate (100/s max), so this should be no
problem to handle at script level.

HTH
R'

[et4]

On 12/23/2022 2:36 AM, Ralf Fassel wrote:
> * et4 <tcl...@rocketship1.me>
> | Looks like it's not safe to call
>>
> | Tcl_SetVar(interp_global, "midi_data_input", data, TCL_GLOBAL_ONLY);
>>
> | in the middle of a midi callback, I got an error that says,
>>
> | alloc: invalid block: 000001A244E22800: 58 44
>>
> | which comes from the panic below given the message and number of
> | args. It's in tclThreadAlloc.c
>
> I haven't verified it by looking at the thread-ids, but the only way
> what I see (and what you describe here) makes sense is that the
> TCL-program and the windows-MIDI-callbacks run in separate threads.
> tclsh does not enter the event-loop, yet I see the windows callbacks
> arriving.

Ahh, so maybe windows has setup an extra thread. But it's probably not a tcl thread, so it's maybe invisible.

>
> The midi.c code does not handle this situation at all, it simply
> modifies the data buffers in both the callback and the TCL-read
> function. This cannot work reliably without proper locking mechanisms.

In the routine Rpmidi_InFunc there's a switch with case MIM_LONGDATA and one for MIM_DATA, the LONGDATA one works, but they didn't code up the DATA which is the one that's actually more important. my midi keyboard only sends 1 type of long, everything else is a short.

>
> Looking at the poor quality of rest of the midi.c code (error checks
> missing, 32/64bit issues, logic errors in some places), I myself would
> discard that project and start from scratch (maybe using some code
> blocks from midi.c).
>
> It's not too complicated:

Maybe for someone who understands this. I moved from C to tcl because I don't like all the typing and defining etc. I can hack it, but not write good C code. After 20 years, I'm still not all that good at tcl. But I love the language.

>
> - you just need interfacing to a few midi-functions (open, close, and
> the callbacks, probably the info functions, too)
>
> - get rid of the channel idea, and just specify a TCL callback in the
> 'open' call for incoming data. Outgoing data is straight forward
> anyway.
>
> - in the input-callbacks, store the data in some data structure and
> notify TCL by some inter-thread-calls
> https://www.tcl.tk/man/tcl8.6/TclLib/Notifier.html
> (eg Tcl_ThreadQueueEvent or the like)
> Take care of proper locking as required.
>
> - then in the TCL main thread, call the callback function (event loop
> required for this) and handle the data at script level.

I'm afraid most of this is over my head. I'm a really old dog. In the 70's I was able to modify other people's C code, and so ported most of unix over to the old DEC systems, like the pdp-11 and vax (we couldn't afford unix back then).

I supported people who were world famous graphics programmers, who later created Pixar. For them, they didn't care about clean code, so I fit in. So, to do this right, is just not something I'm good at.

I was hoping I could have just filled in the short midi data in the same way the long data is returned, but when I tried to just clone that, it didn't work. They are storing it in places deep inside structs with indirection etc. that at my age gives me a headache.

>
> With my Roland UM-ONE connected to a Pioneer keyboard, there are data
> coming in at a very moderate rate (100/s max), so this should be no
> problem to handle at script level.

All I really want to do is be able to combine tcl, midi, and twapi so I can spin dials instead of having only 1 mouse wheel. Not sure what I want to do with it. I thought if this midi code could work enough for that I'd find a use for it.

However, I do think this is worthy of a TIP, since having all these controls could be really cool in Tk, and browsing on amazon (to find my daughter a midi piano) I see there are hundreds of usb midi devices.

But if you've seen any of my TIPs (got 3 approved for 8.7) you'd know that it's people like Brian G. who did all the work (e.g. we're getting a range command called lseq, and underscores in numbers).

Now, if someone young and sharp wanted to do this job.... :)

e

[et4]

Well, I'm done. It's good enough as it is, it's only crashing with my torture test, and most of the time it dies silently w/o the panic dialog. I'll just have it re-run itself in a forever loop.

I've updated the dropbox file, but it's the same link if anyone is interested. I'm not going to post it anywhere else, since it's not good enough.

Thanks all, and especially Ralf, who showed me how to do C compiling on windows. And thanks to Christian for how to run the 64 bit version.