llama.prg

[alkr...@yahoo.com]

Hi,

 I've started a new project, llama.prg - the Harbour bindings to the llama.cpp ( thanks to Antonio for pointing out this project).

At the moment, it allows (see test1.prg) to load the selected local model and have a dialogue with it. Model startup parameters can be set in models.ini.
The context of the dialogue is not remembered yet - I'm figuring out how to do this.

It works with 64-bit compilers only. For Windows msvc should be used ( I cpild build it with 64-bit mingw, too, but it does't work properly). For Linux the standard gcc works without problems.

To make it possible to work with local AI models, using any C compiler, including 32-bit, I suppose to use a part of my guiserver project. Th idea is that you connect to an external program, written on Harbour, using llama.prg, which works directly with AI model, send commands there and get answers.

The project is placed on gitflic.ru : https://gitflic.ru/project/alkresin/llama_prg

Regards, Alexander.

[trepao2...@gmail.com]

Thank you very much Mr. Alexander Kresin !!!

[antonio....@gmail.com]

Dear Alexander,

Congratulations for your initiative and your work! 

One possible solution to remember the context is to save the conversation and provide the complete conversation to the model on each request.

Based on your interest on llama.cpp I take this opportunity to comment you about "embeddings" implementation idea for Harbour: DBFs or SQL use provides

us "literal" searches. In example we can search for a specific "word".  To support "context" searches for DBFs or SQL use, we need to provide DBFs or SQL

with "vector database" capabilities. In example if we search for "future", the search will return "next years", etc. as "similar context" contents. We can have

this in Harbour in we use free https://t.co/LuWhALRWTv HuggingFace embeddings API. OpenAI also provides an embeddings API but it is a paid API so we can 

not take such way.

The idea would be to add a field where we store the "embeddings" of the record, so it will allow us to have "context" searches. This would open a new and

very powerfull way to query DBFs and SQL based on "context" searches. Google started this technology with its "Word2Vec", but embeddings provides the same

functionality based on a simple API to be used.  

Maybe with the help of Przemek, yours and others we can take a giant step on Harbour turning it into a vector database. This would be great!

best regards

[alkr...@yahoo.com]

Antonio,

> One possible solution to remember the context is to save the conversation and provide the complete conversation to the model on each request.

  Yes, sure - and maybe the only one. I dig in main.cpp and server.cpp samples, trying to understand, how to do this correctly.

> Based on your interest on llama.cpp I take this opportunity to comment you about "embeddings" implementation idea for Harbour...

 Sounds interesting. Need to think about it.

Regards, Alexander.

[alkr...@yahoo.com]

I've added a  simple demo  of using the built with llama.prg executable from any harbour program, compiled with any C compiler.

Everything is written there in the readme.

The llama_exsrv.exe from the demo package may be used with hbedit. In the latest version of hbedit, the plug_extllm plugin has been added, it can be activated in hbedit.ini. We need to copy llama_exsrv.exe to hbedit/plugins and create models.ini with paths to the models there. When you launch this plugin, a separate window opens and you are prompted to open the model - after successfully opening, you can start a dialogue.

Regards, Alexander.

 

[alkr...@yahoo.com]

Linux (Debian 11) demo added to the same page.

[cod...@outlook.com]

Could not download ZIP from   https://gitflic.ru/project/alkresin/llama_prg  .

When I click on Код , then on Скачат ZIP архив,  I got page with error  404 Ошибка .

Please, anyone can help me ?

Regards

Simo.

[alkr...@yahoo.com]

Probably, this option works for registered users only - I'll try to find out this.

You may to register on gitflic.ru or use the 'git clone' command:

git clone https://gitflic.ru/project/alkresin/llama_prg.git

Regards, Alexander.

воскресенье, 24 марта 2024 г. в 21:56:10 UTC+3, cod...@outlook.com:

[alkr...@yahoo.com]

I've updated the project and demo fies on my site. Now it keeps the context, so a real dialog is possible.

Hbedit plugin is updated, too.

Regards, Alexander.

[cod...@outlook.com]

Thank you Alexander !

This worked:

git clone https://gitflic.ru/project/alkresin/llama_prg.git

Yesterday I made account on gitflic, then waited verification message on my mail, but never received it.

Best Regards,

Simo.

[antonio....@gmail.com]

I have built it this way using just a go.bat file:

@setlocal
call "%ProgramFiles%\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64
cl -c -TP -Ic:\harbour\include -Illama.cpp -Illama.cpp\common /EHsc source\hllama.cpp
cl -c -TP -Illama.cpp -Illama.cpp\common /EHsc llama.cpp\llama.cpp
cl -c -Illama.cpp -Illama.cpp\common /EHsc llama.cpp\ggml.c llama.cpp\ggml-alloc.c llama.cpp\ggml-backend.c llama.cpp\ggml-quants.c llama.cpp\common\sampling.cpp llama.cpp\common\common.cpp llama.cpp\common\grammar-parser.cpp llama.cpp\common\build-info.cpp
lib /OUT:llama64.lib hllama.obj llama.obj ggml.obj ggml-alloc.obj ggml-backend.obj ggml-quants.obj sampling.obj common.obj grammar-parser.obj build-info.obj
@endlocal

then to build the examples:

@setlocal
call "%ProgramFiles%\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64
c:\harbour\bin\win\msvc64\hbmk2.exe test1.prg -lllama64
@endlocal

regards

[antonio....@gmail.com]

Dear Alexander,

How do you specify in models.ini to use all the CPU threads or to increase the number of used threads ?

many thanks

[antonio....@gmail.com]

Dear Alexander,

It goes slower than using official main.exe, no idea why. I have manually increased the threads to 16 for my pc.

@setlocal
call "%ProgramFiles%\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64

cl -c -TP /Gm- /GS /arch:AVX2 -Ic:\harbour\include -Illama.cpp -Illama.cpp\common /EHsc source\hllama.cpp
cl -c -TP /Gm- /GS /arch:AVX2 -Illama.cpp -Illama.cpp\common /EHsc llama.cpp\llama.cpp
cl -c /Gm- /GS /arch:AVX2 -Illama.cpp -Illama.cpp\common /EHsc llama.cpp\ggml.c llama.cpp\ggml-alloc.c llama.cpp\ggml-backend.c llama.cpp\ggml-quants.c llama.cpp\common\sampling.cpp llama.cpp\common\common.cpp llama.cpp\common\grammar-parser.cpp llama.cpp\common\build-info.cpp

lib /OUT:llama64.lib hllama.obj llama.obj ggml.obj ggml-alloc.obj ggml-backend.obj ggml-quants.obj sampling.obj common.obj grammar-parser.obj build-info.obj
@endlocal

many thanks

[alkr...@yahoo.com]

Antonio,

> How do you specify in models.ini to use all the CPU threads or to increase the number of used threads ?

   I don't specify it now, but I can add it to a list of available parameters.

> It goes slower than using official main.exe, no idea why

   With both test1.prg and test2.prg?

   The test1.prg is based on simple.cpp sample, it creates the context for every quesion and closes it after getting an answer.

   The test2.prg is based on main.cpp code, it creates the context one time at the beginning of the session.

Regards, Alexander.

[alkr...@yahoo.com]

I've wrote to their support and they ask for your mail address to check what happened.

BTW, had you checked your spam folder?

Regards, ALexander.

вторник, 26 марта 2024 г. в 14:30:49 UTC+3, cod...@outlook.com:

[cod...@outlook.com]

Thank you Alexander.

Nothing in my spam and trash folders.

Regards,

Simo.

[cod...@outlook.com]

Hi,

I made llama64.lib using .bat file suggested by Antonio. Windows 10 64 bit.

Also created test1.exe (test1.prg) and test2.exe (test2.prg), again using .bat file suggested by Antonio.

I used model file phi-2_Q4_K_M.gguf.

With test1.exe it takes about 70 seconds to start answering. But it was just start, writing only few words then stops, waiting to continue … very slow.

With test2.exe it takes about 50 seconds to start answering. Again it was just start writing few words then also stops, waiting to continue … very slow.

Then I used Alexander examples, with same model file phi-2_Q4_K_M.gguf.

First used test_exllm.exe (GUI). It starts answering instantly after clicking on Ask button, fast and clean.

Second used llama_exsrv.exe (command line). Also starts answering instantly, fast and clean.

I did not used Alexander HWBuilder nor its .bat files, as Antonio's .bat files was simplier for me.

Regards,

Simo.

[antonio....@gmail.com]

Dear Alexander,

Could you please post a GIF file to see how fast it works on Linux for you ?

If we use our llama64.dll then it works really fast:

https://forums.fivetechsupport.com/viewtopic.php?p=266316#p266316

There must be a difference that it is making it run so slow

many thanks for your support

[alkr...@yahoo.com]

I will try to make a gif later - my Linux computer is at home.

Here, at work, I have to use Windows.

You may try   my copy of test2 - maybe, some compile/link options do the difference. Take a look at the stderr.log and compare it with main.log, which is created by main.exe from llama.cpp.

Regards, Alexander.

пятница, 29 марта 2024 г. в 09:17:01 UTC+3, antonio....@gmail.com:

[alkr...@yahoo.com]

I've built test2.exe with Antonio's bat file and it really works very slowly.

So, some options should be added from my scripts - I don't know exactly, which of them, because I'm not familiar with msvc. I get them, tracing the build process of llama.cpp.

Regards, Alexander

[antonio....@gmail.com]

This version works faster:

@setlocal
call "%ProgramFiles%\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64

set HB_USER_CFLAGS=-TP /Gm- /GS /arch:AVX2 /fp:precise /Zc:inline /Zc:forScope
set HB_USER_LDFLAGS=/INCREMENTAL:NO /NOLOGO /SUBSYSTEM:CONSOLE /TLBID:1 /MANIFEST /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /manifest:embed /DYNAMICBASE /NXCOMPAT

c:\harbour\bin\win\msvc64\hbmk2.exe test1.prg -lllama64
@endlocal

[cod...@outlook.com]

Hi.

Alexander, on my computer your copy of test2.exe works fast and clear. As other your exe examples.

Then I compiled your test2.prg using  my llama64.lib. When compiling it I got error: unknown function LLM_PRINT_TIMINGS(). It is on line 72. After I commented that line test2.exe is created. But new test2.exe worked same slow as I reported for other examples in previous post.

Also on my computer did not helped Antonio's  new build .bat.

Regards,

Simo.

[antonio....@gmail.com]

Much improved. We were not using the right terminal!

go.bat (to build llama64.lib)

@setlocal
call "%ProgramFiles%\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64

set CFLAGS=/Ox /O2 /GL /arch:AVX512 /Oi /Ot /Oy /GS- /sdl- /Zc:inline
cl -c %CFLAGS% /TP /errorReport:queue -Ic:\harbour\include -Illama.cpp -Illama.cpp\common source\hllama.cpp llama.cpp\llama.cpp llama.cpp\common\sampling.cpp llama.cpp\common\common.cpp llama.cpp\common\grammar-parser.cpp llama.cpp\common\build-info.cpp
cl -c %CFLAGS% /errorReport:queue -Illama.cpp -Illama.cpp\common llama.cpp\ggml.c llama.cpp\ggml-alloc.c llama.cpp\ggml-backend.c llama.cpp\ggml-quants.c

lib /OUT:llama64.lib hllama.obj llama.obj ggml.obj ggml-alloc.obj ggml-backend.obj ggml-quants.obj sampling.obj common.obj grammar-parser.obj build-info.obj
@endlocal

go1.bat (using gtwvt)

@setlocal
call "%ProgramFiles%\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64

c:\harbour\bin\win\msvc64\hbmk2.exe test1.prg -gtwvt -lllama64
@endlocal

best regards

[cod...@outlook.com]

Hi Antonio

I made new llam64.lib .

Also made new test1.exe and test2.exe ,  using gtwvt .

When I start  test1.exe, windows opens, I select model, can see message Loading ... and test1.exe dies. Nothing on screen.

As I could see test1.exe dies on function  llm_Open_Model( aModels[i,1] ) in

   IF llm_Open_Model( aModels[i,1] ) != 0
      ? " === Can't open " + aModels[i,1] + " ==="    //  this message is not displayed
      RETURN Nil
   ENDIF

In hb_out.log file I found:

Application Internal Error - D:\llama_prg\test1.exe
Terminated at: 2024-03-30 19:37:17
Unrecoverable error 6005: Exception error:
    Exception Code:C000001D
    Exception Address:00007FF62280ED76
    RAX:0000000000000000  RBX:000000AC1A52F3C0  RCX:000002A68C116020  RDX:000002A68C116040
    RSI:0000000000000000  RDI:0000000000000002  RBP:000000AC1A52F260
    R8 :0000000000000170  R9 :000002A68C116040  R10:00007FF622A11228  R11:0000000000000170
    R12:0000000000000000  R13:0000000000000000  R14:0000000000000000  R15:0000000000000000
    CS:RIP:0033:00007FF62280ED76  SS:RSP:002B:000000AC1A52F160
    DS:002B  ES:002B  FS:0053  GS:002B
    Flags:00010246
Modules:
00007FF622790000 00000000002DD000 D:\llama_prg\test1.exe
00007FFA80110000 00000000001F5000 C:\Windows\SYSTEM32\ntdll.dll
00007FFA7BA60000 0000000000019000 C:\Program Files\AVG\Antivirus\aswhook.dll
00007FFA7E860000 00000000000BE000 C:\Windows\System32\KERNEL32.DLL
00007FFA7D8C0000 00000000002C8000 C:\Windows\System32\KERNELBASE.dll
00007FFA7A3B0000 0000000000090000 C:\Windows\SYSTEM32\apphelp.dll
00007FFA7E230000 00000000001A1000 C:\Windows\System32\USER32.dll
00007FFA7DB90000 0000000000022000 C:\Windows\System32\win32u.dll
00007FFA7F9F0000 000000000002B000 C:\Windows\System32\GDI32.dll
00007FFA7DFE0000 000000000010D000 C:\Windows\System32\gdi32full.dll
00007FFA76D70000 0000000000027000 C:\Windows\SYSTEM32\WINMM.dll
00007FFA7D820000 000000000009D000 C:\Windows\System32\msvcp_win.dll
00007FFA7FCD0000 000000000009E000 C:\Windows\System32\msvcrt.dll
00007FFA7DEE0000 0000000000100000 C:\Windows\System32\ucrtbase.dll
00007FFA7E150000 0000000000030000 C:\Windows\System32\IMM32.DLL
00007FFA7A550000 000000000009E000 C:\Windows\system32\uxtheme.dll
00007FFA7FD70000 0000000000355000 C:\Windows\System32\combase.dll
00007FFA7E930000 0000000000125000 C:\Windows\System32\RPCRT4.dll
00007FFA7F370000 0000000000115000 C:\Windows\System32\MSCTF.dll
00007FFA7E3E0000 00000000000CD000 C:\Windows\System32\OLEAUT32.dll
00007FFA7E710000 000000000009C000 C:\Windows\System32\sechost.dll
00007FFA7A950000 0000000000012000 C:\Windows\SYSTEM32\kernel.appcore.dll
00007FFA7DDA0000 0000000000082000 C:\Windows\System32\bcryptPrimitives.dll
00007FFA75D50000 00000000000F9000 C:\Windows\SYSTEM32\textinputframework.dll
00007FFA79D00000 000000000035E000 C:\Windows\System32\CoreUIComponents.dll
00007FFA7A060000 00000000000F2000 C:\Windows\System32\CoreMessaging.dll
00007FFA7F2A0000 000000000006B000 C:\Windows\System32\WS2_32.dll
00007FFA7E180000 00000000000AD000 C:\Windows\System32\SHCORE.dll
00007FFA7E4B0000 00000000000AE000 C:\Windows\System32\advapi32.dll
00007FFA79620000 0000000000154000 C:\Windows\SYSTEM32\wintypes.dll
00007FFA7D3B0000 0000000000033000 C:\Windows\SYSTEM32\ntmarta.dll
00007FFA7E5C0000 000000000012A000 C:\Windows\System32\ole32.dll

Called from LLM_OPEN_MODEL(0) in test1.prg
Called from MAIN(35) in test1.prg
------------------------------------------------------------------------

The same apply for test2.exe.

Please, advice what can I do.

Regards,

Simo.

[antonio....@gmail.com]

Dear Simo,

What GGUF file are you using ? For initial testing I advice you to use tinyLlama.gguf

Here you have it:

https://wormhole.app/KYJxY#pFxsjBmn0YzJNwAEwx9vNQ

best regards,

[cod...@outlook.com]

Thank you Antonio.

I use  model file phi-2_Q4_K_M.gguf. Also have mistral-7b-v0.1.Q4_K_M.gguf (you recommended it), but do not use it.

I will try with tinyLlama.gguf and report what happened.

Regards,

SImo.

[cod...@outlook.com]

Hi Antonio.

Same happened with  tinyLlama.gguf.

With previous .BAT files and created llam64.lib file, model  file phi-2_Q4_K_M.gguf worked but very slow. With default GT.

Regards,

Simo.

 

[antonio....@gmail.com]

Dear Simo,

We have noticed that Alexander is not using latest llama.cpp files. We have been able to use llama.cpp current files wih his hllama.cpp

We are going to build the lib with the most recent llama.cpp files and we will share it

best regards

[antonio....@gmail.com]

using the most recent llama.cpp repo files:

git clone https://github.com/ggerganov/llama.cpp

@setlocal
call "%ProgramFiles%\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64

set CFLAGS=/EHsc /Ox /O2 /arch:AVX512 /Oi /Ot /Oy /GS- /sdl- /Zc:inline /favor:INTEL64 -D_CRT_SECURE_NO_WARNINGS
cl -c %CFLAGS% -Ic:\harbour\include -Illama.cpp -Illama.cpp\common source\hllama.cpp
cl -c %CFLAGS% -Illama.cpp llama.cpp\ggml.c llama.cpp\ggml-alloc.c llama.cpp\ggml-backend.c llama.cpp\ggml-quants.c
cl -c %CFLAGS% -Illama.cpp llama.cpp\llama.cpp llama.cpp\unicode.cpp llama.cpp\unicode-data.cpp
cl -c %CFLAGS% -Illama.cpp -Illama.cpp\common llama.cpp\common\common.cpp llama.cpp\common\grammar-parser.cpp llama.cpp\common\sampling.cpp
cl -c %CFLAGS% -TP -Illama.cpp -Illama.cpp\common llama.cpp\common\build-info.cpp.in
lib /out:llama64.lib hllama.obj ggml.obj ggml-alloc.obj ggml-backend.obj ggml-quants.obj llama.obj unicode.obj unicode-data.obj common.obj build-info.obj grammar-parser.obj sampling.obj
@endlocal

@setlocal
call "%ProgramFiles%\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64

set HB_USER_LDFLAGS=/NODEFAULTLIB:msvcrt.lib

c:\harbour\bin\win\msvc64\hbmk2.exe test1.prg -gtwvt -lllama64
@endlocal

build-info.cpp.in has to be edited as Alexander does it:

int LLAMA_BUILD_NUMBER = 0;
char const *LLAMA_COMMIT = "unknown";
char const *LLAMA_COMPILER = "MSVC 19.29.30153.0";
char const *LLAMA_BUILD_TARGET = "x64";

No idea yet how to automatize this

happy Easter!

[antonio....@gmail.com]

I missed to use /MT when building the lib:

set CFLAGS=/EHsc /MT /Ox /O2 /arch:AVX512 /Oi /Ot /Oy /GS- /sdl- /Zc:inline /favor:INTEL64 -D_CRT_SECURE_NO_WARNINGS

also when building the EXE:

set HB_USER_CFLAGS=/MT

speed is very fine here using Intel Xeon, 32 gb ram

[antonio....@gmail.com]

In our tests the llama64.dll and Windows GUI use give the best performance:

https://github.com/FiveTechSoft/screenshots/blob/master/fwh_ai.gif?raw=true

The above url shows a recorded GIF with the speed we get

best regards, 

[cod...@outlook.com]

Hi Antonio,

Just to report.

I have started llama from beginning. Reason is that my first versions of llama64.lib and test1,exe somehow worked. Then after some changes in .bat files error appeared.

My computer is 64 bit Windows 10, 4 GB RAM.

I have made fresh new go64lib.bat and go64tst1.bat, first for making lib and second for building example test1.prg. Also, used my first clone of llama_prg, created on 26 march.

These fresh new go*.bat I made copying them from your first .bat suggestions, in mail on 26 march. I got llama64.lib (size about 17 MB)  and test1.exe (size about 3 MB). After starting test1.exe I wait about 40 sec, and response appears now with decent speed. Same was with default GT and with gtwvt. Used phi-2_Q4_K_M.gguf model.

Then I updated go64tst1.bat with one from you mail on 29 march. Again test1.exe works good.

Then I made new go*.bat based on your suggestions in mail on 30 march. New llama64.lib was about 24.5 MB and test1.exe was about 2 MB. Unfortenatly on my computer test1.exe dies, when calling function LLM_OPEN_MODEL(), as I explained. I changed gguf to TinyLlama.gguf but no success. After that I have tried with cloned new llama.cpp, also no success, same error. No other suggested changes helped me. Always same error on LLM_OPEN_MODEL().

Regards,

Simo.

[antonio....@gmail.com]

Dear Simo,

many thanks for your feedback

best regards,

[antonio....@gmail.com]

great finding thanks to Felix!

https://justine.lol/matmul/

[cod...@outlook.com]

Hi.

Can some Linux expert help me ?

I wanted to try llama_prg with Harbour on Linux. Windows is my main development environment, my knowledge on Linux and Harbour on Linux is near newbie level.

I have installed Kubuntu 22.04.4 LTS, as dual boot with Windows 10. Home folder is gama.

Also installed Harbour on Kubuntu with no errors, in  /home/gama/harbour. Made some hello.prg and compiled with success.

Then I cloned llama_prg,

git clone https://gitflic.ru/project/alkresin/llama_prg.git

and got  /home/gama/llama_prg.  In directory llama_prg I created two subdirectories,  lib and obj/gcc.

After that I have created two sh files, go64lib.sh for building libllama.a  and  go64tst1.sh for building test1 Linux executable.

Content for these .sh files is what Alexander suggested in its llama readme file.

go64lib.sh :

#!/bin/bash

# Set your Harbour path here

export HRB_DIR=/home/gama/harbour

export CFLAGS="-c -Wall -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Illama.cpp -Illama.cpp/common -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DLOG_DISABLE_LOGS=1 -c -I$HRB_DIR/include"

export CXXFLAGS="-c -Wall -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Illama.cpp -Illama.cpp/common -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DLOG_DISABLE_LOGS=1 -c -xc++ -I$HRB_DIR/include"

 

gcc $CXXFLAGS -oobj/gcc/hllama.o source/hllama.cpp

gcc $CXXFLAGS -oobj/gcc/llama.o llama.cpp/llama.cpp

gcc $CFLAGS -oobj/gcc/ggml.o llama.cpp/ggml.c

gcc $CFLAGS -oobj/gcc/ggml-alloc.o llama.cpp/ggml-alloc.c

gcc $CFLAGS -oobj/gcc/ggml-backend.o llama.cpp/ggml-backend.c

gcc $CFLAGS -oobj/gcc/ggml-quants.o llama.cpp/ggml-quants.c

gcc $CXXFLAGS -oobj/gcc/common.o llama.cpp/common/common.cpp

gcc $CXXFLAGS -oobj/gcc/sampling.o llama.cpp/common/sampling.cpp

gcc $CXXFLAGS -oobj/gcc/grammar-parser.o llama.cpp/common/grammar-parser.cpp

gcc $CXXFLAGS -oobj/gcc/build-info.o llama.cpp/common/build-info.cpp

ar rc  lib/libllama.a  obj/gcc/hllama.o obj/gcc/llama.o obj/gcc/ggml.o obj/gcc/ggml-alloc.o obj/gcc/ggml-backend.o obj/gcc/ggml-quants.o obj/gcc/common.o obj/gcc/sampling.o obj/gcc/grammar-parser.o obj/gcc/build-info.o

 

go64tst1.sh :

#!/bin/bash

# Set your Harbour path here

export HRB_DIR=/home/gama/harbour

 

$HRB_DIR/bin/harbour -n -q -i$HRB_DIR/include test1.prg

gcc -c -Wall -I$HRB_DIR/include -otest1.o test1.c

gcc  test1.o -otest1 -L$HRB_DIR/lib/linux/gcc -Llib -Wl,--start-group -lgttrm -lhbvm -lhbrtl -lgtcgi -lgttrm -lhbcpage -lhblang -lhbrdd -lhbmacro -lhbpp -lrddntx -lrddcdx -lrddfpt -lhbsix -lhbcommon -lhbct -lhbcplr -lhbpcre -lhbzlib -lllama -lpthread -lm -lz -lpcre -ldl -Wl,--end-group -fPIC -O3 -Wall -lstdc++ -shared-libgcc

Starting ./go64lib.sh I have created libllama.a in /home/gama/llama_prg/lib directory.

With go64tst1.sh got error you can see in attachment. Line 5 is:     $HRB_DIR/bin/harbour -n -q -i$HRB_DIR/include test1.prg .

In directory /gama/harbour/bin I do not have subdirectory with name harbour. Do I need to create it ? Also can not understand other error messages.

Can somebody help me what to do ?

Regards,

Simo.

[Gerald Drouillard]

Do you have a "/home/gama/harbour/bin/harbour" file?

--
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
Unsubscribe: harbour-user...@googlegroups.com
Web: https://groups.google.com/group/harbour-users
---
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to harbour-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/harbour-users/4b8a270a-603d-4fc5-a984-5e6adcf34471n%40googlegroups.com.

[cod...@outlook.com]

Hi Gerald.

That is what confuses me. I found that in /home/gama/harbour/bin directory only have:

3rdpatch.hb

check.hb

commit.hb

harbour.ucf

linux   , this is directory

That is all I have after Harbour installation.

I do not know is /home/gama/harbour/bin/harbour file or directory ? 

Regards, 

Simo.

[Gerald Drouillard]

Did you build harbour from the source?

--
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
Unsubscribe: harbour-user...@googlegroups.com
Web: https://groups.google.com/group/harbour-users
---
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to harbour-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/harbour-users/6e96b942-dd59-4a38-8131-84891c0ce851n%40googlegroups.com.

[cod...@outlook.com]

Hi Gerald.

Yes, I built Harbour from the source. 

Mainly used this link  https://blog.faq400.com/en/03-open-source-en/harbor-linux-installation-guide/  .

Regards,

Simo.

[Gerald Drouillard]

How about your harbour/bin/linux/gcc ?

--
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
Unsubscribe: harbour-user...@googlegroups.com
Web: https://groups.google.com/group/harbour-users
---
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to harbour-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/harbour-users/6dee06a3-1ed0-46bb-a9e9-60b235f9968fn%40googlegroups.com.

[cod...@outlook.com]

Hi Gerald.

Here is attachment . As I can see  I have same content.

Regards,

Simo.

[Gerald Drouillard]

did you run:

ldconfig

after the build?

--
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
Unsubscribe: harbour-user...@googlegroups.com
Web: https://groups.google.com/group/harbour-users
---
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to harbour-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/harbour-users/de2982d9-ad5b-4812-a1ed-68514362e178n%40googlegroups.com.

[cod...@outlook.com]

Hi Gerald.

No, I did not run idconfig after build. 

Installation steps I used did not mention that.

Where to go and what to type ?

Regards,

Simo.

[Gerald Drouillard]

just type: 

ldconfig

--
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
Unsubscribe: harbour-user...@googlegroups.com
Web: https://groups.google.com/group/harbour-users
---
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to harbour-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/harbour-users/04263d1a-9c90-43fc-b619-11654dced4b4n%40googlegroups.com.

[cod...@outlook.com]

Hi Gerald.

I did it with    sudo ldconfig  

Nothing happened, just new line. Nothing new in   /harbour/bin    directory.

Regards,

Simo.

[Gerald Drouillard]

does building your app work now?

--
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
Unsubscribe: harbour-user...@googlegroups.com
Web: https://groups.google.com/group/harbour-users
---
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to harbour-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/harbour-users/c06730c8-c7fd-4679-91e7-cd931821e09bn%40googlegroups.com.

[cod...@outlook.com]

Hi Gerald.

Meanwhile after ldconfig I tried again to build app, but with no success. Same result. Same error.

Regards,

Simo.

[antonio....@gmail.com]

Enhanced llama.cpp support for images recognition from Harbour:

If you are interested in this feature please email me and I will assist you.

best regards

[antonio....@gmail.com]

[alkr...@yahoo.com]

> When I start  test1.exe, windows opens, I select model, can see message Loading ... and test1.exe dies. Nothing on screen.

> As I could see test1.exe dies on function  llm_Open_Model( aModels[i,1] ) i

  I tried it with Antonio's new bat files and test1.exe dies on opening model, too.

So, I strongly recommend to use the bat, which I provided, or insert flags from there to your's. Take attention that flags for c and cpp files should be different.

You may  simplify my bat file, replacing four lines ( SET PATH, SET INCLUDE, SET LIB, SET LIBPATH ) with this one:

call "c:\Program Files (x86)\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64

Regards, Alexander.

[alkr...@yahoo.com]

> We have noticed that Alexander is not using latest llama.cpp files.

  I use the llama.cpp repository of March 4, 2024 - the day when I started dig in it.

I will update it from time to time.

Regards, ALexander.

[cod...@outlook.com]

Hi Alexander.

Thank you. I will try with your instructions.

Also good news are from Antonio, regarding support for images recognition.

Regards,

Simo.

[alkr...@yahoo.com]

Simo,

> With go64tst1.sh got error you can see in attachment. Line 5 is:     $HRB_DIR/bin/harbour -n -q -i$HRB_DIR/include test1.prg .

> In directory /gama/harbour/bin I do not have subdirectory with name harbour.

  This isn't a subdirectory name here, it an executable file - harbour compiler. As I understand, you have it in harbour/bin/linux/gcc/ directory

So, you need to change this line in your go64tst1.sh:

   $HRB_DIR/bin/harbour -n -q -i$HRB_DIR/include test1.prg

to

   $HRB_DIR/bin//linux/gcc/harbour -n -q -i$HRB_DIR/include test1.prg

[cod...@outlook.com]

Hi Alexander.

Just one more help.

I changed line 5 to $HRB_DIR/bin//linux/gcc/harbour -n -q -i$HRB_DIR/include test1.prg    and that is good.

I have another error. You can see it in my attachment.

What to do to resolve errors with lhbpcre and lhbzlib ?

 

Thank you and best rergards,

Simo.

[alkr...@yahoo.com]

Simo,

 just delete -lhbpcre -lhbzlib from the list of libraries ( the last line of the sh ).

Regards, Alexander

вторник, 9 апреля 2024 г. в 19:07:38 UTC+3, cod...@outlook.com:

[cod...@outlook.com]

Hi Alexander.

I have deleted  -lhbpcre -lhbzlib   libs. Building app succeeded and I got test1 executable.

Copied gguf model file in folder and started test1.

When I entered question "explain speed of light"    :)   answer started instantly ! Speed excellent !

There days I will do it again on Windows according to new instructions. 

I must admit that it is all more easier on Linux than on Windows.

Thank you and best regards.

Simo.

[antonio....@gmail.com]

Dear Alexander,

Gregori and his team are constantly updating the llama.cpp project thats why we always do a git pull from his repo before building

best regards

[alkr...@yahoo.com]

Antonio,

 yes, I see, they are doing a tremendous job.

I've updated the llama.prg, using the latest sources of llama.cpp just now.

Regards, Alexander.

[cod...@outlook.com]

Hi Alexander.

Just to report what I did with llama_prg on Windows.

I have managed to build llama.lib and test1.exe using your .bat files, according to your suggestions.

As I have installed Visual Studio Community 2022, a tried to set paths in your .bat files to my situation, but did not succeeded.

So I did what you suggested: I simplified bat files replacing four lines ( SET PATH, SET INCLUDE, SET LIB, SET LIBPATH ) with this one:

call "c:\Program Files (x86)\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64

After that I built llama.lib and test1.exe.

I made test1.exe with default GT. After I start test1.exe I must wait about 45-50 seconds to see response. But response works this way: On the screen is written first sentence of response. Then same sentence continues to repeat, until screen is filled with that repeating text.

Same happened with test1.exe using gtvwt.

I worked with llama_prg version before April 10.

This does not happened on Linux. There is no wait time, and response is normal, all correct response is displayed. No repeating any part of text.

Regards,

Simo.

[alkr...@yahoo.com]

Simo,

  the generated answer depends on a model used, on a memory size currently available (if you have 4Gb only, even closing a browser may help) and on a question itself. If a model is small and it doesn't understand the question (due to grammatical error or some other reason), it may produce something like you see.

I have 4Gb on my home computer and sometimes I get the same.

Regards, Alexander.

среда, 10 апреля 2024 г. в 20:11:37 UTC+3, cod...@outlook.com:

[alkr...@yahoo.com]

Hello,

 I've added a possibilty to process image files, based on llava sample in llama.cpp.

The library should be rebuild with changed make scripts (bat, sh), since few source files added.

Use test3.prg to see how it works.

The following models: ggml-model-q5_k.gguf, mmproj-model-f16.gguf should be downloaded from https://huggingface.co/mys/ggml_llava-v1.5-7b

Regards, Alexander

[cod...@outlook.com]

Hi Alexander.

On Windows could not make llama.lib with clip.cpp and llava.cpp.

Then I built llama.lib without this two .cpp files.

After that I made test1.exe. It works OK, now response displayed with no repeating text (used phi-2_Q4_K_M.gguf).

If you have time please suggest how to solve problem with clipp.cpp. I am sending bat file (renamed to txt) I used to build lib. Also I am sending file with output of that bat file with error for clip.cpp (and llava.cpp).

 

Thank you and Regards,

Simo.

[alkr...@yahoo.com]

Simo,

that's my fault. There was a bug in readme, I fixed it already.

Instead of

cl.exe %CXXFLAGS% /Foobj\msvc64\common.obj llama.cpp\llava\clip.cpp

cl.exe %CXXFLAGS% /Foobj\msvc64\common.obj llama.cpp\llava\llava.cpp

there should be

cl.exe %CXXFLAGS% /Foobj\msvc64\clip.obj llama.cpp\llava\clip.cpp
cl.exe %CXXFLAGS% /Foobj\msvc64\llava.obj llama.cpp\llava\llava.cpp

BTW, tests are updated. It is possible now to set model parameters directly in a command line. Also, in test3.prg, which demonstrates the image processing, it is possible now to set your own prompt, for example: "How many girls are here?". There is not a warranty, though, that it will count them correctly). Probably, bigger model is needed.

Regards, Alexander.

пятница, 12 апреля 2024 г. в 10:58:20 UTC+3, cod...@outlook.com:

[cod...@outlook.com]

Hi Alexander.

Thank you for your prompt answer.

Now I built llama.lib with clip.cpp.

I have cloned llama_prg this morning, so I already have updated project and examples. 

I made test2.exe example, started it from command line with "test3 \AImodels\ggml-model-q5_k.gguf \AImodels\mmproj-model-f16.gguf some.jpg". It works several minutes with

Loading

Image processing ...

I do not know what to expect as result. Do I need to make test3.exe with gwvt ?

Regards,

Simo.

[alkr...@yahoo.com]

Simo,

  After the "Image processing" an answer should appear - a comment about an image file. The time of response depends on the computer's hardware, it memory size.

On my pc with 12Gb RAM I wait about 2-3 minutes.

I recommend also to reduce the image size till ~500 pixels, maybe less.

And I don't think that a type of terminal has any sense here.

Regards, Alexander.

[cod...@outlook.com]

Thank you Alexander.

Yes, comment on image appeared, at speed according to my computer strength.

I will try with different image parameters,.

Regards,

Simo.

[cod...@outlook.com]

Hi Alexander.

I tried new version of llama_prg on Linux.

When building library I have errors that it cannot find some header files, clip.h and llava.h .

Then I added -Illama.cpp/llava to next lines:

gcc $CXXFLAGS -oobj/gcc/hllama.o -Illama.cpp/llava source/hllama.cpp

gcc $CXXFLAGS -oobj/gcc/hllava.o -Illama.cpp/llava source/hllava.cpp

…

…

gcc $CXXFLAGS -oobj/gcc/clip.o -Illama.cpp/llava llama.cpp/llava/clip.cpp

gcc $CXXFLAGS -oobj/gcc/llava.o -Illama.cpp/llava llama.cpp/llava/llava.cpp

After that I succeeded to build library.

Did not have problems with building examples. Executable test1 works OK. No wait time, response start immediately.

With executable test3  ggml-model-q5_k.gguf  mmproj-model-f16.gguf myimg.jpg  I got answer "Loading ..", then nothing in next one minute, and finally it finished without any comment on image. I used small JPG image, same as on Windows.

Regards,

Simo.

[alkr...@yahoo.com]

Simo,

Does your tets3.prg contains the following?

   IF llm_Clip_Open( cModelI, cModelM, cImage ) != 0
      ? " === Can't init llava ==="
      RETURN Nil
   ENDIF

Regards, Alexander

пятница, 12 апреля 2024 г. в 21:45:17 UTC+3, cod...@outlook.com:

[cod...@outlook.com]

Hi Alexander.

No, it does not.

It containes:

   llm_Clip_Open( cModelI, cModelM, cImage, cPrompt )
   ? "Image processing..."
   llm_Clip_SetImage()

Regards,

Simo.

[cod...@outlook.com]

Sorry, sorry ...

On Linux it contains ! On Linux I have most new version.

Regards, Simo.

[alkr...@yahoo.com]

I can't test it on Linux now, because my Linux computer has 4Gb RAM only and test3 doesn't work here at all, it can't load the models. I will find appropriate Linux PC later and test it.

Regards, Alexander.

[alkr...@yahoo.com]

I've added few additional error messages for llm_Clip_Open() - if it doesn't work, they may appear in stderr.log, something like "Failed to load model" or "Failed to create context".

Regards, Alexander.

[cod...@outlook.com]

Hi Alexander.

Test on Linux:

On Sunday morning I cloned fresh new llama_prg .

Shell commands from your README.md for creating library works OK.

Shell commands your README.md for creating examples works OK.

Executables test1 and test2 works OK.

Test3 do not work. It works as I already described. Image is about 5K. I am sending stderr.log from my computer. May be it can help.

Test was on laptop with 16 GB RAM, Intel i7, with 48 GB free on disk.

Regards,

Simo.

[cod...@outlook.com]

Hi Alexander.

New test on Windows:

On Sunday afternoon I made fresh new llama_prg .

Windows commands from your README.md for creating library works OK.

Windows commands from your README.md for creating examples works OK.

I have only simplified bat files replacing four lines ( SET PATH, SET INCLUDE, SET LIB, SET LIBPATH ) with call "c:\Program Files (x86)\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64, as you suggested.

Test1.exe and test2.exe works OK. Wait time to respond was about 30 seconds. Before this version it was about 50 seconds. This time better.

Test3 works this way:

First I see "Loading …".  Then after 60 seconds appears "Image processing …" (this message do not exist on Linux). Then I must wait about 10 minutes to see start of response. Response is very, very slow, but it describes image. 

As you explained this is normal behavior for my computer.

Test done on desktop PC with 4 GB RAM, old Intel i5, 120 GB free on non SSD disk, Windows 10.

 

Regards,

Simo. 

[cod...@outlook.com]

Hi Alexander.

Most new test on Windows 64 bit. I have installed all on computer with better configuration.

Test was on laptop with 16 GB RAM, Intel i7, with 100 GB free on SSD disk.

I have cloned Llama_prg on monday morning.

I have created library and examples without any problems.

Test1.exe and test2.exe works OK. No wait time, response started immediately. Clean ans fast.

Test3 also works very good:

First I see "Loading …".  Message "Image processing …" appears immediately.

Then I must wait about 60 seconds to see start of response. Response was at normal speed, describing image. No pauses in response. 

 

Regards,

Simo. 

[alkr...@yahoo.com]

Simo,

 I changed a bit test3.prg and I think that it should work properly on Linux, too.

Regards, Alexander

воскресенье, 14 апреля 2024 г. в 20:24:21 UTC+3, cod...@outlook.com:

[cod...@outlook.com]

Hi Alexander.

Test on Kubuntu Linux:

Laptop with 16 GB RAM, Intel i7, with 40 GB free on SSD disk.

I have cloned Llama_prg on Tuesday morning.

I have created library and examples without any problems.

Test1.exe and test2.exe works OK. No wait time, response started immediately. Clean and fast.

Test3 also works OK:

First I see "Loading …".  Message "Image processing …" appears immediately.

Then I must wait about 35 seconds to see start of response. Response is at normal speed. It describes image.  At the end it displays some statistics about image processing.

 

Regards,

Simo. 

[alkr...@yahoo.com]

Whisper support added to llama.prg. It is  bindings for https://github.com/ggerganov/whisper.cpp (thanks to Antonio for pointing out this project).

See test4.prg as a sample:

test4 <path_to_model> <path_to_wav> [cParamsString]

It allows to recognize an audio file in wav format  16kHz.

Appropriate models may be found at https://huggingface.co/ggerganov/whisper.cpp

Regards, Alexander

[alkr...@yahoo.com]

I[ve added a related project - stable-diffusion.prg, based on stable-diffusion.cpp (thanks to Nikolay Kozlov for pointing out to it) - generating images by text prompts.

Regards, Alexander

четверг, 18 апреля 2024 г. в 16:19:50 UTC+3, alkr...@yahoo.com:

[cod...@outlook.com]

Hi Alexander,

Can you suggest us which model file to use with  stable_diffusion_prg. 

Thank you and Best Regards,

Simo.

[cod...@outlook.com]

Hi Alexander,

Meanwhile I found in your *.prg that model files can be found on link https://huggingface.co/CompVis/stable-diffusion-v-1-4-original.

For now I tried test1.exe, on Windows.

I used  sd-v1-4.ckpt  model file. As prompt I entered one word    duck.

Result was PNG image of nice duck in color. 

Regards

Simo.

[alkr...@yahoo.com]

You may try also some other links from stable-diffusion.cpp, for example, https://huggingface.co/runwayml/stable-diffusion-v1-5 or https://huggingface.co/stabilityai/stable-diffusion-2-1

I was recommended this one: https://huggingface.co/justinpinkney/miniSD/tree/main

Or may find something else, using google or huggingface search.

Regards, Alexander

[alkr...@yahoo.com]

I've added to Llama.prg a small cross-platform module for recording from a microphone and a new example - test5.prg, which demonstrates how to dictate text and then recognize it. If the computer resources allow, it turns out relatively quickly. You can use this with the editor, or use it to recognize voice commands.

Regards, Alexander.

[cod...@outlook.com]

Hi Alexander.

For building whisper.lib on Windows i had to change line:

cl.exe %CXXFLAGS% /Fo%OBJ%\hwhisper.obj llama.cpp\source\hwhisper.cpp

to this line:

cl.exe %CXXFLAGS% /I%HB_PATH%\include  /Fo%OBJ%\hwhisper.obj source\hwhisper.cpp 

Regards,

Simo.

[cod...@outlook.com]

Hi Alexander.

When building example test5 on Windows cannot find next functions:

audi_setcallback

audi_record

audi_abort

Regards,

Simo.

[alkr...@yahoo.com]

Simo, you need to add source/audiorecord.c to your build script to compile and link it (look at the test5.hwprj).

Regards, Alexander

пятница, 3 мая 2024 г. в 18:24:24 UTC+3, cod...@outlook.com: