SF.net SVN: harbour-project:[17404] trunk/harbour
dru...@users.sourceforge.net
http://harbour-project.svn.sourceforge.net/harbour-project/?rev=17404&view=rev
Author: druzus
Date: 2012-04-20 15:55:44 +0000 (Fri, 20 Apr 2012)
Log Message:
-----------
2012-04-20 17:52 UTC+0200 Przemyslaw Czerpak (druzus/at/poczta.onet.pl)
* harbour/include/hbdefs.h
* moved HB_WCHAR definition from hbapicdp.h to hbdefs.h
* harbour/include/hbapicdp.h
* harbour/include/hbcdpreg.h
* harbour/src/rtl/cdpapi.c
* harbour/src/rtl/cdpapihb.c
+ added support for custom sorting redirected from HVM
+ added support for custom character indexes in strings
+ added support for custom character flags (upper, lower, alpha, digit)
+ added support for custom upper/lower conversions
+ added support for CPs using unicode character values instead of
ASCII ones
+ added new CP functions which respects custom CP settings:
hb_cdpUpperWC(), hb_cdpTextLen(), hb_cdpTextPos(),
hb_cdpTextPosEx(), hb_cdpTextGetU16(), hb_cdpTextPutU16(),
hb_cdpCharEq(), hb_cdpCharCaseEq()
+ added new conversion functions:
hb_cdpGetUC(), hb_cdpGetWC(), hb_cdpGetU16Ctrl()
+ added macros to detect codepage parameters:
HB_CDP_ISBINSORT() - codepage uses simple binary sorting
HB_CDP_ISCUSTOM() - codepage uses custom string decoding
HB_CDP_ISCHARIDX() - codepage use character indexes instead
of bytes ones
HB_CDP_ISCHARUNI() - CHR(), ASC() and similar functions operates
on Unicode values instead of bytes
HB_CDP_ISUTF8() - codepage uses UTF-8 encoding
* harbour/include/inkey.ch
- removed HB_INKEY_EXTENDED - it was not used in Harbour
+ added new flag HB_INKEY_EXT - it allows to used different
event encoding system with automatic translation to
standard Clipper values. It will be used in the future
as base for low level GTs code. Now it's possible to
selectively switch to the new system. New codes are
unique and do not interacts with Clipper.
+ added HB_INKEY_ALL macro - it's similar to INKEY_ALL but
enables also GTEVENT keys.
* harbour/include/hbgtcore.h
* harbour/src/rtl/hbgtcore.c
* changed character values in internal screen buffer to unicode.
Warning: this modification interacts with SAVESCREE()/RESTSCREEN()
data. Now all GTs uses extended definition. If somene
has code which needs VGA like screen buffers returned
by SAVESCREE() then it can be forced by
hb_gtInfo( HB_GTI_COMPATBUFFER, .t. )
Anyhow it will for to use only one CP in output.
+ added new GT methods which operated on HB_WCHAR values
* modified existing methods using HB_WCHAR/HB_USHORT parameters
to operate on HB_WCHAR values
+ added codepage conversion code directly to core code so it can
be eliminated from low level GT drivers which do not need some
special operations on it.
+ added support for extended inkey codes, they allow to encoded
unicode values, mouse events with flags, keyboard events with
modifier flags (shift,alt,ctrl,...), etc.
Now in low level GT code only unicode key values are used but
I plan to updated all GTs and switch to new code - it should
resolve many small problems inheriting with Clipper inkey codes.
* harbour/include/hbapifs.h
* moved hb_fsNameConvU16() definition from hbapicdp.h to hbapifs.h
* harbour/include/hbxvm.h
- removed comment copied by mistake from GT header file
* harbour/include/hbwinuni.h
* changed HB_CHARDUP() and HB_CHARDUPN() macros - now they
operate on functions which can be used without active HVM.
In such case ANSI CP is used as source encoding.
+ added new macros: HB_OSSTRDUP() and HB_OSSTRDUP2().
They make conversions from OS encoding to HVM one.
They can be used without active HVM and in such case ANSI CP
is used as destination encoding.
* harbour/include/hbapi.h
* harbour/src/vm/cmdarg.c
+ added new function hb_cmdargProgName()
It returns application name with path or NULL if not set,
caller must free returned value with hb_xfree() if not NULL.
The string is in HVM encoding so it does not need any additional
encodings.
* modified hb_cmdargString() to return strings in HVM encoding.
% some small code optimizations
; NOTE: parameters passed to MS-WINDOWS GUI programs which use
WinMain() instead of main() as startup entry are translated
to ANSICP before they can be accessed by application.
This can be eliminated though not for console programs
using where C compiler makes such translation to main()
parameters.
in all systems parameters are passed to application
startup and init functions before programmer can set OS CP.
This can be resolved by adding:
INIT PROC CLIPINIT()
SET( _SET_CODEPAGE, <userCP> )
SET( _SET_OSCODEPAGE, <systemCP> )
RETURN
to linked code. INIT procedures called CLIPINIT() are
executed before any other ones.
* harbour/include/hbapi.h
* harbour/src/vm/set.c
+ added new functions for CP converisons which can be used with
and without active HVM:
hb_osStrEncode(), hb_osStrEncodeN(),
hb_osStrDecode(), hb_osStrDecode2(),
hb_osStrU16Encode(), hb_osStrU16EncodeN(),
hb_osStrU16Decode(), hb_osStrU16Decode2(),
* harbour/include/hbvm.h
* harbour/src/vm/hvm.c
+ added new function hb_vmIsReady() - it's similar to hb_vmActive()
but it also checks if current thread has active HVM stack.
* moved hb_cdpReleaseAll() to the end of HVM cleanup code so
CP conversions are longer active.
* harbour/include/hbcomp.h
* harbour/src/compiler/cmdcheck.c
* harbour/src/compiler/hbusage.c
* harbour/src/common/expropt2.c
+ added new compiler switch:
-ku - strings in user encoding
Now it informs compiler that strings use custom encoding so some
optimizations which are byte oriented cannot be used.
It's possible that in the future we will change above definition
to sth like: "strings in UTF8 encoding" but now I would like to
keep more general.
* harbour/src/vm/macro.c
* inform macrocompiler about custom CPs using own character indexes
to disable byte oriented optimizations.
* harbour/include/hbapigt.h
* harbour/src/rtl/gtapi.c
* harbour/src/rtl/inkeyapi.c
+ added HB_B_*_W macros with unicode box character definitions
+ added HB_MBUTTON_* macros
+ added new GT functions: hb_gtHostCP() and hb_gtBoxCP() which
allows to extract CPs used in translations by GTs.
+ added new function hb_inkeyKeyString() - it converts inkey value
to corresponding string
+ added new function hb_inkeyKeyStd() - it converts new extended
key value to standard Clipper one.
* harbour/include/hbapifs.h
* harbour/src/common/hbffind.c
* moved OS codepage translations fully to hb_fsFind*() functions.
It fixes few problems which existed before, i.e. double CP
conversions in MS-Windows builds and simplifies upper level code.
* harbour/src/pp/hbpp.c
* harbour/src/rtl/direct.c
* harbour/src/rtl/fssize.c
* harbour/src/rtl/file.c
* harbour/contrib/hbct/files.c
* eliminated not longer necessary CP conversions in code calling
hb_fsFind*() functions.
* harbour/src/common/hbgete.c
* moved OS codepage translations to hb_getenv(), hb_getenv_buffer()
and hb_setenv() functions. It fixes few problems which existed
before, i.e. double CP conversions in MS-Windows builds and
simplifies upper level code.
* harbour/src/rtl/net.c
* harbour/src/rtl/gete.c
* eliminated not longer necessary CP conversions in code calling
hb_getenv()/hb_setenv() functions.
; NOTE: additional parameters in HB_GETENV() and HB_SETENV() which
disabled CP conversions are not longer supported.
They were strictly platform dependent and ignored in chosen
cases (i.e. in MS-Windows UNICODE builds we always have to
convert strings transferred between HVM and OS. If someone
needs old functionality for other platform then he should
temporary disable _SET_OSCODEPAGE.
* harbour/src/common/hbver.c
* harbour/src/common/hbfsapi.c
* harbour/src/rtl/fstemp.c
* harbour/src/rtl/fslink.c
* eliminated HB_TCHAR_*() macros
* harbour/src/common/strwild.c
+ added supprot for custom CPs using own character indexes in:
hb_strMatchWild(),
hb_strMatchWildExact(),
hb_strMatchCaseWildExact()
* harbour/src/nortl/nortl.c
+ added new dummy function replacement for binaries which are not
linked with HVM.
* harbour/src/rtl/filesys.c
! fixed double CP conversions in MS-Windows builds of hb_fsCurDirBuff()
* use hb_vmIsReady() instead of hb_stackId() in file name conversions.
* use hb_cmdargProgName() in hb_fsBaseDirBuff()
* harbour/src/rtl/philes.c
* use hb_cmdargProgName() in HB_PROGNAME() function.
* harbour/src/rtl/gtcgi/gtcgi.c
* harbour/src/rtl/gtstd/gtstd.c
* harbour/src/rtl/gtpca/gtpca.c
* harbour/src/rtl/gtdos/gtdos.c
* harbour/src/rtl/gtos2/gtos2.c
* harbour/src/rtl/gtwin/gtwin.c
* harbour/src/rtl/gtwvt/gtwvt.h
* harbour/src/rtl/gtwvt/gtwvt.c
* harbour/src/rtl/gttrm/gttrm.c
* harbour/src/rtl/gtcrs/gtcrs.c
* harbour/src/rtl/gtsln/gtsln.c
* harbour/src/rtl/gtsln/kbsln.c
* harbour/src/rtl/gtsln/gtsln.h
* harbour/src/rtl/gtxwc/gtxwc.h
* harbour/src/rtl/gtxwc/gtxwc.c
* harbour/contrib/gtwvg/gtwvg.c
* harbour/contrib/gtwvg/gtwvg.h
* harbour/contrib/gtalleg/gtalleg.c
* harbour/contrib/hbqt/gtqtc/gtqtc.cpp
* harbour/contrib/hbqt/gtqtc/gtqtc.h
* updated to work with new unicode GT API
please make tests with different GTs - I'm not able to test
all of them, i.e. I do not have any OS2 machine.
* harbour/src/rtl/box.c
* harbour/src/rtl/oldbox.c
* harbour/src/rtl/scroll.c
* harbour/src/rtl/console.c
* updated to work with new unicode GT API and
CPs using custom character indexes
* harbour/src/rtl/at.c
* harbour/src/rtl/ati.c
* harbour/src/rtl/rat.c
* harbour/src/rtl/len.c
* harbour/src/rtl/transfrm.c
* harbour/src/rtl/left.c
* harbour/src/rtl/right.c
* harbour/src/rtl/substr.c
* harbour/src/rtl/stuff.c
* harbour/src/rtl/padc.c
* harbour/src/rtl/padl.c
* harbour/src/rtl/padr.c
* updated to work with CPs using custom character indexes
* harbour/src/rtl/chrasc.c
+ added support for HB_CDP_ISCHARUNI() CPs.
* harbour/src/rtl/mlcfunc.c
* rewritten from scratch to work with CPs using custom character
indexes
* harbour/src/rtl/accept.c
* updated to work with unicode inkey values and
CPs using custom character indexes
* harbour/src/rtl/strmatch.c
% small optimization
* harbour/src/rtl/Makefile
+ harbour/src/rtl/chruni.c
+ added new PRG functions which allows to make byte/binary and
unicode/character operations on strings:
HB_UCHAR( <nCode> ) -> <cText>
return string with U+nCode character in HVM CP encoding
HB_BCHAR( <nCode> ) -> <cText>
return 1 byte string with <nCode> value
HB_UCODE( <cText> ) -> <nCode>
return unicode value of 1-st character (not byte) in given string
HB_BCODE( <cText> ) -> <nCode>
return value of 1-st byte in given string
HB_ULEN( <cText> ) -> <nChars>
return string length in characters
HB_BLEN( <cText> ) -> <nBytes>
return string length in bytes
HB_UPEEK( <cText>, <n> ) -> <nCode>
return unicode value of <n>-th character in given string
HB_BPEEK( <cText>, <n> ) -> <nCode>
return value of <n>-th byte in given string
HB_UPOKE( [@]<cText>, <n>, <nVal> ) -> <cText>
change <n>-th character in given string to unicode <nVal> one and
return modified text
HB_BPOKE( [@]<cText>, <n>, <nVal> ) -> <cText>
change <n>-th byte in given string to <nVal> and return modified
text
* harbour/src/rtl/hbdoc.prg
* harbour/src/rtl/memvarhb.prg
* use HB_BCHAR() for binary string definitions
* harbour/src/rtl/hbi18n2.prg
* use hb_utf8CHR( 0xFEFF ) instead of hardcoded binary sting and
HB_BLEN() instead of LEN()
* harbour/src/rtl/inkey.c
+ added new functions:
HB_KEYCHAR( <nKey> ) -> <cChar>
HB_KEYSTD( <nExtKey> ) -> <nClipKey>
! use HB_INKEY_ALL instead of INKEY_ALL in LASTKEY()
* harbour/src/rtl/achoice.prg
* harbour/src/rtl/browse.prg
* harbour/src/rtl/menuto.prg
* harbour/src/rtl/tgetlist.prg
* harbour/src/rtl/teditor.prg
* harbour/src/rtl/tlabel.prg
* harbour/src/rtl/tpopup.prg
* harbour/src/rtl/radiobtn.prg
* harbour/src/rtl/radiogrp.prg
* harbour/src/rtl/wait.prg
* updated to work with unicode inkey() values
* harbour/src/rtl/listbox.prg
* use box.ch macros instead of explicit CHR(...) definitions
* harbour/src/rtl/ttopbar.prg
* updated to work with different type of CPs
* harbour/src/rtl/scrollbr.prg
* formatting
* harbour/src/rtl/mouse53.c
* use HB_MBUTTON_* macros instead of local ones
* harbour/src/codepage/cp_utf8.c
+ harbour/src/codepage/uc16def.c
+ harbour/src/codepage/utf8sort.c
* replaced UTF8ASC with new CP: UTF8EX
This CP uses character indexes instead of bytes one
and operates on unicode characters flags.
Tables for upper/lower conversions and upper/lower/alpha/digit
flags were generated automatically from
http://www.unicode.org/Public/UNIDATA/UnicodeData.txt
It also uses custom collation rules. It's very simple one
level sorting based on UTF8 C collation.
If someone needs some advanced sorting rules, then it's enough
to create copy of this cp with user custom version of UTF8_cmp()
and UTF8_cmpi() functions, i.e. they can be redirected to some
external library like ICU (icu-project.org).
* harbour/contrib/hbct/ctwin.c
* harbour/contrib/hbct/ctwin.h
* harbour/contrib/hbct/ctwfunc.c
* added support for new unicode GT API
* harbour/contrib/xhb/xhbfunc.c
* redirected HB_CMDARGARGV() to HB_PROGNAME()
* harbour/contrib/hbnf/origin.c
* redirected FT_ORIGIN() to HB_PROGNAME()
* harbour/contrib/hbnf/getenvrn.c
! windows version of FT_GETE() fully rewritten - it should
fix well known problems reported to the devel list. Please test.
% small optimization for other systems
* harbour/contrib/hbfship/exec.c
* use hb_cmdargProgName() in EXECNAME()
+ harbour/tests/uc16_gen.prg
+ added code which generates tables with unicode character flags
from http://www.unicode.org/Public/UNIDATA/UnicodeData.txt.
harbour/src/codepage/uc16def.c was generated by this code.
* harbour/tests/inkeytst.prg
* harbour/tests/wvtext.prg
* harbour/tests/gtkeys.prg
* use HB_INKEY_ALL
* harbour/include/harbour.hbx
* harbour/include/hbcpage.hbx
* harbour/include/hblang.hbx
* regenerated
; It was quite big peace of modifications and for sure not everything
is well tested so please make test and report problems you will find.
; This is basic version which introduce to HVM CPs with custom encodings.
Some contrib code has to be updated to work correctly with it.
I hope that developers interesting in will make necessary updates.
I haven't touched GTWVW code at all - sorry but it needs very serious
work to make it production ready and fix all existing problems.
; Special thanks to OTC - this firm sponsored adding basic UTF8 support
to HVM.
Modified Paths:
--------------
trunk/harbour/ChangeLog
trunk/harbour/contrib/gtalleg/gtalleg.c
trunk/harbour/contrib/gtwvg/gtwvg.c
trunk/harbour/contrib/gtwvg/gtwvg.h
trunk/harbour/contrib/hbct/ctwfunc.c
trunk/harbour/contrib/hbct/ctwin.c
trunk/harbour/contrib/hbct/ctwin.h
trunk/harbour/contrib/hbct/files.c
trunk/harbour/contrib/hbfship/exec.c
trunk/harbour/contrib/hbnf/getenvrn.c
trunk/harbour/contrib/hbnf/origin.c
trunk/harbour/contrib/hbqt/gtqtc/gtqtc.cpp
trunk/harbour/contrib/hbqt/gtqtc/gtqtc.h
trunk/harbour/contrib/xhb/xhbfunc.c
trunk/harbour/include/harbour.hbx
trunk/harbour/include/hbapi.h
trunk/harbour/include/hbapicdp.h
trunk/harbour/include/hbapifs.h
trunk/harbour/include/hbapigt.h
trunk/harbour/include/hbcdpreg.h
trunk/harbour/include/hbcomp.h
trunk/harbour/include/hbcpage.hbx
trunk/harbour/include/hbdefs.h
trunk/harbour/include/hbgtcore.h
trunk/harbour/include/hblang.hbx
trunk/harbour/include/hbvm.h
trunk/harbour/include/hbwinuni.h
trunk/harbour/include/hbxvm.h
trunk/harbour/include/inkey.ch
trunk/harbour/src/codepage/cp_utf8.c
trunk/harbour/src/common/expropt2.c
trunk/harbour/src/common/hbffind.c
trunk/harbour/src/common/hbfsapi.c
trunk/harbour/src/common/hbgete.c
trunk/harbour/src/common/hbver.c
trunk/harbour/src/common/strwild.c
trunk/harbour/src/compiler/cmdcheck.c
trunk/harbour/src/compiler/hbusage.c
trunk/harbour/src/nortl/nortl.c
trunk/harbour/src/pp/hbpp.c
trunk/harbour/src/rtl/Makefile
trunk/harbour/src/rtl/accept.c
trunk/harbour/src/rtl/achoice.prg
trunk/harbour/src/rtl/at.c
trunk/harbour/src/rtl/ati.c
trunk/harbour/src/rtl/box.c
trunk/harbour/src/rtl/browse.prg
trunk/harbour/src/rtl/cdpapi.c
trunk/harbour/src/rtl/cdpapihb.c
trunk/harbour/src/rtl/chrasc.c
trunk/harbour/src/rtl/console.c
trunk/harbour/src/rtl/direct.c
trunk/harbour/src/rtl/file.c
trunk/harbour/src/rtl/filesys.c
trunk/harbour/src/rtl/fslink.c
trunk/harbour/src/rtl/fssize.c
trunk/harbour/src/rtl/fstemp.c
trunk/harbour/src/rtl/gete.c
trunk/harbour/src/rtl/gtapi.c
trunk/harbour/src/rtl/gtcgi/gtcgi.c
trunk/harbour/src/rtl/gtcrs/gtcrs.c
trunk/harbour/src/rtl/gtdos/gtdos.c
trunk/harbour/src/rtl/gtos2/gtos2.c
trunk/harbour/src/rtl/gtpca/gtpca.c
trunk/harbour/src/rtl/gtsln/gtsln.c
trunk/harbour/src/rtl/gtsln/gtsln.h
trunk/harbour/src/rtl/gtsln/kbsln.c
trunk/harbour/src/rtl/gtstd/gtstd.c
trunk/harbour/src/rtl/gttrm/gttrm.c
trunk/harbour/src/rtl/gtwin/gtwin.c
trunk/harbour/src/rtl/gtwvt/gtwvt.c
trunk/harbour/src/rtl/gtwvt/gtwvt.h
trunk/harbour/src/rtl/gtxwc/gtxwc.c
trunk/harbour/src/rtl/gtxwc/gtxwc.h
trunk/harbour/src/rtl/hbdoc.prg
trunk/harbour/src/rtl/hbgtcore.c
trunk/harbour/src/rtl/hbi18n2.prg
trunk/harbour/src/rtl/inkey.c
trunk/harbour/src/rtl/inkeyapi.c
trunk/harbour/src/rtl/left.c
trunk/harbour/src/rtl/len.c
trunk/harbour/src/rtl/listbox.prg
trunk/harbour/src/rtl/memvarhb.prg
trunk/harbour/src/rtl/menuto.prg
trunk/harbour/src/rtl/mlcfunc.c
trunk/harbour/src/rtl/mouse53.c
trunk/harbour/src/rtl/net.c
trunk/harbour/src/rtl/oldbox.c
trunk/harbour/src/rtl/padc.c
trunk/harbour/src/rtl/padl.c
trunk/harbour/src/rtl/padr.c
trunk/harbour/src/rtl/philes.c
trunk/harbour/src/rtl/radiobtn.prg
trunk/harbour/src/rtl/radiogrp.prg
trunk/harbour/src/rtl/rat.c
trunk/harbour/src/rtl/right.c
trunk/harbour/src/rtl/scroll.c
trunk/harbour/src/rtl/scrollbr.prg
trunk/harbour/src/rtl/strmatch.c
trunk/harbour/src/rtl/stuff.c
trunk/harbour/src/rtl/substr.c
trunk/harbour/src/rtl/teditor.prg
trunk/harbour/src/rtl/tgetlist.prg
trunk/harbour/src/rtl/tlabel.prg
trunk/harbour/src/rtl/tpopup.prg
trunk/harbour/src/rtl/transfrm.c
trunk/harbour/src/rtl/ttopbar.prg
trunk/harbour/src/rtl/wait.prg
trunk/harbour/src/vm/cmdarg.c
trunk/harbour/src/vm/hvm.c
trunk/harbour/src/vm/macro.c
trunk/harbour/src/vm/set.c
trunk/harbour/tests/gtkeys.prg
trunk/harbour/tests/inkeytst.prg
trunk/harbour/tests/wvtext.prg
Added Paths:
-----------
trunk/harbour/src/codepage/uc16def.c
trunk/harbour/src/codepage/utf8sort.c
trunk/harbour/src/rtl/chruni.c
trunk/harbour/tests/uc16_gen.prg
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
Massimo Belgrano
Now harbour have Unicode in HVM so start test and report problem
Massimo Belgrano
Viktor Szakáts
Thank you very much Przemek. (and OTC for sponsoring)
For those interested in looking into the whole patch (f.e. to
update 3rd party code), use this command in Harbour SVN
sandbox root:
svn diff -r 17403:17404 > uni.dif
One of the next logical questions: How to enable unicode
fields in tables? Plus some more, but I'm still digesting the
changes.
One issue I've found:
fs_win_get_drive() in filesys.c has a call to hb_wcntombcpy()
which needs to be updated to one of the new APIs.
Viktor
[ cut commit msg to be sure ]
Massimo Belgrano
Massimo Belgrano
Viktor Szakáts
Thank you very much Przemek. (and OTC for sponsoring)
For those interested in looking into the whole patch (f.e. to
update 3rd party code), use this command in Harbour SVN
sandbox root:
svn diff -r 17403:17404 > uni.dif
One of the next logical questions: How to enable unicode
fields in tables? Plus some more, but I'm still digesting the
changes.
One issue I've found:
fs_win_get_drive() in filesys.c has a call to hb_wcntombcpy()
which needs to be updated to one of the new APIs.
Viktor