CCSID transaltion with RPGle
Dr.Ugo Gagliardelli
when I should read fields defined with certain CCSID.
Assuming a file like this:
CREATE TABLE RUSSIAN (
CODART CHARACTER ( 12) CCSID 1144 NOT NULL WITH DEFAULT
CONSTRAINT RUSSO_CODART_PRIMARY PRIMARY KEY,
ARTDES CHARACTER ( 20) CCSID 1144 NOT NULL WITH DEFAULT,
TRADES CHARACTER ( 50) CCSID 1154 NOT NULL WITH DEFAULT,
ARTTXT CHARACTER (150) CCSID 1154 NOT NULL WITH DEFAULT
)
As you can see there are 2 fields with CCSID 1144 that's Italian with
euro symbol (the primary language) and 2 fields with 1154 that's
Cyrillic with euro symbol.
I should print russian information with a printer-file like this (just
an experimental one):
R RIGA
CODART 12A 3SKIPB(3)
ARTDES 20A 3SPACEA(2)
TRADES 50A 3SPACEA(2)
FNTCHRSET(C0430380 T1001025)
ARTTXT 150A 3SPACEA(2)
FNTCHRSET(C0430380 T1001025)
using this RPG code (experimental as well):
FRUSSIAN if e k disk rename(RUSSIAN:russo)
Frussoprt o e printer
c dow not %eof and not %error
c read(e) russo
c write(e) riga
c enddo
c eval *inlr = *on
What happens, is that in fields TRADES and ARTTXT all characters not
present in 1144 CCSID (the job ccsid) are traslated as x'3f' that's a
non printable character, so the printer-file substitute them with blanks.
If I change the job CCSID in 65535, will print as expected. Obviously
changing job ccsid is not an option, because possibly a production
program (not an experimental one) would like to run under the mother
language ccsid.
Is there a way to have RPG program disregard job CCSID?
For example if I code the same in C:
_RFILE *in, *out;
in = _Ropen("TRUFILE/AARUSSO", "rr, ccsid=65535");
out = _Ropen("TRUFILE/RUSSOPRT", "ar, ccsid=65535");
I'm able to specify ccsid=65535 in opening mode, so no conversion will
take place, and the program will behave as expected (or better as *I*
expect).
I didn't find any way to declare the file the same way in RPG, hope
that's my fault and someone other will show me the right way.
nicolas
I recently had to deal with a problem that is somewhat similar.
A program that had been initially written to print French had to be
changed to print any of the EU languages including greek.
What I made is create table containing fields whith CCSID 13488 so these
fields are of type C in the program.
In the printer file, I put the keyword CHRID on every field that had to
be printed with NLS, and did not change the ones which had to remain in
French.
At execution, before the opening of the printer file I do OVRPRFT
CHRID(the_good_one), then for each thing to print I used iconv() from
Unicode to the CCSID that matches CHRID (chrid and ccsid values are
stored in a table when language as primary key).
Not sure that's the way it's meant to be used, but that works, tested
against Latin, Greek, Cyrillic, Arab (just characters, it obviously
still prints right-to-left). All with the same programme and PRTF.
Hope this helps
Nicolas
Dr.Ugo Gagliardelli
> Hello,
>
> I recently had to deal with a problem that is somewhat similar.
> A program that had been initially written to print French had to be
> changed to print any of the EU languages including greek.
> What I made is create table containing fields whith CCSID 13488 so these
> fields are of type C in the program.
>
> In the printer file, I put the keyword CHRID on every field that had to
> be printed with NLS, and did not change the ones which had to remain in
> French.
>
> At execution, before the opening of the printer file I do OVRPRFT
> CHRID(the_good_one), then for each thing to print I used iconv() from
> Unicode to the CCSID that matches CHRID (chrid and ccsid values are
> stored in a table when language as primary key).
>
> Not sure that's the way it's meant to be used, but that works, tested
> against Latin, Greek, Cyrillic, Arab (just characters, it obviously
> still prints right-to-left). All with the same programme and PRTF.
iconv makes the program more complex and heavy, then I think that if a C
program is able to read raw bytes from the same table, an RPG program
should do as well. Further more, as I use FNTCHRSET keyword in the
printerfile, I need no translation, while RPG still make data
translations even if not requested at all, and in this case I don't see
the reason why it replace characters that's not able to convert, with
x'3F' instead of just leave them as is.
Peter H. Coffin
Gotta turn off the job CCSID before running the program. *Then* RPG will
read the file without converting the read and write into and out of the
job CCSID.
--
Liberty, equality, diversity. Pick any two.
Dr.Ugo Gagliardelli
> On Tue, 13 Apr 2010 15:23:26 +0200, Dr.Ugo Gagliardelli wrote:
>> il 13.04.2010 13.37, Scrive nicolas 41112032:
> read the file without converting the read and write into and out of the
> job CCSID.
>
My need would be open the file specifying the ccsid as _ROpen C function
does.
CRPence
> I know that uicode and iconv works, but the table is made like
> that and iconv makes the program more complex and heavy, then I
> think that if a C program is able to read raw bytes from the same
> table, an RPG program should do as well. Further more, as I use
> FNTCHRSET keyword in the printerfile, I need no translation,
> while RPG still make data translations even if not requested at
> all, and in this case I don't see the reason why it replace
> characters that's not able to convert, with x'3F' instead of just
> leave them as is.
Of course it is not really the RPG which is converting the data.
It is the database performing CCSID translation for the RPG
program as requester, for the specified CCSID. For lack of any open
option to specify otherwise, that open option will default to
CCSID(*JOB). So when running with job CCSID=*HEX, the database does
not convert the data into any CCSID, so the text remains the
original binary code points. Thus to get the original binary code
points irrespective of job CCSID, the request to open would need to
be requested by the RPG with the same option effected on _ropen with
open option of "ccsid=65535" or to instead use a logical file or ODP
where the columns are redefined to have the CCSID(*HEX). Opening
the file as program described also would make the database treat the
data as binary.
Even without direct capability in RPG [I am not suggesting no
support exists, just that I do not know of such], the C open could
be used with sharing enabled, and then the RPG can open the shared ODP.
Regards, Chuck
Peter H. Coffin
I understand the need, but I haven't found a way to actually DO that.
Fortunately, the job CCSID doesn't have to be off for the whole run of
the job, just any RPG manipulation of *character* data that contain
characters outside the job CCSID. Graphic data's okay. Which got me into
the habit of getting data out of UTF-8 and into UCS-2 as soon as
possible in the process, to the point of writing little RPG converter
programs like the one posted here in June of 2007. (It's Message-ID:
<slrnf7qqje....@abyss.ninehells.com> if you want to rummage it
up in Google Groups or something.)
As far as RPG is concerned, the conversion happens before it even
"reads" the datafile, as the data appears in the input record converted
to job CCSID, without any sense of preservation of what the data is in
the file, and it gets converted again on write, from job to file/field
CCSID, so long as it was ever fetched from the buffer. That is, if you
read a record and write it back out but NEVER EVER refer to it in the
program (such that it doesn't show up in the eval in DBG mode), then it
seems to get passed through intact. I've been looking for a way around
this since 2007, and only way it seems to be able to write in RPG and
not have the automatic conversion happen at runtime is to A) turn off
the job CCSID before calling the RPG program or routine, and then B) use
the APIs if you need to any kind of conversion.
--
12. One of my advisors will be an average five-year-old child. Any
flaws in my plan that he is able to spot will be corrected before
implementation.
--Peter Anspach's list of things to do as an Evil Overlord
dieter...@t-online.de
> I'm facing with a problem using a db file in RPGle, the problem raise
> when I should read fields defined with certain CCSID.
> Assuming a file like this:
> CREATE TABLE RUSSIAN (
> CODART CHARACTER ( 12) CCSID 1144 NOT NULL WITH DEFAULT
> CONSTRAINT RUSSO_CODART_PRIMARY PRIMARY KEY,
> ARTDES CHARACTER ( 20) CCSID 1144 NOT NULL WITH DEFAULT,
SQL might be an option
D*B
Dr.Ugo Gagliardelli
> Dr.Ugo Gagliardelli wrote:
>> I know that uicode and iconv works, but the table is made like
>> that and iconv makes the program more complex and heavy, then I
>> think that if a C program is able to read raw bytes from the same
>> table, an RPG program should do as well. Further more, as I use
>> FNTCHRSET keyword in the printerfile, I need no translation,
>> while RPG still make data translations even if not requested at
>> all, and in this case I don't see the reason why it replace
>> characters that's not able to convert, with x'3F' instead of just
>> leave them as is.
>
> Of course it is not really the RPG which is converting the data. It
> is the database performing CCSID translation for the RPG program as
> requester, for the specified CCSID. For lack of any open option to
> specify otherwise, that open option will default to CCSID(*JOB). So
> when running with job CCSID=*HEX, the database does not convert the data
> into any CCSID, so the text remains the original binary code points.
If it were a db fault (or option, as you prefer) it should happen
anytime. Using SQL from STRSQL, instead, with something like:
insert into newfile (select * from oldfile)
without changing job CCSID I can copy data without any conversion,
further more, if I read the same file by JDBC without changing the
server job CCSID the behaviour is what expected.
My complain is not that RPG should or should not operate a CCSID
conversion, but, if it does, it should not put x'3F' characters
replacing what it's not able to convert properly.
> Thus to get the original binary code points irrespective of job CCSID,
> the request to open would need to be requested by the RPG with the same
> option effected on _ropen with open option of "ccsid=65535" or to
> instead use a logical file or ODP where the columns are redefined to
> have the CCSID(*HEX). Opening the file as program described also would
> make the database treat the data as binary.
I'm wondering that to use CCSID keyword in a logical file field, the
basing field of the phisical file should be a G type field, that's not
permitted for "sigle-byte" CCSID as my file has. Maybe even using CCSID
at file level should have the same resstrictions.
> Even without direct capability in RPG [I am not suggesting no support
> exists, just that I do not know of such], the C open could be used with
> sharing enabled, and then the RPG can open the shared ODP.
I didnt' try this, it could be an option, even if the _ROpen mode or
varparm list, as far as I know, doesn't include any "sharemode" keyword,
so I'm wondering that an ovrdbf share(*yes) command before _ROpen is
still needed, and the ipotetical C open function should be aware the way
the RPG program would have open the file (input, output, update or
delete) to avoid rpg implicitly open a new data-path. Isn't it?
CRPence
> il 13.04.2010 20.30, Scrive CRPence 43070448:
>> Dr.Ugo Gagliardelli wrote:
>>> then I think that if a C program is able to read raw bytes
>>> from the same table, an RPG program should do as well.
>>>
>>> Further more, as I use FNTCHRSET keyword in the printerfile,
>>> I need no translation, while RPG still make data translations
>>> even if not requested at all, and in this case I don't see
>>> the reason why it replace characters that's not able to
>>> convert, with x'3F' instead of just leave them as is.
>>
>> Of course it is not really the RPG which is converting the
>> data. It is the database performing CCSID translation for the
>> RPG program as requester, for the specified CCSID. For lack of
>> any open option to specify otherwise, that open option will
>> default to CCSID(*JOB). So when running with job CCSID=*HEX,
>> the database does not convert the data into any CCSID, so the
>> text remains the original binary code points.
>
> Basing on my knowledge I should disagree your statement.
Since most of my career was working on the DB2 for OS/400,
i5/OS, and a bit on DB2 for i, perhaps that may be sufficient to
accept my contention that the database does the conversions as
requested by the CCSID specified by interface; i.e. when that
interface is RPG, if for lack of any ability to specify CCSID on the
F-spec [as an "open option"], then that CCSID as open option is
established by default to the job CCSID. FWiW it is the Common Data
Management which sets the CCSID to the job CCSID whenever the open
options do not include a CCSID specification, and the database uses
that value as the target CCSID for its conversion activity.
Knowing that the C open interface provides an interface to the
open-option of "ccsid=..." should imply [that one might then infer]
that the above description of the programming interface to the
database is accurate.
> If it were a db fault (or option, as you prefer) it should happen
> anytime. Using SQL from STRSQL, instead, with something like:
> insert into newfile (select * from oldfile) without changing job
> CCSID I can copy data without any conversion,
For the noted SQL, the user has no choice in the matter, because
they [user] are not the recipient of the data. The SQL itself
defines the outcome according to the SQL language for that
statement; i.e. that the data should "map" from the source column
data type [and CCSID attribute] to the target column data type [and
CCSID attribute].
However if attempting within the SQL to open a SELECT cursor [or
SELECT within STRSQL] of a column with an ASCII CCSID, when the job
CCSID is set to *HEX, then notice what the database does to the
column data. The database returns raw\unconverted hex code points;
the binary string. Similarly for SELECT rows of any Unicode column
which has characters that are not represented in the non-hex EBCDIC
job CCSID, the STRSQL report will show the x'3F' in place of any
character that is not known to that EBCDIC CCSID.
> further more, if I read the same file by JDBC without changing
> the server job CCSID the behaviour is what expected.
An ASCII interface to the EBCDIC data, although that is a
somewhat different scenario, is still effectively the same. The
primary difference is that there is no capability to set an ASCII
job CCSID against which to make comparisons. Just like with the RPG
as requester, the JDBC as requester must send the "please return the
data in CCSID ####". In that case the server job CCSID is again
moot, because the client has established the target of conversions.
While running local to the server, the client is the program or
utility asking to open the file; the program or utility must
establish its requested target CCSID, or the database would simply
use CCSID(*JOB). FWiW even the RPG can request an ASCII CCSID, and
its VARPG feature does that, and the database honors that request,
dutifully converting the data from the EBCDIC CCSID to the ASCII
CCSID. I am just not aware of any method to establish any specified
CCSID as open option via the RPG; i.e. I know only of the implicit
capability by CCSID(*JOB).
> My complaint is not that RPG should or should not operate a CCSID
> conversion, but, if it does, it should not put x'3F' characters
> replacing what it's not able to convert properly.
By definition, a CCSID conversion where a character translation
is undefined, the database returns the "substitution character"
which is the hex code point 0x3F [for EBCDIC anyhow]. Those are the
rules. With iconv living under the same rules, in fact as the
implementation for the database conversions, its output should also
have x'3F' substitution character for the same data with the same
from-CCSID and to-CCSID.
>> Thus to get the original binary code points irrespective of job
>> CCSID, the request to open would need to be requested by the
>> RPG with the same option effected on _ropen with open option of
>> "ccsid=65535" or to instead use a logical file or ODP where
>> the columns are redefined to have the CCSID(*HEX). Opening the
>> file as program described also would make the database treat
>> the data as binary.
>
> I'm wondering that to use CCSID keyword in a logical file field,
> the basing field of the phisical file should be a G type field,
> that's not permitted for "single-byte" CCSID as my file has.
> Maybe even using CCSID at file level should have the same
> restrictions.
Hmmm, Yes, the DDS LF rules prevent generally, most any CCSID
specifications; a noted exception, for conversion between Graphic
data type and [Var]Char data types.
To effect a no-xlate for character, possibly the type conversion
to H [DDS Hex; a pseudo-datatype since actually implemented as a
/shift/ attribute, so not legitimately a separate data type] is
available.?
However I was actually thinking of an SQL VIEW logical file,
where the scalar function CAST would specify CCSID 65535. Some of
the same restrictions exist, but I do not remember which.
>> Even without direct capability in RPG [I am not suggesting no
>> support exists, just that I do not know of such], the C open
>> could be used with sharing enabled, and then the RPG can open
>> the shared ODP.
>
> I didn't try this, it could be an option, even if the _ROpen mode
> or varparm list, as far as I know, doesn't include any
> "sharemode" keyword, so I'm wondering that an ovrdbf share(*yes)
> command before _ROpen is still needed, and the hypothetical C
> open function should be aware the way the RPG program would have
> open the file (input, output, update or delete) to avoid rpg
> implicitly open a new data-path. Isn't it?
The share attribute could come from the member or an override.
There actually may be an open option for sharing, even if it is not
externally documented; tricky part is finding out if there is, and
how it is specified. The first open would want to know and effect
what the RPG requires for I\O\U\D, but the database would [in most
cases, just] open the file shared, irrespective of the desires of
the RPG, but logging a message warning of the differences between
the original & shared opens. Thus the open which is shared will be
limited for what access methods are supported, as established by the
original open options.
Regards, Chuck
Dr.Ugo Gagliardelli
> Dr.Ugo Gagliardelli wrote:
>> il 13.04.2010 20.30, Scrive CRPence 43070448:
>>> Dr.Ugo Gagliardelli wrote:
>
> However I was actually thinking of an SQL VIEW logical file, where the
> scalar function CAST would specify CCSID 65535. Some of the same
> restrictions exist, but I do not remember which.
ccsid 65535 and it works.
I tried casting with other ccsid than 65535 but each time, sql runtime
complained that it failedto fetch because of ccsid translation, so i
gave up.
> The share attribute could come from the member or an override. There
> actually may be an open option for sharing, even if it is not externally
> documented; tricky part is finding out if there is, and how it is
> specified.
You're joking with me. isn't it?
:-))
> The first open would want to know and effect what the RPG
> requires for I\O\U\D, but the database would [in most cases, just] open
> the file shared, irrespective of the desires of the RPG, but logging a
> message warning of the differences between the original & shared opens.
> Thus the open which is shared will be limited for what access methods
> are supported, as established by the original open options.
C manual tells that if you try to open a file with a different mode than
the shared one, it will open a new data-path disregarding overrides. Or
better:
"The first open of a shared file determines the open mode for the file
(for example, whether it is open for INPUT, OUTPUT, UPDATE, and
DELETE). If a subsequent open specifies an open mode that was not
specified by the first open, the file will be opened the second time but
the open mode will be ignored. For example, if the first open specifies
an open mode of IO and the second open specifies IOUD, the file will be
opened the second time with a mode of IO." [cit]
So I was wondering on what could happen when RPG try to access an USROPN
declared file with options that don't match the _ROpen options.
The answer is:
CPF417C with error code 1:
1 -- The CCSID specified in the program does not match the value
specified for the original opening.
happening during the open inside the rpg function, while if I omit the
open inside the RPG (tricking the compiler) I get:
RNX1211 Attempt to reading a closed file
while wrkjob + 14 shows the file is opened and shared
This definitely tells us that C _ROpen and RPG open are not the same thing.
CRPence
> C manual tells that if you try to open a file with a different
> mode than the shared one, it will open a new data-path
> disregarding overrides. Or better:
> "The first open of a shared file determines the open mode for the
> file (for example, whether it is open for INPUT, OUTPUT, UPDATE,
> and DELETE). If a subsequent open specifies an open mode that was
> not specified by the first open, the file will be opened the
> second time but the open mode will be ignored. For example, if
> the first open specifies an open mode of IO and the second open
> specifies IOUD, the file will be opened the second time with a
> mode of IO." [cit]
>
The quoted text actually contradicts the earlier statement about
what the "C manual" suggests. The quoted text does not overtly
suggest a new ODP will be created. That text does say the "file
will be opened the second time", but that second open will be shared
according to the share-attribute established for the first open.
The second open will merely receive a diagnostic message CPF4123 rc1
to indicate that the shared open may not be functional as requested,
because its open options were downgraded to match the ODP that will
be shared. Consider the following script and its logged results:
CL> ovrdbf t qsqptabl share(*YES)
CL> opndbf t *inp opnid(i)
Member QSQPTABL file QSQPTABL in QSYS2 opened.
CL> opndbf t *all opnid(a)
Open options ignored for shared open of member QSQPTABL.
Member QSQPTABL file QSQPTABL in QSYS2 opened.
CL> wrkjob option(*opnf)
/* shows QSQPTABL once, SHR=Yes, Nbr=2 */
Whether C or CL, the /second/ open is shared, even though the
second open requester would be unable to effect any of O\U\D if such
a statement were to be issued in the program that requested such an
open. The I\O method is prevented when the I\O is requested, rather
than the file open being prevented. A new ODP may be implicit for
other cases, but not for a scenario with input\output\update\delete
capabilities.
>
> So I was wondering on what could happen when RPG try to access
> an USROPN declared file with options that don't match the _ROpen
> options. The answer is:
> CPF417C with error code 1:
> 1 -- The CCSID specified in the program does not match the
> value specified for the original open.
>
I did not recall that restriction. :-( That result suggests the
requirement remains, to either find a method to get the RPG to open
the file with ccsid=65535 or to process the data retrieved in an
encompassing CCSID followed by converting the data into the final
form rather than depending on unconverted data being retrieved to
avoid the conversion. Of course avoiding Unicode & iconv was noted
as a goal, thus trying instead to get the file to open without data
conversion but not by CHGJOB CCSID(*HEX) during run-time. Ughh.
Sorry, I have no additional suggestions I can think of other than to
make the CCSID retrieval, change, and reset scoped with the module
doing the open, plus making that module capable to back-out the
change if interrupted; e.g. by ENDRQS or an unmonitored exception.
I guess I can rationalize my lack of assist with... My original
intent of clarifying that the DB is converting according to how the
RPG requested the open, was all that I really intended to offer
anyhow. ;-)
> <<SNIP>>
>
> while wrkjob + 14 shows the file is opened and shared
> This definitely tells us that C _ROpen and RPG open are not the
> same thing.
The same /common data management/ processing, so they are the
same except in what options are available from each language or
differ in what options are compatible between an original open for
sharing and a subsequent attempt to share the existing ODP. The
incompatibility of original CCSID mismatch with an attempted shared
open should be diagnosed irrespective of language; i.e. If RPG did
the first open in CCSID 37 and the second open in job CCSID 285 [or
even a C open specified the open option of ccsid=285], then I would
expect the same restriction of msgCPF417C RC1 to be imposed.
Regards, Chuck