How to regenerate icu data files?

[Jiang Jiang]

Hi,

At Opera we are trying to support a few extra locales/languages that are not supported by Google Chrome/Chromium (because our existing version supports them), we had the translations but some of the languages are lacking proper locale data for date formatting and so on, Norwegian Nynorsk for instance, the data from ICU 4.6 is too old to include that information, but current one [1] has it.

However, merely backporting those text files to Chromium doesn't work, because the icudt46l_dat.S files are pre-generated as described in icu.gyp and README.chromium under third_party/icu. But exactly how to regenerate these data files aren't quite clear from README.chromium. Does anyone have instruction to re-generate these files once the locales text files are updated?

From log I can see that Jungshik Shin (js...@chromium.org) updated it a few times so I'm CCing him here.

BTW, any plan to update ICU in Chromium? Say, from 4.6 to 4.8 or more recent versions?

- Jiang

[1] http://source.icu-project.org/repos/icu/icu/trunk/source/data/locales/nn.txt

[Mostyn Bramley-Moore]

Jungshik Shin mentioned that he was going to update ICU to 51.2 soon
(perhaps this week).


-Mostyn.
--
Mostyn Bramley-Moore
mos...@opera.com

[Liu Guang]

we also want regenerate icudt.dll,  but there is no instruction for that,  is anyone generate success?

在 2013年6月18日星期二UTC+8下午6时08分20秒，Jiang Jiang写道：

[Jungshik Shin (신정식, 申政湜)]

Sorry for the delay. 

2013/11/3 Liu Guang <lg8...@gmail.com>

we also want regenerate icudt.dll,  but there is no instruction for that,  is anyone generate success?

It's documented in README.chromium. See item 10 for that.  What's not mentioned in that file is where the file is generated. When you run 'make', *S file is generated in data/out/tmp.  

We'll eventually move away from statically linking data file (on Linux and Mac). See http://crbug.com/72633. 

Jungshik 

[Liu Guang]

tks for reply.

I  try to build like this :  nmake.exe /f makedata.mak icumake=D:\chrome_30\src\third_party\icu\source\data

output is:

ICU data make path is D:\chrome_30\src\third_party\icu\source\data

ICU root path is D:\chrome_30\src\third_party\icu

ICU data source archive is D:\chrome_30\src\third_party\icu\source\data\in\icudt

46l.dat

NMAKE : fatal error U1073: don't know how to make '"D:\chrome_30\src\third_party

\icu\bin\pkgdata.exe"'

Stop.

I google U1073, same guy says the makefile.mak has error, do you have suggestion?

btw: VS2010 Sp1, Win7, no cygwin。 do i need install cygwin?

在 2013年11月14日星期四UTC+8上午9时22分19秒，Jungshik Shin写道：

[Liu Guang]

finally, i build icudt.dll success, this is my step:

1. open source\allinone.sln, select Release and Win32, then build solution . after that, source\tools\makconv\x86\Release\makeconv.exe will be generated. 

    ignore the build error like " cannot open include file:'layout/LEType.h', no such file or directory. 

2. run command in source\data folder:

    nmake.exe /f makedata.mak ICUMAKE=D:\chrome_30\src\third_party\icu\source\data CFG=x86\Release

    change ICUMAKE to your folder.

3. if has error like this: " NMAKE.exe: fatal error U1073: don't nkow how to make 'curr\'. 

    rename source\data\curr\reslocal.mk, 

    rename source\data\coll\collocal.mk

    rename source\data\brkitr\brklocal.mk

   rename source\data\lang\reslocal.mk
   rename source\data\locales\reslocal.mk  
   rename source\data\rbnf\rbnflocal.mk  
   rename source\data\region\reslocal.mk
   rename source\data\translit/trnslocal.mk

  

  rerun command in step2. it will be success.

   

   icudt46l.dat will be generate at folder : source\data\out

   icudt46.dll will be generate at folder : icu\bin

4.  because reslocal.mk was modify by chrome, so if you want make your own data, you can modify resfiles.mk in data folder, then rerun command in step2.

5. i will check why make failed in step2 when reslocal.mk file was exist,  Jungshik Shin (js...@chromium.org)  do you know why?

在 2013年11月17日星期日UTC+8上午10时34分28秒，Liu Guang写道：

[Scott Graham]

Hi jshin,

Per https://code.google.com/p/chromium/issues/detail?id=321626 icudt.dll is based poorly and is cutting renderer's address space in half.

It would be beneficial if we could use the data file on Windows too, or otherwise pull the data dll build into the regular gyp build rather than a checked in binary built by magic steps, so it could have its base address set appropriately.

Are either of those in scope while updating to 51.2? Let me know if I can help.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Jungshik Shin (신정식, 申政湜)]

Sorry again for the late reply. 

I always built icu*.dat file on Linux first (with all the customizations in foolocal.mk in source/data/* and patches applied) and move that over to Windows to icu/source/data/in  (as noted in README.chromium). Then, on Windows, to build icu*.dll, I fire up VS to open icudt_build solution and build icudt_build project. 

The current VS proj/solution files for icudt_build is not for VS 2010, but the automatic conversion works fine. One more thing to do is to suppress warning 4068 (ICU upstream does not have this issue any more) before building icudt_build. 

On Linux/Mac/Android, I've been working on using icudt*.dat directly instead of static linking. (there might be a perf implication in switching from static link to mmap'ing icu*.dat on Linux/Mac). We already do that on iOS).  We can do the same on WIndows. This time, there'll be a different perf implication (dll to mmap'ing dat file).  I'll reply to Scott's email in this thread. 

2013/11/17 Liu Guang <lg8...@gmail.com>

[Jungshik Shin (신정식, 申政湜)]

2013/11/20 Scott Graham <sco...@chromium.org>

Hi jshin,

Per https://code.google.com/p/chromium/issues/detail?id=321626 icudt.dll is based poorly and is cutting renderer's address space in half.

It would be beneficial if we could use the data file on Windows too, or otherwise pull the data dll build into the regular gyp build rather than a checked in binary built by magic steps, so it could have its base address set appropriately.

That's good to hear. If using 'mmap'ing dat' file would be beneficial for Windows, I'd be more than happy to do that. In http://crbug.com/72633, I've been working on that for Linux/Mac/Android/CrOS (the 2nd CL is in CQ after which I'll experiment with flipping the switch to actually use icu*.dat). Doing that on Windows requires a small change in gyp and icu_util.cc.   

One remaining question is whether that affects v8's icu loading. Apparently, it's not affected when v8 is a part of Chrome (based on my testing on Linux), but I'm not 100% positive. If it does not affect v8's icu usage as a part of Chrome (making it work when v8 is stand-alone should be easy), we're almost set. 

Jungshik 

[Jungshik Shin (신정식, 申政湜)]

Dropping an incomplete email address. 

+jochen (v8's icu use), shess (sqlite's icu data loading)

@Jochen , @shess : can you answer my question about v8's and sqlite's ICU data loading in http://crbug.com/72633  ?  

Thank you,

Jungshik

 

2013/12/3 Jungshik Shin (신정식, 申政湜) <js...@chromium.org>