Dump to binary file
20 views
Skip to first unread message
Gareth Murfin
Apr 19, 2019, 8:31:15 AM4/19/19
to CodenameOne Discussions
Im not sure exactly what I want here, but basically I have to read in a .txt file with 164,000 entries, each line needs to be string-tokenized. So the code is easy but it obviously takes a loooong time to do this in Java on a phone or desktop even. So I was wondering is there some other way to do it, like do it once and save it as some sort of binary file that can be loaded back into the ram at highspeed? Its probably a silly question really but I am looking for ways to speed this up because this stuff is used as a "realtime" search within the app so I cant exactly force the user to wait 10 minutes everytime they load the app just to prepare the search data.
MY CODE:
hi.add(new Label("START"));
List<LangHolder> allLineObjects = new ArrayList<LangHolder>();
long start = System.currentTimeMillis();
try
{
//The format of the comma-delimited data is:{romaji},{kanji},{kana},{English gloss}
//eg doujou, 仝, どうじょう, 1. "as above" mark.
_("Reading in search data now!");
final String fileName = "/searchdata.txt";
InputStream in = Display.getInstance().getResourceAsStream(Form.class, fileName);
String fileContents = com.codename1.io.Util.readToString(in);// efficient
_("Finished reading ("+fileContents.length()+"), now parse");
List<String> lines_ = StringUtil.tokenize(fileContents, '\n');
for (int x=0; x<lines_.size(); x++)
{
StringTokenizer st = new StringTokenizer(lines_.get(x),",");
int i=0;
String romaji = "";
String kanji = "";
String kana = "";
String english = "";
while (st.hasMoreElements())
{
String token = (String) st.nextElement();
switch (i)
{
case 0:romaji=token;break;
case 1:kanji=token;break;
case 2:kana=token;break;
case 3:english=token;break;
default:_("UNKNOWN STATE HERE PARSING CVS "+i);
}
i++;
if (i>3)
{
_(""+x+" ROMANJI: "+romaji+" KANJI: "+kanji+" KANA:"+kana+" ENGLISH:"+english);
LangHolder lh = new LangHolder(romaji,kanji,kana,english);
allLineObjects.add(lh);
romaji = "";
kanji = "";
kana = "";
english = "";
i=0;
}
}
}
_("finished making search data, there are "+allLineObjects.size()+" objects holding the info");
long end = System.currentTimeMillis();
_("time took "+((end-start)/1000)/60+" mins.");
hi.add(new Label("finished making search data, there are "+allLineObjects.size()+" objects holding the info"));
hi.invalidate();
hi.add(new Label("time took "+((end-start)/1000)/60+" mins."));
EXAMPLE OF FILE:
doujou,仝,どうじょう,1. "as above" mark.
kansuujizero,漢数字ゼロ,かんすうじゼロ,1. "kanji" zero.
maru,○/〇,まる,1. circle (sometimes used for zero). 2. 'correct' (when marking). 3. symbol used as a placeholder (either because a number of other words could be used in that position, or because of censorship). 4. period. 5. full stop. 6. maru mark. 7. semivoiced sound. 8. p-sound.
e-bi-shi-jixyun,ＡＢＣ順,エービーシーじゅん,1. alphabetical order.
shi-dyi-pure-ya-,ＣＤプレーヤー/ＣＤプレイヤー,シーディープレイヤーシーディープレーヤー/シーディープレイヤー,1. CD player.
shi-dyi-pureiya-,ＣＤプレーヤー/ＣＤプレイヤー,シーディープレイヤーシーディープレーヤー/シーディープレイヤー,1. CD player.
enukixyou,Ｎ響,エヌきょう,1. NHK Symphony Orchestra.
o-bakku,Ｏバック,オーバック,1. O-back. 2. skirt with peek-a-boo hole in rump.
a-ruesunisannike-buru,ＲＳ２３２ケーブル,アールエスにさんにケーブル,1. rs232 cable.
texi-shatsu,Ｔシャツ,ティーシャツ,1. T-shirt.
texi-bakku,Ｔバック,ティーバック,1. T-back. 2. bikini thong.
aun'nokokyuu,あうんの呼吸/阿吽の呼吸,あうんのこきゅう,1. the harmonizing, mentally and physically, of two parties engaged in an activity. 2. singing from the same hymn-sheet. 3. dancing to the same beat.
meihaku,明白,めいはく,1. obvious. 2. clear. 3. plain. 4. evident. 5. apparent. 6. explicit. 7. overt.
akarasama,明白/偸閑/白地,あからさま,1. plain. 2. frank. 3. candid. 4. open. 5. direct. 6. straightforward. 7. unabashed. 8. blatant. 9. flagrant.
akudoi,悪どい,あくどい,1. gaudy. 2. showy. 3. excessive. 4. crooked. 5. vicious.
agetsurau,論う,あげつらう,1. to discuss. 2. to find fault with. 3. to criticize. 4. to criticise.
asoko,彼処/彼所,あこあそこ/あすこ/かしこ/アソコ/あしこ/あこ,1. there (place physically distant from both speaker and listener). 2. over there. 3. that place. 4. yonder. 5. you-know-where. 6. genitals. 7. private parts. 8. nether regions. 9. that far (something psychologically distant from both speaker and listener). 10. that much. 11. that point.
ashiko,彼処/彼所,あこあそこ/あすこ/かしこ/アソコ/あしこ/あこ,1. there (place physically distant from both speaker and listener). 2. over there. 3. that place. 4. yonder. 5. you-know-where. 6. genitals. 7. private parts. 8. nether regions. 9. that far (something psychologically distant from both speaker and listener). 10. that much. 11. that point.
ako,彼処/彼所,あこあそこ/あすこ/かしこ/アソコ/あしこ/あこ,1. there (place physically distant from both speaker and listener). 2. over there. 3. that place. 4. yonder. 5. you-know-where. 6. genitals. 7. private parts. 8. nether regions. 9. that far (something psychologically distant from both speaker and listener). 10. that much. 11. that point.
asuko,彼処/彼所,あこあそこ/あすこ/かしこ/アソコ/あしこ/あこ,1. there (place physically distant from both speaker and listener). 2. over there. 3. that place. 4. yonder. 5. you-know-where. 6. genitals. 7. private parts. 8. nether regions. 9. that far (something psychologically distant from both speaker and listener). 10. that much. 11. that point.
kashiko,彼処/彼所,あこあそこ/あすこ/かしこ/アソコ/あしこ/あこ,1. there (place physically distant from both speaker and listener). 2. over there. 3. that place. 4. yonder. 5. you-know-where. 6. genitals. 7. private parts. 8. nether regions. 9. that far (something psychologically distant from both speaker and listener). 10. that much. 11. that point.
axtsutoiumani,あっという間に/あっと言う間に/あっとゆう間に/アッという間に/アッと言う間に/アッとゆう間に,アッとゆうまにあっというまに/あっとゆうまに/アッというまに/アッとゆうまに,1. just like that. 2. in the twinkling of an eye. 3. in the blink of an eye. 4. in the time it takes to say "ah!".
attoyuumani,あっという間に/あっと言う間に/あっとゆう間に/アッという間に/アッと言う間に/アッとゆう間に,アッとゆうまにあっというまに/あっとゆうまに/アッというまに/アッとゆうまに,1. just like that. 2. in the twinkling of an eye. 3. in the blink of an eye. 4. in the time it takes to say "ah!".
axtsutoyuumani,あっという間に/あっと言う間に/あっとゆう間に/アッという間に/アッと言う間に/アッとゆう間に,アッとゆうまにあっというまに/あっとゆうまに/アッというまに/アッとゆうまに,1. just like that. 2. in the twinkling of an eye. 3. in the blink of an eye. 4. in the time it takes to say "ah!".
attoiumani,あっという間に/あっと言う間に/あっとゆう間に/アッという間に/アッと言う間に/アッとゆう間に,アッとゆうまにあっというまに/あっとゆうまに/アッというまに/アッとゆうまに,1. just like that. 2. in the twinkling of an eye. 3. in the blink of an eye. 4. in the time it takes to say "ah!".
anou,,あのう/あの/あのー,1. say. 2. well. 3. errr ....
ano,,あのう/あの/あのー,1. say. 2. well. 3. errr ....
ano-,,あのう/あの/あのー,1. say. 2. well. 3. errr ....
Gareth Murfin
Apr 19, 2019, 9:23:46 AM4/19/19
to CodenameOne Discussions
actually unzip and parse takes only 3 seconds on my device! so might be just fine!
Steve Hannah
Apr 19, 2019, 9:58:46 AM4/19/19
to codenameone...@googlegroups.com
Two options that come to mind.
1. Use Externalizable, DataInputStream/DataOutputStream to serialize/deserialize entire array.
2. Use SQLite
--
You received this message because you are subscribed to the Google Groups "CodenameOne Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codenameone-discu...@googlegroups.com.
Visit this group at https://groups.google.com/group/codenameone-discussions.
To view this discussion on the web visit https://groups.google.com/d/msgid/codenameone-discussions/ec425134-68f2-4285-a554-d2304c30c476%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages