2.x export

[Gwern Branwen]

With today's HEAD, I decided to try exporting so I could update my
publicly shared flashcards.

Doing so, my Mnemosyne throws scores of dialog errors saying things
like 'Missing filename:
/home/gwern/.local/share/mnemosyne/default.db_media/_latex/855827f0713ffd68bc1e37c05dbaa067.png',
as part of the 'Bundling media files...' step. If I hold down the
enter key, it eventually finishes and leaves me with a 50MB .cards
file.

I'm not entirely sure what this is supposed to be. 50MB is less than
my default.db_media folder weighs (57M), and I'd expect any export to
be at least 57M (plus overhead for any textual encoding, which would
be like 10-20%) and then another 3MB+ for the actual cards.

--
gwern
http://www.gwern.net

[Peter Bienstman]

Since 2.2, Mnemosyne also bundled the latex files with the export. Could
it be that you have a problem in your latex setup? If not, feel free to
mail me your database privately.

Cheers,

Peter

[Gwern Branwen]

On Thu, Jan 3, 2013 at 2:56 AM, Peter Bienstman
<Peter.B...@ugent.be> wrote:
> Since 2.2, Mnemosyne also bundled the latex files with the export. Could it
> be that you have a problem in your latex setup? If not, feel free to mail me
> your database privately.

I haven't noticed any real issues, asides from the detail that \text{}
doesn't seem to be handled right and just results in the argument
appearing as italicized text. Could there be something going on like
the export is assuming that the respective PNGs have been generated
already even if they haven't come up for review and had a PNG
generated yet?

--
gwern
http://www.gwern.net

[Peter Bienstman]

Did you change the preamble / postamble recently?

Does the problem still occur if you do 'compact database - clear unused
media files'?

Peter

[Gwern Branwen]

On Fri, Jan 4, 2013 at 2:39 AM, Peter Bienstman
<Peter.B...@ugent.be> wrote:
> Did you change the preamble / postamble recently?

Well, not *recently* - I changed something there years ago. I forget
how, though, and I might've undone the changes because when I cat
latex/*, it looks pretty normal:

dvipng -D 200 -T tight tmp.dvi
\end{document}
\documentclass[12pt]{article}
\pagestyle{empty}
\begin{document}

> Does the problem still occur if you do 'compact database - clear unused
> media files'?

I actually did that before I even tried exporting.

--
gwern
http://www.gwern.net

[Peter Bienstman]

Could you privately mail me your database and the media directory? Or
perhaps a smaller database which also shows the problem?

Cheers,

Peter

[Peter Bienstman]

The good news is that I can reproduce this with the database you sent me, the bad new is that I haven't found the cause yet...

Peter

[Peter Bienstman]

OK, fixed the latex bug, but it turns out there's another issue in the database you sent me, namely that some strings are not UTF-8 encoded (they happen in Japanese cards like 'like father like son'). This causes the XML export to trip up.

It might be possible to work around this from Mnemosyne's side, but I'm not sure I would be doing you a service then anyway, because, even though there would be no error on import, the Kanji still would not show up correctly...

What do you think?

Peter

PS: if you want, I can prepare you a prerelease installer which fixes the latex export bug

[Gwern Branwen]

On Wed, Jan 9, 2013 at 3:03 PM, Peter Bienstman
<peter.b...@gmail.com> wrote:
> OK, fixed the latex bug, but it turns out there's another issue in the
> database you sent me, namely that some strings are not UTF-8 encoded (they
> happen in Japanese cards like 'like father like son'). This causes the XML
> export to trip up.
>
> It might be possible to work around this from Mnemosyne's side, but I'm not
> sure I would be doing you a service then anyway, because, even though there
> would be no error on import, the Kanji still would not show up correctly...
>
> What do you think?

I have no idea what that problem is or means, so I'm afraid I can't
help you there. I just copy-pasted stuff as I saw fit for
flashcards...

> PS: if you want, I can prepare you a prerelease installer which fixes the
> latex export bug

I've had an out of date export on my site for like a year now, I'm in
no real rush; I'd rather have a correct export later than a flawed
export sooner.

--
gwern
http://www.gwern.net

[Peter Bienstman]

On 01/09/2013 09:19 PM, Gwern Branwen wrote:
> I have no idea what that problem is or means, so I'm afraid I can't
> help you there. I just copy-pasted stuff as I saw fit for
> flashcards...

If you open e.g. the card

Kaeru no ko wa kaeru.
"Child of a frog is a frog."

I guess you don't see Japanese characters, but rather junk, right?

The importer trips up over this junk because it's not valid UTF-8 unicode.

I don't think the solution here is to patch the importer (because junk
in means junk out, and the information is lost anyhow), but rather
figure out what went wrong during pasting when creating the card, and
fix that.

So, could you send me an example of e.g. a website from which you paste
which gives this junk?

As for the existing cards, I'm afraid you'll have to fix them manually...

Cheers,

Peter

[Gwern Branwen]

On Thu, Jan 10, 2013 at 5:33 AM, Peter Bienstman
<Peter.B...@ugent.be> wrote:
> I guess you don't see Japanese characters, but rather junk, right?

Looks like it.

> The importer trips up over this junk because it's not valid UTF-8 unicode.
>
> I don't think the solution here is to patch the importer (because junk in
> means junk out, and the information is lost anyhow), but rather figure out
> what went wrong during pasting when creating the card, and fix that.
>
> So, could you send me an example of e.g. a website from which you paste
> which gives this junk?

I don't know where I got them.

> As for the existing cards, I'm afraid you'll have to fix them manually...

How would I get a list of all of them?

--
gwern
http://www.gwern.net

[Peter Bienstman]

Quoting Gwern Branwen <gwe...@gmail.com>:

>> As for the existing cards, I'm afraid you'll have to fix them manually...
>
> How would I get a list of all of them?

If you use the card browser, you should be able to pick them up very quickly.

Peter

[Gwern Branwen]

On Thu, Jan 10, 2013 at 1:02 PM, Peter Bienstman
<Peter.B...@ugent.be> wrote:
> If you use the card browser, you should be able to pick them up very
> quickly.

What, browse through 19k cards 'very quickly'? Even if I restricted
myself to the Japanese vocab, how do I know that was all of the cards
breaking exporting?

--
gwern
http://www.gwern.net

[Peter Bienstman]

The problem is only in the Japanese cards. If you page-down full
screen, wouldn't you be able scan 50 cards / sec? Anyway, if you
prefer, I'll write a script to identify the culprits and mail it to
you privately.

Cheers,

Peter

[Gwern Branwen]

On Thu, Jan 10, 2013 at 1:23 PM, Peter Bienstman
<Peter.B...@ugent.be> wrote:
> The problem is only in the Japanese cards. If you page-down full screen,
> wouldn't you be able scan 50 cards / sec? Anyway, if you prefer, I'll write
> a script to identify the culprits and mail it to you privately.

I've taken a look and edited 3 or 4. I'm not sure that's all of them,
though. I was going to check by doing an export, but after a rebuild
of HEAD it was still throwing Latex errors, and checking my repo
doesn't show any recent patches even though I did a pull and think I'm
using the right repo:

$ bzr pull
Using saved parent location:
http://bazaar.launchpad.net/~peter-bienstman/mnemosyne-proj/trunk/
No revisions or tags to pull.

...

------------------------------------------------------------
revno: 1629
committer: Peter Bienstman <Peter.B...@UGent.be>
branch nick: mnemosyne-proj-pbienst
timestamp: Wed 2012-12-19 21:17:16 +0100
message:
Fix control-c shortcut.
------------------------------------------------------------
revno: 1628
committer: Peter Bienstman <Peter.B...@UGent.be>
branch nick: mnemosyne-proj-pbienst
timestamp: Fri 2012-12-14 08:59:17 +0100
message:
RTL fixes.
------------------------------------------------------------

--
gwern
http://www.gwern.net

[Peter Bienstman]

The latest fix is on the pbienst branch, not the trunk.

The encoding error only shows up on import, not already on export.

Cheers,

Peter

> --
> You received this message because you are subscribed to the Google
> Groups "mnemosyne-proj-users" group.
> To post to this group, send email to mnemosyne-...@googlegroups.com.
> To unsubscribe from this group, send email to
> mnemosyne-proj-u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Gwern Branwen]

I rebuilt from HEAD and did an export after fixing some media errors
(and ~100 mistaken uses of '<sound' rather than '<audio' tags, which
took a while because I had to do it by hand and couldn't, as I did
with Mnemosyne 1.x, just run a search-replace on the .mem file). I
then nuked my profile, and imported it; all the cards seemed to work
as they should.

Anyone who wishes can download my cards at
http://dl.dropbox.com/u/85192141/gwern.cards.7z ; 35M, 19,074 cards.

--
gwern
http://www.gwern.net/Spaced%20repetition