Local render server can render wikipedia pages but not local mediawiki

152 views
Skip to first unread message

google

unread,
Jul 15, 2009, 6:58:06 AM7/15/09
to mwlib
Hi,

I've managed to get the mw-render and mw-serve programs running
successfully on Ubuntu 9.04. The test "mw-render -c :en -w rl -o
sun.pdf Sun" (from http://code.pediapress.com/wiki/wiki/mwlib.rl-install)
works just fine, but I cannot successfully render any page from a
local (private) wiki.

If I try an equivalent test command with the base URL directed to my
private wiki, I get the following output (note the URL details have
been obfuscated) in the mw-render.log.rl file:

$ mw-render -c http://www.example.com/w/ -w rl -o test.pdf Collections
font not found: arphic/uming.ttc
font not found: ttf-sil-ezra/SILEOT.ttf
font not found: ttf-farsiweb/nazli.ttf
font not found: unfonts/UnBatang.ttf
font not found: ttf-thai-arundina/ArundinaSans.ttf
font not found: ttf-telugu-fonts/lohit_te.ttf
font not found: ttf-devanagari-fonts/Sarai_07.ttf
font not found: ttf-indic-fonts-core/lohit_pa.ttf
font not found: ttf-oriya-fonts/lohit_or.ttf
font not found: ttf-malayalam-fonts/AnjaliOldLipi.ttf
font not found: ttf-kannada-fonts/Kedage-n.ttf
font not found: ttf-bengali-fonts/LikhanNormal.ttf
font not found: ttf-indic-fonts-core/lohit_ta.ttf
font not found: linux-libertine/LinLibertine_Re.ttf
2009-07-16T14:59:16 mwlib.utils.info >> fetching 'http://
www.example.com/w/api.php?action=query&meta=siteinfo&format=json'
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/Twisted-8.2.0-py2.6-
linux-i686.egg/twisted/internet/defer.py", line 328, in _runCallbacks
self.result = callback(self.result, *args, **kw)
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-
linux-i686.egg/mwlib/twisted_api.py", line 142, in done
d.callback(val)
File "/usr/local/lib/python2.6/dist-packages/Twisted-8.2.0-py2.6-
linux-i686.egg/twisted/internet/defer.py", line 243, in callback
self._startRunCallbacks(result)
File "/usr/local/lib/python2.6/dist-packages/Twisted-8.2.0-py2.6-
linux-i686.egg/twisted/internet/defer.py", line 312, in
_startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "/usr/local/lib/python2.6/dist-packages/Twisted-8.2.0-py2.6-
linux-i686.egg/twisted/internet/defer.py", line 328, in _runCallbacks
self.result = callback(self.result, *args, **kw)
File "/usr/lib/python2.6/json/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.6/json/decoder.py", line 319, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.6/json/decoder.py", line 338, in raw_decode
raise ValueError("No JSON object could be decoded")
exceptions.ValueError: No JSON object could be decoded
fatal: could not get siteinfo [[Failure instance: Traceback: <type
'exceptions.ValueError'>: No JSON object could be decoded
/usr/local/lib/python2.6/dist-packages/Twisted-8.2.0-py2.6-linux-
i686.egg/twisted/internet/defer.py:328:_runCallbacks
/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-linux-
i686.egg/mwlib/twisted_api.py:142:done
/usr/local/lib/python2.6/dist-packages/Twisted-8.2.0-py2.6-linux-
i686.egg/twisted/internet/defer.py:243:callback
/usr/local/lib/python2.6/dist-packages/Twisted-8.2.0-py2.6-linux-
i686.egg/twisted/internet/defer.py:312:_startRunCallbacks
--- <exception caught here> ---
/usr/local/lib/python2.6/dist-packages/Twisted-8.2.0-py2.6-linux-
i686.egg/twisted/internet/defer.py:328:_runCallbacks
/usr/lib/python2.6/json/__init__.py:307:loads
/usr/lib/python2.6/json/decoder.py:319:decode
/usr/lib/python2.6/json/decoder.py:338:raw_decode
]]
3/3/3 jobs -- 0/20 running
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-
linux-i686.egg/mwlib/apps/render.py", line 108, in get_environment
self.zip_filename = hack(output=self.options.keep_zip,
options=self.options, env=env, status=self.status,
keep_tmpfiles=self.options.keep_tmpfiles)
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-
linux-i686.egg/mwlib/apps/buildzip.py", line 105, in hack
reactor.run()
File "/usr/local/lib/python2.6/dist-packages/Twisted-8.2.0-py2.6-
linux-i686.egg/twisted/internet/base.py", line 1128, in run
self.mainLoop()
File "/usr/local/lib/python2.6/dist-packages/Twisted-8.2.0-py2.6-
linux-i686.egg/twisted/internet/base.py", line 1137, in mainLoop
self.runUntilCurrent()
--- <exception caught here> ---
File "/usr/local/lib/python2.6/dist-packages/Twisted-8.2.0-py2.6-
linux-i686.egg/twisted/internet/base.py", line 757, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-
linux-i686.egg/mwlib/twisted_api.py", line 734, in dispatch
reactor.stop()
File "/usr/local/lib/python2.6/dist-packages/Twisted-8.2.0-py2.6-
linux-i686.egg/twisted/internet/base.py", line 527, in stop
"Can't stop reactor that isn't running.")
twisted.internet.error.ReactorNotRunning: Can't stop reactor that
isn't running.
done
removing '/tmp/nuwiki-U6LEIa'
Traceback (most recent call last):
File "/usr/local/bin/mw-render", line 8, in <module>
load_entry_point('mwlib==0.11.2', 'console_scripts', 'mw-render')
()
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-
linux-i686.egg/mwlib/apps/render.py", line 222, in main
return Main()()
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-
linux-i686.egg/mwlib/apps/render.py", line 182, in __call__
env = self.get_environment()
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-
linux-i686.egg/mwlib/apps/render.py", line 117, in get_environment
env = wiki.makewiki(self.zip_filename)
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-
linux-i686.egg/mwlib/wiki.py", line 268, in makewiki
script_extension=script_extension,
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-
linux-i686.egg/mwlib/wiki.py", line 218, in _makewiki
res.images = res.wiki = nuwiki.Adapt(zf)
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-
linux-i686.egg/mwlib/nuwiki.py", line 202, in __init__
self.nuwiki = nuwiki(path_or_instance)
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-
linux-i686.egg/mwlib/nuwiki.py", line 39, in __init__
self.nshandler = nshandling.nshandler(self.siteinfo)
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-
linux-i686.egg/mwlib/nshandling.py", line 55, in __init__
self.capitalize = self.siteinfo['general'].get('case') == 'first-
letter'
KeyError: 'general'
$

I've tried a couple of versions of the Collection extension, and
having a look at the code, none of the versions I have seem to refer
to the PDF Template Blacklist like (from what I can ascertain) older
versions in SVN. There is no PDF Template Blacklist virtual page in
the MediaWiki namespace for our local wiki. Could this be a mismatch
between the mwlib.rl software downloaded using the instructions
provided by http://code.pediapress.com/wiki/wiki/mwlib.rl-install, and
the latest (stable or development) version of the Collection
extension?

We are running MediaWiki 1.15.0 with PHP 5.2.9-2 and MySQL 5.1.35.

Of possible relevance (or at least for completeness), I had to insert
the line $text = ""; within the Collection extension's
Collection.body.php file (line 1021) at the start of the post()
function to avoid the function attempting to return an undefined
variable.

Any thoughts would be welcome.

Thanks.

Ralf Schmitt

unread,
Jul 15, 2009, 4:31:52 PM7/15/09
to mw...@googlegroups.com
On Wed, Jul 15, 2009 at 12:58 PM, google<goo...@iscience.com.au> wrote:
>
> Hi,
>
> I've managed to get the mw-render and mw-serve programs running
> successfully on Ubuntu 9.04. The test "mw-render -c :en -w rl -o
> sun.pdf Sun" (from http://code.pediapress.com/wiki/wiki/mwlib.rl-install)
> works just fine, but I cannot successfully render any page from a
> local (private) wiki.
>
> If I try an equivalent test command with the base URL directed to my
> private wiki, I get the following output (note the URL details have
> been obfuscated) in the mw-render.log.rl file:
>
> $ mw-render -c http://www.example.com/w/ -w rl -o test.pdf Collections

password authentication currently does not work. Is this wiki password
protected?

virtuallyme

unread,
Jul 16, 2009, 1:14:00 AM7/16/09
to mwlib
No, there is no password protection for read access - only for write
access. Permissions that have been explicitly set are as follows:

$wgGroupPermissions['user']['collectionsaveasuserpage'] = true;
$wgGroupPermissions['autoconfirmed']['collectionsaveascommunitypage']
= true;
$wgGroupPermissions['*']['read'] = true;
$wgGroupPermissions['*']['createaccount'] = false;
$wgGroupPermissions['*']['edit'] = false;
$wgGroupPermissions['*']['createpage'] = false;
$wgGroupPermissions['*']['createtalk'] = false;
$wgGroupPermissions['*']['writeapi'] = false;
$wgGroupPermissions['user']['createaccount'] = true;
$wgGroupPermissions['emailconfirmed']['edit'] = true;

I have tried changing the writeapi to true, but this didn't resolve
the problem.

Correction: The original post was the stdout/stderr output from mw-
render, not the mw-render.log.rl file itself. The contents of the mw-
render.log.rl are substantially similar to the following:

font not found: arphic/uming.ttc
font not found: ttf-sil-ezra/SILEOT.ttf
font not found: ttf-farsiweb/nazli.ttf
font not found: unfonts/UnBatang.ttf
font not found: ttf-thai-arundina/ArundinaSans.ttf
font not found: ttf-telugu-fonts/lohit_te.ttf
font not found: ttf-devanagari-fonts/Sarai_07.ttf
font not found: ttf-indic-fonts-core/lohit_pa.ttf
font not found: ttf-oriya-fonts/lohit_or.ttf
font not found: ttf-malayalam-fonts/AnjaliOldLipi.ttf
font not found: ttf-kannada-fonts/Kedage-n.ttf
font not found: ttf-bengali-fonts/LikhanNormal.ttf
font not found: ttf-indic-fonts-core/lohit_ta.ttf
font not found: linux-libertine/LinLibertine_Re.ttf
1% 2009-07-17T04:46:43 mwlib.utils.info >> fetching 'http://
www.example.com/w/api.php?action=query&meta=siteinfo&format=json'
2009-07-17T04:46:44 mwlib.options.warn >> Both --print-template-
pattern and --print-template-prefix (deprecated) specified. Using --
print-template-pattern only.
2009-07-17T04:46:44 mwlib.utils.info >> fetching 'http://
www.example.com/w/api.php?redirects=1&titles=MediaWiki:PDF+Template+Blacklist&format=json&action=query&rvprop=content&prop=revisions'
2009-07-17T04:46:44 mwapidb.error >> Could not get template blacklist
article u'MediaWiki:PDF Template Blacklist'
fatal: could not get siteinfo [[Failure instance: Traceback: <type
'exceptions.ValueError'>: No JSON object could be decoded
/usr/local/lib/python2.6/dist-packages/Twisted-8.2.0-py2.6-linux-
i686.egg/twisted/internet/defer.py:328:_runCallbacks
/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-linux-
i686.egg/mwlib/twisted_api.py:142:done
/usr/local/lib/python2.6/dist-packages/Twisted-8.2.0-py2.6-linux-
i686.egg/twisted/internet/defer.py:243:callback
/usr/local/lib/python2.6/dist-packages/Twisted-8.2.0-py2.6-linux-
i686.egg/twisted/internet/defer.py:312:_startRunCallbacks
--- <exception caught here> ---
/usr/local/lib/python2.6/dist-packages/Twisted-8.2.0-py2.6-linux-
i686.egg/twisted/internet/defer.py:328:_runCallbacks
/usr/lib/python2.6/json/__init__.py:307:loads
/usr/lib/python2.6/json/decoder.py:319:decode
/usr/lib/python2.6/json/decoder.py:338:raw_decode
]]
retrying: could not fetch 'http://www.example.com/w/api.php?
redirects=1&tllimit=500&format=json&revids=15&prop=revisions|templates|
images&action=query&imlimit=500&rvprop=ids'
error: could not get siteinfo

Any further suggestions very welcome. Thanks.

virtuallyme

unread,
Jul 21, 2009, 4:42:56 AM7/21/09
to mwlib
(user virtuallyme = user google from first post)

Ok, I discovered the mw-serve was calling our wiki's api.php that was
returning an error within the JSON formatted reply complaining that
the call to api.php cannot include a redirect and revids request at
the same time.

http://www.example.com/w/api.php?redirects=1&tllimit=500&format=json&revids=131&prop=revisions|templates|images&action=query&imlimit=500&rvprop=ids

Rather than attempt to modify the mwlib code, I created a rewrite rule
for IIS7.0 that removed the "redirects=1" query element from the full
URL. This seemed to allow the mw-serve to continue processing a little
further, but now it fails with a KeyError.

The following error was both returned directly to the user via HTML
(when the PDF Render page was updated) and written to the mw-
render.log.rl file in the render server cache:

An error occured on the render server: traceback Traceback (most
recent call last):
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-linux-
i686.egg/mwlib/apps/render.py", line 182, in __call__ env =
self.get_environment()
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-linux-
i686.egg/mwlib/apps/render.py", line 117, in get_environment env =
wiki.makewiki(self.zip_filename)
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-linux-
i686.egg/mwlib/wiki.py", line 268, in makewiki
script_extension=script_extension,
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-linux-
i686.egg/mwlib/wiki.py", line 218, in _makewiki res.images = res.wiki
= nuwiki.Adapt(zf)
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-linux-
i686.egg/mwlib/nuwiki.py", line 202, in __init__ self.nuwiki = nuwiki
(path_or_instance)
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-linux-
i686.egg/mwlib/nuwiki.py", line 39, in __init__ self.nshandler =
nshandling.nshandler(self.siteinfo)
File "/usr/local/lib/python2.6/dist-packages/mwlib-0.11.2-py2.6-linux-
i686.egg/mwlib/nshandling.py", line 55, in __init__ self.capitalize =
self.siteinfo['general'].get('case') == 'first-letter' KeyError:
'general'

Has anyone else managed to get the latest mwlib software to correctly
render pages hosted by a MediaWiki version 1.15.0?

Any other clues or thoughts regarding the KeyError or problems getting
mwlib to correctly interpret JSON responses also welcome. Thanks.

lasipisa

unread,
Aug 26, 2009, 3:07:54 PM8/26/09
to mwlib
> I've tried a couple of versions of the Collection extension, and
> having a look at the code, none of the versions I have seem to refer
> to the PDF Template Blacklist like (from what I can ascertain) older
> versions in SVN. There is no PDF Template Blacklist virtual page in
> the MediaWiki namespace for our local wiki. Could this be a mismatch
> between the mwlib.rl software downloaded using the instructions
> provided byhttp://code.pediapress.com/wiki/wiki/mwlib.rl-install, and
> the latest (stable or development) version of the Collection
> extension?
>
> We are running MediaWiki 1.15.0 with PHP 5.2.9-2 and MySQL 5.1.35.

Hi VirtuallyMe,

I'm trying something similar (MediaWiki 1.15, Xampplite 1.7.1) but in
Windows. I followed the Windows instructions at pediapress (http://
code.pediapress.com/wiki/wiki/
WindowsSetup#InstallingtheMediaWikiCollectionExtensiononWindows) and
there is a line that mentions you must create the MediaWiki:PDF
Blacklist. To create a MediaWiki: page, you must be logged in as
WikiSysOp:

Create "MediaWiki:PDF Template Blacklist" page

* enter "none" and save the page.

My render server error message looks similar to yours (http://
groups.google.com/group/mwlib/t/aeab042ae026b93e).
Reply all
Reply to author
Forward
0 new messages