Python Error tiddlywebplugins.wimporter

[Birgit Ducarroz]

Hi,

Please would someone me give a suggestion how I might correct following error:
http://<servername>:8000/import

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/httpexceptor/__init__.py", line 58, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/wsgi.py", line 190, in __call__
    output = self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/serve.py", line 161, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/serve.py", line 117, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/query.py", line 44, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/wsgi.py", line 128, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/extractor.py", line 34, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/wsgi.py", line 38, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/negotiate.py", line 30, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/selector.py", line 137, in __call__
    return app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlywebplugins/utils.py", line 32, in _entitle
    output = handler(environ, start_response, *args, **kwds)
  File "/usr/local/lib/python2.7/dist-packages/tiddlywebplugins/utils.py", line 60, in _do_html
    output = handler(environ, start_response, *args, **kwds)
  File "/home/data/www-sysadmin/Sysadmin/tiddly/tiddlywebplugins/wimporter.py", line 49, in interface
    return _send_wimport(environ, start_response)
  File "/home/data/www-sysadmin/Sysadmin/tiddly/tiddlywebplugins/wimporter.py", line 180, in _send_wimport
    template = get_template(environ, 'wimport.html')
  File "/usr/local/lib/python2.7/dist-packages/tiddlywebplugins.templates-0.9.7-py2.7.egg/tiddlywebplugins/templates.py", line 74, in get_template
    return TEMPLATE_ENV.get_template(template_name)
  File "/usr/local/lib/python2.7/dist-packages/Jinja2-2.8-py2.7.egg/jinja2/environment.py", line 812, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/usr/local/lib/python2.7/dist-packages/Jinja2-2.8-py2.7.egg/jinja2/environment.py", line 774, in _load_template
    cache_key = self.loader.get_source(self, name)[1]
  File "/usr/local/lib/python2.7/dist-packages/Jinja2-2.8-py2.7.egg/jinja2/loaders.py", line 399, in get_source
    raise TemplateNotFound(template)
TemplateNotFound: wimport.html

Here is my config file

tiddlywebconfig.py

config = {
    'system_plugins': ['tiddlywebplugins.wimporter', 'tiddlywebwiki'],
    'secret': '18e1230221d9333010136477905101005d41f4ec3',
    'twanager_plugins': ['tiddlywebwiki'],

        'server_host': {
        'scheme': 'http', # or https
        'host': '<servername>', # <-- i put my actual server name here
        'port': '8000', #
    }
}

The tiddly instance on

http://<servername>:8000/recipes/default/tiddlers.wiki

is running correctly.

Thank you a lot for some suggestions.

[chris...@gmail.com]

On Mon, 8 Feb 2016, Birgit Ducarroz wrote:

Hi, first off, sorry for your messages not getting through
initially: First time submissions go through a moderation queue.

The issue you're seeing is happening because you installation of
tiddlywebplugins.wimporter is in your tiddly directory where as
tiddlywebplugins.templates is in your dist-packages as shown here:

>> File "/home/data/www-sysadmin/Sysadmin/tiddly/tiddlywebplugins/wimporter.py", line 180, in _send_wimport
>> template = get_template(environ, 'wimport.html')
>> File "/usr/local/lib/python2.7/dist-packages/tiddlywebplugins.templates-0.9.7-py2.7.egg/tiddlywebplugins/templates.py", line 74, in get_template
>> return TEMPLATE_ENV.get_template(template_name)
>> File "/usr/local/lib/python2.7/dist-packages/Jinja2-2.8-py2.7.egg/jinja2/environment.py", line 812, in get_template
>> return self._load_template(name, self.make_globals(globals))
>> File "/usr/local/lib/python2.7/dist-packages/Jinja2-2.8-py2.7.egg/jinja2/environment.py", line 774, in _load_template
>> cache_key = self.loader.get_source(self, name)[1]
>> File "/usr/local/lib/python2.7/dist-packages/Jinja2-2.8-py2.7.egg/jinja2/loaders.py", line 399, in get_source
>> raise TemplateNotFound(template)
>> TemplateNotFound: wimport.html

This is fine, but it means that you need to add a link from your
"instance directory" to where the wimporter templates are. The
instance directory is the one where your tiddlywebconfig.py lives.
Either create a symlink in there from 'templates' to
'/home/data/www-sysadmin/Sysadmin/tiddlyweb/tiddlywebplugins/templates'
or make a directory in there named 'templates' and copy the contents
of the templates directory in there. If you do that it becomes easy
to customize the templates.


--
Chris Dent http://burningchrome.com/
[...]

[Birgit Ducarroz]

Thank you a lot Chris!
The symbolic link did the trick, but now, when trying to upload my local tiddlywiki, I run into the following error message:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/httpexceptor/__init__.py", line 58, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/wsgi.py", line 190, in __call__
    output = self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/serve.py", line 161, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/serve.py", line 117, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/query.py", line 44, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/wsgi.py", line 128, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/extractor.py", line 34, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/wsgi.py", line 38, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/negotiate.py", line 30, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/selector.py", line 137, in __call__
    return app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlywebplugins/utils.py", line 32, in _entitle
    output = handler(environ, start_response, *args, **kwds)
  File "/usr/local/lib/python2.7/dist-packages/tiddlywebplugins/utils.py", line 60, in _do_html
    output = handler(environ, start_response, *args, **kwds)
  File "/home/data/www-sysadmin/Sysadmin/tiddly/tiddlywebplugins/wimporter.py", line 64, in wimport
    _process_file(environ, files[0].file, tmp_bag)
  File "/home/data/www-sysadmin/Sysadmin/tiddly/tiddlywebplugins/wimporter.py", line 149, in _process_file
    store.put(tiddler)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/store.py", line 206, in put
    result = func(thing)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/stores/text.py", line 215, in tiddler_put
    raise NoTiddlerError('unable to put tiddler: %s' % exc)
NoTiddlerError: unable to put tiddler: [Errno 36] File name too long: "/home/data/www-sysadmin/Sysadmin/tiddly/store/bags/import-tmp-638c91d8-5dbd-4f2d-9543-030041cdecad/tiddlers/Warning%3A%20mysqldump%3A%20Couldn't%20execute%20'SELECT%20%2F*!40001%20SQL_NO_CACHE%20script%2F%20ntpstatus.txt%20script%20vi%20FROM%20%60EVENTS%60'%3A%20Cannot%20proceed%20because%20system%20tables%20used%20by%20Event%20Scheduler%20were%20found%20damaged%20at%20server%20start%20(1577)%20Warning%3A%20Failed%20to%20dump%20mysql%20databases%20information_schema"

I must say that my local tiddlywiki has 40'000 lines.
Do you think I should simply remove the tiddlers whose names are too long?

[chris...@gmail.com]

On Mon, 8 Feb 2016, Birgit Ducarroz wrote:

>> NoTiddlerError: unable to put tiddler: [Errno 36] File name too long: "/home/data/www-sysadmin/Sysadmin/tiddly/store/bags/import-tmp-638c91d8-5dbd-4f2d-9543-030041cdecad/tiddlers/Warning%3A%20mysqldump%3A%20Couldn't%20execute%20'SELECT%20%2F*!40001%20SQL_NO_CACHE%20script%2F%20ntpstatus.txt%20script%20vi%20FROM%20%60EVENTS%60'%3A%20Cannot%20proceed%20because%20system%20tables%20used%20by%20Event%20Scheduler%20were%20found%20damaged%20at%20server%20start%20(1577)%20Warning%3A%20Failed%20to%20dump%20mysql%20databases%20information_schema"
>>
>>
>>
> I must say that my local tiddlywiki has 40'000 lines.
> Do you think I should simply remove the tiddlers whose names are too long?

You've got a few options:

* In the tiddlywiki, before importing, either:
* remove the tiddlers with very long names
* rename the tiddlers to have shorter names
* Or during the import instead of selecting all tiddlers, select
only those with short names (or, more likely, select all and then
deselect those with very long names).

[Birgit Ducarroz]

ok, the import worked for a tiddlywiki containing 13618 lines.
It created a new bag http://servername:8000/bags/import-tmp-0438b80c-ba8f-42e8-83ea-aeabc28425b4/tiddlers.wiki
Two more questions:
1) How can I now import this into the

http://diuf-file:8000/recipes/default/tiddlers.wiki

2) The tiddlywiki containing 40'000 gives me still the following error - even after having deleted all the tiddlers containing too long names. And please note: this error appears directly, I don't get the intermetiate screen where I could select or deselect single tiddlers.

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/httpexceptor/__init__.py", line 58, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/wsgi.py", line 190, in __call__
    output = self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/serve.py", line 161, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/serve.py", line 117, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/query.py", line 44, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/wsgi.py", line 128, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/extractor.py", line 34, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/wsgi.py", line 38, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/web/negotiate.py", line 30, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/selector.py", line 137, in __call__
    return app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/tiddlywebplugins/utils.py", line 32, in _entitle
    output = handler(environ, start_response, *args, **kwds)
  File "/usr/local/lib/python2.7/dist-packages/tiddlywebplugins/utils.py", line 60, in _do_html
    output = handler(environ, start_response, *args, **kwds)
  File "/home/data/www-sysadmin/Sysadmin/tiddly/tiddlywebplugins/wimporter.py", line 64, in wimport
    _process_file(environ, files[0].file, tmp_bag)
  File "/home/data/www-sysadmin/Sysadmin/tiddly/tiddlywebplugins/wimporter.py", line 149, in _process_file
    store.put(tiddler)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/store.py", line 206, in put
    result = func(thing)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/stores/text.py", line 210, in tiddler_put
    tiddler_base_filename = self._tiddler_base_filename(tiddler)
  File "/usr/local/lib/python2.7/dist-packages/tiddlyweb/stores/text.py", line 497, in _tiddler_base_filename
    raise NoTiddlerError(exc)
NoTiddlerError

[Birgit Ducarroz]

oh, it seems that question 2) is not as evil as the error message. Indeed, the tiddlers were anyway imported into a bag. I found the 40'000 lines tiddly in URL http://servername:8000/bags/import-tmp-e474db68-9ce7-4ed2-b133-16b45549e6d7/
, but I had to add at the end of the import-tmp-code the string "tiddlers.wiki".

So my question is resuming to 1)

:-)

On Monday, 8 February 2016 13:42:49 UTC+1, Birgit Ducarroz wrote:

ok, the import worked for a tiddlywiki containing 13618 lines.
It created a new bag http://servername:8000/bags/import-tmp-0438b80c-ba8f-42e8-83ea-aeabc28425b4/tiddlers.wiki
Two more questions:
1) How can I now import this into the

http://servername:8000/recipes/default/tiddlers.wiki

2) The tiddlywiki containing 40'000 lines gives me still the following error - even after having deleted all the tiddlers containing too long names. And please note: this error appears directly, I don't get the intermetiate screen where I could select or deselect single tiddlers.

[chris...@gmail.com]

On Mon, 8 Feb 2016, Birgit Ducarroz wrote:

> ok, the import worked for a tiddlywiki containing 13618 lines.
> It created a new bag
> http://servername:8000/bags/import-tmp-0438b80c-ba8f-42e8-83ea-aeabc28425b4/tiddlers.wiki

As you can see from the name this is a temporary bag, not the
eventual destination bag. Wimporter works in two steps: It imports
the source wiki into a tmp bag (as above) and then presents the
intermediate screen where you can select all or some of those
tiddlers _and_ the target bag in which to put them. As you've noticed
you'll only get that screen if the initial import to temporary completes
without errors. You've not had that yet. Your current error seems to be
that you have a tiddler without a title.

To get your content into the /recipes/default/tiddlers.wiki you
could put it in the 'common' bag or create a new bag and add it to
the default recipe.

Note that there is another plugin, called
tiddlywebplugins.twimport[1], which you can use from the command
line to import a tiddlywiki directly into a bag without the
intervening selection step. For what you are doing it might be a
better choice.

[1] https://pypi.python.org/pypi/tiddlywebplugins.twimport

[Birgit Ducarroz]

First of all, thank's a lot for your responses!!
 

Your current error seems to be
that you have a tiddler without a title.

--> Unable to find this in a file containing 40'000 lines :( If you have any suggestions?

Note that there is another plugin, called
tiddlywebplugins.twimport[1], which you can use from the command
line to import a tiddlywiki directly into a bag without the
intervening selection step. For what you are doing it might be a
better choice.

--> This was the first plugin I tried. But unfortuately I get the following error: ERROR: ValueError: content not a tiddlywiki 2.x
I did not found howto update my existing tiddlywiki :(

[Birgit Ducarroz]

ok, I found a trick to import.

Finally, what did NOT work was:
even trough a coreupgrade using website http://tiddlywiki.com/upgrade.html, the
root@vm02:/home/data/www-sysadmin/Sysadmin/tiddly# twanager twimport common https://<servername>/<path>/<tiddlyname>.html
did not work because of several errors like ERROR: TiddlerFormatError: reserved key "type" in fields of tiddler: [warn] NameVirtualHost *:80 has no VirtualHosts

This worked (a bit complicated, but finally it did the trick):
*imported the 40'000 lines tiddly into a bag.
** checked which tiddlers did not import, using the not working backstage - import function
--> all those tiddlers which are NOT bold in the list, are missing
*repeated the above action 3 times (but each time deleted a range of tiddlers and let another range of tiddlers, using the multiple delete lines command of vi)
, in example
vi /<path>/tiddlywiki.html
:516,25056d (delete lines from 516 ti 25056d)

* recheck using the
backstage - import function

Then moved all the tiddlers into

/home/data/www-sysadmin/Sysadmin/tiddly/store/bags/common/tiddlers/

mv /home/data/www-sysadmin/Sysadmin/tiddly/store/bags/import-tmp-2c86db96-c96e-4d68-907a-b1dfcd8f31e5/tiddlers/* /home/data/www-sysadmin/Sysadmin/tiddly/store/bags/common/tiddlers/

Finally, just about 10 tiddlers were not able to be imported due to "To long tiddlernames" or due to "reserved key "type" in fields of tiddler" errors.
So I created those tiddlers manually into the webtiddly, changing their title

Happy it worked, took me 2 hours, but this is ok for me.

:)