Problems after tiddlywebwiki upgrade on tiddlyspace.gir.dk

[Måns]

Hi Chris

I asked my provider/host to upgrade the tiddlywebwiki on my
tiddlyspace http://tiddlyspace.gir.dk ...
It resulted in some odd errors...

Some of them are MySql related (I guess)
This was the error code he reported back to me:

[Wed Mar 13 16:39:54 2013] [error] [client XX.XXX.XXX.XX]
OperationalError: (OperationalError) (1005, "Can't create table
'tiddlyspace.first_revision' (errno: 150)") '\\nCREATE TABLE
first_revision (\\n\\ttiddler_id INTEGER NOT NULL, \\n\\tfirst_id
INTEGER NOT NULL, \\n\\tPRIMARY KEY (tiddler_id), \\n\\tUNIQUE
(tiddler_id, first_id), \\n\\tFOREIGN KEY(tiddler_id) REFERENCES
tiddler (id) ON DELETE CASCADE, \\n\\tFOREIGN KEY(first_id) REFERENCES
revision (number) ON DELETE CASCADE\\n)ENGINE=InnoDB CHARSET=utf8\\n\
\n' ()

"It's a full stacktrace, however both twanager and tiddlyweb fails on
it"

Is this somehow related to the upgraded jQuery in the TiddlyWebWiki??

Do we have to upgrade the whole package now?

Cheers Måns Mårtensson

[Måns]

>my provider/host

Should be sponsor :-)

Cheers Måns Mårtensson

[Måns]

"It's a full stacktrace, however both twanager and tiddlyweb fails on
it"

Should have been: "It's *NOT* a full stacktrace, however both twanager

and tiddlyweb fails on
it"

Cheers Måns Mårtensson

[chris...@gmail.com]

On Wed, 13 Mar 2013, M�ns wrote:

> I asked my provider/host to upgrade the tiddlywebwiki on my
> tiddlyspace http://tiddlyspace.gir.dk ...
> It resulted in some odd errors...

Can you find out what they actually installed/upgraded? Based on the
error you have reported it looks like they have upgrade all of
tiddlyweb, tiddlywebwiki, _and_ tiddlywebplugins.tiddlyspace (the
package). Doing the latter would most likely have brought in new database
code (moving from tiddlywebplugins.mysql or tiddlywebplugins.mysql2
to tiddlywebplugins.mysql3).

If that happened then the database schema in the new code is
different and a manual migration is required.

There's information about the migration process here:

http://tiddlyweb-sql.tiddlyspace.com/#%5B%5BMigration%20Process%5D%5D

The rest of that space has additional information about using
databases with tiddlyweb.

It ought to be possible to upgrade the tiddlywebwiki package without
upgrading the database packages, by just installing tiddlywebwiki.

If you can get the history of the installation that your
friend/sponsor did I can provide more definite information.

--
Chris Dent http://burningchrome.com/
[...]

[Måns]

Hi Chris

...There's information about the migration process here:

> http://tiddlyweb-sql.tiddlyspace.com/#%5B%5BMigration%20Process%5D%5D

> The rest of that space has additional information about using databases with tiddlyweb.

Sorry - We can´t access that space... needs login.

Cheers Måns Mårtensson

[chris...@gmail.com]

On Thu, 14 Mar 2013, M�ns wrote:

> Hi Chris
>
> ...There's information about the migration process here:
>
>> http://tiddlyweb-sql.tiddlyspace.com/#%5B%5BMigration%20Process%5D%5D

Strange the policies for that space are completely messed up.

While I fix it, you can view a copy of the main tiddler's text here:

http://csand.tiddlyspace.com/Migration%20Process

I should have the policies fixed up shortly.

[chris...@gmail.com]

On Thu, 14 Mar 2013, chris...@gmail.com wrote:

> On Thu, 14 Mar 2013, M�ns wrote:
>
>> Hi Chris
>>
>> ...There's information about the migration process here:
>>
>>> http://tiddlyweb-sql.tiddlyspace.com/#%5B%5BMigration%20Process%5D%5D
>
> Strange the policies for that space are completely messed up.

Should be fixed now: http://tiddlyweb-sql.tiddlyspace.com

Sorry about that.

[Kim Rostgaard Christensen]

Hi!

I'm the sponsor and admin.

Can you find out what they actually installed/upgraded? Based on the
error you have reported it looks like they have upgrade all of
tiddlyweb, tiddlywebwiki, _and_ tiddlywebplugins.tiddlyspace (the
package). Doing the latter would most likely have brought in new database
code (moving from tiddlywebplugins.mysql or tiddlywebplugins.mysql2
to tiddlywebplugins.mysql3).

For starters,  tiddlywebwiki was upgraded - but that broke some part of a space that Måns can tell you more about.

After a few desperate attempts to micro-fix and near-blindly upgrade packages (yes, I'm that scientific about it) - I finally decided to upgrade all packages.

After this, the database configuration was reset (in the /usr/something/dist-package subdir, AFAIR). Then I could connect, but it still failed to upgrade.

I followed your guide, but twanager migrate just gives me:

ERROR: No matching command found

Earlier I had the migrate on the usage list, but it seems to have disappeared after I reinstalled tiddlywebplugins.migrate.

/me is confused, and admits that his Tiddly-foo is weak.

[chris...@gmail.com]

On Thu, 14 Mar 2013, Kim Rostgaard Christensen wrote:

> I followed your guide, but twanager migrate just gives me:
> ERROR: No matching command found
>
> Earlier I had the migrate on the usage list, but it seems to have
> disappeared after I reinstalled tiddlywebplugins.migrate.

Did you get this step in the guide

* Add tiddlywebplugins.migrate to twanager_plugins in tiddlywebconfig.py.

?

If tiddlywebplugins.migrate is installed but "no matching command"
happens, then it is either:

* not configured for the instance _or_
* you're running the twanager command in some other directory than
the instance directory

In the instance directory there will be a tiddlywebconfig.py file,
containing a python dict with the name 'config'.

In there will be a key 'twanager_plugins'. The value is a list,
probably looks a bit like this:

'twanager_plugins': ['tiddlywebplugins.tiddlyspace'],

You need to add migrate to that:

'twanager_plugins': ['tiddlywebplugins.tiddlyspace',
'tiddlywebplugings.migrate'],

That should get the migrate command in place. Make sure you have set
target_store correctly, have created the target database and set its
privileges. And when the migration is done make sure that you change
target_store to server_store.

Sorry things have not been easier through this process. The growth of
the tiddlyspace.com server drove a lot of optimizing both in code and
the database schema.

[Kim Rostgaard Christensen]

On Thursday, March 14, 2013 8:23:54 PM UTC+1, cdent wrote:

On Thu, 14 Mar 2013, Kim Rostgaard Christensen wrote:

> I followed your guide, but twanager migrate just gives me:
> ERROR: No matching command found
>
> Earlier I had the migrate on the usage list, but it seems to have
> disappeared after I reinstalled tiddlywebplugins.migrate.

Did you get this step in the guide

* Add tiddlywebplugins.migrate to twanager_plugins in tiddlywebconfig.py.

?

No, I just realized I accidentally put it in system_plugins instead :-\

Now twanager migrate gives me

ERROR: OperationalError: ('(OperationalError) (1005, "Can\'t create table \'tiddlyspace.current_revision\' (errno: 150)")',)

[chris...@gmail.com]

* Is your old database named tiddlyspace
* Is server_store configured for tiddlysace
* Is your new (target) database named tiddlyspace2
* Is target_store configured for tiddlyspace2
* Have you set privileges properly on tiddlyspace
* Are you running `twanager migrate` as the user who is configured to
be able to access the database (both according target_store's values
and the values you used when granting privileges)

If those things aren't true, or if you're customization don't follow a
similar logic then you'll want to go through the steps and make sure
you have the right setup.

If you're an IRC user, I'll be on irc.freenode.net in the #tiddlyspace
and #tiddlyweb channels throughout most of this evening if you continue
to have issues and you prefer that kind of interaction. Here's fine too.

[Kim Rostgaard Christensen]

On Thursday, March 14, 2013 9:03:19 PM UTC+1, cdent wrote:

On Thu, 14 Mar 2013, Kim Rostgaard Christensen wrote:

> Now twanager migrate gives me
> ERROR: OperationalError: ('(OperationalError) (1005, "Can\'t create table
> \'tiddlyspace.current_revision\' (errno: 150)")',)

The errors (1005 and 150) indicate that it is a foreign key constraint that fails.

 http://dev.mysql.com/doc/refman/5.5/en/innodb-foreign-key-constraints.html

* Is your old database named tiddlyspace

Yep 

* Is server_store configured for tiddlysace

Yep, but in /usr/local/lib/python2.6/dist-packages/tiddlywebplugins/tiddlyspace/config.py

See below.

* Is your new (target) database named tiddlyspace2

Yep 

* Is target_store configured for tiddlyspace2

Yep, see below

* Have you set privileges properly on tiddlyspace

How do I test this?

 

* Are you running `twanager migrate` as the user who is configured to
    be able to access the database (both according target_store's values
    and the values you used when granting privileges)

The two users don't have access to each others databases, I will try to give them that, for the time being.

If those things aren't true, or if you're customization don't follow a
similar logic then you'll want to go through the steps and make sure
you have the right setup.

 

Everything is setup according to the guide, with the exception that the db_config is defined in /usr/local/lib/python2.6/dist-packages/tiddlywebplugins/tiddlyspace/config.py;

 'server_store': ['tiddlywebplugins.tiddlyspace.store', {

        'db_config': 'mysql:///tiddlyspace?user=blala&pass=nothintoseehere}]

For the last step, how do I assert that I run as the same user?

My current tiddlywebconfig.py

config = {

    'system_plugins': ['tiddlywebplugins.tiddlyspace'],

    'secret': '???',

    'twanager_plugins': ['tiddlywebplugins.tiddlyspace','tiddlywebplugins.migrate'],

    'server_host': {

            'scheme': 'http', # or https

            'host': 'tiddlyspace.gir.dk',

            'port': '80'

    },

    'target_store': ['tiddlywebplugins.mysql3', {

                'db_config': 'mysql:///tiddlyspace2?user=tiddlyweb&charset=utf8&use_unicode=0'}]

}

 

If you're an IRC user, I'll be on irc.freenode.net in the #tiddlyspace
and #tiddlyweb channels throughout most of this evening if you continue
to have issues and you prefer that kind of interaction. Here's fine too.

I'm also on freenode, but here is fine for now - it's up to you.

Personally, I think it is better to leave it public for others who may be in the same situation.

[chris...@gmail.com]

Sorry for the delay in responding: was looking to see if I could find
anything relevant in my notes from the migrations I've done on
tidldyspace.com. And having dinner.

On Thu, 14 Mar 2013, Kim Rostgaard Christensen wrote:

>>> Now twanager migrate gives me
>>> ERROR: OperationalError: ('(OperationalError) (1005, "Can\'t create
>> table
>>> \'tiddlyspace.current_revision\' (errno: 150)")',)
>>
>
> The errors (1005 and 150) indicate that it is a foreign key constraint that
> fails.
> http://dev.mysql.com/doc/refman/5.5/en/innodb-foreign-key-constraints.html

Hard to know at this point if this is cause or symptom of something
else. Before each migration attempt make sure you are dropping the
new database, to clear any metadata. I found that innodb gets very
very persnickety when the starting conditions aren't just right.

Searching google shows that that error will often show up when
creating tables (that have foreign key relations) fails for any reason.
If the tables aren't all created at the same time, there can be issues.

>> * Have you set privileges properly on tiddlyspace
>
> How do I test this?

I meant "tiddlyspace2", meaning "did you do the GRANT" step? Sorry for
the typo.

>> * Are you running `twanager migrate` as the user who is configured to
>> be able to access the database (both according target_store's values
>> and the values you used when granting privileges)
>>
> The two users don't have access to each others databases, I will try to
> give them that, for the time being.

Assuming you are configuring the user for the databases in the
server_store and target_store entries, then using a different database
user for each should be okay. In my own use of tiddlyweb and mysql I
always use the same Unix user for the tiddlyweb web instance and for
the database user (so don't need to config the user in server_store
and target_store). As you've discovered you can set the user and
password for the database in config and that ought to work.

As far as I know you're the second person to attempt this and you're
doing it under difficult circumstances given the awkward starting
conditions.

My notes from going from mysql2 to mysql3 are here

http://cdent.tiddlyspace.com/20120321

but not likely to be that useful. Most of my problems came about from
mysqld getting very upset when I tried to rename the database.

Actually just now writing "difficult circumstances" has given me an
idea. You've already upgraded the tiddlyspace code, which means that
your server_store is pointing indirectly to mysql3 thus trying to use
the wrong schema on an existing database, so try this:

* Make sure anything (like apache) that would be reading the tiddlyweb
store is shut down.

* Determine if your old tiddlyweb mysql code was
tiddlywebplugins.mysql or tiddlywebplugins.mysql2. Looking in
/usr/local/lib/python2.6/dist-packages/tiddlywebplugins/ will likely
have 2 of mysql, mysql2 and mysql3. Whichever of mysql or mysql2 you
have, use that in server_store below.

* Edit tiddlywebconfig.py in the instance to set target_store and
override server_store:

'server_store': ['tiddlywebplugins.mysql', { 'db_config':
'mysql:///tiddlyspace?user=blala&pass=nothintoseehere}],

'target_store': ['tiddlywebplugins.mysql3', { 'db_config':
'mysql:///tiddlyspace2?user=tiddlyweb&charset=utf8&use_unicode=0'}]

* Do the `twanager migrate`

* Edit tiddlywebconfig.py again to change target_store to

'target_store': ['tiddlywebplugins.tiddlyspace.store', { 'db_config':
'mysql:///tiddlyspace2?user=tiddlyweb&charset=utf8&use_unicode=0'}]

OR update the config.py file in dist-packages you were using originally,
accordingly.

* Do the lbags and other recommended tests.

* Let me know if it works. If it does I'll update the migration
tiddler to include special instructions for tiddlyspace.

>> If you're an IRC user, I'll be on irc.freenode.net in the #tiddlyspace
>> and #tiddlyweb channels throughout most of this evening if you continue
>> to have issues and you prefer that kind of interaction. Here's fine too.
>>
> I'm also on freenode, but here is fine for now - it's up to you.
>
> Personally, I think it is better to leave it public for others who may be
> in the same situation.

Ah nice, if only more people felt this way.

[chris...@gmail.com]

On Thu, 14 Mar 2013, chris...@gmail.com wrote:

> * Let me know if it works. If it does I'll update the migration
> tiddler to include special instructions for tiddlyspace.

Did you manage to get your tiddlyspace instance operational again?

[Kim Rostgaard Christensen]

Hi Chris

Sorry for the high latency.

Sort of. I managed to migrate, which took a while - but eventually it came back up. I used your instructions, thank you very much for them.

However, there are still some issues with rights, that I have only briefly looked into. Nothing appears in the logs unfortunately. Well nothing apart from seemingly unrelated IOerrors.

Måns have more information on this, as this seems less sysadmin-related.

Best

Kim

[chris...@gmail.com]

On Mon, 25 Mar 2013, Kim Rostgaard Christensen wrote:

> Sorry for the high latency.
> Sort of. I managed to migrate, which took a while - but eventually it came
> back up. I used your instructions, thank you very much for them.

Thanks for the update. I've added some notes and references to this
thread to the @tiddlyweb-sql[1] space.

> However, there are still some issues with rights, that I have only briefly
> looked into. Nothing appears in the logs unfortunately. Well nothing apart
> from seemingly unrelated IOerrors.

The IOerrors are likely an artifact of some versions of mod_wsgi.

> M�ns have more information on this, as this seems less sysadmin-related.

I'll look forward to hearing more.

http://tiddlyweb-sql.tiddlyspace.com

[Måns]

Hi Chris

I can create a new user. I can create new tiddlers via the bookmarksplugin, however I cannot create or modify tiddlers in any space!?
Check out the tester space http://tester.tiddlyspace.gir.dk pswd: tester

Cheers Måns Mårtensson

Den tirsdag den 26. marts 2013 14.51.22 UTC+1 skrev cdent:

On Mon, 25 Mar 2013, Kim Rostgaard Christensen wrote:

> Sorry for the high latency.
> Sort of. I managed to migrate, which took a while - but eventually it came
> back up. I used your instructions, thank you very much for them.

Thanks for the update. I've added some notes and references to this
thread to the @tiddlyweb-sql[1] space.

> However, there are still some issues with rights, that I have only briefly
> looked into. Nothing appears in the logs unfortunately. Well nothing apart
> from seemingly unrelated IOerrors.

The IOerrors are likely an artifact of some versions of mod_wsgi.

> M�ns have more information on this, as this seems less sysadmin-related.

[chris...@gmail.com]

On Thu, 28 Mar 2013, M�ns wrote:

> I can create a new user. I can create new tiddlers via the bookmarksplugin,
> however I cannot create or modify tiddlers in any space!?
> Check out the tester space http://tester.tiddlyspace.gir.dk pswd: tester

Have you done a 'twanager update' in the instance? This needs to be
done after upgrading any tiddlywebplugin which includes tiddlers (for
example tiddlywebplugins.tiddlyspace and tiddlywebwiki).

It looks like some of the plugins may be out of date. I was able to
PUT a tiddler just fine with curl.

The console of the browser is saying this:

Error: Syntax error, unrecognized expression: #[task=tiddlyspace]

[Måns]

Hi Chris

 >Have you done a 'twanager update' in the instance? This needs to be

done after upgrading any tiddlywebplugin which includes tiddlers (for
example tiddlywebplugins.tiddlyspace and tiddlywebwiki).

 

 Does Kim have to do a 'twanager update' in each and every space instance?
The tester space was created after his update - still it needs a 'twanager update'?
- or is 'the' instance a single 'unit' which affect all other space instances?

Cheers Måns Mårtensson

[chris...@gmail.com]

On Fri, 29 Mar 2013, M�ns wrote:

> Does Kim have to do a 'twanager update' in each and every space instance?
> The tester space was created after his update - still it needs a 'twanager
> update'?
> - or is 'the' instance a single 'unit' which affect all other space
> instances?

In this case the "instance" I mean is the tiddlyweb instance which
is hosting the tiddlyspace service:

http://tiddlyweb.tiddlyspace.com/instance

So wherever your tiddlywebconfig.py is, in there, as the correct
user you probably need to run 'twanager update', just once.

[Måns]

Hi Chris

Thanks for the clarification.

Now everything seems to work as intended.
Thank you for every bit of advice and support on the migration process
- when it was needed.
Kim did a GREAT job!!! :-D

Thanks

Cheers Måns Mårtensson

On 29 Mar., 11:47, chris.d...@gmail.com wrote:

[chris...@gmail.com]

On Fri, 29 Mar 2013, M�ns wrote:

> Now everything seems to work as intended.

Excellent, very glad to hear it.

How's the behavior on the sever? Faster? Slower? About the same?

[Måns]

Hi Chris

> Now everything seems to work as intended.

Not quite there yet I'm afraid...

"This space" and "Account" http://mama.tiddlyspace.gir.dk/_space -
http://mama.tiddlyspace.gir.dk/_account don't seem to work yet...

"This space" works when I look at other users spaces eg.
http://birthe.tiddlyspace.gir.dk/_space or one of my subspaces eg.:
http://dansk.tiddlyspace.gir.dk/_space

However Birthe wrote that : "Your account Find Space or View all,
don't seem to work." Which indicates that if you're logged in, you
don't get the information about your own user space....

Cheers Måns Mårtensson

[chris...@gmail.com]

On Fri, 29 Mar 2013, M�ns wrote:

> Hi Chris
>
>> Now everything seems to work as intended.
>
> Not quite there yet I'm afraid...

There appear to be a few different problems.

> "This space" and "Account" http://mama.tiddlyspace.gir.dk/_space -
> http://mama.tiddlyspace.gir.dk/_account don't seem to work yet...

In that space _space and _account are not the correct tiddlers. If you
load

http://mama.tiddlyspace.gir.dk/_account.json?fat=0

you'll see that it is coming from the takenote_public bag, and
overloading the default location. Compare with:

http://tester.tiddlyspace.gir.dk/_account.json?fat=0

where it comes from the tiddlyspace bag.

On a working /_account page there is an error from:

http://tester.tiddlyspace.gir.dk/spaces?mine=1

This suggests that somehwere server_store is still set pointing to
tiddlywebplugins.mysql3 instead tiddlywebplugins.tiddlyspace.store.

This will either be in tiddlywebconfig.py or in
tiddlywebplugins.tiddlyspace.config.