Keyspace not found! (But it exists under /var/lib/cassandra)

[Pantelis Sopasakis]

Hi All,
I have a cassandra database with one column family which contains
some tens of thousands of data I wouldn't like to loose. My keyspace is
called "Chemaster". I start the cassandra server and I connect with the
client and I try to use Chemaster. However it says it is not found:

[default@unknown] use Chemaster;
Keyspace 'Chemaster' not found.

I'm using cassandra version 1.1.0.

Note: I can see that there are files
under /var/lib/cassandra/data/Chemaster

any hints?

Thank you in advance,
Pantelis Sopasakis

[David Prinzing]

I've found that when restoring a keyspace on a new Cassandra node, I need to (1) tell Cassandra about the keyspace, usually with a CQL script, so that it can update System tables, and (2) copy the snapshot files into the appropriately-named folder, in your case /var/lib/cassandra/data/Chemaster. The procedure I use: (1) set up new Cassandra node; (2) run the schema CQL script; (3) shut down Cassandra; (4) copy files in, and (5) restart Cassandra. Then all the data expected is present.

Cheers,

-dp-

[Pantelis Sopasakis]

Hi folks,
I'm not sure I found the solution but talking on IRC#cassandra I
found the issue: https://issues.apache.org/jira/browse/CASSANDRA-4235 :
keyspaces missing after restart which looks like a very nasty one. It's
marked as unresolved for the moment but there's another issue, namely:

https://issues.apache.org/jira/browse/CASSANDRA-4219 : Problem with
creating keyspace after drop

which supposedly should fix 4235 as well. I cloned the current unstable
version from github, compiled and I keep fingers crossed... I hope these
issues will be resolved in the coming version 1.1.1.

Regards,
Pantelis Sopasakis

[Pantelis Sopasakis]

Hi David,
After a long discussion in #cassandra I found out that the problem is
related to the 'system' keyspace. However if you delete your keyspace it
will not be refreshed on startup. I did the following experiment and it
worked so I suggest it as a rescue guide:

0. Before it is too late export your KS schema: Create a file called
'schema_export' and add the following lines inside:
use MyKeyspace;
show schema;
then run this cassandra script and save the output in another file:
cassandra-cli < schema_export > my.schema
keep this file under lock and key! If you change your schema, remember
to update the file.
1. Stop the server
1. Delete 'system': rm -r /var/lib/cassandra/data/system
2. Start the server
3. Load the schema: cassandra-cli < my.schema
4. Restart the server and connect using the client. Apply a 'use
MyKeyspace' and you'll see that your keyspace is there. Also your data
is there.

Regards,
Pantelis

Pantelis Sopasakis, ch...@mail.ntua.gr
Dipl. Chemical Engineer NTUA
Msc. Applied Mathematics NTUA
Automatic Control Unit, School of Chem. Eng., NTUA
Heroon Polytechneiou 9, 15780 Zografou Campus, Athens, Greece
Tel. (+30) 210 772 3236 (office)

Research fellow at Helmholtz Zentrum München
Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
Ingolstädter Landstr. 1
85764 Neuherberg
pantelis....@helmholtz-muenchen.de

[Pantelis Sopasakis]

I forgot to mention an important detain in this workflow. When you
create the file my.schema, edit it and remove certain lines that are not
executable like:

Connected to: "Test Cluster" on 127.0.0.1/9160
Welcome to Cassandra CLI version 1.2.0-SNAPSHOT

Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.

or

Authenticated to keyspace: Chemaster

otherwise you will not be able to load it.

Regards,
Pantelis Sopasakis

[David Prinzing]

Ah, yes. We do something similar, only using cqlsh instead of cassandra-cli:

1. Create a file named, for example, 'getChemasterSchema.cql' containing the following single line as its contents:

DESCRIBE KEYSPACE Chemaster;

2. Run cqlsh with a command like the following (substitute your hostname):

cqlsh hostname < getChemasterSchema.cql > ChemasterSchema.cql

3. This creates the CQL script that can be used to restore the schema in the future with the following command:

cqlsh hostname < ChemasterSchema.cql

Note that you might need to modify the cqlsh python script, depending on your version.

See: https://issues.apache.org/jira/browse/CASSANDRA-4113

As a database backup regimen, we have a nightly cron job that takes a snapshot of the database, exports the matching schema at that time (using the above), and copies both the schema and the snapshot off to another storage medium. Each daily snapshot is kept in a separate folder so that we can go back in time to a previous state of the database if we need to. For more on database backup and restore, this reference was helpful: http://www.datastax.com/docs/1.0/operations/backup_restore

Cheers,

-dp-

P.S. It's nice to see a fellow chemical engineer using Cassandra :)

I'm a Professional Chemical Engineer (PE), licensed to practice in the States of Washington and California, here in the USA.

pantelis.sopasakis@helmholtz-muenchen.de

[Nate McCall]

Thanks for all the details guys - I'm glad this got worked out.

>> pantelis....@helmholtz-muenchen.de
>>
>>
>>
>