step by step mezzanine deployment

[Federico Bruni]

I'm opening a new thread and try to make the deploy of a simple test. Let's see if I understand the basics...

First of all, the DATABASES options are divided in 3 files:

- local_settings.py: the setting for local development. By default, sqlite is used. I wonder what's the best approach if I know that I want to deploy on PostgreSQL on the server. Should I work on PostgreSQL locally as well?

- live_settings.py: this also is pretty clear, it contains the server settings (I found out that it can be left untouched, as long as settings.py is complete)

- settings.py: IIUC, any settings here will be overridden by the two settings above, so I can ignore the DATABASE section. Is it correct?

Now, I assume that Fabric will handle the migration from sqlite to postgresql.

So all I have to do to deploy is defining the FABRIC={} variables in settings.py

I've learned that I'd better split 'fab all' in three steps. I'm trying to understand which options/settings are required for each step (it would be nice to organize them this way, sequentially, in default settings.py):

1. fab install

This is the easier part. It just installs the required packages on the server.

Options to be defined on FABRIC section of settings.py are: SSH_USER, SSH_KEYPATH, HOSTS, LOCALE, SECRET_KEY, NEVERCACHE_KEY.

2. fab create

Another note to add in the documentation is that a remote git/mercurial server is needed.

In this step I have to fill: VIRTUALENV_HOME, PROJECT_NAME, REQUIREMENTS_PATH, REPO_URL, DB_PASS.

I see that DB_PASS is shadowed so it's safe putting the password in the file (repository is private, of course). Or there's a better method?

3. fab deploy

I had nothing more to configure.

This test just worked smoothly. No error.

Now I should find out why my real project is failing. Maybe I could start it from scratch step by step and see what happens.

Anyway, I'd love to hear comments on above.. I may try writing a patch for the documentation to help other users, even if my understanding of mezzanine is still poor.

[Federico Bruni]

2014-03-03 13:01 GMT+01:00 Federico Bruni <fede...@gmail.com>:

Now I should find out why my real project is failing. Maybe I could start it from scratch step by step and see what happens.

I've just one more test: added a page in my local project and then launched 'fab deploy' to send the changes to the server database.

But nothing happens (I'm sure I've forced the reload of the page).

So I've managed to reproduce my original problem. Local and deployment databases are not synced when I launch 'fab deploy'.

No error if I manually launch 'python manage.py runserver' from the virtualenv in the server (after 'sudo supervisorctl stop all').

[Ken Bolton]

Fabric does not sync the content of the databases.

Fabric does sync the database schemas!

--
You received this message because you are subscribed to the Google Groups "Mezzanine Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mezzanine-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Federico Bruni]

Ah! Here's what I was missing!

Now I'm trying to manually sync the databases:

pg_dump -C -h localhost -U localuser dbname | psql -h remotehost -U remoteuser dbname

But I have problem with the port 5432. I already tried to add ip_private to listen_addresses in postgresql.conf, as suggested here:

http://stackoverflow.com/questions/19916720/psql-cant-connect-to-postgresql-server-postmaster-on-ip-and-port-5432

But it doesn't help:

psql: could not connect to server: Connection refused

Is the server running on host "xx.xx.xx.xx" and accepting

TCP/IP connections on port 5432?

pg_dump: [archiver (db)] connection to database "xxxxxx" failed: fe_sendauth: no password supplied

--
You received this message because you are subscribed to a topic in the Google Groups "Mezzanine Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mezzanine-users/FSCD3RAdLww/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mezzanine-use...@googlegroups.com.

[Ross Laird]

Postgresql authentication can be tricky.

You may need to make changes to your pg_hba.conf file. Like so:

# Database administrative login by Unix domain socket

local   all             postgres                                peer

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only

local   all             all                                     peer

# IPv4 local connections:

host    all             all             127.0.0.1/32            md5

# IPv6 local connections:

host    all             all             ::1/128                 md5

# Allow replication connections from localhost, by a user with the

# replication privilege.

#local   replication     postgres                                peer

#host    replication     postgres        127.0.0.1/32            md5

Also, this tutorial might help:

http://rosslaird.com/blog/building-a-project-with-mezzanine/

[Federico Bruni]

2014-03-03 23:33 GMT+01:00 Ross Laird <ro...@rosslaird.com>:

Postgresql authentication can be tricky.

You may need to make changes to your pg_hba.conf file. Like so:

# Database administrative login by Unix domain socket

local   all             postgres                                peer

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only

local   all             all                                     peer

# IPv4 local connections:

host    all             all             127.0.0.1/32            md5

# IPv6 local connections:

host    all             all             ::1/128                 md5

# Allow replication connections from localhost, by a user with the

# replication privilege.

#local   replication     postgres                                peer

#host    replication     postgres        127.0.0.1/32            md5

Ross, thanks for the tips!

My pg_hba.conf is exactly the same.

 

Also, this tutorial might help:

http://rosslaird.com/blog/building-a-project-with-mezzanine/

I realized that I was not using the correct owner of the database on the server
I've tried changing listen_addresses to accept any using * (as in your tutorial) and now I get a different error, which I translate as:

 no entry in pg_hba.conf for host "x.x.x.x", user "project", database "project", SSL disabled

The host is my public IP. It's listening to any address but it needs a specific entry for my IP, so I've added:

host    all             all             x.x.x.x/32            md5

I've repeated the commands above to sync the databases and I got no messages in the terminal.

However, the problem persists. I've also tried to repeat the deploy with 'fab remove & fab create & fab deploy', but it's not using my database.

My last option is manually dumping the database. So I tried:

1. fab create

2. pg_dump -Fc dbname > db.dump (on localhost and copied db.dump on the server)

3. pg_restore -d project db.dump (on the server)

4. fab deploy

But I get some errors when I run pg_restore and when I run 'fab deploy' I get this error:

django.db.utils.ProgrammingError: ERROR:  permission denied for the relation django_content_type

I'm starting running out of hope...

Your tutorial  about deployment is still on your TODO list?

[Ken Bolton]

Pro Tip

Assuming you have reasonable values in your settings.FABRIC, the following Fabric commands run within your virtualenv will do the backup and restore. I've pasted the code they run below each command.

$ fab backup

"pg_dump -Fc %s > %s" % (env.proj_name, filename)

$ fab restore

"pg_restore -c -d %s %s" % (env.proj_name, filename)

hth,

ken

[Ross Laird]

I don't actually use Fabric, so I have been a bit hesitant to make suggestions here. But my guess is that you have one problem and one problem only, and it just keeps changing how it looks depending on what you do. To step through a solution to this, I would take Fabric out of the equation and get the database connection running properly from the command line only (using pg_dump, pg_restore, and syncdb). If those commands work from the command line, then the problem is in the Fabric configuration. If those commands do not work from the command line, then you have a connection problem that sounds like it is related to permissions. In that case, try using psql from the command line to see who owns the database you are trying to connect to. Take as many things out of this equation as you can. Get right down to the core of it: who owns the database, what it's called, what its permissions are, and so on. Your problem lies in one of these areas, I think. Everything else that you try to do (using Fabric, Django, etc.) will fail -- and will give you different error messages -- until this one thing is resolved.

Here's a working example. On my live system, if I type "psql" then "\l" (that's a backslash, then the lowercase letter L), I see this kind of thing:

rosslaird=# \l

                                  List of databases

   Name    |   Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   

-----------+-----------+----------+-------------+-------------+-----------------------

 db_name   | joe       | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 

 other_db  | mary      | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 

 third_db  | ed        | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 

 postgres  | postgres  | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 

... 

(8 rows)

I can see, very easily, who owns what. Then I can look at the permissions of the user who owns the database in question, by typing "\du" (backslash, the lowercase letters D and U). 

Start there and see what you find.

[Federico Bruni]

2014-03-04 18:15 GMT+01:00 Ross Laird <ro...@rosslaird.com>:

I don't actually use Fabric, so I have been a bit hesitant to make suggestions here. But my guess is that you have one problem and one problem only, and it just keeps changing how it looks depending on what you do. To step through a solution to this, I would take Fabric out of the equation and get the database connection running properly from the command line only (using pg_dump, pg_restore, and syncdb). If those commands work from the command line, then the problem is in the Fabric configuration. If those commands do not work from the command line, then you have a connection problem that sounds like it is related to permissions. In that case, try using psql from the command line to see who owns the database you are trying to connect to. Take as many things out of this equation as you can. Get right down to the core of it: who owns the database, what it's called, what its permissions are, and so on. Your problem lies in one of these areas, I think. Everything else that you try to do (using Fabric, Django, etc.) will fail -- and will give you different error messages -- until this one thing is resolved.

Thanks Ross! This is indeed the best approach.

I've found out that part of my problem is that the database and owner names are different between local and server. I've solved following this tip:

http://stackoverflow.com/questions/12233046/django-permission-denied-when-trying-to-access-database-after-restore-migratio

and in particular:

psql database -c "GRANT ALL ON ALL TABLES IN SCHEMA public to user;"

psql database -c "GRANT ALL ON ALL SEQUENCES IN SCHEMA public to user;"

psql database -c "GRANT ALL ON ALL FUNCTIONS IN SCHEMA public to user;"

After this change, I did:

sudo supervisorctl stop all

python manage.py runserver

The server starts without errors but still it's not my local website.

I'll try again from scratch a new project and set up immediately PostgreSQL also in local_settings.py, before running 'manage.py createdb'.

[Federico Bruni]

2014-03-05 17:25 GMT+01:00 Federico Bruni <fede...@gmail.com>:

I'll try again from scratch a new project and set up immediately PostgreSQL also in local_settings.py, before running 'manage.py createdb'.

I've done the sync local>server manually, set up nginx on the server and started mezzanine in background. Everything works fine finally!

It was probably something related to the live settings. I'll try again a deployment with fabric as soon as I have time.

Big thank to Ross, who found the good words to put me on the right track.

[Ross Laird]

Glad you got it to work.

Web development is always an adventure...

[Federico Bruni]

2014-03-03 13:01 GMT+01:00 Federico Bruni <fede...@gmail.com>:

First of all, the DATABASES options are divided in 3 files:

- local_settings.py: the setting for local development. By default, sqlite is used. I wonder what's the best approach if I know that I want to deploy on PostgreSQL on the server. Should I work on PostgreSQL locally as well?

- live_settings.py: this also is pretty clear, it contains the server settings (I found out that it can be left untouched, as long as settings.py is complete)

- settings.py: IIUC, any settings here will be overridden by the two settings above, so I can ignore the DATABASE section. Is it correct?

Can anyone please reply on the last question above?

I've managed to sync local and server manually, but I'd like to be able to do it with Fabric.

One problem may be that the owner of the database created by Fabric is always %(proj_name)s, no matter what I put in the settings. I've tried to write the server user name (string) both in settings and live_settings.py, but fabric seems to ignore it.

[Federico Bruni]

The awful "workaround" is deleting the database created by fabric and then doing a manual dump.

My site is live now, but I'd really like to know what's wrong with my configuration.

If someone wants to help me privately, send me a message. I can offer some small donation for the help.

Thanks

Federico

[Eduardo Rivas]

Hi Federico. The DATABASE setting that is applied is the one in live_settings.py. In production, this file is renamed to local_settings.py by Fabric (the local_settings.py in your dev machine should NOT be in version control). You'll see at the end of settings.py that it imports everything from local_settings.py, thus overriding the DATABASE from settings.py.

[Federico Bruni]

Hi Eduardo

thanks for the explanation! It would be nice if it was included in the documentation...

2014-03-07 1:06 GMT+01:00 Eduardo Rivas <jeriva...@gmail.com>:

Hi Federico. The DATABASE setting that is applied is the one in live_settings.py. In production, this file is renamed to local_settings.py by Fabric (the local_settings.py in your dev machine should NOT be in version control). You'll see at the end of settings.py that it imports everything from local_settings.py, thus overriding the DATABASE from settings.py.

What's the purpose of DATABASE in settings.py then?

[Federico Bruni]

Or rather, why I don't keep a message in the drafts and send it just the day after? :-)

I've already been told that Fabric doesn't sync the content of the database. So the manual operation _is_ required.

Sorry for the noise, I've had few time to spend on Mezzanine lately

[Eduardo Rivas]

To he honest I don't know why the DATABASE setting is left in settings.py, it definitely can be confusing.

I believe the reason databases are not synced by the fabfile is because that's not a very common (or advised) operation. Most of the time developers want to keep production and development DBs completely separated. However, I can think of a few situations where this could be useful, and the general algorithm for a Fabric task would be:

1. Run pg_dump in your dev machine, output to file.
2. Upload file to the project folder in the server.
3. (Optional) Backup current production DB.
   
4. Run pg_restore in the server from the latest uploaded SQL dump.
   

[Ken Bolton]

Hi Eduardo,

I wrote some Fabric scripts for my own use to do a more complicated pg dump and restore. I had a production system to migrate from Mezzanine pre 1.x to 1.x. I used Fabric and Vagrant to iterate over write, build, test against a restore from a recent production dump. https://github.com/stephenmcd/mezzanine/blob/master/mezzanine/project_template/fabfile.py#L289-L302 contains the `backup` and `restore` tasks. I used  the `backup` task to build a `backup_production` task. The `create` task got modifications to `restore` the backup from production.

It was all quite complicated. I would have used fixtures, but Mezzanine had an issue with dumping FKs, although it might have been me.

k

--

You received this message because you are subscribed to the Google Groups "Mezzanine Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mezzanine-use...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Eduardo Rivas]

That's great, Ken! Maybe you can share some of your tasks with Federico?

I've gone down the fixture path, and the two things I needed to get it working were:

1. Always use the --natural flag.
2. Don't do a huge, all-apps dump; rather specify the apps you wish to dump to the fixture. Off the top of my head, if you need all Page entries from the db: python manage.py dumpdata --natural pages forms galleries. Combine this with the fact that the fabfile creates the site and user objects for you, and it might be all you need to sync.