Admin - saveasnew to another database

[Mike Dewhirst]

I'd like to use the [Save as new] infrastructure to transfer records and
their children to a different database.

Can anyone give me some advice on whether this is a reasonable project
to attempt? Would it be valuable for anyone else?

My alternative is to build a separate mechanism for my own use. Which I
have started but I can see it is going to get overly complex.

Thanks

Mike

[Tim Graham]

Interesting idea. I'd be interested to here more about the use case.

How would it work for entities with a foreign key? Django assumes foreign keys can't be cross-database since the constraint can't be enforced.

[Mike Dewhirst]

On 9/11/2016 9:13 AM, Tim Graham wrote:
> Interesting idea. I'd be interested to here more about the use case.

Simple. I permitted my users to "play" on the staging site and now one
of them wants to transfer their work to the production site. If I can
get it going it becomes a nice-to-have feature. Plan B is to manually
copy the records and laboriously ensure nothing has been missed. The
actual antithesis of having a computer doing stuff in the first place.
Bit like using liquid paper on the screen instead of backspace.

I'm using Django 1.8.x Admin. I thought I'd add a field called
"Transfer" and if transfer was triggered then I would call a utility
function to transfer the records.

The use case is a database of substances and mixtures of substances. A
mixture happens when multiple substances are linked to another substance
(the mixture) via a through table called Substance_Ingredients each
record of which carries the ingredient proportion. Then there are many
FK relationships between other tables containing different sets of
substance properties as applicable. Some of those are 1:1.

>
> How would it work for entities with a foreign key? Django assumes
> foreign keys can't be cross-database since the constraint can't be
> enforced.

Agreed. However we don't need to maintain relationships between
databases after the transfer. To help explain, here is the help_text for
the above-mentioned "transfer" field ...

help_text='Select /To the production database/ and click [Save] to
transfer this substance to the production database. Here are the
transfer rules:
1. The exact same Division must exist at the destination
2. This substance must not exist at the destination. A transfer can only
be done once.
3. Visit the production site to verify that the transfer was successful.
4. Recreate the SDS (which is not transferred)
Note that if this is a mixture, its ingredients will also be transferred
unless those ingredients belong to other Divisions/Companies. This means
some mixtures may be incomplete at the destination and thus may require
some rework.'

... which help_text might change if we can add a kwarg, say
db='production' so _saveasnew (in contrib/admin/options.py) can cross
the divide.

In any case, transferred records will necessarily have different ids in
the destination database so all transfers will need to be done using
queries. This is why piggy-backing on [Save as new] beats doing things
either manually or programmatically in my own utility functions.

Your thoughts?

Mike

>
> On Tuesday, November 8, 2016 at 4:29:48 PM UTC-5, Mike Dewhirst wrote:
>
> I'd like to use the [Save as new] infrastructure to transfer
> records and
> their children to a different database.
>
> Can anyone give me some advice on whether this is a reasonable
> project
> to attempt? Would it be valuable for anyone else?
>
> My alternative is to build a separate mechanism for my own use.
> Which I
> have started but I can see it is going to get overly complex.
>
> Thanks
>
> Mike
>

> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-users...@googlegroups.com
> <mailto:django-users...@googlegroups.com>.
> To post to this group, send email to django...@googlegroups.com
> <mailto:django...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/7d1bd5c7-cffa-4b4a-bf1b-7dbb1e18f504%40googlegroups.com
> <https://groups.google.com/d/msgid/django-users/7d1bd5c7-cffa-4b4a-bf1b-7dbb1e18f504%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Melvyn Sopacua]

On Wednesday 09 November 2016 13:59:35 Mike Dewhirst wrote:
> On 9/11/2016 9:13 AM, Tim Graham wrote:
> > Interesting idea. I'd be interested to here more about the use case.
>
> Simple. I permitted my users to "play" on the staging site and now one
> of them wants to transfer their work to the production site.

Your approach is wrong. You think of this as one django project and
trying to cheat under the hood, while in fact they are two different
projects. And by design, staging is not always structually identical to
production.

Your best approach is to use serialization with natural keys and a job
scheduler. Cron will do, but there's plenty other schemes allowing you
to provide feedback about the job queue and expected time of completion.
--
Melvyn Sopacua

[Tim Graham]

I see. Although I would be interested to see what you come up with, my initial reaction is that we wouldn't accept a patch for Django to do this. It seems too complex and brittle.

I agree with Melvyn that it seems unusual to use Django's multi-database facilities to transfer data between staging and production.

[Mike Dewhirst]

On 10/11/2016 12:06 AM, Tim Graham wrote:
> I see. Although I would be interested to see what you come up with,

@Tim
It is getting there so if you are really interested I'll post it
somewhere when it is working. I'm getting router errors at the moment
which I might be able to workaround by permitting null in a FK fields
during obj.save(using=db). Might ask some questions on this later. I
don't really want to permit null in this particular case.

> my initial reaction is that we wouldn't accept a patch for Django to
> do this. It seems too complex and brittle.

Having done some work on this I agree with you but disagree with Melvyn.

The task is complex and would be almost impossible to generalise. For
example, from a number of substances the user may have done serious work
on only a few and won't want the others transferred.

So I'm building a facility to transfer one substance at a time. If there
is a future requirement for bulk transfer I can iterate.

The transferred substance might be a mixture in which case I have to
collect the ingredient substances plus the m2m through records which are
effectively the recipe. Some of the ingredients might themselves be
mixtures. Not worrying about that at the moment.

For each substance there are 6 OneToMany, up to 15 OneToOne and two m2m
relationships plus a OneToOne where the sibling has lots of its own
complex relationships including a recursive OneToOne to form a
daisy-chain of history. I'm skipping that one for the moment.

> I agree with Melvyn

See below

> that it seems unusual to use Django's multi-database facilities to
> transfer data between staging and production.
>
> On Wednesday, November 9, 2016 at 4:51:39 AM UTC-5, Melvyn Sopacua wrote:
>
> On Wednesday 09 November 2016 13:59:35 Mike Dewhirst wrote:
> > On 9/11/2016 9:13 AM, Tim Graham wrote:
> > > Interesting idea. I'd be interested to here more about the use
> case.
> >
> > Simple. I permitted my users to "play" on the staging site and
> now one
> > of them wants to transfer their work to the production site.
>
> Your approach is wrong. You think of this as one django project and
> trying to cheat under the hood, while in fact they are two different
> projects. And by design, staging is not always structually
> identical to
> production.
>

@Melvyn

I was wrong to think saveasnew might do the trick.

However, migration is fully implemented so I can easily abort any
transfer if the structures are not identical. As it happens, there are
only two databases at the moment. Staging and production and while they
are not always structurally identical they are currently the same during
testing.

>
> Your best approach is to use serialization with natural keys
>

Maybe but the complexity and a couple of business rules persuade me to
write code to do it "manually" so I can unit test it. Also, if I get it
going, the user can select a substance to transfer and doesn't have to
bother me.

I also realise it is very brittle so it needs to be revised if the
structure changes.

Thanks for your advice. I appreciate it even if I think I need to tackle
the problem my way.

Cheers

Mike

> and a job
> scheduler. Cron will do, but there's plenty other schemes allowing
> you
> to provide feedback about the job queue and expected time of
> completion.
> --
> Melvyn Sopacua
>

> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-users...@googlegroups.com
> <mailto:django-users...@googlegroups.com>.
> To post to this group, send email to django...@googlegroups.com
> <mailto:django...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-users/ccfa9aef-d1c3-45e0-a894-c07a2701c1aa%40googlegroups.com
> <https://groups.google.com/d/msgid/django-users/ccfa9aef-d1c3-45e0-a894-c07a2701c1aa%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Mike Dewhirst]

To finish this thread, I have decided to abandon the attempt. My
solution is to tell the users to avoid doing real work on the staging
server. It exists only to check out new features.

I feel better now.

Thanks Melvyn and Tim

Mike

[Fred Stluka]

Mike,

On 11/11/16 8:00 AM, Mike Dewhirst wrote:
> ... My solution is to tell the users to avoid doing real work on the
> staging server.  It exists only to check out new features. ...

I have a similar issue with users of my TEST vs PROD web sites.  They
develop experimental data in TEST and then want me to move it to
PROD once they have it ready.

So far, I've been able to accommodate them by dumping part or all
of the TEST DB and importing it into the PROD DB using native tools
that come with the DBMS (MySQL in this case).

But, the best solution, I think, is to add a "Draft" field to the models,
so that they can perform their experimental data changes in PROD
flagging them as draft, and then clear the draft flag when they are
ready for other users to see the new data.

This may require an access control system that allows privileged
users to mark data as draft, and to specify which other users can
see/edit the draft data, etc.  May also need multiple different drafts,
so you get into version tracking, etc.  Not simple, but may be the
best answer to your problem.

--Fred

---

Fred Stluka -- mailto:fr...@bristle.com -- http://bristle.com/~fred/
Bristle Software, Inc -- http://bristle.com -- Glad to be of service!
Open Source: Without walls and fences, we need no Windows or Gates.

---

[Mike Dewhirst]

Fred

Spot on!Saving to the production database didn't work. It couldn't get
past the atomicity protection in Postgres. I gave up and did it manually
for the fellow who had got himself into that pickle. He loves me now.
It's embarrassing!

We also adjusted the user docs to say the staging server gets blown away
frequently - use it at your peril!

I do like your DRAFT notion.

Thanks

Mike

On 3/01/2017 11:52 AM, Fred Stluka wrote:
> Mike,
>
> On 11/11/16 8:00 AM, Mike Dewhirst wrote:
> > ... My solution is to tell the users to avoid doing real work on the
> > staging server. It exists only to check out new features. ...
>
> I have a similar issue with users of my TEST vs PROD web sites. They
> develop experimental data in TEST and then want me to move it to
> PROD once they have it ready.
>
> So far, I've been able to accommodate them by dumping part or all
> of the TEST DB and importing it into the PROD DB using native tools
> that come with the DBMS (MySQL in this case).
>
> But, the best solution, I think, is to add a "Draft" field to the models,
> so that they can perform their experimental data changes in PROD
> flagging them as draft, and then clear the draft flag when they are
> ready for other users to see the new data.
>
> This may require an access control system that allows privileged
> users to mark data as draft, and to specify which other users can
> see/edit the draft data, etc. May also need multiple different drafts,
> so you get into version tracking, etc. Not simple, but may be the
> best answer to your problem.
>
> --Fred

> ------------------------------------------------------------------------

> Fred Stluka -- mailto:fr...@bristle.com -- http://bristle.com/~fred/
> Bristle Software, Inc -- http://bristle.com -- Glad to be of service!
> Open Source: Without walls and fences, we need no Windows or Gates.

> ------------------------------------------------------------------------

> https://groups.google.com/d/msgid/django-users/f54523dc-9022-77ac-d76a-57588ddea3bc%40bristle.com
> <https://groups.google.com/d/msgid/django-users/f54523dc-9022-77ac-d76a-57588ddea3bc%40bristle.com?utm_medium=email&utm_source=footer>.