A new approach to deployments

149 views
Skip to first unread message

Aaron Powell

unread,
Mar 24, 2014, 5:33:49 AM3/24/14
to umbra...@googlegroups.com

Been a while since I’ve been on here!

 

I’ve always found deployments of changes across multiple Umbraco instances a bit unpleasant, the primary problem is that things are stored in the database, in particular your Content Types. I know there’s things like uSync, I’ve even written one myself, but the problem I’ve always found is that they require the website to actually be running, so it’s generally done on app startup. This means whoever does the first request cops a hit from that process being run.

 

Recently I’ve been playing with Orchard and have really liked the way they handle it, they have a CLI which you run commands through. So like any developer seeing something interesting I’ve started implementing it myself!

So here’s Chauffeur, your Umbraco delivery service - https://github.com/aaronpowell/Chauffeur

 

Chauffeur uses the Umbraco service layer under the hood so if nothing else it’s an interesting exercise in what can be done with the API, although I’ve found some pointy edges when dealing with some of the internal types such as:

·         https://github.com/aaronpowell/Chauffeur/blob/master/Chauffeur/ApiWorkarounds/MappingResolver.cs

o   MappingResolver is all internal and needs to be setup if you want to use ContentTypeService.GetContentType(string alias)

·         https://github.com/aaronpowell/Chauffeur/blob/master/Chauffeur/Deliverables/ContentTypeDeliverable.cs#L191-L211

o   This is the only way I can find to ensure the PropertyGroupId is setup as if it’s not set then the PackagingService fails

 

Chauffeur is all extensible, create a new class that implements the Deliverable class, override the Run method and you’re good to go.

 

So, do people think that it could be a good idea? Got any ideas of what it should be able to do? Let me know!

 

 

Aaron Powell

unread,
Mar 24, 2014, 5:37:33 AM3/24/14
to umbra...@googlegroups.com

--
You received this message because you are subscribed to the Google Groups "Umbraco development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to umbraco-dev...@googlegroups.com.
To post to this group, send email to umbra...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/umbraco-dev/BLU402-EAS1953AC9DA76E59E3507076DEE7A0%40phx.gbl.
For more options, visit https://groups.google.com/d/optout.

Lee Kelleher

unread,
Mar 24, 2014, 6:35:34 AM3/24/14
to umbra...@googlegroups.com
Hey Aaron,

Chauffeur looks great!  I'll give it a go.

As for ideas, there's an old thread about an "umbraco.exe" here: https://groups.google.com/d/msg/umbraco-dev/IL1f_gkKWAM/b7XsknFl9xwJ

Cheers,
- Lee

Aaron Powell

unread,
Mar 24, 2014, 7:17:52 AM3/24/14
to umbra...@googlegroups.com

Thanks for the link Lee, I thought I remembered seeing something “CLI-ish” was demoed a while back, guess it never made it passed the PoC stage.

 

My goal with Chauffeur is that it should be extensible, so anything I don’t implement doesn’t mean it can’t be done, so much of those points ­can be done, but whether they are in the box depends on whether I can extend the numbers of hours in the day (and what is possible in the Umbraco API).

 

Initially I’m focused on deployment though, being able to script an environment creation so you could use PowerShell to setup an IIS website, configure permissions and then install the “structure” to an Umbraco instance.

I’d like to tackle content but full content replication across multiple environments is really bloody hard (which is why Courier has so many pointy edges).

 

But I was talking to someone today about it and a crazy idea that you could host an OWIN/Katana server inside a Chauffeur Deliverable, which then loads the Umbraco API through REST endpoints, so you could reserve Umbraco out of your own self-hosted API :P

--

You received this message because you are subscribed to the Google Groups "Umbraco development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to umbraco-dev...@googlegroups.com.
To post to this group, send email to umbra...@googlegroups.com.

Morten Christensen

unread,
Mar 24, 2014, 7:38:11 AM3/24/14
to umbra...@googlegroups.com
The last workaround you mention for the "PropertyGroupId" was that only a problem only in 7.0.0 as you wrote in the comment or is it still a problem in latest version of v7 ?
I had a quick peek at the code base and at a glance it looks like its loaded okay. But if its still an issue we should get it fixed as that would be a problem for packages (export).

- Morten


--
You received this message because you are subscribed to the Google Groups "Umbraco development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to umbraco-dev...@googlegroups.com.
To post to this group, send email to umbra...@googlegroups.com.

Morten Bock Sørensen

unread,
Mar 24, 2014, 10:32:09 AM3/24/14
to umbra...@googlegroups.com
Hi Aaron (and welcome back)

I've been thinking about having a console tool of some sort to run "migrations" against umbraco for a while. Currently we are using uSync, but I really like the idea of just running this outside of the web process.

Have you thought about what the workflow would be with this tool? Is it supposed to be for continuos delivery, or is it more for a one-shot setup of the site to run CI tests against? 

Say I have a deployment coming with updates to a couple of document types, what would be the steps to deploy this? Would it be somthing like 
1) Run export on all items ind dev environment
2) copy exported files to target environment
3) Have a powershell script loop the files and import them via the CLI?

The above would be very similar to uSync, just using the CLI to do the actual execution of the export/import? If this is the case, would it make sense to make this a "uSync task runner", since that already supports datatypes and that sort of thing? Or is that too mixed up in the HttpContext for it to be realistic?

/Bock

Aaron Powell

unread,
Mar 24, 2014, 4:14:36 PM3/24/14
to umbra...@googlegroups.com
So far I haven’t tested beyond 7.0.4 (7.0.0 is my min version) so I’d assume that it goes backwards at least that far. I think that 7.1.0 doesn’t have PropertyGroupId as a public member though. 

Sent from Windows Mail

From: Morten Christensen
Sent: ‎Monday‎, ‎24‎ ‎March‎ ‎2014 ‎10‎:‎38‎ ‎PM
To: umbra...@googlegroups.com

The last workaround you mention for the "PropertyGroupId" was that only a problem only in 7.0.0 as you wrote in the comment or is it still a problem in latest version of v7 ?
I had a quick peek at the code base and at a glance it looks like its loaded okay. But if its still an issue we should get it fixed as that would be a problem for packages (export).

- Morten
On Mon, Mar 24, 2014 at 10:33 AM, Aaron Powell <m...@aaron-powell.com> wrote:

Been a while since I’ve been on here!

 

I’ve always found deployments of changes across multiple Umbraco instances a bit unpleasant, the primary problem is that things are stored in the database, in particular your Content Types. I know there’s things like uSync, I’ve even written one myself, but the problem I’ve always found is that they require the website to actually be running, so it’s generally done on app startup. This means whoever does the first request cops a hit from that process being run.

 

Recently I’ve been playing with Orchard and have really liked the way they handle it, they have a CLI which you run commands through. So like any developer seeing something interesting I’ve started implementing it myself!

So here’s Chauffeur, your Umbraco delivery service - https://github.com/aaronpowell/Chauffeur

 

Chauffeur uses the Umbraco service layer under the hood so if nothing else it’s an interesting exercise in what can be done with the API, although I’ve found some pointy edges when dealing with some of the internal types such as:

·         https://github.com/aaronpowell/Chauffeur/blob/master/Chauffeur/ApiWorkarounds/MappingResolver.cs

o   MappingResolver is all internal and needs to be setup if you want to use ContentTypeService.GetContentType(string alias)

·         https://github.com/aaronpowell/Chauffeur/blob/master/Chauffeur/Deliverables/ContentTypeDeliverable.cs#L191-L211

o   This is the only way I can find to ensure the PropertyGroupId is setup as if it’s not set then the PackagingService fails

 

Chauffeur is all extensible, create a new class that implements the Deliverable class, override the Run method and you’re good to go.

 

So, do people think that it could be a good idea? Got any ideas of what it should be able to do? Let me know!

 

 

--
You received this message because you are subscribed to the Google Groups "Umbraco development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to umbraco-dev...@googlegroups.com.
To post to this group, send email to umbra...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/umbraco-dev/BLU402-EAS1953AC9DA76E59E3507076DEE7A0%40phx.gbl.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Umbraco development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to umbraco-dev...@googlegroups.com.
To post to this group, send email to umbra...@googlegroups.com.

Aaron Powell

unread,
Mar 24, 2014, 4:16:48 PM3/24/14
to umbra...@googlegroups.com
Morten,

> Have you thought about what the workflow would be with this tool? Is it supposed to be for continuos delivery, or is it more for a one-shot setup of the site to run CI tests against? 

Both essentially, here’s two scenarios:
  • You’re a new developer joining a team that’s using Umbraco. You clone the repo which the code is within. How do you get all the content types setup? Either the team is using a shared database which you’ve got to hope is in a good state or you have to get a database backup from another team member
  • You’ve got a UAT environment, there are some new content types (or other structural elements) that you need to add, others that need updating, and you want to push those changes up with minimal effort

These two scenarios can both be addressed with Chauffeur and a PowerShell/Bat file, reading exported items that you are storing in your SCM.

At present I’ve only implemented a export/import single content-type, but I think you could do it in a much better, more bulk fashion, see https://github.com/aaronpowell/Chauffeur/issues/2
Whether or not Chauffeur is intelligent enough to know which scripts have/haven’t been run on an environment (ala DbUp - http://dbup.github.io/) I’m not decided yet. I’m thinking that DeliveryPackages would be agnostic of whether they’ve been run before and you could make another command to track them.

And there’s no reason that Chauffeur couldn’t leverage the uSync exported files and run them, I’d have to look into it (or someone who knows uSync better could always make it 😉).


Sent from Windows Mail

From: Morten Bock Sørensen
Sent: ‎Tuesday‎, ‎25‎ ‎March‎ ‎2014 ‎1‎:‎32‎ ‎AM
To: umbra...@googlegroups.com

Hi Aaron (and welcome back)

I've been thinking about having a console tool of some sort to run "migrations" against umbraco for a while. Currently we are using uSync, but I really like the idea of just running this outside of the web process.

Have you thought about what the workflow would be with this tool? Is it supposed to be for continuos delivery, or is it more for a one-shot setup of the site to run CI tests against? 

Say I have a deployment coming with updates to a couple of document types, what would be the steps to deploy this? Would it be somthing like 
1) Run export on all items ind dev environment
2) copy exported files to target environment
3) Have a powershell script loop the files and import them via the CLI?

The above would be very similar to uSync, just using the CLI to do the actual execution of the export/import? If this is the case, would it make sense to make this a "uSync task runner", since that already supports datatypes and that sort of thing? Or is that too mixed up in the HttpContext for it to be realistic?

/Bock


On Monday, March 24, 2014 10:33:49 AM UTC+1, Aaron Powell wrote:

Been a while since I’ve been on here!

 

I’ve always found deployments of changes across multiple Umbraco instances a bit unpleasant, the primary problem is that things are stored in the database, in particular your Content Types. I know there’s things like uSync, I’ve even written one myself, but the problem I’ve always found is that they require the website to actually be running, so it’s generally done on app startup. This means whoever does the first request cops a hit from that process being run.

 

Recently I’ve been playing with Orchard and have really liked the way they handle it, they have a CLI which you run commands through. So like any developer seeing something interesting I’ve started implementing it myself!

So here’s Chauffeur, your Umbraco delivery service - https://github.com/aaronpowell/Chauffeur

 

Chauffeur uses the Umbraco service layer under the hood so if nothing else it’s an interesting exercise in what can be done with the API, although I’ve found some pointy edges when dealing with some of the internal types such as:

·         https://github.com/aaronpowell/Chauffeur/blob/master/Chauffeur/ApiWorkarounds/MappingResolver.cs

o   MappingResolver is all internal and needs to be setup if you want to use ContentTypeService.GetContentType(string alias)

·         https://github.com/aaronpowell/Chauffeur/blob/master/Chauffeur/Deliverables/ContentTypeDeliverable.cs#L191-L211

o   This is the only way I can find to ensure the PropertyGroupId is setup as if it’s not set then the PackagingService fails

 

Chauffeur is all extensible, create a new class that implements the Deliverable class, override the Run method and you’re good to go.

 

So, do people think that it could be a good idea? Got any ideas of what it should be able to do? Let me know!

 

 

--
You received this message because you are subscribed to the Google Groups "Umbraco development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to umbraco-dev...@googlegroups.com.
To post to this group, send email to umbra...@googlegroups.com.

Morten Christensen

unread,
Mar 24, 2014, 4:29:26 PM3/24/14
to umbra...@googlegroups.com

Okay, and its when you get a ContentType through the ContentTypeService that its not initialized with the PropertyGroupId and thus explodes in the PackagingService export, right?

Let me know and I'll try and get that fixed for 7.1.0.

Thanks,

Morten Christensen

Morten Christensen

unread,
Mar 24, 2014, 4:33:04 PM3/24/14
to umbra...@googlegroups.com

FWIW the uSync files are just xml from the PackagingService's export, so essentially the same thing right?

- Morten

Aaron Powell

unread,
Mar 24, 2014, 4:34:02 PM3/24/14
to umbra...@googlegroups.com
Yep that’s right, the value isn’t set, check out ContentTypeRepository.GetPropertyTypeCollection. It doesn’t appear set in 7.1.0 either, but looks like an oversight.
That said I’ll have to keep my hack in there to support earlier 7.0.x implementations anyway.

Sent from Windows Mail

From: Morten Christensen
Sent: ‎Tuesday‎, ‎25‎ ‎March‎ ‎2014 ‎7‎:‎29‎ ‎AM
To: umbra...@googlegroups.com

Okay, and its when you get a ContentType through the ContentTypeService that its not initialized with the PropertyGroupId and thus explodes in the PackagingService export, right?

Let me know and I'll try and get that fixed for 7.1.0.

Thanks,

Morten Christensen

Den 24/03/2014 21.16 skrev "Aaron Powell" <m...@aaron-powell.com>:
So far I haven’t tested beyond 7.0.4 (7.0.0 is my min version) so I’d assume that it goes backwards at least that far. I think that 7.1.0 doesn’t have PropertyGroupId as a public member though. 

Sent from Windows Mail

From: Morten Christensen
Sent: Monday, 24 March 2014 10:38 PM
To: umbra...@googlegroups.com

The last workaround you mention for the "PropertyGroupId" was that only a problem only in 7.0.0 as you wrote in the comment or is it still a problem in latest version of v7 ?
I had a quick peek at the code base and at a glance it looks like its loaded okay. But if its still an issue we should get it fixed as that would be a problem for packages (export).

- Morten
On Mon, Mar 24, 2014 at 10:33 AM, Aaron Powell <m...@aaron-powell.com> wrote:

Been a while since I’ve been on here!

 

I’ve always found deployments of changes across multiple Umbraco instances a bit unpleasant, the primary problem is that things are stored in the database, in particular your Content Types. I know there’s things like uSync, I’ve even written one myself, but the problem I’ve always found is that they require the website to actually be running, so it’s generally done on app startup. This means whoever does the first request cops a hit from that process being run.

 

Recently I’ve been playing with Orchard and have really liked the way they handle it, they have a CLI which you run commands through. So like any developer seeing something interesting I’ve started implementing it myself!

So here’s Chauffeur, your Umbraco delivery service - https://github.com/aaronpowell/Chauffeur

 

Chauffeur uses the Umbraco service layer under the hood so if nothing else it’s an interesting exercise in what can be done with the API, although I’ve found some pointy edges when dealing with some of the internal types such as:

·         https://github.com/aaronpowell/Chauffeur/blob/master/Chauffeur/ApiWorkarounds/MappingResolver.cs

o   MappingResolver is all internal and needs to be setup if you want to use ContentTypeService.GetContentType(string alias)

·         https://github.com/aaronpowell/Chauffeur/blob/master/Chauffeur/Deliverables/ContentTypeDeliverable.cs#L191-L211

o   This is the only way I can find to ensure the PropertyGroupId is setup as if it’s not set then the PackagingService fails

 

Chauffeur is all extensible, create a new class that implements the Deliverable class, override the Run method and you’re good to go.

 

So, do people think that it could be a good idea? Got any ideas of what it should be able to do? Let me know!

 

 

--
You received this message because you are subscribed to the Google Groups "Umbraco development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to umbraco-dev...@googlegroups.com.
To post to this group, send email to umbra...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Umbraco development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to umbraco-dev...@googlegroups.com.
To post to this group, send email to umbra...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Umbraco development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to umbraco-dev...@googlegroups.com.
To post to this group, send email to umbra...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Umbraco development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to umbraco-dev...@googlegroups.com.
To post to this group, send email to umbra...@googlegroups.com.

Aaron Powell

unread,
Mar 24, 2014, 4:38:31 PM3/24/14
to umbra...@googlegroups.com
Sweet, then it’s just a matter of making Deliverables that can take those types, which comes down to having enough time in the day or Pull Requests *cough cough*

Sent from Windows Mail

From: Morten Christensen
Sent: ‎Tuesday‎, ‎25‎ ‎March‎ ‎2014 ‎7‎:‎33‎ ‎AM
To: umbra...@googlegroups.com

FWIW the uSync files are just xml from the PackagingService's export, so essentially the same thing right?

- Morten

Den 24/03/2014 21.28 skrev "Aaron Powell" <m...@aaron-powell.com>:
Morten,

> Have you thought about what the workflow would be with this tool? Is it supposed to be for continuos delivery, or is it more for a one-shot setup of the site to run CI tests against? 

Both essentially, here’s two scenarios:
  • You’re a new developer joining a team that’s using Umbraco. You clone the repo which the code is within. How do you get all the content types setup? Either the team is using a shared database which you’ve got to hope is in a good state or you have to get a database backup from another team member
  • You’ve got a UAT environment, there are some new content types (or other structural elements) that you need to add, others that need updating, and you want to push those changes up with minimal effort

These two scenarios can both be addressed with Chauffeur and a PowerShell/Bat file, reading exported items that you are storing in your SCM.

At present I’ve only implemented a export/import single content-type, but I think you could do it in a much better, more bulk fashion, see https://github.com/aaronpowell/Chauffeur/issues/2
Whether or not Chauffeur is intelligent enough to know which scripts have/haven’t been run on an environment (ala DbUp - http://dbup.github.io/) I’m not decided yet. I’m thinking that DeliveryPackages would be agnostic of whether they’ve been run before and you could make another command to track them.

And there’s no reason that Chauffeur couldn’t leverage the uSync exported files and run them, I’d have to look into it (or someone who knows uSync better could always make it 😉).


Sent from Windows Mail

From: Morten Bock Sørensen
Sent: ‎Tuesday‎, ‎25‎ ‎March‎ ‎2014 ‎1‎:‎32‎ ‎AM
To: umbra...@googlegroups.com

Hi Aaron (and welcome back)

I've been thinking about having a console tool of some sort to run "migrations" against umbraco for a while. Currently we are using uSync, but I really like the idea of just running this outside of the web process.

Have you thought about what the workflow would be with this tool? Is it supposed to be for continuos delivery, or is it more for a one-shot setup of the site to run CI tests against? 

Say I have a deployment coming with updates to a couple of document types, what would be the steps to deploy this? Would it be somthing like 
1) Run export on all items ind dev environment
2) copy exported files to target environment
3) Have a powershell script loop the files and import them via the CLI?

The above would be very similar to uSync, just using the CLI to do the actual execution of the export/import? If this is the case, would it make sense to make this a "uSync task runner", since that already supports datatypes and that sort of thing? Or is that too mixed up in the HttpContext for it to be realistic?

/Bock


On Monday, March 24, 2014 10:33:49 AM UTC+1, Aaron Powell wrote:

Been a while since I’ve been on here!

 

I’ve always found deployments of changes across multiple Umbraco instances a bit unpleasant, the primary problem is that things are stored in the database, in particular your Content Types. I know there’s things like uSync, I’ve even written one myself, but the problem I’ve always found is that they require the website to actually be running, so it’s generally done on app startup. This means whoever does the first request cops a hit from that process being run.

 

Recently I’ve been playing with Orchard and have really liked the way they handle it, they have a CLI which you run commands through. So like any developer seeing something interesting I’ve started implementing it myself!

So here’s Chauffeur, your Umbraco delivery service - https://github.com/aaronpowell/Chauffeur

 

Chauffeur uses the Umbraco service layer under the hood so if nothing else it’s an interesting exercise in what can be done with the API, although I’ve found some pointy edges when dealing with some of the internal types such as:

·         https://github.com/aaronpowell/Chauffeur/blob/master/Chauffeur/ApiWorkarounds/MappingResolver.cs

o   MappingResolver is all internal and needs to be setup if you want to use ContentTypeService.GetContentType(string alias)

·         https://github.com/aaronpowell/Chauffeur/blob/master/Chauffeur/Deliverables/ContentTypeDeliverable.cs#L191-L211

o   This is the only way I can find to ensure the PropertyGroupId is setup as if it’s not set then the PackagingService fails

 

Chauffeur is all extensible, create a new class that implements the Deliverable class, override the Run method and you’re good to go.

 

So, do people think that it could be a good idea? Got any ideas of what it should be able to do? Let me know!

 

 

--
You received this message because you are subscribed to the Google Groups "Umbraco development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to umbraco-dev...@googlegroups.com.
To post to this group, send email to umbra...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Umbraco development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to umbraco-dev...@googlegroups.com.
To post to this group, send email to umbra...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Umbraco development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to umbraco-dev...@googlegroups.com.
To post to this group, send email to umbra...@googlegroups.com.

Morten Christensen

unread,
Mar 24, 2014, 4:55:16 PM3/24/14
to umbra...@googlegroups.com

I'll double check tomorrow, but as far as I remember there are two types of collections: PropertyGroups with PropertyTypes that belong in those groups and then there are PropertyTypes, which contains the PropertyTypes that don't belong to any group (listed under Generic Properties in the UI).

- Morten

Aaron Powell

unread,
Mar 24, 2014, 5:29:22 PM3/24/14
to umbra...@googlegroups.com
That may be the intention and prior to 7.1.0 it wasn't how it was used. Looking at the source for 7.0.4 (which is the stable at time of writing) you'll see https://github.com/umbraco/Umbraco-CMS/blob/7.0.4/src/Umbraco.Core/Services/PackagingService.cs#L376
 
The problem I kept hitting was that PropertyGroupId was null on all PropertyType objects, which resulted in a null ref within the FirstOrDefault.
 
Now if you compare that to 7.1.0 - https://github.com/umbraco/Umbraco-CMS/blob/7.1.0/src/Umbraco.Core/Services/PackagingService.cs#L379 you'll see the null check first.
I haven't checked to see if the value is initialized for PropertyType's that have a group but at least the null ref is guarded against.
 

Date: Mon, 24 Mar 2014 21:55:16 +0100

Subject: Re: A new approach to deployments
Reply all
Reply to author
Forward
0 new messages