Subversion respository structure to support dev/test/prod environments

[Chris Muir]

Hi gang

A not-so JDev ADF question, but related.

Many Oracle and enterprise sites support three environments, namely
development, test and production, for their Oracle based systems
(particularly those coming from an Oracle Forms background in my own
experience). Basically code is developed in the development
environment, promoted to the test environment for testing, and then
promoted to the production environment to be used by the end-users.

(For an example of this code-promotion structure, see the section
entitled "Code-Promotion Branches" in the following link "Branching
and Merging Primer" by Chris Birmele:
http://msdn.microsoft.com/en-us/library/aa730834.aspx)

In running such environments its useful to have your Software Control
Management (SCM) tools (you do use a SCM don't you ;-) mirror the same
structure to assist the overall workflow of code-promotion.

Has anybody any thoughts on what sort of repository structure to setup
for Subversion (SVN), potentially for an ADF project?

We've proposed the following SVN repository structure:

<root>
<root>\<app>
<root>\<app>\trunk (dev)
<root>\<app>\branches
<root>\<app>\branches\test
<root>\<app>\branches\prod
<root>\<app>\tags

Where <app> is the application name. Multiple applications can exist
in their own separate directories off <root> with similar sub-
directory structures.

For the application at hand the trunk location will be where the
initial development code resides. The test and prod branches exist to
be the single point of reference of what code is running in the test
and production environments.

The tags directory (of which initially there will be no
subdirectories) will eventually contain separate releases. This may
be releases that currently exist in the test or production branches,
or releases that are historic and kept for versioning reasons.

Is this an approach others have taken with SVN to support dev/test/
prod environments?

This becomes slightly more complicated for a JDev 11g ADF application
I'd guess? Potentially each ADF Library would become it's own <app>
tree? Anybody any thoughts on this?

Your ideas and comments appreciated.

Cheers,

CM.

[Sandra Muller]

Hi Chris,

We use a similar svn repository structure as you described, at the top
level. We also have trunk, branches, and tags. Except when we promote
a revision to the test environment, we don't create a branch, but we
create a tag with a build number: <root>\<app>\tags\build.<nn>, for
example build.32. In the pages of the application we show which build
number is running, so that when we solve an issue, we can record in
which build number it is fixed, and the tester can check in the test
environment if he is looking at the build in which the issue should be
fixed.

Our build system automatically creates the build number tags, and also
stores that build number in a properties file in the application so
that it can be shown on the screen.

What do you mean by:

> This becomes slightly more complicated for a JDev 11g ADF application
> I'd guess? Potentially each ADF Library would become it's own <app>
> tree?

Do you mean the kind of ADF Library where you create your own jar file
that contains ADF components, and which you want to reuse in other ADF
projects?

If that is the case, I'd say that the sources of the ADF Library are
indeed another <app> with its own subversion folder structure. The jar
does not get released to the test environment independently, but only
as part of a build of another application. You can manage the
dependency with maven, for example.

Hope this helps,
Sandra

[Chris Muir]

Hi Sandra

Thanks for the response.

When you say "We also have trunk, branches, and tags. Except when we

promote a revision to the test environment, we don't create a branch,

but we create a tag...", what do you use branches for then?

Our idea (probably the same as yours) is each tag (or your equivalent
build) will be branched-merged with branches\test and eventually
branches\prod as the tag/build is migrated to those environments. As
such we only have a single branches\test and branches\prod and each
represents what's currently in either environment. Is this what you
meant?

As for your question about what I meant about ADF Libraries, you
guessed correctly, essentially ADF components we want to reuse in
other ADF projects. I guess dependent on the type of ADF components
it might be worth having a separate root level sub-directory to
categorise them. For example I could imagine Declarative Components
going in their own directories under <root>\DecComponents
\<component1> .... <component2> etc.

I assume how Maven helps you, is if you have say <app1>\tag\buildX
(representing a certain version of app1) that is dependent on another
app's <app2>\tag\buildY (representing a different app2 version), Maven
allows you to define those dependencies and build the applications in
the correct order?

Thanks for your help,

CM.

[Sandra Muller]

Hi Chris,

We don't use branches yet, but intend doing so as soon as real
parallel development needs to take place. Up to now we only develop in
the trunk. This might change when for example you need to apply an
urgent fix on the production version, while you are already developing
a new release in trunk.

You are right that maven helps with defining the dependencies
(identifying exactly on which version of the jar you are dependent)
and building things in the correct order.

Hope this helps,
Sandra

[John Flack]

We are using ClearCase, not Subversion, so I'm not sure how this maps,
but:
We have a separate "application" named "common" for anything that is
shared by more than one application. This is a leftover from when I
originally established an Oracle schema named "COMMON" in which I put
tables, PL/SQL packages, and any other Oracle object that is shared.
It works well for Oracle, and so far so good for ADF. Fortunately, in
ClearCase directories are versioned, so if we create something in one
application that we later decide to move to "common", that isn't a
problem because the versions are still there from when it was in a
different folder.
We don't branch for test or production releases. Instead we create a
"label", and label each file version that belongs to that release.
ClearCase lets us create a "view" - think of it as a working copy
based on a label.
Most development is done on the trunk of the tree. We mostly branch
for one of two purposes:
Maintenance on a production release, once development has started on
the next production release. Here the intention is to eventually
merge the changes into the next production release which is being
developed on the main branch(trunk).
An experimental development project. I have one of these going right
now as I attempt to migrate a 10.1.3 application to 11. If the
experiment is successful, it will be merged back into the main branch
for continued development. Otherwise, the experimental branch will be
abandoned.

Of course, keep in mind that ours is an extremely small shop. We are
very informal; some would say haphazard.

[Chris Muir]

Thanks for the follow up John. Sounds like ClearCase "labels" are the
same as SVN's "tags".

In talking to a colleague about the directory structure I'd proposed
for SVN... ie:

<root>
<root>\<app>
<root>\<app>\trunk (dev)
<root>\<app>\branches
<root>\<app>\branches\test
<root>\<app>\branches\prod
<root>\<app>\tags

He'd suggested that the tags directory should further be divided into
categories for releases, developer backups and anything else. ie:

<root>\<app>\tags\backup\TomBackup01
<root>\<app>\tags\experimental\ThisMayNotWork07
<root>\<app>\tags\releases\REL1 (or CR1 if a release is a change
request within your org)
etc

He also suggested that the standard SVN structure doesn't need to be
followed at all. Thus something like the following would be fine:

<app>\backup
<app>\development
<app>\releases

Possibly this is the case of the standard SVN examples leading us up
the garden trail; if we sit back and realise SVN allows us to create
any directory structure we have full flexibility to create structures
that mirror our change control work processes.

I'll see if I can get him on the forum to give us a better
explanation.

CM.

[Aino]

Hi,

Indeed, the trunk/tags/branches directories are not required. But it seems that they are a de-facto standard and some tools rely on (expect) them, e.g. the Maven release plugin (although it allows for a different configuration, if needed).

One mistake we once made in a project was to have each JDeveloper project, model (1 times) and viewController, have its own trunk/tags/branches. Fortunately, svn allows for easy re-organization :-).

Actually, I don't really have an opinion about organizing the tags and branches directory. In most projects it didn't really apply or incidentially. The only thing that matters is a clear name of the label/tag, which is handled (suggested) by the Maven release plugin that we use.

btw, I often get inspired how things are organized at Apache. 

Ciao

  Aino

[Duncan Mills]

Sorry I meant to pitch in earlier in this, now I have the time though.

Firstly on the subject of overall repository structure I think that it's best to stick to the standard trunk/branches/tags layout that the SVN doc outlines. There is little reason not to use this layout and it make it a hell of a lot easier to look at a repository and work out what's what.

On the matter of tags and branches, just from our experience of using SVN to manage the Product Demos. We tag a lot, in fact every successful Cruise Control build will create a tag using the build number. Then we are in a position to restore the stripe of files before or after a particular checkin. Tags have a very low overhead so certainly if you are using any kind of build automation either a daily or per build tag is a good idea.
Branching is used to peel off for a particular release only, we don't need to do private branches given the size of the PM team, everyone works on the main line. (This infact reflects the way we build JDeveloper as a whole)

I've actually written a whole chunk about this subject in the new Fusion book because I sense that this is an area that needs guidance (or at least reassurance)
Merry Christmas All!

Duncan

[Chris Muir]

I think Duncan's definitely on the right track here and I'd like to
express why. What follows is a rant. Please put rant-safety glasses
on.

This is also somewhat of a follow up to Susan Duncan's post on the
following thread; somehow the project-best-practices thread and SVN
threads were merged, this is an attempt to split them again:
http://groups.google.com/group/adf-methodology/msg/3bba8b83b01d068c?hl=en

To be honest from my own experience it's not working with SVN via JDev
at issue though I appreciate SVN's different locking mechanisms throw
some programmers out (including me), particularly those coming from a
lock-modify-unlock VCS such as PVCS used at many Oracle Forms sites.
Rather at issue with JDev is a suitable change control *process*
supported by SVN that allows me to take a new *enterprise* project
through the very common dev, test, prod environments, then put in a
longer term process of patches & releases through again the dev, test
and prod environments.

In other words I need to know how we get the ball rolling on the new
change control process for the enterprise, not just how to check stuff
in, out and merge via SVN. To be successful with a JDev project I
need guidelines on what to tell junior programmers about the
enterprise version control process besides read the SVN manual, and I
need guidelines to inform IT management how their processes need to
change, because it has a business impact.

While it may sound harsh to dump this on the JDeveloper camp, the
problem is most organisations already have some sort of change control
process on board. Change is hard. Where change is hard, it stops
JDeveloper being accepted, because JDeveloper requires change to such
processes. I see architects having trouble convincing their
organisations to adopt JDev because of organisational resistance to
change existing change control processes (ha!) that an enterprise IT
section is based around. Those processes have normally been put in
place by seasoned (read: old and bitter? ;-) programmers with years of
experience, little reason to change, and the power-base behind them to
stop change in this case.

Therefore to accomodate the change that JDev introduces in a
structured way into an enterprise, we first need to work out a change
control process that is suitable for JDeveloper (seems to be complete
releases due to the nature of a single packed WAR/EAR, as separate to
Oracle Forms where at minimum you can squeeze through a single Form
update) and matches the current technical environment (ie. dev, test,
prod), and armed with that we can then show an organisation how to
adapt their existing processes.

This is why working out the process for moving between the common
enterprise site environment of dev then test then prod (to me at
least) is so important. And (yes another unreasonable demand on
Oracle and how it should improve the world) I think Oracle needs to
give more focus on the change control processes side, rather than just
the SVN integration into JDev (I know, I know, Duncan, Susan, Sandra
et all are doing this, but I'm ranting, so leave me be).

So in summary the above is an attempt to make it clear the need for
more discussion on the change control processes based around JDev &
SVN, rather than SVN's integration into JDev itself.

Oh yeah, that's why we setup the ADF Methodology Group isn't it ;-)

Off my box, love to hear other opinions. I'm wrong 8 times out of 10
by the way (though I did predict the sub-prime crises 3 years ago).

Cheers,

CM.

[Aino]

Hi,

I agree with Duncan about the standard svn directory structure... on application level. We once did it on project level, the model and viewController had their own trunk, branches and tags directories. That caused lots of problems :-(. Fortunately, svn allows easy reorganization.

We have all our project maven enabled. All releases are (automatically) tagged and the deliverable (artifact in maven terminology) is stored in a repository. All releases can thus be identified. This gives us a better insight in the delivery process.

For incident management, we use Jira that allows incidents to be 'connected' with a release and release notes to be created. Again, more informormation about the delivery and release process. Note, that it relies on the developer to do his work incident driven.

However, all our work is done in the main branch (the svn trunk directory). Once an incident is fixed or a feature is added, it is part of the next release of the application. It is not possible to to test these units of work individually before they are added to the main branch (or not at all). In such a process, work is done in separate branches and delivered as patches that are merged into the main branch, typically by a release manager and not by all the developers.

Subversion has a sofisticated patch and merge mechanism to support such a process. I have no experience with such a delivery process and have some doubts if it would work on ADF projects because of the dependencies between objects, within a project, between projects (model and viewController) and with the database. In addition, patches can also be dependent upon each other. In addition, JHeadstart generation doesn't make things easier.

Maybe, itt could be done by oragnizing the project into small chunks, that are quite independent from each other, share a minimal amount of files. 

It is interesting to know if anybody has experience with this way of working on ADF (JHeadstart) projects.

Ciao

  Aino

[John Stegeman]

Hi gang,

Sorry that it's taken me so long to jump in here - been pretty busy
lately.

I'll echo the sentiments previously expressed here that the "standard"
structure is probably the best (trunk/branches/tags). I'll provide a
little information about how we use that structure as well as a few
comments.

First, in the 10g days, the capability of the Subversion plugin just
wasn't up to what was needed for enterprise-level development. For the
most part, I used the svn command-line tools; I did have the plugin
installed for JDev, but the only thing I did with it was to look at
the state overlay icons and the revision numbers in the navigator to
see what was going on. All of the checkouts/commits/merges/etc were
done from the command line (other team members preferred TortoiseSVN -
but I'm a command-line jockey at heart).

As far as tagging - I tag:

* upon successful build (done automatically by the CI tool) - tag
names are automatically constructed and include the revision number
* upon release of a version of the application into the production
environment - tag names use a naming convention including the date the
software was released for production use

For branches:

* We sometimes use branches when doing side or prototyping
development. This was a pretty rare use case for us, though.
* When we go through a major enhancement cycle for the application,
we tend to develop the enhancements on the trunk, and create a branch
for production fixes (should we need to fix anything in production
before the enhancements are ready). We create the branch by copying
the release tag. We do it this way because (in hope as well as theory)
the volume of code change for production fixes should be much less
than the volume of code fixes for enhancements. This means it is
easier (fewer changes) to merge the production fixes back into the
trunk than it is to develop enhancements on a branch and then merge
them back into the trunk.

As far as incident management, we use a hosted environment that
includes SVN and Trac - we do cross-reference the Trac tickets with
SVN revisions by using ticket numbers in commit messages and using
revision numbers in ticket logs.

Getting off-topic with the rest of this post:

I am intrigued more and more by the Maven process, but am still
holding back on using it because of:

* Lack of support in JDev 11.
* Laziness, or more accurately, lack of time.
* Lack of a public repository, or at least a publically-agreed
taxonomy for, the ADF libraries. I know there is an ER out there for
this, but if we simply had an "Oracle sanctioned" structure for the
ADF libraries (hint hint hint hint), we could create our own Maven
repository containing the Oracle libraries and then consume the public
repository if and when it comes available.

A question for Aino - did you develop your own archetypes for Fusion
apps (I guess this would be equivalent to an application template in
JDev)? What would be really cool would be one or more of the
following:

* A way to create Maven archetypes from JDeveloper application
templates.
* First-class Maven support in JDeveloper (much like the ability to
use ANT as the default build process).
* A JDeveloper extension to create a pom.xml file from a JDeveloper
application/project.

Best,

John

[Liran Zelkha]

I agree.
A much better Maven integration is needed. Even the ability to create
JDeveloper projects from Maven itself (like eclipse:eclipse)


Thank You,
Best Regards,

Liran Zelkha
Founder & Co-CEO
Aluna
Phone: 972-3-7382555
Mobile: 972-54-3010714
E-Mail: liran....@alunasoft.com
http://www.linkedin.com/in/liranz

[Sandra Muller]

I started a wiki page http://wiki.oracle.com/page/Maven+and+JDeveloper+ADF+projects
in which I added my tips for how to setup JDeveloper projects for use
with Maven, and how to run Maven commands from within JDeveloper. I
hope that this will be the starting point for more best practices with
Maven and JDeveloper.

kind regards,
Sandra

[shay shmeltzer]

There is a maven plug-in that will generate a JDeveloper project file for a maven structured project.
See info here:
http://www.oracle.com/technology/products/jdev/howtos/1013/maven/maven.html
http://kingsfleet.blogspot.com/2008/12/jdeveloper-and-maven-2x.html

[John Stegeman]

Hi Shay,

Yes, I've seen that from the wiki page. At the risk of getting off-
topic...

1). The plugin works fine if you already have a project structured to
use Maven. It would be nice if JDev had an extension/capability to
create a new project from an application template but would build the
directory structure as appropriate for Maven and create the pom.xml
files. In other words, we need archetypes for the JDeveloper
application templates.

2). The plugin hard-codes the path to the libraries in the local Maven
repository - I guess this is OK, as in theory, each user would
generate their own jws/jpr using the plugin.

3). We still need an "approved" taxonomy for the Oracle-supplied
libraries. Of course, we can each create our own, but then if/when
Oracle publishes a standard taxonomy, we will all have to alter our
repositories.

J

[Aino]

Hi,

I am a bit confused. Maven and JDev certainly certifies for a separate thread.

What about comments, discussion and improvements on the wiki page,do we haver them here or on the wiki page? 

Ciao

  Aino

[John Stegeman]

Yes, i apologize for getting this thread off-track.

I would think that the wiki page is the best place for a discussion on
the wiki page

John