Mark,

[Long, and rather dithery in places]

The question underlying much of this discussion is that of how
expensive tags and branches are.  The cost in terms of CPU time and
disk space is pretty trivial, so all the important cost is in the
administration of them, and the psychological and productivity costs of
handling them.  If those costs are small or can be made small, then we
shouldn't worry about exploiting branches to the full.  That is, of
course a big `if'. (Is this the only issue, by the way, or are there
aesthetic or style aspects to your points?)

One way that use costs go down is by having simple branch policies and
straightforward branch and tag name conventions.

The other is just familiarity.  Having used them more, more recently,
I've found branches less of a tar pit than I expected.  In the automake
and autoconf components, I've been developing the modifications which I
hope to submit back to the maintainers on one branch, and the
Starlink-private ones on the trunk with the other branch merged onto it
periodically.  Automake and autoconf updates go on the import branch,
and are merged onto first one then the other branch.  That's more
complicated than anything I suggested on the wiki page, and I haven't
significantly confused myself yet.

On Thursday, May 6, 2004, at 12:26 PM, Mark Taylor wrote:

>> Secondly, it makes it more likely that we'd have to go back to a
>> non-current release and fix a bug that the integration had exposed,
>> without changing an interface, and therefore without simply replacing
>> the component with the latest-and-greatest one, since this may have a
>> different interface or feature-set, so that the replacement might
>> cause
>> other components to stop working.
>
> I'm not suggesting that for a full USSC release bugfixes are made
> by dropping in the bleeding-edge version, and that's not what we are
> doing for this release.

Right -- thus we're occasionally going to have to go back to a previous
(and previously tagged) version of a component and prepare a bugfix on
a bugfix branch.

>> In other words, what you're doing with this broad-brush tagging is
>> simultaneously unit testing (`Topcat has reached a temporarily
>> releasable
>> stage') and integration testing (`Topcat works with the other packages
>> in the repository').
>
> I appreciate these points, and I'm not enthusiastic about tagging in
> this way for just the reasons that you state.  However, tagging the
> java/source/topcat directory on its own is not very helpful -
> if at a later date you want to rebuild that release you need to
> have the right versions of all the libraries that it depends on

Very much so, reproducing a release is very much the goal here.

Tagging java/source/topcat in isolation is useful.  It says that here
is a set of files which is in a releasable state, with (say) the
documentation up-to-date, a given set of application bugs fixed, and a
particular set of behaviours.  That's useful, even if it doesn't make
any claims about what libraries you need to have to make it run, which
might, as you say, be a very specific set.

(And by the way I agree with you that...

> To put it another way, unit testing in the strict sense in which
> you're using it here doesn't really make sense for application
> packages,
> or in general, for packages with dependencies on (many) other packages
> which are subject to rapid change.

...  While there are some unit-testing -- that is, contract-fulfulling
-- tests one can apply to an application, it's certainly a much
blurrier boundary.)

Tying the application to the library versions it needs is a
conceptually separate operation, to do with _packaging_.  I can think
of a couple of ways to do that.

1. One way is to have a list of required library tags in a file in the
application directory.  You'd need to have a make/ant target for
constructing such a release (presumably be a zip file containing a
jar). Part of that rule or its dependencies (manifest?) would be a list
of the library component tags which are required for this release of
the application component.  That means that the required tag list is
kept with the application component and tagged along with it.  It does
mean that checking this release out again becomes a two-step process.
But how often do we expect to have to do this?

2. Alternatively, add another tag to the repository, say
topcat-x-y-z-package, then you can export that and build the interim
release package from that.  I don't think you need to tag the entire
repository -- just the components that are required for this interim
release, which are listed in the file mentioned in (1).  The problem
with this is (a) that if a problem subsequently appears with a
particular component then you might not have a well-defined point to
branch in order to make a bugfix release (we're much more likely to do
this for the USSC than for interim releases, of course), and (b) you
will end up with pages of uninformative tags on barely-related
components.

Put another way, it is _irrelevant_ to (the developer of) library
component X that Topcat deigns to play well with it, and so there
should not be _any_ such `topcat-x-y-z-likes-this-one' tags on
component X.

When we're preparing a USSC release, you tell Steve that Topcat version
x.y-z, with tag topcat-x-y-z that is, is the one you think should go in
the release, and that that needs a given other set of libraries -- this
is the information in the tag-list file in (1).  Splat will have a set
of library versions it requires, and so will Frog.  These are unlikely
to all agree, so there will be a certain amount of to and fro. But
since the conversation is about particular tagged versions of library
components, it's better contained than if it were about random tagged
snapshots of the entire tree.

This means that `The USSC' consists of a file containing a list of
tags, consisting of  the component tags which integration testing has
shown _will_ work together.  Come to think of it, this means that you
don't actually have to tag the whole repository with a USSC tag at all!
  Mind you, since there will be adjustments needed before the release
and possibly bugfixes afterwards, this should probably be on a branch.
This returns to very much the model of this USSC release, _except_ that
the initial branch point is that implied by a negotiated set of
component tags.  This preserves the orthogonality between components as
much as possible.

But no: what's supposed to happen on that branch?  Do you do
post-release bugfixes on that branch?  If so, what do you name them?
You have to name them somehow, so that you can release a named jar file
for folk to drop in to their installation as a replacement.
array-1-2-3-ussc-xxx-patch-1, perhaps (yuk)?  You can't call the fixed
version array-1-2-4 since there might be more than one USSC branch
branching off from array-1-2-3, and you don't want the obvious next
tag, array-1-2-4, appearing on more than one branch (saying `don't do
that, then' requires you to specify the bureaucracy which will tell
someone what they _are_ allowed to tag versions -- detective work on
`cvs status -v' output?).

Hmm, I think I need a diagram!

Organising this is, I think, the big problem: in this scenario what, in
detail, am I supposed to do when I discover a bug in the released USSC?

Or else you freeze the branch at release time (in which case there may
not be a great deal of point in having it).  If a bug appears in the
USSC release of the array component, which needs fixing, you know which
version of array was included in the USSC, so you make a patch/release
branch on the array component at that point, make your fixes, and tag
that array-1-2-4.  Then you can release a drop-in jar containing just
that version.  Next time there's a problem with array-1-2-*, you spot
there's a branch, so you don't make a new one, and start working on the
head of that branch.

>> If you want to make a release by snapshotting the HEAD of the trunk,
>> then you're forced to require that the HEAD of the trunk be always in
>> a releasable state -- not just buildable, but ready for the public.
>
> No, you're forced to require that it be in a candidate releasable
> state at one given point in time, at which the snapshot is taken
> and the release branch (on which integration testing is subsequently
> done) is created; this is we're doing for USSC219.

In other words a feature freeze, with all the disadvantages that
brings.  If instead you simply agree on a list of tags which are to be
in the release candidate -- which may, after discussion, turn out to be
a couple of versions back from the HEAD -- then noone has to care, and
noone has to freeze anything.  What is it that isn't simple about this?

> I can see this
> has a down-side (it's hard to do big-feature development around
> the time of the release branch creation), but my feeling is that
> the simplification of the release preparation and testing process
> that it allows makes it a better bet than the full-on approach that
> you've documented on the wiki.

[...]

> Only non-dead-cert ones near the release epoch.  If you're doing one
> of those then you can do it on a branch and merge it to the trunk
> following the release branch creation.  I reckon that would add up
> to less branches overall.

I don't agree that this is simpler.  Having to negotiate a feature
freeze, and decide whether you have time to put a particular feature in
before the freeze time, or what to do when you squeak it in and
discover it breaks something else, or whether you should put it on a
development branch, and so on, would be hard work.  This scenario is
potentially profligate with development branches, and would discourage
working on the trunk. Or worse, encourage different behaviour at
different times in the release cycle.  This is moving away from the
`latest and greatest' (LAG) pattern and towards some other pattern
which has development work kept away from the trunk.

The point of the LAG pattern is that releases come from somewhere a
little _behind_ the HEAD of the trunk.  By not requiring the HEAD to be
anything more than buildable, you avoid having to create more branches.

> An important up-side is that it
> makes it easy for anyone to grab the current version of the release
> candidate by just updating on the release branch.

Yes, so it probably _would_ be useful to have an actual ussc-xxx
branch, starting at the nominated versions.

Gasp,

Norman


--
----------------------------------------------------------------------
Norman Gray                   http://www.astro.gla.ac.uk/users/norman/
Physics and Astronomy, University of Glasgow    [log in to unmask]