 |
 |
A progress report on where I'm at on this question, with thanks to Lalit Garg, Richard Sedcole, Stephen Senn, Brett Houlding, Brendan Murphy, Allan Reese and Bill Venables. Lalit Garg refers to some of his joint work on clustering classified survival data including 1. Garg L, McClean SI, Meenan BJ, Millard PH (2011). Phase-type survival trees and mixed distribution survival trees for clustering patients’ hospital length of stay. INFORMATICA. 22(1): 57-72. and 2. Garg L., McClean SI, Barton M, Meenan BJ and Fullerton K (2010). The extended mixture distribution survival tree based analysis for clustering and patient pathway prognostication in a stroke care unit. International Journal of Information Sciences and Application. 2(4): 671-675. I am always interested in mixture modelling approaches so I will try to look at these papers, though the type of data is a lot different from that I am thinking about. Brendan Murphy also mentions model-based clustering. I am very keen on model-based clustering but I actually think that this situation may be more amenable to a more traditional approach. Richard Sedcole suggests that Canonical Variate Analysis or Discriminant Function Analysis would be a good approach. this indeed was the approach that I had most in mind before asking anzstat and allstat. Brett Houlding sent me a pre-print about Constrained Clustering. My first thought was that this was not really what I was interested in but in fact it may be. I will trickle out a few more details about the application. I am considering the clustering of different geological layers at a number of sites based on geochemical composition, with several observations per layer-within-site. Now there are indeed some constraints: one would not wish to cluster together two classes from different layers at the same site, the interest is in which classes from different sites cluster. Stephen Senn referred me to the work of David Wishart on the clustering of single-malt whisky by flavour. Unfortunately most of this work seems to be in books that I don't have ready access too, though I have ordered his 2006 book for our Library. Allan Reese suggests Confirmatory Factor Analysis. In the clustering situation the latent variables are discrete rather than continuous so I'm not sure that CFA (let me admit, I've forgotten what that does!) would be the way to go. Bill Venables offers the following suggestions, which I will repeat for the benefit of allstat: Isn't this just a problem of clustering the clusters? In my naivete I would have thought all you need to do is a) define a distance measure between groups, (as opposed to the primary entities), that reflects the importance you put on the separations that exist already, b) define an overall objective you want the clustering to achieve (e.g. as an objective function to optimise, that balances costs and benefits) and c) employ some appropriate clustering algorithm to see if there is any advantage in merging any of your existing groups. Which one to use will depend on your circumstances. (At this stage it might sometimes be useful to look at a hierarchical clustering of your groups first, rather than to go for a hard clustering straightaway - or it may not!) My own thinking has evolved towards the following modest proposal (which has some similarity to Bill's ideas, but which, I swear, was arrived at independently. 1. Form a 'distance' matrix between the groups based on pairwise Mahalanobis distance w.r.t. the pooled variance-covariance matrix. 2. Begin the hierarchical clustering process by merging the two groups the smallest distance apart. 3. There is no need to bother about linkage methods, the changed pairwise Mahalanobis distances are recomputed. 4. Continue merging until things you know shouldn't be merged start getting merged. 5. Meditate on dendrogram. Further comments welcome. Regards, Murray Jorgensen Cheers, Murray -- Dr Murray Jorgensen http://www.stats.waikato.ac.nz/Staff/maj.html Department of Statistics, University of Waikato, Hamilton, New Zealand Email: [log in to unmask] [log in to unmask] Fax 7 838 4155 Phone +64 7 838 4773 wk Home +64 7 825 0441 Mobile 021 0200 8350 You may leave the list at any time by sending the command SIGNOFF allstat to [log in to unmask], leaving the subject line blank.