yes your suspicion about the estimation of the total length from the "hight" of the hyperostotic part of the haddock cleithrum is justified. If the formula for smaller (normal size fishes) is used you would get a TL of around 1.6 meter (the largest specimen caught was 112 cm). The problem I think is that the hyperostotic growth is not allometric to other parts of the skeleton, these parts grow faster in older/larger fish. I don't know from where your archaeological specimen comes but we have the experience with Late Glacial haddock from the swedish west coast that the cleithra are extremely massive, one quaternary geologist who found such bones (eroded in beach sediments) thought they were teeth of killerwhale (Orcinus orca). With the cleithra of very large haddock one should try some other measurement. I know that it can be hard to define such measurements. Try to estimate the dorsal lenght from the anterior (distal) tip to the point where the bone bends up. I guess that that distance is more isometric in growth.
Some years ago I got a 1 m haddock in Iceland that I sent to Eufrasia and Arturo in Madrid. Try with them to get measurements and a scan of the whole bone which you could scale to the size of your "monster-haddock" and compare other measurements with.