 |
 |
I certainly agree that there are times when you would want to *include* stopwords in your search/indexing/analysis, but I can think of one use-case for omitting them. When we compiled the indices for IAph2007, we originally planned to exclude "common words" (and have them in a secondary index for those interested... after all in a digital publication generating an index is not a human task, so there's no reason not to do something that you're doing anyway). In the end we ran out of time before we implemented this secondary index, and so never got around to drawing up a list of such common words. See for example: <http://insaph.kcl.ac.uk/iaph2007/inscriptions/indices/word/word-grcK.html> where the entry for καί (kai) is pretty ridiculous, or even better <http://insaph.kcl.ac.uk/iaph2007/inscriptions/indices/word/word-grcO.html> where three-quarters of the document is taken up by the lemma ὁ (ho), whose inflected forms appear on almost every line of almost every inscription. You might argue no harm done, but when we produce a printed index of these texts, we'll certainly want to omit stopwords. (I vote +1 to creating a list of such words in the DC wiki. On what grounds would such words be selected? By frequency? On grammatical grounds? What have existing projects done? Could someone approach the Dartmouth Dante team and ask for their list? What *useful* format would such a list be in? (Text, XML, SQL table, Python list...) G Hugh Cayless a écrit : > I don't know of one, and I wonder whether anyone's ever seen a need > for one. Stopwords can help as a sort of performance optimization in > search engines with a restricted set of use cases, but once you get > beyond a certain domain limit, they just aren't useful (you can search > for 'a' on Google for an example of what I mean). Philologists are > often very interested in words that might get dropped by a stopword > list. I might want to find particular uses of 'et' for example, and > be very irritated if the results told me I couldn't. > > I've implemented search engines a few times now and honestly never had > a use for stopwords in the end for any of them. I sort of don't > believe in them anymore...so my question would be: what's the use > case, and do you really need one? > > Hope this helps, > Hugh > > On Aug 24, 2008, at 6:28 PM, Neven Jovanović wrote: > >> Hello, >> >> does anybody know where could one look for a list of stop words for >> Latin? >> I have seen an English stop words list on Perseus >> (http://www.perseus.tufts.edu/Texts/engstop.html), but have not been >> able >> to find anything similar for Latin. Yes, the Dartmouth Dante >> (http://dante.dartmouth.edu/help.php) mentions "stopword list" for >> Latin, >> but does not make it available. >> >> It seems that such a list is something that always gets compiled from >> scratch. Perhaps a version of it, made freely available, could be a >> welcome contribution to the Digital Classicist wiki. >> >> (Not to mention a Greek stop word list...) >> >> Yours, >> >> Neven Jovanovic >> >> Zagreb, Hrvatska / Croatia -- Dr Gabriel BODARD (Epigrapher & Digital Classicist) Centre for Computing in the Humanities King's College London 26-29 Drury Lane London WC2B 5RL Email: [log in to unmask] Tel: +44 (0)20 7848 1388 Fax: +44 (0)20 7848 2980 http://www.digitalclassicist.org/ http://www.currentepigraphy.org/