Just want to note that both examples are of what I'd call "restricted
use cases" :-). I.e. cases where you're dealing with a fairly small,
closed domain.
Slightly more useful would be a list of the hundred or so most common
words in each language and their frequency. That way you could decide
where you wanted your cutoff to be. It ought to be possible to
generate that off the Perseus dataset. In fact I'd bet this is
something they've already done. Any Perseids on the list? If not, I
can make enquiries.
H
On Aug 25, 2008, at 7:16 AM, Gabriel Bodard wrote:
> I certainly agree that there are times when you would want to
> *include* stopwords in your search/indexing/analysis, but I can
> think of one use-case for omitting them. When we compiled the
> indices for IAph2007, we originally planned to exclude "common
> words" (and have them in a secondary index for those interested...
> after all in a digital publication generating an index is not a
> human task, so there's no reason not to do something that you're
> doing anyway). In the end we ran out of time before we implemented
> this secondary index, and so never got around to drawing up a list
> of such common words. See for example: <http://insaph.kcl.ac.uk/iaph2007/inscriptions/indices/word/word-grcK.html
> > where the entry for καί (kai) is pretty ridiculous, or even
> better <http://insaph.kcl.ac.uk/iaph2007/inscriptions/indices/word/word-grcO.html
> > where three-quarters of the document is taken up by the lemma ὁ
> (ho), whose inflected forms appear on almost every line of almost
> every inscription. You might argue no harm done, but when we produce
> a printed index of these texts, we'll certainly want to omit
> stopwords.
>
> (I vote +1 to creating a list of such words in the DC wiki. On what
> grounds would such words be selected? By frequency? On grammatical
> grounds? What have existing projects done? Could someone approach
> the Dartmouth Dante team and ask for their list? What *useful*
> format would such a list be in? (Text, XML, SQL table, Python list...)
>
> G
>
> Hugh Cayless a écrit :
>> I don't know of one, and I wonder whether anyone's ever seen a need
>> for one. Stopwords can help as a sort of performance optimization in
>> search engines with a restricted set of use cases, but once you get
>> beyond a certain domain limit, they just aren't useful (you can
>> search
>> for 'a' on Google for an example of what I mean). Philologists are
>> often very interested in words that might get dropped by a stopword
>> list. I might want to find particular uses of 'et' for example, and
>> be very irritated if the results told me I couldn't.
>> I've implemented search engines a few times now and honestly never
>> had
>> a use for stopwords in the end for any of them. I sort of don't
>> believe in them anymore...so my question would be: what's the use
>> case, and do you really need one?
>> Hope this helps,
>> Hugh
>> On Aug 24, 2008, at 6:28 PM, Neven Jovanović wrote:
>>> Hello,
>>>
>>> does anybody know where could one look for a list of stop words for
>>> Latin?
>>> I have seen an English stop words list on Perseus
>>> (http://www.perseus.tufts.edu/Texts/engstop.html), but have not been
>>> able
>>> to find anything similar for Latin. Yes, the Dartmouth Dante
>>> (http://dante.dartmouth.edu/help.php) mentions "stopword list" for
>>> Latin,
>>> but does not make it available.
>>>
>>> It seems that such a list is something that always gets compiled
>>> from
>>> scratch. Perhaps a version of it, made freely available, could be a
>>> welcome contribution to the Digital Classicist wiki.
>>>
>>> (Not to mention a Greek stop word list...)
>>>
>>> Yours,
>>>
>>> Neven Jovanovic
>>>
>>> Zagreb, Hrvatska / Croatia
>
> --
> Dr Gabriel BODARD
> (Epigrapher & Digital Classicist)
>
> Centre for Computing in the Humanities
> King's College London
> 26-29 Drury Lane
> London WC2B 5RL
>
> Email: [log in to unmask]
> Tel: +44 (0)20 7848 1388
> Fax: +44 (0)20 7848 2980
>
> http://www.digitalclassicist.org/
> http://www.currentepigraphy.org/
|