Discussion Groups / ESL Teaching / May 2005

I think the Oxford Wordsmythe tools at oup.com are what you're looking
for, but you should carefully check them out before you sign up. I
might not have a real handle on what you really need.

You might want to hit postgraduate student discussion groups at major
applied linguistics places like U. Edinburgh and Nottingham.

Good luck.

Is there any possibility that you might have the stones to try a
little programming? There are a variety of computer languages that do
this sort of thing. You might be on a learning curve for a while, but
you would end up having complete control of your results. Compilers,
interpreters and user manuals for these languages are available on the
net for free, so all you need invest is your time. PERL used to be the
language of choice: http://www.perl.com/ but it has been surpassed in
my opinion by PYTHON http://www.python.org

Chances are any software packcage that offers what you want will have
been written in one of those two, since such parsing and analysis is
their forte.

   Steve> Hi, Occasionally I go to Project Gutenberg (
   Steve> http://www.gutenberg.org ) for public domain books that
   Steve> have been digitized and made available for downloading.

Digitized?  An  *image* (e.g.  in  JPEG, GIF, PNG formats)  of scanned
book pages and documents is also  in digitized form.  I hope you don't
mean that.  If you're talking  about text format, then you have better
luck.

   Steve> I  thought it might  be interesting  to examine  them using
   Steve> text analysis  software that would  give me an idea  of the
   Steve> difficulty   level

No.  Unless you define  "difficulty level".  Computers can't read your
mind.  They  won't know  what you mean  by "difficult".  If  you don't
give them a clear and unambiguous mathematical formula for "difficulty
level", they can't do it.

   Steve> and frequency count of words and phrases.  

Frequency count of words is trivial.

Phrases...   that's more  difficult, esp.   with  ambiguous sentences.
("Time flies  like an arrow;  fruit flies like  a banna."  How  do you
break  them  up into  phrases?)   Computers  are  pretty incapable  of
dealing with ambiguities.

   Steve> I would also like to have the word count and frequency
   Steve> results graphed or made available for export to Excel.

   Steve> Is anyone familiar with a software package, freeware or
   Steve> otherwise, that can handle this level of analysis and
   Steve> charting?

Word frequency is trivial.  A simple unix command does it:

       tr -dc a-zA-Z < *.txt | sort | uniq -c | sort -nr

would give you  the frequency of words, listed  in descending order of
frequency, appearing in the  files matching "*.txt".  Pipe the results
into a charting software (e.g. gnuplot) and you're done.

   Steve> I    did     find    the    web     site    textalyser    (
   Steve> http://www.lexicool.com/text_analyzer.asp  ) that  has some
   Steve> of these capabilities but  no way to specify varying phrase
   Steve> lengths or  export options for charting nor  does it handle
   Steve> book size jobs.

And I guess it won't do "difficult level".  Right?

Thank you everyone for your help, Steve.