def test_segmenter ( segmenter , tests ): "Try segmenter on tests; report failures; return fraction correct." return sum ([ test_one_segment ( segmenter , test ) for test in tests ]), len ( tests ) def test_one_segment ( segmenter , test ): words = tokens ( test ) result = segmenter ( cat ( words )) correct = ( result == words ) if not correct : print 'expected' , words print 'got ' , result return correct proverbs = ( """A little knowledge is a dangerous thing A man who is his own lawyer has a fool for his client All work and no play makes Jack a dull boy Better to remain silent and be thought a fool that to speak and remove all doubt; Do unto others as you would have them do to you Early to bed and early to rise, makes a man healthy, wealthy and wise Fools rush in where angels fear to tread Genius is one percent inspiration, ninety-nine percent perspiration If you lie down with dogs, you will get up with fleas Lightning never strikes twice in the same place Power corrupts; absolute power corrupts absolutely Here today, gone tomorrow See no evil, hear no evil, speak no evil Sticks and stones may break my bones, but words will never hurt me Take care of the pence and the pounds will take care of themselves Take care of the sense and the sounds will take care of themselves The bigger they are, the harder they fall The grass is always greener on the other side of the fence The more things change, the more they stay the same Those who do not learn from history are doomed to repeat it""" . splitlines ())