[cheesecake-dev] Comments from Connelly Barnes

[Michał Kwiatkowski]

Worst case scenario: The authors of this project assert that the
kwalitee metric is almost always "right," that all packages should use
unit testing, and follow standards such as including a setup.py. The
Python community comes to believe that a project with low kwalitee
must have low quality. The major Python projects such as scipy,
pycrypto, PIL, PyOpenGL, etc make only halfhearted attempts at
achieving a high kwalitee score, and end up not having a high kwalitee
score because their project architecture is not really built to
accommodate the things Cheesecake is looking for (for example, scipy
might have the unit tests in a directory unit_tests/ instead of tests/
which Cheesecake looks for, and it may use its own unit testing
framework). Thus either Cheesecake incorporates specific fixes which
allow the major Python projects to keep their eccentricities and still
achieve a high kwalitee score, or people ignore the kwalitee metric
for the "major" projects, assuming that if a project is widely used
then it MUST have quality. Thus people are willing to make an
exception to the rule that "kwalitee = quality" for major projects,
but not for minor projects. This forces minor projects to adhere to
the "Cheesecake way" or be ignored. In the end we have (a) more
bigotry in the Python community and (b) many people wasting a large
amount of time trying to tweak packages endlessly and get a "high
score" on Cheesecake.

It should be pretty easy to prevent this worst case scenario from
happening. One solution is to clearly state that the Cheesecake
recommendations are only that -- the Cheesecake score is a subjective
measure, and following the ideals put forth by Cheesecake may in the
authors' opinion often be "good" for the average package, but may not
be "good" for every package. It might also help to state that
achieving a high Cheesecake score per se is not the goal, rather if
the ideas espoused by Cheesecake are followed and a low Cheesecake
score results, then this should be considered a bug in the Cheesecake
program and not something that the package author should fret about.
My major point here is that the Python community is pretty uptight and
rather prone to have religious assertions of subjective "fact," and
two things that I would very much not like to see is an entire
community converted to the "Cheesecake is absolutely correct" opinion
or an endless flamewar between the "Cheesecake is absolutely correct"
and "Cheesecake is absolutely wrong" factions.

I have not been terribly clear, so let me illustrate with some
examples. When I asked about some obscure thread issue in the Python
chat channel, I got lectured that threads should never be used and
also told that threads are just fine. When I asked about globals I was
told that I should never use globals. When I asked about looking up an
execution frame's f_back attribute, I was told not to use this. I
won't pretend that I like such patronizing attitudes -- in fact, I
find them to be the single worst aspect of the otherwise excellent
Python world! I don't think my experiences are unique, indeed after
reading a rather insightful rant by Stevie Yegge it seems that in the
Python community there is something endemic which causes people to
believe that certain things are absolutely "correct" while others are
"incorrect." For some issues, there are factions which debate these
ideas back and forth endlessly. I think that if you emphasize that
Cheesecake need not be in either the "correct" or "incorrect"
category, then people may calm down and instead take it to be a
generally useful metric but not Gospel nor the Book of Satan.

Finally, if you aren't convinced that unit tests and a setup.py are
not always appropriate, I'll try to convince you. In the popular
package pygame, many modules cause graphical output, interface with
the video card, interface with the sound card, or interface with input
devices. While the pygame package currently has only 3 unit tests, and
more unit tests might help the project, it is not possible to have a
really comprehensive unit test suite for the project because outputs
are not consistent on different computers/graphics cards/sound cards
(OpenGL does not specify exact behavior for pixels, color depth may
change, the sound card may only support a fixed number of channels,
etc). Also while unit tests for input devices might be useful, it
would be impossible to run these in an automated testing process! In
many cases it is more appropriate to simply run an example program and
determine whether the graphics animate correctly on the screen,
whether the mouse works, and whether the sound works. For an example
of where a setup.py is inappropriate, you might look at htmldata by
myself -- this module includes unit tests, docs, etc but does not
include a setup.py because it is only one Python source file. It seems
easier to drop the file in the appropriate directory than to use the
setup.py process. I might be wrong! However, the general principle
stands: it's probably "better" for everyone if package maintainers
generally try to use "good" coding practices, however if they
intentionally bend a few rules then it's OK, and the entire Python
world doesn't need to freak out or devolve into a flamewar.

Finally, I apologize if I have hurt the feelings of anyone on the
Cheesecake team. I do not expect that the Cheesecake team will say
that their metric is "always right" -- it was rather a hypothetical,
dystopian exaggeration to emphasize the potential problems that may
arise in a somewhat "religious" community. I hope most of my post was
just silly worrying, and that none of these problems will arise. I do
think Cheesecake is a good idea, and will probably use it myself to
determine if projects are well-commented, use unit tests, and so forth
(and perhaps even to determine for my own projects whether I
remembered to add docstrings!). Thanks for your time. I'd be
interested in hearing your thoughts.

Connelly Barnes