[cheesecake-dev] Comments from Connelly Barnes

[Grig Gheorghiu]

First of all, thank you very much, Connelly, for taking the time to give us
your feedback. One of the goals of the Cheesecake project is to make people
aware of these issues and to generate a discussion within the Python
community, which discussion will hopefully prove constructive and will help
raise the 'kwalitee' of Python projects.

I agree with you that the Cheesecake scores are not absolute, and must be
taken with a grain of salt. We'll test our algorithms on the best-known
projects in the Python world, and it those projects will have abysmal
scores, we'll know we'll have to tweak the algorithms. However, a certain
minimum standard should be met by all projects. I think all of us Python
programmers expect to be able to install a package via "python
setup.pyinstall". Simply dropping the package files in a directory
leads to all
kinds of problems, as I have repeatedly seen. I personally think it would be
much better if people started to use setuptools more heavily and made their
packages available as eggs. Cheesecake will reward packages that do this.

As far as unit tests are concerned, let me disagree with you. I'm a *very*
strong believer in unit tests. In the example you give, the pygame
developers should be able to mock the external interfaces, and still have
plenty of unit tests for the core logic in their code. I understand that
code that deals with external interfaces is hard to test automatically, but
those are integration tests, not proper unit tests. If all packages adhered
to the rule that their unit tests can be run via "python setup.py test",
then the naming of the directories where their unit tests reside would not
even matter (and as a matter of fact, Cheesecake *will* look into any
directory with 'test' in its name, so it will discover a directory called
unit_tests for example).

Again, I'm glad we're starting to have a discussion about these things, and
I'm sure Michal's work over the summer will uncover many such issues that
need to be brought out in the open and discussed in the community.

Grig
-- 
http://agiletesting.blogspot.com

On 6/2/06, Michał Kwiatkowski <constant.beta at gmail.com> wrote:
>
> Worst case scenario: The authors of this project assert that the
> kwalitee metric is almost always "right," that all packages should use
> unit testing, and follow standards such as including a setup.py. The
> Python community comes to believe that a project with low kwalitee
> must have low quality. The major Python projects such as scipy,
> pycrypto, PIL, PyOpenGL, etc make only halfhearted attempts at
> achieving a high kwalitee score, and end up not having a high kwalitee
> score because their project architecture is not really built to
> accommodate the things Cheesecake is looking for (for example, scipy
> might have the unit tests in a directory unit_tests/ instead of tests/
> which Cheesecake looks for, and it may use its own unit testing
> framework). Thus either Cheesecake incorporates specific fixes which
> allow the major Python projects to keep their eccentricities and still
> achieve a high kwalitee score, or people ignore the kwalitee metric
> for the "major" projects, assuming that if a project is widely used
> then it MUST have quality. Thus people are willing to make an
> exception to the rule that "kwalitee > but not for minor projects. This forces minor projects to adhere to
> the "Cheesecake way" or be ignored. In the end we have (a) more
> bigotry in the Python community and (b) many people wasting a large
> amount of time trying to tweak packages endlessly and get a "high
> score" on Cheesecake.
>
> It should be pretty easy to prevent this worst case scenario from
> happening. One solution is to clearly state that the Cheesecake
> recommendations are only that -- the Cheesecake score is a subjective
> measure, and following the ideals put forth by Cheesecake may in the
> authors' opinion often be "good" for the average package, but may not
> be "good" for every package. It might also help to state that
> achieving a high Cheesecake score per se is not the goal, rather if
> the ideas espoused by Cheesecake are followed and a low Cheesecake
> score results, then this should be considered a bug in the Cheesecake
> program and not something that the package author should fret about.
> My major point here is that the Python community is pretty uptight and
> rather prone to have religious assertions of subjective "fact," and
> two things that I would very much not like to see is an entire
> community converted to the "Cheesecake is absolutely correct" opinion
> or an endless flamewar between the "Cheesecake is absolutely correct"
> and "Cheesecake is absolutely wrong" factions.
>
> I have not been terribly clear, so let me illustrate with some
> examples. When I asked about some obscure thread issue in the Python
> chat channel, I got lectured that threads should never be used and
> also told that threads are just fine. When I asked about globals I was
> told that I should never use globals. When I asked about looking up an
> execution frame's f_back attribute, I was told not to use this. I
> won't pretend that I like such patronizing attitudes -- in fact, I
> find them to be the single worst aspect of the otherwise excellent
> Python world! I don't think my experiences are unique, indeed after
> reading a rather insightful rant by Stevie Yegge it seems that in the
> Python community there is something endemic which causes people to
> believe that certain things are absolutely "correct" while others are
> "incorrect." For some issues, there are factions which debate these
> ideas back and forth endlessly. I think that if you emphasize that
> Cheesecake need not be in either the "correct" or "incorrect"
> category, then people may calm down and instead take it to be a
> generally useful metric but not Gospel nor the Book of Satan.
>
> Finally, if you aren't convinced that unit tests and a setup.py are
> not always appropriate, I'll try to convince you. In the popular
> package pygame, many modules cause graphical output, interface with
> the video card, interface with the sound card, or interface with input
> devices. While the pygame package currently has only 3 unit tests, and
> more unit tests might help the project, it is not possible to have a
> really comprehensive unit test suite for the project because outputs
> are not consistent on different computers/graphics cards/sound cards
> (OpenGL does not specify exact behavior for pixels, color depth may
> change, the sound card may only support a fixed number of channels,
> etc). Also while unit tests for input devices might be useful, it
> would be impossible to run these in an automated testing process! In
> many cases it is more appropriate to simply run an example program and
> determine whether the graphics animate correctly on the screen,
> whether the mouse works, and whether the sound works. For an example
> of where a setup.py is inappropriate, you might look at htmldata by
> myself -- this module includes unit tests, docs, etc but does not
> include a setup.py because it is only one Python source file. It seems
> easier to drop the file in the appropriate directory than to use the
> setup.py process. I might be wrong! However, the general principle
> stands: it's probably "better" for everyone if package maintainers
> generally try to use "good" coding practices, however if they
> intentionally bend a few rules then it's OK, and the entire Python
> world doesn't need to freak out or devolve into a flamewar.
>
> Finally, I apologize if I have hurt the feelings of anyone on the
> Cheesecake team. I do not expect that the Cheesecake team will say
> that their metric is "always right" -- it was rather a hypothetical,
> dystopian exaggeration to emphasize the potential problems that may
> arise in a somewhat "religious" community. I hope most of my post was
> just silly worrying, and that none of these problems will arise. I do
> think Cheesecake is a good idea, and will probably use it myself to
> determine if projects are well-commented, use unit tests, and so forth
> (and perhaps even to determine for my own projects whether I
> remembered to add docstrings!). Thanks for your time. I'd be
> interested in hearing your thoughts.
>
> Connelly Barnes
>
> _______________________________________________
> cheesecake-dev mailing list
> cheesecake-dev at lists2.idyll.org
> http://lists2.idyll.org/listinfo/cheesecake-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists2.idyll.org/pipermail/cheesecake-dev/attachments/20060602/b18c0de4/attachment.htm