This week we added six new tests to the Falcon test suite. We had six tests, which where labeled as big tests. Big test means here, that the runtime or the resource requirements of that test are higher than usual. As our internal test systems are running steady integration tests on a 24/7 basis, we have to make a trade off and limit the test resource requirements to get an acceptable test throughput. However, from time to time these big tests should be run. In the Falcon team we agreed on running these tests in our Weekly Falcon Test Overview (WFTO) cycles.
We also see that there are two tests, which are not failing anymore. We disabled two tests, which will not be fixed in any near future:
We are interested in you! Where do you use Falcon? What do you do with Falcon? Are there any features you want to see in Falcon? You can test Falcon and get famous by providing valuable bug reports or even test cases for Falcon!
At MySQL we run a whole regression test suite on various platforms after each code “push”. If you are coming from CVS or SVN and alike, then think about a “commit” when we say “push”.
The test language we use is described here: MySQL Test Framework
We at the Falcon team wanted to see our progression in terms of bugs fixed, so for over 18 months we additionally are running Falcon related tests on a weekly basis. With lot’s of teams working on MySQL sometimes there are tests, which fail in rather random fashion. Such tests are usually disabled, so that other teams don’t get distracted. For our Weekly Falcon Test Overview we enable all disabled test, to see where we are.
I will try to do a public weekly report about the status of Falcon along with the Weekly Falcon Test Overview results, bugs we have fixed, and noteworthy news.
Last week we saw this bug fixes for Falcon:
As we are an Open Source company, our bug system is open to all: http://bugs.mysql.com/
If you are interested in Falcon related bugs, you can use our saved search at: http://bugs.mysql.com/saved/FalconBugs
As a special add-on for fans of numbers and statistics, here is the list of the Falcon test overview status in plain text:
2006-09-01 Failed 30/67 tests, 55.22% were successful. 2006-09-15 Failed 28/75 tests, 62.67% were successful. 2006-09-22 Failed 30/76 tests, 60.53% were successful. 2006-09-29 Failed 26/77 tests, 66.23% were successful. 2006-10-06 Failed 26/79 tests, 67.09% were successful. 2006-10-13 Failed 25/83 tests, 69.88% were successful. 2006-10-20 Failed 26/83 tests, 68.67% were successful. 2006-10-27 Failed 28/88 tests, 68.18% were successful. 2006-11-03 Failed 31/92 tests, 66.30% were successful. 2006-11-10 Failed 31/99 tests, 68.69% were successful. 2006-11-17 Failed 26/101 tests, 74.26% were successful. 2006-11-24 Failed 26/102 tests, 74.51% were successful. 2006-12-01 Failed 25/102 tests, 75.49% were successful. 2006-12-08 Failed 25/105 tests, 76.19% were successful. 2006-12-15 Failed 22/106 tests, 79.25% were successful. 2006-12-22 Failed 23/107 tests, 78.50% were successful. 2006-12-29 Failed 24/107 tests, 77.57% were successful. 2007-01-05 Failed 24/107 tests, 77.57% were successful. 2007-01-12 Failed 25/110 tests, 77.27% were successful. 2007-01-19 Failed 26/110 tests, 76.36% were successful. 2007-01-26 Failed 23/110 tests, 79.09% were successful. 2007-02-02 Failed 22/110 tests, 80.00% were successful. 2007-02-11 Failed 22/110 tests, 80.00% were successful. 2007-02-16 Failed 23/114 tests, 79.82% were successful. 2007-02-23 Failed 27/123 tests, 78.05% were successful. 2007-03-02 Failed 27/123 tests, 78.05% were successful. 2007-03-10 Failed 43/127 tests, 66.14% were successful. 2007-03-17 Failed 30/127 tests, 76.38% were successful. 2007-03-24 Failed 29/129 tests, 77.52% were successful. 2007-04-15 Failed 28/132 tests, 78.79% were successful. 2007-04-20 Failed 32/142 tests, 77.46% were successful. 2007-04-27 Failed 28/145 tests, 80.69% were successful. 2007-05-04 Failed 30/148 tests, 79.73% were successful. 2007-05-11 Failed 27/148 tests, 81.76% were successful. 2007-05-18 Failed 26/149 tests, 82.55% were successful. 2007-05-25 Failed 33/150 tests, 78.00% were successful. 2007-06-01 Failed 32/155 tests, 79.35% were successful. 2007-06-08 Failed 29/155 tests, 81.29% were successful. 2007-07-06 Failed 23/158 tests, 85.44% were successful. 2007-07-13 Failed 19/159 tests, 88.05% were successful. 2007-07-20 Failed 20/159 tests, 87.42% were successful. 2007-07-27 Failed 20/160 tests, 87.50% were successful. 2007-08-03 Failed 20/161 tests, 87.58% were successful. 2007-09-07 Failed 24/170 tests, 85.88% were successful. 2007-09-14 Failed 29/172 tests, 83.14% were successful. 2007-09-21 Failed 28/172 tests, 83.72% were successful. 2008-01-10 Failed 21/185 tests, 88.65% were successful. 2008-01-24 Failed 21/185 tests, 88.65% were successful. 2008-02-01 Failed 22/185 tests, 88.11% were successful. 2008-02-28 Failed 17/191 tests, 91.10% were successful. 2008-03-13 Failed 21/194 tests, 89.18% were successful. 2008-03-20 Failed 21/194 tests, 89.18% were successful. 2008-03-28 Failed 19/189 tests, 89.95% were successful. 2008-04-04 Failed 21/194 tests, 89.18% were successful.
Hotshot is bundled in the standard library and provide useful data. Maybe you have to install an extra package on some linux distribution if I recall it correctly, because it’s not GPL; iPython provides a nice interface to perform live debugging, like [...]
ContinuousIntegration: http://del.icio.us/popular/ContinuousIntegration