Changes between Version 5 and Version 6 of TestingNotes

08/24/10 15:56:40 (10 years ago)

Added links to newly filed tickets.


  • TestingNotes

    v5 v6  
    11== Various enhancements to the data-driven tests == 
    3 Most of these should probably be turned into tickets... 
    5  * Reading of sub-images.  Similar to the md5 test for full planes, it would be a good idea to check the md5 of a fixed-size sub-image (or at the very least, verify that an exception is not thrown). 
    6  * Leaking file handles.  File handles that are left open are known to cause a problem on Windows, so we should somehow find a way to test that there are no open file handles after close() has been called on a reader.  
    7  * Reading planes in a random order.  All of our tests right now read planes sequentially from 0 to n, but most formats should support reading in any order (compressed QuickTime is the notable exception). 
    8  * Consistent type checking.  We could add a field to the config files that gives the name of the reader that should be used for that file.  Not particularly useful for files with unique extensions, but for many of the TIFF variants this could be useful. 
    9  * There should be an option to map all of the files into memory before running the tests. 
    10  * A lot of the config files need updating anyway, so now might be an OK time to change the config file format to something a bit more readable.  What we have now is fine for files with one series, but for big plates it gets to be a bit unwieldy.  Something either XML- or INI-based might be easier to work with. 
    11  * Also, it would be nice to be able to opt out of specific tests in the config files.  E.g. OME-XML sanity tests are sometimes expected to fail, so it would be better to just skip the test rather than reporting it as failed and having to re-investigate the failure every time. 
    12  * Also also, it would be nice to be able to test metadata beyond core metadata (so we'd need to be able to define this metadata in the config file).  Physical sizes, channel names, and various Instrument/Detector/Laser linkages in particular would be good to test. 
    13  * We should be able to specify an alternate MD5 hash for the first plane.  JPEG-compressed planes, for instance, have different MD5 sums on Linux and Mac OS X; so if you generated the MD5 on Mac OS X and test on Linux, the test will fail even though the pixels are correct. 
    14  * Experiment with migrating the data-driven tests to JUnit (see also #498). 
    15  * If a test fails because setId failed, then we should skip all subsequent tests for that file. 
    16  * Once we have a reasonable set of public test data, set up a Hudson job to run the data-driven tests on the public test data. 
     3 * #498 migrate (back) to JUnit 
     4 * #531 MD5 test for sub-images 
     5 * #532 test for leaking file handles 
     6 * #533 reading planes in a random order 
     7 * #534 consistent type checking 
     8 * #535 allow files to be mapped into memory 
     9 * #536 consider changing config file format 
     10 * #537 non-core metadata consistency 
     11 * #538 alternate MD5 hash 
     12 * #539 skip tests when setId fails 
     13 * #540 add Hudson jobs to run data-driven tests 
    1815== Public test data ==