wiki:TestingNotes

Version 3 (modified by melissa, 9 years ago) (diff)

--

Various enhancements to the data-driven tests

Most of these should probably be turned into tickets...

  • Reading of sub-images. Similar to the md5 test for full planes, it would be a good idea to check the md5 of a fixed-size sub-image (or at the very least, verify that an exception is not thrown).
  • Leaking file handles. File handles that are left open are known to cause a problem on Windows, so we should somehow find a way to test that there are no open file handles after close() has been called on a reader.
  • Reading planes in a random order. All of our tests right now read planes sequentially from 0 to n, but most formats should support reading in any order (compressed QuickTime is the notable exception).
  • Consistent type checking. We could add a field to the config files that gives the name of the reader that should be used for that file. Not particularly useful for files with unique extensions, but for many of the TIFF variants this could be useful.
  • There should be an option to map all of the files into memory before running the tests.
  • A lot of the config files need updating anyway, so now might be an OK time to change the config file format to something a bit more readable. What we have now is fine for files with one series, but for big plates it gets to be a bit unwieldy. Something either XML- or INI-based might be easier to work with.
  • Also, it would be nice to be able to opt out of specific tests in the config files. E.g. OME-XML sanity tests are sometimes expected to fail, so it would be better to just skip the test rather than reporting it as failed and having to re-investigate the failure every time.
  • Also also, it would be nice to be able to test metadata beyond core metadata (so we'd need to be able to define this metadata in the config file). Physical sizes, channel names, and various Instrument/Detector/Laser linkages in particular would be good to test.
  • Experiment with migrating the data-driven tests to JUnit (see also #498).
  • If a test fails because setId failed, then we should skip all subsequent tests for that file.
  • Once we have a reasonable set of public test data, set up a Hudson job to run the data-driven tests on the public test data.

Public test data

Most of the data-driven tests would benefit from having a comprehensive set of public sample data (see also #157).

Formats for which we already have public sample data:

A '*' indicates that we could generate more public data in this format.

  • ICS (*)
  • Leica LEI
  • IPLab
  • BMP (*)
  • Image-Pro SEQ
  • QuickTime (*)
  • Bio-Rad PIC
  • Image-Pro Workspace
  • Fluoview/ABD TIFF
  • Perkin Elmer Ultraview
  • Gatan DM3
  • Zeiss LSM
  • Openlab LIFF (*)
  • Leica LIF (*)
  • TIFF (*)

Formats for which we can definitely generate public sample data:

  • PNG/APNG
  • JPEG
  • PGM
  • FITS
  • PCX
  • GIF
  • Openlab Raw
  • OME-XML
  • OME-TIFF
  • AVI
  • PICT
  • LIM
  • PSD
  • Targa
  • Bio-Rad Gel
  • Fake
  • ECAT-7
  • NRRD
  • JPEG-2000
  • Micromanager
  • Text
  • DICOM

Formats for which I need to check whether or not we can generate public sample data:

  • IPLab Mac (Ivision)
  • Deltavision
  • MRC
  • Gatan DM2
  • Imaris (HDF)
  • SDT
  • EPS
  • Alicona AL3D
  • MNG
  • Khoros
  • Visitech
  • InCell
  • L2D
  • FEI
  • NAF
  • MINC
  • MRW
  • TillVision
  • ARF
  • LI-FLIM
  • Oxford Instruments
  • VG-SAM
  • Hamamatsu HIS
  • WA-TOP
  • Seiko
  • TopoMetrix
  • UBM
  • Quesant
  • RHK
  • Molecular Imaging
  • JEOL
  • NIfTI
  • Analyze 7.5
  • Olympus CellR/APL
  • Amira
  • Unisoku
  • Perkin Elmer Densitometer
  • FV1000 .oib/.oif
  • Zeiss ZVI
  • Nikon ND2
  • SimplePCI .cxd
  • Imaris (raw)
  • Imaris (TIFF)
  • Molecular Devices Gel
  • Aperio SVS
  • Imacon .fff
  • LEO
  • JPK
  • Leica TCS
  • Nikon NEF
  • Nikon TIFF
  • Prairie
  • Metamorph TIFF/STK/ND
  • Improvision TIFF
  • Photoshop TIFF
  • FEI TIFF
  • SimplePCI TIFF
  • Burleigh
  • SM-Camera
  • SBIG

Formats for which we definitely cannot generate public sample data:

  • Slidebook
  • Cellomics
  • CellWorX
  • Olympus ScanR
  • BD Pathway
  • Opera Flex
  • MIAS