binaries/assimp-3.1.1/test/regression/README

Assimp Regression Test Suite
============================

1) How does it work?
---------------------------------------------------------------------------------
run.py checks all model in the <root>/test/models folder and compares the result
against a regression database provided by us (db.zip). If the test passes
successfully, Assimp definitely WORKS perfectly on your system. A few failures
are totally fine as well (see sections 7+). You need to worry if a huge
majority of all files in a particular format or post-processing configuration 
fails - this might be a sign of a recent regression in assimp's codebase. 

2) What do I need?
---------------------------------------------------------------------------------
 - you need Python installed - 3.x !!
 - you need to build tools/assimp_cmd as described in the INSTALL file (
	make && make install on unixes,release-dll target with msvc).

3) How to add more test files?
---------------------------------------------------------------------------------
Use the following procedure:
 - verify the correctness of your assimp build - run the regression suite.
   DO NOT continue if one or more tests fail.
 - add your additional test files to <root>/test/models/<fileformat>, where
   <fileformat> is the file type (typically the file extension)
 - rebuild the regression database using gen_db.py
 - run the regression suite again - all tests should pass, including
   those for the new files.

 - contributors: commit the db.zip plus your additional test files to
   the SVN repository. 

4) I made a change/fix/.. to a loader, how to update the database?
---------------------------------------------------------------------------------
 - rebuild the regression database using gen_db.py
 - run the suite - all tests should pass now. If they do not, don't continue
 
 - contributors: commit the db.zip to the SVN repository 

5) How to add my whole model repository to the suite?
---------------------------------------------------------------------------------

Edit the reg_settings.py file and add the path to your repository to
<<model_directories>>. Then, rebuild the suite.

6) So what is actually verified?
---------------------------------------------------------------------------------
The regression database includes mini dumps of the aiScene data structure -
the scene hierarchy plus the sizes of all data arrays MUST match. Floating-point
data buffers, such as vertex positions, are handled less strictly: min,max and
average values are stored with lower precision. This takes hardware- or 
compiler-specific differences in floating-point computations into account. 
Generally, almost all significant regressions will be detected while the
number of false positives is relatively low.

7) The test suite fails, what do do?
---------------------------------------------------------------------------------
Get back to ../results and check out regression_suite_failures.txt 
It contains a list of all files which failed the test ... they're copied to
../results/tmp. Both an EXPECTED and an ACTUAL file is produced per test.
The output of `assimp cmpdump` is written to regressions_suite_output.txt. 
To quickly find all all reports pertaining to tests which failed, I'd
recommend grepping for 'but' because its a constituent of all error messages
produced by assimp_cmd :) Error reports contain detailed information
regarding the point of failure inside the data structure, the values of
the two corresponding fields that were found to be different from EXPECTED
and ACTUAL, respectively, ... this should offer you enough information to start
debugging.

8) fp:fast vs fp:precise fails the test suite (same for gcc equivalents)
---------------------------------------------------------------------------------
As mentioned above, floating-point inaccuracies between differently optimized
builds are not considered regressions and all float comparisons done by the test
suite involve an epsilon. Changing floating-point optimizations can, however,
 lead to *real* changes in the output data structure, such as different number 
of vertices or faces, ... this violates one of our primary targets, that is
produce reliable and portable output. We're working hard on removing these
issues, but at the moment you have to live with few of them.

Currently, the regression database is build on Windows using MSVC8 with 
fp:precise. This leads to a small number of failures with fp:fast and
virtally every build with gcc. Be patient, it will be fixed.