2 | | Any living code base requires checks to make sure that its continued development does not introduce any new bugs. AstroBEAR's developers are working to create a comprehensive lists of tests (the ''test suite'') which runs a set of simulations in order to ensure the fundamental behavior of the code is robust and unchanged. |
| 2 | Any living code base requires checks to make sure that its continued development does not introduce any new bugs. AstroBEAR's developers are working to create a comprehensive lists of tests (the ''test suite'') which runs a set of simulations in order to ensure the fundamental behavior of the code is robust and unchanged. Tests should run and report results with minimal interaction required. Ideally, running a test and viewing the result should be as easy as executing a single script. |
| 3 | |
| 4 | The basic design philosophy is as follows: |
| 5 | * There exist 3 levels of interactivity and completeness when running the tests -- |
| 6 | 1. Verification tests, run via verifytests.s (Est. time: ? min) |
| 7 | 1. Individual tests, run via their own {{{runtest.s}}} files, located in their respective folders (within modules/tests/). |
| 8 | 1. General tests, run via alltests.s (Est. time: ? min) |
| 9 | 1. Showcase tests, run via show.s (Est. time: ? min) |
| 10 | |
| 11 | The above scripts are all located in the code's main directory, and are intended to be executed in that location after a successful build of the code. Test result may be different depending on the test. For simple tests the result may consist of "pass" or "fail". For other tests, or in cases where failure is reported, the result may need to include certain figures, such as comparison figures (comparing the result versus the expected result). |
| 12 | |
| 13 | '''Verification tests''' (verifytests.sh) should be ran by developers after ANY changes to the code, before checking them in. efter every code checkin in order to to ensure that. ''For simplicity, all verification tests will run {{{mpibear}}} on the current machine using 2 processors. More scripting will be needed to run larger showcase simulations if they need to run in parallel with 2+ cores in order to finish in a reasonable amount of time. |
| 14 | '' |
| 15 | |
| 16 | '''General tests''' are meat to be automated periodical (overnight) test with no user interaction which will post results on web. |
| 19 | * This will require [TestScripts scripts] which correctly runs the executable on a given platform. |
| 20 | * The run should be designed to produce data and exit in a repeatable fashion (i.e., not requiring the user to stop the simulation after a certain time). |
| 21 | * Upon completion, the script needs to invoke {{{bear2fix}}} which, via the input-file option, processes the data and produces a result. |
| 22 | |
| 23 | |
| 24 | '''Showcase tests''' are meant to illustrate whether the code functions as expected in a general sense when running well-understood simulations. These tests will produce images which show the endstate of the simulation. Inspection of these images by the user should confirm the simulation run; an indicator, or start point, that something is going wrong. |
| 25 | |
| 26 | ---- |
29 | | ---- |
30 | | |
31 | | The basic design philosophy is as follows: |
32 | | * There exist 3 levels of interactivity and completeness when running the tests -- |
33 | | 1. ALL tests may be run via alltests.sh (Est. time: ? min) |
34 | | 1. All STRICTLY CODE VERIFICATION tests may be run via verifytests.sh (Est. time: ? min) |
35 | | 1. Individual tests may be run via their own {{{runtest.sh}}} files, located in their respective folders. |
36 | | * The 3 run scripts are all located in the contrib/astro/tests folder, and are intended to be run in that location after a successful build of the code. |
37 | | * Note: Prior to checkins, we should require developers to run verifytests.sh, which performs barebones verification tests. |
38 | | * Periodically run alltests.sh on the code with no user interaction & post results to web. |
39 | | |
40 | | * The tests should run and report results with minimal interaction required. Ideally, running a test and viewing the result should be as easy as executing a single script. |
41 | | * This will require [TestScripts scripts] which correctly runs the executable on a given platform. |
42 | | * The run should be designed to produce data and exit in a repeatable fashion (i.e., not requiring the user to stop the simulation after a certain time). |
43 | | * Upon completion, the script needs to invoke {{{bear2fix}}} which, via the input-file option, processes the data and produces a result. |
44 | | |
45 | | * The exact nature of the result will be different depending on the test. For simple tests, the result may consist of little beyond "pass" or "fail." |
46 | | * For other tests, or in cases where failure is reported, the result may need to include certain figures, such as comparison figures (comparing the result versus the expected result). |
47 | | * Some tests will not necessarily produce success/failure results. These "showcase" tests will instead produce images which show the endstate of the simulation. Such images will then be viewed by the user(s) for confirmation. These tests in other words are meant to illustrate whether the code functions as expected in a general sense when running well-understood simulations. Since it is difficult, especially from the beginning, to design tests to cover every possible failure mode of the code, these tests serve as an important indicator that something is going wrong. (They therefore also serve as an important starting point in diagnosing such.) |
48 | | |
49 | | * For simplicity, all verification tests will run {{{mpibear}}} on the current machine using 2 processors. More scripting will be needed to run larger showcase simulations if they need to run in parallel with 2+ cores in order to finish in a reasonable amount of time. |
50 | | |