-
-
Notifications
You must be signed in to change notification settings - Fork 21
Managing a Test Repository
Sorting test alphabetically by application works best if your test are written to test a particular software package. This technique makes it easy for end user to find software test by application fairly quick.
(siddis14-TgVBs13r) buildtest[master ?+] $ tree apps
apps
├── a
├── b
├── c
├── d
├── e
├── f
├── g
├── h
├── i
├── j
├── k
├── l
├── m
├── n
├── o
├── p
├── q
├── r
├── s
├── t
├── u
├── v
├── w
├── x
├── y
└── z
The sub directory for each application can look as follows
(siddis14-TgVBs13r) apps[master ?+] $ tree a
a
├── amber
│ ├── include
│ │ └── header.h
│ └── src
│ ├── file1.c
│ └── file2.c
└── ansys
The source code is placed in src and header files stored in include
Pros:
- Simple and intuitive way for finding test
Cons:
- Does not work for application test that use multiple modules
- Can lead to duplicate tests where some tests can be compiled with different compilers + mpi
In this method we will categorize test into different components based on the test use case. As a general rule we will start off by covering compilers, openmp, mpi and gpu.
The directory format can look as follows
The root of component tree would look like this
(siddis14-TgVBs13r) buildtest[master ?+] $ tree -L 1 components/
components/
├── compilers
├── gpu
└── mpi
Any serial test using compilers such as gnu, intel, pgi, clang, etc... can go under compilers/apps directory
(siddis14-TgVBs13r) components[master ?+] $ tree compilers/
compilers/
├── apps
│ ├── clang
│ ├── gnu
│ ├── intel
│ └── pgi
└── commons
├── dotprod
├── helloworld
└── jacobi
Test that can be cross compiled between compilers such as hello world program can reside under compilers/commons. In commons directory, the test should be organized into sub-directory that highlight the name or functionality of the test.
The inner directory structure will be same as outlined above.
Similarly gpu application can be written in cuda, opencl, openacc, stl, etc... All of these language
programming language can be categorized in gpu/apps and the common gpu apps can be stored in gpu/commons
(siddis14-TgVBs13r) components[master ?+] $ tree gpu
gpu
├── apps
│ ├── cuda
│ ├── openacc
│ ├── opencl
│ └── stl
└── commons
├── helloworld
└── matrixmultiplication
Similarly mpi test specific to an mpi wrapper will go under mpi/apps with the name of the compiler and test that
are common between mpi can be stored in mpi/commons. At the moment we have not classified test that could be supported with MPI-1, MPI-2, MPI-3.
(siddis14-TgVBs13r) components[master ?+] $ tree mpi
mpi
├── apps
│ ├── intelmpi
│ ├── mpich
│ ├── mvapich2
│ └── openmpi
└── commons
├── latency
├── p2p_bandwidth
├── ping-pong
└── ring-test
We can extend this concept similar to how OpenHPC manages its components see https://github.com/openhpc/ohpc/tree/obs/OpenHPC_1.3.6_Factory/components
There are some questions we need to think about apart from categorizing test
Add list of maintainers per test
This can be done relatively easy. As part of YAML configuration add the following field
maintainers:
- Shahzeb Siddiqui <shahzebmsiddiqui@gmail.com>
- John Doe <john.doe@example.com>
User email address can change one option could be to tie email associated with their github profile to ensure git commits
match the user.email field in git configuration
How many total test do we support?
This could be a threshold defined in the framework BUILDTEST_TEST_LIMIT=1000 let's say and total test library must be within this limit. We want a sustainable set of test that are useful.
The project maintainer would need to define this limit during each release depending on volume of test and make decision on what test to drop.
Should we leverage test from other vendors, third party, hpc sites? If so what about incorporating test with existing license?
To be on safe side we should try to develop our own test and not rely on other maintainers unless we know what the test is doing
Should we accept a test for an unpopular software or rarely used in HPC sites?
This needs to be determined by project maintainers.
At some point we need to archive test that as demand changes. If we categorize by application then we can use the same format with a directory __archive__
(siddis14-TgVBs13r) apps[master ?+] $ tree __archive__
__archive__
├── a
├── b
├── c
├── d
├── e
├── f
├── g
├── h
├── i
├── j
├── k
├── l
├── m
├── n
├── o
├── p
├── q
├── r
├── s
├── t
├── u
├── v
├── w
├── x
├── y
└── z
26 directories, 0 files
In this scheme __archive__ will reside in the same directory level as all the alphabets
(siddis14-TgVBs13r) buildtest[master ?+] $ ls apps/
a __archive__ b c d e f g h i j k l m n o p q r s t u v w x y z
Archiving test can be done during every release which would be a decision made by the project maintainers. Test archival which will be discussed later.
The framework should provide means for users to vote on test through the framework.
- Like/Dislike
- Test Satisfaction Score (0-100)
Like and Dislike is a pretty simple scoring mechanism where +1 is awarded for Like and -1 for Dislike. The framework can codify this in yaml configuration or maintain a database with scores for each test.
The command line interface could be something like
buildtest vote --id=<test> --choice=like or buildtest vote --id=<test> --choice=dislike. The framework
will record the entry based on the git configuration user.email. buildtest will need to verify the email
is valid otherwise vote will not be accepted. User can change the vote from like to dislike but rerunning
same command to like or dislike should not be allowed.
In this method a user is requested to score the test from 0-100 and average or median can be take from all
results to determine test satisfaction score.
The command line usage could be
buildtest vote --id=<test> --score=67.2
The project maintainer may want to sort the satisfaction score or put a minimal threshold such as 60% for making it to core test library. Similarly any test that falls below threshold can be moved to archival if it was present in core test library.
User can vote on all test in a core or archive and project maintainer will need to decide which method works best.