Test management tool expectations

Suggested considerations

The important aspects for any test management tool are:

Must have

• Be appreciated by the users of all main target stakeholder groups (possibly prioritized and ranked).
• Support current process or way-of-working rather than imposing a specific way-of-working (unless you don't have a way-of-working and are willing to let a tool guide you into one).
• Be able to express regression test cases in suitable format.
• Be able to tag test cases for categories for different testing purposes.
• Description possibility for test data descriptions for different test environments and other variations.
• Tagging possibilities for different test suite combinations.
• Access management - viewers vs editors.
• Correspond to technical requirements for installation platform, security restraints, and so forth.
• Related costs that are within budget, both initial and running costs (support and operations).
• Cater for the expected number of users - and more.

Should have

• Status reporting capabilities.
• Statuses for test cases (e.g. under development/design, ready, obsolete).
• Storing of test results.
• Trend graphs and analysis features.
• Support for exploratory testing (recording, mind-mapping, screenshotting).
• Test execution progress reporting capabilities.
• Manual test execution mode for viewing test cases and reporting test step results.
• Integration to build engine for automated test execution results propagation.
• Integration to bug tracking system.
• Possibilities to upload pictures for test step descriptions.
• Notification system.
• Stable vendor that will be around for foreseeable future.
• Licensing model that doesn't limit usage due to license constraints.
• Explicit support for Gherkin syntax/BDD/Specification by example.
• Easily googleable solutions to problems.
• Appreciated way of working and GUI so people don't object spending time in the tool.
• Suitable billing options.
• Active support for older products to enable upgrade delays when necessary.
• Should include functionality for multiple products and multiple teams for scalability.
• Should have appealing response times when used under nominal load, and during report generation/dashboard use with relevant data volumes.
• The tool should be well documented.
• The tool should have a wide user base.
• Should be backup-able without taking down the database.
• Any email server integration, or any other integration, should have technical specification to match current setup.
• Version control of relevant test assets.
• Accessibility features.
• Support regional characters for all relevant fields.
• Customization of workflow and data fields.
• Traceability to requirements/acceptance criteria/feature cards/user stories/use cases, or whatever describes the described expectations that tests could be based on.
• Security features for business-critical secrets.
• Features to enable communication, comments, and discussions around tests.
• Test licenses for tool test environment for free to test out product upgrades, platform upgrades, assess addons/plugins, or other tests or evaluations regarding the tool.

Could have

• Accessible and understandable by business to discuss regression tests and to communicate test progress and status.
• Possibility to assign test cases to specific individuals.
• Possibility to assign test steps to specific individuals.
• Support escalation routines.
• Upgrading without need to close down system.
• Test automation tool integration for manually starting automated test execution of selected tests.
• Formal education or courses on usage and administration of the tool.
• Tool code base ownership if tool is abandoned by vendor should not force an immediate tool switch.
• AD/LDAP integration for user management and possible single-sign-on.
• Data export mechanisms for relevant server-side data entities.
• Usage statistics reporting.
• Templating for data assets.
• Performance metrics for operation.
• Possibilities to import data from previous test management systems.
• Scheduling features for test execution, both in parallel and sequence.
• Good data structure to enable easy transition to next test management tool in the far future.

General tool selection process

When choosing a tool you might have to adhere to rigid and formal process of requirement gathering, vendor evaluation, proof-of-technology (POT), proof-of-concept (POT), organisation setup, aquisition, introduction and more.

However, as a minimum I'd recommend doing the following:

1. List expectations that constitute a good tool in your opinion
2. Make a sweep of relevant tools to identify what main categories of tools there are
3. Identify what category of tool is relevant for you and create a short-list of about five tools to analyze further
4. Quite quickly bring that list down to two or three candidates for proof-of-concept