Total: our choice the introduction of Test Rail.
- This is the receipt of real-time test results (creating reports on completely different criteria: Defect Summary, Comparison for Cases, test results for projects / components, etc.)
- This is an easy organization and convenient tracking of the load of the testing department (the ability to fully customize the "working dashboard" of the QA department)
- Easy installation, deployment and support
- "Communication" with the Jira (this is very convenient for regression and autotests)
- easy to receive reports on the launch of test suites
- beautiful and understandable reports and diagrams- This is an effective management of test cases, plans and test cycles (creating / storing / editing test scenarios, managing test plans, launching test cycles, entering test results)
- User-friendly interface (clear, simple, customizable)
- Effective management of the following scenario: "technical requirements -> test cases -> overview -> test suite -> test cycle -> results"
- convenient storage of test scenarios
- This is a significant improvement in the quality of testing (a clear description of test scenarios, their reviews, correlation with requirements, division into areas - all this makes it possible to evaluate both the completeness of coverage by functional tests, and is necessary material for the entire project team)
As one of the drawbacks - until recently, Test Report system was poorly implemented reporting system: scant results on test-wounds, lack of customization of reports, etc.
Working on a Global team, TestRail enabled everyone to have a seat at the Test Case management/creation table. Editing existing test cases was very easy to do, and the GUI makes it easy to find any existing Test Case you are looking for.
First of all, TestRail is an extremely smarter way to track and maintain test cases than using Excel. I like how easy it is to create new projects, create custom fields, create test runs, and generate good looking reports to submit to your managers. The GUI takes a little bit to get used to, but once you get a hand of all of the key-shortcuts moving around the application is easy!
My least favorite part about this software is the JIRA plug in. In order to see the linked test cases from TestRail in JIRA, you need to have a TestRail account. In my company only the testing team had TestRail accounts, while the rest of R&D didn't. While it was helpful for me as a tester to see the linked test cases on a JIRA bug, when we started off with TestRail we had assumed everyone in the R&D department would be able to see this information so we could be more transparent in our testing. My company was not willing to pay for 100 TestRail accounts, so this was a bummer. If yours is then TestRail might be right for you!
Having using various test management suites TestRail does excel in that it was built to run manual test cases and display results in an easy glance for the whole team. Linking to defects is simple, generating a report when the test run is complete. I feel like reusing regression suites could be improved, it does what it is meant to however you have to repeat several steps each time you want to generate a regression suite.
Some areas for improvement would be a test case type for API tests, I am sure everyone needs this but struggles to find a way to record this data.
I think report generation is the strong suit of Test rail, the percentage pie chart of passed, failed blocked cases at a glance make daily reporting for scrum stand up meetings ideal.
Integration with Jira is excellent, although some room for improvement.
Trying to do something simple like import test case steps from a .csv is not so easy. You can import a suite but that is not ideal.
Trying to duplicate cases can be tedious, let's say you have a lot of cases, trying to copy paste them in to various projects is a lot of hassle, it is surprising they have not addressed issues like this after so many years.
i have using this tool for my every day test execution before this i was using our own company proprietary tool and then we switched to TestRail as we switched to agile methodology. i have found this quite useful in agile environment. it is scalabe and can be customised accordingly to suite the customer/test requirement.
it is now integrated to our companies other tools like JIRA(bug tracker tool),github,automated test suites, bitbucket etc and many others and is working flawlessly. Reports generation and tracking the various result is quite simple and have real time insights for these with this tool. We can easily link our test cases, requirements, use cases and stores using this.
Really its a great tool with so many functionalities and i love it. it increase the team collaboration in a great way
Thanks so much from the TestRail team Bheesam for taking the time to complete this review.
Coming from a Quality Assurance team which had zero test cases and no system TestRail has become a huge asset. It allows for our team which has members working in a different time zone to continually work independently while still having the entire team engaged.
Right off the bat the software is very easy to use The software allow us to manage our time effectively focussing on running regression tests on any particular task. It has been a useful tool for our Quality Assurance team to prioritize and organize test cases. The fact that we can implement the use of Jira is also a huge plus since this is a tool we also use within our organization.
Not many cons, the only two minor aspects that come to mind are one if you have no knowledge of how to use it does take some time to learn it. However if you have even the basic background on a tool like this the learning time does shorten. The other is the fact that results can only be shared with the registered users, so we are having to take screen shots of results to share with the rest of our company teams.
I test software builds for a system which is installed into Taxis around the US. Each city has it's own configuration, rules and templates for printing & displaying info. There are several hardware configurations any taxi might have. By creating Projects based on functionality, I can copy/paste each required Project into a master Project then as a Test Run is created, check on/off individual Sections as needed and then test each build accordingly.
Having a report for each test Run allows management to go back & view each build's tes Run results when any "issue" is complained about by fleets/drivers to verify whether it was working or not in that build - as drivers are sometimes worse lairs than Lawyers or Politicians
1- versatility to create Projects which allow base line Test Cases to be used in a variety of different test runs for different products.
2- ease of being able to import test cases from existing methods
The main issue I have is the Projects are listed as a flat-file. We have several different groups which have created Projects and even though they might be testing a given entity from different sides, they have to name their Projects differently. We prefix each project with a Group Name so the Projects can be "grouped" and easily referenced from the main dashboard.
Another less important issue is there isn't really an easy way to make a backup of our entire Dashboard. We can Export each Project but I don't see any way to make backups of anything else outside of offloading each individual entity which really isn't all that hard.
Compared to other test case management systems TestRail is relatively easy to set up and configure and maintain. The online documentation is readily available, concise and thorough with useful links to further information. The UI is familiar enough to users of other commonly used test management systems that they don't take long to feel comfortable with TestRail, making implementation and adoption easier. The vendor support is quick and helpful.
It would be helpful to have more examples of how to realize the effectiveness of Milestones versus Test Runs and creating Test Case Templates. All of these are configurable within the UI however it takes a lot of trial and error to find how to make use of these features to best effect. Fields can be customized but some of the choices for required fields seems unnecessary, such as "Estimate" for test case. Adding the option to enter a "Description" or "How to Use This Field" for custom fields would be a useful addition as well.
First of all we easily implemented integration with test automation framework that saved manual team work. Second thing is reports that our managers could use for proper decision making based on product quility understanding.
The best thing about TestRail is their REST API. Thanks to it we were able to easily implement its integration with our test automation framework. In particular we were able to map test exucution statuses to TestRail test case statuses and allow updating them easily for test reruns. Second good thing is reports they allow to generate. That's what our managers wanted.
Sometimes UI is not intuitive. For instance, it took a while to figure out how to change test cases status in bulk.
TestRail has been useful in my QA position for tracking the progress of test cases in our weekly regression runs. The framework it gives you for test case management is flexible enough to be used in a variety of ways for our company, whether for bug testing or tracking what tests have been automated over time.
- The system of test suites (a master list of all your test cases) and test runs (individual lists of specific test cases) is well-built and allows you to manage your test cases easily.
- Product is consistently stable.
- Test cases in different parts of the product have synced states, which makes it easy to make changes to a test case when necessary.
- Tools for sorting test cases in a run/suite are powerful.
- There isn't a good tutorial for the product (or at the very least I never saw anything like a tutorial). As a result, it took me a while to notice some of the more unintuitive features.
- Collaboration can be a bit clunky sometimes (multiple users editing a single test case, notifications at email level only).
Test management became easier.
Valuable reports (that's the main purpose of testing)
Easy to onboard
Supports is repsonsive and helps with your case
Good API and integration with JIRA
There is no integration with confluence (to get bubbles like for JIRA defects/tasks)
You can't push an attachment in the jira (only can show an attachment in jira's plugin)
API needs some extra methods to work with plans and runs with configuration (you allow to manipulate with it, but only in strict cases)
Sometimes it takes too muck time to get a reply from staff on the forum.
Thanks so much for taking the time to do this. Much appreciated..
TestRail can be a helping hand to the QA engineers. It can now be integrated to JIRA for easier project management and testcase management, User need not to keep track of both of them seperately.
-TestRail provides easy management of test-suites, test-cases, test-runs. Easy management for maintaining agile methodology in the project.
-Feature to export/import files for test-cases from xml, csv and xls files.
-The provision to define milestones can be useful for maintaining release-wise goals.
-The User can define the expected duration and priority for a test-case.
- The representation and reporting are the best part of this software, the user can see the progress of the test-run in various formats as pie-chart.
- Test results should be editable as required in some cases.
Overall the solution is one of the best I used in the past 10 years. Powerful test case management capabilities and API for automation so we can aggregate manual testing results as well as results from automation. But missing some collaboration features we are used to from google suite.
- managed in cloud
- powerfull customization options
- easy to use
- option of templates for TC
- some reports are complicated to get, as who worked on what in the past quarter
- to customize some stuff is sometimes quirky
- hard to use for exploratory testing
- collaboration on preparing test plan is a pain, it does not allow two testers add stuff to test plan simultaneously
Good tool for test case Management
1. It is simple to login in. It takes your windows login/pwd
2. Test cases copying from the previous release is easy.
3 The pie chart display of TC run is precise and informative.
4. Its filtering and selection option is superb. And I like this feature very much.
1. I have administrative right to deleted folder, cases etc. But I feel delete option could have a bigger icon. Many times I have to search the tiny red button.
2. Artifact/screenshots addition option is good but after the run, you cannot add any artifact. You have to rerun test cases in testrail for future reference
3. Loading of GUIs takes little time [ but not often]
Thanks so much from the TestRail team Manoj for taking the time to complete this review.
We are a start up and as we grow, our test cases and test executions are growing. We could successfully manage our tests and executions very well. With a little work on integrating Test rail into our build process, we could get make it part of our build process.
Good way to organize test plans, cases and runs
Report is good at test planning level
Configuring runs for multiple devices/browsers in run configurations instead of creating new runs
APIs are good,
Can be integrated into our build process well.
nothing much. may you can support hippa complaint could offering?
We are able to clearly track the health of a build. With creating testplan we are able to track release readiness through milestones. Management can view report on mail or printout
Appreciate easy integration with JIRA, this allows us to map test cases to JIRA task & defects. Test plan is another feature which we appreciate. It allows us to track our progress on milestones. Adding additional attributes to test cases also helped us to resolve some requirements like multi level reviews. Configuration is another option which we have used extensively.
A direct integration with Confluence could be appreciated. We have always felt that tester needs to navigate to JIRA and to confluence to review any requirement change. As confluence is a widely used documentation tool and direct integration with the same would be widely appreciated.
We are using TestRail to manage automated tests on a variety of projects, from API tests to application acceptance tests. We've integrated TestRail into various CI platforms with ease.
TestRail is powerful yet easy to learn and use. It is extremely flexible and we have not encountered any application of it that was not a perfect fit. It has been widely adopted by our QA and Dev teams.
There are some admin features that could be easier such as managing users and grouping them for team access and policy rights.
We believe that testrail is basic like any other test case management system, features are fairly generic, workflows are fairly dated. However UI is good, probably due to the app being basic. We use it because teams have already comfortable with it, but by no means is this a tool that can solve new development methodologies such as continuous integration test management. If you understand the tools limitations, then may work for you, but we find it to be lacking and at this stage have stopped integrating our CI test runs into the tool.
TestRail's simple UI and functionality is straight forward. Easy for people to jump in and use the tool, assuming that the tool fits Dev/QA's team workflow and method of usage. For the administrator, access is easily manageable, and provides multiple categories for restricting access and defining user roles within the organization.
TestRail is no different than other test management systems, beside the front end and UI benefits. TestRail is good for teams who initiate test planning, test case creation, and management specific to older QA management styles. When introducing continuous integration testing into a development/QA' teams methodology, this is where TestRail needs improvement . TestRail does not do a good job logging individual test runs for historic tracking, DB backend cannot handle the data that is pushed to the system, and we often have to delete DB records to help with performance and lookup. Usually without these backend hacks, the system performs very poorly and is slow. TestRail may be good for teams that have not adopted advanced development and QA practices, but if teams want to integrate CI testing with test cases and suites within TestRail this is where we find it to fail.
We have had very good success with TestRail managing multiple products/features within the same instance. We have tight delivery deadlines and TestRail has allowed us to increase the speed we complete our tests by about 20%. I would highly recommend it!
- Very easy to use, the team was using it full time within 1 week
- Great price vs some of competition
- Reporting is very helpful for management
- Easy integration with automated testing tools
- Custom fields allows us to customize for our needs
- It would be nice if TestRail included bug tracking functionality, but it just integrates with other tools
- I wish you could add "read-only" users for free like some tools
Overall, we love the tool and have directed more teams to use it with the caveat that they do not have a lot of reporting needs.
Super easy to use and organize test cases. It works great and allows multiple structures depending on how your team manages test cases
The reporting capabilities are extremely limited. There are some canned reports that you can tweak. We struggle to find what we need without a custom report. You cannot download them as PDF. You need to download them as HTML which is not ideal in most work place scenarios. We can get most of the info we need from the dashboard, but it would be nice to download as a pdf to excel/csv.
It's faster than free product, better interface and easy to use API
The Test Cases Organization and flexibility to create tests. The possibility to add custom fields and that after, you can filter by those fields
The way the Test plans are managed, there is no easy way to delete tests from a given configuration in a test plan and the reporting is very basic, not allowing Leads or managers to see progress for individuals
Thanks so much from the TestRail team Sergio for taking the time to complete this review.
- Web based - doesn't require a localized server installation (unless you want to) and can be accessed by anyone with credentials, no matter where they are located without any effort
- Very easy to use and gives you a very neat category division when it comes to test cases, test runs and milestones
Can't search and replace words in multiple test cases at the same time which makes it harder to re-use existing test cases with only small changes
We were able to import our large set of tests in just a few short days and be up and running with a more robust tool that offered us an agile transformation.
*customized for any project
* easy to get reports from
* web based
* can import from variety of other tools
* good forums for help and ideas
* can easily make it complex for others depending on how you setup
* a few known issues that can cause data loss
Thanks so much from the TestRail team Juan for taking the time to complete this review.
Strong test case management, meaning more organized testing, meaning higher quality product.
- Intuitive UI system that allows you yo easily organize test suites, test runs, and projects.
- Sync logged bugs to the test cases you're testing for transparency.
- Reporting and the ability to estimate testing allows for you to gather more information easily.
- It can be quite expensive for small teams, but it's well worth it.
- Creating test cases in TestRail is pretty tedious (which is why I typically just import them).
- Reporting system is a little complicated to understand and utilize effectively.
My team is using this tool to add testcases for all the projects and we uses testplan for the releases.
This tools helps us to organize the testcases and testplan. Adds lot of visibility to the entire team and it’s reporting is very useful
We can’t add testcases in the testplan across different project. Each platform has its own project so when we want to combine the common cases in one testplan, it doesn’t provides us the way.
Our employees enjoyed the user interface, and the ease to organize test cases and move them around. We have seen value in the ability to link up to Jira for tracking bugs that have been found and also linking to requirements that are also tracked in Jira.
The ability to have reporting at a higher level across all projects within TestRail. Need to have a better manager level view of everything.