ProdPerfect

ProdPerfect was exactly what we needed as a small software business that did not want to hire a full time front-end QA engineer.

Onboarding was easy because their software automatically measured the user flow in our web app, which is then used to create test cases.

The PP team created the test cases and get them up and running without very much input from me. Once testing was operational, we held a highly effective weekly meeting series to discuss the system use, testing issues, and new test cases.

The ProdPerfect team has good handle on what is needed in the roadmap. In general it would be ideal for the test execution cases to be run in parallel with no-dependency executions. This is something they have been working on but is not ready yet.

Not having this feature means that testing of each case is done in series which takes more time and there are some reporting challenges with failures. For example the JavaScript environment is not aware of which test case is running, so the test case ID cannot be included in logging reports (e.g. Sentry)

The worst part about setting up an outsourced QA team is training them on the flows users go through - simultaneously how users behave and where they are interacting with the product. The highest value proposition of ProdPerfect to me is the tracking beacon that lets them not only figure out what flows to monitor from the start, but as we launch feature changes, ProdPerfect notices the change in behavior and builds out the next suite of tests. It's incredibly hands off but offering us continued and accurate protection.

The data viz tool to look at how users are behaving on the site could use some tuning. It's a bit information overload and unintuitive to manipulate to make it useful. Luckily, I have never had to actually use it and after talking to the team at ProdPerfect I was relieved to know I didn't ever actually have to use it - and they're working on a better way to represent the heaps of data they have on our user behavior already.

PP removes a class of problem from the list of things I need to "do something" about. Regression is a no-win situation for eng management. Do it well and nobody notices. Screw it up and you never live it down.

Lack of UI/visibility. If you deal with regression internally you have the same problem ... that you're doing a lot of work that doesn't visibly contribute to forward movement. Buy ProdPerfect and the only visible indicator of effort is a product that works.

- Setup is hands on where you want it to be, hands off where you want it to be. ProdPerfect determined which tests would be the most effective, wrote and delivered them to us, but we got to decide exactly how they fit into our pipelines.
- Test maintenance and writing is completely automatic – we know the outcome of the tests that are written but don't need to be involved in the creation or upkeep of them.
- Tests are high quality – important workflows have been saved on multiple occasions now from being broken due to the suite catching issues that weren't caught by our unit tests.
- Communication is top notch – very professional, quick to respond, friendly.
- Multiple times cheaper than doing it ourselves. We would have either had to hire QA or have existing engineers context switch to writing regression tests alongside regular development and unit tests.

This sounds ridiculous but we really don't have any. We got exactly what we expected.