Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I'm curious how your mobile testing compares to https://revyl.com
I've been experimenting with Revyl and it's really nice. I think this agent-driven testing is the future.
by tcoff91 - Try the OSS alternative - https://github.com/vostride/agent-qa
- We support both web and mobile, which is what a lot of companies prefer, just one agent for both. Also, I'm pretty sure Revyl relies only on vision models, which tend to be slower. We built the platform around a hybrid approach that combines vision and accessibility APIs, which is much faster.
Would love to hear your feedback after you try it out!
by okwasniewski - Curious how they frame testing on real devices: “Real devices on demand: spin up iOS simulators and Android emulators”. So those are not real/physical devices?
Physical devices for AI agents is something we at TestingBot do provide: https://testingbot.com/support/ai/mcp
by defied - Does it work of mobile native applications or expo apps that have native modules?
Pricing question, the usage on the plans seems low considering in the demo you said that you have 25 tests per pr which would mean you get only 10 PRs per month on the hobby plan?
by yohguy - Yes, it works for any framework. We just get the built native binary and run it in the cloud.
Regarding pricing, the self serve options are currently only for lower usage. We will add more plans further down the line. Currently the most popular one is the startup plan. If you need more usage I’m happy to discuss it on a call!
by okwasniewski - Great presentation
On a slight tangent, since we are all here...
Does anyone still believe there is a long-term future in traditional UI/UX?
It feels like a lot of attention is still going into landing pages, dashboards, and CRUD apps, while overlooking a bigger shift where fewer people will actually need to interact with those interfaces directly when the same tools can perform the underlying tasks automatically, without much UI at all.
So the bigger question is does UI/UX evolve into something else, or does a large part of it simply disappear?
I might be a bit too early. Recently I started a project and decided to skip all of that and focus to make it more friendly to AI agents and frankly so far it has been great purely from user experience but also what it delivers.
by _pdp_ - Is there a long term future in hand-crafted UI/UX? Maybe not.
Is there a future where we still have traditional UX? Absolutely.
I don't want to write a whole dissertation on this topic, so I'm just going to mention that we tried to build AI voice assistants for a decade, and while LLMs have basically solved understanding, they have not solved the UX portion.
by Eridrus - How can it perform tasks automatically? It's not magic, there has to be an UI/UX for interacting with it. Will that UI/UX be more optimized and easier to use is the question. Like would you prefer saying "close window computer" or press alt+f4 or just click on the little cross thing or equivalent. Why are we assuming all AI automagic UI/UX will be better for all tasks?by altmanaltman
- I wonder how does it compare to mobileboost.io, which has been used by some companies like Duolingo?by j0sip
- Our approach is heavily focused on agents, both for executing tests and for managing the platform. We want to provide the best and simplest way to conduct agentic testing, with a strong focus on details. It looks like their platform also requires a sales call.by okwasniewski
- Love your approach to product. It feels like TesterArmy will become the "Vercel for testing". Refreshing stuff!
- Thank you! That's the goalby okwasniewski
- Some digging FAST_MODEL = "google/gemini-3-flash" (fast mode primary) DEEP_MODEL = "openai/gpt-5.4" (deep mode primary) VISION_CLICK_MODEL= "openai/gpt-5.4" (the visual grounder)
fast: gemini-3-flash, falls back to gpt-5.4, 15-min run timeout, max 2 visual calls/step. deep: gpt-5.4, 15-min timeout, max 3 visual calls/step.
Why such a hard timeout, and why not latest models?
- We found gemini 3 flash to be the best model as of now, when it comes to bang for the buck, GPT 5.5 is also a bit more expensive than 5.4, if we run tests at scale it has to be affordable. Once a newer model that fits into the criteria is released, we will update it!by okwasniewski
- Was writing E2E tests ever a problem that needs automation? Also E2E tests need to be updated everytime a new feature is added. TesterArmy sounds great. But config overhead and potential security leaks makes it a no goby negamax
- Try the OSS alternative - https://github.com/vostride/agent-qa
- Have you been able to nail down a loop where your tool can take an open pr, guess the code path and do some testing?
We use cypress heavily for our core flows which has a similar ai prompt thing but it’s not quite ad hoc enough for smaller fixes which is where the bottleneck still comes in for us.
by msencenb - Yes! We spent quite a lot of time on this, and we are currently creating a test plan based on PR changes and sending an agent to verify it. We have some customers who are only using this feature.by okwasniewski
- On the mobile side, what's your strategy for state across cold starts? Agentic test runs tend to fall over the moment auth tokens expire mid-flow, because the agent has no notion of 'this state should persist across this device reboot but not that one'. Curious how you handle keychain/secure-storage in your test rigs.by hugorus87
- Has anyone tried to build their own version of this?
It's cool, but I'm not super excited about using some 3rd party SaaS as a critical part of my testing.
by Eridrus - I did. Checkout the OSS alternative - https://github.com/vostride/agent-qa
- "Traditional E2E tests are slow to set up and expensive to maintain." I don't really understand this. If I'm already using Opus to write the code, surely it would know best what E2E tests to write to be able to verify its own output? This seems like an unnecessary external step.by dbbk
- Unfortunately from our experience tests don’t scale as well as code. First of all static tests are very brittle, you rely on selectors, need wait times and can’t really test a lot of dynamic content (think AI chats/interactions). Then it’s all the infrastructure around it: solving captchas, handling auth, handling email OTP (each of our agents has access to its own inbox) and handling video recording and screenshots. So with the traditional testing approach you end up mocking a lot of services. I highly recommend you to give it a try!by okwasniewski
- E2E tests are now quick to write due to LLMs, and are then deterministic AND cheap to run. How would this compare to the token costs of running an agent the whole time for each test? How do you make sure results stay stable regardless of the nondeterministic nature? Do customers still need to create test cases - any way to import from test case management system - based on which they could have already generate e2e tests locally?by poisonborz
- Unfortunately from our experience tests don’t scale as well as code. First of all, static tests are very brittle: you rely on selectors, need wait times, and can’t really test a lot of dynamic content (think AI chats/interactions). Then it’s all the infrastructure around it: solving captchas, handling auth, handling email OTP (each of our agents has access to its own inbox) and handling video recording and screenshots.
To ensure stable results we do a lot of harness engineering, where we inject trajectories of previous tests to ensure the stability and also the split into smaller steps helps to prevent context overload and decision fatigue.
Regarding test case management, our customers have used our CLI to migrate their existing test cases from whatever system they were using before.
by okwasniewski