Experienced Devs

3959 readers

9 users here now

A community for discussion amongst professional software developers.

Posts should be relevant to those well into their careers.

For those looking to break into the industry, are hustling for their first job, or have just started their career and are looking for advice, check out:

Logo base by Delapouite under CC BY 3.0 with modifications to add a gradient

founded 1 year ago

MODERATORS

[email protected]

How much flakiness do you tolerate in end to end tests? (programming.dev)

submitted 1 year ago* (last edited 1 year ago) by [email protected] to c/[email protected]

22 comments fedilink hide all child comments

End to end and smoke tests give a really valuable angle on what the app is doing and can warn you about failures before they happen. However, because they're working with a live app and a live database over a live network, they can introduce a lot of flakiness. Beyond just changes to the app, different data in the environment or other issues can cause a smoke test failure.

How do you handle the inherent flakiness of testing against a live app?

When do you run smokes? On every phoenix branch? Pre-prod? Prod only?

Who fixes the issues that the smokes find?

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 3 points 1 year ago (1 children)

Having the e2e smokes be a requirement for PR merge is frustrating. But I’ve been on the fence before with this on my own teams. It’s enticing to have a completely “clean” main branch that has not been infested with regressions caused by a PR.

It also gives you confidence in the crummy cases where you need to push a fix to prod right now.

If the e2e’s flap too much, then it is not an option. I’ve tried it and it lasts one sprint before we nix it. It’s just too frustrating and development comes to a standstill.

We have a retry policy on any smokes that need to run in a step by step order, and we aggressively prune and remove smokes that frequently fail or don't test for real issues.

I actually think that’s the best way to handle it.

Who fixes the issues that the smokes find?

On teams I’ve been on, typically a junior dev. Sounds crummy, but it actually gives them more experience with the product/code. I have been that junior dev before and I found it a positive experience.

[–] [email protected] 5 points 1 year ago

I find it interesting that the junior would fix these issues. In my team we established a fictive role which each developer takes in sprint rotation. The developer in this role would then handle these cases and also drive the investigation of incoming support requests / bug reports, playing third level support in a sense.

How do you handle support requests in your team/company? Is it the developers or do you have a dedicated team for that?