We're a small ML team for a project and we keep running into the same wall: large preprocessing jobs (think 50–100GB datasets) running on a single machine take hours, and when something fails halfway through, it's painful.
We've looked at Prefect, Temporal, and a few others — but they all feel like they require a full-time DevOps person to set up and maintain properly. And most of our team is focused on the models, not the infrastructure.
Curious how other teams are handling this:
- Are you distributing these jobs across multiple workers, or still running on single machines?
- If you are distributing — what are you using and is it actually worth the setup overhead?
- Has anyone built something internal to handle this, and was it worth it?
- What's the biggest failure point in your current setup?
Trying to figure out if we're solving this the wrong way or if this is just a painful problem everyone deals with. Would love to hear what's actually working for people.
[link] [comments]
