How are you managing long-running preprocessing jobs at scale? Curious what's actually working [R]

Reddit r/MachineLearning / 4/28/2026

💬 OpinionIdeas & Deep AnalysisTools & Practical Usage

共有:

Key Points

The article is a Reddit-style question asking how practitioners manage long-running preprocessing jobs at scale for machine learning workloads.
The author specifically wants to understand what “actually works” in real trials, rather than just reviewing documentation.
It invites discussion on what the main breaking points are, such as setup complexity, ongoing maintenance, or other operational challenges.
The focus is on practical, operational experiences and failure modes encountered when running preprocessing pipelines in production.

Did anyone actually trial these properly for Machine Learning Jobs before walking away, or was it more of a ‘looked at the docs and noped out’ situation? Specifically curious what the breaking point was — setup complexity, ongoing maintenance, or something else entirely.

submitted by /u/krishnatamakuwala
[link] [comments]