Buyer Beware – Runpod Is Just Not a Production Environment, not for us anyway. This is a tale of customer service woes like one we have never seen before.
[Don’t have time to read this? Video in production – watch this space]
Note somthing we were unaware of when this article was written – To clarify people with a blue name are not RunPod staff, and are volunteering. They are giving their time for free to try to help users, but are not paid by RunPod for doing this.
In 15 years of trading, I cannot recall us ever writing a negative post on our own blog. But this is an exception, and this is something any of our clients, the other businesses we work with, and anyone who might stumble upon this thread needs to be aware of…. Our RunPod nightmare.
Firstly for those unfamiliar with RunPod; They offer leased server hosting. So you pay for server on a need-be basis, and one of the key USPs is you only pay for the time you use, not 24/7.
Our scenario was very specific, but not uncommon. We required an R&D platform for some AI implementations we are putting in place across the business and for clients.
- We needed a powerful GPU (5090)
- 300GB Storage
- Hosting OLLAMA _ Multiple models – QWEN/FLUX & QWAN
We found runpod which we thought matched our requirements. It would be perfect for Ollama, and the AI models we were using. They also had templates which included Ollama, so we pulled the trigger and ordered.
All was good. A Few months in and we were happy. The configuration isnt easy, as they have isolated docker containers running the software, and separate workspaces for files/software which you need to retain after reboots…. But all in all, a solid solution beyond the learning curve. Or so we thought…and then the problems began. We needed support…
We opened a ticket via their support platform;
Keep it simple, explain the issue and expect some support…so I hoped.
Now we were in a semi-production environment, so I didn’t flag this as urgent. However, 48 hours went by without a reply, except to say ‘Hector’ had been assigned. So I dropped a chaser;
After around 56 hours we got a reply;
Not great. 56 hours in and all I get is that someone will look into it. Had this been a production situation… had clients been losing business we would now be having seriously problems. The assigned guy is saying “talk soon”. Not giving me any information, no timescales, and not telling me what the specific problem is. “An issue with the host name” I knew already, that’s why I reported the problem!
I somehow hold my composure. I remain polite;
The next day I finally get a positive reply from Hector
So he explained that they ‘found the issue’. And they fixed it.
So without being arrogant, or negative even, I get this email late at night, and when I read it I am immediately sceptical. Those 5 words that make me wonder…’have they even tested this’;
“It should come up normally”
Now I have worked in the I.T industry long enough to know, that if you fix something for a customer, you say “The issue is resolved”, or “The problem is fixed….this was the cause”.
When someone says “it should come up normally’ my immediate thought ‘they haven’t tested it’, or ‘they haven’t done anything and they are sending the email as a holding email’.
However, off I went to the office (10pm), to see if the issue was indeed fixed….
I wasn’t surprised, the issue was not fixed. No change from 4 days ago when I reported the problem. However now the pod reported an additional error, that said “We have detected a critical error on this machine which may affect some pods. We are looking into the root cause and apologize for any inconvenience. We would recommend backing up your data and creating a new pod in the meantime.”
I took some solace from his message which had said “If you still see it stuck or encounter any errors, let me know and I’ll have the team take another look right away.”, however the message about taking a backup was alarming. How can we take a backup when the server is down!
So I replied;
Another 24 hours passes and a new support person pops up;
Please note “highest priority” and “urgency”.
At this point we are 5 days deep, and I haven’t seen any evidence at all of anything being worked on. Like zero improvement, zero changes, and definitely no improvement on our server.
Now imagine… if this had been rolled out and we had 100 customers reliant on its performance. We would probably now be losing business, our clients would be losing money, and we have had just a few lines of communication and not improvements.
At this point I reply to Namrata, still polite, and explain our services are down, critical work is being lost. I ask for timescales. To my delight, after12 hours I get a reply;
WONDERFUL I thought. However, notice that line again
“Are you able to restart it ? ”
5 days down….services and over 100 hours work critically lost….. And the agent says “could YOU test” and “IF you are able to restart it.”
Surely they wouldnt tell me its fixed, if it isn’t fixed. So off I went to try again.
Lo and behold…it’s still down. No change from 5 days ago, exactly the identical problem with no improvement whatsoever, and no evidence that anyone has hands on and diagnosed the problem.
I hate to harp on…. But just imagine if this was mission critical. This could be a business ending situation, and RunPod, honestly, don’t really seem to care.
I reply, and follow it up with a second reply…at this point I have lost hope that they care, let alone in them fixing it.
So then, I decide, how about the Discord group….maybe just maybe someone there can help. Someone who can, might see it, and might help.
So I join the Discord and I open a thread there;
Notice the little button “Help us help you, link your account”. I better do that….so I linked our discord to our RunPod.
What words of wisdom do you think came back. I mean, have a guess, how might they give me some advice on how to get this pod up and running again…..
Wowsers. “Open a ticket”. If only I had thought of that.
Trying to hold it together, I hold off from replying, before another reply drops in;
So he is now asking another member of staff to check my ticket, on an account for which he apparently cant see my ticket, since he asked me to open a new ticket.
Now slightly losing patience (as well as productivity), I reply…
And the reply…
So Jason says “all good”.
Sorry Jason, all is not good. You cannot treat customers like this, when what they are buying takes tens or even hundreds of hours of investment.
Watch this space for the inbound YouTube video of our RunPod customer service experience.