So, we had some trouble with Shapeoko 5 Pro PCBs that we hope is behind us.
(This is going to get nerdy; I apologize in advance.)
In the scope of electronics manufacturing, we're not exactly Apple, but we're not ordering prototype quantities either. We're trapped squarely in the middle, which makes it tough to find a manufacturer that's a good fit for us.
For about four years, we've used a small PCB manufacturer founded by a group of guys that broke off from our old vendor. We're a good customer for them, and they've always treated us well- until our last order.
We still don't know what changed there, but we had two crucial problems in our last batch of Shapeoko 5 Pro boards.
The Cold Solder Joint
The component (an optoisolator) we use for the BitSetter input had a cold solder joint. Not every board had the problem, just some of the boards. For those who don't know, a cold solder joint didn't get to a high enough temperature to flow correctly between the PCB and the component mounted on it. The challenging part is that a failure from a cold solder joint can be intermittent; it will work for a while and then fail.
In the case of the BitSetter input, the boards worked fine for testing and then failed in the field or when the board warmed up enough to grow from the temperature (which is totally normal). This one took a long time to figure out because it was an intermittent problem for many affected people.
The Missing Component
Problem #2 is that they completely forgot a crucial component on the output for the spindle (a pull up resistor). This missing component causes the spindle to go right to full RPM all the time. This one was not intermittent; it happens all the time.
The Resolution
The resolution to both problems is to send out a new set of electronics, which we've done in large quantities. We've sent out a lot of replacement units, and it's cost us more than we've paid the PCB vendor in the past year or two. Right now, we're not sure if we'll continue with them, or find a new vendor.
Any machine shipped after 9/20/2023 definitely has a PCB from the new batch.
The Crushing Support Burden
As you can imagine, this problem crushed the support team, which was trying to diagnose intermittent issues by trying to discover the common problems. Until the BitSetter problem was found to be the solder joint, it wasn't clear if it was the PCB, the wiring harness, or the BitSetter itself, so we sent a lot of replacement components.
Once we figured out the problem, we distilled the troubleshooting into a quick 3-step process so we didn't have to put customers through a long troubleshooting session.
To make matters worse, right in the middle of this, I sent out a newsletter to everyone on the mailing list announcing the release of Carbide Motion V6, which led to another problem. Many customers downloaded the new version of Motion and then misconfigured their machines- leading to a second wave of support tickets.
The Silver Lining
At the height of the problem, our support ticket queue had never been higher in the history of the company, which led to everyone chipping in to help out. As we got "fresh eyes" involved, we found significant inefficiencies. I understand why they happened, we've grown a lot over the past ten years, and it's primarily been organic, just-hire-another-person growth.
By changing some of the priorities and processes, we've managed to get to zero active tickets at the end of the day several times in the past week. Time will tell, but it seems sustainable right now, which means our support response time should be dropping below the times we had before PCB-geddon.
We've also made it mandatory for all product development folks to be in the support queue during the week. We want them to know the pain inflicted on users when they make a bad assumption about how the product will be used and what it means to a customer when a product ships with a manufacturing error. Jorge, Edward, and I have always tried to be in the support queue, but the rest of the people have already had some "a-ha" realizations.
The support trouble caused by Carbide Motion V6 is more challenging to figure out. We can't make changes if they lead to a crushing support burden, but we also don't want to cut off support for older machines. We've got code for some good stuff paused because we can't risk another support wave.
Right now, we've settled on adding "feature flags" to Carbide Motion, letting us invite groups of people to try a new feature in Carbide Motion by entering a code. This will allow us to open up changes to wider and wider groups of people so we don't have massive numbers of people all jumping on at once. When a feature is "done," we'd remove the need for a code.
Time will tell if this is a good idea, but it's a path forward for now.
Conclusion
Summarizing everything above, this sucked for us and new Shapeoko 5 Pro owners. This is not what we want the first experience with Carbide 3D to be for a new user.
Once we end up in the middle of a problem, all we can do is try and treat people's time with the respect we'd want to be treated. We've tried our best to do that and that will continue to be the goal going forward.