Tuesday, February 21, 2017

Taken by a Strange Mood, Part 5 - The shutdown problem

We're going to call this blog post "The Shutdown Problem".  The Raspberry Pi, though small in stature and large in utility, is in fact a full linux computer.  That means that all the things that come with a full linux computer apply, included the need to shut the computer down before pulling the plug on the power.  This isn't normally a problem because most normal people just have their pis plugged into a wall outlet until they've actually done a manual shutdown.  I don't have this luxury.  My Pi is powered by the TV itself, and when I hit the off button on the TV, so goes the Pi.

This is a problem.  In my setup, there is no guarantee of a safe shutdown.  This TV is ultimately going to live in my kids' playroom, so they're just going to be hitting the off button when they're done with it.  If I don't do something about this problem, we'll eventually get file system corruption, and I'll have to start the whole process over again.  Fortunately, there is a solution.

I'm going to take this moment to apologize for not having this blog post up sooner.  I believe its been nearly a month or more since my last post.  Its been that long because its taken me that long to get my elegant solution actually working.  I went through many trials, more errors, and intermittent success before working out all the kinks.  And now that I have, I can officially call this project "finished".

So, where to start?  I guess we'll define the specific problem.  When the power goes off, my pi goes off, without a shutdown.  How do we fix it?  Shutdown at power down.  But we can't complete a shutdown at power down because power is already down.  We'll need some power stored to keep the pi alive during shutdown, and we'll need a way to sense when power has been cut in order to initiate the shutdown automatically.

Solution#1:  In line rechargeable battery.  You know? One of those big secondary batteries that you get to charge your iPhone when you don't have a wall to plug into.  Fully packaged, simple, easy.  It also won't work.  When a Pi shuts down, it isn't doing a true shut down, its doing a system halt.  Effectively the same thing, but different enough to give me trouble.  Once a pi is halted, the only way to unhalt it and cause it to boot is to pull the plug, or hit the reset pins.  Unfortunately, I didn't know about the reset pins until after I had already permanently glued my pi into the TV.  I couldn't reach them.  So since we can't boot without pulling the power, that means I wouldn't be able to start up the pi again until the battery completely died. With a battery the size of the ones available, even the smallest ones, that could take hours, if not days or weeks.  We need a better solution.

Solution#2, provided by the nice folks at the adafruit forums: create a supercapacitor circuit and place it in line with the power supply.  I'm not going to go into the nitty gritty of capacitors, but suffice it to say that they are capable of storing energy, and then releasing it when the charging voltage is removed.  This is a much better solution because capacitors charge relatively quickly, and more importantly for my application, they also discharge relatively quickly.  After I did some math and estimations I determined I needed about 10 farads worth of capacitance, at 5 volts.  Oh, and the capacitor also needed to be able to discharge with a current of at least 1.5 amps.

I jumped onto the internet, to get buying.  I found one that I thought was perfect.  A single 5V capacitor that could store like 15F.  I bought it.  It came in.  I wired up my circuit.  I plugged it all in.  I let it charge.  I pulled the plug. It didn't work.  I double checked the data sheet: this capacitor was only capable of supplying 35 miliamps.  Back to the internet.  This is when I hit a snag.  I could not find a single capacitor rated at 5V, 9-15F, and also 1.5 amps or more.  But I did find capacitors at 2.7V, 30F, and like 9 amps (it was alot, more than I would draw).  This complicated the circuit, because now I'd need two capacitors in series.  Caps in series add their voltages, just like batteries.  But they also half their capacitance. (Actually, its a formula. 1/C = 1/C1 + 1/C2 + ...)  The other problem with using more than one capacitor is that, due to manufacturing tolerances, no two capacitors are exactly alike, and if they are different enough they will be imbalanced, and eventually one or both of them will break.  So I needed a balancing circuit.

Behold My Circuit Diagram. Yes, I know my diode symbol is backwards.
Another problem with capacitors is that when they are completely discharged, they look like a short circuit to a power supply.  And most supplies freak out when they think they've been shorted.  Some burn out.  Others just cut their supply so as to not burn out.  This is what mine does.  How to get around this?  You have to put a small resistor in front of the capacitors so that it no longer looks like a short.  It has to be small because the resistors themselves reduce the amount of charging voltage available to the caps, thus reducing the maximum charge I can put into the caps.  So 1 or 2 ohms is sufficient.  Unfortunately, the smallest resistors I had at the time were 100 ohm.  And I discovered that this was way too much resistance.  What's a guy to do?  I took all 11 of my 100 ohm resistors, twisted their ends together, and put them in parallel.  This produced an equivalent resistance of just under 10 ohms.  This worked much better, but was not ideal.
Actual Circuit.  The two large caps are connected, the smaller one isn't.

The other problem with these resistors and capacitors was that if the resistors were just put straight into the charge lane, they'd also be in the way of the discharge lane, and reduce discharge voltage, giving me headaches.  The solution to this is to put a diode in parallel with the resistors.  The diode prevents current from skipping over the resistors when charging, but allows current to flow around the resistors when discharging.  You see, capacitors discharge in the reverse direction of charging.

So, after much trial and error, I finally had a circuit put together that actually worked.  Mostly.  See how my large caps are bent over in the picture.  Turns out this caused me a world of pain that I wasn't able to correct until today.  You see, when I bent the pins on the caps like that, I accidentally damaged whatever internal connections there were.  Damaged, but not broken.  So they worked... sometimes.  It was very frustrating, and I blew up 2 different voltmeters while trying to figure out just why I was having the trouble I was having.  Anyway, here's the final circuit in place.

I'm skipping over a huge part of this, because its less interesting because it worked the first time, every time.  The method for sensing a power down was super easy.  I picked of a "voltage regulator".  The specific regulator I got takes a 5-15 volt signal, and converts it to 3.3V, which is the logic level of the raspberry pi GPIO pins.  I connected the regulator straight to the original 12V supply (before my 5V converter), put a large (1000ohm) resister in series with the 3.3v signal, and connected it to a GPIO pin.  Then, I have my fan control script watch that pin, and whenever it goes from on to off, we initiate a shutdown.  Here's how it looks.  You can also see my fan control transistor in this picture too.

That was the easy part.  Here's how the whole thing looks now that its done.

So, it was quite a struggle, but I persevered, and I was all done.  Or so I had thought.  After I tested it, I put everything back together, hooked it up, and it worked great.  All the emulators running on the Pi ran without a hitch, and when I killed the power I could verify from the activity lights that it kept alive through the shutdown.

And then one day, it didn't.  I was getting some software set up, and I noticed that the standard raspberry pi low power warning was constantly popping up on my screen.  "Huh", I thought.  I didn't think my little circuit would pull that much current, especially not after it was charged up.  I turned off the power and checked the activity lights.  Dead.  Nothing.  No shutdown.  Something was broken.  Turns out, I broke the capacitors when I bent the connectors, because the bend point was actually just inside the cap, instead of outside.  Oops, rookie mistake.  I bought two more this week, and just got finished installing them.  They charge up faster, discharge consistently, and no more low power indicator.  Thank goodness.

I hope you all enjoyed this little technical journey.  Here's some pictures of the finished product again.

If you want to see ALL the pictures from this build, I've linked them on a google plus album here.  Thanks for following!