24

I'm a member of my high school's robotics club, and am responsible for programming the robot. One suggestion I keep hearing from various adults is that I should be writing unit tests to help validate my code. The code base is getting a bit big, and I do agree that unit tests would be really helpful in helping me catch bugs quicker.

However, I'm not entirely sure how I could accomplish this. To the best of my knowledge, unit testing is done by taking a function (or a subsystem of the code) and feeding it a set of input to make sure it comes out with the same output each time. The code that I currently have doesn't do any heavy data crunching, but rather directly manipulates the hardware components on the robot. Most of the complexity comes from making sure that the electronics are sound, that the code at the moment matches the actual hardware on the robot, etc. Often times, I can only see if there's a problem by loading the code to the robot itself, and attempting to run it.

By extension, how can unit tests be written for code meant to operate any mechanical device? It seems to me that you can only catch errors by physically observing the operation of the machine.

Or am I just misunderstanding how unit tests should work?

(If it matters, here's the code, it's written in C++, and I'm participating in FRC)

Michael0x2a
  • 1,119
  • 1
  • 7
  • 23

4 Answers4

22

You will need to mock the hardware layer to do this testing. The idea is that instead of your code talking to the actual hardware, you talk to a fake version of the hardware that you can control and then use to verify that your code is working correctly.

Unfortunately there are some problems you have to overcome:

  • Mocking things in relatively low-level languages is more difficult (and thus, a lot more work)
  • Mocking hardware-level stuff is more difficult (and thus, a lot more work)

Also, most of the value of automated, unit testing comes from being able to run your tests at any time to catch regression bugs for long periods of time after you write the original code. With these sorts of competitions, your code won't be used at all after the contest is over, so you won't really get most of the value out of the tests. And considering how difficult it would be to add tests to your particular case, it might be better use of your time to just do manual testing and focus on features for the contest.

Oleksi
  • 11,894
  • 2
  • 54
  • 54
  • 1
    Nice answer. Especially the bit about the fact that the code won't be used after the competition, and that the great benefit of automated unit tests comes well after the tests have been written. You might consider automating some of your tests in the case where you find yourself running the same test over and over; but until this happens, there's not much point. – Dawood ibn Kareem Apr 25 '12 at 08:02
  • No need to Mock the hardware at all. If the robot has logging, run the test program and observe the logs. The final test needs observation "turn left" in the log should correspond tpo the robot turning left.

    You will need to write a test harness to mock the input devices - hook the input device code as close to the hardware layer as possible

     – mattnz Apr 25 '12 at 08:23
  • 4
    @DavidWallace As a little food for thought, when using TDD/BDD, the benefits of unit testing occur immediately. In the first place by allowing the code to be confidently refactored immediately, and secondly by encouraging implementation to be limited to the minimum implementation needed to satisfy the tests. – S.Robins Apr 25 '12 at 08:26
  • You're absolutely right, @SRobins, but the OP seems to be asking whether they should unit test their existing code. – Dawood ibn Kareem Apr 25 '12 at 08:28
  • 4
    @mattnz bad idea, and I know from experience. What if the code under test fails very very hard and the robotarm crashes against the wall, ruining a piece of xxxx$ hardware??? – stijn Apr 25 '12 at 09:14
  • @DavidWallace Just to be picky, OP has asked How, not Why, which is why I wrote my answer as I did, whereas my comment was more to address your statement that ...the great benefit of automated unit tests comes well after the tests have been written. I agree with your implication however that it is questionable whether there is value writing tests after implementation given the relative effort required, however I'd assume there is work to do at the hardware layer, in which case it would make sense to implement some unit testing there in order to better support and debug calls to hardware. – S.Robins Apr 25 '12 at 13:34
  • I wonder if googlemock might help here. The question Is Google Mock a good mocking framework? suggests that it might be. – Mark Booth Apr 25 '12 at 14:20
  • @DavidWallace: Hes a member of a high school robitics club - I doubt they have anything powerful enough to do that much damage.....Most of these clubs use robots from kits, the worst you do is break a $10 part. You do not "NEED" to mock the hardware layer. Many robots have been built successfully without one. – mattnz Apr 25 '12 at 21:37
  • The competition is over for this year, so it's not as crucial that I create unit tests for the code. However, I thought it might be a good idea to learn how to do so now, so that I can get more experience now rather then in the middle of the build season next year. But I definitely get the point that it might not be worth it, especially since we won't really be reusing after this summer. – Michael0x2a Apr 25 '12 at 22:24
  • 2
    @mattnz: Our robots are about 2 feet by 3 feet by 4 feet, and weigh about 150 pounds. The kit/registration costs 5 thousand dollars each year, and we usually fundraise another 5 to 10k to purchase additional parts. The worst case scenario would probably cost over $10 ;) – Michael0x2a Apr 25 '12 at 22:28
10

I can think of a couple of things that you will need to consider. The first is to make your hardware access layer as thin as you can, even if that means creating a basic wrapper-type of layer for it. This offers you a couple of advantages. The first is that it allows you to isolate the hardware-specific behaviours of your code from the hardware access itself, which means you can test everything down to the very bottom of your hardware comms without needing to access the hardware itself.

For example, if you need to design an I2C-based signalling protocol, you can test your code is generating the correct I2C signals without needing to hook the hardware into your tests.

For calls to the actual hardware, you can test that they are behaving correctly by mocking your hardware layer, and this is where keeping a very thin hardware layer really pays off, because you can reduce your mock to needing only handle the minimum functions required to actually address the hardware, but you don't necessarily need to test the individual signals themselves, as all of your signalling should have been testable at a higher level. This means then that you use your mock to check that calls are made to specific hardware methods that cause your signals to be sent to hardware. If you need to poll your hardware, then your mock needs to be able to trigger events or methods in your code only, because again, your return signalling should be handled in a higher layer.

This basically fits with what Oleksi said in his answer, in that it is usually more work to mock hardware-level stuff, however it isn't so difficult if you keep to the leanest possible minimal-code/call layer you can make for the hardware.

When you have code that passes all of its tests, you will still need to run through a series of manual tests in order to be sure you hoked everything up correctly in your hardware layer.

The other thing that comes to mind aside from the mocking and the layering, is to use a test-first development practice. Essentially, you code your requirements as test criteria, and you base your implementation off your tests. This will help you to ensure you keep your implementation code to a minimum, while ensuring all of your test cases are driving your development efforts. By not wasting too much time on other potentially non-critical code that you might be tempted to do "just because", test first helps you to stay focused, and will make it easier to change your code as you debug, as will the use of your unit tests and mocks. Debugging software bugs through the hardware is notoriously complicated and sucks up large amounts of your time that you would find better spent on other tasks.

Community
  • 1
S.Robins
  • 11,485
  • 3
  • 37
  • 52
2

I can tell you how they do it on Flight Simulators

First, you're only going to get half the answer if you ask this question to only programmers so you should probably cross post this on http://electronics.stackexchange.com while you're at it.

I haven't done any work with robots but I did spend 5 years doing hardware on flight simulators so I can tell you how their architecture works.

The hardware layer is dumb

It contains a basic interface where you can adjust simple input/output values and set interpolation breakpoints for analog signals. When you're working with 'fresh' hardware, everything will work as expected with little or no calibration but over time the parts will undergo mechanical wear and need to be adjusted.

The calibration is a simple table that contains sectioned ranged between the min/max values. To measure input on these, a servo is typically used (ex a linear potentiometer, transducer, accelerometers, etc). Or in the case of instrumentation, you just judge accuracy visually and adjust until calibrated.

The software layer is the opposite

Everything is complex and interconnected so it's important to isolate some variables to test the functionality. There's no need to give yourself a headache thinking up scenarios because it's much easier to run some realistic scenarios where you can collect data. When you run the tests, you're basically measuring the stored data against the current output.

On a flight simulator this is referred to as a QTG (Qualification Test Guide). At its core it plots the data on a 2D grid where one dimension is time and the other is the output.

Believe it or not, that's the essence of how they develop the models. A real plane is equipped with a ton of sensors and flown around doing controlled scenarios. Because all of the controls can be driven without human interaction, the tests are run (ie the sim flies itself) by the computer and the data is compared.

Even though robotics are created on a much different scale the principles are the same. The traditional approach is to completely sever the hardware and software layers so both can be tested individually. Hardware input is collected through servos, and set through an independent interface. Sofware input can be set/read by independently measuring and comparing the signaling that would otherwise go to the hardware and plotting it against known 'good' data.

The tests themselves don't necessarily need to be complex as long as the results are predictable, measurable, and reproducible.

Evan Plaice
  • 5,755
1

As previously said, mock and stub out the hardware parts. As an example, if you have an interface to the robot, you can inherit from that interface, and then make simple stub implementations of it. Then you can have test that the stub implementation have been called as expected. If that is expected functions or expected parameters.

martiert
  • 359
  • 1
  • 6