Friday, July 29, 2011

The Great GPS Test

Many people out there wonder "Which GPS watch should I get? Garmin? Timex? The new Nike?" That question has been unanswered until now. I went and got all of the available watches with integrated GPS and tested them out with a series of runs. Here is the watch list:
  • Garmin Forerunner 305
  • Garmin Forerunner 405
  • Garmin Forerunner 210
  • Garmin Forerunner 310XT
  • Garmin Forerunner 610
  • Timex Ironman GPS
  • Nike+ Sportwatch GPS
  • Nike+ Sportband (just for fun)
To get a good feel of how all of them perform I took them on three runs:
  • Track workout consisting of 6x800m sprints with 400m jog rest
  • Trail run consisting of a 2.2 mile loop 3 times
  • Road run of 9.45 miles around White Rock Lake
This is the fairest way to test all the watches because it gets them on a variety of terrain, varying distances, and tests the signal in both covered and uncovered areas. Before I go in depth on the results of the tests, let me break down the watches:
Run #1: Track
I took all 8 watches to the track for some 800m repeats. I stayed in lane one on the inside of the lane the whole time so I wouldn't stray from exactly .25 miles each lap. With the 400m recovery after each repeat, the total mileage for the run was 4.50 miles. Here are the results from the workout:
So the total workout took me 29:32 for the 6 800's and the recovery jog. I didn't do splits for the run because that would have been way too hard to press the lap button on all those watches. But as you can see, the Nike+ GPS is the clear winner for sure. It was spot on for the whole workout. I am very impressed with Nike on the track. Both of their watches ranked in the top 3. As long as you have the Sportband calibrated properly, it can hang in there with the expensive watches. I threw in the column for displayed pace to show how much of a difference .05 of a mile makes even in a short run like this one. So from the best of the watches closing in at .01 miles off to the worst watch at .27 miles off, GPS has a long way to go for track workouts. Kings of the Track:
  1. Nike+ Sportwatch GPS
  2. Garmin Forerunner 305
  3. Timex Ironman GPS
Interestingly enough, even though I ran with 5 Garmins, only one made the top three. Another surprise is that the 305 and the Timex both measure less than the actual distance while all the others (disregarding the Sportband) measure farther.
The reason GPS watches have such a hard time with distance on the track is the constant turning. Because the watch gets signals from the satellites every 3-6 seconds, the watch connects the points linearly instead of on a curve. When you think about it logically, however, the GPS path should be shorter instead of longer. It could be that the watch is trying to accommodate for the curve and adding extra distance, or it could be that people actually don't run on the 400m section of the track, they could waver a little bit and go into other lanes or not take the innermost section of the track.


Run #2: Trail
Depending on the trail, GPS watches have never really been the best in tracking distance. Elevation changes, switchbacks, and tree cover all make pinpointing a location very hard. Often you will see watches that are way off in distance both above and below the actual distance. So I hit the trail on a 2.2 mile loop that has tons of elevation change, loads of switchbacks, and about 75% tree cover. I did the loop three times to make sure I could really test to see how the watches were doing and see if they were consistent throughout the multiple laps. Here are the results from test #2:
I expected there to be a big difference amongst the watches, but I had no idea it would be this bad. The closest to being the actual distance was the 310XT and it really was the stand out in the group. The probable reason for the watches performing so poorly was the tree cover. The signal gets blocked temporarily and then the watch has to guess where you went, which means you could lose an entire switchback or a long stretch of random trail. The only watch to display a farther distance than actual was the Sportband, and that makes perfect sense. The Sportband goes based off number of footstrikes, not GPS location. Since you take smaller steps when you are trail running, and your watch is calibrated for your typical longer stride, it thinks you are going further than you actually are. Kings of the Trail:
Aerial view of the trail run
  1. Garmin Forerunner 310XT
  2. Garmin Forerunner 610
  3. Garmin Forerunner 305/Nike+ Sportband
Looks like Garmin dominated this run. The Nike+ GPS and Timex GPS really were nowhere to be seen. Both had very poor performance pretty much from the beginning. For the second run in a row, the "old school" Garmin 305 has placed in the top 3, which for me is surprising. But without a doubt, none of the watches could even compare to the 310XT, so trail runners, go pick it up and hit the trails.


Run #3: Road 
Your typical runner is going to be running primarily on the roads, which is why this test was so important. I went and did a lap around a lake, which is typically 9.3 miles, but I took a wrong turn and it added .15 miles. So I did a 9.45 mile run on the road in a completely open area with no tree cover. I expected to get the most accurate results here and the tightest span from worst to best. Here are the results:
 Between the GPS watches, the span wasn't too bad. The worst was only .16 off, which isn't horrible over the course of 9.45 miles. The Sportband obviously wasn't calibrated correctly for this run, so it performed the worst of all of them. The 610 and 310XT were exactly correct which is very impressive by both watches! Looking at these results though, you can obviously tell Garmin is king of the road. Official Kings:
  1. Garmin Forerunner 610/310XT
  2. Garmin Forerunner 210
  3. Garmin Forerunner 405
This was the first test that the 305 wasn't in the top 3, but it definitely was right there with them. This test is the most fair of all the tests since it was run in open air with nothing to interfere with the signal.

Overall Results
 When we go based just off the average percent error of all 8 watches, here is the order of most to least accurate in varying conditions:
  1. Garmin Forerunner 310XT
  2. Garmin Forerunner 610
  3. Garmin Forerunner 305
  4. Garmin Forerunner 405
  5. Garmin Forerunner 210
  6. Nike+ Sportwatch GPS
  7. Timex Ironman GPS
  8. Nike+ Sportband
In the end, all Garmin's seemed to be the best. But the hands down winner for most accurate is the 310XT with an average of .13 miles off for 3 runs totaling 20.05 miles. The least accurate GPS watch was the Timex Ironman GPS with an average of .44 miles off. 
Overall, very interesting results coming from these tests, I never would have expected what came about. I will leave you with my GPS watch accolades:
  • Best GPS Watch Overall: Garmin Forerunner 310XT
  • Best GPS Watch for Track Running: Nike+ Sportwatch GPS
  • Best GPS Watch for Trail Running: Garmin Forerunner 310XT
  • Best GPS Watch for Road Running: Garmin Forerunner 310XT and Garmin Forerunner 610
  • Best Value GPS Watch Based on Accuracy: Garmin Forerunner 305
Well that wraps that up! I hope you found this review helpful and enjoyed The Life of a Runner. 

14 comments:

  1. You realize your distance you ran on the track (6*800+6*400) is 7200 meters? A 400 meter lap is not a quarter mile, it's a little less. So the actual distance was 4.47 miles.

    ReplyDelete
  2. I've heard that a track is measured at the outside of the first lane, not the inside. (not sure on this, but figured i'd throw it out there :)

    ReplyDelete
  3. Actually a track is measured 20 cm (roughly 10 in) out from the inside line. This is roughly where they figure the average runner will run. Close to the inside edge but not all the way on the inside.

    Heathur is correct the distance on the track should be 4.47 mi.

    ReplyDelete
  4. How did you determine the "actual" distance on the trail and road? Unless you used a calibrated wheel your "actual" is likely no more accurate than your GPS measurements. Please don't tell me you drove the road route with your car or mapped it online. Don't get me wrong, this was still a cool comparison of the different models since they all traveled the same route but the only test that says much about absolute accuracy was the track. Also withy the Garmin 405, did you enable 1sec recording by pairing it with an Ant+ power meter first? And did you set the 305 and 610 for 1sec recording?

    ReplyDelete
  5. It would be interesting to see how repeatable the results are.

    ReplyDelete
  6. Did you use the shoe pod w/ the Nike+ GPS watch as well or just the satellite?

    ReplyDelete
  7. Nice job on the comparison tests. I've run with the Garmin 305 for years. It's very accurate and a terrific value for the money.

    ReplyDelete
  8. If you tested them as you are wearing them in the picture, the units on your left arm traveled a shorter distance on the track than those on your right. That figure could be significant, considering the width of your body and the number of laps run.

    For the trail loop test, it is not valid so stated how far each unit was "off", since you have no accurately measured and strictly run course. Even if you have a wheeled measurement of the course, unless you ran in exactly the same foot step on every loop, you are the variable in exactly how far the trail measures.

    Finally, the same problem applies to your road loop. It is interesting to compare the GPS units to each other, but which one is giving you the proper distance? You have no way of knowing unless all of those units were strapped to a measuring wheel to compare the measured distance to the GPS reported distance.

    ReplyDelete
  9. I bet you got lots of stares running around with all those watches.  I wonder how an iPhone using the Runkeeper app would stack up... I have a garmin 405 but i usually use the iPhone to track my runs cause it's with me all the time.

    ReplyDelete
  10. Foregoing criticism aside, this was a great test -- thanks for doing it!

    ReplyDelete
  11. The probable reason for the watches performing so poorly was the tree cover. The signal gets blocked temporarily and then the watch has to guess where you went. Buy watches

    ReplyDelete
  12. Great test and great post. A lot of the people who are commenting are just trying to show how smart they are - but did any of them test all these different gps watches and blog about the results?

    ReplyDelete
  13. I went and got all of the available watches with integrated GPS and ... gpsforerunner.blogspot.de

    ReplyDelete