I’ve been reading about stories of GPS inaccuracies on Strava, and this post by Suffolk cyclist got me thinking. How could two riders, ascending a hill together end up recording quite different times?
I decided that some more experimentation was required. Evidence is a wonderful thing, after all, except if you’re a creationist.
Hypothesis:
There may be variation in the way different GPS units record tracks,which results in spurious timings when the data is uploaded to sites such as Strava, which allows the comparison of specific segments of a route. This would essentially undermine the whole principle of Strava leaderboards and make the coveted KOMs rather meaningless.
Aims:
To determine if there is consistency between different GPS tracking devices.
To determine the reliability of segment timings as calculated by Strava.
Apparatus:
One Sunday club run with the fabulous Rhyl Cycling Club.
4 GPS equipped riders, as follows:
 |
Paul, iPhone 4S, strava app. |
| Tim, Garmin 500 |  |
 |
Craig, Strava app on HTC Sensation |
| Me, Garmin edge 800, plus (for this test only) android app MyTracks running on Samsung Galaxy S2 |
 |
A nice hill called Glascoed road, which is exactly 2 miles long, and gains 405ft, average gradient 3.7%. The strava segment is called Craig’s Favourite.
Yes, named after the Craig in our group, as he doesn’t like that particular hill very much.
Method:
For the test hill, the gps riders stayed together in a tight group, so that their segment times should all be within a few seconds of each other.
One rider (myself) ran two GPS logs – one from a Garmin 800, and one from an android phone app.
The data was uploaded to Strava and compared.
Results:
iPhone 4S (running Strava app)
Strangely, although the ride did register many of the segments on the route, this particular segment did not register. On closer inspection, the actual track is all over the place, and I thought the issue was so significant that it deserved its own post.
HTC Sensation (running Strava app)
This phone crashed at some point in the day, and the data was lost.
Galaxy S2 (running Google App MyTracks)
This phone/app combo produced a very accurate track which matched the Garmin Edge 800 exactly.
Garmin
The two Garmin devices in the test both performed impeccably, creating accurate tracks.
Here is the data we did manage to record and review.
| GPS device | Time | Rider |
| Garmin 500 | 13:06 | Tim |
| Garmin 800 | 13:09 | Alan |
| MyTracks app on Galaxy S2 | 13:09 | Alan |
| Strava app on HTC Sensation | n/a | Craig |
| Strava app on iPhone | n/a | Paul |
Paul contacted Strava about the missing segment, and somehow they tweaked it so that it did register, 36 hours later. The time it gave was 13:06, the same as Tim and his Garmin 500. Paul and I did the exact same ride as we live on the same street. Over 58 miles, it was a surprise to discover that the elevation data were very close – only 31 metres difference, in over 1200m of climbing.
I’ve left the iPhone timing data out of the table because it shouldn’t be necessary to have to email Strava to put your data right. The quality of the iPhone track in this experiment was very poor. See the screenshots.
The Garmin 800 and MyTracks andoid app which were on the same rider came out with identical scores.
The Garmin 500 recorded 3 seconds faster, as did the adjusted iPhone data, but there may have been genuine slight variation as it is impossible to know exactly where the segment begins and ends. A more scientific approach would be for one rider to carry all the GPS units. But this was a sunday club ride, not a laboratory controlled experiment, so take it with a pinch of salt.
Conclusion
We didn’t witness any wild variation in timings, as others have reported. It would appear that there is consistency between the Garmin 500 and 800, and it was very reassuring to see the Garmin 800 and the MyTracks app (both carried by the same rider) produce exactly the same times for the segment.
It would be fair to say, from the evidence gathered, that the quality of the iPhone track was inconsistent. It was so poor at times that the recorded track departed significantly from the actual road, which caused the segment to not be recognised by Strava. In contrast, the Garmin tracks from the 500 and the 800 were consistently following the road with no deviation at any time.
I wonder how many iPhone users have stolen KOMs from people simply because their hardware does not track adequately?
Good bit of research. A couple more things that would be worth investigating further:
1. quality of tracking in amongst trees (on the road). That segments looks like it is right out in the open making it fairly easier for the gps units.
2. Also try it in amongst some steep sided dales where satellites might get hidden by the hills.
Those are where I see the most discrepancies.
RaceShape gives a good comparison tool for the timings. See this segment for us Lowe brothers (4th & 5th at time of writing): http://raceshape.com/strava-segments/668080 We were riding next to each other the whole way up, me recording on my 305 and Tones on his Android HTC. Lots of fluctuations although only to a couple of seconds, but even so, that could make all the difference. But if you compare the elevation graphs in strava (Tony’s: http://app.strava.com/rides/10933852#192020139) then you can see the android’s recording is really jittery compared to the Garmin. I wonder if the iPhone’s elevation is worse again?
How about comparing the elevation graphs of your test recordings as well?
One other thing, what do you reckon is the minimum length of a segment for the timings to be reliable (given accurate gps recordings)? The gpx files can have a number of seconds between recorded points, particularly on straight roads, so when Strava adds you to a Segment it seems to use the closest point to the start end, even if that might be a few seconds out.
A good example is this sprint:
KOM: http://app.strava.com/rides/6344524#120283818 – this rider’s effort for the segment (blue line on map) goes from the junction to just over halfway to the A1 bridge. But look at this athlete’s effort http://app.strava.com/rides/11867463#204692309 and he’s gone almost twice the distance! (note: both using Garmin 705′s). More of an art than a science!
Seems like I am being childish given that its free. But I did have a KOM for a sprint. Suddenly someone I know rode 2 mph and 5 seconds faster. Got the same result as above. “His” segment ( which I created ) was shorter than mine by about a third.
Then today I rode with a friend with an app, I have a garmin 500. I toasted him by 100 m over the 3 / 10s course, and he ended up with the same time.
Its not just a GPS thing. I think strava is tweaking to find some similarity and has created some issues.
As an aside, my KOM has disappeared entirely, and a slower day has been ranked. I can’t even locate the other.
And of course there is an issue with the smoothing the watts. I get the explanation, but I do have a watt hub, so I know what my watts are. The new KOM averaged 36.1 mph on a 2.7% grade downhill, (soft) yet pulled only 65 w.
Weird stuff which takes away from the enjoyment of the program. Hopefully its being addressed. I might even pay for it if its wasn’t so bogus.
Malco, after commenting above I put together this page (http://veloviewer.com/NewLeaderboard.php) on my site that, in a fairly lazy way, compares the actual distance travelled and from that the average speed for each athlete for a segment and provides an alternative leaderboard based on that result. Doesn’t work on all segments due to technical reasons but I’d be interested to know the segment you refer to shows a significantly different result to what Strava shows. From what I hear Strava are reworking their segment matching algorithm to make the start/end points for a segment the same for everyone (rather than the closest recorded gps point) although there’s a good chance they won’t apply this to historical data.
You failed to mention what iPhone you were testing. I’ve used an iPhone 4S alongside and Garmin 800 and a Garmin 910XT and found little difference. The iPhone 3gs I used to use was poor, however.
It was a 4S.
I’ll amend the article, thanks.