0

I have a system that's running a bunch of calculations and for some reason, everytime 0.07 shows up it appears as 0.07000000000000001

I'm well aware that floats are approximations, however I'm writing code as such:

number = '%.2f' % ( num1 - num2 )
number.to_f

Despite this, 0.07000000000000001 keeps showing up in my code and ultimately causes array sort to fail:

sort! { |a,b| b <=> a }      <-- b equals 0.07000000000000001 and a equals 0.08

It raises an error saying it can't compare these two numbers. Any thoughts on how to fix this?

Update

So it appears as though the original value was 0.07351923338974675 at which point I call .round(2). I then store it in Redis temporarily and then pull it back out. When it goes to run a calculation it ends up showing up as 0.07000000000000001. It sounds like the integer comparison may be best here, unless there's an alternative solution.

Improve this question
3
  • 1
    Integers and floats can be compared. One of them must be something else. You should also show what num1 - num2 is. Commented Feb 1, 2013 at 22:43
  • 2
    What is the error that is being raised? If the values truly are what you say they are, I can't see any reason why they shouldn't be comparable, even with the drift in the ~0.07 number. Also, as @Cthulhu mentions, round() is probably a better way to remove floating point drift. Commented Feb 1, 2013 at 22:46
  • It's going to take me 10 min or so to test ... will have an update. Trying .round(2) first Commented Feb 1, 2013 at 22:52

1 Answer 1

Reset to default
2

All numbers are ordered, but a String cannot do ordered comparisons against numbers, only equality comparisons to see if they are the same object. (A comparison that will always return false, obviously.)

I imagine this is the type of error you are seeing, something like: ArgumentError: comparison of String with Float failed.

The issue with low-order bits in Float objects could possibly interfere with your sort, though raising an actual exception seems highly unlikely. Like most decimal string fractions, 0.07 does not have an exact representation in the underlying Float format. It is conceivable that different ways of getting to 0.07 (literal constant vs. result of calculation) could result in a every-so-slightly different value.

But also be aware that if you attempt to print out more fraction digits than the quantity really has, you will see various remainders of the last bits that were there. These digits can be non-zero but they don't tell you anything useful about your number.

The safest thing to do is to scale to integral values, call #to_i, and compare the integers.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

I think you nailed it ... the .0700000000001 was being converted into a string. Do you know why that would be the case?
In fact, sawa and mdunsmuir nailed it about an hour ago. If you had given the actual error message it would probably have been obvious immediately.
I can't get the error as Sidekiq isn't reporting it :( It only provides the following: {"retry"=>false, "queue"=>"default", "class"=>"LeaderboardMonitor", "args"=>[], "jid"=>"ff70ecfc6a0437bd6f3a84f8"} 2013-02-01T23:05:15+00:00 app[worker.1]: 2013-02-01T23:05:15Z 2 TID-1h2weo WARN: comparison of LeaderboardEntry with LeaderboardEntry failed 2013-02-01T23:05:15+00:00 app[worker.1]: 2013-02-01T23:05:15Z 2 TID-1h2weo WARN: /app/app/models/leaderboard_data.rb:51:in sort!' 2013-02-01T23:05:15+00:00 app[worker.1]: /app/app/models/leaderboard_data.rb:51:in sort_by'
Isn't so obvious is it? It's funny when I see people like you on here that effectively insult the person asking the question ...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.