• nogrub@lemmy.world
    link
    fedilink
    arrow-up
    1
    ·
    1 year ago

    it’s almost like computers are not that accurate when calculating floating point numbers

    • afraid_of_zombies@lemmy.world
      link
      fedilink
      arrow-up
      1
      ·
      1 year ago

      About a year ago I ended up with a floating point value that was something like 1.0000000000078 when it should have been 1. Tore my hair out for hours trying to get the piece of crap embedded vendor locked device to just make it 1.