# Questions tagged [floating-point]

Floating point numbers are approximations of real numbers that can represent larger ranges than integers but use the same amount of memory, at the cost of lower precision. If your question is about small arithmetic errors (e.g. why does 0.2 + 0.1 equal 0.300000001?) or decimal conversion errors, please read the "info" page linked below before posting.

11,315 questions
10 views

### Rounding off a float value without decimal points in python 2

Problem is to round-off the resulting value without any decimals after summation of percentage values using Python 2. provided 3 inputs Input 1 (meal cost) : 10.25 Input 2 (tip percentage) : 17 ...
75 views

### How to produce infinity in Haskell?

Similarily to How to produce a NaN in Haskell ... In C, there is the INFINITY macro, defined by math.h. Again, in http://hackage.haskell.org/package/ClassyPrelude-0.1/docs/Prelude-Math.html I can ...
54 views

### How to produce a NaN in Haskell [duplicate]

In C, math.h defines nan. Is there a similar facility in Haskell whose only purpose is to produce a NaN? Or is something like 0/0 or acos(2) my only choice? http://hackage.haskell.org/package/...
34 views

### PHP Floating point precision inconsistent [duplicate]

0.01 === 0.01 // true 1/100 === 0.01 // true 3.6 / 360 === 0.01 // true 0.001 * 10 === 0.01 // true 0.0049 + 0.0051 === 0.01 // true 0.005 + (0.0025 * 2) === 0.01 // true 36 - 35.99 === 0.01 // false ...
23 views

### Specific inputs of experiment return parser error

#!/bin/bash for tracelength in 50 100 150 200 250 300 350 400 450 500 ; do step=0.2 short=0 long=1 for i in {1..4}; do ratio=0 for j in {1..4}; do declare -...
50 views

### Why does rounding the floating-point number 1.4999999999999999 produce 2?

I've been reading a book Write Great code - Understanding the Machine. In the section about rounding it says: Numbers should be rounded to the smallest bigger number if the decimal bit value is ...
64 views

### Multiply *float32 and int value

I have 2 fields that I have to multiply. One is a *float32 field and another is an int field. How can I multiply these fields? var totalPrice *float32 var price *float32 var volume int this code ...
62 views

### What is the max real number in computer I can get TRUE as a result of is.finite in R?

(reproducible example is added) The maximum positive double (floating point) number in computer in my 64-bit OS and 32-bit R is: .Machine\$double.xmax # 1.797693e+308 So, the following is TRUE: is....
69 views

### Mathematically correct result from Math.sqrt for int arguments?

Given an int x, is (int)Math.sqrt(x) guaranteed to return the mathematically correct result, truncated to an int (i.e., the exact mathematical value of floor(sqrt(x))) for all non-negative x?
27 views

### Checking if a string is a float or integer [duplicate]

I'm creating a function that checks if a string is either a float or negative but my code does not account for decimal points or "-" for negative numbers. def is_num(s): if s.isdigit() == False: ...
26 views

### How can I pass the selective part of fraction number to string for displaying?

I would like to display the first 4 digit of fraction number and pass it to string to display in title of my plot. I checked this post but I couldn't find an elegant way. I've tried following code as ...
42 views

### Different floating point matrix multiplication result on two GPUs

I am running a MPSMatrixMultiplication from Metal Performance Shader on two separate GPUs, one is internal GPU and one is AMD GPU. The matrices are exactly the same. But the problem is that the ...
38 views

### python converting strings to floats resulting in error

I'm trying to convert strings from a txt file with charset: us-ascii. to np.float64. The data are just decimal numbers (positive and negative). with open(path) as fp: data=fp.readlines() ...
96 views

### Why float division is faster than integer division in c++?

Consider the following code snippet in C++ :(visual studio 2015) First Block const int size = 500000000; int sum =0; int *num1 = new int[size];//initialized between 1-250 int *num2 = new int[size];//...
55 views

### How to strip and convert string to float (decimals)?

I am trying to convert a string which is like " 8,191.55 MB " to " 8191.55" so that I can divide it by 1024 to get it in GigaBytes. Code I am trying gives me " 819155" instead of "8191.55" Code : ...
57 views

### How to automatically eliminate double-precision math from generated code?

In our embedded project, targeted for the ARM Cortex M* cores, we are using the ARM clang-based toolchain (v6). We use float type for our floating-point math. As we try to minimize footprint, we found ...
30 views

### Is there a generic standard way to store floats in a std::set and/or a std::unordered_set?

We know that NaN != NaN for IEEE floats. As a result, a number of "obvious" operations on floating pint numbers have a hidden gotchas in which NaNs in the data can mess things up terribly. For example:...
28 views

### Floating point not iterable python

I wanted to store floating point values in a list iteratively but it says floating point not iterable even if the iteration value was integer n=int(input('Please enter value of N: ')); for i in range(...
76 views

### C#: Wrong result when converting expression with floats to int

Take the following code: float a = 100.0f; float b = 0.05f; If I want to convert the result of division between a and b to an integer, I can do it with this code: float c = a / (b * 1000.0f); // c =...
33 views

### Issues with converting from floating point little-endian to big-endian and back again

I am trying to write a save parser which saves floating point numbers in little endian, however, Java is in big-endian, so i need to convert the FP and back again when writing but this does cause some ...
57 views

### Double constants usage with suffix 'f' [duplicate]

double f_64 = 3.35f; double f1_64 = 3.35; What is the effect if suffix 'f' is used? With the online FPU compilers, hexadecimal results are as below With suffix f - 0x400ACCCCC0000000, without ...
55 views

### Will division result of 2 numbers with the same ratio always be the same?

a is an integer. b is also an integer, but declared as double. c = a/b, and c is also declared as double. And, there are a2, b2, c2, with similar rules as a, b, c. Also, in term of decimal algebra, ...
35 views

### Where can I find some free benchmarks to evaluate a MCU? [closed]

At present, I'm designing a soft processor with single-precision floating point unit (FPU). I am going to put my soft core into an FPGA and do some performance evaluation. The benchmarks should ...
24 views

### Read only the numbers in a specified line in a text file

I am trying to read two floats from lines 18 and 19 of a text file and find their average. Please only look at the section of code separated by #########. The file has a mixture of words and numbers, ...
24 views

### How do I avoid floating point rounding errors in WebVTT?

I'm trying to build a WebVTT metronome and I'm specifying the cues with millisecond precision for both the start and end times. Unfortunately, it is showing up inconsistently in different browsers. ...
21 views

### My app is work in my emulator but on real device the error Error inflating class android.support.design.widget.BottomNavigationView" is appear

my app is run in emulator good but when i try it with real device the error appear and the app is stopped please can any one help me I think the problem is with api but I don't know helpppp pleaseeee ...
99 views

### Precision of floating-point data types in C++

Why don't the precision of floating-point data types grow proportional to its size? E.g.: std::cout << sizeof(float) << "\n"; // this gives 4 on my machine "debian 64 bit" with "gcc 6.3....
52 views

### C float initialization result is unexpected

Visual Studio 2019 Community float variable is not what I initialize it to. I made it less precise in hopes of avoiding truncation. Is this not "controllable"? tried to force init to round in other ...
35 views

### Vector double-double floating point arithmetic

There exist workloads for which double precision floating point is not quite adequate, hence a need for quad precision. This is rarely supplied in hardware, so a workaround is to use double-double, ...
43 views

### Representation of double numbers

In an 8 bit representation, we know that the number 4 is stored as 00000100, and the number -4 is stored as 11111100. But how the number 4.6 is stored in a double?
91 views

### Recommended settings for floating-point contraction in C, C++, and Fortran

The C standard (and C++: [expr.7.1.6], and Fortran), states: The values of the floating operands and the results of floating expressions may be represented in greater precision and range than that ...
16 views

### Custom Binning function doesn't return Float data type

I've built a custom function that bins data based on a chosen min, max and bin size. It works as intended for integers, however when using float's that represent percentages, it won't return the ...
16 views

### Why when converting SQL Real to Numeric does the scale slightly increase?

I'm storing a value (0.15) as a Real datatype in a Quantity field in SQL. Just playing around, when I cast as numeric, there are some very slight changes to scale. I'm unsure why this occurs, and ...
15 views

### How to insert float numbers from mysqldump

I have sql dump file coming from FirebirdSQL. Looks same as exported from MySQL (MariaDB). In the said file there are table creation statements one of which are the following: CREATE TABLE VALIORS ( ....
35 views

### represent decimal numbers in javascript as “%g” formatting in C

I have a number generated as a finite decimal: var x = k * Math.pow(10,p) with k and p integers. Is there a simple way to convert it to an exact string representation? If I use implict string ...
69 views

### Is there a C++ function for converting single precision IBM floats to IEEE-754 floats? [duplicate]

I'm trying to read single precision floating point numbers from a binary (.segy) file on windows, using C++. These numbers follow the IBM floating point architecture, so I need to convert them into ...
100 views

### How to change the rounding mode for floating point operations in MATLAB?

I want to change the rounding mode for floating point operations in MATLAB. According to IEEE 754-2008, there are 5 strategies for rounding: round to nearest, ties to even round to nearest, ties away ...
33 views

### Replacement for ceil function for arbitrary ranges?

I have a couple of situations where I need to clamp a floating-point number to the upper or lower end of an arbitrary[1] range. Thus far, I have been using floor and ceil functions with pre- and post-...
52 views

### precision between float and double in C

I understand there are several topics same as mine, but I still don't really get it, so I'm expecting someone could explain this in a more simple but explicit way for me instead of pasting other ...
54 views

### identifying and assigning +NaN and -NaN in C

I have to get input in float as +NaN or -NaN and convert to another artificial floating form called tinyfp (1 sign bit, 3 exponent bits, 4 fraction bits) for one question, or get input in the tinyfp ...
101 views

### Why the cos function in math.h faster than x86 fcos instruction

The cos() in math.h run faster than the x86 asm fcos. The following code is compare between the x86 fcos and the cos() in math.h. In this code, 1000000 times asm fcos cost 150ms; 1000000 times cos() ...
40 views

### How can I base-convert a large/high-precision floating-point number in PHP?

As generic of a question as this seems, I'm having a really hard time learning specifically about how to base-convert large high-precision float values in PHP using BCMath. I'm trying to base-...
206 views

### best practices on floating point precision in python

Earlier tonight, a friend of mine just handed me this cute problem. The problem says: Make a program in MATLAB to check whether a point is inside a triangle or not. Not to forget to check if the ...
43 views

### How to convert floating point to binary on the command line?

I'd like a simple way to convert a float string on the command line to a binary float. For example: \$ echo "1.23" > file.bin Except with the output as a 4 byte IEEE float I'd expect the result ...
56 views

### Correct way to use > and < comparison on floats in Java?

I am well aware that using the == operator on two floats does not work/is bad practise, however from what I can see, using the > and < operators on two floats should be valid. However, in my ...
31 views

### Odd `gawk` filtering of very small floating point number

gawk filters out very small positive number differently depending on threshold used, but all thresholds should retain the entry. Example input file, tmp: A 3.92e-373 B 5e-300 C 5e-20 D 5e-6 E 5e-...
34 views

### echo a variable with floating point value

Declare total=O for repeat in {1..100}; do executiontime=\$(gtime -f "%U" python3 main.py | tail -0) total=\$(echo "scale=2; \$total + \$executiontime" | bc) done echo "\$total/100" The code ...
23 views

### How to use random.uniform function in for Loop and store the values in a variable for each execution

I have 1000 rows * 100 columns dataset. wanted to generate Random number with the help of these data. specifically for a some columns. say 20 columns. have performed - random.randrange(min(dataset["...