# Questions tagged [floating-point]

Floating point numbers are approximations of real numbers that can represent larger ranges than integers but use the same amount of memory, at the cost of lower precision. If your question is about small arithmetic errors (e.g. why does 0.2 + 0.1 equal 0.300000001?) or decimal conversion errors, please read the "info" page linked below before posting.

**0**

votes

**0**answers

14 views

### Is this a Floating-Point Error or something else entirely?

Given the following R code:
i = 1.0
while(i > 0.03)
i=i-0.05
print(i)
The output is as follows:
[1] 0.95
[1] 0.9
...
..
.
[1] 0.1
[1] 0.05
[1] -3.191891e-16
Can anyone explain why the ...

**0**

votes

**0**answers

30 views

### Equality operation does not work for Pandas dataframe value in one or more cells but work for others

I am working with dataframes read from csv file using pandas. I wanted to do some equality operation to find the rows where the column is of a specific value. However, it seems that the equality ...

**0**

votes

**1**answer

55 views

### How to get data in an array in a pattern?

I am trying to get the x and y values out of a PointCloud2 data. This data is stored in an array in bytes of float32's. My question is, how can I get only the x and y data?
I will add a snippet of ...

**1**

vote

**1**answer

37 views

### How to convert HEX string to Float (Little Endian)

I just learned Python (3.x) and I am stuck with HEX String conversion to Float. I have this HEX String values:
'...

**0**

votes

**0**answers

28 views

### Checking and extracting data from a string

I've spent ages trying to edit similar solutions to work for my situation, but can't get it working, can someone please help?
I am reading from either cin or file, and I'd like to check if the first ...

**1**

vote

**4**answers

60 views

### When to use %d and %f in C?

I'm new to C programming, but decent in Java, my question is when do we use %d and %f? In what situation? For example, based on the code block given below, if I was asked to take (int)a*(float)y, do I ...

**0**

votes

**1**answer

43 views

### Why did IEEE754 choose 11 exponent bits for double aka binary64?

I have been reading and it seems the IEEE 754 defines a 64 bit float's (double) exponent as 11 bits. (https://en.wikipedia.org/wiki/Double-precision_floating-point_format)
My question is why?
A 64 ...

**3**

votes

**4**answers

65 views

### C: while loop using floating point never terminates [duplicate]

By executing this code:
float f = 1.0;
while (f != 0.0) {
f = f - 0.1;
printf("%.1f\n", f);
}
It is expected that it would run 10 times and stop, but what turns out is that it inevitably ...

**2**

votes

**1**answer

79 views

### How to detect differences in floating point behaviour across platforms

What checks can I perform to identify what differences they are in the floating point behaviour of two hardware platforms?
Verifying IEE-754 compliance or checking for known bugs may be sufficient (...

**-2**

votes

**1**answer

50 views

### what should i change in my function so it calculates real numbers?

i am trying to write a function that repeatedly adds 0.001 to 't' and then plugs it into 'y' until 't' reaches 0.3 however the numbers come out wrong, but i've noticed that if i change float to int ...

**2**

votes

**1**answer

51 views

### how to validate that a floating number is not “0”

I have an <Entry> control where the user is allowed to enter decimal numbers, for examenple ...
0,2
0,02
5,405
But I do not want to enter a "0" (as a decimal), as follows
0,0
...

**0**

votes

**1**answer

37 views

### Why does f(50) differ radically from f(50.0)? [duplicate]

I have defined a function in Python 2.7.15 which takes two inputs (b,v), but I have noticed that f(50,0.1) produces very different results to those of f(50.0,0.1). This is the function:
def f(b,v):
...

**0**

votes

**2**answers

57 views

### Convert FP32 to Bfloat16 in C++

How can I convert from float (1bit sign, 8bit exp, 23bit mantissa) to Bfloat16 (1bit sign, 8bit exp, 7bit mantissa) in C++?

**-1**

votes

**0**answers

34 views

### Performing floating-point calculations as fixed-point in Linux kernel code

I am trying to perform the following computation in Linux kernel code.
float rm_least_upper_bound;
rm_least_upper_bound = (task_count*((2^(1/task_count))-1));
Since kernel code does not normally ...

**0**

votes

**2**answers

57 views

### Bash float calculations from last line in a file

I have a file text.txt as follows:
A B C D E F G F
1.1 2 3 4 5 6 7 8
1.2 20 30 40 50 60 70 80
6.3 70.5 10 20.5 ...

**1**

vote

**1**answer

23 views

### IEEE 754 identity laws: Literal meaning and value-changing optimizations

The IEEE 754-2008 standard says for expressions like 0 + x and 1 * x that the identity property may be applied (§ 10.4):
Applying the identity property 1 × x when x is not a signaling NaN and the ...

**-3**

votes

**4**answers

127 views

### How a float number can store a number greater than 256? [closed]

I'm confused about the representation of floating points (IEEE_754-1985) spec. How can a float number store a number greater than 256 if only 8 bits (1 byte) is available to store the significand part?...

**0**

votes

**1**answer

29 views

### What should the maximum difference be between a float and a 16 bit calculation? [on hold]

Lets say I have a formula that calculates a value. I can calculate the value either with floating point arithmetic or with fixed point arithmetic (using 16 bit integers).
What should be the maximum ...

**0**

votes

**0**answers

18 views

### Weird behavior of python in accurate math [duplicate]

Look at this code in python:
a=2**.5
b=np.sqrt(1-a**2/2)
__main__:1: RuntimeWarning: invalid value encountered in sqrt
The actual result shall be 0, because a**2/2=1 and sqrt(1-1)=0
but python ...

**0**

votes

**0**answers

44 views

### Python performance: square roots and hot-loop operation strategy

In comparing math.sqrt(…) to the built-in binary operator, … ** 0.5 in a REPL using timeit.timeit(…), it looks as though the binary op has a slight edge over the math module function:
… although I ...

**0**

votes

**1**answer

53 views

### software floating point and x87 or sse disabled

I am trying to enable software floating point under gcc following suggestions in this question but I've hit a snag:
The -msoft-float flag causes:
/usr/include/c++/4.8.2/bits/basic_string.h: In ...

**1**

vote

**2**answers

72 views

### How to correctly floor the floating point pair sum

Let me have two floating point variables coming as function arguments:
float fun(float x, float y) {
// ...
}
I would like to calculate the floor of their sum. Is it possible to do it correctly ...

**0**

votes

**1**answer

26 views

### Check for integer value vs float value

I
My vba doesnt seem to quite work. I need it to multiply a value in column N by 2 if it is a decimal and do nothing for all other values.
currently it is
Sub multiply()
Dim i, j, k, l As Integer
...

**0**

votes

**2**answers

66 views

### Reading and writing non-standard floating point values

I'm working with a binary file (3d model file for an old video game) in C#. The file format isn't officially documented, but some of it has been reverse-engineered by the game's community.
I'm having ...

**0**

votes

**1**answer

17 views

### Pandas Float Formating -

I am a newbie to Python/Pandas and do not know how to present my dataframe.
I understand that the data may not need to be changed but rather how it is displayed.
I have a dataframe as follow :
...

**3**

votes

**2**answers

42 views

### What's a method that works exactly like Math.floorMod() but with floats instead of ints?

Here's a set of inputs and outputs for Math.floorMod(x, 5) as an example.
int x;
inputs:
x = -15 | -14 | -13 | -12 | -11
-10 | -09 | -08 | -07 | -06
-05 | -04 | -03 | -02 | -01
+00 | +01 ...

**0**

votes

**1**answer

16 views

### unable to execute java program [duplicate]

When I tried to compile the program, it compiled successfully. But when I executed the program, this is the output I recieved in command prompt
Exception in thread "main" java.lang....

**0**

votes

**0**answers

33 views

### Python: mysql double and float

In MySQL table column balance has a type DOUBLE without specifying precision. And I can't change column type to DECIMAL or DOUBLE with precision (I know that is epic fail to store money in DOUBLE ...

**0**

votes

**1**answer

20 views

### Shift bits of a float in MIPS without MTC1

I'm currently working on a school assignment that requires me to find the biased exponent of a single precision float without using the mtc1 command that comes with MIPS. I am wondering how it would ...

**0**

votes

**0**answers

20 views

### Getting wrong output when doing simple float point addition on turbo c [duplicate]

So, i wrote up a C program to accept a float number from user and then add it with 34.2 and display the output. When i enter 1 or 1.0 as the input. The output i get is 35.200001 or 35.200002. But i ...

**-1**

votes

**3**answers

70 views

### Putting value of a char array into a float variable

I want to put the value of an array into a float integer.
main(){
float a;
char array[4]="12.1";
a=atoi(array);
printf("%f",a);
}
When I uses this program, it gives 12.000000 as ...

**22**

votes

**12**answers

4k views

### Python if-else code style for reduced code for rounding floats

Is there any shorter, more legible code style to solve this problem?
I am trying to classify some float values into interregional folders.
def classify(value):
if value < -0.85 and value &...

**0**

votes

**1**answer

35 views

### How to store state flags in floating point numbers for GLSL / WebGL

I have recently learned about storing boolean flags in integers using bitmasks. I'm wondering how you can do this to store and retrieve boolean flags to/from JavaScript, and store and retrieve flags ...

**0**

votes

**2**answers

52 views

### Why does my double round my variables sometimes? [duplicate]

I really dont unterstand why, but my double array just sometimes round my variables, even though it shouldn't. The weird thing is, that it only does this sometimes, as you see in the picture. All of ...

**0**

votes

**0**answers

29 views

### Fortran: Floating-Point Arithmetic for really small number [duplicate]

I want to define two small number as my variables in fortran. I tested them in two ways, one is by using Kind library, and the other is use ISO_FORTRAN_ENV. Here is my program:
implicit none
...

**2**

votes

**1**answer

82 views

### Cuda programming cannot have the same computation accuracy compared with CPU program in terms of float type

I try to use the GPU to accelerate my program which computes L2 distance between two float array. In order to check the computation accuracy, I write both CUDA program and CPU program. However, I ...

**-3**

votes

**1**answer

55 views

### How float data type is stored in memory? [duplicate]

If we create one variable of data type float and assign any value to it then how this is stored in memory?
float var = 13.34;

**0**

votes

**0**answers

17 views

### Find FLOATS between FLOATS query with MySQLi PHP?

I'm working with latitude and longitude to get points that are closer to others points. I need to find the latitudes that may exist between 2 numbers in my DataBase, unfortunately I can't find -as an ...

**2**

votes

**1**answer

29 views

### Looking at user input to see if it is a float, always coming back as true

I trying to prompt the user fora number, and scan it to see if it is a float. Even when entering a number without a decimal it is coming back saying it is a float. The function should return 1 if the ...

**48**

votes

**3**answers

1k views

### Do std::min(0.0, 1.0) and std::max(0.0, 1.0) yield undefined behavior?

The question is pretty clear. The following gives the reason why I think these expressions might yield undefined behavior. I would like to know whether my reasoning is right or wrong and why.
Short ...

**1**

vote

**0**answers

8 views

### Control when IPython/Jupyter switches to exponential notation for floating point numbers

Sometimes the result of some calculation in IPython is some hard to read floating point number like 436132624031153.8. I absolutely don't mind the 16 significant digits, my problem is that its very ...

**-1**

votes

**0**answers

22 views

### Python : missing data in if…else [duplicate]

I'm a little new to Python. I have attached a snippet of code below.
threshold = 100.0
percentage=0.0
for key in map:
percentage=percentage+(key[1]["estimate"]*100/total_estimate)
if ...

**0**

votes

**1**answer

14 views

### Convert unsigned byte value to float from -1 to 1 [duplicate]

I'm not sure this is a programming question or more of a math problem, but since it's in the context of a javascript program I'm writing, I'm posting here.
I have an array of byte values from 0-255 (...

**-3**

votes

**1**answer

70 views

### Converting IPv6 addresses to 64-bit double

IPv6 addresses are 128-bit unsigned integers under the hood, so their range of
values is [0, 3.40 × 10^38]
A 128-bit register can store 2^128 (over 3.40 × 10^38) different values. The range of ...

**0**

votes

**2**answers

43 views

### ValueError: could not convert string to float, values in CSV can't be converted to floats

I am trying to do some mathematical operation on the values of a column fetched from a csv file. For that I wrote the code given below:
rows = csv.reader(open('sample_data_ml.csv', 'r'))
newrows = []
...

**0**

votes

**1**answer

53 views

### Sprintf and float values on serial port IAR

I've the following snippet of code which I'm compiling on my IAR workbench.
// global declaration
float fval = 3.14f;
uint8_t uartTxBuffer [100];
void function(void)
{
memset(uartTxBuffer,'\0',...

**0**

votes

**1**answer

22 views

### Decimal scale in Redshift after division by literal

Trying to understand this behavior
CREATE TABLE tab (x DECIMAL(10,0), y DECIMAL(10,0))
INSERT INTO tab VALUES(1, 3)
select x / y from tab will return 0.3333.... to 11 decimal places according to ...

**0**

votes

**2**answers

29 views

### Sum of two 'np.longdouble' yielding numerical error unrelated to printing

The issue I am trying to both understand and solve relates to the one asked years ago in this question: Sum of two "np.longdouble"s yields big numerical error, but is unrelated to the mere ...

**3**

votes

**1**answer

65 views

### Number formatting functions: What algorithm is used for rounding values ending in 5?

I had a value, 0.2435, that I wanted to format in D3 as a percent, rounded to the tenths place. I was expecting "24.4%" but got "24.3%".
I know there are different rounding methods, but couldn't ...

**3**

votes

**2**answers

47 views

### float data type precision in Java

The value of the number 0.1 when stored as a single precision floating point number is 0.100000001490116119384765625 (source : https://www.h-schmidt.net/FloatConverter/IEEE754.html) but it is printed ...