Questions tagged [floating-point]

Floating point numbers are approximations of real numbers that can represent larger ranges than integers but use the same amount of memory, at the cost of lower precision. If your question is about small arithmetic errors (e.g. why does 0.2 + 0.1 equal 0.300000001?) or decimal conversion errors, please read the "info" page linked below before posting.

0
votes
0answers
14 views

Is this a Floating-Point Error or something else entirely?

Given the following R code: i = 1.0 while(i > 0.03) i=i-0.05 print(i) The output is as follows: [1] 0.95 [1] 0.9 ... .. . [1] 0.1 [1] 0.05 [1] -3.191891e-16 Can anyone explain why the ...
0
votes
0answers
30 views

Equality operation does not work for Pandas dataframe value in one or more cells but work for others

I am working with dataframes read from csv file using pandas. I wanted to do some equality operation to find the rows where the column is of a specific value. However, it seems that the equality ...
0
votes
1answer
55 views

How to get data in an array in a pattern?

I am trying to get the x and y values out of a PointCloud2 data. This data is stored in an array in bytes of float32's. My question is, how can I get only the x and y data? I will add a snippet of ...
1
vote
1answer
37 views

How to convert HEX string to Float (Little Endian)

I just learned Python (3.x) and I am stuck with HEX String conversion to Float. I have this HEX String values: '...
0
votes
0answers
28 views

Checking and extracting data from a string

I've spent ages trying to edit similar solutions to work for my situation, but can't get it working, can someone please help? I am reading from either cin or file, and I'd like to check if the first ...
1
vote
4answers
60 views

When to use %d and %f in C?

I'm new to C programming, but decent in Java, my question is when do we use %d and %f? In what situation? For example, based on the code block given below, if I was asked to take (int)a*(float)y, do I ...
0
votes
1answer
43 views

Why did IEEE754 choose 11 exponent bits for double aka binary64?

I have been reading and it seems the IEEE 754 defines a 64 bit float's (double) exponent as 11 bits. (https://en.wikipedia.org/wiki/Double-precision_floating-point_format) My question is why? A 64 ...
3
votes
4answers
65 views

C: while loop using floating point never terminates [duplicate]

By executing this code: float f = 1.0; while (f != 0.0) { f = f - 0.1; printf("%.1f\n", f); } It is expected that it would run 10 times and stop, but what turns out is that it inevitably ...
2
votes
1answer
79 views

How to detect differences in floating point behaviour across platforms

What checks can I perform to identify what differences they are in the floating point behaviour of two hardware platforms? Verifying IEE-754 compliance or checking for known bugs may be sufficient (...
-2
votes
1answer
50 views

what should i change in my function so it calculates real numbers?

i am trying to write a function that repeatedly adds 0.001 to 't' and then plugs it into 'y' until 't' reaches 0.3 however the numbers come out wrong, but i've noticed that if i change float to int ...
2
votes
1answer
51 views

how to validate that a floating number is not “0”

I have an <Entry> control where the user is allowed to enter decimal numbers, for examenple ... 0,2 0,02 5,405 But I do not want to enter a "0" (as a decimal), as follows 0,0 ...
0
votes
1answer
37 views

Why does f(50) differ radically from f(50.0)? [duplicate]

I have defined a function in Python 2.7.15 which takes two inputs (b,v), but I have noticed that f(50,0.1) produces very different results to those of f(50.0,0.1). This is the function: def f(b,v): ...
0
votes
2answers
57 views

Convert FP32 to Bfloat16 in C++

How can I convert from float (1bit sign, 8bit exp, 23bit mantissa) to Bfloat16 (1bit sign, 8bit exp, 7bit mantissa) in C++?
-1
votes
0answers
34 views

Performing floating-point calculations as fixed-point in Linux kernel code

I am trying to perform the following computation in Linux kernel code. float rm_least_upper_bound; rm_least_upper_bound = (task_count*((2^(1/task_count))-1)); Since kernel code does not normally ...
0
votes
2answers
57 views

Bash float calculations from last line in a file

I have a file text.txt as follows: A B C D E F G F 1.1 2 3 4 5 6 7 8 1.2 20 30 40 50 60 70 80 6.3 70.5 10 20.5 ...
1
vote
1answer
23 views

IEEE 754 identity laws: Literal meaning and value-changing optimizations

The IEEE 754-2008 standard says for expressions like 0 + x and 1 * x that the identity property may be applied (§ 10.4): Applying the identity property 1 × x when x is not a signaling NaN and the ...
-3
votes
4answers
127 views

How a float number can store a number greater than 256? [closed]

I'm confused about the representation of floating points (IEEE_754-1985) spec. How can a float number store a number greater than 256 if only 8 bits (1 byte) is available to store the significand part?...
0
votes
1answer
29 views

What should the maximum difference be between a float and a 16 bit calculation? [on hold]

Lets say I have a formula that calculates a value. I can calculate the value either with floating point arithmetic or with fixed point arithmetic (using 16 bit integers). What should be the maximum ...
0
votes
0answers
18 views

Weird behavior of python in accurate math [duplicate]

Look at this code in python: a=2**.5 b=np.sqrt(1-a**2/2) __main__:1: RuntimeWarning: invalid value encountered in sqrt The actual result shall be 0, because a**2/2=1 and sqrt(1-1)=0 but python ...
0
votes
0answers
44 views

Python performance: square roots and hot-loop operation strategy

In comparing math.sqrt(…) to the built-in binary operator, … ** 0.5 in a REPL using timeit.timeit(…), it looks as though the binary op has a slight edge over the math module function: … although I ...
0
votes
1answer
53 views

software floating point and x87 or sse disabled

I am trying to enable software floating point under gcc following suggestions in this question but I've hit a snag: The -msoft-float flag causes: /usr/include/c++/4.8.2/bits/basic_string.h: In ...
1
vote
2answers
72 views

How to correctly floor the floating point pair sum

Let me have two floating point variables coming as function arguments: float fun(float x, float y) { // ... } I would like to calculate the floor of their sum. Is it possible to do it correctly ...
0
votes
1answer
26 views

Check for integer value vs float value

I My vba doesnt seem to quite work. I need it to multiply a value in column N by 2 if it is a decimal and do nothing for all other values. currently it is Sub multiply() Dim i, j, k, l As Integer ...
0
votes
2answers
66 views

Reading and writing non-standard floating point values

I'm working with a binary file (3d model file for an old video game) in C#. The file format isn't officially documented, but some of it has been reverse-engineered by the game's community. I'm having ...
0
votes
1answer
17 views

Pandas Float Formating -

I am a newbie to Python/Pandas and do not know how to present my dataframe. I understand that the data may not need to be changed but rather how it is displayed. I have a dataframe as follow : ...
3
votes
2answers
42 views

What's a method that works exactly like Math.floorMod() but with floats instead of ints?

Here's a set of inputs and outputs for Math.floorMod(x, 5) as an example. int x; inputs: x = -15 | -14 | -13 | -12 | -11 -10 | -09 | -08 | -07 | -06 -05 | -04 | -03 | -02 | -01 +00 | +01 ...
0
votes
1answer
16 views

unable to execute java program [duplicate]

When I tried to compile the program, it compiled successfully. But when I executed the program, this is the output I recieved in command prompt Exception in thread "main" java.lang....
0
votes
0answers
33 views

Python: mysql double and float

In MySQL table column balance has a type DOUBLE without specifying precision. And I can't change column type to DECIMAL or DOUBLE with precision (I know that is epic fail to store money in DOUBLE ...
0
votes
1answer
20 views

Shift bits of a float in MIPS without MTC1

I'm currently working on a school assignment that requires me to find the biased exponent of a single precision float without using the mtc1 command that comes with MIPS. I am wondering how it would ...
0
votes
0answers
20 views

Getting wrong output when doing simple float point addition on turbo c [duplicate]

So, i wrote up a C program to accept a float number from user and then add it with 34.2 and display the output. When i enter 1 or 1.0 as the input. The output i get is 35.200001 or 35.200002. But i ...
-1
votes
3answers
70 views

Putting value of a char array into a float variable

I want to put the value of an array into a float integer. main(){ float a; char array[4]="12.1"; a=atoi(array); printf("%f",a); } When I uses this program, it gives 12.000000 as ...
22
votes
12answers
4k views

Python if-else code style for reduced code for rounding floats

Is there any shorter, more legible code style to solve this problem? I am trying to classify some float values into interregional folders. def classify(value): if value < -0.85 and value &...
0
votes
1answer
35 views

How to store state flags in floating point numbers for GLSL / WebGL

I have recently learned about storing boolean flags in integers using bitmasks. I'm wondering how you can do this to store and retrieve boolean flags to/from JavaScript, and store and retrieve flags ...
0
votes
2answers
52 views

Why does my double round my variables sometimes? [duplicate]

I really dont unterstand why, but my double array just sometimes round my variables, even though it shouldn't. The weird thing is, that it only does this sometimes, as you see in the picture. All of ...
0
votes
0answers
29 views

Fortran: Floating-Point Arithmetic for really small number [duplicate]

I want to define two small number as my variables in fortran. I tested them in two ways, one is by using Kind library, and the other is use ISO_FORTRAN_ENV. Here is my program: implicit none ...
2
votes
1answer
82 views

Cuda programming cannot have the same computation accuracy compared with CPU program in terms of float type

I try to use the GPU to accelerate my program which computes L2 distance between two float array. In order to check the computation accuracy, I write both CUDA program and CPU program. However, I ...
-3
votes
1answer
55 views

How float data type is stored in memory? [duplicate]

If we create one variable of data type float and assign any value to it then how this is stored in memory? float var = 13.34;
0
votes
0answers
17 views

Find FLOATS between FLOATS query with MySQLi PHP?

I'm working with latitude and longitude to get points that are closer to others points. I need to find the latitudes that may exist between 2 numbers in my DataBase, unfortunately I can't find -as an ...
2
votes
1answer
29 views

Looking at user input to see if it is a float, always coming back as true

I trying to prompt the user fora number, and scan it to see if it is a float. Even when entering a number without a decimal it is coming back saying it is a float. The function should return 1 if the ...
48
votes
3answers
1k views

Do std::min(0.0, 1.0) and std::max(0.0, 1.0) yield undefined behavior?

The question is pretty clear. The following gives the reason why I think these expressions might yield undefined behavior. I would like to know whether my reasoning is right or wrong and why. Short ...
1
vote
0answers
8 views

Control when IPython/Jupyter switches to exponential notation for floating point numbers

Sometimes the result of some calculation in IPython is some hard to read floating point number like 436132624031153.8. I absolutely don't mind the 16 significant digits, my problem is that its very ...
-1
votes
0answers
22 views

Python : missing data in if…else [duplicate]

I'm a little new to Python. I have attached a snippet of code below. threshold = 100.0 percentage=0.0 for key in map: percentage=percentage+(key[1]["estimate"]*100/total_estimate) if ...
0
votes
1answer
14 views

Convert unsigned byte value to float from -1 to 1 [duplicate]

I'm not sure this is a programming question or more of a math problem, but since it's in the context of a javascript program I'm writing, I'm posting here. I have an array of byte values from 0-255 (...
-3
votes
1answer
70 views

Converting IPv6 addresses to 64-bit double

IPv6 addresses are 128-bit unsigned integers under the hood, so their range of values is [0, 3.40 × 10^38] A 128-bit register can store 2^128 (over 3.40 × 10^38) different values. The range of ...
0
votes
2answers
43 views

ValueError: could not convert string to float, values in CSV can't be converted to floats

I am trying to do some mathematical operation on the values of a column fetched from a csv file. For that I wrote the code given below: rows = csv.reader(open('sample_data_ml.csv', 'r')) newrows = [] ...
0
votes
1answer
53 views

Sprintf and float values on serial port IAR

I've the following snippet of code which I'm compiling on my IAR workbench. // global declaration float fval = 3.14f; uint8_t uartTxBuffer [100]; void function(void) { memset(uartTxBuffer,'\0',...
0
votes
1answer
22 views

Decimal scale in Redshift after division by literal

Trying to understand this behavior CREATE TABLE tab (x DECIMAL(10,0), y DECIMAL(10,0)) INSERT INTO tab VALUES(1, 3) select x / y from tab will return 0.3333.... to 11 decimal places according to ...
0
votes
2answers
29 views

Sum of two 'np.longdouble' yielding numerical error unrelated to printing

The issue I am trying to both understand and solve relates to the one asked years ago in this question: Sum of two "np.longdouble"s yields big numerical error, but is unrelated to the mere ...
3
votes
1answer
65 views

Number formatting functions: What algorithm is used for rounding values ending in 5?

I had a value, 0.2435, that I wanted to format in D3 as a percent, rounded to the tenths place. I was expecting "24.4%" but got "24.3%". I know there are different rounding methods, but couldn't ...
3
votes
2answers
47 views

float data type precision in Java

The value of the number 0.1 when stored as a single precision floating point number is 0.100000001490116119384765625 (source : https://www.h-schmidt.net/FloatConverter/IEEE754.html) but it is printed ...

http://mssss.yulina-kosm.ru