# Questions tagged [floating-point]

Floating point numbers are approximations of real numbers that can represent larger ranges than integers but use the same amount of memory, at the cost of lower precision. If your question is about small arithmetic errors (e.g. why does 0.2 + 0.1 equal 0.300000001?) or decimal conversion errors, please read the "info" page linked below before posting.

**0**

votes

**1**answer

10 views

### Rounding off a float value without decimal points in python 2

Problem is to round-off the resulting value without any decimals after summation of percentage values using Python 2. provided 3 inputs
Input 1 (meal cost) : 10.25
Input 2 (tip percentage) : 17
...

**3**

votes

**1**answer

75 views

### How to produce infinity in Haskell?

Similarily to How to produce a NaN in Haskell ...
In C, there is the INFINITY macro, defined by math.h.
Again, in http://hackage.haskell.org/package/ClassyPrelude-0.1/docs/Prelude-Math.html I can ...

**4**

votes

**0**answers

54 views

### How to produce a NaN in Haskell [duplicate]

In C, math.h defines nan. Is there a similar facility in Haskell whose only purpose is to produce a NaN?
Or is something like 0/0 or acos(2) my only choice?
http://hackage.haskell.org/package/...

**-1**

votes

**0**answers

34 views

### PHP Floating point precision inconsistent [duplicate]

0.01 === 0.01 // true
1/100 === 0.01 // true
3.6 / 360 === 0.01 // true
0.001 * 10 === 0.01 // true
0.0049 + 0.0051 === 0.01 // true
0.005 + (0.0025 * 2) === 0.01 // true
36 - 35.99 === 0.01 // false
...

**0**

votes

**1**answer

23 views

### Specific inputs of experiment return parser error

#!/bin/bash
for tracelength in 50 100 150 200 250 300 350 400 450 500 ; do
step=0.2
short=0
long=1
for i in {1..4}; do
ratio=0
for j in {1..4}; do
declare -...

**3**

votes

**1**answer

50 views

### Why does rounding the floating-point number 1.4999999999999999 produce 2?

I've been reading a book Write Great code - Understanding the Machine. In the section about rounding it says:
Numbers should be rounded to the smallest bigger number if the decimal bit value is ...

**-1**

votes

**1**answer

64 views

### Multiply *float32 and int value

I have 2 fields that I have to multiply. One is a *float32 field and another is an int field. How can I multiply these fields?
var totalPrice *float32
var price *float32
var volume int
this code ...

**0**

votes

**1**answer

62 views

### What is the max real number in computer I can get TRUE as a result of is.finite in R?

(reproducible example is added)
The maximum positive double (floating point) number in computer in my 64-bit OS and 32-bit R is:
.Machine$double.xmax # 1.797693e+308
So, the following is TRUE:
is....

**1**

vote

**1**answer

69 views

### Mathematically correct result from Math.sqrt for int arguments?

Given an int x, is (int)Math.sqrt(x) guaranteed to return the mathematically correct result, truncated to an int (i.e., the exact mathematical value of floor(sqrt(x))) for all non-negative x?

**-1**

votes

**0**answers

27 views

### Checking if a string is a float or integer [duplicate]

I'm creating a function that checks if a string is either a float or negative but my code does not account for decimal points or "-" for negative numbers.
def is_num(s):
if s.isdigit() == False:
...

**-2**

votes

**3**answers

26 views

### How can I pass the selective part of fraction number to string for displaying?

I would like to display the first 4 digit of fraction number and pass it to string to display in title of my plot. I checked this post but I couldn't find an elegant way.
I've tried following code as ...

**2**

votes

**1**answer

42 views

### Different floating point matrix multiplication result on two GPUs

I am running a MPSMatrixMultiplication from Metal Performance Shader on two separate GPUs, one is internal GPU and one is AMD GPU. The matrices are exactly the same.
But the problem is that the ...

**0**

votes

**0**answers

38 views

### python converting strings to floats resulting in error

I'm trying to convert strings from a txt file with charset: us-ascii.
to np.float64. The data are just decimal numbers (positive and negative).
with open(path) as fp:
data=fp.readlines()
...

**1**

vote

**1**answer

96 views

### Why float division is faster than integer division in c++?

Consider the following code snippet in C++ :(visual studio 2015)
First Block
const int size = 500000000;
int sum =0;
int *num1 = new int[size];//initialized between 1-250
int *num2 = new int[size];//...

**0**

votes

**5**answers

55 views

### How to strip and convert string to float (decimals)?

I am trying to convert a string which is like " 8,191.55 MB " to " 8191.55"
so that I can divide it by 1024 to get it in GigaBytes.
Code I am trying gives me " 819155" instead of "8191.55"
Code :
...

**2**

votes

**1**answer

57 views

### How to automatically eliminate double-precision math from generated code?

In our embedded project, targeted for the ARM Cortex M* cores, we are using the ARM clang-based toolchain (v6). We use float type for our floating-point math. As we try to minimize footprint, we found ...

**1**

vote

**0**answers

30 views

### Is there a generic standard way to store floats in a std::set and/or a std::unordered_set?

We know that NaN != NaN for IEEE floats. As a result, a number of "obvious" operations on floating pint numbers have a hidden gotchas in which NaNs in the data can mess things up terribly. For example:...

**0**

votes

**2**answers

28 views

### Floating point not iterable python

I wanted to store floating point values in a list iteratively but it says floating point not iterable even if the iteration value was integer
n=int(input('Please enter value of N: '));
for i in range(...

**2**

votes

**1**answer

76 views

### C#: Wrong result when converting expression with floats to int

Take the following code:
float a = 100.0f;
float b = 0.05f;
If I want to convert the result of division between a and b to an integer, I can do it with this code:
float c = a / (b * 1000.0f); // c =...

**0**

votes

**1**answer

33 views

### Issues with converting from floating point little-endian to big-endian and back again

I am trying to write a save parser which saves floating point numbers in little endian, however, Java is in big-endian, so i need to convert the FP and back again when writing but this does cause some ...

**0**

votes

**1**answer

57 views

### Double constants usage with suffix 'f' [duplicate]

double f_64 = 3.35f;
double f1_64 = 3.35;
What is the effect if suffix 'f' is used?
With the online FPU compilers, hexadecimal results are as below
With suffix f - 0x400ACCCCC0000000,
without ...

**2**

votes

**1**answer

55 views

### Will division result of 2 numbers with the same ratio always be the same?

a is an integer.
b is also an integer, but declared as double.
c = a/b, and c is also declared as double.
And, there are a2, b2, c2, with similar rules as a, b, c.
Also, in term of decimal algebra, ...

**-4**

votes

**0**answers

35 views

### Where can I find some free benchmarks to evaluate a MCU? [closed]

At present, I'm designing a soft processor with single-precision floating point unit (FPU). I am going to put my soft core into an FPGA and do some performance evaluation. The benchmarks should ...

**0**

votes

**2**answers

24 views

### Read only the numbers in a specified line in a text file

I am trying to read two floats from lines 18 and 19 of a text file and find their average. Please only look at the section of code separated by #########.
The file has a mixture of words and numbers, ...

**1**

vote

**0**answers

24 views

### How do I avoid floating point rounding errors in WebVTT?

I'm trying to build a WebVTT metronome and I'm specifying the cues with millisecond precision for both the start and end times. Unfortunately, it is showing up inconsistently in different browsers. ...

**0**

votes

**0**answers

21 views

### My app is work in my emulator but on real device the error Error inflating class android.support.design.widget.BottomNavigationView" is appear

my app is run in emulator good but when i try it with real device the error appear and the app is stopped please can any one help me
I think the problem is with api but I don't know helpppp pleaseeee ...

**2**

votes

**3**answers

99 views

### Precision of floating-point data types in C++

Why don't the precision of floating-point data types grow proportional to its size? E.g.:
std::cout << sizeof(float) << "\n"; // this gives 4 on my machine "debian 64 bit" with "gcc 6.3....

**-1**

votes

**3**answers

52 views

### C float initialization result is unexpected

Visual Studio 2019 Community
float variable is not what I initialize it to.
I made it less precise in hopes of avoiding truncation.
Is this not "controllable"?
tried to force init to round in other ...

**1**

vote

**1**answer

35 views

### Vector double-double floating point arithmetic

There exist workloads for which double precision floating point is not quite adequate, hence a need for quad precision. This is rarely supplied in hardware, so a workaround is to use double-double, ...

**0**

votes

**1**answer

43 views

### Representation of double numbers

In an 8 bit representation, we know that the number 4 is stored as 00000100, and the number -4 is stored as 11111100. But how the number 4.6 is stored in a double?

**0**

votes

**0**answers

91 views

### Recommended settings for floating-point contraction in C, C++, and Fortran

The C standard (and C++: [expr.7.1.6], and Fortran), states:
The values of the floating operands and the results of floating expressions may be represented in greater precision and range than that ...

**0**

votes

**0**answers

16 views

### Custom Binning function doesn't return Float data type

I've built a custom function that bins data based on a chosen min, max and bin size.
It works as intended for integers, however when using float's that represent percentages, it won't return the ...

**0**

votes

**1**answer

16 views

### Why when converting SQL Real to Numeric does the scale slightly increase?

I'm storing a value (0.15) as a Real datatype in a Quantity field in SQL.
Just playing around, when I cast as numeric, there are some very slight changes to scale.
I'm unsure why this occurs, and ...

**1**

vote

**1**answer

15 views

### How to insert float numbers from mysqldump

I have sql dump file coming from FirebirdSQL. Looks same as exported from MySQL (MariaDB). In the said file there are table creation statements one of which are the following:
CREATE TABLE VALIORS (
....

**0**

votes

**2**answers

35 views

### represent decimal numbers in javascript as “%g” formatting in C

I have a number generated as a finite decimal:
var x = k * Math.pow(10,p)
with k and p integers. Is there a simple way to convert it to an exact string representation?
If I use implict string ...

**2**

votes

**0**answers

69 views

### Is there a C++ function for converting single precision IBM floats to IEEE-754 floats? [duplicate]

I'm trying to read single precision floating point numbers from a binary (.segy) file on windows, using C++. These numbers follow the IBM floating point architecture, so I need to convert them into ...

**6**

votes

**1**answer

100 views

### How to change the rounding mode for floating point operations in MATLAB?

I want to change the rounding mode for floating point operations in MATLAB. According to IEEE 754-2008, there are 5 strategies for rounding:
round to nearest, ties to even
round to nearest, ties away ...

**0**

votes

**0**answers

33 views

### Replacement for ceil function for arbitrary ranges?

I have a couple of situations where I need to clamp a floating-point number to the upper or lower end of an arbitrary[1] range. Thus far, I have been using floor and ceil functions with pre- and post-...

**-1**

votes

**2**answers

52 views

### precision between float and double in C

I understand there are several topics same as mine, but I still don't really get it, so I'm expecting someone could explain this in a more simple but explicit way for me instead of pasting other ...

**0**

votes

**3**answers

54 views

### identifying and assigning +NaN and -NaN in C

I have to get input in float as +NaN or -NaN and convert to another artificial floating form called tinyfp (1 sign bit, 3 exponent bits, 4 fraction bits) for one question,
or get input in the tinyfp ...

**1**

vote

**1**answer

101 views

### Why the cos function in math.h faster than x86 fcos instruction

The cos() in math.h run faster than the x86 asm fcos.
The following code is compare between the x86 fcos and the cos() in math.h.
In this code, 1000000 times asm fcos cost 150ms; 1000000 times cos() ...

**0**

votes

**1**answer

40 views

### How can I base-convert a large/high-precision floating-point number in PHP?

As generic of a question as this seems, I'm having a really hard time
learning specifically about how to base-convert large high-precision float values in PHP using BCMath.
I'm trying to base-...

**3**

votes

**4**answers

206 views

### best practices on floating point precision in python

Earlier tonight, a friend of mine just handed me this cute problem. The problem says:
Make a program in MATLAB to check whether a point is inside a triangle or not. Not to forget to check if the ...

**1**

vote

**2**answers

43 views

### How to convert floating point to binary on the command line?

I'd like a simple way to convert a float string on the command line to a binary float.
For example:
$ echo "1.23" > file.bin
Except with the output as a 4 byte IEEE float
I'd expect the result ...

**0**

votes

**2**answers

56 views

### Correct way to use > and < comparison on floats in Java?

I am well aware that using the == operator on two floats does not work/is bad practise, however from what I can see, using the > and < operators on two floats should be valid. However, in my ...

**1**

vote

**1**answer

31 views

### Odd `gawk` filtering of very small floating point number

gawk filters out very small positive number differently depending on threshold used, but all thresholds should retain the entry.
Example input file, tmp:
A 3.92e-373
B 5e-300
C 5e-20
D 5e-6
E 5e-...

**0**

votes

**2**answers

34 views

### echo a variable with floating point value

Declare total=O
for repeat in {1..100}; do
executiontime=$(gtime -f "%U" python3 main.py | tail -0)
total=$(echo "scale=2; $total + $executiontime" | bc)
done
echo "$total/100"
The code ...

**-1**

votes

**0**answers

23 views

### How to use random.uniform function in for Loop and store the values in a variable for each execution

I have 1000 rows * 100 columns dataset. wanted to generate Random number with the help of these data. specifically for a some columns. say 20 columns.
have performed - random.randrange(min(dataset["...

**1**

vote

**2**answers

104 views

### Can we store a floating point in a regular register?

As I understand, floating points are stored in XMM registers and not the general purpose registers such as eax, so I did an experiment:
float a = 5;
in this case, a is stored as 1084227584 in the ...

**0**

votes

**3**answers

67 views

### Convert float to string without point/comma in python

I was wondering it there is a simple way to convert a float, for example 0.25, into a string without the point/float, i.e. like 025.
Is it possible? I'm searching for something that works for all ...