# Questions tagged [floating-point]

Floating point numbers are approximations of real numbers that can represent larger ranges than integers but use the same amount of memory, at the cost of lower precision. If your question is about small arithmetic errors (e.g. why does 0.2 + 0.1 equal 0.300000001?) or decimal conversion errors, please read the "info" page linked below before posting.

11,315 questions
1answer
10 views

### Rounding off a float value without decimal points in python 2

Problem is to round-off the resulting value without any decimals after summation of percentage values using Python 2. provided 3 inputs Input 1 (meal cost) : 10.25 Input 2 (tip percentage) : 17 ...
1answer
75 views

### How to produce infinity in Haskell?

Similarily to How to produce a NaN in Haskell ... In C, there is the INFINITY macro, defined by math.h. Again, in http://hackage.haskell.org/package/ClassyPrelude-0.1/docs/Prelude-Math.html I can ...
0answers
54 views

### How to produce a NaN in Haskell [duplicate]

In C, math.h defines nan. Is there a similar facility in Haskell whose only purpose is to produce a NaN? Or is something like 0/0 or acos(2) my only choice? http://hackage.haskell.org/package/...
0answers
34 views

### PHP Floating point precision inconsistent [duplicate]

0.01 === 0.01 // true 1/100 === 0.01 // true 3.6 / 360 === 0.01 // true 0.001 * 10 === 0.01 // true 0.0049 + 0.0051 === 0.01 // true 0.005 + (0.0025 * 2) === 0.01 // true 36 - 35.99 === 0.01 // false ...
1answer
23 views

### Specific inputs of experiment return parser error

#!/bin/bash for tracelength in 50 100 150 200 250 300 350 400 450 500 ; do step=0.2 short=0 long=1 for i in {1..4}; do ratio=0 for j in {1..4}; do declare -...
1answer
50 views

### Why does rounding the floating-point number 1.4999999999999999 produce 2?

I've been reading a book Write Great code - Understanding the Machine. In the section about rounding it says: Numbers should be rounded to the smallest bigger number if the decimal bit value is ...
1answer
64 views

### Multiply *float32 and int value

I have 2 fields that I have to multiply. One is a *float32 field and another is an int field. How can I multiply these fields? var totalPrice *float32 var price *float32 var volume int this code ...
1answer
62 views

### What is the max real number in computer I can get TRUE as a result of is.finite in R?

(reproducible example is added) The maximum positive double (floating point) number in computer in my 64-bit OS and 32-bit R is: .Machine\$double.xmax # 1.797693e+308 So, the following is TRUE: is....
1answer
69 views

### Mathematically correct result from Math.sqrt for int arguments?

Given an int x, is (int)Math.sqrt(x) guaranteed to return the mathematically correct result, truncated to an int (i.e., the exact mathematical value of floor(sqrt(x))) for all non-negative x?
0answers
27 views

### Checking if a string is a float or integer [duplicate]

I'm creating a function that checks if a string is either a float or negative but my code does not account for decimal points or "-" for negative numbers. def is_num(s): if s.isdigit() == False: ...
3answers
26 views

### How can I pass the selective part of fraction number to string for displaying?

I would like to display the first 4 digit of fraction number and pass it to string to display in title of my plot. I checked this post but I couldn't find an elegant way. I've tried following code as ...
1answer
42 views

### Different floating point matrix multiplication result on two GPUs

I am running a MPSMatrixMultiplication from Metal Performance Shader on two separate GPUs, one is internal GPU and one is AMD GPU. The matrices are exactly the same. But the problem is that the ...
0answers
38 views

### python converting strings to floats resulting in error

I'm trying to convert strings from a txt file with charset: us-ascii. to np.float64. The data are just decimal numbers (positive and negative). with open(path) as fp: data=fp.readlines() ...
1answer
96 views

### Why float division is faster than integer division in c++?

Consider the following code snippet in C++ :(visual studio 2015) First Block const int size = 500000000; int sum =0; int *num1 = new int[size];//initialized between 1-250 int *num2 = new int[size];//...
5answers
55 views

### How to strip and convert string to float (decimals)?

I am trying to convert a string which is like " 8,191.55 MB " to " 8191.55" so that I can divide it by 1024 to get it in GigaBytes. Code I am trying gives me " 819155" instead of "8191.55" Code : ...
1answer
57 views

### How to automatically eliminate double-precision math from generated code?

In our embedded project, targeted for the ARM Cortex M* cores, we are using the ARM clang-based toolchain (v6). We use float type for our floating-point math. As we try to minimize footprint, we found ...
0answers
30 views

### Is there a generic standard way to store floats in a std::set and/or a std::unordered_set?

We know that NaN != NaN for IEEE floats. As a result, a number of "obvious" operations on floating pint numbers have a hidden gotchas in which NaNs in the data can mess things up terribly. For example:...
2answers
28 views

### Floating point not iterable python

I wanted to store floating point values in a list iteratively but it says floating point not iterable even if the iteration value was integer n=int(input('Please enter value of N: ')); for i in range(...
1answer
76 views

### C#: Wrong result when converting expression with floats to int

Take the following code: float a = 100.0f; float b = 0.05f; If I want to convert the result of division between a and b to an integer, I can do it with this code: float c = a / (b * 1000.0f); // c =...
1answer
33 views

### Issues with converting from floating point little-endian to big-endian and back again

I am trying to write a save parser which saves floating point numbers in little endian, however, Java is in big-endian, so i need to convert the FP and back again when writing but this does cause some ...
1answer
57 views

### Double constants usage with suffix 'f' [duplicate]

double f_64 = 3.35f; double f1_64 = 3.35; What is the effect if suffix 'f' is used? With the online FPU compilers, hexadecimal results are as below With suffix f - 0x400ACCCCC0000000, without ...
1answer
55 views

### Will division result of 2 numbers with the same ratio always be the same?

a is an integer. b is also an integer, but declared as double. c = a/b, and c is also declared as double. And, there are a2, b2, c2, with similar rules as a, b, c. Also, in term of decimal algebra, ...
0answers
35 views

### Where can I find some free benchmarks to evaluate a MCU? [closed]

At present, I'm designing a soft processor with single-precision floating point unit (FPU). I am going to put my soft core into an FPGA and do some performance evaluation. The benchmarks should ...
2answers
24 views

### Read only the numbers in a specified line in a text file

I am trying to read two floats from lines 18 and 19 of a text file and find their average. Please only look at the section of code separated by #########. The file has a mixture of words and numbers, ...
0answers
24 views

### How do I avoid floating point rounding errors in WebVTT?

I'm trying to build a WebVTT metronome and I'm specifying the cues with millisecond precision for both the start and end times. Unfortunately, it is showing up inconsistently in different browsers. ...
0answers
21 views

### My app is work in my emulator but on real device the error Error inflating class android.support.design.widget.BottomNavigationView" is appear

my app is run in emulator good but when i try it with real device the error appear and the app is stopped please can any one help me I think the problem is with api but I don't know helpppp pleaseeee ...
3answers
99 views

### Precision of floating-point data types in C++

Why don't the precision of floating-point data types grow proportional to its size? E.g.: std::cout << sizeof(float) << "\n"; // this gives 4 on my machine "debian 64 bit" with "gcc 6.3....
3answers
52 views

### C float initialization result is unexpected

Visual Studio 2019 Community float variable is not what I initialize it to. I made it less precise in hopes of avoiding truncation. Is this not "controllable"? tried to force init to round in other ...
1answer
35 views

### Vector double-double floating point arithmetic

There exist workloads for which double precision floating point is not quite adequate, hence a need for quad precision. This is rarely supplied in hardware, so a workaround is to use double-double, ...
1answer
43 views

### Representation of double numbers

In an 8 bit representation, we know that the number 4 is stored as 00000100, and the number -4 is stored as 11111100. But how the number 4.6 is stored in a double?
0answers
91 views

### Recommended settings for floating-point contraction in C, C++, and Fortran

The C standard (and C++: [expr.7.1.6], and Fortran), states: The values of the floating operands and the results of floating expressions may be represented in greater precision and range than that ...
0answers
16 views

### Custom Binning function doesn't return Float data type

I've built a custom function that bins data based on a chosen min, max and bin size. It works as intended for integers, however when using float's that represent percentages, it won't return the ...
1answer
16 views

### Why when converting SQL Real to Numeric does the scale slightly increase?

I'm storing a value (0.15) as a Real datatype in a Quantity field in SQL. Just playing around, when I cast as numeric, there are some very slight changes to scale. I'm unsure why this occurs, and ...
1answer
15 views

### How to insert float numbers from mysqldump

I have sql dump file coming from FirebirdSQL. Looks same as exported from MySQL (MariaDB). In the said file there are table creation statements one of which are the following: CREATE TABLE VALIORS ( ....
2answers
35 views

### represent decimal numbers in javascript as “%g” formatting in C

I have a number generated as a finite decimal: var x = k * Math.pow(10,p) with k and p integers. Is there a simple way to convert it to an exact string representation? If I use implict string ...
0answers
69 views

### Is there a C++ function for converting single precision IBM floats to IEEE-754 floats? [duplicate]

I'm trying to read single precision floating point numbers from a binary (.segy) file on windows, using C++. These numbers follow the IBM floating point architecture, so I need to convert them into ...
1answer
100 views

### How to change the rounding mode for floating point operations in MATLAB?

I want to change the rounding mode for floating point operations in MATLAB. According to IEEE 754-2008, there are 5 strategies for rounding: round to nearest, ties to even round to nearest, ties away ...
0answers
33 views

### Replacement for ceil function for arbitrary ranges?

I have a couple of situations where I need to clamp a floating-point number to the upper or lower end of an arbitrary range. Thus far, I have been using floor and ceil functions with pre- and post-...
2answers
52 views

### precision between float and double in C

I understand there are several topics same as mine, but I still don't really get it, so I'm expecting someone could explain this in a more simple but explicit way for me instead of pasting other ...
3answers
54 views

### identifying and assigning +NaN and -NaN in C

I have to get input in float as +NaN or -NaN and convert to another artificial floating form called tinyfp (1 sign bit, 3 exponent bits, 4 fraction bits) for one question, or get input in the tinyfp ...
1answer
101 views

### Why the cos function in math.h faster than x86 fcos instruction

The cos() in math.h run faster than the x86 asm fcos. The following code is compare between the x86 fcos and the cos() in math.h. In this code, 1000000 times asm fcos cost 150ms; 1000000 times cos() ...
1answer
40 views

### How can I base-convert a large/high-precision floating-point number in PHP?

As generic of a question as this seems, I'm having a really hard time learning specifically about how to base-convert large high-precision float values in PHP using BCMath. I'm trying to base-...
4answers
206 views

### best practices on floating point precision in python

Earlier tonight, a friend of mine just handed me this cute problem. The problem says: Make a program in MATLAB to check whether a point is inside a triangle or not. Not to forget to check if the ...
2answers
43 views

### How to convert floating point to binary on the command line?

I'd like a simple way to convert a float string on the command line to a binary float. For example: \$ echo "1.23" > file.bin Except with the output as a 4 byte IEEE float I'd expect the result ...
2answers
56 views

### Correct way to use > and < comparison on floats in Java?

I am well aware that using the == operator on two floats does not work/is bad practise, however from what I can see, using the > and < operators on two floats should be valid. However, in my ...
1answer
31 views

### Odd `gawk` filtering of very small floating point number

gawk filters out very small positive number differently depending on threshold used, but all thresholds should retain the entry. Example input file, tmp: A 3.92e-373 B 5e-300 C 5e-20 D 5e-6 E 5e-...
2answers
34 views

### echo a variable with floating point value

Declare total=O for repeat in {1..100}; do executiontime=\$(gtime -f "%U" python3 main.py | tail -0) total=\$(echo "scale=2; \$total + \$executiontime" | bc) done echo "\$total/100" The code ...
0answers
23 views

### How to use random.uniform function in for Loop and store the values in a variable for each execution

I have 1000 rows * 100 columns dataset. wanted to generate Random number with the help of these data. specifically for a some columns. say 20 columns. have performed - random.randrange(min(dataset["...
2answers
104 views

### Can we store a floating point in a regular register?

As I understand, floating points are stored in XMM registers and not the general purpose registers such as eax, so I did an experiment: float a = 5; in this case, a is stored as 1084227584 in the ...
3answers
67 views

### Convert float to string without point/comma in python

I was wondering it there is a simple way to convert a float, for example 0.25, into a string without the point/float, i.e. like 025. Is it possible? I'm searching for something that works for all ...

http://mssss.yulina-kosm.ru