Questions tagged [floating-point]

Floating point numbers are approximations of real numbers that can represent larger ranges than integers but use the same amount of memory, at the cost of lower precision. If your question is about small arithmetic errors (e.g. why does 0.2 + 0.1 equal 0.300000001?) or decimal conversion errors, please read the "info" page linked below before posting.

0
votes
1answer
10 views

Rounding off a float value without decimal points in python 2

Problem is to round-off the resulting value without any decimals after summation of percentage values using Python 2. provided 3 inputs Input 1 (meal cost) : 10.25 Input 2 (tip percentage) : 17 ...
3
votes
1answer
75 views

How to produce infinity in Haskell?

Similarily to How to produce a NaN in Haskell ... In C, there is the INFINITY macro, defined by math.h. Again, in http://hackage.haskell.org/package/ClassyPrelude-0.1/docs/Prelude-Math.html I can ...
4
votes
0answers
54 views

How to produce a NaN in Haskell [duplicate]

In C, math.h defines nan. Is there a similar facility in Haskell whose only purpose is to produce a NaN? Or is something like 0/0 or acos(2) my only choice? http://hackage.haskell.org/package/...
-1
votes
0answers
34 views

PHP Floating point precision inconsistent [duplicate]

0.01 === 0.01 // true 1/100 === 0.01 // true 3.6 / 360 === 0.01 // true 0.001 * 10 === 0.01 // true 0.0049 + 0.0051 === 0.01 // true 0.005 + (0.0025 * 2) === 0.01 // true 36 - 35.99 === 0.01 // false ...
0
votes
1answer
23 views

Specific inputs of experiment return parser error

#!/bin/bash for tracelength in 50 100 150 200 250 300 350 400 450 500 ; do step=0.2 short=0 long=1 for i in {1..4}; do ratio=0 for j in {1..4}; do declare -...
3
votes
1answer
50 views

Why does rounding the floating-point number 1.4999999999999999 produce 2?

I've been reading a book Write Great code - Understanding the Machine. In the section about rounding it says: Numbers should be rounded to the smallest bigger number if the decimal bit value is ...
-1
votes
1answer
64 views

Multiply *float32 and int value

I have 2 fields that I have to multiply. One is a *float32 field and another is an int field. How can I multiply these fields? var totalPrice *float32 var price *float32 var volume int this code ...
0
votes
1answer
62 views

What is the max real number in computer I can get TRUE as a result of is.finite in R?

(reproducible example is added) The maximum positive double (floating point) number in computer in my 64-bit OS and 32-bit R is: .Machine$double.xmax # 1.797693e+308 So, the following is TRUE: is....
1
vote
1answer
69 views

Mathematically correct result from Math.sqrt for int arguments?

Given an int x, is (int)Math.sqrt(x) guaranteed to return the mathematically correct result, truncated to an int (i.e., the exact mathematical value of floor(sqrt(x))) for all non-negative x?
-1
votes
0answers
27 views

Checking if a string is a float or integer [duplicate]

I'm creating a function that checks if a string is either a float or negative but my code does not account for decimal points or "-" for negative numbers. def is_num(s): if s.isdigit() == False: ...
-2
votes
3answers
26 views

How can I pass the selective part of fraction number to string for displaying?

I would like to display the first 4 digit of fraction number and pass it to string to display in title of my plot. I checked this post but I couldn't find an elegant way. I've tried following code as ...
2
votes
1answer
42 views

Different floating point matrix multiplication result on two GPUs

I am running a MPSMatrixMultiplication from Metal Performance Shader on two separate GPUs, one is internal GPU and one is AMD GPU. The matrices are exactly the same. But the problem is that the ...
0
votes
0answers
38 views

python converting strings to floats resulting in error

I'm trying to convert strings from a txt file with charset: us-ascii. to np.float64. The data are just decimal numbers (positive and negative). with open(path) as fp: data=fp.readlines() ...
1
vote
1answer
96 views

Why float division is faster than integer division in c++?

Consider the following code snippet in C++ :(visual studio 2015) First Block const int size = 500000000; int sum =0; int *num1 = new int[size];//initialized between 1-250 int *num2 = new int[size];//...
0
votes
5answers
55 views

How to strip and convert string to float (decimals)?

I am trying to convert a string which is like " 8,191.55 MB " to " 8191.55" so that I can divide it by 1024 to get it in GigaBytes. Code I am trying gives me " 819155" instead of "8191.55" Code : ...
2
votes
1answer
57 views

How to automatically eliminate double-precision math from generated code?

In our embedded project, targeted for the ARM Cortex M* cores, we are using the ARM clang-based toolchain (v6). We use float type for our floating-point math. As we try to minimize footprint, we found ...
1
vote
0answers
30 views

Is there a generic standard way to store floats in a std::set and/or a std::unordered_set?

We know that NaN != NaN for IEEE floats. As a result, a number of "obvious" operations on floating pint numbers have a hidden gotchas in which NaNs in the data can mess things up terribly. For example:...
0
votes
2answers
28 views

Floating point not iterable python

I wanted to store floating point values in a list iteratively but it says floating point not iterable even if the iteration value was integer n=int(input('Please enter value of N: ')); for i in range(...
2
votes
1answer
76 views

C#: Wrong result when converting expression with floats to int

Take the following code: float a = 100.0f; float b = 0.05f; If I want to convert the result of division between a and b to an integer, I can do it with this code: float c = a / (b * 1000.0f); // c =...
0
votes
1answer
33 views

Issues with converting from floating point little-endian to big-endian and back again

I am trying to write a save parser which saves floating point numbers in little endian, however, Java is in big-endian, so i need to convert the FP and back again when writing but this does cause some ...
0
votes
1answer
57 views

Double constants usage with suffix 'f' [duplicate]

double f_64 = 3.35f; double f1_64 = 3.35; What is the effect if suffix 'f' is used? With the online FPU compilers, hexadecimal results are as below With suffix f - 0x400ACCCCC0000000, without ...
2
votes
1answer
55 views

Will division result of 2 numbers with the same ratio always be the same?

a is an integer. b is also an integer, but declared as double. c = a/b, and c is also declared as double. And, there are a2, b2, c2, with similar rules as a, b, c. Also, in term of decimal algebra, ...
-4
votes
0answers
35 views

Where can I find some free benchmarks to evaluate a MCU? [closed]

At present, I'm designing a soft processor with single-precision floating point unit (FPU). I am going to put my soft core into an FPGA and do some performance evaluation. The benchmarks should ...
0
votes
2answers
24 views

Read only the numbers in a specified line in a text file

I am trying to read two floats from lines 18 and 19 of a text file and find their average. Please only look at the section of code separated by #########. The file has a mixture of words and numbers, ...
1
vote
0answers
24 views

How do I avoid floating point rounding errors in WebVTT?

I'm trying to build a WebVTT metronome and I'm specifying the cues with millisecond precision for both the start and end times. Unfortunately, it is showing up inconsistently in different browsers. ...
0
votes
0answers
21 views

My app is work in my emulator but on real device the error Error inflating class android.support.design.widget.BottomNavigationView" is appear

my app is run in emulator good but when i try it with real device the error appear and the app is stopped please can any one help me I think the problem is with api but I don't know helpppp pleaseeee ...
2
votes
3answers
99 views

Precision of floating-point data types in C++

Why don't the precision of floating-point data types grow proportional to its size? E.g.: std::cout << sizeof(float) << "\n"; // this gives 4 on my machine "debian 64 bit" with "gcc 6.3....
-1
votes
3answers
52 views

C float initialization result is unexpected

Visual Studio 2019 Community float variable is not what I initialize it to. I made it less precise in hopes of avoiding truncation. Is this not "controllable"? tried to force init to round in other ...
1
vote
1answer
35 views

Vector double-double floating point arithmetic

There exist workloads for which double precision floating point is not quite adequate, hence a need for quad precision. This is rarely supplied in hardware, so a workaround is to use double-double, ...
0
votes
1answer
43 views

Representation of double numbers

In an 8 bit representation, we know that the number 4 is stored as 00000100, and the number -4 is stored as 11111100. But how the number 4.6 is stored in a double?
0
votes
0answers
91 views

Recommended settings for floating-point contraction in C, C++, and Fortran

The C standard (and C++: [expr.7.1.6], and Fortran), states: The values of the floating operands and the results of floating expressions may be represented in greater precision and range than that ...
0
votes
0answers
16 views

Custom Binning function doesn't return Float data type

I've built a custom function that bins data based on a chosen min, max and bin size. It works as intended for integers, however when using float's that represent percentages, it won't return the ...
0
votes
1answer
16 views

Why when converting SQL Real to Numeric does the scale slightly increase?

I'm storing a value (0.15) as a Real datatype in a Quantity field in SQL. Just playing around, when I cast as numeric, there are some very slight changes to scale. I'm unsure why this occurs, and ...
1
vote
1answer
15 views

How to insert float numbers from mysqldump

I have sql dump file coming from FirebirdSQL. Looks same as exported from MySQL (MariaDB). In the said file there are table creation statements one of which are the following: CREATE TABLE VALIORS ( ....
0
votes
2answers
35 views

represent decimal numbers in javascript as “%g” formatting in C

I have a number generated as a finite decimal: var x = k * Math.pow(10,p) with k and p integers. Is there a simple way to convert it to an exact string representation? If I use implict string ...
2
votes
0answers
69 views

Is there a C++ function for converting single precision IBM floats to IEEE-754 floats? [duplicate]

I'm trying to read single precision floating point numbers from a binary (.segy) file on windows, using C++. These numbers follow the IBM floating point architecture, so I need to convert them into ...
6
votes
1answer
100 views

How to change the rounding mode for floating point operations in MATLAB?

I want to change the rounding mode for floating point operations in MATLAB. According to IEEE 754-2008, there are 5 strategies for rounding: round to nearest, ties to even round to nearest, ties away ...
0
votes
0answers
33 views

Replacement for ceil function for arbitrary ranges?

I have a couple of situations where I need to clamp a floating-point number to the upper or lower end of an arbitrary[1] range. Thus far, I have been using floor and ceil functions with pre- and post-...
-1
votes
2answers
52 views

precision between float and double in C

I understand there are several topics same as mine, but I still don't really get it, so I'm expecting someone could explain this in a more simple but explicit way for me instead of pasting other ...
0
votes
3answers
54 views

identifying and assigning +NaN and -NaN in C

I have to get input in float as +NaN or -NaN and convert to another artificial floating form called tinyfp (1 sign bit, 3 exponent bits, 4 fraction bits) for one question, or get input in the tinyfp ...
1
vote
1answer
101 views

Why the cos function in math.h faster than x86 fcos instruction

The cos() in math.h run faster than the x86 asm fcos. The following code is compare between the x86 fcos and the cos() in math.h. In this code, 1000000 times asm fcos cost 150ms; 1000000 times cos() ...
0
votes
1answer
40 views

How can I base-convert a large/high-precision floating-point number in PHP?

As generic of a question as this seems, I'm having a really hard time learning specifically about how to base-convert large high-precision float values in PHP using BCMath. I'm trying to base-...
3
votes
4answers
206 views

best practices on floating point precision in python

Earlier tonight, a friend of mine just handed me this cute problem. The problem says: Make a program in MATLAB to check whether a point is inside a triangle or not. Not to forget to check if the ...
1
vote
2answers
43 views

How to convert floating point to binary on the command line?

I'd like a simple way to convert a float string on the command line to a binary float. For example: $ echo "1.23" > file.bin Except with the output as a 4 byte IEEE float I'd expect the result ...
0
votes
2answers
56 views

Correct way to use > and < comparison on floats in Java?

I am well aware that using the == operator on two floats does not work/is bad practise, however from what I can see, using the > and < operators on two floats should be valid. However, in my ...
1
vote
1answer
31 views

Odd `gawk` filtering of very small floating point number

gawk filters out very small positive number differently depending on threshold used, but all thresholds should retain the entry. Example input file, tmp: A 3.92e-373 B 5e-300 C 5e-20 D 5e-6 E 5e-...
0
votes
2answers
34 views

echo a variable with floating point value

Declare total=O for repeat in {1..100}; do executiontime=$(gtime -f "%U" python3 main.py | tail -0) total=$(echo "scale=2; $total + $executiontime" | bc) done echo "$total/100" The code ...
-1
votes
0answers
23 views

How to use random.uniform function in for Loop and store the values in a variable for each execution

I have 1000 rows * 100 columns dataset. wanted to generate Random number with the help of these data. specifically for a some columns. say 20 columns. have performed - random.randrange(min(dataset["...
1
vote
2answers
104 views

Can we store a floating point in a regular register?

As I understand, floating points are stored in XMM registers and not the general purpose registers such as eax, so I did an experiment: float a = 5; in this case, a is stored as 1084227584 in the ...
0
votes
3answers
67 views

Convert float to string without point/comma in python

I was wondering it there is a simple way to convert a float, for example 0.25, into a string without the point/float, i.e. like 025. Is it possible? I'm searching for something that works for all ...

http://mssss.yulina-kosm.ru