Rounding in Calculations and Conversions

In this section:

All computations are done in floating-point or decimal arithmetic, as defined by the FLOATMAPPING parameter. Packed fields are converted to D, M, or X internally, then back to P.

SET FLOATMAPPING enables you to take advantage of decimal-based precision numbers available in DB2 and Oracle, and extends that functionality to all numeric processing for floating point numbers. With this processing, you gain both precision, including improved rounding, and enhanced performance. The syntax is: 

SET FLOATMAPPING = {D|M|X}

where:

D

Uses the standard double-precision processing. This is the default value.

M

Uses a new internal format that provides decimal precision for double-precision floating point numbers up to 16 digits.

X

Uses a new internal format that provides decimal precision for double-precision floating point numbers up to 34 digits.

Note: If the field is passed to a HOLD file, the internal data types X or M will be propagated to the USAGE and ACTUAL formats in the HOLD Master File.

When a field with decimal places is computed to an integer field, the decimal places are truncated, and the resulting value is the integer part of the input value.

When a field with decimal places is computed from one format to another, two conversions take place:

  1. First, the field is converted internally to the data type specified by the FLOATMAPPING parameter.
  2. Second, the result of this conversion is converted to the specified format. At this point, the rounding algorithm described previously is applied.

Example: Redefining Field Formats

The following example illustrates some differences in the way packed fields, floating-point fields, and integer fields are stored and displayed. It also shows database values redefined to a format with a larger number of decimal places.

Master File

FILE=EXAMPLE, SUFFIX=FOC
 SEGNAME=ONLY, SEGTYPE=S1,$
  FIELD=PACKED2,,P9.2,$
  FIELD=DOUBLE2,,D9.2,$
  FIELD=FLOAT2,, F9.2,$
  FIELD=INTEGER,,I9  ,$
  FIELD=INTEGER,,I9  ,$
  FIELD=MATH2,,M9.2,$  
  FIELD=XMATH2,,X9.2,$

Program to Load Data

This MODIFY creates a file with four fields: a P field with two decimal places, a D field with two decimal places, an F field with two decimal places, and an integer field. The same data values are then loaded into each field. 

CREATE FILE EXAMPLE
MODIFY FILE EXAMPLE
FIXFORM PACKED2/9 X1 DOUBLE2/9 X1 FLOAT2/9 X1 INTEGER/9 X1  
FIXFORM MATH2/9 X1 XMATH2/9                                 
MATCH PACKED                                                
  ON MATCH REJECT                                           
  ON NOMATCH INCLUDE                                        
DATA                                                        
1.6666666 1.6666666 1.6666666 1.6666666 1.6666666 1.6666666 
125.16666 125.16666 125.16666 125.16666 125.16666 125.16666 
5432.6666 5432.6666 5432.6666 5432.6666 5432.6666 5432.6666 
4.1666666 4.1666666 4.1666666 4.1666666 4.1666666 4.1666666 
5.5       5.5       5.5       5.5       5.5       5.5       
106.66666 106.66666 106.66666 106.66666 106.66666 106.66666 
7.2222222 7.2222222 7.2222222 7.2222222 7.2222222 7.2222222 
END

Report Request

A DEFINE command creates temporary fields that are equal to PACKED2, DOUBLE2, and FLOAT2, with redefined formats containing four decimal places instead of two. These DEFINE fields illustrate the differences in the way packed fields and floating-point fields are stored and displayed.

The request prints the values and a total for all four database fields, and for the three DEFINE fields. 

DEFINE FILE EXAMPLE
PACKED4/P9.4=PACKED2;
DOUBLE4/D9.4=DOUBLE2;
FLOAT4/D9.4=FLOAT2;
MATH4/M9.4=MATH2;  
XMATH4/X9.4=XMATH2;
END
TABLE FILE EXAMPLE
PRINT PACKED2 PACKED4 DOUBLE2 DOUBLE4 FLOAT2 FLOAT4
MATH2 MATH4 XMATH2 XMATH4 INTEGER
ON TABLE SUMMARIZE
ON TABLE SET PAGE NOLEAD
ON TABLE HOLD FORMAT HTML
ON TABLE SET STYLE *
GRID=OFF,$
ENDSTYLE
END

The resulting report follows:

PAGE     1
 
  PACKED2   PACKED4    DOUBLE2    DOUBLE4    FLOAT2     FLOAT4   INTEGER
  -------   -------    -------    -------    ------     ------   -------
     1.67    1.6700       1.67     1.6667      1.67     1.6667         2
   125.17  125.1700     125.17   125.1667    125.17   125.1667       125
  5432.67 5432.6700   5,432.67 5,432.6666   5432.66 5,432.6641      5433
     4.17    4.1700       4.17     4.1667      4.17     4.1667         4
     5.50    5.5000       5.50     5.5000      5.50     5.5000         6
   106.67  106.6700     106.67   106.6667    106.67   106.6667       107
     7.22    7.2200       7.22     7.2222      7.22     7.2222         7
 
TOTAL
  5683.07 5683.0700   5,683.06 5,683.0555   5683.04 5,683.0529      5684

In this example, the PACKED2 sum is an accurate sum of the displayed values, which are the same as the stored values. The PACKED4 values and total are the same as the PACKED2 values.

The DOUBLE2 sum looks off by .01. it is not the sum of the printed values but a rounded sum of the stored values. The DOUBLE4 values show that the DOUBLE2 values are actually rounded from the stored values. The DOUBLE4 values and sum show more of the decimal places from which the DOUBLE2 values are rounded.

The FLOAT2 total seems off by .02. Like the DOUBLE2 total, the FLOAT2 total is a rounded total of the stored FLOAT2 values. F fields are not accurate beyond eight digits, as the FLOAT4 column shows.

The XMATH2 and MATH2 sums look off by .01. They are not the sum of the printed values but a rounded sum of the stored values. The XMATH4 and MATH4 values show that the XMATH2 and MATH2 values are actually rounded from the stored values. The XMATH and MATH values and sum show more of the decimal places from which the XMATH2 and MATH2 values are rounded.

The integer sum is an accurate total. Like packed fields, the storage values and displayed values are the same.

DEFINE and COMPUTE

DEFINE and COMPUTE may give different results for rounded fields. DEFINE fields are treated like data source fields, while COMPUTE fields are calculated on the results of the display command in the TABLE request. The following example illustrates this difference: 

DEFINE FILE EXAMPLE
DEFP3/P9.3=PACKED2/4;
END
TABLE FILE EXAMPLE
PRINT PACKED2 DEFP3
COMPUTE COMPP3/P9.3=PACKED2/4;
ON TABLE SUMMARIZE
END

The following report results:

  PACKED2      DEFP3     COMPP3
  -------      -----     ------
     1.67       .417       .417
   125.17     31.293     31.293
  5432.67   1358.168   1358.168
     4.17      1.042      1.042
     5.50      1.375      1.375
   106.67     26.668     26.668
     7.22      1.805      1.805
                               
TOTAL                          
  5683.07   1420.768   1420.767

The DEFP3 field is the result of a DEFINE. The values are treated like data source field values. The printed total, 1420.765, is the sum of the printed DEFP3 values, just as the PACKED2 total is the sum of the printed PACKED2 values.

The COMPP3 field is the result of a COMPUTE. The printed total, 1420.767, is calculated from the total sum of PACKED2 (5683.07 / 4).