x

Data harvesting is the process of taking data from a source and processing it into information that is then loaded into one or more datapoints needed by the PMF source.

Note: The result of any data harvesting operation is always numeric.

Typical data harvesting actions are:

• Summarizing aggregated values of an inbound numeric field.
• Summarizing values of an inbound numeric field, but only if certain conditions are true. Otherwise, they are ignored.
• Counting occurrences of a particular set of field conditions (Counting when).
• Distinctly counting occurrences of a particular set of field conditions. Distinct counting is a method to prevent double-counting fixed assets, people, and so on.
• Writing a custom load for a datapoint, where the logic is freeform. For example, it is possible to perform custom operations or calculations on the data inbound or implement more complex or nested filters. These use WebFOCUS programming logic, functions, and operations. The PMF source loader allows all the data harvesting methods above.

In PMF, all data harvesting is done across a potential Cartesian cross product of dimensional intersection. Nearly every load involves a fairly high degree of sparsity against this cross product, but in most cases, it also involves creating multiple records that follow a particular pattern against the source data.

In many cases, depending on the degree of granularity of the inbound loadable data from the source, PMF requests the data to be aggregated even when loaded at the lowest level of its dimensionality. This happens when the physical data table or view contains more detailed records that go to a lower logical level than the source requests. Some examples of this are:

• If the physical data of the source is stored at a daily or hourly level, and the source load needs it summarized at a monthly level.
• If there is a location dimension, and the source stores data at the geocode or ZIP code level, and the PMF source wants this data at the city or regional level.

In the examples listed, the PMF loader will be aggregating records from the source inbound during the load process.

x

Loadable Sources are primary sources for data in PMF. PMF treats them as first generation in any lineage, along with user-entered datapoints and generated datapoints.

Loadable data is treated as updated on the date of load, but is effective as of the time dimension linkages for the data.

x

To set up a new Loadable source:

1. In the Manage tab, click the Sources panel button.
2. Click New.

   The New Source panel opens.

3. Name your new source and select Loaded from Data Source from the first drop-down menu, as shown in the following image. This option lets PMF know that you want this source to harvest data from an existing table, view, flat file, and so on.

   

4. Select the metadata file for the existing table from the second drop-down menu. If the file you want to use is not listed, expand the File Picker by selecting Show More... from the drop-down menu.

   The FIle Picker allows you to look at all of the available metadata files in your currently configured WebFOCUS app path. The icons to the left of each file name let you explore the contents of the file to make sure that it is the correct one. To select the file to use, click the name once and close the File Picker.

5. Specify the datapoints to be harvested. For each line where you want to add another field from the source, you can do the following:
   • To harvest by summarizing the values of the field, select the field you want to harvest from the Source field name drop-down menu. You can use the default name assigned to the datapoint, or change the name at anytime.
   • To harvest by summarizing the values of the field only when conditions are matched, select the field you want to harvest from the Source field name drop-down menu and create filters by clicking the Filters button to choose your filters.
   • To harvest by counting, select [Count] from the Source Field Name drop-down menu and specify the conditions to count by clicking the When button. You are required to provide a name for the datapoint.
   • To harvest by counting distinctly, select [Distinct Count] from the Source Field Name drop-down menu. Specify the conditions to count by clicking the When button, and specify the matching field for distinct counting. You are required to provide a name for the datapoint.
   • To create a custom formula for calculating the field during harvesting, select [User Defined]. Specify filters to be used by clicking the Filters button and specify the WebFOCUS code-based formula for harvesting the field by clicking the Code button. You are required to provide a name for the datapoint.

   Your datapoints should look similar to the following image.

   

   Note: You can save your work-in-progress at any time. PMF will not be able to actually harvest data into the datapoints that were set up until the needed harvesting details and dimensionality have been specified.

6. Click the Dimension Links tab and specify Time, as well as other dimensional linkages, as shown in the following image.

   

   Note: The fields you select for dimensional linkages must contain values that match up to those you loaded for the dimension keys in the Dimension Loader. PMF will alert you if there is an issue.

7. Once the dimensional linkages are set up, click the Preview tab. The data that is being set up for harvesting can be seen, as shown in the following image.

   

   Note:

   • The Preview pane is very flexible and shows the row values that will be added, changed, or deleted from your source, in separate sections.
   • You can resort the contents of the Preview pane by clicking a column heading. The sort toggles between ascending and descending order.
   • You can scroll the Preview pane vertically and horizontally, and drag the navigation bar divider to make more room onscreen.
   • You can refresh the Preview pane at any time by clicking the Refresh button on the Preview tab.
   • You can switch back and forth between the Datapoints, Dimension Links, and Preview tabs to alter the patterns you are using for harvesting, adjust the linkage keys and dimensions that you are linking to, and refresh the preview, as needed.
8. Click Save before concluding your session to ensure that the datapoints and source you have set up are correctly stored in the system. After saving the data, you can click Load to harvest initial data for your new source.

x

To update a loadable source:

1. From the Manage tab, click the Sources panel button.
2. Select the source you want to update. The Edit Source panel opens.
3. Make the desired changes. You can rename the datapoints, change their field linkages, format, filters, when conditions, or code. You can also revise dimensional linkages and change Advanced properties of the source.
4. Click Save when you are done making edits. PMF will perform the actions on the source and/or save the changes into each datapoint for the source.

x

If the name of the Master File or other physical connection information used by a loadable source changes, you can adapt those changes into the source in PMF.

1. From the Manage tab, click the Sources panel button.
2. Select the source you want to edit. The Edit Source panel opens.
3. Select the new Master File for the new data source you want to use from the drop-down menu. PMF will automatically clear the fields from the data source, so you can choose the correct new ones from your new source.
4. Click Save to save your changes.

x

1. From the Manage tab, click the Sources panel button.
2. Click New.

   The New Source panel opens.

3. Name the source, and from the first drop-down menu, select Entered by Users. This lets PMF know that the source will harvest data from the user-entered source.
4. Define each user-entered datapoint for this source by entering a name and format for each datapoint. You can quickly enter this information by pressing the Tab key to move from field to field. You can create multiple user-entered sources, and each user-entered source can represent all the datapoints you would collect from a particular user population.

   For example, if you are collecting input from HR staff for HR metrics, you could create a user-entered source called HR Input, and in that source, you could define the datapoints to be collected from that group.

   For each datapoint you define, you can also define the numeric validation format. The PMF input facility will enforce that data format during collection, as shown in the following image.

   

   Note: You can save your work and leave the session at any time. If all the steps are not completed, the Source panel will mark this source as incomplete. End users will not be able to enter data until the source is complete, and incomplete components do not participate in recalculation.

5. Click the Dimension Links tab and define the dimensionality to be collected for the user-entered datapoint. Select the dimensions and levels to link for each, as shown in the following image.

   

6. Click Save.
7. Click the Enter Data tab to input data, as shown in the following image.

   

   Click the small Save button within the Data Entry tab to save the data you entered into the individual datapoints on the Data Entry tab.

Note:

• If you leave any open boxes blank, PMF will not create an intersection for them. That intersection will be treated as a NULL, for example, MISSING.
• Zeros entered will be treated as entered data, and an intersection will be created.
• If you blank out an entered number, its intersection will be deleted when you click Save.

x

User-entered data differs from standard loaded data in the following ways:

• It is dependent on users supplying their data. If users fail to supply their data, no data will be available for the datapoints, or downstream, to be used in any calculations for derived datapoints, or for copying into measures.
• The timing of the data entry is unpredictable. It is therefore difficult to predict when the automatic population of the data should be done.

  As a result, you need to schedule recalculation more frequently.

Note: User-entered data is treated as updated on the date of entry, and the downstream datapoints and measure copies are treated as loaded on the day they were scheduled to update.

x

You can update user-entered sources at any time. PMF adjusts existing user-entered data for the datapoints in a user-entered source as follows:

• Changing the name for any datapoint causes the datapoint to be renamed, with no side effects.
• Adding new datapoints causes them to be added to the source, and to take on the dimensional linkages already configured for that source.
• Changing the format for any datapoint has no effect on the underlying data, because PMF stores all numbers internally in floating-point format. If you change the numeric format to remove or shorten the mantissa (for example, changing the format from D12.2 to D12) when the data is redisplayed for entry, PMF will automatically truncate the mantissa when the data is saved. This might have an effect on values shown for measures on various views and charts in PMF.
• Adding dimensional linkages causes PMF to wipe the data. This is because PMF does not know how to reallocate data that has already been entered.
• If the dimensional level is changed to a lower level for any linked dimension, it causes PMF to wipe the user-entered data for the datapoints. This is because PMF does not know how to reallocate the rolled-up data. The individual dimension-linked values will be lost.
• If the dimensional level is changed to a higher level, PMF attempts to roll the data up. Once the rollup is completed, changing the level back to a lower one will cause PMF to wipe the data, for the same reason as above.