Data Sources
A data source is connected via FTP to parse the content of a text file. Once configured it will generate mappings based on the labels in the file header. There are several settings that need to be set to process the file correctly.
Parser Settings
| Number | Label | Description |
|---|---|---|
| 1 | FTP server | FTP server to collect data from. |
| 2 | Folder path | The folder path on the FTP server to collect files from. |
| 3 | File match | Only filenames matched by the file match pattern will be read by the data source, if the file match is blank, the data source will read all files in the folder. Partial matches are also valid, for example if the file in the FTP folder is called: data-123-abc.csv, then the following file matches are all valid: 123, data-, .csv. Advanced pattern syntax: '*' matches any sequence of characters.'?' matches any single character.'[' { character-range } ']' matches characters in the character-range within the square brackets.'[' [ '^' ] { character-range } ']' matches characters that are not in the character-range within the square brackets.c matches character c (c cannot be: '*', '?', '\\', '[').'\\' c matches character c.Example filename: data-123-abc.csvExample valid matches: data*data*abc*data-???-abc.csvdata-???*data-[0-9][0-9][0-9]-[a-z][a-z][a-z].csvdata-[0-9][0-9][0-9]*data-[^a-z][^a-z][^a-z]* |
| 4 | Delimiter | The delimiter used to seperate the data in the file, this will often be a comma in a CSV type file. |
| 5 | Datetime format | The format of the datetime in the datetime column. For a full example YYYY-MM-DD HH:mm:ss would be 2023-12-24 23:59:00.Here is the complete list of the datetime format tokens: Year YY : 70 71 ... 29 30YYYY : 1970 1971 ... 2029 2030Month M : 1 2 ... 11 12MM : 01 02 ... 11 12MMM : Jan Feb ... Nov DecMMMM : January February ... November DecemberDay D : 1 2 ... 30 31DD : 01 02 ... 30 31Hour H : 0 1 ... 22 23HH : 00 01 ... 22 23h : 1 2 ... 11 12hh : 01 02 ... 11 12Minute m : 0 1 ... 58 59mm : 00 01 ... 58 59Second s : 0 1 ... 58 59ss : 00 01 ... 58 59Factional Second S : 0 1 ... 8 9SS : 00 01 ... 98 99SSS : 000 001 ... 998 999AM/PM A : AM PMa : am pmTime Zone Z : -07:00 -06:00 ... +06:00 +07:00ZZ : -0700 -0600 ... +0600 +0700z or zz : EST CST ... MST PSTUnix Timestamp X : 1360013296x : 1360013296123 |
| 6 | Timezone | Timezone of the data in the file, this can differ from the project timezone. |
| 7 | Datetime/Labels/Data | The column in the file with the datetime, row in the header of the file with the labels of what is in the file and row in the file where the data starts, see file mapping. |
| 8 | Collection Interval | The collection interval in minutes. |
| 9 | Out of date period | If no data is collected in this period, the data parser is flagged as OVERDUE. |
| 10 | Last reading datetime | The most recent reading collected for the parser, also see reprocessing data. |
| 11 | Last file datetime | The most recent datetime of the file collected for the parser, this is based on when the file was edited or updated, not to do with the data within the file. When a file is moved to an FTP server this datetime will be set at that point, also see reprocessing data. |
File Mapping
The file mapping is done after the file parser is set up. The system will connect to the server and pull an example of the first file it can find. This will confirm that the setup has been done correctly and the file you want to parser is collected.
The column of the datetime, row of the labels and the row in which the data starts after the header are selected.
Column
Column format maps each column to a unique sensor input.
Row
Row format includes the sensor in each row which is used to determine which sensor to map each column input to.
Sensor Mapping
Mappings link data to the input of a sensor. When accessed from within a file parser you only have access to the data from that parser.
These can also be configured on individual sensors, which also allows for the data to be connected from other parsers and sensors.
Logs
The data source log can be used to see what data has been collected and whether there has been any issues.
Reprocessing Data
On the data source, we store the timestamp of the last file processed and the last reading in that file. This prevents redundant data processing during each collection. These timestamps can be reset or adjusted to specific dates to reprocess old data. A reset icon is available for each timestamp, and there is a "Run" button at the bottom to initiate a manual collection. You can view the number of files and readings processed in the logs tab.
