TSTool / Command / WebGet
WebGet command retrieves content from a website and writes the content to a local file.
The transfer occurs using binary characters and the local copy is the same as that shown by
View / Source (or View / Page Source) in the web browser.
This command is useful for mining time series data and other content from data provider web sites.
The local file can then be processed with additional commands, for example
ReadDelimitedFile if the content is
Extraneous content (such as HTML markup around text) and inconsistencies in newline characters
\r\n for windows and
\n on other systems) may lead to some issues in processing the content.
Additional command capabilities may be implemented to help handle these issues.
The following dialog is used to edit the command and illustrates the command syntax.
WebGet Command Editor (see also the full-size image)
The command syntax is as follows:
|The Uniform Resource Identifier (URI) for the content to be retrieved. This is often also referred to as the Uniform Resource Locator (URL). Global properties can be used with the
||None - must be specified.|
||The local file in which to save the content. Global properties can be used with the
||Output file will not be written.|
||Name of the processor property to set the retrieved contents. For example, a snippet of data can be set as a property for processing by other commands, in which case the content will be accessed using
||Content will not be set in a property.|
||The HTTP response code returned by the request. This can be used to check whether the request was successful. For example, code
||Property value is not set.|
See the automated tests.