How-To: Utilize Store & Forward

Jump to Section

What is in this article

  • What is Store & Forward?
  • Store & Forward Settings
  • Store & Forward Examples
  • Store & Forward Considerations

What is Store & Forward?

The Store and Forward mechanism helps prevent data loss during network connectivity issues with the target consumer. When Store & Forward takes effect, the payload is temporarily stored in a built-in SQLite database and the output retries are based on the configured wait interval. Once the connection is healthy again, the data is outputted in a first-in-first-out order, as fast as possible to catch up to the latest data, while maintaining the initially attempted timestamp.

The Store & Forward setting can be found on the connection overview page for each connection. When turned on the Intelligence Hub will create a SQLite file per connection after the first write. This file size can be determined based the max entries per output setting. The max file size is dependent on the max entries per output setting multiplied by the size of each write.

Store & Forward Settings

Store & Forward has three settings;

  • Enabled / Disabled flag
  • Max Entries per Output: Specifies max entries that can be stored in the database
  • Failure Wait Interval: Specifies how long the Intelligence Hub should wait until retrying the connection

Default Settings:

Store & Forward Examples

  • Handling Data Surges

    • In cases where data transmission experiences bursts of activity followed by periods of lulls, Store & Forward can smooth out the data flow. By buffering the incoming data, it ensures that the downstream systems or network bandwidth are not overwhelmed, thereby maintaining a steady flow of information.
    • Store & Forward may not be as effective when dealing with massive volumes of data due to the potential for bottlenecks.

  •  Network Connectivity Challenges

    • Intermittent Disconnects: For scenarios where connectivity to cloud services or databases is unreliable, Store & Forward acts as a buffer, holding data during disconnects and ensuring no data is lost.
    • Random Connectivity Breaks: In situations where outbound connectivity is sporadically interrupted, or there are unpredictable bursts of inbound data, the mechanism effectively queues data, ensuring that all information is eventually transmitted without loss.

Store & Forward Considerations

  • Managing Store & Forward limits

    • Users should carefully determine the Store & Forward limit based on the connectors processing speed and network conditions. This limit is vital in balancing data storage against processing capabilities.

  • Data Loss

    • Store & Forward does not guarantee that you will not lose your data but is intended as an additional fail safe that can be implemented when connection interruptions occur. Implementing Store & Forward is not a setting that is constrained to your configuration but instead on underlying infrastructure.

  • Handling Limit Exceedance

    • In instances where the Store & Forward limit is reached, strategies like increases the limit or adjusting the data flow rate become necessary.

  • System Load

    • Enabling Store & Forward does add additional load to the system the Intelligence Hub is deployed onto. When enabling this setting please consider your machine/environmental resources.

  • Network Stability and Distance Considerations

    • The further the distance between the systems, the higher the likelihood of network instability. Store & Forward can counteract this by storing the data locally until a stable connection is re-established, thus ensuring data integrity across distant systems.

  • Store & Forward Database File Management

    • When you enable Store & Forward a database files is created on a per connection basis. This means that each connection’s data is isolated. If you wish to isolate a specific output create an identical connection with a single output.
    • Turning off Store & Forward will delete the database file.
    • By default Store & Forward files are stored within your runtime directory. You can have these files sent to a dedicated directory at a location of your choice by setting up the storeForwardData directory. Setting up a distinct directory for Store & Forward data can organize your buffered data efficiently.

  • Troubleshooting Store & Forward

    • If Store & Forward Data is invalid or corrupted your connection output might not work as expected. In order to troubleshoot for this, it is recommended you clone your connection, disable Store & Forward, and map the usage of the cloned connection to match the original connection. Doing this will preserve your Store & Forward Database file in the event it is not the cause of your issue.

  • Statistics Tab and Clearing Writes

    • The ‘Statistics’ tab for each connection displays various metrics to assess the connections health and write status. This includes the current health status, details about any errors if the connection is not in a healthy state, and information on pending and dropped writes associated with the connection. Additionally, write statistics for each output will be displayed.
    • Under the ‘Output Statistics’ section, you can select an output and click clear writes to clear the Store & Forward database file for that output. This action is permanent and not reversible.