Bin2DataStreamer

The new Bin2DataStreamer component is provided as a modern default alternative to the original BinDataStreamer, which is of course still supported for backward compatibility. Like their predecessors, Bin2 streamers output a binary format, which is compatible with all editions of Data Abstract.

Data Streamer components are responsible for encoding and decoding data packets that are transmitted between server and client applications over the network.

Why Produce an Alternative to BinDataStreamer?

When BinDataStreamer was written for the first version of Data Abstract, data transfer requirements were far different from those needed these days. Although it still provides fairly efficient data transfer, particularly for small loads, it has become increasingly obvious that a replacement was needed because:

  • Over three major versions of the product, the requirement for full backward compatibility made it increasingly difficult to extend and improve.
  • BinDataStreamer was based on streaming functionality provided by Delphi's TReader/TWriter classes, which were not really designed for the type of data transfer needed.
  • For other platforms, quirks in Delphi's TReader/TWriter classes had to be reproduced on .NET and FPC for backwards compatibility.
  • Several of the planned features for 'Vinci' required a much tighter integration with the data streamer, which was not possible with the existing component.
  • BinDataStreamer has a known issue: it can cause deadlock, which can occur in the FormDestroy event, caused while using global critical sections while reading/writing.

Bin2DataStreamer Benefits

The main benefits that users of Data Abstract gain from the new streaming formats are: speed, speed and more speed! Except for the different component name, you will see no logical or conceptual changes in how the streamers are used or configured.

Under the hood there are three core features that the Bin2 streamer benefits from:

New Streaming Format

Bin2DataStreamer provides a completely new well-defined data format.

Each individual data field is stored/read independently. This allows you to parse data in a more efficient and flexible manner. Using custom streaming methods provides much better compatibility between all editions of Data Abstract.

The new streaming format provides efficient validation of the stream structure before it is processed, in order to avoid data corruption. All data in the stream can be validated before it is written to the actual table in the database. Data may be checked by field names, types and size.

Reduction of network traffic

Apart from simplifying data validation already covered in the previous section and general improvements in data encoding that save network traffic, there are two items to discuss here: SendReducedDelta mode and non-transmission of null values.

If SendReducedDelta mode is turned on, it can significantly reduce packet sizes when applying updates, because only key fields and changed fields are added to the delta and sent to the server. If turned off, Bin2DataStreamer streams all fields, similar to the old BinDataStreamer.

In certain circumstances, there is a small downside to turning the feature on: with the fields being transmitted changing row by row, unique SQL needs to be generated for each record, so the process of applying many record changes in one go will not benefit from having a single SQL statement "prepared" and reused. However, this effect is certainly minor for all but large bulk updates.

Non-transmission of nulls values is achieved by analyzing the data row's fields to build a single bit mask which holds a reference to all null fields. This is transmitted instead of individual nulls and can have a significant effect on the streaming size for tables containing many null values.

Components

See Also