The Wiert Corner - irregular stream of stuff: jpluimers

Variant Records are a feature that has been in the Pascal language since Standard Pascal.

A cool page for historic perspective is R3R: Pascal Features in Popular Compilers, hopefully someone will update it to more modern versions of the mentioned compilers.

There is not much official documentation on the Delphi side on this, so below some parts of a case I used for a project that started in 1997 and is still in use to day.

The project involves communication between applications running on PCs (in 1997 there were many written in Borland Pascal 7 and a few in Delphi 3) and AS/400 (now called “System i” after an initial rename to “iSeries”) using CPI-C and APPC (both parts of SNA).

Back then, it was top notch for a few reasons:

• Most of the LAN and WAN was already IP-based, but HOST (back then System/390, now called System z after an initial rename to “zSeries”), AS/400 and 3270 terminal emulation were still SNA based. Which meant using Microsoft SNA Server (meanwhile renamed to HIS or Host Integration Services).
• Both an architect and the board were involved and because of strategic alliances, the solution had to make use of
  • IBM DTAQ data queues
  • NetSoft Router for SNA routing
  • SNA servers in a 2-by-2 failover configuration

  It was impossible to get this stable (the combination of these 3 requirements caused any cobination of high CPU usage, or undefined memory growth, or unreliable communications) which took 4 months to debug with overseas help form Microsoft, NetSoft and IBM to prove that direction – that worked in the labs of the 3 partners – was impractical in the real world.

• After the initial failure, the AS/400 programmer and I sat together without the architect and the board to make up our minds.
  The AS/400 programmer was a very smart guy, and since we had a closet full of IBM Red Books, we decided to see what was available in the SNA protocol. Low and behold, it was very versatile, included authentication, automatic encryption over the wire (very useful for a huge financial), automatic release of resources used by broken sessions and much more.

The communications layer on the PC side is still in Delphi (as of mid 2012 in Delphi XE2) but in 2005 the DOS client software got rewritten in India by a company advertising to be CMM level 6 (impossible) and having deep experience with COOL:Gen (also named as IEF, Composer, COOL:Gen, Advantage Gen, AllFusion Gen, CA Gen) and C/C++.

In India, they had some C/C++ gurus that agreed it would be OK if the communications layer became a DLL and they would write a small C interface inbetween COOL:Gen and the DLL. It ended up that after half a week of me explaining how they should do that in Visual C++ 6 (modern as they were, they still opted to use the 1998 release while it was 2005), I wrote that portion for them as well.

The communication layer on the AS/400 side is still mainly COBOL with a bit of RPG IV.

In 2011, the whole communication part got rewritten on the PC side using a Delphi layer to MQ Series 5.3 (now called WebSphere MQ) that was officially already unsupported, and on the AS/400 side WebSphere MQ v6 with an interface layer still mostly in COBOL and RGP IV.

In retrospect that should have been done a lot earlier, as in 2006 the AS/400 got moved about a 1000 miles away, not a distance that the SNA protocol would easily cope with, not even with an Enterprise Extender. Back then I recommended rewriting the middle layer based on a solution sitting on top of TCP/IP, but they didn’t want to as “it worked fine”.

It took another 4 years for major interruptions to occur and go away spontaneously, and finally not to go away in 2011.

In 2012 another, but shorter, refactoring round was done to bring the PC side to Windows 7 and WebSphere MQ version 7.x, and the AS/400 side to e WebSphere MQ 7.x (and also move the AS/400 a lot closer).

Between 2011 and 2012, there have not been any outages apart from a few configuration issues (where someone forgot to make a couple of AS/400 subsystems automatically start upon AS/400 LPAR reboot).

The really cool thing about the SNA to WebSphere MQ transition is that the communications layer instantly became twice as fast for “normal” communications, an order of magnitude faster on heavy traffic, and orders better in case of communications trouble (now there are even reasonable error messages to the end users).

Back to Variant Records (:

The goal of Variant Records is to map different kinds of memory structures onto the same memory. It is similar to unions in C/C++.

The Delphi implementation is very similar to the Standard Pascal one.

Variant records are like absolute (where you define two variables to have the same memory location):

You use the a variation of case to distinguish the different structures to start at the same memory position with the record.
As with a regular case, the tags need to be unique.

Both support for RTTI and Windows messages relies heavily on variant records.

A record can start as a regular record, and towards the and can have one variant part in a record definition.

The cool thing is that Variant Records are not going away in the NEXTGEN compiler, just look at the documentation of the TTypeData variant record to see the NEXTGEN conditional defines.

Before looking at the example code, lets first list a few limitations.

• You cannot used managed types inside variant records (but you can use them in the regular portion), as the compiler would not know which of the tagged cases it should initialize/finalize.
  So these are out: Strings, Open Dynamic Arrays, Interfaces, Anonymous methods, Variants or Records containing other managed types.
  If you still try to, you get error messages like E2154 or E2418, or E2569.
• I think you can create a generic record as long as you apply a record constraint on the generic parameter.
• In Delphi .NET (now defunct) you could get the E2417 error when the compiler could not determine the record size in the first compile pass.

The example code below starts out with a few simple things:

1. single byte character type TChar (we are dealing with DOS and COOL:Gen, they don’t get Unicode yet)
2. single byte character arrays (zero based called TChar###, 1-based called T1Char###).
3. the TVariantData part (simplified from the communications code, but still shows enough detail on how and why)
   the trick here is that the record needs to be exactly 33 bytes long, and the “Contents” portion takes care of that.
4. the TVariantKey part (shows how ID’s were done in the SNA implementation and are now done in the MQ implementation)
   with this one, we could run both the SNA version and the MQ version at the same time and do refactoring and regression testing.

This is basically all you need to know: look at how the case portions are formulated, you can use it with any ordinal (Boolean, Integer, enumeration) tag.

Have fun with it!

–jeroen

unit VariantRecordUnit;

interface

{ First a few basic types}

const
  TGuidStringSize = 38;

type
  TChar = AnsiChar; { single byte character, as it interfaces with DOS and CoolGen programs through C interface }
  TChar2    = array[0..   1] of TChar;
  TChar8    = array[0..   7] of TChar;
  TChar10   = array[0..   9] of TChar;
  TChar20   = array[0..  19] of TChar;
  T1Char33 = array[1..33] of TChar; { 1-based because the DOS CAS sources expect this }
  TGuidChar   = array[0..TGuidStringSize-1] of TChar;
  TMessageId = array[0..23] of Byte;

{ now the record types }

type
  TVariantData = record
  case Boolean of
    False: (
      ProgramName: TChar10;
      InterChangeFormat: TChar10;
      FunctionCode: TChar2;
      ReturnCode: TChar2;
      ErrorCode: TChar2;
      Zero: TChar2);
    True: (Contents: T1Char33);
  end; { total: 33 bytes }

  TId = packed record
    NetBiosName: TChar20; { historically, as DOS app defined it wrongly }
    TimeStamp: TChar8; { HHMMSShh because a DOS directory name can be no longer than 8 characters }
  end; { total: 28 bytes }

  TVariantKey = packed record
  case Integer of
    0: ( // SNA
      ConversationId: TId; { 28 bytes }
      GuidChars: TGuidChar); { 38 bytes }
    2: ( // MQ
      ConversationIdFiller: TId;
      MessageId: TMessageID); // 24 bytes
  end; { total: 66 bytes }

  TPacket = packed record
    EntryType : Byte;
    ReturnKey : TVariantKey;
    DataType  : Byte;
    Data      : TVariantData;
  end;

implementation

end.

–eof–