Extended Memory Overlay Handler ------------------------------- Earl Chew 26 Gardenia Street Blackburn Victoria Australia, 3130. This handler supplements the overlay handlers provided in the RT11 SYSLIB. It is modelled upon the RT11 extended memory overlay handler but provides some features not present in the RT11 handler. This handler allows the user to move an entire program, including the root section, into extended memory and thus minimising the low memory requirements of the program. This is especially important in the PRO/RT environment and even allows several large programs to be run concurrently as system jobs. 1.0 INTRODUCTION The recommended method for linking large programs to be run under the PRO/RT environment is to create a dummy program stub which calls the program proper as an extended memory overlay at runtime. This method is very successful when the program proper does not contain overlays. Any global references are thus contained entirely within the extended memory overlay and there are no inter-overlay references for the linker to resolve. Large programs which are overlaid, create special problems for the linker. Inter-overlay references may require the linker to force some psects to be located in the root section, and each unique subprogram call across overlay boundaries will require 4 words of memory to be allocated in low memory for the dummy subprogram stub. In certain applications, the amount of low memory required by a job becomes critical, especially if these jobs are to be run as system jobs. The XHANDL overlay handler solves this problem by forcing the low memory root section to be a constant 332 words in size (excluding RT11 impure area requirements). 2.0 LINKING PROGRAMS USING XHANDL XHANDL requires an extra program module. This module must define the following symbols; V$JSW, V$START and V$STACK. V$JSW defines the job status word and will be inclusively or'ed with the XHANDL jsw requirements. V$START defines the location of the word pointing to the entrypoint of the user's program. V$STACK defines the location of the word pointing to the top of the user's stack. A zero indicates that no explicit stack has been specified and that the program will use the default stack starting at 01000. Note that there is very little stack space available here and programs written in high level languages like C or Pascal will need an explicit stack region (/BOTTOM will not work in the environment provided by XHANDL). Page 2 A typical module would resemble the template shown in figure 1. .title XHANDL Data ; v$jsw == 0 ; .psect .root. v$stack:: .word 0 v$start:: .word start .end Figure 1 Entrypoint template used by XHANDL to determine the requirements of the user's program. In the file specifications given to the linker, it is necessary to specify XHANDL first, followed by the user's files which must include the extra XHANDL module. In addition to this, the /NOBITMAP (/X) option must be specified. A typical overlay command file is shown in figure 2. The user must observe the restrictions listed in the following section. ! rtbldx.com ! ! RT11 link command file for overlayed version of RUNOFF. ! ! The input files are assumed to be on device INP. ! The output files will be put on device OUP. ! r link oup:runoff,oup:runoff=sy:xhandl,inp:startx,inp:startn,inp:runoff/x// inp:gcin,inp:rnodyn,inp:rt11io,inp:cmtab inp:rnort/v:1 inp:hyphen,inp:index,inp:comnd,inp:rncmd/v:1 inp:init,inp:escape,inp:style/v:1 inp:fmtcm,inp:pindx,inp:rnoif,inp:ermsg/v:1 // ^C Figure 2 Bonner Lab Runoff link file. This links an overlaid version of Runoff using XHANDL. STARTX.OBJ is the XHANDL entrypoint file. 3.0 RESTRICTIONS The following restrictions must be observed for XHANDL to function properly. Page 3 3.1 JSW Initialisation The user must not store anything into the JSW using the .asect directive. This will override the defaults placed there by XHANDL and may be catastrophic. Use the V$JSW symbol in the entrypoint module. 3.2 Low Memory Usage XHANDL will load its initialisation code into locations below 400 octal. The user may use this space at execution time but must not load code in this space using the .asect directive. 3.3 Job Type XHANDL forces the program to be virtual and overlaid by setting the appropriate bits in the JSW. Do not attempt to run the program as a privileged job since the XHANDL initialisation code will attempt to load over the interrupt vectors. 3.4 XM Settop The user's program must be linked with at least one extended memory overlay (/V:n:m) or with the /XM settop feature enabled (/V). Failure to do this will result in undefined globals at link time. 3.5 Entrypoint Module One entrypoint module must be present. This module must be placed in the root segment. Undefined globals will result if the entrypoint module is not present. 3.6 Base Address The /BOTTOM (/B) option must not be used when linking the program. In addition to this the user's code must not use the .asect directive to load code into locations greater than 777 octal. These restrictions are present since XHANDL must align the root segment of the user's code at the PAR1 boundary (20000 octal). 3.7 Loading of Overlays During its initialisation phase, the handler will load all virtual overlay regions. This may mean that all overlay disk access will be confined to the initialisation phase only for some programs. Page 4 3.8 TSX+ TSX+ version 5.1 hates programs linked with XHANDL. It does not appear to be a difficult problem to solve, although the author has not had the opportunity to look into this any further at this stage. 4.0 MEMORY USAGE Overlaid programs consist of three sections; the root section, the low memory overlay region and the extended memory overlay region. This is the way the linker will allocate programs. Such a program is illustrated in figure 3. The unmapped regions are present due to PAR alignments enforced by the linker. +-------------------+ | XM settop | |-------------------| | | | unmapped | | | +-------------------+ | | | | | /V overlay region | | | | | |-------------------| | | | unmapped | |-------------------| | | | | | /O overlay region | | | | | |-------------------| | root segment | |-------------------| | overlay tables | |-------------------| | overlay handler | |-------------------| | free memory | |-------------------| 000070 | SYSCOM | +-------------------+ Figure 3 Typical memory allocation for a virtual job linked with low and extended memory overlays. Page 5 The runtime environment is altered subtly when using XHANDL. Memory is divided into four sections; the low memory root section, the extended memory root section, the /O overlay region and the /V overlay region. This structure is illustrated in figure 4. +-------------------+ | XM settop | |-------------------| | | | unmapped | | | +-------------------+ | | | | | /V overlay region | | | | | |-------------------| | | | unmapped | |-------------------| | | | | | /O overlay region | | | | | |-------------------| | root segment | |-------------------| | entrypoint module | |-------------------| | overlay tables | | | |-------------------| 020000 | | | unmapped | | | |-------------------| | overlay handler | |-------------------| 001000 | free memory | |-------------------| 000400 | XHANDL initialise | |-------------------| 000070 | SYSCOM | +-------------------+ Figure 4 Typical memory allocation for a virtual job linked with XHANDL and low and extended memory overlays. The XHANDL code makes up the entire low memory root section. It is for this reason that the low memory requirements of all programs linked with XHANDL is constant. Page 6 The extended memory root section is made up of the user's root section code and the overlay table requirements. This code begins at location 20000 in the address space of the program. The /O overlay region also resides in extended memory and forms a continuous addressing space with the extended memory root section. The /V overlay region resides in extended memory but there may be a gap between the top of the /O overlay region and the next free address. It is sufficient for the user to consider the program image as a normal program with a root section, a low memory overlay region and an extended memory overlay region. The reallocations will be handled automatically by the linker and XHANDL. 5.0 SQUISH Because of the requirements of the linker, there are 14 unused blocks in the executable image. The SQUISH program removes these 14 ununsed blocks and compresses the executable image. The command syntax is: .r squish *outfil=infil if the program resides on the system device or, .run dev:squish *outfil=infil if the program resides on another device. The default input and output extensions are .SAV. 6.0 BUGS Some work should be done to XHANDL to move the start up code above 01000. This has already been done with XVHNDL and the code could just be ported over. 7.0 BUG FIXES 9 March 1987 The .module and .audit macros utilise locations 110 and onwards. Now allow 32(decimal) bytes for this. Previously, use of .module or .audit would result in mysterious memory management faults. Page 7 8.0 CONCLUSION This overlay handler has been used successfully under RT11 V5.02 in conjunction with the DECUS C compiler and Bonner Lab Runoff. APPENDIX A XVHNDL - XHANDL EXTENDED XVHNDL allows massive programs to be loaded and run. It extends the concepts used in XVHNDL by removing the constraints imposed on the subroutine calls in an overlaid program. XVHNDL maintains an internal call stack and tracks the calls in the user program. On each return, XVHNDL checks to ensure that the overlay from which the call emanated is in memory. If not, it is loaded before the return is completed. This feature allows the user to concentrate on the optimal placement of overlays without regard to the sequence of calls. A.1 INTERNAL STACK The internal stack maintained by XVHNDL is located just below address 01000. This stack grows downwards and its lower limit is set at about 0500. This limits the recursion to about 192 calls. This should be sufficient for most overlay applications. For this reason highly recursive calls should not cross overlay boundaries. Keeping such calls within overlay boundaries is also more efficient since there is a large overhead involved when the overlay handler is invoked compared with a normal subroutine call. A.2 JUMPS The overlay handler will not track jumps - it's just not possible. You have been warned. A.3 LINK REQUIREMENTS All the statements about XHANDL apply to XVHNDL. There is an added restriction that the user MUST supply a stack region, ie the word at V$STACK must be non-zero and yield the address of the top of the user's stack. The following two examples are taken from the rogue clone port. In the first example use has been made of the cxvir module. This interacts with XVHNDL to move to stack to low memory giving as much space as XVHNDL - XHANDL EXTENDED Page A-2 possible in extended memory addressing space for the program code and data. r link exe:rogue[-1],map:rogue[-1]=lib:xvhndl,c:clib/x// lib:cxvir,c:suport obj:lowcod,obj:term,obj:random.obj obj:monsin.obj/o:1 obj:termca.obj/o:1 obj:curses.obj/v:2 obj:hit.obj/v:2 .. many lines omitted .. obj:zap.obj/v:2 // ^C Figure 5 Rogue link file. This links humungous executable using XVHNDL and cxvir. All of extended memory addressing space is dedicated to program code and data. A large stack is available for the program at 020000. r link exe:rogue[-1],map:rogue[-1]=lib:xvhndl,c:clib/x/e:2000// lib:cxir,cstk,c:suport obj:lowcod,obj:term,obj:random.obj obj:monsin.obj/o:1 obj:termca.obj/o:1 obj:curses.obj/v:2 obj:hit.obj/v:2 .. many lines omitted .. obj:zap.obj/v:2 // .stk. ^C Figure 6 Rogue link file. This links humungous executable using XVHNDL and cxir. This will use a much smaller stack which will be located in extended memory. The second example shows how the cxir module can be used instead. This will have much smaller low memory requirements. Some extended memory is sacrificed to provide the user's stack. A.4 TRADE OFFS XVHNDL requires a little more memory than XHANDL but gives the user more functionality. It is suggested that XHANDL will satisfy most XVHNDL - XHANDL EXTENDED Page A-3 requirements but XVHNDL will be required to cope with those really huge programs that crop up once in a while. --ooOoo--