BATCH - Batch Processing for RSX-11M ==================================== User Guide ========== As RSX-11M does not (yet) support batch processing, a small version of it has been developed using the virtual terminal driver (VTDRV) developed by Dave Elderkin. The main purpose of BATCH is simply to run an indirect command file without tying up a terminal, and creating a log file of the output. It also has a facility for supplying data 'inline' to a program. Before BATCH processing can be initiated, the virtual terminal driver must be loaded, and the tasks BATCH.TSK and SUB.TSK installed. Files are then submitted to the processor via a command of the form:- SUBMIT logon-text <$>filename where:- logon-text is any valid command-line to HELLO, containing account and password. $ is specified if the file to be processed is a BATCH data (see later) filename is the name of the command or data file to be processed. The extension must be omitted and defaults to .CMD for command files, and .DAT for data files. The filename may be up to 9 characters long for command files, or 6 for data files. Examples:- SUB 1/10/FRED BUILD SUB PHIL/ $XDAT The command (or data) files must reside under the user's current UIC (irrespective of the account he specifies on the SUBMIT line) and on his current system disk. BATCH will create an output log file (with the same name as the input file but with an extension of .LOG and a version number of 1) on the same disk/UIC. Note that in certain cases the output log file will be in a slightly odd format which, while it will print successfully on the terminal or printer, could cause odd results if edited by any editor other than TECO. The 'data files' mentioned above (and characterised by a leading '$' sign in the SUBMIT line) are files containing instructions/data to be processed by the Batch Processor itself. These files may only contain lines in one of two forms:- MCR commands - These are passed to the Virtual Terminal Driver in the form required by MCR Data Lines - These are identified by a leading ! and are passed (minus the !) to the Virtual Terminal Driver in the form required by most utility programs. If the first character after the ! is a ^ it, too, is omitted and the next character is interpreted as the corresponding control character. For example, a data file containing:- PIP !/LI !^Z would produce a PIP listing in the log file. There is also a small program, called BTQ, which will display the current contents of the batch queue (not including the file currently being processed). It is run without arguments and produces a list of the form:- BATCH WAIT QUEUE PHIL/ ZZZ TT1: 1/10/SYS $DAT TT5: FRED/FRED BUILD TT3: A measure of security is supplied in that if a non-privileged user runs the program, then all of the logon-text after the second / will be suppressed - i.e. if the second format above was used the password would not be printed for a non-privileged user. The following error messages may be output from the BATCH system: 1. SUB -- COMMAND SYNTAX ERROR You made a mistake in your submit line. Try again. 2. SUB -- SEND DATA OR REQUEST ERROR The SUBMIT task could not communicate with the BATCH processor. Probably it is either not installed or has the wrong taskname (should be BAT...). 3. BAT -- I/O ERROR ON XXX FILE An I/O error occurred on one of the files BATCH accesses (CMD, DAT or LOG). Check to see that your input files are readable and that there is enough space on the disk for your LOG file. 4. BAT -- ATTACH FAILURE ON VIRTUAL TERMINAL BATCH could not attach to the virtual terminal VT1:. The most likely reason is that the Virtual Terminal Driver is not loaded. load it and try again. 5. BAT -- USER CANNOT LOG ONTO VT, TRY AGAIN BATCH could not log on to the virtual terminal using the login text you gave it. Probably you typed part of it incorrectly - if so, correct it and try again. 6. BTQ -- BATCH PROCESSOR NOT INSTALLED BTQ cannot list out the Batch Queue as the BATCH task is not installed. If you wish to do any batch processing install BATCH first. 7. BTQ -- OPEN FAILURE ON RSX11M.STB Because of the way BTQ works, the pseudo-device LB: must be pointing to a disk holding the RSX11M.STB file (in [1,54]) for the current system. If this message occurs it probably means that LB: is not correctly set - adjust it and try again. 8. BTQ -- SYMBOL NOT FOUND IN RSX11M.STB If this message ever occurs then you have something seriously wrong with your RSX11M.STB file. I wouldn't go much further without carefully checking it. BATCH - Batch Processing for RSX-11M ==================================== System Guide ============ BATCH is a simple attempt to provide batch processing for RSX-11M. In this context Batch Processing is interpreted as implying:- 1) No terminal is tied up during processing 2) A log of the run is available afterwards 3) Single-streaming of jobs will be provided A facility has also been added whereby input data can be supplied to programs that prompt for it. The intention of this was to facilitate the running of benchmarks, but there has not yet been time to see how successfully it does so. As it seems very likely that RSX-11M will officially support Batch Processing before long (as it is available on RSX-11M-PLUS) no real attempt has been made to make this into a full 'bells-and-whistles' product. Rather it is intended merely to be a useful adjunct to the system. The original code was written by Eric Rosdol in SWS in Vienna and was published in the Software Dispatch in 1978. Several modifications were then made to it by James Downward, who also placed it in the DECUS Library. This version was further modified by Phil Stephensen-Payne as described in the code. The programs also require the use of the Virtual Terminal Driver as originally designed by Dave Elderkin in 1977. Minor chages have been made to it by Phil Stephensen-Payne for it to run on RSX-11M V3.2 and to support this version of BATCH. Again no attempt has been made to turn this into a polished product as it seems likely that virtual terminals will be supported by RSX-11M in the near future. The following sections of this document contain brief discussions and comments on the various elements of the BATCH system. The code is, however, pretty well commented and should be used as the ultimate reference source if any modifications are to be made. 1. SUB - Program to submit a Batch Request ======================================= SUB is the program responsible for submitting a request to BATCH. This is a major area in which this BATCH cannot really perform like a proper BATCH. In the latter the user need only supply a filename and the job will be run with his UIC and privileges automatically. BATCH does not really have this option as the Virtual Terminal driver must (currently at least) be treated like an ordinary terminal. There are a number of ways of coping with this, i.e. 1) Keep the VT terminal always logged in & just change UIC 2) Modify the VT driver to allow BATCH to force its default UIC & privileges 3) Force the user to specify an account/password each time Method 1) would allow any user access to all accounts, and method 2) would be difficult to implement without similar problems. Thus (taking the easy way out) method 3) has been used. The basic format of the Submit command line is therefore:- SUB logintext filename where 'logintext' is any valid account/password combination for HELLO (e.g. 1/2/FRED or FRED/FRED) and filename is the name of the file to be executed. This file is assumed to live, not under the account/system disk corresponding to the login text, but under the account/system disk the user is currently set to. This is because RSX-11M users often have one account they log in under, but then work in several accounts, just resetting their UIC. By adopting the method just described they can then use batch files from any account without having to set up an account entry for each one. (For those users who do stick to the account they log in under, this will of course make no difference.) In general 'filename' is a 9-character filename representing the command file to be executed (it will be given an extension of .CMD). For data files, however (i.e. ones containing embedded data to be passed directly to a prompting task) the filename must be at most 6 characters long and preceded by a '$'. Ones of this type will be given an extension of .DAT. SUB first parses the command-line it is given. It assumes that everything from the first non-blank character, to the first space after that is the text for HELLO, and the only requirements imposed on that text are that it should contain at least one slash (/) and at most two. It then assumes that everything from there to the end comprise the filename, and checks to make sure this contains only valid RAD50 characters. SUB then sets up a 13-word RSX-11M Send/Receive packet as follows:- Words 1 to 7 - Login text (unchanged) Words 8 to 10 - Filename (in RAD50) Word 11 - User's TI: device (special format) Word 12 - User's SY: device (special format) Word 13 - User's UIC (2 octal bytes) If the filename was specified with a leading '$' then that is stripped off, the filename is put in Words 8 & 9 and Word 10 is set to -1 (not a valid RAD50 combination). The 'special format' of the TI: and SY: devices consists of the two-character ASCII code translated as RAD50 plus the unti number in binary - this will adequately cope with units up to 50 (octal) which should be enough. The TI: device is required by Batch for completion reporting, and the SY:/UIC are needed to find the input file and to place the output (.LOG) file. Having built the packet, SUB then sends it to BATCH and request BATCH (in case it is not running). SUB then exits. SUB is built by the command file SUB.CMD. It is a non-privileged task, with only 1 unit and a taskname of ...SUB. 2. BATCH - Program to process Batch Requests ========================================= BATCH is the main program that actually processes the batch requests. It is basically very simple and the major complications are introduced by the different ways in which various MCR tasks react. The major problems are:- 1) Most tasks, when terminating, output a '>' sign, which can be picked up by BATCH, but some (e.g. BYE) do not and a timeout must be put on every read to the VT driver to cope with this. (Note that the driver could be modified to time out such reads automatically, but I was reluctant to change the driver more than I had to.) 2) HELLO and AT., if they hit an error, do not terminate normally, and so it is necessary to check for their error messages as alternate termination possibilities. In addition AT. uses the same format for its error messages as for its informational messages and while none of the latter should occur it was felt that a check should be put in for the DELAYING/CONTINUING combination (output in response to a .DELAY) as, should a user include them in his file, the reason for the file terminating is not at all obvious. 3) Tasks that prompt for input require a different format input line to MCR. It would appear that every line passed to MCR must have a trailing carriage-return, and every line passed to a prompting task must not have a trailing carriage return. Hence special code is needed to distinguish the two cases. 4) Tasks which output a lot of information to the 'terminal' appear to send out 80 bytes chunks with embedded carriage return. This works fine when it appears on the terminal, but looks a bit strange in a file. The method adopted sets up the file so that it will print on a terminal (or printer) as the original utility would - but any attempt to edit the output file (unless you use TECO) can produce some very odd results. [I'm not sure whether this situation is caused by the utilities, MCR or the VT driver.] 5) For the VT driver to succeed, the virtual terminals must pretend that they are terminals (or MCR will get very upset). Because of this, if the system is built with 'Checkpoint during Terminal Input' support, the system will try and checkpoint BATCH during VT I/O if it is made checkpointable - and, of course, the poor VT driver does not have the code to cope with it. BATCH must therefore be made non-checkpointable for the duration of all such I/O. Bearing the above in mind, the processing of BATCH is fairly simple. It dequeues a request from its receive queue (exiting if there are none) and sets itself to the UIC specified and assigns units to the given SY: and TI: devices. It then opens the output file and, if processing a 'data file', the input file. It attempts to log off the virtual terminal (in case it was already logged on) and to log it on with the login text supplied. If that succeeds it sets the virtual terminal to the user's account and writes the headers to the log file, and calls ATPRC or BATPRC to process 'AT. command files' and 'data files' respectively. When they complete it logs out the virtual terminal and closes and truncates the log file and looks for the next request. ATPRC merely sends the line '@filename' to the Virtual Terminal, logs any output it gets, and waits for either an '@' or an AT. error message to be logged, at which point it terminates. BATPRC basically reads the input file a line at a time, passing each one to the virtual terminal, logging it and any reply, and terminating when the input file is finished. 'Data lines' are flagging by a preceding '!' and these are passed to the terminal without the leading '!' and as 'solicited writes' (see the VTDRV writeup later). In addition if the first character of a data line (after the !) is a '^' then it too is omitted and the next character is interpreted as a control character (so that the user may include ^Z in his file). Each line is considered 'acted upon' if the read to the VT driver either times out, is rejected with an IE.EOF (which often means the other unit is trying to read) or if the last character of the input buffer is a '>' - this should be borne in mind if this facility is used with user programs. BATCH is assembled and built with the command file BATCH.CMD. It is a privileged, slave task (/PR:0/SL) which references EXEMC.MLB and RSXMC.MAC. It has a taskname of BAT... and should really have an 'above-average' priority (e.g. 65). 3. BTQ - Batch Queue Lister ======================== BTQ is a small task that will list out the current Batch Queue. Unlike most 'professional' Batch Queue Listers it cannot list the name of the job currently being run - this would require some rather hairy code. BTQ works on the standard principle of switching into system state, finding the TCB of BAT... and hence tracing its Receive Queue. Note that BTQ uses the 'RSX11M.STB Lookup' method described in the writeup of BOILER.MAC. BTQ simply copies the Receive Queue for BAT... into its own buffer, and then outputs a line for each entry, listing the login text, the filename and the originating terminal. An attempt is made at security for the program checks to see if the person who requested the listing is privileged, and if not only lists the login text as far as the second slash (/). This will only work if the submit request used the full HELLO format (e.g. 1/2/FRED) but as I have no concern about security on my system I did not bother to improve on it (i.e. only listing up to the last slash specified) - that is left as a user exercise. BTQ is assembled and built by BTQ.CMD. It is a privileged task (/PR) and references EXEMC.MLB and RSXMC.MAC. It uses only two units and should have a task-name of ...BTQ. 4. VTDRV - The Virtual Terminal Driver =================================== The Virtual Terminal Driver is an essential part of the BATCH system. Basically the driver is very simple in that it sets up the data structures (in VTTAB) for two terminals known as VT0: and VT1:. If the user sends a Write request to VT1: then the driver will use it as a reply to a read from VT0:, or will (usually) pass it to MCR as 'unsolicited input' from VT0:. Conversely, if the user does a read to VT1: then the driver will satisfy it the next time somebody does a write to unit VT0:. Because of the tendency of the system to crash when you start playing with user-written drivers, not a lot of work has been done on the Virtual Terminal Driver to see how it could be improved, or what would happen if various things were changed. For instance it would seem feasible to use this for communicating between two non-privileged tasks (one reading/writing to VT0: and one to VT1:) but I don't guarantee it would work, and have not tried it. It would also be nice if the UCBs could be simplified but, because of the need for the virtual terminals to look to the system like ordinary terminals, this is no easy job and has not been attempted. For more information on the driver the (brave) user is referred to the source code itself. The only changes I have made from Dave Elderkin's original design are:- 1) Updated the UCBs in line with RSX-11M V3.2 changes. 2) Added a 'Solicited Write' feature to the interface to VT1:. A 'Solicited Write' to VT1: will never be passed to MCR as unsolicited input but will instead be held until a read is issued to VT0:. This was implemented for the 'data file' aspect of BATCH as otherwise the prompting task could miss the data because it wasn't fast enough. Note that as this request will hang forever if no read request is issued to VT0: it should probably be timed out. 3) A problem with the original VT driver came if you tried to abort a task running 'from' VT0:. First TKTN tried to output an 'exit' message to VT0: and then MCR tried to output a '>'. As there was no soliciting task, the driver ignored them both and both TKTN and MCR hung the system timing them out (a total of about 15 seconds in which the system hung completely). The driver has been modified to terminate immediately any unsolicited writes from TKTN or MCR. The Virtual terminal driver is built with VTDRV.CMD in the usual way for a Loadable Driver.