The Software Tools You can judge a craftsman by looking in his toolbox. The master craftsman has separate tools for each job. New tools replace old ones when the new tool has a better balance, keener edge, or tighter grip. A master craftsman's toolbox will also contain a few special tools he made himself because no stock tool does the job exactly right. Programmers also have tools and toolboxes. Our tools are compilers and editors. Our toolbox is a directory. In it we keep favorite programs and special files such as LOGIN.CMD, EDTINI.EDT, and USERCCL.CCL. One special set of tools available to RSX programmers is known as LBL Software Tools Virtual Operating System or more simply the Software Tools. This package of over 100 utilities provides UNIX-style program development capabilities for RSX systems. A glance at the one line description of each tool listed in Figure 1 does not reveal the power available from the Software Tools. Software Tools features are better illustrated by the command I will use to check spelling of this article: @@ % ospell pro8605.wrd @@@ The above syntax is very familar to any UNIX user. The @@name@@@ specifies the output file. These files are known as standard input and standard output. However, @@ospell@@@ is not some spelling checking utility. Instead, it is the following Software Tools script. @@ tr A-Z a-z | tr !a-z '@n' | sort | uniq | comm -2 dict @@@ The @@ospell@@@ script checks spelling by processing text through a series of utility programs known as filters. The input of one filter is the output of the previous filter. This connection, known as a @@pipe@@@, is denoted by the vertical bar symbol (@@|@@@). The first filter, @@tr@@@ for character transliterate, changes all upper case to lower case. The same program is used again to change all non-letter characters (!a-z) to carriage return/linefeed (@n). These first two commands put each word in this article on a separate line. The resulting input is then fed to the @@sort@@@ utility. The resulting sorted word list is processed by @@uniq@@@ to remove multiple occurrences of words. This list is then compared with the dictionary file. The '-2' switch tells @@comm@@@ to output all words in the second file (implied standard input from @@uniq@@@) which are not found in the first file (the previously sorted dictionary file @@dict@@@). Thus the file @@pro8605.wrd@@@ is a sorted list of all misspelled or unknown words in this article. I simply scan the list for each incorrectly spelled word and use EDT to find and correct the errors. I find this approach to spelling checking to much more productive than an interactive spelling checker. Looking up misspelled words in the dictionary and fixing the mistakes takes the same amount of time. However, While the @@ospell@@@ script is running as a background process, I can be working on some other activity. <<<<<<<< start of potential section to remove >>>>>>>>>>>>> The power of the Software Tools derives from the ability to hook simple tools together to perform useful work. Before looking at other features and uses of the Software Tools, it is useful to look back at their origin. The Software Tools originated in the late 1970's with Brian Kernighan and P.J. Plauger's book @@Software Tools@@@ (Addision-Wesley, 1976). This excellent book teaches good programming "by seeing how significant programs can be made easy to maintain and modify, human-engineered, efficient, and reliable, by the application of common sense and good programming practices." The code presented in the book was not artificial but actual programs and subroutines. Kernighan and Plauger provided complete code for an editor, source file librarian, text formatter, sort utility and a variety of text filters, string functions, and I/O primitives. Kernighan and Plauger primarily addressed good software engineering principles: "keep it simple, build in stages, and let someone else do the hard part". About the same time Software Tools was published, various universities and government laboratories started to look at the problems caused by proliferation of different computers and operating systems. Their user communities were confronted with learning multiple environments when moving from one vendor's system to another or being tied to one vendor's systems because the cost of moving to different systems outweighed advantages. Furthermore, a useful program developed on one system was useless on another. Lawrence Berkeley Laboratory (LBL) attacked the problem from the point of view of a Virtual Operating System or VOS. Their VOS consisted of three components: a standard set of primitives which provide the virtual machine run-time library, utility programs such as compilers, linkers, and editors, and the command language by which users access system resources from a terminal. LBL was working on defining the functionality of their VOS when the Kernighan and Plauger's Software Tools appeared and help resolve most of the design decisions. The Virtual Operating System would run on many different systems. LBL defined sets of primitive functions for file access, I/O, process (task) control, directory manipulation, and program environment. Each system to which VOS was ported would implement these primitives. Utilities based on the virtual machine run-time library would be completely portable. VOS also needed a portable language. Kernighan and Plauger's Rational Fortran or RATFOR solved the problem. Almost all machines, especially in the late 1970's, offered a ANSI 66 Fortran compiler. Unfortunately, it is difficult to write clear, structured code in strict ANSI 66 Fortran. Furthermore, the unique extensions every vendor added to Fortran made source code portability across systems unrealistic. RATFOR solves these problems. RATFOR translates its dialect of Fortran into strict ANSI 66 Fortran code. RATFOR adds structured programming constructs such as @@if-then-else@@@, @@for@@@, @@while@@@, and @@switch@@@ necessary for structured code. The VOS primitives solved the problem of incompatible I/O calls. LBL's final step was to implement the critical portable utility, a user interface to link the simple tool programs together. The UNIX command line processor, called the shell, was used as a model. The Software Tools shell is the glue which makes the entire package incredibly useful. The LBL Virtual Operating System was described by its authors - Dennis Hall, Deborah Scherrer, and Joe Sventek - in the September 1980 issue of the Communications of the ACM. Other universities have also made major contributions to the current tool set, notable Dave Hanson at the University of Arizona and Phil Enslow at Georgia Tech. <<<<<<<< end of section >>>>>>>>>>>>> Describing the Software Tools Virtual Operating System is a little like a blind man feeling his way around an elephant. The package is so rich in functionality that looking at only one aspect will not give you a true picture of the package. Perhaps a look at how the tools fit into a typical day of mine last week will help show some the potential uses you could make. I start using the tools as soon as I log into my RSX system. The last line in my login command file starts the history version of Software Tools shell (@@hsh@@@). The shell passes any command it does not recognize to MCR for normal processing. The combined abilities of the shell, MCR, and the CCL catch-all task let me intermix Unix, MCR, and DCL style command lines almost without thinking. Even if the Software Tools commands are never used, the history shell is still useful because of its command line editting features. Any of the previous 25 command lines can be recalled, editted, and executed as a new command. This features is invaluable for cycling through repetitive commands such as a series of Macro-11 assembles. The first set of Software Tools I use are @@delta@@@, and @@get@@@. These programs implement a simple source file control system. A part of my business is writing and marketing software products. I use these tools to keep an edit history and track of source file changes. For example, I used the following @@get@@@ command to pull the latest version of ladkit.mac from its source control file ladkit.tcs: @@ % get [301,77]ladkit.tcs ladkit.mac @@@ When I have finished making and debugging my changes, @@delta@@@ is used to update the @@tcs@@@ file with all changes made to ladkit.mac. @@Delta@@@ also prompts for an edit history which can also be output using @@get@@@. Two years from now, I will still be able to recreate any version of ladkit.mac which every existed. Later in the day I was debugging some code on another terminal with ODT. The Software Tools includes @@dc@@@, a desk calculator which supports octal and hexadecimal notation and allows variables and simple equations. I used @@dc@@@ as my scratch pad while debugging. I also updated some personal files I keep encrypted using the @@crypt@@@ tool. Because I do not often reference these files, it had been a long time since I had used @@crypt@@@. This is no problem because the Software Tools documentation is kept online. I simple had to type @@man crypt@@@ to get all the usage notes I needed. The Software Tools documentation is complete and very well written. My day ended by looking into a RSX-11M-Plus V3.0 system crash. Simple analysis showed a task using the memory management directives (PLAS) would crash as soon as the first window was mapped. Other tasks using the directives were not affected. Digging a little deeper revealed the operating system had incorrectly assumed the task was using the new fast mapping feature and trashing executive data structures. Specifically, offset H.FMAP in the task header was non-zero. Now the magic question was who stored what into H.FMAP. I got the answer from the Software Tools. I used the following commands to scan all the executive sources looking for the pattern H.FMAP. @@ % set /uic=[11,10] @@@ @@ % ls | lam '-find H.FMAP <' | sh -v @@@ The @@ls@@@ utility outputs a list of files which match the supplied pattern to standard output. In this case, @@ls@@@ outputs all files in [11,10], one file per line. @@Lam@@@ is used to laminate text strings together. Each filename is changed into a @@find@@@ command for the string H.FMAP. Finally, newly constructed command lines are passed to the shell again to be executed. The @@-v@@@ switch simple tells @@sh@@@ to echo each command line so I can watch which files are being searched. @@Find@@@ will output every line containing H.FMAP. The answer turned out to be the task loader (LOADR.MAC). The code would zero H.FMAP when a task was loaded if it did not use fast mapping only for task using external headers. A header loaded into pool was not touched. The particular program causing the problem was built using a /-XH switch. The ability to recursively invoke the shell is used quite often in connection with @@ls@@@ to construct commands to operate on each found file. For example, the following shell commands updated the copyright statement in all my Macro-11 source files for 1986. @@ % ls .mac | ch '%?*$' 'ch <& "(c) 1985" "(c) 1985, 1986" >& | sh @@@ Here, @@ls@@@ find only file which include the pattern @@.mac@@@. The @@ch@@@ is then used to change change each filename into a command to update the copyright text. Software tools pattern matching uses a syntax called regular expressions. Both the @@.mac@@@ and @@'%?*$' clauses are regular expressions. In the case of the later, each character is a special control character. @@%@@@ matches beginning of a line, @@?@@@ matches any character, and @@$@@@ matches the end of a line. The @@*@@@ repeats the preceding the pattern for 0 to n occurrences. Thus the pattern @@@'%?*$'@@@ matches all lines. In the second @@ch@@@ pattern, the @@&@@@ means the entire pattern matched by the first pattern, or in this case, the filename. Each filename is changed to the following and feed to the shell. @@ ch file @@@ <<<<<<<< start of potential section to remove >>>>>>>>>>>>> Once mastered, regular expressions is one of the most powerful features of the Software Tools. My best example shows how the Software Tools help me answer a question about RSX-11M which not even Digital RSX Software Engineering could answer. In early 1981, some systems I was working with ran into critical pool shortages. The question was raised how much memory optional but useful features such as DCL, error logging, and parity memory cost. We ended up with a list of 21 different executive features not needed for the application but included in the current system. What was the potential pool savings for each feature? In thinking through the problem, we realized the Software Tools could be used to provide exact answers. We would use the tools to reduce the RSX executive sources to a data base of how much code was controlled by each conditional symbol combination. The first step was to reduce each source file to a common notation. Each source was proceed by the follow shell command: @@ % mstd file.std @@@ The @@mstd@@@ scripts started the process by converting all text to lower case and changing all tabs to spaces. @@ tr A-Z a-z | detab @@@ The @@m1@@@ and @@m2@@@ scripts continued to standardize the text by removing all text following comments, all labels, trailing blanks, blank lines, and leading spaces. @@ ch ';?*$' | ch '%[a-z0-9$.]+:+' | ch ' +@n' '@n' @@@ @@ find '?+' | ch '%[ ]+' @@@ The @@mopr@@@ then used the extended change utility (@@@xch@@@) to reduce the PDP-11 instruction set to lines of @@(word)@@@. @@ xch spec.pat | xch inst.pat | xch reg2.pat | xch reg1.pat @@@ Each of the .pat file has lines with pairs of regular expressions. The first is the matching pattern and the second is the replacement text. For instances, the inst.pat file has lines like those shown below to convert each PDP-11 instruction name found at the beginning a line to @@(word)@@@. /%adcb/(word)/ /%adc/(word)/ /%add/(word)/ /%ashc/(word)/ The reg2.pat file processed all 2-operand instructions. Its patterns output a second @@(word)@@@ line whenever an instruction did not use Rn, (Rn), (Rn)+, -(Rn), @(Rn)+, or @-(Rn) syntax for the source operand. Reg1.pat repeated the scan for destination and 1-operand instructions. The spec.pat file was used to handle Macro-11 directives. In particular, all .if statements where normalized. The resulting text was not perfect. Hand editting was used to normalize .ASCII, .BLKW and other such statements into @@(word)@@@ notation. However such lines were isolated to only a few modules. The RSX sources now looked like the sample code for DRABO shown in figure 2. The next step was to reduce this code so each line describe how much code was controlled but some set of conditionals. A special tool was written for this step. Its output is shown in figure 3. The code controlled by each conditional level is noted on a separate line. We had now constructed our raw data base. The usefulness of the Software Tools did not stop here. Software Tool scripts could also be used to evaluate how much code was controlled by each conditional symbol. The first step was to use @@xch@@@ and change any constant conditions into either @@@@@ or @@@@@. For instance, since we only cared about systems with memory management, the symbol @@m$$mge@@@ would always be enabled. @@Xch@@@ can also solve the resulting boolean equations. The @@truth.pat@@@ file shown in Figure 4 shows how the various true-false conditions reduce. The @@{...}@@@ is a regular expression for defining a class. The @@$1@@@ then refers to the class. The data base is now ready to be checked for the 21 different features. The shell script below was invoked for each symbol. The @@sh@@@ substitutes the symbol name for the @@$1@@@ argument. @@ find temp @@@ @@ ch ' | truth | elim | good | find -x '(' | sums @@@ @@ ch ' | truth | elim | good | find -x '(' | sums @@@ @@ ch ' | truth | elim | good | find '(' @@@ Our reduced database was stored in exec.min. The first line found all lines using the conditional of interest and outputs to a temporary file. The next line enables the conditional and outputs the amount of code which would result. The third line checks to see if any code is generated when the feature is not present. Finally, the last line outputs any code segments which cannot be resolved. The @@elim@@@ script removes segments which do not generate code, that is true conditionals which evaluate as false and false conditionals which are true. The @@find -x@@@ command finds all lines which do not match the supplied regular expression. @@ find -x '(ift) ' | find -x '(iff) ' @@@ The 'good' script discards conditionals which evaluate to be true. Thus any line remaining without parentheses is code which is controlled by the conditional in question. @@ ch ' (ift) ' | ch ' (iff) ' @@@ The total code can be summed using @@dc@@@ and the @@sum@@@ script: @@ (echo tot=0\ field 6-10 'tot=tot+$1'\ echo tot) | dc @@@ The parentheses group a set of commands together to use as a single filter. The @@@echo@@@ utility outputs its arguments to standard output so the first command sets the dc variable @@tot@@@ to zero. The @@field@@@ utility extracts the specified columns from each input line and outputs using the prototype line using the input in place of the @@$1@@@ parameter. As you can see from Figure 3, column 6-10 is the number of bytes controlled by the set of conditionals. The final @@echo@@@ causes @@dc@@@ to output the accumulated value of @@tot@@@. The above procedure took two days to setup and execute. However, when we were finished we had exact answers to our questions about how much code was generated for different executive features. For instance, the costs of DCL and other CLI's turned out to be insignificant as most of the code is put into executive common's. <<<<<<<< end of section >>>>>>>>>>>>> The Software Tools do have drawbacks. The tools take up disk space and memory. The biggest objection raised by most people is execution speed. The portable nature of the tools does not lend itself towards CPU efficiency. For example, the RSX Software Tools uses temporary files to implement the pipe mechanism. But I find the typical tool only 10% less efficient than the equivalent native program. Also, The tools lend themselves to background and batch processing. The scripts used in the conditional study were run at night and in background mode. We would simple start the scripts at 5:00 pm and get the results the next morning. The portability of Software Tools Virtual Operating System succeeded beyond the wildest belief of its inventors. When you leave RSX for other systems, you will not have to relearn how to use the Tools. You can obtain from the public domain RSX, IAS, VAX/VMS, TOPS-20, UNIX, IBM/CMS, IBM/MVS, Univac 1100, SEL MPX, HP 1000/3000, CP/M and MS-D0S versions. All require the appropriate Fortran compiler for the target machine. There are two different sources for the latest RSX Software Tools version. The software is on the RSX SIG Fall 1983 tape. This two tape set may be ordered for $154 from the DECUS library (11-SP-60). A DECUS library order form can be obtained from DECUS Order Processing, 249 Northboro Road, BP02, Marlboro, MA 01752. The latest VAX/VMS version will be placed on the Spring 1986 tape from the Dallas Symposium. The same RSX distribution as well as the basic tape and any of the other versions may ordered from the Software Tools Users Group. Each tape cost $80. Contact the organization at 140 Center Street, El Segundo, CA 90245. Finally, a Software Tools Catalogue is available from Intelligent Decisions Inc., P.O. Box 50174, Palo Alto, CA 94303 for $8.95. This reference describes work done on the Software Tools on various systems. There is a lot to the Software Tools. But once the tools are mastered, you have solutions to problems which you can take with you were ever you go.