.RIGHT MARGIN 80.LEFT MARGIN 5
.FIRST TITLE
.TITLE Sort-11 V3.0
.SUBTITLE B.#Z.#Lederman
.PARAGRAPH
I was very supprised to read Terry Medlin's letter in the December 1984
Multitasker. We have been using V3.0 since the beginning of July, 1984, and
have not had any significant problems.
.PARAGRAPH
My work is in an area where correctly operating software is very
important, so I did considerable testing of V3.0 before passing it on to the
applications people. I found that for sorting very short files, V3.0 was
slower (probably due to the more complicated command parsing code), but for
long files (hundreds or thousands of blocks long) it was slightly faster. A
very big improvement is in the sort specification language. When I first heard
of V3.0 at the spring 1984 symposium (Cincinatti), the DEC speaker polled the
audience and I believe there was at most one other person besides myself
who had ever used the V2.0 specification file, and I've never met anyone else
before or since who has used it;
but with V3.0 use of the specification files has been simplified by
orders of magnitude, and this, plus the fact that you can now merge multiple
input files, means that V3.0 has a wider range of applications than V2.0 had.
V3.0 also has at least one V2.0 problem fixed: it will correctly assign input
and output files to your local SY: when it is not the system SY:, whereas with
V2.0 you always had to put in explicit devices on the input and output file
names. A further convenience is that you do not have to put in a /FO switch:
the output file will default to the same file organization as the input file.
.PARAGRAPH
This is not to say that V3.0 is perfect. One disapointing aspect of
Terry Medlin's letter is that he does not give any details of the problems he
encountered or SPRed; so for the benefit of others, I will describe the few
problems I have seen.
.PARAGRAPH
If you do have V2.0 specification files and use the supplied utility TRN to
translate them to V3.0 specification files, explicit comparisons to numeric
literals might not be translated properly. In my case, comparisons with an
octal constant were always translated into a comparison with a hexidecimal
constant of zero. This is easy to fix by hand, but you should watch out for
it. The default to hexidecimal is also a bit annoying.
.PARAGRAPH
A more serious error is that comparisons between a binary data field and a
numeric constant fail unless the constant is expressed in hexidecimal. The
failure is accompanied by an error message which states that there is an
invalid specification, but does not give the line number. This will occur with
a command file such as: (only part of the file shown)
.BLANK.NO JUSTIFY.NO FILL
/PROCESS=RECORD
/FIELD=(NAME:IDENT, POSITION:1, SIZE:2, BINARY)
/CONDITION=(NAME:TASK__REC, TEST:(IDENT EQ %X0003))
/INCLUDE=(CONDITION:TASK__REC,
	KEY= . . . .
.BLANK.JUSTIFY.FILL
This specification file works correctly, but changing the constant from %X0003
to %D0003 (a decimal constant) will cause the sort to fail. I have not yet
received an answer from DEC from my SPR on this.
.PARAGRAPH
The manual shows fields in specification files seperated with equals signs
("="), but the translation utility uses colons (":"): apparently, both are
acceptable, but only one is shown in the documentation.
.PARAGRAPH
The manual shows comparisons between fields, but not between a field and a
numeric literal. It turns out that the format to enter numeric literals is the
same as for the /PAD command, or as shown in the example given above.
.PARAGRAPH
A minor point is that Sort does not truncate output files. If you normally
sort an entire file, you will never see this problem. If you do use
specification files to reject unwanted records, or reduce the size or
re-format the output record, then the output file may be smaller than the
input file, but it will have the same number of blocks allocated to it as were
in the input file. Pre-allocating blocks in a file makes writing to the file
faster, so that is O.K., and you can always use PIP to truncate the output
file to it's actual size. It does mean that if your output file is very much
smaller than your input file, you will want to truncate the file to avoid
wasting disk space.
.PARAGRAPH
A more interesting set of circumstances was uncovered when I tried to improve
Sort performance through the use of resident libraries. I have found that
other tasks were improved when linked to the RMS resident libraries (rather
than use disk overlays), but Sort-11, as distributed, does not use any of the
resident library options. If nothing else is gained, at least there are fewer
disk overlay reads, and this is useful on moderate to heavily loaded systems.
In some cases, use of the resident libraries will increase the amount of task
space that Sort can use as it's work area, and that also will improve
performance. I found that it was very easy to link Sort to the RMS resident
library, and more particularly to the Supervisor mode resident library, with a
considerable reduction in system load. The actual sorts did not run
significantly faster when tested on an unloaded system (PDP-11/70 with RP06
disks), but when I placed a constant and fairly heavy disk and CPU load on the
system, the improvement was considerable. In some cases, the increased
internal working space allowed sorts to be done entirely in memory, where the
distributed version of Sort had to use work files (Sort V3.0 also has the
advantage of optionally printing out a considerable amount of information on
it's internal state during the sort, and the amount of resources were used). An
incidental benefit is that the task image itself is smaller when it does not
have all of the RMS code in it, so it occupies less space on disk (nice if you
have small disks).
.PARAGRAPH
An even further increase in performance was obtained by building Sort as an
I-and-D space, non-overlayed task: all of the sorts worked, but we found that
if the task image went over a certain size, evidence was obtained that it
might not always work, as it was apparently using signed arithmatic in
calculating it's data area, and it could not cope with a work space greater
than 32Kwords. I do not think it appropriate to repeat all of the SPR
response to these tests, except that DEC might re-consider linking to the RMS
resident libraries. The version of Sort-11 V3.0 that we have been using in
production for several months is linked to the Supervisor mode RMS resident
library. We would perfer to go to the non-overlayed I-and-D space version, but
a bug in VMR will not allow it to be installed (it thinks too many APR's are
being used), and we are still a little worried about it's performance, even
though it passed all of our tests.
.PARAGRAPH
Incidentally, if you are still using Sort-11 V2.0 and have RMS V2.0, you can
link to the RMS resident libraries, especially the Supervisor mode library,
and obtain similar improvements, and it can also be built non-overlayed.  V2.0
cannot be built as an I-and-D space task due to poor code management: there is
data mixed into instruction PSECTs, and other problems. I do not know if V2.0
can be linked to the old RMS V1.8 resident library.
.PARAGRAPH
Sort-11 V3.0 comes with provisions for building a Sort resident library. This
might be useful if you want to call Sort routines from within a program, and
it looks as if it will be easier to use than was V2.0, but we havn't tried it.
The library itself appears to be very poorly organized. Sort-11 is actually
two products, Sort and Merge. Although they appear to share most of their code
(which would be reasonable), nearly every module in the resident library is
duplicated: once for Sort and once for Merge, which seems absurd. The
resulting library is a memory resident overlayed monster, and I cannot believe
that both products which are so similar are so totally different internally.
The library might pay off if you could link Sort itself to it, and there is a
lot of the Sort-11 task code in the library; but DEC does not supply a build
file to link the Sort-11 task to the Sort-11 resident library, and I was not
successful in building a library or a task where this could be done. This
would have been useful for situations where many sorts are being done at the
same time, as physical memory would be saved, but fortunately (for us) we
don't have a shortage of memory.
.PARAGRAPH
We also use PRO-350s here, and I found that there is no real sort utility for
P/OS. (There is sort of a callable utility, but nothing you can get from DCL
or in the tool kit.)# Both versions of Sort-11 (V2.0 and V3.0) will build and
run well on the PRO with the RMS resident library, but I don't know how you
would go about obtaining a license for this setup.