Subject: Object file symbols limited to 8 characters [+FIX] (#166 - #9 of 19) Index: cc,as,ld,ar,ranlib,nm,nlist,adb,... (2.11BSD) Description: For some time now (seem like eons;-)) the object file format used by Unix for the PDP-11 has restricted symbols to 8 significant characters (actually 7 due to the C compiler prefixing symbols with a leading tilde (~) or underscore (_)). Aside from the "creative constraints" this imposes on the programmer there was the continuing problem of 'name collisions', especially when porting applications from machines whose object file format permitted longer symbol names. Numerous workarounds have been employed in the past. The most common one relied on a combination of a name collision detection program ('shortc') and the flexname capability of the C preprocessor ('cpp'). This served to mask the problem while making debugging difficult due to mangled/synthetic symbol names. Repeat-By: Attempt to compile the following program: int this_is_a_long_name; int this_is_a_long_name_too; main() { exit(0)}; Fix: This section is repeated in each of the 19 parts which make up the update kit. You should read it perhaps once or twice, but then skip over it (how to do that is mentioned below). Taking a "hint" from the a.out(5) man page: "The compiler will note name collisions when they occur within a single file... There is really little that can be done about this. Some thought is being given to modifying the loader to flag detectable collisions, but the real solution would be to change over to the 4BSD a.out format. This would involve modifying the compiler, assembler and adb and then simply porting the 4.3BSD ld, nm, ranlib, strip and nlist. Or perhaps simply porting the entire 4.3BSD suite might be best ... Anyone interested in a project?" This I have done. No more volunteers for the project need apply ;-) The new limit on symbol length is 32 characters! There is still a limit (but it is _much_ more reasonable now) simply because of address space constraints - it needs to be possible to hold at least one of the 'symbol' or 'string' tables in memory in many cases (nice to hold both, but - i know, get a 486;-)). It must be noted though that it is almost trivial now to raise the limit if that is desired - the programs which need to know the maximum length of a symbol string all have an easily changed #define statement now (usually MAXSYMLEN but there are a couple exceptions). The 'string table' format itself doesn't care how long the strings are. The actual a.out format won't have to change again to accomodate a higher limit on symbol name length! The "string table" object file format has been ported and all the necessary changes made throughout the entire system. The changes were *massive* and widespread. Programs affected of course included the assembler and compiler. Other programs affected were anything which accessed a symbol table entry either via nlist(3) [ps, pstat, fstat, vmstat, etc] or by reading object files [ld, ranlib, nm, adb, strip, etc]. The actual changes to the compiler and assembler were minor because those programs had already been modified earlier (updates #142, 143, 152, 153). The compiler only needed to have the maximum size of a symbol name raised. The assembler already knew how to generate 'string table' object files - all that needed to be done in 'as' was to flip a bit telling it to generate the new object format instead of the old style. +++++++++++++++ And now for a bit of a narrative about what was done. The detailed instructions for applying this part (#9 of 19) of the update kit follow the 'story' below. This started out as a semi-organized accounting of what was done but then devolved into a semi-rambling tale due to the sheer bulk of the changes. You can skip to the details for applying #166 by searching for the string "=======" below - this header is replicated in all parts of this kit. +++++++++++++++ Alas, the remaining changes were not so simple. Complete replacements for ranlib(1), ar(1), nlist(3) were ported from the Net-2 release. Other programs such as symorder(1) and two new programs 'symcompact' and 'strcompact' (used to compress/compact symbol and string tables) were written from scratch. Perhaps the two hardest parts of the whole effort were rewriting the linker 'ld' and making *large* modifications to the debugger 'adb'. This was a very difficult job. 'ld needed to scan new style ranlib archives, as well as using the "virtual memory" facility (the 'libvmf' routines posted earlier) for symbol table management and so on. 'adb' was a MESS (having been written in a pseudo block structured macro language). Since the new symbol table entry could be so much larger than the old it was no longer possible for adb to hold as much of the symbol table in memory - an alternate method took a while to develope and implement, more on that in the patch which deals with adb (actually the changes to adb are so large there are two substantial parts of this update kit just for adb!). After the basic programs (ar, ld, ranlib, etc) were running the system had to be completely recompiled from sources, beginning with the object libraries. After those were done the process of recompiling the rest of the system could proceed. Guess what happens when you recreate libc.a with a buggy linker? Yep - the system is rendered useless until backup copies of everything can be reloaded. Don't let this happen to you - be sure (and i'll repeat the point later) to back up the system (or at least key executables and .a files) before installing this upgrade. In all there were about 330 files modified during the change of object file format. Some of these were not directly related to the new object file format. There were a number of (obsolete) references to "BSD2_10" lingering in the system. Those have been replaced with "pdp11" and the 'BSD2_10' define has been removed from the C preprocessor (cpp). DO NOT use 'BSD2_10' to #ifdef pdp-11 sensitive code, use "pdp11" instead. During the recompile of the libraries a fairly large number of "shortened" names were lengthened - these included syscall routines such as "gethostname" which no longer had to be munged into "gethname". Also a surprising number of typographical errors were uncovered (mainly in the Fortran libraries) where an extra character (beyond the 7th character) was left off or accidentally added. These were all fixed and eventually, after a couple evenings, the libraries were built and installed. After the libraries were done it was the application programs' turn to be recompiled. This took the better part of a couple weeks to finally make it thru due to (as it turned out) the iterative nature of the task. A symbol would come up undefined and have to be tracked down exactly where the wrong definition/use was coming from. Finally, however, the task was done and it was time to move on to the kernel. The kernel proved to be suprisingly easy - no real complications arose except when it came time to reboot, a bug had been introduced into 'autoconfig' (who uses 'nlist' to scan the kernel symbol table). Ouch! That was another couple late nights. Since the compiler supports unsigned longs now a number of small changes which ifdef'd 'u_long' to 'long' were removed. REMEMBER - you need to recompile 'autoconfig' and install it before rebooting the new kernel ;-) The performance of 'ps' though (and anything else which used nlist(3), 'fstat', 'w' are good examples) was unacceptably slow. So, amidst other delays (real work, the earthquake - which almost tossed the disc drive to the floor, etc) the "symorder" program was written (with ideas borrowed from the Net-2 version). The symorder(1) program rather insists on holding both the symbol and string tables in memory - this was a problem (or could be if the kernel symbol table grows much more) so two new and original programs were written: 'symcompact' and 'strcompact'. The first program compacts the symbol table by removing 'register' local variables (they're of no use to anyone - the debugger doesn't/can't do anything with them) and redundant global text symbols (symbols in an overlaid program which are in the root segment do not need both the '~' and '_' symbols present). The second program 'strcompact' is one that any 'string table' based object file system can use. It implements "shared strings" for symbols - if a program has many references to 'error' as a local symbol, why store the string 'error' more than once? Simply store one instance and then update the symbol table entries to all point to the same string! Using both 'strcompact' and 'symcompact' on the /unix image resulted in a file that was 15kb smaller. Running 'symorder' then puts the most frequently used symbols at the front of the symbol table, the performance of 'w', 'pstat', and other programs which nlist(3) the kernel was now acceptable. Some of the parts of this kit are large. The large patch files have been split into pieces which the 'patch' will handle, other parts (the replacement 'ar' sources) were left as a single 'shar' file rather than split them up. Each part of this kit consists of: a 'patchfile' - this is used with the "patch" program to update files. an optional 'script' - this is run ("sh script") to perform initialization, remove files, create directories and so on. an optional 'new.sources' - this is a "shar" file containing complete sources for a program. ALL pathnames are _absolute_ - this way you do not have to "cd" around the system, you should be able to apply all the patches while you are in /tmp (or /usr/tmp - wherever you have the most free space). Be sure that you have at least 40mb free on /usr before rebuilding the system - if you do not then building in stages will be necessary. Part 19 contains the detailed instructions for rebuilding the system _after_ the previous 18 patches have been applied. The patches (#158 thru #175) should be applied in order following the directions in each part. DO NOT recompile anything once the patching has begun until requested to do so in part 19. Many of the system include files are modified and the object file format is being changed - recompilation will not be possible until the transformation of the system and object libraries is complete. AT A MINIMUM you will want to back up the following files (unless you have a known good backup already made) in case you need to recompile something before part 19 is done: /bin/ar /bin/ld /bin/nm /bin/as /usr/bin/ranlib /lib/c0 /lib/crt0.o /lib/mcrt0.o /lib/libc.a /bin/nm /usr/include/*.h /usr/include/sys/*.h In part 19 there is a *complete* list of all files affected (all 336 of them) - you may wish to back those up also. And now the common header ('boilerplate') is over (at last ;-)), let the installation guide begin. As always, the complete 2.11BSD updates are available via anonymous FTP to 'ftp.iipo.gtegsc.com' in the directory /pub/2.11BSD ========== #166 (Part #9 of 19) This part updates the following files. BACK THESE UP if you have any worries about the proceedure or do not have a bootable backup already at hand. /usr/src/games/pdp/chess/old.h /usr/src/games/pdp/chess/data.c /usr/src/games/pdp/chess/stat.c /usr/src/games/warp/intrp.c /usr/src/games/warp/Makefile /usr/src/games/snake/busy.c /usr/src/games/hack/config.h /usr/src/games/sail/player.h /usr/src/games/canfield.c 0) Be in a temp directory ("cd /tmp" or "cd /usr/tmp") 1) Save the following shar archive to a file (/tmp/166 for example) 2) Unpack the archive: sh 166 3) Patch the files: patch -p0 < patchfile 6) rm 166 patchfile Part 9 of 19 is done. DO NOT rebuild or compile _anything_ at this point! ====== cut here #! /bin/sh # This is a shell archive, meaning: # 1. Remove everything above the #! /bin/sh line. # 2. Save the resulting text in a file. # 3. Execute the file with /bin/sh (not csh) to create: # patchfile # This archive created: Fri Feb 4 22:50:05 1994 export PATH; PATH=/bin:/usr/bin:$PATH if test -f 'patchfile' then echo shar: "will not over-write existing file 'patchfile'" else sed 's/^X//' << \SHAR_EOF > 'patchfile' X*** /usr/src/games/pdp/chess/old.h.old Sat May 5 00:39:31 1979 X--- /usr/src/games/pdp/chess/old.h Thu Jan 6 21:16:18 1994 X*************** X*** 19,26 **** X X int attacv[64]; X int center[64]; X! int wheur[]; X! int bheur[]; X int control[64]; X int clktim[2]; X int testf; X--- 19,26 ---- X X int attacv[64]; X int center[64]; X! int (*wheur[])(); X! int (*bheur[])(); X int control[64]; X int clktim[2]; X int testf; X*** /usr/src/games/pdp/chess/data.c.old Mon Aug 3 23:23:44 1987 X--- /usr/src/games/pdp/chess/data.c Thu Jan 6 21:15:10 1994 X*************** X*** 16,22 **** X int wheur4(); X int wheur5(); X int wheur6(); X! int wheur[] = X { X &wheur1, X &wheur2, X--- 16,22 ---- X int wheur4(); X int wheur5(); X int wheur6(); X! int (*wheur[])() = X { X &wheur1, X &wheur2, X*************** X*** 33,39 **** X int bheur4(); X int bheur5(); X int bheur6(); X! int bheur[] = X { X &bheur1, X &bheur2, X--- 33,39 ---- X int bheur4(); X int bheur5(); X int bheur6(); X! int (*bheur[])() = X { X &bheur1, X &bheur2, X*** /usr/src/games/pdp/chess/stat.c.old Mon Aug 3 23:19:33 1987 X--- /usr/src/games/pdp/chess/stat.c Thu Jan 6 21:19:05 1994 X*************** X*** 1,5 **** X--- 1,6 ---- X #include "old.h" X X+ int * X statl() X { X int *p1, *p2, *p3; X*** /usr/src/games/warp/intrp.c.old Wed Aug 26 01:13:35 1987 X--- /usr/src/games/warp/intrp.c Sat Jan 1 00:56:44 1994 X*************** X*** 1,8 **** X! /* $Header: /usr/src/games/warp/RCS/intrp.c,v 1.2 87/07/03 00:56:37 games Exp $ X * X * $Log: intrp.c,v $ X * Revision 7.0.1.2a 87/07/03 00:56:37 games X! * Included shortnames.h for BSD2_10 X * X * Revision 7.0.1.2 86/12/12 16:59:04 lwall X * Baseline for net release. X--- 1,11 ---- X! /* $Header: /usr/src/games/warp/RCS/intrp.c,v 1.3 93/12/31 22:40:37 games Exp $ X * X * $Log: intrp.c,v $ X+ * Revision 7.0.2 93/12/31 23:40:37 games X+ * Removed shortnames.h for new version of 2.11BSD X+ X * Revision 7.0.1.2a 87/07/03 00:56:37 games X! * Included shortnames.h for 2.10BSD X * X * Revision 7.0.1.2 86/12/12 16:59:04 lwall X * Baseline for net release. X*************** X*** 14,23 **** X * Split into separate files. Added amoebas and pirates. X * X */ X- X- #ifdef BSD2_10 X- #include X- #endif X X #include "EXTERN.h" X #include "warp.h" X--- 17,22 ---- X*** /usr/src/games/warp/Makefile.old Sun Jan 2 19:58:21 1994 X--- /usr/src/games/warp/Makefile Sun Jan 2 19:58:23 1994 X*************** X*** 217,223 **** X intrp.o: /usr/include/errno.h X intrp.o: /usr/include/fcntl.h X intrp.o: /usr/include/sgtty.h X- intrp.o: /usr/include/short_names.h X intrp.o: /usr/include/signal.h X intrp.o: /usr/include/stdio.h X intrp.o: /usr/include/sys/ioctl.h X--- 217,222 ---- X*** /usr/src/games/snake/busy.c.old Thu May 30 08:44:09 1985 X--- /usr/src/games/snake/busy.c Thu Jan 6 21:25:25 1994 X*************** X*** 4,12 **** X * specifies the terms and conditions for redistribution. X */ X X! #ifndef lint X! static char sccsid[] = "@(#)busy.c 5.1 (Berkeley) 5/30/85"; X! #endif not lint X X /* X * busy: print an indication of how busy the system is for games. X--- 4,12 ---- X * specifies the terms and conditions for redistribution. X */ X X! #if defined(DOSCCS) && !defined(lint) X! static char sccsid[] = "@(#)busy.c 5.1.1 (2.11BSD GTE) 1/06/93"; X! #endif X X /* X * busy: print an indication of how busy the system is for games. X*************** X*** 35,42 **** X exit(0); X } X X! #include X! #include X X struct nlist nl[] = { X { "_avenrun" }, X--- 35,41 ---- X exit(0); X } X X! #include X X struct nlist nl[] = { X { "_avenrun" }, X*** /usr/src/games/hack/config.h.old Sun May 18 04:49:38 1986 X--- /usr/src/games/hack/config.h Sat Jan 1 00:57:03 1994 X*************** X*** 23,29 **** X #define BSD /* delete this line on System V */ X X /* X! * On V7 / BSD2.9 / Ultrix 2.0 and other V7 look-a-likes, a signal trap during X * a read from the tty will cause the read to terminate prematurely with an X * error (EOF) when the signal trap returns. Note that on most version 7 X * type systems, defining `BSD' (above) in addtion to VERSION7 is appropriate. X--- 23,29 ---- X #define BSD /* delete this line on System V */ X X /* X! * On V7 / 2.9BSD / Ultrix 2.0 and other V7 look-a-likes, a signal trap during X * a read from the tty will cause the read to terminate prematurely with an X * error (EOF) when the signal trap returns. Note that on most version 7 X * type systems, defining `BSD' (above) in addtion to VERSION7 is appropriate. X*** /usr/src/games/sail/player.h.old Sun Aug 30 15:37:56 1987 X--- /usr/src/games/sail/player.h Mon Jan 10 21:57:42 1994 X*************** X*** 3,17 **** X * All rights reserved. The Berkeley software License Agreement X * specifies the terms and conditions for redistribution. X * X! * @(#)player.h 5.1 (Berkeley) 5/29/85 X */ X X #include X #include "externs.h" X- X- #ifdef BSD2_10 X- #define initscreen initsreen X- #endif X X /* sizes and coordinates for the screen */ X X--- 3,13 ---- X * All rights reserved. The Berkeley software License Agreement X * specifies the terms and conditions for redistribution. X * X! * @(#)player.h 5.1.1 (2.11BSD GTE) 1/1/94 X */ X X #include X #include "externs.h" X X /* sizes and coordinates for the screen */ X X*** /usr/src/games/canfield.c.old Tue Aug 25 21:59:58 1987 X--- /usr/src/games/canfield.c Mon Jan 10 21:58:04 1994 X*************** X*** 4,18 **** X * specifies the terms and conditions for redistribution. X */ X X! #ifndef lint X char copyright[] = X "@(#) Copyright (c) 1980 Regents of the University of California.\n\ X All rights reserved.\n"; X- #endif not lint X X! #ifndef lint X! static char sccsid[] = "@(#)canfield.c 5.4 (Berkeley) 1/13/86"; X! #endif not lint X X /* X * The canfield program X--- 4,16 ---- X * specifies the terms and conditions for redistribution. X */ X X! #if defined(DOSCCS) && !defined(lint) X char copyright[] = X "@(#) Copyright (c) 1980 Regents of the University of California.\n\ X All rights reserved.\n"; X X! static char sccsid[] = "@(#)canfield.c 5.4.1 (2.11BSD GTE) 1/1/94"; X! #endif X X /* X * The canfield program X*************** X*** 29,42 **** X #include X #include X #include X- X- #ifdef BSD2_10 X- #define printtopbettingbox ptopbetx X- #define printtopinstructions ptopins X- #define printbottombettingbox pbotbetx X- #define printbottominstructions pbotinx X- #define cleanupboard cleanboard X- #endif X X #define decksize 52 X #define originrow 0 X--- 27,32 ---- SHAR_EOF fi exit 0 # End of shell archive