Subject: chflags - immutable|appendonly files come to 2BSD (#197 - Part 1 of 14) Index: bin,etc,lib,sys/many_files 2.11BSD Description: Files/directories can not be declared append-only, immutable, archived or not to be dumped under 2.11BSD. dump(8) lacked the ability to bypass files marked for 'nodump'. Upper case only and Hazeltine terminal support made the tty driver (and several utilities - 'vi' and 'getty' to name two) needlessly complicated and larger (especially since those devices are obsolete). open(2) can not provide for atomically obtaining a flock(2)'d file descriptor. open(2)ing with O_NDELAY (now called O_NONBLOCK) would not wait for carrier on the first i/o operation. Obsolete stub system calls 'setdopt' and 'getdopt' were still present in the kernel even though they have never done anything except take up space. Repeat-By: This section does not apply this time since this is an 'update' which adds new functionality rather than fixing an existing bug. Fix: Please read this entire file before proceeding further. It is lengthy but the changes made to the system are widespread. In particular the kernel is heavily modified because several internal calling conventions had to be changed (this is explained in some detail further on). This preamble will not be repeated in each of the following 13 parts of this update. Each of the remaining parts (patches #198 thru #210 inclusive) will contain: 1) A reference back to #197 (this file). 2) A list of files modified by that patch file. 3) The actual patch file, shell script or shar file. This is part 1 of 14 (#197). Contained in this file is is a brief description of the new functionality being added to the 2.11BSD system. The contents of this file are: 1) The introduction (this section). 2) General notes on what was done (my chance to improve my typing skills ;-)). 3) A manifest of the update kit contents - what is in each of the 14 parts. 4) Directions how to apply the update kit. The 4.4BSD 'chflags' capability has been ported (using the 4.4-Lite CD as a source of inspiration and guide) to 2.11BSD. Perhaps an excerpt from the manual page (as modified from 4.4 for 2.11) for chflags(2) would be useful at this point: --------- "int chflags(path, flags) char *path; u_short flags; int fchflags(fd, flags) int fd; u_short flags; The flags specified are formed by or'ing the following values UF_NODUMP Do not dump the file. UF_IMMUTABLE The file may not be changed. UF_APPEND The file may only be appended to. SF_ARCHIVED File is archived. SF_IMMUTABLE The file may not be changed. SF_APPEND The file may only be appended to. The UF_IMMUTABLE and UF APPEND flags may be set or unset by either the owner of a file or the super-user. The SF_IMMUTABLE and SF_APPEND flags may only be set or unset by the super-user. They may be set at any time, but normally may only be unset when the system is in single- user mode. (See init(8) for details.)" ---------- NOTE: the mention of 'single user mode' is premature, the security level features of 4.4BSD have not been implemented yet in 2.11BSD. This is on the TODO list and scheduled for some time in 1995. At present the 2.11BSD kernel runs with "securelevel" (to use 4.4's terminology) set to -1. What might this feature be used for? Logfiles are a good example. You may wish to may a file publically writeable so that any program can log activity to it - but you do not want programs to be able to write anywhere but at the end. By setting one of the APPEND bits all programs must specify O_APPEND when opening the file or the open(2) will fail. Files marked as 'append only' may not be deleted until the append flag is turned off. Directories marked as APPEND may only grow - no files may be removed until the APPEND flag is turned off. Immutable files may not be rename(2)'d, unlink(2)'d, or open(2)'d for writing. Period. Any access except opening for reading will be denied. It is almost eerie to see a file with 666 permissions that you can not remove, overwrite, rename, etc. In the process of modifying the tty driver to support the new calling convention for read/write/ioctl entry points the support for 'LCASE' and 'TILDE' (uppercase only and Hazeltine terminal tilde character handling) were removed for three reasons: 1) the space was needed for more useful features, 2) the logic was simplified (and made a bit faster), 3) terminals which do not know that '~' is an ASCII character or which do not generate lowercase characters are obsolete. Since the kernel's handling of open() was being modified anyways (to support APPEND and IMMUTABLE) it was almost trivial to add O_SHLOCK and O_EXLOCK support to the open(2) syscall. General Notes: The changes to the kernel ended up being _far_ more extensive than originally planned. Originally (back in May/June 1994) when the thought of porting 'immutable/append-only' files struck it appeared to be a simple matter of putting a 'flags' word in the inode and testing a couple bits in several places in the kernel. Alas things rapidly became more complex. The 'append' and 'nonblocking' flags of the file descriptor needed to be passed down from the top level (user program does a read/write) to the lower levels of the kernel. In the case of 'character' devices the 'ioflags' needed to be passed all the way down to the device driver. In addition to 'append' and 'nonblocking' there is the concept of 'synchronous' I/O. Directory I/O is always synchronous - when the kernel writes a directory it must specify the IO_SYNC flag to 'rdwri()'. Thus many of the device drivers had their read, write and ioctl entry points augmented with another argument. The kernel's internal read/write routines - rdwri() and rwip() - were modified to accept the additonal parameter. As if that was not enough the 'uio' structure changed too! The type of request (UIO_READ or UIO_WRITE) is now encapsulated in the 'uio' structure rather than being passed along as a separate calling parameter. This did have a pleasant side effect of reducing the code size in several places. The 'tty' driver was a bit of a mess. The 'uio' handling worked, but had quite a bit of needless code present. Writing to a tty port that was opened in nonblocking mode will now wait for carrier correctly (it did not before). All in all the changes ended up being ~300kb of diffs and new programs. Adding 'chflags' and 'fchflags' to the kernel was the starting goal, but along the way quite a bit of cleanup was done as well. To reduce the size of several modules (so that changes to existing overlay structures would not be needed or would be minimal) much of the 'inlining' (use of ILOCK() instead of ilock() for example) was removed. Some of the macros generated quite a bit of code and the benefit was quite small (performance wise). The function call overhead was likely over estimated or was much higher on the Vax. Even with almost all of the inlining removed from the kernel 2.11BSD has not slowed much at all - kernel recompile times are within seconds of what they always have been. The incore inode structure grows by two bytes (i_flags is a unsigned short). To make room for this (new features should at least attempt to pay their own way!) the following steps were taken: 1) a couple panic strings were shortened or ifdef'd out (since they'd never occured during the last 7 years) 2) repeated error message strings were assigned to a static 'char []' variable (tmscp.c and ts.c are good examples of this). The savings were enough that the GENERIC kernel is actually smaller now than it was before. 3) The 'sccs' string was removed - this saved almost 100 bytes alone! 4) The floating exception table was changed from 'int' to 'u_char' since the codes are all small in size (hey - it's 16 bytes reclaimed!). Manifest: Part Patch Contents ---- ----- ---------- 1 #197 introduction description of the update kit. 2 #198 remove shell script to remove old files and prepare for part 2. 3 #199 new.shar shar file of _new_ sources (chflags.c, new modules for 'ls', and so on) 4 #200 sys1.patch first half of /sys/sys patch file 5 #201 sys2.patch second half of /sys/sys/patch file 6 #202 include.patch patch file for /usr/include. NOTE: this patches /usr/include, _not_ /usr/src/include! 7 #203 pdp.patch patch file for /sys/pdp 8 #204 pdpuba.patch patch file for /sys/pdpuba 9 #205 lib.patch patch file for /usr/src/lib 10 #206 bin.patch patch file for /usr/src/bin 11 #207 ex.patch patch file for /usr/src/ucb/ex 12 #208 etc.patch patch file for /usr/src/etc 13 #209 man.patch patch file for /usr/src/man 14 #210 misc.patch small patch file for remaining (3) programs (tabs, usw.) Directions: And now the moment you have all been waiting for! ;-) To install the update kit make sure you have all the parts. There are 14 files as shown in the table above. Optional (but recommended) step: purge the system of files created by previous patches. This patch kit is large, you may need the space. find / \( -name '*~' -o -name '*.rej' -o -name '*#' \) -print > /tmp/XXX then examine /tmp/XXX to see if there are any files you really wish to preserve (if there are files you want to save, edit /tmp/XXX and remove their names). Finally if all the files in /tmp/XXX are to be deleted do: rm `cat /tmp/XXX` Parts 2 and 3 _must_ be done first and in that order. This is because part 2 creates directories and renames files needed by part 3. A convention I have found useful is to save the parts of the kit into /tmp files and name those files after the patch number - thus you should have the files: /tmp/198, /tmp/199, /tmp/200, ... /tmp/210 if you have the entire kit present. Part 2 is a shell script - you apply it by one of two methods (first removing any mail/news headers): chmod +x /tmp/198 /tmp/198 or sh /tmp/198 Part 3 is a shar file, after removing the headers you install it with: chmod +x /tmp/199 /tmp/199 or sh /tmp/199 ALL of the remaining parts (4 thru 14, files 200 thru 210) are 'patch' files made with absolute pathnames. While not strictly necessary it is recommended that the patches be applied in order like this: patch -p0 < /tmp/200 patch -p0 < /tmp/201 ... patch -p0 < /tmp/210 Now look for any rejected patches. There should be none if the system was currently at patch level 196 before starting the update. find / \( -name '*.rej' -o -name '*#' \) -print will give a list of patches which did not apply correctly. You will need to manually examine the files to find out what was rejected and why. If you get stuck send me a mail item with the details and I will try to help out. IMPORTANT: If any kernel patches fail to apply properly do not attempt to build and use a new kernel! You may proceed with the building of the libraries and utilities. Next you will need to recompile the kernel. NOTE: many of the kernel modules changed size - it is highly probable that the overlay structure will need minor adjustments. Moving 'mem.o' to an overlay (or to a different overlay) is all that was necessary for the GENERIC kernel, your changes may be similarily trivial. You do not need to reconfigure the kernel - a simple recompilation is enough. Then install the kernel and reboot. The 'pstat' program will return garbage until it is recompiled, but the new kernel can be used to perform any future recompilations: cd /sys/YOUR_KERNEL make tags # time to recreate the tags file make mv /unix /ounix # save the old kernel mv /netnix /onetnix # only if making a networking system mv unix / chmod 744 /unix mv netnix / chmod 744 /netnix # only for networking systems After the system successfully reboots you should 'rm /ounix /onetnix'. Optional: remake the GENERIC kernel and install as /genunix. This can be deferred until later but it is something all sites should do. Having a copy of the GENERIC kernel present is inexpensive insurance against losing /unix or /netnix. cd /sys/GENERIC make install -m 744 unix /genunix The next step is to recompile and install the libraries. If you are familiar with the proceedure it is possible to recompile the new/modified files and update the libraries manually. It is HIGHLY recommended that you simply recompile the libraries from source to avoid leaving something out: cd /usr/src/usr.lib/libcurses make make install cd /usr/src/lib/libc make make install cd /usr/src/usr.bin/lint ./libs NOW it is time to recompile and install the new and modified programs: cd /usr/src/bin/sh make make install cd /usr/src/bin/csh make make install cd /usr/src/bin/ls make make install cd /usr/src/bin/chflags make all make install cd /usr/src/bin make stty install -s -m 755 stty /bin/stty cd /usr/src/etc make pstat install -s -m 2755 -g kmem pstat /etc/pstat cd /usr/src/etc/getty make make install cd /usr/src/ucb/ex make make install cd /usr/src/ucb/tset make all make install cd /usr/src/usr.bin make tabs install -m 755 tabs /usr/bin/tabs then clean up: cd /usr/src make clean and lastly the man pages: cd /usr/src/man/man2 make chflags.0 fcntl.0 open.0 install -m 444 chflags.0 fcntl.0 open.0 /usr/man/cat2 ln /usr/man/cat2/chflags.0 /usr/man/cat2/fchflags.0 cd /usr/src/man/man1 make ls.0 install -m 444 ls.0 /usr/man/cat1/ls.0 cd /usr/src/man/man4 make tty.0 install -m 444 tty.0 /usr/man/cat4/tty.0 cd /usr/src/man/man8 make dump.0 install -m 444 dump.0 /usr/man/cat8/dump.0 Congratulations - you are now done. Have fun with 'chflags'!