July 1987

                           The History and Technology
                   of TSR Terminate and Stay Resident software

                                       by
                                  Steve Gibson
                        InfoWorld TechTalk Columnist and
                       President of Gibson Research Corp.


                                 HISTORY BEGINS
        ----------------------------------------------------------------
        Once upon a time a Big Blue company named IBM decided that it 
        wanted to make personal computers too.  For reasons known only 
        to the Gods of Boca Raton Florida, IBM decided to use the 8088 
        microcomputer from Intel rather than Motorola's vastly "cleaner" 
        68000 (as is used in Apple's macintosh).  Viewed in retrospect, 
        this has turned out to be a ridiculously expensive, i.e. WRONG, 
        decision.  But anyway... IBM journeyed to the West to get its 
        software for this new machine.  

        IBM first visited the company who was then the acknowledged king 
        of operating systems, Digital Research, Inc.  DRI had developed 
        CP/M, the defacto standard operating system for the very popular 
        8080 and Z80 microcomputer systems.  IBM explained that it 
        needed an operating system for the new 8088 microprocessor in 
        their new, blue, personal computer and asked if DRI would please 
        provide them with one.  DRI said, quite truthfully, that they 
        didn't have one.  Big blue must have then asked if they would 
        like to make one, to which DRI must have again said: "No 
        thanks." 

        IBM shrugged and headed North. . . . . 

        Though IBM may have only planned to ask Microsoft for an 8088 
        BASIC language interpreter for their new machine, they found 
        themselves in Washington without an operating system.  Microsoft 
        said they would be pleased to provide IBM with both an 8088 
        operating system and the BASIC language interpreter.  IBM said 
        "Thank you" and said they needed it very soon please. 

        But Microsoft didn't have an 8088 operating system either, and 
        being so very much smaller then than now, didn't have the 
        resources to build one from scratch by IBM's deadline.  So they 
        found an even tinier company called Seattle Computer who had 
        built a little unpretentious operating system for the 8088 and 
        8086 micros called 86-DOS.  It was also known by some as QDOS.  
        QDOS was said to stand for Quick and Dirty Operating System. 

        So Microsoft, then able to meet IBM's 8088 machine timetable, 
        delivered both an operating system (albeit one of unassuming 
        parentage and questionable virtue) and a BASIC interpreter. 

        Everything seemed fine until we all began really using the new 
        operating system.  We wanted additional and rather reasonable 
        things from MS-DOS and PC-DOS.  Things which had not been 
        designed into 86-DOS, like printing in the background while the 
        user was doing other things.  Or supporting RS-232 serial 
        printers, which required routing the printer data out through a 
        serial port instead of the normal parallel printer port. 

        To meet these reasonable demands Microsoft made a series of 
        small changes and additions to the guts of MS-DOS which allowed 
        extra "add-on" code to be "hung" onto the outside of MS-DOS and 
        also provided for limited communication between DOS and the add-
        on code.  Microsoft's idea was that these MS-DOS limitations 
        would be "fixed" by writing some new DOS commands (in the above 
        examples PRINT and MODE) which would make use of special 
        undocumented aspects of their latest version operating system. 

        The problem was that MS/PC-DOS, still really old 86-DOS (QDOS) 
        in disguise, was never built to support multitasking.  This 
        means that it was never meant to serve more than a single master 
        at a time.  A command like PRINT, which can continue to operate 
        even AFTER the user has begun doing something else, could very 
        easily confuse it, since now there could be service requests 
        coming into MS-DOS from two quite different places at once. 

        Since this began making the entire operating system rather 
        fragile and weird, Microsoft's grand plan was to retain all 
        these special hooks and communications paths as a company 
        secret.  They figured that as long as they were the only ones to 
        use these new MS-DOS secrets things wouldn't get out of hand. 

        But Microsoft must have gravely underestimated the cleverness 
        and determination of the development community.  When the MODE 
        command proudly stated: "Resident portion installed" developer's 
        eyebrows shot up!  RESIDENT portion?  Hmmmmmm.  And if the PRINT 
        command could access the disk for file printing when we were 
        messing around with a spreadsheet, why couldn't we build a 
        little resident notepad or two, too? 

        But EVEN after these new resident goodies began appearing in 
        increasing numbers, Microsoft refused to "disclose" anything 
        whatsoever.  They said they "couldn't" document these things or 
        they'd be forced to make all FUTURE versions of the operating 
        system support them in upwardly compatible fashion.  Since these 
        were awkward "kludge" solutions to the real need for a true 
        multitasking operating system, Microsoft's delicate condition 
        can be readily understood. 

        However, developers sensing massive market opportunities were 
        not to be stopped.  Debugging software which allowed programmers 
        to peer inside the still working machine sold like wildfire.  
        Using these debuggers (like PERISCOPE, my personal favorite), 
        developers probed into the very heart of Microsoft's own PRINT 
        and MODE commands, unscrambling and deciphering the undocumented 
        secrets of their operation.  Using the same secrets Microsoft's 
        own developers had used, they learned how to give their OWN 
        programs similar capabilities. 

        In an old episode of Star Trek, Mr. Spock said, after capturing 
        the Romulan's cloaking device, "...military secrets are the most 
        fleeting of all, Commander."  This applies equally well to 
        technological secrets in general, and as we've seen, to Resident 
        Software in particular! 

        It's unfortunate that Microsoft did not have time back then in 
        the beginning to build their own DOS from scratch.  They might 
        have done it right.  But we're stuck with what we have today, 
        and making the most of it is certainly our best strategy. 

        It remains unfortunate, though certainly understandable, that 
        Microsoft isn't willing to disclose the details of MS-DOS's 
        resident software hooks and techniques.  This policy keeps such 
        knowledge quite secret, since even the third-party developers 
        are reluctant to disclose their own hard-won "insider's" 
        knowledge and proprietary techniques for forcing reliable 
        multitasking behavior from good ol' single tasking DOS.  It's 
        quite interesting to speculate that perhaps today even Microsoft 
        does not KNOW how to write MS-DOS multitasking applications as 
        well and robustly as several of the larger commercial resident 
        software publishers!  The tricks required to make DOS multitask 
        are real nasty stomach-turners. 

        So... what are these tricks and techniques?  How DO resident 
        programs "pop up over" applications at the simple press of a 
        "hot key"? 



                       THE TECHNOLOGY OF RESIDENT SOFTWARE
        ---------------------------------------------------------------

        Since we've now know how we got to wherever it is we are, let's 
        answer the those questions posed above.... 

        Two things are required for the functioning of TSR technology:  
        First, the TSR program needs to somehow "hang around" well after 
        any other program has started to run.  Secondly, it has to 
        "know" what's going on in the system to be able to jump in and 
        help out when called upon for action.  It also needs some way of 
        re-assuming control of the machine when the time comes for it to 
        take action. 

        The requirement of continuing RAM residency is satisfied by a 
        service provided by the operating system from which TSR's get 
        their name: Terminate and Stay Resident (TSR). 

        The main operating RAM memory of an IBM compatible PC is a 
        contiguous block beginning at address zero (called the bottom of 
        RAM memory) and extending "upward" for a maximum of 640K bytes 
        (called the top of memory).  Note that 640K is actually 655,360 
        bytes since "Ks" in computerdom are really 1024 bytes, not 1000.  
        Since the very bottom of this large contiguous block of RAM is 
        reserved for system housekeeping purposes, DOS is "booted" by 
        copying it from disk into RAM just "above" this low-RAM region. 

        Once DOS has been loaded, all the space "above" it, to the top 
        of RAM, is available for application program loading and use.  
        We'll call this location just above DOS the LOAD POINT for 
        application software.  So, when a program is started, it is 
        copied from disk into RAM starting at this LOAD POINT, 
        continuing upward until the entire program has been loaded.  The 
        memory available, and allocated, to the program for its own data 
        storage and manipulation extends from the end of itself to the 
        physical top of the RAM memory. 

        Most standard programs word processors, spreadsheets, or 
        databases have "control" of the computer until the user tells 
        the program to terminate.  After performing whatever cleanup and 
        file saving might be required, the program simply informs DOS 
        that it is all finished by saying the equivalent of "I'm all 
        finished now, terminate me please". 

        Responding to this "Terminate" request from a program, DOS 
        simply "forgets" all about the program and whatever data it 
        might have had in RAM.  It displays the command line prompt and 
        awaits another command.  The next program to run is loaded at 
        the same LOAD POINT as before, thus overwriting whatever 
        terminated program was just running, and implicitly making the 
        same amount of memory available to this next program. 

        Now consider the special case of the RESIDENT PROGRAM... 

        It begins innocently, just like any other application program, 
        when its name is typed at the DOS prompt.  DOS copies it from 
        disk into the user's RAM memory starting at the same LOAD POINT 
        as any prior application software, implicitly leaving the rest 
        of RAM memory for the program's own use until done, then hands 
        control of the computer system over to this program. 

        Now things start getting interesting! 

        The resident program will generally say hello to the user 
        announcing its intention to hang around for the duration.  Then 
        it does a few "non-standard things" to the system.  These are 
        the very things MICROSOFT wanted to keep all to itself.  These 
        special "things" keep the program from being ex-communicated 
        from system activities after it terminates.  Then, rather than 
        simply saying "terminate me" to DOS, it says: "I'm xxx bytes 
        long.  Now terminate me but DON'T OVERWRITE ME."  Which is to 
        say: Terminate me, but let me Stay Resident. 

        Upon receiving this request, DOS takes control from the program, 
        moves the software LOAD POINT UPWARD by those xxx bytes, and 
        only THEN returns the DOS prompt to the user.  Since the LOAD 
        POINT has been moved up by "xxx" bytes, all subsequent programs 
        will be loaded AFTER the programs which ASKED to remain 
        resident, thus leaving it undisturbed in the system's memory! 

        Additionally, since it's the resident program which tells DOS 
        how big it is, it is able to declare itself to be ANY LENGTH IT 
        CHOOSES!  This is how programs like SideKick, which are only 39K 
        bytes long while on the disk, can have a 50K notepad file and 
        take up 89K bytes while in RAM!  It effectively LIES to DOS 
        about how big it is, so DOS obligingly leaves it space for 
        itself and for however much working memory it wants!  This is 
        the way resident programs are able to eat up so much of our main 
        RAM memory!  When this process is repeated many time (as is 
        frequently the case) a resident program "stack" is created in 
        main RAM memory. 

        So, we've now seen how programs are able to remain in memory 
        after being "officially" terminated.  The final question is:  
        How are these "terminated" programs able to remain "aware" of 
        the activities around them?... and how are they able to regain 
        control of the computer whenever they see fit to do so?  In 
        other words, how can a notepad or other utility "pop up" ??  To 
        answer these last questions let's step back for a moment, 
        reviewing a bit about how computer's are controlled: 

        A computer's actions are directed by its reading of instructions 
        which are stored in its main memory.  These instructions may be 
        temporary, as in the case of programs loaded from disk into RAM, 
        or permanent, as in the case of instructions permanently wired 
        into ROM chips.  In either case, each instruction has a unique 
        address which is just the number of the byte it occupies in the 
        machine's memory. 

        We cause the computer to perform a given task by having it read 
        a series of instructions which detail the task.  In a sense we 
        CALL upon those instructions whenever we need to exert their 
        intended function.  Such a "CALLED" instruction sequence 
        generally has a clever instruction called RETURN which is able 
        to send the computer back to wherever it was called from! 

        This notion of FUNCTIONS which are being performed by 
        instruction sequences called upon in memory, and which will 
        return to us when finished, is very powerful.  IBM compatible 
        computers come equipped with a BIOS ROM which contains a host of 
        very useful general purpose functions which return control to 
        their caller as soon as they have performed their designated 
        duty.  This is EXACTLY what that often mentioned BIOS ROM is all 
        about!  B.I.O.S. stands for "Basic Input Output System", and 
        consists of a collection of standard functions for reading and 
        writing to the system keyboard, screen, mass storage, and many 
        other standard devices. 

        Naturally, we must know the exact address of these "functional" 
        services in order to CALL them into service.  By convention, the 
        starting address of every built-in function in the IBM PC is 
        located in a predefined table of addresses located at the very 
        start of the computer's RAM memory.  Every function is numbered 
        simply by the location of its address in this table.  This means 
        that in order to call for any built-in IBM PC function we only 
        need to know its standard function number - from this we're able 
        to calculate the location in the table where that function's 
        starting address can be found. 

        There's even a clever computer instruction which lets you 
        directly call upon a function whose ADDRESS is stored at a 
        specific location in this table.  So in effect you can simply 
        ask the computer to call "standard function number 7".  In 
        response to this function request the computer will look into 
        the 7th slot in the table for the address of function number 7, 
        and go there directly, returning to you when the function has 
        been performed.  In computer lingo, a place which contains the 
        address of something else is a POINTER or a VECTOR. 

        Even the electrical hardware of the computer is able to call 
        upon these standard functions.  In fact many of the standard 
        functions in the ROM BIOS are designed to operate hand-in-hand 
        with the PC's hardware.  For example, every time a keyboard key 
        is either pressed or released the standard function number 9, 
        whose address is naturally in location 9 of the Address Table, 
        is called upon by the hardware.  This function is called EVEN IF 
        THE SOFTWARE WAS BUSY DOING SOMETHING ELSE AT THE TIME! 

        This ability of the hardware to INTERRUPT whatever the software 
        might be doing at the time is called a "hardware interrupt".  
        Essentially the keyboard hardware says to the processor: 

        "Hey there CPU, hold on a second, remember where you are and 
        what you are doing, now go perform the function whose address is 
        in slot number 9, and when that function returns to you, go back 
        to whatever is was you were doing when I bugged you just like 
        nothing ever happened." 

        This is EXACTLY how we are able to type ahead of our computers.  
        Even if our spreadsheet is lost in thought recalculating our tax 
        refund, we're able to type ahead of it!  Each depression and 
        release of a key steals a fraction of a moment, a slice of time, 
        away from the spreadsheet's recalc.  The function invoked by the 
        keyboard's hardware interrupt places the key into a type-ahead 
        buffer ... which the spreadsheet will read from as soon as its 
        good and ready ... then returns control to the recalc as if 
        nothing at all had happened. 

        Now imagine what would happen if some program (like our resident 
        one) were to point the POINTER stored in the 9th table location 
        to somewhere else, such as to ITSELF, prior to issuing the 
        Terminate and Stay Resident (TSR) request to DOS! 

        THEN the hardware interrupt occurring at every keyboard action 
        would turn control over to THIS TSR'd RESIDENT PROGRAM instead 
        of to the normal built-in keyboard handling function!  If the 
        resident program wasn't "interested" in that particular keyboard 
        action or keystroke, it could simply pass control along to the 
        place where the POINTER HAD BEEN POINTING BEFORE just as if it 
        hadn't been evesdropping at all!  However, if the key action 
        happened to be THIS resident program's assigned "HOT KEY", it 
        could retain control of the system as long as it wished, before 
        passing it along as if nothing at all had happened! 

        That's all there is to it! 

        As we've seen, the clever application of these various tricks 
        and techniques can significantly expand the power and 
        flexibility of MS-DOS and PC-DOS based personal computers. 

        There are those who feel that resident software is a makeshift 
        temporary solution which is not long for this world.  They say 
        that TSR programs will be eliminated by the presence of full 
        multitasking operating systems.  This author could not disagree 
        more.  I believe TSR programs will be here for the duration as 
        necessary and viable add-ons to otherwise incomplete solutions. 

        So there you have it.  The history and technology of TSR 
        Resident Software in a nutshell.  I hope this gives you a better 
        feel for what's going on under the hood of your machine! 

        Thanks for listening ... and REMEMBER, if you would like to 
        receive notification of other free information, software, and 
        our future TSR and utility products, simply pickup the phone or 
        drop us a note or a card.  We'd be pleased to keep you up to 
        date. 

        Steve Gibson. 
                               <<< END OF FILE >>>