AOH :: P23-08.TXT

Getting Serious About VMS Hacking

                                ==Phrack Inc.==

                      Volume Two, Issue 23, File 8 of 12

                    |                                    |
                    |  Getting Serious About VMS Hacking |
                    |                                    |
                    |     by VAXbusters International    |
                    |                                    |
                    |            January 1989            |

The VAX/VMS operating system is said to be one of the most secure systems
currently available.  It has been massively extended in the past to provide
features which can help system managers getting their machines locked up to
abusers and to trace back any attempts to indiscriminate system security.  As
such, it is not easy getting into VMS machines now without having insider
information, and it's even harder to stay in.

The following article describes some of the internals which make up the VMS
security features, and tries to give hints what to do to remain undiscovered.
The reader should be familiar with the VMS system from the programmer's point
of view.

Some of the things mentioned are closely related to the internal workings of
the VAX/VMS operating system.  All descriptions are held as general as
possible.  It is tried to point out where weak points in the system are
located, not to give step-by-step instructions on how to hack VMS machines.
The main reason for this is, that it is very hard to remain undiscovered in a
VMS system without having good knowledge of the whole system.  This knowledge
is only aquirable by experience.

To use some of the techniques described herein, some literature is recommended:

     "The VAX Architecture Handbook," published by DEC.  This book describes
     the VAX processor, it's instruction set and it's hardware.  It is a good
     book to have on your desk, since it costs nothing (just go to your local
     DEC store and ask for it) and is only in paperback format.

     "MACRO and Instruction Set," part of the VMS documentation kit.  This is
     needed only if you want to program bigger things in MACRO.  It's
     recommended reading, but you don't need to have it on your own normally.

     "VAX/VMS Internals and Data Structures" by L.Kenah and S.Bate.  This is
     the bible for VMS hackers.  It describes the inner workings of the system
     as well as most of the data structures used within the kernel.  The
     Version published always is one version number behind the current VMS
     release, but as the VAX architecture doesn't change, it is the best source
     on a description how the system works.  After you've read and understood
     this book, the VAX won't look more mysterious than your C64.  You can
     order this book from DEC, the order number for the V3.0 version of the
     book is EY-00014-DP.  The major drawback is the price, which is around

A good source of information naturally is the source code of the VMS system.
The easiest way to snoop around in it is to get the microfiche set, which is
delivered by DEC to all bigger customers of the system. The major disadvantage
is that you need a fiche reader to use it.  The fiche is needed if
modifications to the system code are intended, unless you plan to disassemble
everything you need.  The VMS system is written in BLISS-32 and FORTRAN.  BLISS
is quite readable, but it might be worthwhile having a FORTRAN hacker around if
you intend to do patch any of the programs implemented in FORTRAN.  The source
fiche always contains the current release, so it's useful to check if the
information in "Internals and Data Structures" is still valid.

Hacker's Tools
There are several programs which are useful when snooping around on a VMS

The most important utility might be the System Dump Analyzer (SDA), which is
started with the command ANALYZE/SYSTEM.  Originally, SDA was developed to
analyze system crash dumps, which are created every time the machine crashes in
a 'controlled' manner (bugcheck or opcrash).  SDA can also be used to analyze
the running system, which is the more useful function.  A process which wants
to run SDA needs the CMKRNL privilege.  With SDA, you can examine any process
and find out about accessed files and devices, contents of virtual memory (like
typeahead and recall buffers), process status and more.  SDA is a watching
tool, so you normally can't destroy anything with it.

Another helpful tool is the PATCH utility, called up by the command PATCH.  As
VMS is distributed in a binary-only fashion, system updates are normally
distributed as patches to binaries.  PATCHES can be entered as assembler
statements directly.  Combined with the source fiche, PATCH is a powerful tool
for your modifications and improvements to the VMS operating system.

To do interesting things on the VMS system, you normally need privileges.  The
following lists describes some of the privileges which are useful in the
onliner's daily life.

CMEXEC  These two privileges enable a user to execute arbitrary routines with
        KERNEL and EXECUTIVE access mode.  These privileges are needed when one
        plans to access kernel data structures directly.  CMKRNL is the most
        powerful privilege available, everything which is protected can be
        accessed utilizing it.

SYSPRV  A process which holds this privilege can access objects via the system
        protection.  A process holding the this privilege has the same access
        rights as a process running under a system UIC.

SHARE   This allows a process to assign channels to nonshareable devices which
        already have channels assigned to them.  This can be used to prevent
        terminal hangups and to assign channels to system mailboxes.

Process States And The Process Control Block
When you get into kernel hacking, you should pay special attention to the field
PCB$L_STS.  This field tells about the process status.  Interesting bits are
PCB$V_DELPEN, PCB$V_NOACNT and PCB$V_BATCH.  There can be achieved astonishing
effects by setting these bits.

A nice possibility to have is to be unseen by a system manager.  There are many
ways to get invisible to SHOW USERS, but hiding from SHOW SYSTEM is another
story, as it doesn't even use standard system calls to get a list of the
currently running processes.  And in fact, hiding from SDA is even harder,
since it directly peeks kernel data structures.  Anyway, being invisible to
SHOW USERS is useful on small systems, where one user more could ring the alarm
bell of the system operator.

One possibility to do this is to become a subprocess of some non-interactive
job (like a BATCH or NETWORK process).  The other way is to patch the PCB to
become a BATCH process or to delete the terminal name (which makes SHOW USERS
think you are non-interactive as well).  Patching the PCB has a disadvantage:
The system global variable SYS$GW_IJOBCNT which contains the number of
interactive users must be directly decremented before you hide, and MUST be
incremented before you log out.

If you forget this, the interactive job count will be wrong.  If it becomes
negative, strange effects will show up, which will confuse every system

Accounting And Audits
The most nasty thing about VMS since release 4.2 is the security auditing
feature.  It enables the system manager to log almost every security relevant
event he desires.  For example, access to files, login failures and
modification user authorization data base can all be monitored, logged and
written to the system printer.  The first thing to find out in a new, unknown
system is the awareness of the system management.  The status of the accounting
system is easily determinable by the command SHOW ACCOUNTING.  Normally,
everything except IMAGE accounting is enabled.  When IMAGE accounting is also
enabled, this is the first hint to be careful.  The second thing to check out
is the status of the security auditing system.  You need the SECURITY privilege
to execute the command SHOW AUDIT.

If no audits are enabled, and image accounting is not turned on, the system
normally is not set up to be especially secure.  Such systems are the right
playground for a system hacker, since one doesn't have to be as careful as one
has to be on a correctly managed system.

The main intention for running accounting on a system is the need to charge
users for resources (cpu time, printer usage etc.) they use on the machine.  On
the other hand, accounting can be very useful to track down invaders.  Luckily,
accounting information is being logged in the normal file system, and as such
one can edit out information which isn't supposed to be seen by sneaky eyes.
The most important utility to handle accounting files is, naturally, the
ACCOUNTING utility.  It has options to collect information which is stored in
accounting files, print it in a human readable manner, and, most importantly,
edit accounting files.  That is, you can edit all information out of an
accounting file which you don't want to appear in reports anymore.  The
important qualifier to the ACCOUNTING command is /BINARY.

File Access Dates
One way for system managers to discover unwanted guests is to look out for
modified system files.  Fortunately, there are ways to modify the modification
dates in a file's header.  This can be done with RMS system calls, but there is
no easy way to do that with pure DCL.  There are several utilities to do this
kind of things in the public domain, so look out in the DECUS catalog.

OPCOM is a process which logs system and security relevant events (like tape
and disk mount transactions, security auditing messages etc.).  OPCOM receives
messages via a mailbox device, formats them, logs the event in the operator
logfile (SYS$MANAGER:OPERATOR.LOG) and notifies all operators.  Additionally,
it sends all messages to it's standard output, which normally is the system
console device _OPA0:.  When OPCOM is started, one message is sent to the
standard output announcing that the operator logfile has been initialized.
Thus, it's not recommended to kill OPCOM to remain undiscovered, since the
system manager most likely will get suspicious if the operator logfile has been
initialized without an obvious reason.  The elegant solution to suspend OPCOM,
for the time where no operator messages shall come through.  While OPCOM is
suspended, all messages will be buffered in the mailbox device, where every
process with sufficient privilege can read them out, thus avoiding that OPCOM
reads those messages after it is restarted.

There is one problem with this solution though:  OPCOM always has a read
pending on that mailbox, and this read will be there even if the OPCOM process
is suspended.  Unless you're heavily into kernel hacking, there is no way to
get rid of this read request.  As such, the easy solution is to generate an
unsuspicious operator message as soon as OPCOM is suspended.  Afterwards, your
own process (which can be a DCL procedure) reads all subsequent messages off
the OPCOM mailbox until you feel save enough to have OPCOM resume it's work. By
the way, the OPCOM message mailbox is temporary and has no logical name
assigned to it.  You'll need SDA to get information about the device name.

Command Procedures
Timely, you'll need DCL procedures to have some routine work done
automatically.  It is important not to have strange command procedures lying
around on a foreign system, since they can be easily read by system managers.
Fortunately, a command file may be deleted while someone is executing it.  It
is good practice to do so, utilizing the lexical function F$ENVIRONMENT.  If
you need access to the command file itself from the running procedure, just
assign a channel to it with OPEN.

It's not normally a good idea to add new, possibly privileged accounts to a
foreign system.  The better approach is to to use accounts which have been
unused for some months and to hide privileged programs or piggybacks which gain
privilege to the caller by some mechanism.  A piggyback is a piece of code
which is added to a privileged system program, and which gives privileges
and/or special capabilities to callers which have some kind of speciality (like
a special process name, for example).  Be careful not to change file sizes and
dates, since this makes people suspicious.

This file just tries to give an impression how interesting VMS kernel hacking
can be, and what possibilities there are.  It of course is not complete, and
many details have been left out.  Hopefully, it has been useful and/or
interesting lecture.

               (C)opyright 1989 by the VAXBusters International.
   You may give around this work as long as you don't pretend you wrote it.

AOH Site layout & design copyright © 2006 AOH