Search                        Top                                  Index
REF EXTERNAL                             John Gibson, Jon Meyer Aug 1992
                                            Revised John Gibson Dec 1996

        COPYRIGHT University of Sussex 1996. All Rights Reserved.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<<                             >>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<<  LOADING EXTERNAL DATA AND  >>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<< CALLING EXTERNAL FUNCTIONS  >>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<<                             >>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

This file, along with REF * EXTERNAL_DATA and REF * DEFSTRUCT, describes
a basic, integrated set of facilities for loading external functions and
data, for calling  functions, and  for accessing data.

         CONTENTS - (Use <ENTER> g to access required sections)

  1   Introduction

  2   Lib External

  3   Overview of External Loading with exload

  4   Syntax of exload
      4.1   Input File Arguments
            ... Unix Systems with Static Linking
            ... Unix Systems with Dynamic Linking
            ... VAX VMS
            ... Alpha VMS
      4.2   Identifier Specifications
            ... First Form
            ... Second Form
            ... The = Form is a Runtime Action
            ... External Names
            ... Note On Variables, Structures and Arrays in VMS
      4.3   Attribute Lists
            ... prefix Attribute
            ... language Attribute
            ... type Attribute
            ... reloading Attribute
            ... batching Attribute
      4.4   Characters in External Names

  5   Calling External Functions
      5.1   External Function Argument Processing
      5.2   Floating-Point Arguments
      5.3   Full Fields in Argument Structures
      5.4   Arguments Passed `By Reference'
      5.5   Example: Passing Arrays
      5.6   Other Types of Arguments
      5.7   VMS: Arguments Passed by Descriptor
      5.8   External Function Results

  6   Re-compiling and Unloading an exload
      6.1   Alpha VMS Note

  7   Saving and Restoring Images
      7.1   Unix
      7.2   VMS
            ... VAX Note On message Utility

  8   Utility Procedures for External Loading

  9   Raw External Loading

 10   External Callback
      10.1  C Interface
            ... Synopsis
            ... Description
      10.2  Callback and Fixed-Address Structures
      10.3  External Function Closures
      10.4  Callback and Abnormal Exit
      10.5  Restrictions on Operations Inside Callback
            ... Processes
            ... Prolog

 11   General Warning




---------------
1  Introduction
---------------

In  Poplog,  external  data  and  functions  are  data  structures   and
function/procedure code associated  with non-Poplog  languages (e.g.  C,
FORTRAN, PASCAL,  etc),  and  which  can  be  dynamically  loaded  (from
operating system object  files) into memory  alongside a running  Poplog
system. External  functions  can  then  be  called,  and  external  data
accessed (or updated), from a Poplog program.

Reasons for wanting  to do this  include the possibility  of doing  some
things more efficiently  than they  can be  done in  Poplog (e.g.  heavy
numerical computation),  or gaining  access to  facilities not  provided
directly by Poplog.

Note that  while these  facilities have  been considerably  expanded  in
Poplog Version  14, and  are  now much  more  powerful, they  are  still
somewhat clumsy  in  respect  of calling  functions  and  procedures  in
languages such as  FORTRAN and PASCAL  (the reason concerns  the way  in
which arguments  are  passed  in  those  languages).  Calling  functions
written in C, on the other hand, is much more straightforward (since its
argument passing mechanism is simpler, and more compatible with Pop-11).

However, the LIB * EXTERNAL facilities, described in HELP * EXTERNAL and
HELP * NEWEXTERNAL, provide  a  higher-level, more  language  orientated
approach to external  loading and calling.  While this too  concentrates
mainly on C, it  also supports a FORTRAN  interface which eases some  of
the problems with calling that  language. (Unfortunately, it still  does
not (as yet) provide any similar support for PASCAL.)



---------------
2  Lib External
---------------

The  identifiers  in  this  section  are  used  by  the   LIB * EXTERNAL
facilities.


external                                                        [syntax]
        The  main   syntax  construct   in  LIB * EXTERNAL   facilities,
        described  in  HELP * EXTERNAL.   LIB * NEWEXTERNAL  is  a   new
        improved version of this, see HELP * NEWEXTERNAL.


c_dec()                                                      [procedure]
        Used  by   LIB * EXTERNAL  and   LIB * NEWEXTERNAL  to   parse C
        declarations. See also HELP * NEWC_DEC.


fortran_dec()                                                [procedure]
        Used by LIB * EXTERNAL to parse Fortran declarations.



-------------------------------------------
3  Overview of External Loading with exload
-------------------------------------------

Object files containing external functions  and data may be loaded  with
the syntax form  exload. To understand  and use this,  you will need  to
have read the following:

    REF * EXTERNAL_DATA
        (at least the  section Representation  of External  Data
        which describes external pointers), and

    REF * DEFSTRUCT
        (which describes type_spec  syntax for external  pointer
        type specifications, as  well as the  exacc syntax  form
        used for calling functions and accessing data).

exload operates by running the operating  system linker to link in  with
the Poplog system user object files produced by language compilers,  and
then provides access to the named external functions and data defined in
those files.

This is an 'additive'  incremental process in the  sense that a  running
list of all external names brought  in by previous loads is  maintained,
and everything accumulated  is accessible  at each new  load. The  input
files specified to a  new exload (plus  any operating system  libraries,
etc) must provide  definitions for  all names that  have not  previously
been loaded (linker error messages will  result if any are undefined  or
multiply defined).

The exload syntax form will  be briefly summarised before describing  it
in detail in the following sections  of this file. It consists of  three
parts, bracketed by  the opening  exload and the  closing endexload,  as
follows:

            exload
                mark_item
                input_file_list
                identifier_specs
            endexload;

The mark_item part is just an item to identify this particular  external
load,  and  may   be  anything   (word,  string,   integer,  etc).   The
input_file_list part  names the  object  files, etc  to be  loaded,  and
consists of a list of strings or words in square brackets, e.g.

            [foo baz grum]

Finally, the  identifier_specs  part  specifies  declarations  for,  and
values to be assigned to, a  set of Pop-11 identifiers (i.e.  variables,
constants, etc).  Each  identifier will  have  assigned to  it  a  value
derived from  an external  name (such  as a  function name),  where  the
external name either defaults to the  same as the identifier, or can  be
specified separately if different.

The value to be assigned  in each case is  specified by a type_spec.  In
the most general form, this can be used to access any value through  the
external name, but an abbreviated form  allows for the most common  case
of loading a simple external pointer.

With a suitable type_spec for an external pointer, exload will also make
an automatic typename  declaration for  the identifier  name (i.e.  with
p_typespec or l_typespec), which allows it  to be used in an exacc  form
without further type declarations.



-------------------
4  Syntax of exload
-------------------

exload                                                          [syntax]
        This construct has the form

                exload
                    mark_item
                    input_file_list
                    identifier_specs
                endexload;

        where the parts are as follow:

        mark_item
            An item that  identifies this particular  exload. It may  be
            any Poplog item (usually a  number, word, string, etc),  but
            must   be   different   from    the   mark_item   for    any
            previously-compiled exload (that is, different according  to
            the Pop-11 "=" operator, not "==").

            This item will come into  play if you re-compile an  exload.
            See Re-compiling  and Unloading  an  exload below  for  more
            details.

        input_file_list
            A square  bracket enclosed  list  of object-file  names  and
            other file specifications (strings or words) required to  be
            loaded. The exact contents of this list are operating system
            dependent; see Input File Arguments below.

            Note that this  argument is optional  (if omitted, then  all
            external names  being loaded  must  be either  present  from
            previous loads, or in default libraries, etc).

            Note also  that this  list  is read  as  a normal  Pop  list
            construct (that is, it may contain 'evaluate' keywords  such
            as %, ^, ^^, etc).

        identifier_specs
            Overall, this part  is like a  sequence of Pop-11  constant,
            vars, lconstant or lvars statements, each ending with a  ";"
            (which is optional for  the last one);  however, to get  the
            current default  constant  or  vars declaration  in  use  by
            define, the declarator keyword may be omitted.

            That is, identifier_specs basically  consists of any  number
            of repetitions of

                declarator
                    identifier_spec,
                    identifier_spec,
                        ...
                    ;

            where declarator is either empty, or one of

                constant  vars  lconstant  lvars

            Each identifier_spec can take one of two forms. The meanings
            of these  are  described  below in  the  section  Identifier
            Specifications for exload:

                identifier_name     type_spec  from_name
                identifier_name  =  type_spec  from_name

            type_spec is optional  in the first  form, but mandatory  in
            the second;  from_name is  optional in  both cases,  and  if
            present consists of  <- (read 'comes  from') followed by  an
            external name, i.e.

                <-  external_name

            Whereas identifier_name  must be  a word,  if  present  then
            external_name  may  be either a word or a quoted  string. If
            external_name is not  present, it  defaults to  the same  as
            identifier_name.

            A bracketed comma  separated list of  attributes may  appear
            anywhere  between  statements  beginning  with   declarator.
            Attributes  specified   in   this   way   apply   to   every
            identifier_spec that follows upto  endexload, or until  that
            attribute is redefined in another attribute list. Attributes
            are used to  control identifier  prefixes and  if, how,  and
            when  identifiers  are  loaded.  Attribute  lists  have  the
            syntax:

                ( attribute, attribute, ... )

            Each attribute is one of:

                # prefix prefix
                # no prefix
                # batching
                # no batching
                # reloading
                # no reloading
                # language name

            Where both prefix  and name  may be either  a string  or
            word. A typical attribute list would therefore be:

                (no reloading, language C)

            The following default attributes:

                (no prefix, batching, reloading, language C)

            are used by exload.


Before completing the description of  exload in the following  sections,
the example below gives an idea of what an exload statement looks like:

            exload 'example'

            (language C)
                malloc(nbytes) :exptr,
                atan2(x, y) :dfloat,
                c_exit(status) :void        <- exit,
                c_printf(string, ...) :int  <- printf,
                errno :int

            endexload;


This uses no object  files, but loads the  standard C library  functions
malloc, atan2,  exit, printf  and  the variable  errno, where  exit  and
printf are loaded under the names  c_exit and c_printf (note the  latter
renaming would be essential, since  printf is already a standard  Pop-11
procedure).  In  each  case,  the  identifiers  have  external  pointers
assigned to them (and typename declarations made for them).

Alternatively, using the prefix attribute, all the above could be loaded
as the Pop identifiers c_name (i.e. c_malloc, ..., c_errno):

            exload 'example'

            (language C, prefix c_)
                malloc(nbytes) :exptr,
                atan2(x, y) :dfloat,
                exit(status),
                printf(string, ...) :int,
                errno :int

            endexload;



4.1  Input File Arguments
-------------------------
The input_file_list argument to exload  may contain strings or words  to
specify object files,  etc to be  loaded; the exact  contents and  their
interpretation differs between Unix and VMS Poplog, as described below.

On all systems, an argument beginning  with == is assumed to specify  an
environment variable/logical  name,  and  is replaced  by  the  list  of
strings (i.e.  the  exlibs result)  returned  by  * sys_translate_exlibs
applied to it.

(Note however, that  this translation  is not done  when compiling  with
Popc. The input_file_list is  merely written to  the output w-file,  and
the corresponding translation of  environment variables is performed  at
link time by Poplink.)

input_file_list may also contain  identifiers whose idvals are  strings,
words or  lists; these  are dereferenced  to their  values, lists  being
processed recursively.


...  Unix Systems with Static Linking
-------------------------------------
This currently includes  older versions  of SunOS  (4.1.X and  earlier),
Ultrix, versions of  Irix prior  to 5.0  and systems  based on  System V
Release 3.

On Unix  system where  static linking  is employed,  exload invokes  the
operating system linker ld to link in relocatable object files and  code
extracted from static archives. Since the  input file list is passed  to
ld, it may contain any valid  arguments for that utility (see  ld(1)). A
word argument (not  beginning with "-")  is taken to  be an object  file
name without an extension, and the  standard extension '.o' is added  to
it. String arguments may  specify library files  (or -L directories)  as
appropriate;      both      the       Poplog      external       library
'$popexternlib/libpop.olb'. and the  C default library  '-lc' are  added
automatically to the end of the list.


...  Unix Systems with Dynamic Linking
--------------------------------------
This currently includes  OSF1, AIX,  HP-UX (post version  8.0) and  Unix
systems based on System V Release  4 (SVR4), including Sun Solaris  2.X,
Silicon Graphics IRIX 5.X and ICL's implementation of SVR4.

On these systems, exload uses the system call interface to the operating
system dynamic linker to  attach a shared object  to the Poplog  process
and make  the functions  and data  defined in  the shared  object/shared
library available to Poplog. The input file list determines which shared
objects will be  opened and  where they  are to  be found.  The list  is
processed by Poplog and may contain a subset of the options supported by
ld: namely, shared object  files, library specifications (-l  arguments)
and additional library search paths (-L arguments). A word argument (not
beginning with "-") is taken to be a shared object file name without  an
extension, and either of the  standard extensions '.so' (SVR4, OSF1  and
AIX) or  '.sl' (HP-UX)  is added  to it.  String arguments  may  specify
library  files  as  appropriate;   both  the  Poplog  external   library
'$popexternlib/libpop.olb'. and the  C default library  '-lc' are  added
automatically to the end of the list.

The  creation  of  a  shared  object/shared  library  as  opposed   to a
relocatable object  file  is quite  trivial.  It has  two  requirements.
Firstly, that the source file  is compiled to position independent  code
(PIC): this  is  accomplished by  an  option to  the  external  language
compiler. Secondly. it requires a call to the operating system linker ld
to create a shared object from  the relocatable object file produced  by
the external language compiler.

Solaris
On Solaris systems,  these two steps  may be combined.  For example,  to
produce a shared  object 'test.so'  corresponding to the  C source  file
'test.c':

        % cc -o test.so -G -K pic test.c

Silicon Graphics
Silicon  Graphics  compilers   generate  position-independent  code   by
default, but building a shared object has to be done in a separate  link
step; so to get the same effect under IRIX 5:

        % cc -c test.c
        % ld -o test.so -shared test.o

OSF1
Similar to  Silicon  Graphics, but  any  C libraries  required  must  be
specified to the link command. In addition, if any of the object modules
use Poplog callback procedures (see External Callback below), these will
result in undefined symbol warnings unless the -expect_unresolved option
is specified for symbols beginning with pop_ :

        % cc -c test.c
        % ld -o test.so -shared -expect_unresolved pop_\* test.o -lc

HP-UX
To produce a shared library 'test.sl' corresponding to the C source file
'test.c' on HP-UX:

        % cc -c +z test.c
        % ld -o test.sl -b test.o

AIX
To produce a shared object 'test.so' corresponding to the C source  file
'test.c' on AIX:

        % cc -c test.c
        % ld -o test.so -G -bexpall -bnoentry test.o -lc

(In AIX, shared objects may also  be modules in archives created by  the
ar utility, i.e. files whose extensions  are '.a'. For -l arguments,  .a
archives are searched for first  before .so files, and  if a .a file  is
found, the module name is assumed to be 'shr.o'.)


For  further  details  see  the  appropriate  operating  system/language
processor manual.

Note that  there is  a subtle  restriction on  SVR4 systems:  given  two
shared objects A and B where B depends on A (e.g. a function defined  in
B calls a function  defined in A), you  can load A on  its own, but  you
can't load B on its own, regardless of whether you've loaded A before or
not. You must explicitly specify the dependency in the exload; e.g.

    exload 'example' ['-lB' '-lA'] ... endexload; ;;; B depends on A

It is actually possible to make  the dependency explicit when you  build
the library. E.g.

    cc -o B.so -G -K pic B.c -lA

Then  you  can  exload  B  on  its  own:  the  linker  will  bring  in A
automatically if it's needed.  However, there may  be a problem  arising
from where the  run-time linker looks  for library A.  A solution is  to
specify a list of library search  directories using the -R option  which
will be passed to the run-time linker. See the manual page for ld(1) for
details.

Note also that it is quite in  order to exload a shared object  multiple
times: exloads  subsequent to  the first  return a  handle to  the  same
instance of the shared object: there is  thus no added cost in space  or
time. This also applies to HP-UX shared libraries.


...  VAX VMS
------------
On VAX VMS, exload invokes the  operating system linker link to link  in
object files. Apart from  linker command qualifiers  (which may also  be
included), the list goes into a VMS linker options file, with one string
per line, and so may  contain any lines valid for  such a file (see  the
VMS Linker Reference Manual). Words are treated the same as strings, and
no default extensions are added to  object file names (since for a  name
with no extension, '.obj' is assumed by the linker anyway). The list may
also contain the names of shareable images to be loaded (see below),  as
well as libraries, etc; the Poplog external library

    popexternlib:libpop.olb/library

is included automatically.

    Linker Command Qualifiers
        Strings beginning with a  `/` character are  taken to be  linker
        command qualifiers, and are extracted from the list and added to
        the link command.  However, you  should not attempt  to use  any
        qualifiers in this context  other than '/nosyslib',  '/nosysshr'
        and '/nouserlib' (and their abbreviations).

    Shareable Images
        A shareable image (specified with 'filename/share' etc), may  be
        either a system image  in sys$share:, or  a private image  file.
        For example,

          'vaxcrtl/share'   ;;; system image "vaxcrtl"
          'myshare/share'   ;;; where "myshare" is a
                            ;;; logical name for a file myshare.exe

        Although filename may include components such as disk, directory
        or extension, only the  actual file name  part is relevant  (and
        the other parts may be omitted).  The VMS linker uses only  this
        part, and looks to see  if it is defined  as a logical name;  if
        not, it assumes the image is  in sys$share. If the logical  name
        is defined, it must translate to the full filename of a  private
        shareable image.

        In other  words,  you  cannot load  a  private  shareable  image
        without  first  having  defined  a  logical  name  for  it,  and
        moreover, the logical name must be the same as the file name.

        Also, each  shareable image  must appear  in a  separate  string
        (i.e. do not  use commas  to separate  two or  more images  in a
        single string, even though this would normally be allowed by the
        linker).

    Linker Special Options
        Of  these,   only  'symbol=',   'psect_attr=',  'collect='   and
        'cluster=' are safe to use (but on no account must 'cluster=' be
        used to specify a base address).


...  Alpha VMS
--------------
The behaviour  of exload  on VAX  VMS systems  is not  supported on  the
Alpha. Instead, you  are restricted  to loading  external symbol  values
from shareable images only  (this is similar in  spirit to Unix  systems
with  dynamic   loading).  exload   uses  the   VMS  library   procedure
lib$find_image_symbol to effect the loading.

Each input_file_list  item  must be  a  word or  a  string  specifying a
shareable image, in the form

        name/share

where the /share qualifier is optional. The name part may be either

    (1) A logical name (optionally ending with ':') which translates  to
        the full filename of a shareable image, e.g.

            'myshare/share'

        where "myshare"  is  defined  as  a  logical  name  for  a  file
        disk:[dir]myshare.exe, etc.

    (2) A filename containing only a  file name part (i.e. no  directory
        or extension), which names a shareable image in sys$share, e.g.

            'sys$ssishr/share'

        refers to sys$share:sys$ssishr.exe.

    (3) A filename containing a directory and/or extension, which  names
        a file in the usual way, e.g.

            'myshare.exe/share'

        means myshare.exe in the current directory.

    (However, note that in standalone  programs compiled with Popc,  the
    /share qualifier is mandatory, and (3) is disallowed. With Popc, the
    shareable image  spec is  processed through  the VMS  linker  (which
    requires /share),  and  the  resulting  executable  would  require a
    logical name to activate any image not in sys$share.)

Unlike the VAX system, input files do not accumulate from one exload  to
the next;  the  input_file_list to  each  exload must  specify  all  the
shareable images required to define the external symbols for that  load.
(Also unlike the VAX system, you  cannot implicitly refer to symbols  in
other shareable images that are used by the ones specified; every symbol
must be a universal of one of the images given.)

Creating Shareable Images
In order to  load object files  into Alpha Poplog,  you must first  turn
them into shareable image(s). The VMS Linker Reference Manual  describes
how to  do  this  in  general; however,  Poplog  provides  a  simplified
approach with the command file

        popcom:linkshare.com

whose main feature is to relieve you of the burden of having to  declare
as "universal" the global symbols you require to access. It simply links
an image in which all global symbols are declared universal.

The format of the command is as follows:

        @popcom:linkshare image-name file1 file2 ... file7

where image-name is the name of the shareable image to link, followed by
upto 7 file arguments. Each file argument may be

      # An object file spec (wildcards allowed, e.g. *.obj);

      # A shareable image name with /share appended;

      # A library name with /library or /include (etc) appended;

      # A linker options  file with  /options appended:  any such  files
        will be  placed last  in  the link.  (If  you have  more  than 7
        arguments, you  can  put  the  remainder  in  an  options  file.
        However, since these are processed by the linker, wildcards will
        not be accepted in object file names.)

      # If required, a  "gsmatch=" option  (note that  this defaults  to
        "gsmatch=lequal,1,1000" if not specified).

All global symbols defined in the object files/object libraries used are
made universal in the output image (and thus accessible with exload).

For example,

        $ @popcom:linkshare myshare.exe mycode.obj

will  turn  the  object  file   'mycode.obj'  into  a  shareable   image
'myshare.exe'.

Callback Shareable Image Library
Note that linking a shareable  image for external loading which  employs
Poplog external  callback (see  External  Callback below)  requires  the
Poplog external library

        popexternlib:libpop.olb/library

in   the   link   command    (this   is   supplied   automatically    by
popcom:linkshare).  libpop.olb  is  a  shareable  image  library   which
(currently) contains only one image, namely

        popexternlib:pop$c_callback.exe

(given by the logical name "pop$c_callback").



4.2  Identifier Specifications
------------------------------
As described above, each identifier_spec in an exload statement controls
the value which will be assigned to one Pop-11 identifier. Ignoring  for
a moment the optional from_name part, this can have two forms:

          identifier_name     type_spec     (type_spec optional)
          identifier_name  =  type_spec     (     "     required)

(In all cases, the actual Pop-11 identifier involved is  identifier_name
with any current  prefix added  -- in  the descriptions  below, this  is
simply referred to as identifier_name.)


...  First Form
---------------
This is  the most  common case,  and is  used when  the 'value'  of  the
external name being loaded is the address of something in memory  (where
this includes functions, data  structures, arrays, variables, etc);  the
Pop-11 identifier  is  assigned an  external  pointer pointing  to  that
object in memory.

If in addition  type_spec is  supplied, then  it specifies  the type  of
object pointed to, and  the identifier name is  declared as a  typename.
That is, a declaration

            x_typespec identifier_name type_spec;

is made (using p_typespec for a permanent identifier, and l_typespec for
a lexical one). This allows the identifier to be used with exacc without
having to specify a type cast.

For example, in the identifier_spec

            atan2(x, y):dfloat

identifier_name  is  "atan2",  while  '(x,y):dfloat'  is  the  value  of
type_spec. This  specifies a  pointer to  a function  with 2  arguments,
returning a double float  result (produced as  a Poplog ddecimal).  This
could be called directly with exacc as follows:

            exacc atan2(x, y) -> ddecimal;

(See External Function Results below for more on function results.)

Alternatively,

            errno :int

loads into errno a pointer to the  C "int" variable of that name,  whose
value could then be accessed or updated with

            exacc errno -> error_code;
            0 -> exacc errno;

etc.


...  Second Form
----------------
Instead of just loading  an external pointer as  in the first form,  the
second form of identifier_spec  using = allows an  actual access on  the
pointer to be performed. It can be used when

    # The value  of the  external name  is not  an address,  but  an
      integer in  its  own right.  Examples  of this  are  the  many
      symbolic status  return codes  used in  VMS (e.g.  SS$_NORMAL,
      whose value is 1). In Unix systems, this kind of external name
      is hardly ever used, because such integer codes are  #define'd
      as macros  in  C header  files  (and the  integer  values  are
      substituted for the names at compile-time).

    # The external name  is the address  of an object,  but what  is
      required in the  Pop-11 identifier  is not a  pointer to  that
      object, but something derived from it, or accessed through it,
      etc.

As for the first  form, the value  of the external name  is put into  an
external pointer record, but then instead  of assigning that to the  Pop
identifier, exacc  is  applied  to  it with  the  given  type_spec;  the
identifier then receives whatever results from that exacc. That is,  the
identifier is assigned the result of

            exacc type_spec temp

where temp contains the external pointer.

Thus for example, whereas

            errno :int

would set errno to a pointer to an integer,

            errno = :int

would actually access the integer (and assign that to errno). Note  that
type_spec can consist of access and/or conversion procedures only, e.g.

            baz = #convert

is valid.

Where the value of the external  name itself is required as an  integer,
this can be got with a `pointer value' access using "^", e.g.

            foo = ^uint

would set identifier foo to the value  of the external name "foo" as  an
unsigned integer.

    In general, since type_spec can include implicit access  procedures,
conversion procedures, etc, etc, the item resulting from the exacc  (and
put in  the identifier)  may turn  out as  anything. However,  so as  to
provide the same automatic typename  declarations as in the first  form,
if the result  of the type_spec  access is a  `typed' external  pointer,
e.g.

        identifier_name = :exptr.{pointer_type}

then as before, x_typespec  is used to  declare identifier_name as  type
pointer_type.


...  The = Form is a Runtime Action
-----------------------------------
An important aspect of the = form not so far mentioned is the following:
the evaluation of

            exacc type_spec temp

(and the assignment of the result  to the identifier) has to be  delayed
by the system until `run-time'.

That is, instead of directly evaluating the exacc at load-time, it,  and
the  assignment  of  the  result   to  the  identifier,  are  done   via
sys_runtime_apply  as  a  runtime   action.  (See  Runtime  Actions   in
REF * SYSTEM.)

There are two reasons for this. The first is that although Poplog  saved
images  can  correctly  save  and  restore  external  name  values  from
operating system  shareable  libraries when  they  are confined  to  the
`original' external pointers, they cannot  guarantee to do the same  for
anything accessed through those pointers (or indeed deal with any  other
external pointer class  structures into which  the external name  values
may get copied). The second  reason is that Popc compilation  similarily
cannot deal  with  anything but  the  original external  pointers.  (The
`original' external pointers  are actually constructed  and returned  by
external_do_load.)

The principal consequence of this delayed action is that, if the current
image has been run with %nort in order to build a saved image, then  any
code that attempts to use the  value of an identifier at load-time  will
fail (since the assignment will not have been done). On the other  hand,
when building a saved image, the current  image MUST be run in this  way
(because if the actions ARE done at compile-time, the saved inage cannot
guarantee to save and restore the results correctly).

Thus compile-time executed  "top level" code should  not attempt to  use
the values from identifiers declared with the = form. Moreover, so as to
allow their assignments to be done at run-time, all such identifiers are
coerced to variables (i.e. lvars or vars as appropriate).

(Note that this issue does not  concern the first form of exload,  since
that only assigns the original external pointers to identifiers.)


...  External Names
-------------------
The optional from_name part at the end of an identifier_spec allows  the
name of  the  Pop-11  identifier  being  declared  and  assigned  to  be
completely different from the external name being loaded (as opposed  to
just having a simple prefix added). When present, it consists of

            <-  external_name

where external_name is generally a word  (but may also be a string,  see
below). E.g.

          the_c_printf_function(string,...):int  <- printf

(N.B. If there is a current prefix specified with the prefix  attribute,
then this  will still  be added  to the  Pop identifier  name even  when
from-name is present.)

You can  use  from_name  if  you simply  want  to  call  the  identifier
something different  from  the external  name,  or  when it  has  to  be
different because  the external  name clashes  with an  existing  Pop-11
identifier (e.g. a built-in  system one). There  are also other  reasons
for using from_name, but before describing these something must be  said
about external symbols as opposed to external names.

By external  symbol is  meant the  name by  which the  operating  system
linker (and other utilities) actually know a given external  definition.
The point here  is that  compilers in  many Unix  systems generally  add
characters to names used in programs,  so that the symbols appearing  in
object files are not quite the same as those used in the original source
code. (The standard 'mappings' in this respect are for the C compiler to
prepend an underscore (so  that name "foo"  becomes symbol '_foo'),  and
for the FORTRAN compiler  to both prepend and  append one (so that  name
"foo" becomes symbol `_foo_`).)

Note that the above does not apply in VMS, where the external symbol  is
always the  same as  the  original name.  The  situation is  similar  in
SVID-compatible Unix systems using COFF format object files, except that
FORTRAN compilers will usually append an underscore (so that name  "foo"
becomes symbol `foo_`).

In any  case,  when  given an  external  name  as a  word,  exload  will
automatically perform any necessary mapping from name to symbol; this is
the normal case where either from_name  is omitted, or is supplied  with
external_name as a word. Supplying external_name as a string says  "this
is the actual external symbol", and bypasses any translation.


...  Note On Variables, Structures and Arrays in VMS
----------------------------------------------------
The treatment of non-procedures such as variables, structures and arrays
in some  VMS compilers  is a  little strange:  unlike procedures,  their
names are not by default declared  as ordinary external symbols, but  as
psects (program sections).

Since Poplog cannot  handle such  declarations, you  must declare  these
structures globaldef in order to  load them. Structures not so  declared
will result in undefined symbol messages.

The problem applies particularly to the C language on the VAX (note, not
Alpha), where the default behaviour is wholly inappropriate (having more
to do with FORTRAN  common blocks than  C). For example,  if you try  to
load my_structure and my_variable from the following

        struct {
            int a;
            char b;
            long c;
        } my_structure;

        int my_variable;

you will get both undefined. They must be redeclared as

        globaldef struct {
            int a;
            char b;
            long c;
        } my_structure;

        globaldef int my_variable;

(and declared globalref in other external files that reference them).

This problem  is  cured  in  the  Alpha  VMS  C  compiler:  by  default,
variables,  structures  and  arrays  are  declared  as  ordinary  global
symbols. (Indeed, the  default mode  of compilation  will issue  warning
messages for globaldef/globalref.)

See VMS language manuals for further details.



4.3  Attribute Lists
--------------------
As well as controlling the current Pop identifier name prefix, attribute
lists in exload  are used  to control  mappings from  external names  to
symbols  (the  language  attribute),   reloading  of  identifiers   (the
reloading attribute),  and  when  loading  is  actually  performed  (the
batching attribute).

All  attributes  apply  to  each  identifier_spec  which  follows,  upto
endexload, or until  that attribute  is redefined  in another  attribute
list. The initial attributes used by exload are:

        (no prefix, no type, batching, reloading, language C)


...  prefix Attribute
---------------------
The current prefix added to all  Pop identifier names can be  controlled
with

            ( prefix prefix )

where prefix can be a word or string (it makes no difference which).


...  language Attribute
-----------------------
In order to  allow exload  to perform  language-dependent mappings  from
external names to symbols, the attribute

            ( language name )

can be used, where  name can be  a word or  string (makes no  difference
which).

You can always supply  a language attribute if  you wish, but  currently
there is only one  case in which exload  actually takes any notice  of a
specific language name.  This is "FORTRAN"  in (non-COFF) Unix  systems,
where (as noted  above), the mapping  from name to  symbol is  different
from the C mapping (the C one being the default for such systems).


...  type Attribute
-------------------
The operating system linker may need to be told whether a given external
symbol is a code symbol or  a data symbol (currently, this applies  only
to HP-UX on  HPPA machines,  and then only  when compiling  a file  with
Popc). The attribute

            ( type type )

can be used for this purpose, where type is "code" or "data".

However, you need only use  this attribute for identifier_specs that  do
not supply a type_spec. With  the default (no type), an  identifier_spec
having a function type_spec will  assume "code", and an  identifier_spec
having any other type_spec will assume "data".

For identifier_specs that do  not supply a  type_spec, the default  with
(no type) is "code".


...  reloading Attribute
------------------------
The reloading attribute controls whether an identifier which already has
a value in Poplog should be loaded again. If you enable reloading, using
the attribute list  (reloading), then exload  will load every  specified
identifier regardless of whether that  identifier already has a  defined
value in Poplog.  Previous values  of the  identifier are  lost. If  you
disable reloading, using  the (no  reloading) attribute  list, then  any
identifier_spec which refers  to an existing  identifier in Poplog  will
effectively be ignored. This allows you to "demand load" symbols from an
external object file or library.


...  batching Attribute
-----------------------
The batching attribute  is used  to collect  identifier-spec lists  from
multiple exload  commands  together,  and  load them  in  a  single  raw
external_do_load call.  This  "deferred loading"  is  desirable  because
invoking the operating system linker is time consuming.

The batching  attribute is  used in  conjunction with  the  exload_batch
command.


exload_batch                                                    [syntax]
        This construct has the form:

            exload_batch
                statement_sequence
            endexload_batch;

        Where statement_sequence contains any statements which are valid
        Pop-11,  and   may   include  multiple  exload  or  exload_batch
        commands.

        Within the context of an  exload_batch command, exload will  add
        each identifier_spec for which batching  is enabled to a  buffer
        which  is  flushed  when  endexload_batch  is  reached.   Nested
        exload_batch commands  can  occur;  the  batch  buffer  is  only
        flushed  when  the  outermost  endexload_batch  is  reached.  If
        batching is disabled by  specifying the (no batching)  attribute
        to  exload,  or  outside  an  exload_batch  ...  endexload_batch
        statement, exload  will add  each  identifier_spec to  a  buffer
        which is loaded as soon as the closing endexload is reached.


To   manually   flush    the   batch   buffer,    use   the    procedure
exload_do_batch_load, which takes no arguments:


exload_do_batch_load()                                       [procedure]
        This  calls  external_do_load  to   load  any  symbols  in   the
        exload_batch buffer.


Within an exload_batch, the default is to batch unless (no batching)  is
specified on a particular exload.

The following example shows  how exload_batch can be  used to group  two
exload commands into a single external_do_load:

        exload_batch

        exload 'test1'
            c_printf <- printf;
        endexload;

        exload 'test2'
            c_exit <- exit;
        endexload;

        endexload_batch;

exload_batch does not  expect there to  be any particularly  complicated
dependencies between different object file specifications in the various
input_file_lists, and  merges these  lists  from the  different  exloads
using an algorithm which (usually) ensures the correct result.

If this causes a problem  for a particular exload,  you can use the  (no
batching) attribute to resolve  it. (This will  not, however, work  with
Popc batch-compilation, since  Poplink use the  same algorithm to  merge
the input_file_lists from all input files).



4.4  Characters in External Names
---------------------------------
A problem  that arises  in VMS  (but  not generally  in Unix),  is  that
external names can contain 'sign'  characters that make them invalid  as
normal Pop-11 words when read as input in a program; the main culprit in
this respect is the character $, which occurs frequently in VMS names.

One  way  to  solve  this  problem   in  VMS  is  to  use  a   different
identifier_name with external_name specified as  a string (in which  a $
doesn't matter). Another  (and probably  the best)  way is  to make  the
character \ an alphabeticiser with

        12 -> item_chartype(`\\`);

(which is  best  done in  your  'init.p' file),  after  which $  can  be
escaped, e.g.

        SS\$_NORMAL = ^uint

Note that in VMS all external symbols are case-blind, that is

        ss\$_normal = ^uint

gives the  same external  symbol as  the  above (but  not the  same  Pop
identifier).

(Note also that there is a limit of 31 characters on VMS external symbol
names; exload will automatically truncate any names over this limit.)




-----------------------------
5  Calling External Functions
-----------------------------

The basic way  to call an  external function pointed  to by an  external
pointer (or  external pointer  class  record) is  with the  syntax  form
exacc. (It can  also be done  with an 'apply'  procedure defined by  the
syntax form defexacc.)  Both exacc  and defexacc are  not restricted  to
calling external  functions,  but  can also  be  used  to  access/update
external data; their general forms are described in REF * DEFSTRUCT.

However, an exacc to  call a function is  quite simple, e.g. an  example
already given above:

            exacc atan2(x, y) -> ddecimal;

This particular example depends on the identifier atan2 already having a
typename declaration for  it as  type '(x,y):dfloat' (if  this were  not
already declared, it would have  to be given as  a type cast before  the
name, i.e.

            exacc (x,y):dfloat atan2(x, y) -> ddecimal;

etc). Note that the  type_spec declaration for a  function allows it  to
have a variable number of arguments, e.g. the C function "printf" loaded
as c_printf

            c_printf(string, ...):int  <- printf

where the "..." specifies an indeterminate number of further  arguments.
In this case, the  actual number of  arguments to a  given call must  be
supplied as an additional last argument. (A convenient way to do this is
with the Pop-11 construct #|, which  counts the number of things put  on
the stack between #| and |#, e.g.

            exacc c_printf(#| string_arg, x, y, z, ... |#)

would supply the appropriate last argument.)

    While the actual call of  a function is straightforward, passing  it
the correct arguments may  be less so.  The following sections  describe
how arguments are converted  when passed, and what  should be passed  to
ensure a function receives the intended data.



5.1  External Function Argument Processing
------------------------------------------
Not surprisingly,  the way  in which  arguments are  passed to  external
functions is conditioned by the way in which they are passed in  Pop-11.
This means (as  noted in Introduction)  that it is  rather more work  to
call languages whose argument passing conventions are not the simple one
used by Pop,  and whereas  the C language  does (roughly)  use the  same
convention, languages like  FORTRAN and PASCAL  do not. The  description
below is therefore of the basic  argument passing mechanism, and how  it
relates to function arguments in C; the following sections consider  the
requirements of other languages.

Although the result of an  external function is `statically typed',  its
arguments are `dynamically typed'. That is, the actual values passed  to
a function depend solely on the run-time arguments you supply to it, and
(with the  exception  of (d)decimals),  any  particular kind  of  Poplog
object is always passed  in the same way.

Any Poplog item may be passed  an a argument; the actual value  received
by the function depends on its processing, as follows:

    Simple integers or bigintegers
        These are converted from Poplog encoded representation to normal
        machine representation, and passed as for a long or int argument
        in C (or a short or char, etc since these are passed as ints).

        A biginteger  exceeding the  size  of an  machine word  has  its
        high-order part truncated (i.e.  is passed modulo 2**N,  where N
        bits is  the machine  wordsize  (64 bits  on  Alpha, 32  on  all
        other current machines)).

    Decimals or ddecimals (floating-point)
        For a  given argument,  both  of these  are passed  as  either a
        machine  single  float,  or  as  a  machine  double  float.  See
        Floating-Point Arguments below.

    Booleans (false and true)
        These are passed  as integer arguments,  0 for false  and 1  for
        true. (They are  in fact  external deref  class, with  word-size
        fields containing 0 and 1 at their structure pointers.)

    External pointer class records
        (I.e. record classes with the attribute "external_ptr".)

        These are  replaced by  their "exptr"  fields at  the  structure
        pointer (see REF * EXTERNAL_DATA), which is then passed as for a
        pointer type in C.

    External deref class records
        (I.e. record classes with the attribute "external_deref".)

        These are replaced  by their word-size  fields at the  structure
        pointer (see * conskey), which is then passed as for an  integer
        in C.

    All other kinds of data structure (records, vectors, etc)
        These are passed  (unchanged)  as  their structure pointers, (in
        the same way as a pointer type in C.

        For a vector class (e.g.  strings, intvecs), the pointer  points
        to the first element of the vector; for a record class it points
        to whatever field is at the pointer position (the section Format
        of Poplog Data Structures in REF * DATA explains this further).

        All Poplog byte  vectors such  as strings are  guaranteed to  be
        null-terminated, so  you can  pass these  directly as  C  string
        arguments.

        Note that if you pass a Poplog array to be processed externally,
        the actual data  is contained  in its arrayvector,  so you  must
        pass this rather than the array procedure itself.

        NOTE ALSO  that when  passing any  Poplog data  structure to  an
        external function which will store the structure pointer in some
        external location for use  in a LATER  external call, a  garbage
        collection while  back  in  Poplog between  the  two  calls  may
        invalidate the pointer in the external location. This problem is
        dealt with  in detail  by  Fixed-Address Poplog  Structures  for
        External Use in REF * EXTERNAL_DATA.



5.2  Floating-Point Arguments
-----------------------------
As mentioned above, Poplog decimals or ddecimals may be passed as either
machine single floats, or machine double floats. The default is to  pass
them as double, but you can  select single by appending  the  flag  <SF>
to any argument in the type_spec declaration for the function. E.g. with
the type_spec

        (x, y<SF>, z) :void

a (d)decimal would be passed as  single for the second argument, but  as
double for the first and third. Where a function is variadic, e.g.

        (x, y<SF>, z, ...<SF>) :void

the ...<SF> means pass (d)decimals as singles for any arguments from the
fourth onwards.

In C, one  place you will  need to use  <SF> is for  float arguments  to
functions which are declared in the new ANSI form

        foobaz(int x, float y, int z)

as opposed to the old pre-ANSI style

        foobaz(x, y, z)
        float y;

etc. Although  declared  as  float rather  than  double,  the  old-style
declaration has  always  in fact  expected  a double  as  argument  (and
continues  to,  even  with  an   ANSI  compiler).  Only  the   new-style
declaration actually requires a single.

Another place to  use <SF> is  where floats are  being passed under  the
guise of general int-size values, e.g.  in a C procedure using  varargs,
or taking some argument which represents an anonymous data value and  is
just declared as int or void * etc.

Note that <SF> has  no effect if  any value other  than a (d)decimal  is
supplied: if you give  an <SF>-flagged argument  an integer, an  integer
will be passed.



5.3  Full Fields in Argument Structures
---------------------------------------
An important point to bear in  mind when passing Poplog data  structures
to external functions (i.e, the last case in External Function  Argument
Processing above)  is  that most  structures  used by  a  normal  Poplog
program hold their data in "full" record fields or vector elements.

As can be seen from the way  in which arguments are converted (and  also
Representation of  Data  in  Poplog in  REF * DATA),  the  main  problem
concerns  integer  and   floating-point  values,  which   are  held   in
Poplog-encoded form. Whereas these are converted to normal machine  form
when  passed  as  direct  argument  values,  when  passed  inside   data
structures they  are  (of course)  not  converted. This  means  that  an
external function trying to access integer or floating-point data from a
"full" field will not get the correct value.

The upshot  is that  integer/floating-point data  intended for  external
processing must  not be  in "full" fields, but  in packed  (i.e.  normal
machine) form, in record  fields or vector elements  with types such  as
:int, :sfloat, :dfloat, etc. For example, if you wish to pass an integer
array to be processed  externally, you cannot  use one constructed  with
newarray, because its arrayvector  is a full  vector; instead, you  must
use one whose  arrayvector is an  intvec (or some  other packed  integer
type).



5.4  Arguments Passed `By Reference'
------------------------------------
One of  the  principal  problems  with  calling  FORTRAN  functions  and
subroutines is the fact that in that language, single integer, real  and
double arguments are passed `by reference' rather than `by value'.  This
means that  a POINTER  to the  data is  expected, rather  than the  data
itself; thus e.g.  for an  integer, what  has to  be passed  is not  the
integer value,  but  a  pointer  to a  memory  location  containing  the
integer.

In the C language, int, float and double arguments are passed by  value;
passing an argument `by reference'  corresponds to passing a pointer  to
the value. Thus the C equivalent of the FORTRAN function

        double precision function square(x)
        double precision x
        square = x * x
        end

is not (as might be expected)

        double square(x)
        double x;
        { return(x * x); }

but

        double square(x_ptr)
        double *x_ptr;
        { double x = *x_ptr; return(x * x); }

etc.

Pop-11 too has its version of this, namely passing an identifier  rather
than its value; the Pop equivalent would be

        define square(x_id);
            lvars x_id, x = idval(x_id);
            x * x
        enddefine;

where square could be called using the ident syntax word, e.g.

        square(ident x)

Now, ident COULD  be used with  an external function,  and would  indeed
supply a pointer to the value;  unfortunately, idval is a "full"  field,
meaning that  its  use  is  ruled out  by  the  Poplog-encoding  problem
discussed in the last section.

So for this type of  argument, what has to be  passed instead is a  data
structure containing  a single  `packed'  element, for  which  purpose a
one-element vector is the usually  the most convenient. In FORTRAN,  the
element type will be :int, :sfloat or :dfloat (corresponding to integer,
real and double respectively). The :int type is provided already by  the
built-in intvec vector class, defined as

        defclass intvec :int;

and suitable floating-point ones can be defined by, e.g.

        defclass sfltvec :sfloat;
        defclass dfltvec :dfloat;

Thus, if either  the FORTRAN or  C version of  the function square  were
loaded as

        square(xptr) :dfloat

then passing it a value, say from the variable x, requires that value to
be put into a 1-element dfltvec first:

        lconstant argvec = initdfltvec(1);
        x -> argvec(1);
        exacc square(argvec) -> result;

(declaring argvec as an lconstant here means a single vector is  created
at compile-time and used for every call, which avoids creating a new one
every time).

    Note that  one  of  the  main advantages  of  passing  arguments  by
reference as opposed to by  value is that a  called function can (if  it
wishes) modify the value pointed to by  an argument, and thus use it  to
return data to  the caller (in  the Poplog  case with a  vector as  used
above, the vector element would then have a new value after the external
call).



5.5  Example: Passing Arrays
----------------------------
As with C, array  arguments in FORTRAN are  expected as pointers to  the
array data. Note that  this too is passing  `by reference', but in  this
case the data in Poplog will already be in a suitable vector (unlike the
case of a single FORTRAN integer,  etc, for which in effect a  1-element
`array' has to be constructed). If (as normal) your array processing  in
Poplog is done with Poplog array procedures (see REF * ARRAYS), then the
vector containing the  data will  be the  arrayvector of  one of  those;
alternatively (e.g.  for  1-dimensional  arrays),  you  might  just  use
vectors on their own.

Note that as mentioned above,  Poplog full vectors (or array  procedures
using them) are not suitable for passing to external functions requiring
integer/floating-point data,  since  they hold  data  in  Poplog-encoded
form.

The following example shows how to define a Pop-11 procedure that  calls
a FORTRAN subroutine to `threshold' a 2-dimensional integer array  (i.e.
replace all elements less than a given limit with zero):

      c
      c This subroutine thresholds an integer array IMAGE of
      c dimension XSIZE by YSIZE with the threshold LIMIT.
      c
          subroutine threshold(image, xsize, ysize, limit)
          integer limit, xsize, ysize
          integer image(xsize, ysize)

          do 10 j=1,ysize
          do 10 i=1,xsize
      10      if (image(i,j) .lt. limit) image(i,j) = 0
          end

After compiling the  above subroutine  to an  object file  'threshold.?'
(where the extension ? is operating system dependent), it can be  loaded
with

        exload 'array-example'
            [threshold]     ;;; object file threshold.?

        (language FORTRAN)
            threshold(image,xsize,ysize,limit)
            ;

        endexload;

A Pop procedure to call the subroutine is shown below. A suitable Poplog
array for this would be one constructed on an intvec or equivalent, e.g.

        newanyarray([1 10 1 10], intvec_key) -> array;

Note that  a 1-element  intvec is  required to  pass each  of the  three
integer arguments  to  the  subroutine;  the  size  arguments  for  each
dimension are computed using the * boundslist of the given array:

        define threshold_array(array, lim);
            lvars
                array, lim, bounds = boundslist(array);
            lconstant
                xsize   = initintvec(1),
                ysize   = initintvec(1),
                limit   = initintvec(1);

            bounds(2)-bounds(1)+1 -> xsize(1);
            bounds(4)-bounds(3)+1 -> ysize(1);
            lim -> limit(1);
            exacc threshold(arrayvector(array), xsize, ysize, limit)
        enddefine;

For comparison, a C function to do the same job might look as follows:

        void threshold(image, xsize, ysize, limit)
        int limit, xsize, ysize;
        int image[];
        { int i, j, r, t = xsize*ysize;
          for (j=0; j < t; j = r)
            { r = j+xsize;
              for (i=j; i < r; i++)
                if (image[i] < limit) image[i] = 0;
            }
        }

This is simpler  to call from  Pop, because the  int arguments are  just
passed by value and no extra vectors are required:

        define threshold_array(array, lim);
            lvars array, lim, bounds = boundslist(array);

            exacc threshold( arrayvector(array),
                             bounds(2)-bounds(1)+1,
                             bounds(4)-bounds(3)+1,
                             lim)
        enddefine;

For an additional example in C see REF * INTVEC.



5.6  Other Types of Arguments
-----------------------------
In C,  character strings  are null-terminated.  From Version  14.1,  all
Poplog strings are  guaranteed to  have a  0 byte  following their  last
byte, which means you can pass them directly to C. (The same applies  to
any byte vectorclass.)

The way  in  which FORTRAN  character  string arguments  are  passed  is
generally host system dependent; you may  be able to ascertain this  and
other details for a particular system  by consulting a manual on how  to
call FORTRAN from C -- if one exists. (For example, on Sun systems,  see
Sun FORTRAN Programmers Guide, section 4.5 Interprocedure Interface.  In
VMS FORTRAN, all character strings are passed by descriptor, see below.)



5.7  VMS: Arguments Passed by Descriptor
----------------------------------------
Many VMS  functions and  procedures,  including library  procedures  and
system services,  make  use of  another  kind of  argument  passing:  by
descriptor.  There  are  actually  lots  of  different  descriptors  for
different kinds of data, but currently the only one explicitly supported
by Poplog is a simple string descriptor:


consdescriptor(string) -> desc                               [procedure]
        Returns a  VMS  simple string  descriptor  record desc  for  the
        string string.


VMS  FORTRAN   character   string   arguments  can   be   passed   using
consdescriptor: the following example illustrates its use in calling the
VMS system service sys$trnlog to  translates a logical name  (sys$trnlog
is actually now superseded by the more complicated sys$trnlnm).

In this example, the input and output strings are passed by  descriptor,
and the argument for the result string length is passed by reference (so
a value can  be passed  back in  it). Note  that a  system service  call
requires all its arguments, so the last 3 unused ones default to zero:

        include sysdefs.ph;         ;;; needed for #_IF DEF ALPHA

        exload 'vms-example'

        ;;; no input files needed on VAX, but ALPHA requires the
        ;;; shareable image sys$share:sys$ssishr.exe
        #_IF DEF ALPHA
            ['sys$ssishr/share']
        #_ENDIF

        (language VMS)
            sys\$trnlog(lognam,rsllen,rslbuf,table,acmode,dsbmsk) :uint,
            ;

        endexload;


        define trans_logical(string);
            lvars
                string, status;
            lconstant
                result_len      = initintvec(1),
                result_string   = inits(128),
                result_desc     = consdescriptor(result_string),
                SS\$_NORMAL     = 1,        ;;; return status code
                SS\$_NOTRAN     = 1577,     ;;;   "       "     "
            ;

            exacc sys\$trnlog(
                    consdescriptor(string),
                    result_len,
                    result_desc,
                    0, 0, 0 ) -> status;

            if status == SS\$_NORMAL or status == SS\$_NOTRAN then
                ;;; extract result using length returned in result_len(1)
                substring(1, result_len(1), result_string)
            else
                ;;; no translation
                false
            endif
        enddefine;



5.8  External Function Results
------------------------------
Direct  results   from  external   functions   can  only   be   integer,
floating-point, or  pointer  values.  For  example,  you  cannot  call C
functions which return  structures by-value (nor  indeed can you  call C
routines that take by-value structure arguments).

One   result   type   requiring   special   mention   is   single-length
floating-point. In  some pre-ANSI  C compilers  (e.g. Suns,  but not  HP
Bobcat or VAX VMS),  a C function declared  as returning a float  result
actually returns a double-float (as if it were declared double); on  the
other hand, this is not so for a real*4 FORTRAN function on any  system.
To allow for  this, the result  type float is  provided: this  assumes a
C-style result as  appropriate to  the host  system, as  opposed to  the
standard sfloat  type  which  always  assumes a  single  (in  all  other
contexts, float and sfloat are identical as types).

Thus the exload declaration

        foo(x) :sfloat

is appropriate for FORTRAN, but not for  C (at least, not on all  Poplog
hosts); for C, always use

        foo(x) :float

In the case of double-float results,

        foo(x) :dfloat

is appropriate for all languages.



---------------------------------------
6  Re-compiling and Unloading an exload
---------------------------------------

The purpose  of the  mark_item argument  to exload  is to  specify  that
particular load uniquely. If you  re-compile an exload that has  already
been compiled one  (or more)  times, the system  uses the  mark item  to
detect that  happening,  and  undoes  the  existing  load  first  before
re-doing it  (without  which, duplication  of  object files  etc,  would
occur).

However, because external loading is an `additive' incremental  process,
to undo a given load requires that all later loads are undone also. Thus
if loads with mark items  'A', 'B', 'C' and  'D' have been compiled  (in
that order), then re-compiling 'B' forces the system to unload not  only
'B' itself, but 'C' and 'D' as well (but 'A' remains untouched).

This means that to re-compile one of a sequence of loads, you must  also
re-compile any that came after it.

You can also explicitly unload back  to a given load with the  procedure
external_unload, and  the  current  state of  external  loading  can  be
displayed with  the  procedure external_show  (for  both of  these,  see
Utility Procedures for External Loading below).

6.1  Alpha VMS Note
-------------------
When a shareable  image is  first loaded, exload  remembers the  version
number of the file.  Additional loads using that  image will always  use
the same version number.

Thus if you recompile and relink a  shareable image, to pick up the  new
version you must recompile your Pop code back to the load mark where the
image was first loaded. (You cannot invoke a new version of a  shareable
image that is already used in the system, or in a saved image.)

(Because VMS  in fact  provides no  support for  unloading or  reloading
shareable images,  each time  exload has  to load  a new  version of  an
existing image it does so by loading a temporary copy of the image under
another name. Although the memory occupied by the first load of an image
cannot be reused (since it may contain other shareable images invoked by
the outer one), the memory  occupied by subsequent 'temporary' loads  is
recycled. This may cause problems if you try to make too drastic changes
to an image,  i.e. making it  use new  shareable images it  did not  use
previously. In addition, every 'temporary' reload of a new version  will
add a further entry  to VMS's internal database  of images, which  could
eventually overflow.)

None of the above applies when simply reloading from the same version of
an existing image.




------------------------------
7  Saving and Restoring Images
------------------------------

A saved image created with  either syssave or sys_lock_system will  save
the state of external loading, and sysrestore on the image will  restore
that state. The saving and restoring is done in such a way that  further
incremental loading can  continue normally after  the restore.  However,
all loads done before the saved  image was created are locked, that  is,
cannot be unloaded (a mishap will result if you try to re-load a  locked
load).

It is important to note that the Poplog saved image mechanism saves only
external functions  and  the  initial  (load-time)  state  of  any  data
associated with them.  It does not  save their run-time  data, i.e.  you
cannot save and restore the  results of actually running external  code.
(In particular, areas of dynamic memory allocated by the library  memory
allocators malloc (Unix) and lib$get_vm (VMS) are not saved.)

In other words, saved  images are not  intended as a  way of saving  the
running state of external functions, but only for saving external  loads
after compilation. While  Poplog does  not prevent  you from  creating a
saved image after  external code has  been run, this  will, in  general,
lead  to  unpredictable  external  code  behaviour  when  the  image  is
restored.


7.1  Unix
---------
On Unix  systems with  dynamic  linking, no  external  code or  data  is
actually saved  in an  image;  instead, a  record  of files  loaded  and
symbols accessed during all  external loads is kept.  When the image  is
restored, all the external loads are redone in one go.

A consequence of  this is that  any shared objects  recorded in a  saved
image must be accessible in their  original locations when the image  is
restored. This is  not quite as  restrictive as it  might seem,  since a
shared object file may  be referenced using a  path name which  includes
one or more  environment variables.  These will be  translated when  the
saved image  is restored,  thereby allowing  saved objects  to be  moved
(providing the relevant environment variables are changed accordingly).


7.2  VMS
--------
In VMS, all shareable images used by external functions are  remembered,
and reactivated when the saved image is restored.

On the VAX,  activation of a  private shareable image  (i.e. one not  in
sys$share:) requires the definition  of a logical  name for that  image.
Logical names for all  such images reactivated by  a Poplog saved  image
must therefore be defined when the saved image is restored.

On Alpha systems, this only applies if you actually loaded the shareable
image via a logical name. For an image loaded via a filename, e.g.

            'somedir:myshare.exe/share'

the exact filename  given at  load time  will be  reactivated (i.e.  any
logical name in the device part  will use the run-time translation,  not
the load-time translation).

Note that, owing to possible expansion of system shareable images in new
versions of  VMS, a  Poplog saved  image may  fail to  work in  a  later
version of VMS than  the one it was  built in -- see  Note on VMS  Saved
Images in REF * SYSTEM for more on this point.

...  VAX Note On message Utility
--------------------------------
A problem exists  on VAX  systems concerning the  interaction of  Poplog
saved images and  object files  containing messages created  by the  VMS
message utility. While a program that loads such object files will  work
correctly when run directly after loading,  it will fail to work if  run
from a saved  image. (What  happens is  that although  the messages  are
mapped into  memory, they  do  not get  registered with  the  sys$getmsg
facility).

However, this  problem applies  only  to object  files  per se,  not  to
shareable images:  a simple  workaround is  therefore to  make any  such
message  files  into   a  shareable   image.  A   message  object   file
'messages.obj' is easily turned into one with the DCL command

        $ link/shareable messages.obj

This  produces  the  shareable  image  'messages.exe';  after   defining
'messages' as a logical name pointing to the image,

        $ define messages [mydir]messages.exe

you can  then include  'messages/share' in  an exload  in place  of  the
original object  file. A  saved  image created  after this  will  behave
correctly (remembering that, as mentioned  above, the logical name  must
also be defined when the saved image is restored).

(This problem does not apply on  Alpha systems, since you can only  load
shareable images anyway.)




------------------------------------------
8  Utility Procedures for External Loading
------------------------------------------


external_show()                                              [procedure]
        Produces a listing of all current external loads.


external_load_count() -> N                                   [procedure]
        Returns the current number of external loads.


external_load_mark(N) -> item                                [procedure]
        Returns the mark item  for the N-th  external load, where  loads
        are numbered from 1 (the most recent).


external_unload(mark_item)                                   [procedure]
        This can be used with  a given mark to  undo the effects of  all
        loads back to (and including) the one indicated. Because loading
        is an  'additive'  incremental process,  to  undo a  given  load
        requires that all later loads are undone also. Thus if  external
        loads 'A', 'B', 'C' and 'D' have been done (in that order), then

                external_unload('B')

        will unload back to mark 'B',  i.e. will unload all of 'B',  'C'
        and 'D' (and only 'A' will remain loaded).




-----------------------
9  Raw External Loading
-----------------------

The  `raw'  procedure  for  loading  object  files  containing  external
functions and data is external_do_load, described below; this is used by
the syntax  construct  exload. (You  do  not  need to  know  about  this
procedure  unless  you   want  to  construct   other  external   loading
interfaces.)

The input  files (or  default libraries,  etc) for  each new  load  must
provide definitions  for  all  symbols  in  symbol_list  that  have  not
previously been loaded (linker or  Poplog error messages will result  if
any are undefined or multiply defined).

Systems with Static Linking
On Unix  systems  with  static linking, and  VAX  VMS,  external_do_load
operates by running  the operating system  linker (ld in  Unix, link  in
VMS) in a sub-process,  to link the given  files with a special  `symbol
table' object file. This  symbol table file contains  a running list  of
symbols defined either in the base Poplog system or in previous external
loads; symbols accumulate  from one  external_do_load to  the next,  and
each one generates a new symbol table file.

On VAX VMS, external_do_load also maintains a running list of  shareable
images that have been loaded  (including those extracted from  shareable
image libraries),  all  of  which are  automatically  included  in  each
subsequent load: thus if you have specified 'foo/share' in one load, you
need not specify it in a later one (if you do, it will be ignored).

The base `symbol table' object file contains those external symbols that
are  used   already   in   the  standard   Poplog   system.   In   Unix,
external_do_load assumes this file  is given by  the translation of  the
environment   variable   "popexlinkbase"   (which   will   normally   be
'$popsys/basepop11.stb').  In  VAX  VMS,  the   file  is  taken  to   be
'image.stb' where image is the name of the Poplog image being run (again
normally   'popsys:basepop11.stb',   since   the   image   is   normally
'popsys:basepop11.exe').

Systems with Dynamic Linking
On Unix systems with dynamic  linking, external_do_load uses the  system
call interface to the operating system dynamic linker to attach a shared
object to the Poplog process and to access symbols defined therein.

In Alpha VMS, external_do_load uses lib$find_image_symbol to dynamically
load symbols from shareable images.

In neither case is there any  need for accumulating symbol table  files.
The base `symbol table' object file will be merely an empty dummy.


external_do_load(mark_item, input_file_list, symbol_list)    [procedure]
        Loads object files containing  external functions and data,  and
        provides for the values of external symbols from those files  to
        be accessed.

        The arguments are as follows:

        mark_item
            An item that  identifies this particular  external load.  It
            may  be  any  Poplog  item  (number,  string,  word,   etc),
            providing it is different from any previously used mark item
            (N.B. different according to the = operation, not ==).

            If mark_item  specifies  an  existing  load,  then  that  is
            unloaded first, i.e. external_unload(mark_item) is called.

        input_file_list
            Same as  the  input_file_list  to  exload.  See  Input  File
            Arguments above.

        symbol_list
            Specifies the external symbols to  be loaded, and is a  list
            of 5-element full vectors, one  vector for each symbol.  The
            elements in each vector are used for either input or output,
            as follows:

                1.  (Input/Output) The name of the symbol, as a word  or
                    string. If a  string, the  name is  taken as  given;
                    otherwise, a word is  transformed to an actual  name
                    string  according  to  the  host  operating  system,
                    taking into account the language name in  element 2,
                    and  the result  is assigned back into element 1.

                    In VMS the only translation performed is to truncate
                    all  symbols  to  31  characters  (by  chopping  off
                    trailing characters).  Otherwise,  for  a  word  the
                    result string is the same as the word.

                    In Unix  systems  without COFF  or  ELF  object-file
                    format,  the   basic  translation   (as  for   the C
                    compiler) is to prefix the name with an  underscore,
                    i.e. "foo"  becomes  '_foo';  if  in  addition,  the
                    language  name  in   (2)  is   'FORTRAN',  then   an
                    underscore is also postfixed to the name, i.e. "foo"
                    becomes `_foo_`.

                    In Unix systems with COFF/ELF object-file format, no
                    translation is performed  (like VMS)  except in  the
                    case where the  language name in  (2) is  'FORTRAN',
                    when an underscore is postfixed  to the name in  the
                    same way as other Unix systems.

                2.  (Input) A language name  string, ignored except  for
                    'FORTRAN' in Unix systems as above.

                    This element may also be  a pair whose front is  the
                    language name string  and whose back  is a type  for
                    the symbol. If supplied, the type must be the string
                    'code'  or  'data',  and  defaults  to  'code'  when
                    omitted.

                3.  (Input/Output) Return  value;  after the  call  this
                    will be an  external pointer  record, whose  pointer
                    value is  the value  of the  external linker  symbol
                    given by (1). The external_ptr_props of this  record
                    is set to the name of the symbol from (1).

                    In all systems except  VAX/VMS, you can also  supply
                    as input in  this field  a pre-constructed  external
                    pointer record  to use  for  the output  value.  The
                    procedure * external_load_consptr will return a  new
                    external  pointer  if   this  allowable,  or   false
                    otherwise. Thus if  you wish to  pass in a  pointer,
                    always use the result from external_load_consptr.

                4.  (Output) Set  to  a "tmpval"  property  entry  whose
                    fast_prop_entry_arg is the external symbol  string
                    from 1.   This  is  stored on  the system's  history
                    list     of     external     loads,     and      its
                    fast_prop_entry_value is printed out for the  symbol
                    'value'  by  external_show;  thus  you  can   assign
                    whatever is required to show  as the value into  the
                    fast_prop_entry_value of this.

                    In addition,  if  the  entry  value  is  set  to  an
                    external   pointer    class    record,    then    an
                    external_unload for  mark_item will  assign null  to
                    the pointer value of that record.

                    If no value  is assigned,  or the  value is  garbage
                    collected, then  external_show  will not  print  the
                    symbol at all.

                5.  (Ignored) May contain any user data associated  with
                    the symbol.

        NOTE:
            There are  certain  restrictions on  the  way in  which  the
            external pointers  returned  in  element (3)  can  be  used:
            basically,  they   must  be   treated  as   `opaque'   until
            `run-time'.

            In practice this means the following: data accessed  through
            the pointers,  or other  external pointer  class  structures
            into which the  pointer values have  been copied, cannot  be
            handled either (a) by  the saved image  mechanism or (b)  by
            Popc compilation.

            The reason for (a) is  that saved images can correctly  save
            and restore  external symbol  values from  operating  system
            shareable libraries only  when they reside  in the  original
            external pointer records returned by external_do_load;  they
            cannot,  however,  guarantee  to  deal  correctly  with  any
            shareable-library  data  or  code  accessed  through   those
            pointers, or deal  with the  external name  values when  put
            into other  external pointer  class structures.  (Note,  for
            example, that  when loading  in Alpha  VMS with  pop_runtime
            false (i.e. system started  with %nort), the pointer  values
            are not even  assigned, since the  shareable images are  not
            actually activated.)

            The reason for (b)  is the Popc  compiler can only  generate
            object code for external pointers where the external name is
            given by the external_ptr_props of the pointer.

            Thus accesses through the  pointers, or copying the  pointer
            values into other external pointer structures, must be  done
            at run-time.  Any such  operation which  would nominally  be
            done   at   compile-time   must   be   deferred   by   using
            * sys_runtime_apply (see Runtime Actions in REF * SYSTEM).


external_load_consptr() -> exptr_or_false                    [procedure]
        If external_do_load can take  input external pointer records  in
        element (3) of symbol vectors  (see above), then this  procedure
        returns a  new external  pointer; otherwise,  it returns  false.
        (Currently, the  only  system  in  which  it  returns  false  is
        VAX/VMS.)




---------------------
10  External Callback
---------------------

From Version 14  of Poplog,  external functions called  from within  the
system may  call  back  to  Poplog procedures.  The  interface  to  this
consists of a  set of  callback functions,  described below;  currently,
there are only  C versions  of these, and  no direct  support for  other
languages is provided (although of course, if a particular language  can
call C, then that provides an interface).


10.1  C Interface
-----------------
These functions are in the  Poplog external library 'libpop.olb' in  the
directory given by the environment variable/logical name  "popexternlib"
(this library is automatically included for external loading).

Declarations for the functions are in the C header file 'callback.h'  in
the same directory. To use this with  #include in Unix you will need  to
give the  host  C  compiler  a  -I  argument  for  the  directory  (i.e.
-I$popexternlib). In VMS you can either  give the C compiler a  /include
argument (i.e. /include=popexternlib:), or specify

    #include "popexternlib:callback.h"

directly in the source file.


...  Synopsis
-------------

    #include "callback.h"

    int pop_mishap(char *message);

    int pop_call(POPOBJ obj, void *argptr);

    POPOBJ pop_get_ident(char *idname);

    int pop_free_fixed_hold(POPOBJ obj);

    int pop_check_interrupt();

    POPOBJ *pop_exfunc_closure_arg;

    unsigned *pop_external_flags;


...  Description
----------------

    int pop_mishap(char *message);

        Re-enters Poplog  and  then  generates  a  mishap  with  message
        message (i.e. calls mishap(0, message_string) ).

        The  result  value  is  1  for   a  normal  return,  0  for   an
        abnormal-exit return (see  below). Since it  generates a  Poplog
        mishap (which should cause an abnormal exit, usually a  setpop),
        this function should always return 0 (if it returns at all).


    int pop_call(POPOBJ obj, void *argptr);

        Re-enters Poplog and then calls the procedure specified by  obj,
        giving it as argument an external pointer whose pointer value is
        argptr.

        obj must  be  a  Poplog  structure  pointer  which  supplys  the
        procedure P to be called: it  can be the procedure directly,  an
        identifier whose idval  is the procedure,  or a reference  whose
        cont is either of those. (A Poplog mishap will result if obj  is
        not valid.)

        P is  called as  P(exptr), where  exptr is  an external  pointer
        pointing to  whatever  data  structure  argptr  points  to.  The
        content of this  data structure is  entirely between the  caller
        and P, and it is up to P to access its components through  exptr
        (with exacc,  etc) to  extract input  arguments, and  to  update
        components through exptr to return output values (P itself  must
        not return any result on the Poplog userstack).

        The  result  value  is  1  for   a  normal  return,  0  for   an
        abnormal-exit return (see below).


    POPOBJ pop_get_ident(char *idname);

        Gets the  permanent identifier  with  pathname idname  from  the
        Poplog dictionary. (For  an identifier  containing a  procedure,
        the result of this  function could be  supplied directly as  the
        obj argument to pop_call.)

        idname may  be  any  valid pathname,  including  sections  where
        necessary, and may  be either  absolute (beginning  with $-)  or
        relative (not  beginning with  $-); if  relative, it  refers  to
        whatever * current_section is at the time of the call.

        The normal  return value  is a  (held) fixed-address  reference,
        whose cont is either the permanent identifier record  associated
        with idname, or its idval for an assigned constant (i.e, one for
        which isconstant returns true). That is,

                cons_fixed(id_or_idval, ref_key, true)

        where id_or_idval is either the identifier or its idval.

        Note that  if  idname  is not  declared,  the  usual  'DECLARING
        VARIABLE' message will result inside  Poplog; thus for a  normal
        return this function  will always return  an identifier. For  an
        abnormal-exit return the result is 0 (see below).


    int pop_free_fixed_hold(POPOBJ obj);

        Re-enters Poplog and calls the procedure free_fixed_hold on obj.
        (For example,  this  could  be  used  to  free  a  fixed-address
        reference returned by pop_get_ident.)

        The  result  value  is  1  for   a  normal  return,  0  for   an
        abnormal-exit return (see below).


    int pop_check_interrupt();

        Checks to see  if there  are any asynchronous  traps or  signals
        pending inside Poplog (i.e. queued but not processed), and if so
        (and  asynchronous  trap  and  signal  processing  is  currently
        enabled),  re-enters  the   system  and   services  them.   (See
        REF * ASYNC.)

        The  result  value  is  1  for   a  normal  return,  0  for   an
        abnormal-exit return (see below).


    POPOBJ *pop_exfunc_closure_arg;

        A pointer to a POPOBJ variable that receives the argument  value
        from an external function closure ("exfunc_closure") as  created
        by make_exfunc_closure. See External Function Closures below.


    unsigned *pop_external_flags;

        A pointer to an integer  variable that holds status and  control
        flags, all  defined in  callback.h (note  that these  flags  are
        local to  each  block  of  external  calls).  See  Callback  and
        Abnormal Exit below.

        The  first  is  a  status  flag  for  testing  only  (i.e.   `if
        (*pop_external_flags&FLAG) ...' etc):

            PEF_DOING_ABEXIT    Set if  the  current block  of  external
                                calls will continue an abnormal exit  on
                                return to Poplog.

        The following flags can be  set for control over abnormal  exits
        (*pop_external_flags|=FLAG):

            PEF_RETURN_ABEXIT_NEXT
                                Return  from  abnormal   exit  on   next
                                callback only (cleared after  callback).
                                An    abnormal    return    will     set
                                PEF_DOING_ABEXIT.

            PEF_RETURN_ABEXIT_ANY
                                Return  from   abnormal  exit   on   any
                                callback  (in  this  block  of  external
                                calls).  An  abnormal  return  will  set
                                PEF_DOING_ABEXIT.

            PEF_CATCH_ABEXIT_NEXT
                                Catch and return  from abnormal exit  on
                                next  callback   only   (cleared   after
                                callback).

            PEF_CATCH_ABEXIT_ANY
                                Catch and return  from abnormal exit  on
                                any callback (in this block of  external
                                calls).

        These flags  can  be  set  for  user  control  over  asychronous
        callback:

            PEF_ASYNC_CALLBACK
                                If  set,  then  asynchronous  traps  and
                                signals  that  interrupt  user  external
                                code will  call  pop_check_interrupt  to
                                service   the   interrupt    (otherwise,
                                servicing waits  until return  from  the
                                current block  of external  calls).  You
                                should not  set  this flag  unless  your
                                program is  `garbage-collection  proof',
                                i.e.   makes   proper   allowance    for
                                fixed-address structures etc. See below.

            PEF_ASYNC_RETURN_ABEXIT
                                If  this  is  set  in  combination  with
                                PEF_ASYNC_CALLBACK,      causes       an
                                asynchronous     callback     to     set
                                PEF_RETURN_ABEXIT_NEXT before making the
                                callback.

            PEF_ASYNC_CATCH_ABEXIT
                                If  this  is  set  in  combination  with
                                PEF_ASYNC_CALLBACK,      causes       an
                                asynchronous     callback     to     set
                                PEF_CATCH_ABEXIT_NEXT before making  the
                                callback.

        Note: Pop-11 macros for all these flags are  also  available  in
        INCLUDE * EXTERNAL_FLAGS.



10.2  Callback and Fixed-Address Structures
-------------------------------------------
pop_call is the basic callback function: to use this successfully, it is
essential to read  Fixed-Address Poplog Structures  for External Use  in
REF * EXTERNAL_DATA.

The obj argument to  pop_call (supplying the procedure  to call) can  be
passed down through an external function, either as a direct argument or
in a "full" field of  some data structure (an  external one or a  Poplog
one). In all cases, ordinary Poplog structure pointers stored externally
may become  invalid after  a return  to Poplog  which causes  a  garbage
collection.

Thus it is almost always essential to use fixed-address structures  in a
callback context, either by creating structures as fixed-address in  the
first place, by making fixed copies, or by putting ordinary  relocatable
structures inside fixed  ones. For  a procedure argument  like obj,  the
simplest way  is  to  pass  a  fixed-address  reference  containing  the
procedure, etc (which is why  pop_call allows a reference as  argument).
This is  the method  used by  pop_get_ident when  returning a  permanent
identifier or its value.

It is also essential to  make provision for `holding' any  fixed-address
structure which has  no other  pointers to  it inside  Poplog. Thus  the
references returned by pop_get_ident  are kept on  the fixed hold  list,
and can only become garbage if freed with pop_free_fixed_hold.



10.3  External Function Closures
--------------------------------
One of the main  potential uses of the  Poplog callback mechanism is  to
allow externally-defined callbacks to be extended to Poplog. That is, in
the situation where  an external  function, F say,  requires a  variable
callback function CF as argument, we wish (in effect) to allow CF to  be
a variable Poplog procedure.

While it is straightforward to write a function CF which packages up its
arguments in  a temporary  struct  to pass  as  the second  argument  to
pop_call, the problem arises as to how to to pass the Poplog  procedure.
A well-designed function F will take an additional argument P, which can
be any user data to be passed on as one of the arguments to CF, and this
can be used to pass the  procedure (either directly or inside some  data
structure, etc).

However, F may not be so designed; in this case, the only alternative is
to create a new external function, which is essentially CF but with  a P
argument frozen in -- that is, a closure of CF. Poplog provides for this
with make_exfunc_closure, which allows the creation of external function
closures. These are records containing a small piece of executable code,
together with a pointer  to an external function  (the CF) and a  frozen
argument value  (the  P), and  which  can be  called  from C  just  like
ordinary external functions. When executed,  such a closure assigns  its
frozen argument value P to the standard location *pop_exfunc_closure_arg
and then calls  its base  function CF  (which can  then pick  up P  from
*pop_exfunc_closure_arg). In C, this might look something like

        CF(arg1, ..., argN)
        type1 arg1; ... typeN argN;
          {
            struct { type1 arg1; ... typeN argN; } args;
            args.arg1 = arg1;
                ...
            args.argN = argN;
            pop_call(*pop_exfunc_closure_arg, &args);
          }

etc. After  externally loading  CF, a  closure for  a  given   procedure
P could be created with

        make_exfunc_closure(CF, P, true)

and passed as argument to the F function.

    A disadvantage (from the Poplog  programmer's point of view) of  the
example just given  is that it  requires a C  'stub' procedure CF  which
knows what arguments F is going to pass to its callback procedure. Every
different callback  context  is  likely to  require  a  different  stub,
written in C and  externally loaded into Poplog.  To avoid this  problem
Poplog provides a 'generic' stub procedure which simply calls back on P,
passing a pointer to any arguments  it has been passed. This allows  the
argument decoding to be carried out  within a Poplog procedure which  is
then 'exported'  as the  closure  argument for  the generic  stub.  This
generic C stub looks something like

        CG(va_alist)
        va_dcl
        {
            va_list argp;

            va_start(argp);
            pop_call(*pop_exfunc_closure_arg,argp);
            va_end(argp);
        }

(note the use of the C 'varargs' mechanism for passing the args  through
to pop_call). The Poplog  procedure provided as  *pop_exfunc_closure_arg
receives a single external pointer argument, pointing to an array of the
args. So to achieve the equivalent of  CF above, one might use a  Poplog
procedure such as:

        define pop_CF(exptr);
            lvars exptr, arg1, ..., argN;
            l_typespec exdata
                { type1 arg1; ... typeN argN; }

            exacc :exdata exptr.arg1 -> arg1;
            ...
            exacc :exdata exptr.argN -> argN;

            P(arg1, ..., argN);
        enddefine;

(assuming the types type1, ...  typeN had been previously declared).  To
export  this   procedure   using   the   generic   callback   stub   use
exfunc_export:

        exfunc_export(pop_CF, 0, false)

which will return an external function closure that can be used directly
as a callback argument. (The  additional arguments to exfunc_export  are
discussed in External Function Closures in REF * EXTERNAL_DATA.)

    In practice, one would not want to rewrite pop_CF for each procedure
P used as a callback, but Poplog's ordinary procedure closure  mechanism
can be used to avoid this:

        define pop_CF(exptr, p);
            lvars exptr, procedure p, arg1, ..., argN;
            l_typespec exdata
                { type1 arg1; ... typeN argN; }

            exacc :exdata exptr.arg1 -> arg1;
            ...
            exacc :exdata exptr.argN -> argN;

            p(arg1, ..., argN);
        enddefine;

        exfunc_export(pop_CF(% P %), 0, false)

Examples of this  closure technique in  the X Toolkit  interface can  be
found in LIB * fast_xt_callback and LIB * fast_xt_action.

    Note that exfunc closures are automatically fixed-address  (although
the issue of `holding'  them still has to  be considered). On the  other
hand, since an exfunc closure is an ordinary Poplog record in which  the
P value is  held in a  "full" field,  so there is  no requirement  for P
itself  to  be   fixed-address.  See  External   Function  Closures   in
REF * EXTERNAL_DATA for more details.



10.4  Callback and Abnormal Exit
--------------------------------
Any callback to Poplog can potentially result in an abnormal exit,  that
is, procedures like setpop, exitto, chainto, etc being called to  unwind
the callstack to a point prior to the current block of external calls.

The default action  of all  the callback  functions in  this respect  is
simply to allow the abnormal exit to erase the current block of external
calls altogether, and to carry on exiting from the Poplog call (i.e. the
exacc that initiated it) -- thus the callback function never returns.

However, this default  behaviour may be  altered by setting  one of  the
flags

        PEF_RETURN_ABEXIT_NEXT
        PEF_RETURN_ABEXIT_ANY
        PEF_CATCH_ABEXIT_NEXT
        PEF_CATCH_ABEXIT_ANY

in *pop_external_flags. In general, setting any of these will cause  any
of the callback  functions to handle  an abnormal exit  by returning  an
error value of 0. A "NEXT" flag is cleared after the callback -- and  so
affects only the next  callback function called --  while an "ANY"  flag
remains set  (and  so affects  any  callback  in the  current  block  of
external calls).

The difference between the "RETURN" and "CATCH" versions in each case is
what happens when the current block of external calls returns back  into
Poplog: In the "RETURN"  case, the abnormal  exit is simply  `suspended'
while back  in  the  external  calls,  but  takes  control  again  after
returning to Poplog; the "CATCH" versions, on the other hand, catch  the
abnormal exit altogther, i.e. abort it (whence return from the  external
calls will be as normal).

If both  "CATCH" and  "RETURN"  are set,  then  the "CATCH"  flag  takes
precedence. However,  once  an  abnormal  exit  has  been  handled  with
"RETURN", the `suspension' of  that exit is  irreversible, i.e. it  will
continue on returning to Poplog (regardless of whether further exits are
aborted with  "CATCH").  To indicate  this  state, an  exit  handled  on
"RETURN" sets  the flag  PEF_DOING_ABEXIT in  *pop_external_flags;  once
this is set, the external calls cannot return normally.



10.5  Restrictions on Operations Inside Callback
------------------------------------------------
Certain operations  cannot be  performed  inside external  callback:  in
particular, there  are problems  with Poplog  processes and  the  Prolog
subsystem, as noted below.

In general,  doing any  significant  amount of  work  in a  callback  is
deprecated, and  should be  avoided if  possible. To  aid in  this,  the
procedure external_defer_apply  allows  you to  specify  actions  from a
callback which  are executed  only  when the  callback has  returned  to
top-level Poplog:

external_defer_apply(p)                                      [procedure]
external_defer_apply(p, arg1, arg2, ..., argN, N)
        This  procedure  is  intended  to  be  called  within   external
        callbacks; it permits  operations that  are not  allowed or  are
        inadvisable within  a  callback to  be  safely executed  by  the
        procedure p.

        It  raises   p   as   an  asynchronous   trap   procedure   with
        * sys_raise_ast,  but  specifying  that  p  be  blocked   inside
        external calls. Thus  when inside  external calls,  p is  merely
        queued, and  will  not  actually  be  executed  until  the  next
        interrupt checkpoint encountered on returning from the outermost
        external call.

        (Note for X Toolkit users:  When inside a Toolkit callback,  the
        queueing of p  also causes  an implicit  * XptSetXtWakeup to  be
        performed.)

        If external_defer_apply  is  called  when  not  inside  external
        calls,  p   will  be   executed  immediately   (i.e.   providing
        asynchronous traps are not blocked because * pop_asts_enabled is
        false   --   otherwise,   p   will   be   executed   only   when
        pop_asts_enabled reverts to true).

        In the second  form of  the call,  the procedure  p is  supplied
        together with N items arg1, ..., argN that should be passed to p
        as arguments (this form is the same as * consclosure, and causes
        external_defer_apply to  call consclosure  to create  a  closure
        which will be garbaged after execution).


...  Processes
--------------
The operation  of Poplog  processes  (see REF * PROCESS)  is  restricted
inside callback, as follows: Although processes can be run and suspended
as normal inside callback, you cannot suspend a process across  callback
(that is, so as to include external calls in the process's context).  On
re-entering  Poplog  from  external  callback,  pop_current_process   is
locally set  false;  this prevents  any  attempt to  suspend  a  process
running outside the original external function call.

(N.B. One consequence of the above  is that a procedure inside  callback
which reads input from charin will not work inside Ved "immediate  mode"
-- since that relies  on suspending the immediate  mode process to  wait
for input to  be entered. It  is therefore inadvisable  to use  callback
procedures that read interactive input.)


...  Prolog
-----------
The Prolog  subsystem  in  Poplog  uses  an  area  of  memory  to  store
continuations and the  trail of instantiated  variables (see The  Prolog
Memory  Area   in  REF * PROLOG).   This  area   is  normally   expanded
automatically as  required, or  its  size set  explicitly with  the  Pop
variable pop_prolog_size (or the Prolog predicate prolog_area_size).

However, in all current implementations  of the system, the Prolog  area
is stored in the same region as the callstack, which means that changing
its size  requires  moving  all  the callstack  frames  to  make  space.
Although Poplog procedure stack frames  are designed to be  relocatable,
external function stack frames are  not, and hence the callstack  cannot
be moved while external frames are active.

Thus the  size of  the Prolog  area cannot  be changed  during  external
callback. Attempting to do so will result in the mishap

    'CAN'T CHANGE PROLOG AREA SIZE DURING EXTERNAL CALLBACK'

While this limitation may be overcome in a future version of the system,
for now if  you use Prolog  procedures during callback  you must  ensure
that the prolog  area is large  enough for your  program's needs  BEFORE
starting it (by  assigning an  appropriate value  to pop_prolog_size  or
prolog_area_size).




-------------------
11  General Warning
-------------------

External functions should not access data outside of the data structures
they themselves  define, or  that are  passed to  them from  Poplog;  in
particular they  must  not write  into  memory areas  outside  of  these
structures. If this happens, it will more than likely corrupt the Poplog
system.



--- C.all/ref/external
--- Copyright University of Sussex 1996. All rights reserved.