NASA ADS Abstract Service Mirroring Information

# make sure to have at least the following modules configured
# in your http daemon.  If these are provided as dynamic shared
# modules then you will need to load them using the LoadModule
# directives below, although the path to the shared object may
# vary between installations
LoadModule env_module         modules/mod_env.so
LoadModule config_log_module  modules/mod_log_config.so
LoadModule cgi_module         modules/mod_cgi.so
LoadModule action_module      modules/mod_actions.so
LoadModule alias_module       modules/mod_alias.so
LoadModule access_module      modules/mod_access.so
LoadModule auth_module        modules/mod_auth.so

AddModule mod_env.c
AddModule mod_log_config.c
AddModule mod_cgi.c
AddModule mod_actions.c
AddModule mod_alias.c
AddModule mod_access.c
AddModule mod_auth.c
AddModule mod_so.c

# disable DNS look-ups for efficiency
HostnameLookups off

# run as user ads
User ads

# define new log format so that the ADS cookie ID and 
# Proxy information is saved
LogFormat "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" \"%{Cookie}i\" \"%{Via}i\"" cookieid
CustomLog /ads/logs/access_log cookieid

# The timeout is high for our server because we serve article images,
# which may require long transfers.  For a mirror site that only runs
# the abstract server, setting the timeout to 10 minutes (600 seconds)
# should be plenty.
Timeout 600

# define the directory index and access control file names
DirectoryIndex index.html
AccessFileName .htaccess

# allow .htaccess files to override options set for the
# document root directory and the cgi-bin directory
<Directory "/ads/htdocs">
   AllowOverride All
   Options FollowSymLinks
   Order allow,deny
   Allow from all
</Directory>

# define the cgi-bin directory
ScriptAlias /cgi-bin/ /ads/cgi/bin/
<Directory "/ads/cgi/bin">
   AllowOverride All
   Options FollowSymLinks
   Order allow,deny
   Allow from all
</Directory>

• Incoming http connections (port 80): this port must be open to allow users to query our databases. To test that this has been setup properly, try accessing your website from a machine outside your network.

• Incoming ssh connections (port 22): we will use ssh access to log into your server in order to maintain and update the database as user ads. If you wish to restrict ssh access to just the IP addresses of our servers for increased security, please make sure that connections from the following machines are enabled:

      adsduo.cfa.harvard.edu   (131.142.185.23)
  

  Please let us know once you have enabled ssh access so we can test the setup.

• Outgoing rsync connections (port 873): we use a network syncronization tool called rsync to maintain our database up to date. You will need to make sure that your firewall allows an rsync connection to be estabilished from your machine to our servers:

      adsduo.cfa.harvard.edu   (131.142.185.23)
  

  To test this connection, you should be able to simply type:

      rsync rsync://adsduo.cfa.harvard.edu
  

  and verify that our server responds to the query with a listing of modules available for updating. (If rsync isn't installed on your system, don't panic, we will take care of installing a private copy under user ads's home directory.)

• Outgoing SIMBAD connections (port 1674): we use a simple TCP/IP socket connection to the SIMBAD name resolver to perform object queries. You will need to make sure that your firewall allows such a connection to be estabilished from your machine to the SIMBAD servers:

      simbad.u-strasbg.fr (130.79.128.4)
      simbad.harvard.edu  (131.142.185.22)
  

  To test this connection, you should be able to simply type:

      telnet simbad.u-strasbg.fr 1674
  

  and verify that you get a prompt back that looks like this:

      Trying 130.79.128.4...
      Connected to simbad.u-strasbg.fr (130.79.128.4).
      Escape character is '^]'.
      simbad/smbservc=1674:5077.6
  

• Outgoing NED connections (port 10011): we also use a simple TCP/IP socket connection to the NED name resolver to perform object queries. You will need to make sure that your firewall allows such a connection to be estabilished from your machine to the NED server:

      nedsrv.ipac.caltech.edu  (134.4.36.101)
  

  To test this connection, you should be able to simply type:

      telnet nedsrv.ipac.caltech.edu 10011
  

  and verify that you get a prompt back that looks like this:

      Trying 134.4.36.101...
      Connected to abell-z1.ipac.caltech.edu (134.4.36.101).
      Escape character is '^]'.
  

• Outgoing SMTP connections: The ADS abstract service offers the option of emailing results of a query to users. This means that the user under which the http daemon runs must be able to successfully execute /usr/lib/sendmail, and that outgoing email must be allowed to go through.

#!/bin/sh
#
# Startup script for the ADS Services
# For more information, please see:
# http://ads.harvard.edu/mirror
# 
# To install this script under RedHat linux, do the following (as root):
#    cp init.d_ads /etc/init.d/ads
#    chmod 744 /etc/init.d/ads
#    chkconfig --add ads
#    chkconfig ads on
#
# To install the startup procedure under solaris, do the following (as root):
#    cp init.d_ads /etc/init.d/ads
#    chmod 744 /etc/init.d/ads
#    ln -s /etc/init.d/ads /etc/rc3.d/S93ads
#
# chkconfig: 5 86 16
# description: NASA Astrophysics Data System services
#

case "$1" in
'start')
    # load shared memory segments under ads's uid
    su - ads -c '$HOME/mirror/local/ads_startup'
    ;;
'stop')
    # remove shared memory segments
    su - ads -c '$HOME/mirror/local/ads_shutdown'
    ;;
*)
    echo "Usage: $0 start|stop" 1>&2
    ;;
esac

• Since our servers are part of Harvard's networking infrastructure, we benefit from a high-speed link to the Internet2 consortium. If you have a way to configure your local routing to take advantage of this, you should definitely do so as we have seen transfer rates increase an order of magnitude when routed through the internet2 grid.
• We recommend mounting the partition containing ADS data files with the option noatime, which prevents the OS from updating inodes in the filesystem every time a single file is accessed. Since the ADS search engine accesses these files a lot, this can be a big win for a busy mirror site.
• We currently use EXT3 as the filesystem where our data is stored. If you do use EXT3 or EXT4 as your filesystem, make sure you disable file system checking at boot time to avoid having the server unavailable for a long time when it boots up. This is done using tune2fs (tune2fs -c 0 /dev/sda1). Please see the man page for tune2fs for more info.
• We used to have or data stored on an XFS file system for greater performance, but with today's hardware this does not seem to make much of a difference anymore. One of the advantages of XFS is that it eliminates the need for the extensive file checking a boot time if the system has been shutdown uncleanly and offers very good performance. Google for "xfs benchmarks" for current details. The downside of this is that when something goes wrong with the filesystem data recovery is quite painful.
• You should avoid having updatedb crawl through the filesystem hosting the ADS data since it is just a big waste of I/O resources. To disable this, simply edit /etc/updatedb.conf and add the top-level directory containing ADS data to the PRUNEPATHS variable.
• We strongly recommend that you setup your storage on a RAID partition, either software or hardware. If you do go with a hardware raid solution, make sure you read the fine print about fine-tuning the system for maximum performance for your particular OS and hard drives. We found that tweaking the parameters controlling the readahead block buffering made a big difference on our linux server.

Additional steps required to get the abstract files transferred to the mirror site and to get the ADS database search interface up and running will be taken by ADS staff (needs ADS admin authentication).