Software Distribution & Syncronization

By jbayer - Last updated: Monday, February 28, 2011 - Save & Share - Leave a Comment

I recently had to update about 30 different servers with the identical software.  This is an all-too-common occurance for an administrator, and there are many different systems available to do this.  However, most of them do too much and are generally non-trivial to set up and manage.  I wanted a system which would be easy to install, and easy for a non-administrator to use.  Enter PullIt, a simple system which solves these requirements:

  pullit.tar.gz (32.5 KiB, 313 hits)

PullIt Software Distribution System

TOC

0.  Introduction
1.  Description
2.  Installation Details
3.  Usage
4.  Parameter Definitions
5.  Commands
6.  Error Codes
7.  Example

0.  Introduction

Pullit is a small system which gives administrators the ability to remotely install files on various systems.  It has been designed to be simple to understand and simple to operate.  Based on the KISS principle, it doesn’t try to do everything.  There are many other packages which  provide much more extensive services, if you need more than this can provide.

This system does not have any security attached.  The code for all packages will be copied to all systems, only the required packages will be installed. What this means is that if you have a malicious user who wants to install a package and they are not supposed to, they will be able to copy the files and install them by hand.

1.  Description

PullIt consists of two parts, pullit.sh and pullitd.sh  The first, pullit.sh,  is the shell by which the administrator/user queues up a set of file to be distributed.  The second, pullitd.sh, is a daemon which is run periodictly by cron.

The basic idea is that there is a master server, on which the users/administrators will place files for distribution.  The files will be packaged up into  a shar file and placed into a distribution directory.

The clients will have the pullitd.sh script running in a cronjob.  The script firsts uses rsync to syncronize it’s local directory of packages with those on the server.  This design allows each client to also function as a server, which both relieves the load on the main server in cases where either there are a large number of clients, or if the server is a very low-powered machine. It also allows remote distribution of packages

The system is flexible enough to allow for multiple architectures, and multiple distribution groups.

So for example, you can have a “manager” and “employee” group, and inside each group there could be both 32 bit, 64 bit, and noarch architectures.  Packages intended for the manager group will not get installed on the  employee group, and vice-versa.  Currently a client can only have one group active; this will be modified in the future to allow for multiple  groups.

Also, when installing files, there is a configuration option which says that a backup should be kept.  When this is set, if the file exists then the system will first createa subdirectory in the destination directory and then MOVE (not copy) the files to it.  The reason for the move is in case the file is currently being executed, it will be able to continue to execute without having to worry about the file being overwritten.

When installing a client, the group the client belongs to is specified in the configuration file.

2.  Installation Details

On the server, the file pullit.sh will be installed to /usr/local/sbin (or whereever you specify), a configuration directory will be created in /etc, and a configuration file installed in the configuration directory.

On the client, the file pullitd.sh will be installed to /usr/local/sbin, a configuration directory will be created in /etc, and a configuration  file installed in the configuration directory.  Also, a file will be installed into /etc/cron.d

It is your responsibility to make sure that rsync is working properly on  all systems.

To install, unpack the tar file into a directory and run the  script:

install.sh

The install script can optionally use the “dialog” command to have a curses- based installer.  If you would like to use the text-based mode even if you have the dialog command available, invoke the installer as follows:

install.sh text

The installer should usually be run as root, however it will allow you to run as a regular user.  Obviously, a regular user will not be able to change system files or directories, so if a regular user is installing this then it will be installed local to that user.

The installer has the ability to create an installation script for the  clients.  I strongly recommend you use this, since the installation script  will also create ssh keys on the client and install them on the server.  This makes the complete installation process extemely easy.  The client installation creates the necessary ssh keys, and also asks for the group that the  client is in, and the architecture of the client.

3.  USAGE

The system runs as a simple shell with a limited number of commands.  The system is actually a bash shell script, and does not do any fancy screen manipulation, so you can very easily have this run in a batch mode.

The command to run is:

pullitd.sh

The valid shell commands are:

start packagename
end (see done)
done [ packagename [groupname]]
group groupname
noarch (shortcut for “arch noarch”)
arch 32 | 64 | noarch  (anything will work, as long as all clients use the same)
noarch is special, it means that ALL platforms in the group will install the package
add [ file [file]] destinationDirectory
del file [file...]  | all
commit all | packagename [group]
list [ files | packages | groups | archtypes ]  (defaults to files)
runbefore filename [in directory] [rc expected]
runafter filename [in directory] [rc expected]
prune packagename [packagename...] | all
at [ yyyy-mm-dd [ hh:mm:ss ] ]
quit
exit

4.  Parameter Definitions

packagename Name of the package being worked on.
groupname Name of the group the current package is intended for
filename Name/path of the file being installed/deleted

You can use quotes, either single or double, to enclose a parameter with spaces.  Also, you can use a backslash to escape the next character; this will allow you to embed a quote inside a single parameter.  All quotes,  except for a quoted parameter at the end, must be closed with the same quote which started the quoting.

5.  Commands

start [ packagename ] Use this command to start building a package.  Specify the name of the package with the command.  The packagename needs  to obey the rules for directory names.  The packagename is optional at this point, but will need to be specified on  the “end” command if not entered here.
group groupname An alternate method of specifying what group the current package will be distributed to.  If specified with this command, it will not need to be specified with the “end”  command
(groupname)
A shortcut to entering a group.  This relies on the allowable groups being defined in the configuration file.
done | end [ packagename [ groupname ] ] This command finishes a package.  However, the package is NOT released for distribution yet, use the “commit” command for that.  The packagename and groupname are optional here, in  that if they were specified using the “start” and “group” command you wouldn’t need to specify them here.
arch archtype | noarch The archicture specifies the destination system.  While it would be advisable to use the same arch as the basic OS, there is no technical reason to do so.  This allows you to specify any number of archictures.  One reason to do so is to use a single server to distribute files to 32 bit, 64 bit, x86 x86_64, arm avr32 blackfin m68k m68knommu microblaze  mips powerpc.  However, if you specify any architecture types during the server install, only those types will be allowed.
noarch This is a shortcut for “arch noarch”
(archtype) If the archtype is simply typed on a line by itself, and it doesn’t match a command name, then the architecture will be set to the entered archtype.  This is similar to the “noarch” command as a shortcut.  This relies on the default architectures being specified in the configuration file.
add [ filename [ filename ]…] destinationDirectory Multiple files can be specified here.  This command adds files to the package; the last paramenter is the destination directory on the client computers.  The files don’t have to be in the current directory, the full paths will be stripped before building the package.
del filename | all Delete the specified file from the current package.  If “all” is specified, then delete all files from the current package.
commit all | packagename [ groupname ] Commit a package for distribution.  If a groupname is  specified, then only commit packages which are ready for the specified group.  If “all” is specified, then commit  all available packages for distribution.
list [ files | packages | groups | archtypes ] This will list the following types of information:

files            All files in the current package
packages    All packages ready for commit
groups       All groups with packages ready for commit
archtypes   All allowed architecture types

runbefore filename [ inDirectory [ expectedReturncode ] ] Specify a command or program to run before the files are installed.  These commands are run BEFORE any files are  renamed for backup.  If inDirectory is specified, then  the command will be run in the specified directory.  If specified, the return code from the command must match the expectedReturncode, otherwise the install will abort.  If the directory the command is run in is not important, but you do want to specify the returncode, then use a . as the directory.
runafter filename [ inDirectory [ expectedReturncode ] ] Specify a command or program to run after the files are installed.  These commands are run BEFORE any files are  renamed for backup.  If inDirectory is specified, then  the command will be run in the specified directory.  If specified, the return code from the command must match the expectedReturncode, otherwise the install will abort.  Note that since the files have been installed and backups made if specified, that an abort at this point could leave your systems in an unexpected state.
prune packagename [packagename...] | all Prune specified packages from the cache directory, or all packages
at [ yyyy-mm-dd [ hh:mm:ss ] ] Install the package at or after the specified date/time
quit Both quit and exit will exit the shell.
exit

6.  Error Codes

The following error codes can be returned by the pullit.sh script.  These may be useful if you are calling it from a script:

0       All ok
1       Missing configuration file
2       Missing bash_arrays.sh file
3       Unable to create libdir
4       Unable to install sharutils with yum
5       Niether yum nor apt-get detected
6       unable to install sharutils with apt-get
7       Unable to create the cache dir

The following error codes can be returned by the pullitd.sh script:

0       All ok
1       Missing configuration file
2       Unable to create libdir
3       Unable to create the top journaldir
4       Unable to create the cachedir
5       Transfer program missing
6       Unable to create the lower journal directory
7       Not running as specified user

7.  Example

Lets say that you have a number of web servers, all being load-balanced together in a colo facility.  You need to update some files in a directory on all the servers.  A problem is that some of the servers are always down for maintenance or other problems.

Here is the situation:

Main HTML directory: /var/www/html
Group: webserver
Directory containing files to be updated: /var/www/html/mydir
Files to be updated/added: index.php .htaccess config.php

First, copy the files to the master server:

scp index.php .htaccess config.php jbb@master:

Then log in to the master:

ssh jbb@master

Now start the pullit shell:

pullit.sh

The shell gives the following prompt:

pullit>

Enter the following commands, one per line:

start mydir
group webserver
arch noarch
add index.php .htaccess config.php /var/www/html/mydir
runbefore "/etc/init.d/httpd stop"
runafter "/etc/init.d/httpd start"
end
commit all
quit

Note that because there are spaces in the commands you want to run, you  need to surround the command with quotes.

The commands do the following:

start mydir
Start a package called "mydir"
group webserver
This package is intended for the group called "webserver"
arch noarch
No architecture, so this will be installed on all servers in the group
add index.php .htaccess config.php /var/www/html/mydir
Add the three files to the package, they will be installed to the /var/www/html/mydir directory
runbefore "/etc/init.d/httpd stop
Stop the webserver before doing the installation
runafter "/etc/init.d/httpd start
start the webserver after the install is done
end
End the package, ready for commit
commit all
Commit all packages.  At this point, the packages are available for the servers to pull them down and install
quit
exit the pullit shell
Posted in Administration, Bash, Open Source • • Top Of Page
574 views

Write a comment

asd