InterLex alternate resolver setup
Table of Contents
InterLex alt is a reduced set of the InterLex codebase for serving
directly from the mysql database. The necessary subset of the code
that is needed is copied into this directory and installed from the
main interlex source.
Setup
Install wheel on build machine
The build machine will work with any version of python, however it that
version does need to have wheel installed e.g. via pip install wheel.
Install python on server
Install python3.7-devel (or greater), pip, and pipenv on the server that
will run InterLex Alt. For example on CentOS
sudo yum install centos-release-scl-rh sudo yum install rh-python38-python rh-python38-python-devel # devel needed for pyxattr sudo /opt/rh/rh-python38/root/usr/bin/python3.8 -m ensurepip sudo /opt/rh/rh-python38/root/usr/local/bin/pip3.8 install pipenv sudo ln -s /opt/rh/rh-python38/root/usr/local/bin/pipenv /usr/bin/pipenv
Set environment variables
Set INTERLEX_DEPLOY_USER, INTERLEX_USER, and INTERLEX_SERVER
environment variables. The INTERLEX_DEPLOY_USER should have sudo
access, INTERLEX_USER should NOT have sudo or wheel access.
An example would be
INTERLEX_DEPLOY_USER=user INTERLEX_USER=interlex INTERLEX_SERVER=localhost
Create interlex runtime user account
Do this manually right now for sanity.
The GID is matched to the conventions we use for gentoo docker images.
GID=839 groupadd -g ${GID} interlex useradd -m -k /etc/skel -u ${GID} -g ${GID} -d /var/lib/interlex interlex chmod 0755 /var/lib/interlex
Build and deploy code and config
Get the absolute path to the script and use it to find the alt folder.
SOURCE="${BASH_SOURCE[0]}" while [ -h "$SOURCE" ]; do # resolve all symlinks DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )" SOURCE="$(readlink "$SOURCE")" [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE" # resolve relative symlinks done ABS_PATH="$( cd -P "$( dirname "$SOURCE" )" && pwd )" ALT_PATH="${ABS_PATH}/../"
Go to the alt folder and clean out any existing build files.
pushd "${ALT_PATH}" && git clean -dfx && # cleans only the alt subdir git checkout HEAD -- resources/filesystem/ # prevent stale user popd || exit 1
Build python wheels and package them for deployment.
pushd "${ALT_PATH}" && python setup.py bdist_wheel --universal && python setup.py clean --all && rm -rf ./*.egg-info && mv dist/* run/ && rmdir dist && #pipenv install # leave this out for now due to gunicorn detection issues rm alt.zip; zip -r alt.zip README.org && zip -r alt.zip run/ && popd || exit 2
Deploy the build artifact to the server.
SOURCE="${BASH_SOURCE[0]}" while [ -h "$SOURCE" ]; do # resolve all symlinks DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )" SOURCE="$(readlink "$SOURCE")" [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE" # resolve relative symlinks done ABS_PATH="$( cd -P "$( dirname "$SOURCE" )" && pwd )" ALT_PATH="${ABS_PATH}/../" TEMP_DIR=$(ssh ${INTERLEX_DEPLOY_USER}@${INTERLEX_SERVER} "mktemp -d") TD_EXIT=$? if [ $TD_EXIT -ne 0 ]; then exit $TD_EXIT fi pushd "${ALT_PATH}" && # so apparently we're deploying on things so old that rsync doesn't have the commands on the remote it needs #rsync --rsh ssh --archive --verbose alt.zip ${INTERLEX_DEPLOY_USER}@${INTERLEX_SERVER}:${TEMP_DIR}/alt.zip || exit 20 scp alt.zip ${INTERLEX_DEPLOY_USER}@${INTERLEX_SERVER}:${TEMP_DIR}/alt.zip || exit 20 popd || exit 3
The first time InterLex Alt is set up on a server run https://github.com/tgbugs/interlex/blob/master/alt/bin/config-build.sh.
<<&alt-path>> <<&pushd-clean>> grep -rl interlex resources/filesystem/ | xargs sed -i "s/{:interlex-user}/${INTERLEX_USER}/g" && <<&build-alt-zip>> zip -r alt.zip resources/filesystem/ # first time only add deploy files
Subsequently run https://github.com/tgbugs/interlex/blob/master/alt/bin/build.sh.
<<&alt-path>> <<&pushd-clean>> <<&build-alt-zip>>
Remote commands
After deploying the files to the server run the following remote
commands on INTERLEX_SERVER. NOTE: Make sure you create a
~/.mypass file that conforms to the syntax of ~/.pgpass i.e.
each line should look like server.url.org:port:dbname:user:password
and should have read write permission only for your user (chmod 0600).
If you do not a warning will appear and the server will not start.
The first time run https://github.com/tgbugs/interlex/blob/master/alt/bin/config-remote.sh.
<<&rsync-zip>>
ssh ${INTERLEX_DEPLOY_USER}@${INTERLEX_SERVER} "
<<&config-remote-command>>
"
SSH_EXIT=$?
if [ $SSH_EXIT -eq 15 ]; then
echo you need to edit ~/.mypass on ${INTERLEX_SERVER} as ${INTERLEX_USER} to complete setup
echo the pattern used to set the password is deocumented in step five of README.org on the server
exit $SSH_EXIT
elif [ $SSH_EXIT -ne 0 ]; then
echo remote command failed with $SSH_EXIT
exit $SSH_EXIT
fi
mv ${TEMP_DIR}/alt.zip /var/lib/interlex/alt.zip
rmdir ${TEMP_DIR}
chown ${INTERLEX_USER}:${INTERLEX_USER} /var/lib/interlex/alt.zip
rm -rf run/
rm -rf resources/filesystem/
unzip -o alt.zip || exit 1
/bin/cp -f resources/filesystem/etc/systemd/system/ilxalt.service /etc/systemd/system/ || exit 2 /bin/cp -f resources/filesystem/etc/systemd/system/ilxalt.socket /etc/systemd/system/ || exit 3 /bin/cp -f resources/filesystem/etc/tmpfiles.d/ilxalt.conf /etc/tmpfiles.d/ || exit 4 /bin/cp -f resources/filesystem/etc/nginx/sites-available/uri.interlex.org.conf /etc/nginx/sites-available/ || exit 5 # carful here XXX DO NOT NUKE FROM ORBIT THANKS unlink /etc/nginx/sites-enabled/uri.interlex.org.conf ln -s /etc/nginx/sites-available/uri.interlex.org.conf /etc/nginx/sites-enabled/uri.interlex.org.conf || exit 6 systemd-tmpfiles --create || exit 7 systemctl daemon-reload || exit 8 systemctl enable ilxalt || exit 9
pipenv --rm # the very first time this can fail pipenv install --skip-lock || exit 11
touch .mypass || exit 13 chmod 0600 .mypass || exit 14
sudo mv ${TEMP_DIR}/alt.zip /var/lib/interlex/alt.zip
sudo rmdir ${TEMP_DIR}
sudo chown ${INTERLEX_USER}:${INTERLEX_USER} /var/lib/interlex/alt.zip
pushd /var/lib/interlex || exit 22
sudo -u ${INTERLEX_USER} rm -rf run/
sudo -u ${INTERLEX_USER} rm -rf resources/filesystem/
sudo -u ${INTERLEX_USER} unzip -o alt.zip || exit 1
sudo /bin/cp -f resources/filesystem/etc/systemd/system/ilxalt.service /etc/systemd/system/ || exit 2
sudo /bin/cp -f resources/filesystem/etc/systemd/system/ilxalt.socket /etc/systemd/system/ || exit 3
sudo /bin/cp -f resources/filesystem/etc/tmpfiles.d/ilxalt.conf /etc/tmpfiles.d/ || exit 4
sudo /bin/cp -f resources/filesystem/etc/nginx/sites-available/uri.interlex.org.conf /etc/nginx/sites-available/ || exit 5 # carful here XXX DO NOT NUKE FROM ORBIT THANKS
sudo unlink /etc/nginx/sites-enabled/uri.interlex.org.conf
sudo ln -s /etc/nginx/sites-available/uri.interlex.org.conf /etc/nginx/sites-enabled/uri.interlex.org.conf || exit 6
sudo systemd-tmpfiles --create || exit 7
sudo systemctl daemon-reload || exit 8
sudo systemctl enable ilxalt || exit 9
pushd run || exit 10
sudo -u ${INTERLEX_USER} pipenv --rm # the very first time this can fail
sudo -u ${INTERLEX_USER} pipenv install --skip-lock || exit 11
popd || exit 12
sudo -u ${INTERLEX_USER} touch .mypass || exit 13
sudo -u ${INTERLEX_USER} chmod 0600 .mypass || exit 14
if [ ! -s .mypass ]; then
echo ~/.mypass has no records
exit 15
fi
popd || exit 16
sudo systemctl restart ilxalt &&
sleep 5
sudo systemctl is-active --quiet ilxalt
if [ $? -ne 0 ]; then
sudo journalctl -u ilxalt.service -n 50
exit 100;
fi
sudo systemctl restart nginx
sudo systemctl restart ilxalt &&
sleep 5
sudo systemctl is-active --quiet ilxalt
if [ $? -ne 0 ]; then
sudo journalctl -u ilxalt.service -n 50
exit 100;
fi
Subsequently run https://github.com/tgbugs/interlex/blob/master/alt/bin/remote.sh.
mv -f run/*.whl .
rm run/Pipfile.lock
unzip -o alt.zip || exit 1
pipenv --rm
pipenv install *.whl --skip-lock || exit 3
sudo mv ${TEMP_DIR}/alt.zip /var/lib/interlex/alt.zip
sudo rmdir ${TEMP_DIR}
sudo chown ${INTERLEX_USER}:${INTERLEX_USER} /var/lib/interlex/alt.zip
pushd /var/lib/interlex
sudo -u ${INTERLEX_USER} mv -f run/*.whl .
sudo -u ${INTERLEX_USER} rm run/Pipfile.lock
sudo -u ${INTERLEX_USER} unzip -o alt.zip || exit 1
pushd run || exit 2
sudo -u ${INTERLEX_USER} pipenv --rm
sudo -u ${INTERLEX_USER} pipenv install *.whl --skip-lock || exit 3
popd || exit 4
popd || exit 5
sudo systemctl restart ilxalt &&
sleep 5
sudo systemctl is-active --quiet ilxalt
if [ $? -ne 0 ]; then
sudo journalctl -u ilxalt.service -n 50
exit 100;
fi
<<&rsync-zip>>
ssh ${INTERLEX_DEPLOY_USER}@${INTERLEX_SERVER} "
<<&remote-command>>
"
SSH_EXIT=$?
if [ $SSH_EXIT -ne 0 ]; then
exit $SSH_EXIT
fi
Testing
On a redeploy, the easiest way to test whether everything is working is
to change TestRoutes.host in test/test_alt.py to match the test server
and then run python -m unittest test/test_alt.py. TODO add this to the
deploy scripts for the test server? Simple testing
TEST_HOST=test.host; curl --header 'Host: uri.interlex.org' http://${TEST_HOST}/base/ilx_0109470.ttl TEST_HOST=test.host; curl --header 'Host: uri.interlex.org' http://${TEST_HOST}/sparc/ontologies/community-terms.ttl