Ontology development processes (START HERE!)

Development

• Set up the catalog file
  

  There is an example ontology catalog catalog-v001.xml.example that should be installed by running cp catalog-v001.xml.example catalog-v001.xml. The file itself is not tracked to prevent conflicts when switching branches.

• Load remote imports from local copies
  

  Sometimes we don't want to have to retrieve remote copies of files every time we start protege, or we need to run tests for patches that are applied to remote imports when loading the ontology into SciGraph.

  cd ttl/
  cat <(head -n -1 catalog-v001.xml) ../catalog-extras <(tail -n 1 catalog-v001.xml) > catalog-new
  mv catalog-new catalog-v001.xml
  

  Local copies can be obtained using the following.

  cd ttl
  curl --header 'Accept-Encoding: gzip' --compressed --location --create-dirs \
  $(cat ../catalog-extras | cut -d'=' -f4 | cut -d'"' -f2 | sed 's/^/-o /') \
  --remote-name-all $(cat ../catalog-extras | cut -d'=' -f3 | cut -d'"' -f2)
  

• Running the reasoner
  

  Due to a bug in protege we need to patch our import chain in order to for the reasoner to run correctly.

  patch -p1 -i dev-reasoner.patch
  # OR
  cd ttl/
  patch -p2 -i ../dev-reasoner.patch
  # Then
  protege  # can be launched as you see fit
  

  1. In protege menu bar File -> Open and choose ttl/nif.ttl (after first load File -> Open recent is faster).
  2. Make sure reasoner is set to ELK (menu bar Reasoner -> ELK 0.4.3)
  3. Ctrl-R

  Protege does not have to be run from ttl/ in order to find ttl/catalog-v001.xml, however the file opened must be in ttl/.

• Changing an ontology file
  
  1. Edit your file and save.
  2. If using protege git diff to make sure that any new triples have landed in the correct file.
  3. Execute prior to commit processes.
• Adding a new ontology file
  

  touch filename.ttl
  add filename.ttl
  `make_catalog`  # NOTE this is broken at the moment
  

  1. Add an entry in catalog-v001.xml by hand. Note that you will need to make sure that catalog-v001.xml is copied from the example order to make use of the new entry.

  Before you run make_catalog make sure that the import chain is local (unpatched) otherwise it will take an extremely long time to fetch files.

• Prior to commit
  

  ttlfmt $(git status -s | grep ttl | grep M | cut -d' ' -f3)
  # OR if you are working on files that need to be normalized
  qnamefix $(git status -s | grep ttl | grep M | cut -d' ' -f3)
  

• Adding a new remote import
  

  All remote imports should be imported into NIFSTD via a bridge file. This file will hold any local additions that we want to make.

  1. Create a new bridge file by copying an existing bridge file and modifying as needed to import the remote file from its canonical iri, and to import filename-dead.ttl.
  2. Add an entry in catalog-extras for the new import.
  3. Create ttl/generated/filename-dead.ttl by running necromancy http://myurl.org/filename.owl.
  4. Whenever there is a new release repeat step 3.
• Integrating a new set of terms
  

  If there is an existing ttl file that has been formatted with ttlfmt as described above then create a new branch, add the file, and submit a pull request for review.

  If there is a chance that this file may need to be regenerated from an external source please add the external source and the generation code via the process described in pyontutils/nifstd/development/README.md. This does not need to be done if ttl file will become the source of truth once integrated into NIFSTD.

  Any set of terms that requires more than direct transformation to ttl should also follow the process described in pyontutils/nifstd/development/README.md.

• Check for incorrect or unimported predicates
  
  1. Remove /tmp/ttlcmp.patch if it exists
  2. ttlcmp.sh (lives in pyontutils/pyontutils)
  3. Review /tmp/ttlcmp.patch for annotation and object properties.

External source synchronization

cd ~/git/pyontutils &&  # adjust accordingly
pipenv shell

• SciCrunch Registry
  

  BRANCH="registry-$(date +%s)"
  registry-sync &&
  cd ${PATH_TO_ONTOLOGY} &&
  git pull &&  # alternately git clone and set pushurl -> script-user or something
  git checkout -B ${BRANCH} &&
  git push --set-upstream origin ${BRANCH} &&
  echo success
  # TODO auto github api pull
  

• parcellation
  

  TODO

• slimgen
  

  TODO

Release processes

• Run sanity checks on staging
  

  Besides the checks listed as part of the development process we need to integrate automated tests, probably using ROBOT. See also https://github.com/SciCrunch/NIF-Ontology/issues/60.

• versionInfo
  

  Only bump the versionInfo string on nif.ttl for major releases (e.g. one where nif.ttl is pushed to bioportal). Our versionIRI implementation on top of the unix epoch provides the logical end of granular versioning so we will use that for all practical versioning needs. It seems unlikely that we will bump to a 4.0 release any time in the near future so simply increase the count on the version by one for each release.

• versionIRI
  

  Run ontutils version-iri ttl/nif.ttl from the root of the ontology git directory. Then run ontutils iri-commit NIF-Ontology -l ${git-base} and copy and paste the git commit command and execute it. This can now be run on any branch and will resolve correctly when merged to master.

  TODO: Should we add a process for adding a versionIRI to individual files whenever there is a substantial non-cosmetic change? It is already possible to use the epoch based versionIRIs to refer to files even if we do not include the versionIRI in the file explicitly.

• Merge to master
  

  If the changes are minor don't bother with a pull request, run git checkout master && git merge staging and then push. If there are more major changes, such as changes to modelling of a domain or deprecation of ontology files, submit a pull request, even if you are the only person who will participate in it. This can provide a nice summary for someone who needs to review the history of changes but shouldn't have to dig through the full commit log.

• Build graph for SciGraph release
  

  This process has been automated via run-load-graph-nifstd or run-load-graph-sparc. See the nifstd SciGraph README for details.

• Build and deploy services config
  

  This only needs to be done if the services config has changed.
  This process has been automated via run-build-deploy-nistd or run-build-deploy-sparc. See the nifstd SciGraph README for details.

• Deploy to SciGraph
  

  See the RPM secion of the the pyontutils scigraph readme for the rest of the instructions for deployment.

• Restart dependent services
  
  1. Restart the ontree process on aux-resolver. This is needed to flush the cache of the import chain.

Resolver

Building documentation

• Setup
  

  See pyontutils/README.md#requirements for details on pandoc and orgstrap. For the best results use >=app-text/emacs-26.1 and >=app-text/pandoc-2.2.3.2. For centos 7 (i.e. the current aux-resolver sever) this means

  1. download, extract, build, and install emacs
  2. download, extract, and install the latest version of pandoc.
  3. git clone https://github.com/tgbugs/orgstrap.git and run orgstrap/orgstrap
• Build
  

  On aux-resolver SCICRUNCH_API_KEY=$(cat api-key) ont-docs

• Check links
  

  Use a crawler to check for broken links. For example using pylinkvalidator.py one can run the following.

  pylinkvalidate.py \
  --progress \
  --depth 1 \
  --workers 4 \
  --parser=lxml \
  --types=a \
  --accepted-hosts=github.com,scicrunch.org,bioontology.org,fairsharing.org,neuinfo.org,interlex.org,wordpress.com,w3.org \
  http://ontology.neuinfo.org/docs/
  

• Deploy
  

  On aux-resolver cp -a pyontutils/pyontutils/doc_build/docs /var/www/ontology-docs

External records

• Registries
  

  There are a couple of places where there are external records of the ontology that a maintainer or curator needs to keep up to date.

  1. https://scicrunch.org/resolver/SCR_005414 (NIF Ontology)
  2. https://scicrunch.org/resolver/SCR_005402 (NeuroLex)
  3. https://scicrunch.org/resolver/SCR_016178 (InterLex)
  4. https://bioportal.bioontology.org/ontologies/NIFSTD
  5. https://bioportal.bioontology.org/ontologies/NIFSUBCELL
  6. https://fairsharing.org/bsg-s002584 (NeuroLex)
  7. https://fairsharing.org/bsg-s002628 (NIF Ontology)
  8. https://fairsharing.org/bsg-s002835 (subcellular)
• Other references
  
  1. https://www.biccn.org/tools/atlas
• About pages
  

  Pages that need to be kept up to date / sourced from material here.

  1. https://neuinfo.org/about/components
  2. https://neuinfo.org/about/nifvocabularies
• Related search pages
  
  1. http://interlex.org
  2. https://scicrunch.org/browse/terminology