Invenio Installation Guide (Ubuntu)

This guide will take the reader through a step-by-step installation of Invenio and it's demo site in a virtual environment on recent Ubuntu versions.

Back to developer pages

Install requirements

(Based on http://invenio-software.org/wiki/Installation/InvenioOnDebian)

In this guide we are using aptitude to download packages, apt-get may also be used. To install aptitude run:

sudo apt-get install aptitude

Update and upgrade system

sudo aptitude -y update

sudo aptitude -y upgrade

Now install required packages.

PIP and virtualenv

Install Distribute, PIP (Python module manager), virtualenv and virtualenvwrapper. Read more about virtualenv here: http://www.virtualenv.org/en/latest/

wget https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py
sudo pip install -U pip
sudo pip install virtualenv virtualenvwrapper

System wide packages

sudo aptitude -y install redis-server \
                     libxml2-dev libxslt-dev python-software-properties g++ make \
                     install nfs-common libhdf5-dev \
                     libjpeg-dev libfreetype6-dev libtiff-dev \
                     python-dev virtualenvwrapper

Apache packages:

sudo aptitude -y install libapache2-mod-wsgi apache2-mpm-prefork libapache2-mod-xsendfile

Database:

For MySQL:

sudo aptitude -y install mysql-server mysql-client libmysqlclient-dev

(or alternatively:) MariaDB:

sudo aptitude -y install mariadb-server mariadb-client libmariadbclient-dev

Development tools

sudo aptitude -y install build-essential git-core subversion mercurial vim

Postfix

Used by Invenio to send error messages to your email inbox.

sudo aptitude install postfix

Just choose "Internet site" and leave the defaults.

If you want to check if Postfix is configured correctly you can use sendmail to test, for example like this: http://scratching.psybermonkey.net/2011/03/sendmail-how-to-test-sending-email.html

Installing Invenio

Fork and setup Invenio git repository:

• First fork the Invenio repo on github: http://github.com/inveniosoftware/invenio
• Then clone your fork:

cd ~/src
git clone git@github.com:yourgithubusername/invenio.git invenio

Install git-new-workdir

We suggest to get git-new-workdir for easy management of working tree's (will be used later).

command -v git-new-workdir >/dev/null 2>&1 || { mkdir $HOME/bin && cp /usr/share/doc/git/contrib/workdir/git-new-workdir $HOME/bin && chmod +x $HOME/bin/git-new-workdir }

Create the virtual environment

cd
mkvirtualenv master --no-site-packages
workon master
cdvirtualenv
mkdir src; cd src
~/src/git-new-workdir ~/src/invenio/ invenio master   # watch out, here "master" refers to the actual branch called master and not the venv.
cd invenio
pip install -r requirements.txt

PS: In this guide we assume the name "master" as virtualenv name. You are free to substitute it, just make sure you know what you are doing

Configuration and installation using GNU Autotools

Invenio uses GNU autotools for installation. This configuration step is mandatory. Usually, you do this step only once or sometimes after changing branches.

sudo aptitude install -y automake1.9 autoconf
workon master
cdvirtualenv src/invenio
aclocal
automake -a
autoconf

Launch the configuration and make. Here we have specified the virtualenv prefix and python executable. (Do not do this as sudo!)

./configure --prefix=$VIRTUAL_ENV --with-python=$VIRTUAL_ENV/bin/python
make -s
make -s install

Note that you will be instructed to perform a command similar to. *Please check output for exact command*:

ln -s $VIRTUAL_ENV/lib/python/invenio $VIRTUAL_ENV/local/lib/python2.7/site-packages/invenio
ln -s $VIRTUAL_ENV/lib/python/invenio $VIRTUAL_ENV/lib/python2.7/site-packages/invenio

ONLY SYMLINK ONE OF THEM DO NOT RUN BOTH - you will get cyclic links.

make -s install-jquery-plugins

Installs the necessary Javascript libraries.

Optional installations

make -s install-ckeditor-plugin (optional)

This will automatically download and install in the proper place CKeditor, a WYSIWYG Javascript-based editor (e.g. for the WebComment module). Note that in order to enable the editor you have to set the CFG_WEBCOMMENT_USE_RICH_EDITOR to True.

make -s install-pdfa-helper-files (optional)

This will automatically download and install in the proper place the helper files needed to create PDF/A files out of existing PDF files.

make -s install-mediaelement (optional)

This will automatically download and install the MediaElementJS HTML5 video player that is needed for videos on the DEMO site.

make -s install-solrutils (optional)

This will automatically download and install a Solr instance which can be used for full-text searching. See CFG_SOLR_URL variable in the invenio.conf. Note that you need to install java: sudo aptitude -y install openjdk-7-jdk

Also the admin later has to take care of running init.d scripts which would start the Solr instance automatically.

make install-js-test-driver (optional)

This will automatically download and install JsTestDriver which is needed to run JS unit tests. Recommended for developers.

Note that you need to install java: sudo aptitude -y install openjdk-7-jdk

Configuration

Now the software is installed, but Invenio configuration remains.

First you need to create your local Invenio configuration file. Here is an example of one:

workon master
cdvirtualenv etc

Here you can see invenio.conf. This is the main configuration file of Invenio and it contains all the defaults. We need to overwrite those defaults.

vim invenio-local.conf
[Invenio]
CFG_DATABASE_HOST = localhost
CFG_DATABASE_PORT = 3306
CFG_DATABASE_NAME = master
CFG_DATABASE_USER = master
CFG_SITE_URL = http://localhost:4000
CFG_SITE_SECURE_URL = http://localhost:4000
CFG_SITE_ADMIN_EMAIL = yourusername@cern.ch
CFG_SITE_SUPPORT_EMAIL = yourusername@cern.ch
CFG_SITE_NAME = Atlantis Fictive Institute of Science
CFG_SITE_NAME_INTL_fr = Atlantis Institut des Sciences Fictives
CFG_SITE_EMERGENCY_EMAIL_ADDRESSES = {'*': 'yourusername@cern.ch'}
CFG_BIBDOCFILE_ENABLE_BIBDOCFSINFO_CACHE = 1
CFG_BIBSCHED_PROCESS_USER = yourlocaluser

NB! Remember to replace the placeholder values with your own! Also yourlocaluser is NOT your CERN user name, it is the one you are logged in as in your development machine

To apply this configuration we use the inveniocfg program:

inveniocfg --update-all
inveniocfg --load-bibfield-conf

Next we need to create MySQL tables and database for the Invenio instance and set the MySQL user which have access to create the database and user:

NB: this part assumes you set your MySQL root password to "mysql". Change the commands as needed depending on your virtualenv name.

NB2: Make sure you understand what the commands below are doing. Otherwise you may lose data!

export VENV = master  # name assumed to be "master" for the rest of the tutorial
workon $VENV
export MYSQL_ADMIN_USER=root
mysql -u $MYSQL_ADMIN_USER --password=mysql -e "DROP DATABASE IF EXISTS $VENV"
mysql -u $MYSQL_ADMIN_USER --password=mysql -e "CREATE DATABASE IF NOT EXISTS $VENV DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci" 
mysql -u $MYSQL_ADMIN_USER --password=mysql -e "GRANT ALL PRIVILEGES ON $VENV.* TO $VENV@localhost IDENTIFIED BY 'my123p\$ss'"
mysqladmin -u $MYSQL_ADMIN_USER --password=mysql flush-privileges 

We use inveniocfg again to create database tables and demo site automatically:

inveniocfg --create-tables 
inveniocfg --create-demo-site
inveniocfg --load-demo-records

Run Invenio

In order to run the invenio server, it is recommended to use a dev-server with reload, such as invenio-devserver (https://pypi.python.org/pypi/invenio-devserver/0.4)

workon master
pip install invenio-devserver
serve

Done. Now visit your brand new Invenio installation as directed by serve.

Install INSPIRE overlay

If you want to install the INSPIRE overlay on top of Invenio, follow these steps:

First make sure to fork the repository on GitHub to your GitHub account and clone it locally.

cd ~/src
git clone git@github.com:yourusername/inspire.git

Then let's install it inside our virtualenv:

workon master
cdvirtualenv src
~/src/git-new-workdir ~/src/inspire/ inspire master   # watch out, here "master" refers to the actual branch called master and not the venv.
cd inspire
make -s install

Now to setup the DB properly:

export CFG_INSPIRE_BIBTASK_USER=admin
workon master
inveniocfg --remove-demo-records --drop-demo-site --drop-tables --create-tables
cdvirtualenv src/inspire
make install-dbchanges
make load-demo-records

The invenio-local.conf will now be a bit different:

workon master
cdvirtualenv etc
vim invenio-local.conf
[Invenio]
CFG_DATABASE_HOST = localhost
CFG_DATABASE_PORT = 3306
CFG_DATABASE_NAME = master
CFG_DATABASE_USER = master
CFG_SITE_URL = http://localhost:4000
CFG_SITE_SECURE_URL = http://localhost:4000
CFG_SITE_ADMIN_EMAIL = yourusername@cern.ch
CFG_SITE_SUPPORT_EMAIL = yourusername@cern.ch
CFG_SITE_EMERGENCY_EMAIL_ADDRESSES = {'*': 'yourusername@cern.ch'}
CFG_BIBDOCFILE_ENABLE_BIBDOCFSINFO_CACHE = 1
CFG_BIBSCHED_PROCESS_USER = yourlocaluser
CFG_WEBSTYLE_INSPECT_TEMPLATES = 0
CFG_SITE_LANG = en
CFG_SITE_LANGS = bg,ca,de,el,en,es,fr,hr,it,ja,no,pl,pt,ru,sk,sv,zh_CN,zh_TW
CFG_SITE_NAME = HEP
CFG_SITE_NAME_INTL_en = HEP
CFG_SITE_NAME_INTL_fr = HEP
CFG_SITE_NAME_INTL_de = HEP
CFG_SITE_NAME_INTL_es = HEP
CFG_SITE_NAME_INTL_ca = HEP
CFG_SITE_NAME_INTL_pt = HEP
CFG_SITE_NAME_INTL_it = HEP
CFG_SITE_NAME_INTL_ru = HEP
CFG_SITE_NAME_INTL_sk = HEP
CFG_SITE_NAME_INTL_cs = HEP
CFG_SITE_NAME_INTL_no = HEP
CFG_SITE_NAME_INTL_sv = HEP
CFG_SITE_NAME_INTL_el = HEP
CFG_SITE_NAME_INTL_uk = HEP
CFG_SITE_NAME_INTL_ja = HEP
CFG_SITE_NAME_INTL_pl = HEP
CFG_SITE_NAME_INTL_bg = HEP
CFG_SITE_NAME_INTL_hr = HEP
CFG_SITE_NAME_INTL_zh_CN = HEP
CFG_SITE_NAME_INTL_zh_TW = HEP
CFG_BIBINDEX_FULLTEXT_INDEX_LOCAL_FILES_ONLY = 1
CFG_WEBSTYLE_TEMPLATE_SKIN = inspire
CFG_WEBSEARCH_INSTANT_BROWSE = 0
CFG_WEBSEARCH_SPLIT_BY_COLLECTION = 0
CFG_INSPIRE_SITE = 1
CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS = 5
CFG_WEBCOMMENT_ALLOW_COMMENTS = 0
CFG_WEBCOMMENT_ALLOW_REVIEWS = 0
CFG_WEBCOMMENT_ALLOW_SHORT_REVIEWS = 0
CFG_WEBSEARCH_DEF_RECORDS_IN_GROUPS = 25
CFG_WEBSEARCH_NB_RECORDS_TO_SORT = 5000
CFG_WEBSEARCH_USE_MATHJAX_FOR_FORMATS = hb,hd
CFG_BIBDOCFILE_FILESYSTEM_BIBDOC_GROUP_LIMIT = 20000
CFG_WEBSEARCH_FULLTEXT_SNIPPETS = {'': 5}
CFG_WEBSEARCH_FULLTEXT_SNIPPETS_CHARS = {'': 100}
CFG_WEBSEARCH_FIELDS_CONVERT = {'eprint':'reportnumber','bb':'reportnumber',
                                'bbn':'reportnumber','bull':'reportnumber',
                                'r':'reportnumber','rn':'reportnumber',
                                'cn':'collaboration','a':'author',
                                'au':'author','name':'author',
                                'ea':'exactauthor','exp':'experiment',
                                'expno':'experiment','sd':'experiment',
                                'se':'experiment','j':'journal',
                                'kw':'keyword', 'keywords':'keyword',
                                'k':'keyword', 'au':'author', 'ti':'title',
                                't':'title', 'irn':'970__a',
                                'institution':'affiliation',
                                'inst':'affiliation', 'affil':'affiliation',
                                'aff':'affiliation', 'af':'affiliation',
                                'topic':'695__a','tp':'695__a','dk':'695__a',
                                'date':'year','d':'year','date-added':'datecreated',
                                'da':'datecreated','dadd':'datecreated',
                                'date-updated':'datemodified','dupd':'datemodified',
                                'du':'datemodified','vol':'volume',}
CFG_BIBRANK_SHOW_DOWNLOAD_GRAPHS_CLIENT_IP_DISTRIBUTION = 0
CFG_BIBRANK_SHOW_DOWNLOAD_GRAPHS = 0
CFG_BIBRANK_SHOW_DOWNLOAD_STATS = 0
CFG_BIBINDEX_AUTHOR_WORD_INDEX_EXCLUDE_FIRST_NAMES = True
CFG_WEBSEARCH_SEARCH_CACHE_SIZE = 0
CFG_WEBSTYLE_HTTP_STATUS_ALERT_LIST = 400,5*,41*
CFG_BIBAUTHORID_EXTERNAL_CLAIMED_RECORDS_KEY = external_arxivids
CFG_BIBSCHED_LOG_PAGER = /usr/bin/less
CFG_BIBAUTHORID_MAX_PROCESSES = 1
#
# Coden, journal, type code lookups
#
CFG_WEBSEARCH_SYNONYM_KBRS = {
  'journal': ['JOURNALS', 'leading_to_comma'],
  'collection': ['COLLECTION', 'exact'],
  'subject': ['SUBJECT', 'exact'],
  }
CFG_BIBEDIT_EXTEND_RECORD_WITH_COLLECTION_TEMPLATE = [('INSTITUTION', 'record_institution'),
                                                      ('CONFERENCES', 'record_conference'),
                                                      ('THESIS', 'record_thesis'),
                                                      ('PROCEEDINGS', 'record_proceedings'),
                                                      ('BOOK', 'record_book'),
                                                      ('HEP', 'record_article')]
CFG_BIBEDIT_QUEUE_CHECK_METHOD = regexp
# CFG_PLOTEXTRACTOR_SOURCE_BASE_URL = http://export.arxiv.org/
CFG_BIBSCHED_MAX_NUMBER_CONCURRENT_TASKS = 1
# CFG_BIBSCHED_NODE_TASKS = {
#   'pcudssw1507.cern.ch' : ['webauthorprofile','webcoll','refextract','bibupload','bibexport'],
#   'pcudssx1506.cern.ch' : ['bibindex']
#   }
CFG_SELFCITES_USE_BIBAUTHORID = 1
CFG_SELFCITES_PRECOMPUTE_FRIENDS = 0
CFG_WEBSEARCH_DISPLAY_NEAREST_TERMS = 0
CFG_WEBSEARCH_SPIRES_SYNTAX = 9
CFG_REFEXTRACT_TICKET_QUEUE = Inspire-References
CFG_BIBUPLOAD_STRONG_TAGS = 999,084
CFG_BIBFORMAT_DISABLE_I18N_FOR_CACHED_FORMATS = ha,hb,hs,hx,hdref
CFG_REFEXTRACT_KBS_OVERRIDE={'journals': 'kb:docextract-journals'}
CFG_WEBAUTHORPROFILE_USE_BIBAUTHORID = 1
CFG_WEBSEARCH_CITESUMMARY_SELFCITES_THRESHOLD = 0
CFG_BIBDOCFILE_ENABLE_BIBDOCFSINFO_CACHE = 1
CFG_BIBDOCFILE_DESIRED_CONVERSIONS = {}
CFG_SELFCITES_PRECOMPUTE = 1
CFG_WEBSEARCH_PREV_NEXT_HIT_FOR_GUESTS = 0

#CFG_BIBFORMAT_CACHED_FORMATS = wapdat,wapaff,hdref,hb,hx,ha
CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_DOCTYPES = [('INSPIRE-PUBLIC', 'INSPIRE-PUBLIC'),
                                                 ('arXiv', 'arXiv'),
                                                 ('Springer','Springer'),
                                                 ('JHEP','JHEP'),
                                                 ('Hindawi','Hindawi'),
                                                 ('APS','APS'),
                                                 ('Plot','Plot'),
                                                 ('PlotMisc', 'PlotMisc'),
                                                 ('Supplementary Material','Supplementary Material'),
                                                 ('Data','Data')]
CFG_BIBSORT_BUCKETS = 0
CFG_OAI_ID_PREFIX = inspirehep.net
CFG_OAI_SAMPLE_IDENTIFIER = oai:inspirehep.net:123
CFG_OAI_IDENTIFY_DESCRIPTION = <description>
  <eprints xmlns="http://www.openarchives.org/OAI/1.1/eprints"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://www.openarchives.org/OAI/1.1/eprints
                               http://www.openarchives.org/OAI/1.1/eprints.xsd">
      <content>
       <URL>http://inspirehep.net/</URL>
      </content>
      <metadataPolicy>
       <text>Free and unlimited use by anybody with obligation to refer to original record</text>
      </metadataPolicy>
      <dataPolicy>
       <text>Full content</text>
      </dataPolicy>
      <submissionPolicy>
       <text>Submission restricted. Submitted documents are subject of approval by OAI repository admins.</text>
      </submissionPolicy>
  </eprints>
 </description>
## CFG_BIBMATCH_FUZZY_MATCH_VALIDATION_LIMIT -- Determines the minimum percentage of the
## amount of rules to be positively matched when comparing two records. Should the number
## of matches be lower than required matches but equal to or above this limit,
## the match will be considered fuzzy.
CFG_BIBMATCH_FUZZY_MATCH_VALIDATION_LIMIT = 0.65

## CFG_BIBMATCH_SEARCH_RESULT_MATCH_LIMIT -- Determines the maximum amount of search results
## a single search can return before acting as a non-match.
CFG_BIBMATCH_SEARCH_RESULT_MATCH_LIMIT = 25

## CFG_BIBMATCH_LOCAL_SLEEPTIME -- Determines the amount of seconds to sleep
## between search queries on LOCAL system.
CFG_BIBMATCH_LOCAL_SLEEPTIME = 0.5
CFG_BIBMATCH_REMOTE_SLEEPTIME=5.0

Remember to update config:

workon master
inveniocfg --update-all

Et voila, you should have Inspire installed:

workon master
serve

Final notes

Once we are working on a virtualenv (after using workon master) in order to go to the root folder where invenio is installed, type:

cdvirtualenv

Inside this folder, Invenio is installed:

bin/

Contains all the Invenio executables, such as bibupload, inveniocfg, wellcoll, etc

etc/

Configuration files for all Invenio modules. It also includes the files invenio.conf and invenio-local.conf where all the configuration variables for Invenio are set

lib/

Contains the installed Invenio python modules in the virtual environment. The files related to Invenio can be found in lib/python/invenio

var/

var/www contains all static files used by Invenio

var/www/js JavaScript files

var/www/img Images and CSS files

var/log Log files from Invenio. Some important files:

• invenio.err: contains all the exceptions that happen on your Invenio site
• bibsched.err and bibsched.log: all the info related to bibsched

Back to developer pages