NCBO Wiki - User contributions [en]

File:Ncbo architecture.png

2016-03-30T15:52:45Z

Vemonet:

A diagram describing the NCBO architecture.
Generated using [https://www.yworks.com/products/yed yEd].
Feel free to contact vincent.emonet@lirmm.fr to get the graphml version of the file to edit it

Advanced Administration

2016-03-30T15:47:36Z

Vemonet:

[[Category:NCBO Virtual Appliance]]

On this page you will find some guidelines on how Advanced administration of the BioPortal Virtual Appliance. To proceed you will need a ssh connexion to your Appliance.<br/>
To simplify the examples we will consider we are using a Virtual Appliance deployed at '''http://my-bioportal-vm/'''

= NCBO architecture diagram =

[[File:Ncbo_architecture.png]]

= Solr search index =

== Access to solr ==

It is hosted by the tomcat on the port 8082
http://my-bioportal-vm:8082/solr

There are 2 cores:

* core1 is used for search
* core2 is used when doing a full reindexing (it allows the search to work fine when reindexing the whole bioportal)

To swap the 2 cores:

* Go to core admin tab
* Select core1
* Hit the "Swap" button

== Reindex all ontologies in core2 ==

Re-indexing in the core2 allows the BioPortal Appliance to not stop working during the reindexing (you just have to swap the cores when the reindexing is over)

bin/ncbo_ontology_index -a -l logs/reindexing_all.log -c http://my-bioportal-vm:8082/solr/core2

== Reindex only the specified ontologies in core2 ==

bin/ncbo_ontology_index -o STY,SNOMED -l logs/reindexing_STY_SNOMED.log -c http://my-bioportal-vm:8082/solr/core2

= Updating the Annotator cache and dictionary =

See the [http://www.bioontology.org/wiki/index.php/Annotator_Documentation#Update_cache_and_dictionary Annotator Documentation]

= Common troubles =

== ontologies_report.json is missing ==

When running some job you can face the following error:
ontologies_report.json : No such file or directory

To avoid it you have to create a ''ontologies_report.json'' file at the following location:
/srv/reports/ontologies_report.json

It has to be owned by ncbobp:ncbobp and to have permission 666

File:Ncbo architecture.png

2016-03-30T15:45:08Z

Vemonet: A diagram describing the NCBO architecture

A diagram describing the NCBO architecture

Google Analytics

2016-03-06T23:23:45Z

Vemonet: Created page with "Category:NCBO Virtual Appliance = Adding Google Analytics to the BioPortal Appliance = == Google developer account == The Google API to process our website data. Create..."

[[Category:NCBO Virtual Appliance]]

= Adding Google Analytics to the BioPortal Appliance =

== Google developer account ==
The Google API to process our website data. Create an account at https://console.developers.google.com/project
* Create an account
* Generate credentials (p12 key & email) and save it for later
** API Manager > Credentials > Add credentials > Service account
** Note: default password of the p12 keys is notasecret

* Enable Analytics API
** API Manager > Overview > Advertising APIs > Analytics API

* Copy p12 keys to the appliance (in the /srv directory for example)
** Example: scp mybioportal-ffad56jdic91ky.p12 my_bioportal_va:/srv/

== Google analytics account ==
Get our website data. Create an account at https://www.google.com/analytics/web
* Create an account
* Add the "tracking ID" of this account in /srv/rails/BioPortal/current/config/bioportal_config_production.rb
** Example: `$ANALYTICS_ID = "UA-63691424-3"`

* Give permission to your developer credential (previously generated at the console.developer step) to be able to retrieve the data
** Admin tab > Property (in general, it is the website your doing analytics on) > User Management
** Add permissions for the google developer credential email (generated in the previous section)

* Add in /srv/ncbo/ontologies_api/current/config/environments/appliance.rb & /srv/ncbo/ncbo_cron/config/appliance.rb

<code>
LinkedData.config do |config|
config.enable_ontology_analytics = true
end
NcboCron.config do |config|
config.enable_ontology_analytics = true
config.cron_ontology_analytics = "30 */4 * * *"
# Google Analytics config
config.analytics_service_account_email_address = "account-1@mybioportal-1145.iam.gserviceaccount.com" # The email address you get when creating the google developer account
config.analytics_path_to_key_file = "/srv/agroportal-ffad56jdic91ky.p12" # you have to get this file from the first step
config.analytics_profile_id = "ga:111821946" # replace with your ga view id
config.analytics_app_name = "mybioportal"
config.analytics_app_version = "1.0.0"
config.analytics_start_date = "2015-10-01"
config.analytics_filter_str = ""
end
</code>

= Tests using API explorer =

https://developers.google.com/apis-explorer/#search/analytics/m/analytics/v2.4/analytics.data.get

https://ga-dev-tools.appspot.com/query-explorer/

<code>
id = ga:111821946
start-date = 30daysAgo
end-date = yesterday
metrics = ga:pageviews
dimensions = ga:pagePath,ga:year,ga:month
filters
</code>

= Removing Analytics from UI =

* In /srv/rails/BioPortal/current/app/views/home/index.html.haml add:

<code>
display: none;
%fieldset{style: "display: none;"}
%legend
Ontology Visits #{"in full #{$SITE} " if at_slice?} (#{@analytics.date.strftime("%B %Y")})
</code>

* In /srv/rails/BioPortal/current/app/views/ontologies/_visits.html.haml add:

<code>
display:none;
#visits_content{:style => "margin: 2em 1em 0;"}
</code>

Main Page

2016-02-03T17:00:16Z

Vemonet: /* Technical Documentation */

Welcome to the '''NCBO Public WIKI'''. This WIKI provides information about collaborative ontology development efforts, dissemination events conducted by the NCBO and publically shared materials from the NCBO.
Please visit our [http://www.bioontology.org Main Site] for more information and materials.

=== Software ===
*[http://bioportal.bioontology.org NCBO BioPortal] - Home page for BioPortal Web site
* [http://sparql.bioontology.org/ NCBO BioPortal SPARQL Endpoint] - Access BioPortal via SPARQL queries (obsolete)
* [http://www.bioontology.org/wiki/index.php/Category:NCBO_Virtual_Appliance BioPortal Virtual Appliance] - Contains BioPortal, Annotator, and Resource Index.
* [http://www.bioontology.org/wiki/index.php/BioPortal_Release_Notes BioPortal Software Release Announcements]
*[[NCBO-OOR Development]] - Documentation on how the Open Ontology Repository community can customize and advance BioPortal capabilities.

=== User Support Documentation ===
*[[BioPortal Help]] - General BioPortal help documentation.
* [http://www.bioontology.org/wiki/index.php/Using_NCBO_Technology_In_Your_Project Using NCBO Technology in Your Project] - Documentation on how to use NCBO Technology in your project

=== Technical Documentation ===
* [http://data.bioontology.org/documentation NCBO REST Web services] - Listing of all NCBO web services with usage documentation
* [http://www.bioontology.org/wiki/index.php/SPARQL_BioPortal BioPortal SPARQL Endpoint] - BioPortal SPARQL documentation
* [http://www.bioontology.org/wiki/index.php/Category:NCBO_Virtual_Appliance BioPortal Virtual Appliance] - Contains BioPortal, Annotator, and Resource Index.
* [http://www.bioontology.org/wiki/index.php/NCBO_Developer_Documentation Developer Documentation] - Some documentation for developers

=== Ontology Development Projects ===
Below are links for ontology development projects that NCBO is involved in.
*[[BRO:Main_Page]] - Biomedical Resource Ontology
*[[TMO:]] - Translational Medicine Ontology
*[[NPO:Main_Page]] - NanoParticle Ontology
*[[PATO:Main_Page]] - An ontology of phenotypic qualities
*[[RO:Main_Page]] - OBO Relations ontology
*[[CARO:Main_Page]] - Reference anatomical ontology
*[[CL:Main_Page]] - OBO Cell ontology
*[[SO:Main_Page]] - Sequence Ontology
*[[Disease:Main_Page]] - Disease Ontology, and other ontologies pertaining to diseases and disorders
*[[OCI:Main_Page]] - Ontology of Clinical Investigation

Importing UMLS To Virtual Appliance

2016-02-01T17:12:36Z

Vemonet:

[[Category:NCBO Virtual Appliance]]

<p>The NCBO Virtual Appliance supports [http://www.geneontology.org/GO.format.obo-1_2.shtml OBO] and [http://www.w3.org/TR/owl-features/ OWL] ontology formats but not UMLS in its native form. To bridge this gap, we have developed a project called [https://github.com/ncbo/umls2rdf/ UMLS2RDF] that transforms UMLS ontologies into OWL/RDF.</p>

<p>UMLS2RDF is a Python script that connects to a UMLS MySQL installation and extracts the UMLS ontologies in a format that the Appliance can work with.</p>

<h3>Install UMLS MySQL</h3>

<p>To import UMLS ontologies, a local installation of the [http://www.nlm.nih.gov/research/umls/implementation_resources/scripts/index.html UMLS MySQL release] needs to be available. Please refer to the [http://www.nlm.nih.gov/research/umls/new_users/index.html UMLS documentation] for instructions on how to install the UMLS MySQL distribution.</p>

<h3>Install UMLS2RDF</h3>

<ol>
<li>First clone the github project:<br/><code>git clone https://github.com/ncbo/umls2rdf/</code></li>
<li>Install the MySQL Python driver. We recommend to use <code>pip</code> for this:<br/><code>pip install MySQL-python</code></li>
</ol>

<h3>Configure UMLS2RDF</h3>

<p>UMLS2RDF has two configuration files:</p>

<ol>
<li><strong>conf.py</strong> where the database configuration (host,name,user and password) needs to be specified. Also the output folder.</li>
<li><strong>umls.conf</strong> where one can specified the UMLS ontologies to be extracted. This is a comma separated file with the following 4 fields:
<ol type="a">
<li>SAB</li>
<li>This is legacy. Any value works.</li>
<li>Output file name.</li>
<li>Conversion strategy. Accepted values (load_on_codes, load_on_cuis).</li>
</ol>
</li>
</ol>

<p>With <em>load_on_codes</em> the original source of the ontology will be used as strategy. The Class IDs will be constructed with the MRCONSO.CODE field. If <em>load_on_cuis</em> is selected then the strategy to transform the ontology will use CUIs to construct the Class IDs.</p>

<p>In our [https://github.com/ncbo/umls2rdf/blob/master/umls.conf configuration file], you can see the settings used by our production system. These are all the UMLS ontologies that are publicly available in BioPortal.</p>

<h3>Run UMLS2RDF</h3>

<p>Once the configuration files have the settings run the command:</p>

<p><code>python umls2rdf.py</code></p>

<p>Depending on how many ontologies are extracted the run time can range from a few minutes to four hours. This process is memory intensive and to transform the largest UMLS ontologies (i.e: SNOMED) one needs at least 16G RAM available.</p>

<h3>Upload files to the NCBO Virtual Appliance</h3>

<p>The output files will be located in the folder specified in <strong>conf.py</strong>. Use the BioPortal Web form available in your appliance to submit the extracted ontologies. <strong>IMPORTANT:</strong> The ontology format in the submission form should be UMLS.</p>

<h3>Hardware Considerations</h3>

<p>NCBO dedicates a fair amount of resources (powerful servers) to handle a good portion of UMLS ontologies. Some of the UMLS ontologies contain millions of classes. To import the largest UMLS ontologies (i.e: RXNORM or SNOMEDCT) Users will have to run the Appliance in a powerful dedicated environment with 8GB RAM and 5GB hard disk space available.</p>

<h2>Example Workflow</h2>

<p>This workflow for importing UMLS data has been provided by Vincent Emonet, and has been provided here without testing by the NCBO team. Vincent comments:</p>
<blockquote>This offers more details on how to generate UMLS turtle files, It is simplified to provide the essentials needed to generate the RDF files you want. Note that I am using linux and the 2015AB release for this tutorial.</blockquote>

<h3>Install UMLS using mmsys</h3>

<ul>
<li>Download everything from https://www.nlm.nih.gov/research/umls/licensedcontent/umlsknowledgesources.html</li>
<li>Unzip mmsys.zip</li>
<li>Put the following files in the now unzipped mmsys directory : </li>
<ul>
<li>2015AB-1-meta.nlm</li>
<li>2015AB-2-meta.nlm</li>
<li>2015AB-otherks.nlm</li>
<li>mmsys.zip (why not?)</li>
<li>2015AB.CHK</li>
</ul>
<li>./run_linux.sh (or run.bat or run_mac.sh)</li>
<code>Install UMLS</code>
<li>Source: path to mmsys directory and Destination: path of a directory where the subset file will be generated</li>
<li>Semantic Network -> Choose Database Load Scripts: Mysql 5.6 (to generate the mysql load script for the semantic network subset, aka STY in bioportal)</li>
<li>Select "New configuration.."</li>
<li>Select Default Subset: select the default subset you want (it is not really important, you can specificaly choose each thesaurus in the next step)</li>
<li>Go to the "Output Options" tab > Write Database Load Scripts</li>
<ul><li>Select database > MySQL 5.6</li></ul>
<li>Go to the "Source List" and select the sources (aka ontologies in bioportal) you want</li>
<ul>
<li>hold ctrl to select many</li>
<li>careful there is an option to define if you want the selected source excluded or included in the subset</li>
</ul>
<li>Then click on "Done" in the top command on the window</li>
<li>And wait for UMLS to install</li>
<li>Note: mmsys will generate RRF files. You can use the mmsys software to browse it, but we will just use it to populate a SQL database</li>
</ul>
<h3>Load subset in MySQL</h3>
<ul>
<li>Go to mysql</li>

<code>>create database umls2015ab;</code></li>
<li>Configure database character encoding to UTF-8</li>
<code>>ALTER DATABASE umls2015ab CHARACTER SET utf8 COLLATE utf8_unicode_ci;</code></li>
<li>Go to the 2015AB directory (where UMLS has been installed) in the META directory</li>
<li>Open the populate_mysql_db.sh script and change the first lines with your MySQL credentials:</li>
<ul><li>(note: for MYSQL_HOME it adds "/bin/mysql" to find the mysql bin)</li></ul>
<code>
MYSQL_HOME=/usr
user=<username>
password=<password>
db_name=umls2015ab
</code>

<li>Then run <code>./populate_mysql_db.sh</code></li>
</ul>
<h3>Generate RDF from MySQL</h3>
<ul>
<li>Clone https://github.com/ncbo/umls2rdf</li>
<li>Rename conf_sample.py as conf.py and configure the access to your database (example when database in local)</li>
<pre>
#Folder to dump the RDF files.
OUTPUT_FOLDER = "output"

#DB Config
DB_HOST = "localhost"
DB_NAME = "umls2015ab"
DB_USER = "root"
DB_PASS = "<password>"

UMLS_VERSION = "2015ab"
UMLS_BASE_URI = "http://purl.bioontology.org/ontology/"
</pre>

<li>Define the ontology you want to generate in umls.conf</li>
<ul>
<li>Example for LNC-RU-RU: "LNC-RU-RU,LNC-RU-RU.ttl,load_on_codes"</li>
<li>If you want to get the name used for you ontology in the database you can get them by going to mySQL, select the umls database and get it from the following query: "select distinct SAB from MRCONSO;"</li>
</ul>
<li>Run ./umls2rdf.py</li>
<li>Get the ttl files in the output directory</li>
</ul>

Advanced Administration

2016-01-28T17:18:38Z

Vemonet: /* Annotator cache and dictionary update */

[[Category:NCBO Virtual Appliance]]

On this page you will find some guidelines on how Advanced administration of the BioPortal Virtual Appliance. To proceed you will need a ssh connexion to your Appliance.<br/>
To simplify the examples we will consider we are using a Virtual Appliance deployed at '''http://my-bioportal-vm/'''

= Solr search index =

== Access to solr ==

It is hosted by the tomcat on the port 8082
http://my-bioportal-vm:8082/solr

There are 2 cores:

* core1 is used for search
* core2 is used when doing a full reindexing (it allows the search to work fine when reindexing the whole bioportal)

To swap the 2 cores:

* Go to core admin tab
* Select core1
* Hit the "Swap" button

== Reindex all ontologies in core2 ==

Re-indexing in the core2 allows the BioPortal Appliance to not stop working during the reindexing (you just have to swap the cores when the reindexing is over)

bin/ncbo_ontology_index -a -l logs/reindexing_all.log -c http://my-bioportal-vm:8082/solr/core2

== Reindex only the specified ontologies in core2 ==

bin/ncbo_ontology_index -o STY,SNOMED -l logs/reindexing_STY_SNOMED.log -c http://my-bioportal-vm:8082/solr/core2

= Updating the Annotator cache and dictionary =

See the [http://www.bioontology.org/wiki/index.php/Annotator_Documentation#Update_cache_and_dictionary Annotator Documentation]

= Common troubles =

== ontologies_report.json is missing ==

When running some job you can face the following error:
ontologies_report.json : No such file or directory

To avoid it you have to create a ''ontologies_report.json'' file at the following location:
/srv/reports/ontologies_report.json

It has to be owned by ncbobp:ncbobp and to have permission 666

Main Page

2016-01-28T17:17:10Z

Vemonet: /* Software */

Welcome to the '''NCBO Public WIKI'''. This WIKI provides information about collaborative ontology development efforts, dissemination events conducted by the NCBO and publically shared materials from the NCBO.
Please visit our [http://www.bioontology.org Main Site] for more information and materials.

=== Software ===
*[http://bioportal.bioontology.org NCBO BioPortal] - Home page for BioPortal Web site
* [http://sparql.bioontology.org/ NCBO BioPortal SPARQL Endpoint] - Access BioPortal via SPARQL queries (obsolete)
* [http://www.bioontology.org/wiki/index.php/Category:NCBO_Virtual_Appliance BioPortal Virtual Appliance] - Contains BioPortal, Annotator, and Resource Index.
* [http://www.bioontology.org/wiki/index.php/BioPortal_Release_Notes BioPortal Software Release Announcements]
*[[NCBO-OOR Development]] - Documentation on how the Open Ontology Repository community can customize and advance BioPortal capabilities.

=== User Support Documentation ===
*[[BioPortal Help]] - General BioPortal help documentation.
* [http://www.bioontology.org/wiki/index.php/Using_NCBO_Technology_In_Your_Project Using NCBO Technology in Your Project] - Documentation on how to use NCBO Technology in your project

=== Technical Documentation ===
* [http://data.bioontology.org/documentation NCBO REST Web services] - Listing of all NCBO web services with usage documentation
* [http://www.bioontology.org/wiki/index.php/SPARQL_BioPortal BioPortal SPARQL Endpoint] - BioPortal SPARQL documentation
* [http://www.bioontology.org/wiki/index.php/Category:NCBO_Virtual_Appliance BioPortal Virtual Appliance] - Contains BioPortal, Annotator, and Resource Index.

=== Ontology Development Projects ===
Below are links for ontology development projects that NCBO is involved in.
*[[BRO:Main_Page]] - Biomedical Resource Ontology
*[[TMO:]] - Translational Medicine Ontology
*[[NPO:Main_Page]] - NanoParticle Ontology
*[[PATO:Main_Page]] - An ontology of phenotypic qualities
*[[RO:Main_Page]] - OBO Relations ontology
*[[CARO:Main_Page]] - Reference anatomical ontology
*[[CL:Main_Page]] - OBO Cell ontology
*[[SO:Main_Page]] - Sequence Ontology
*[[Disease:Main_Page]] - Disease Ontology, and other ontologies pertaining to diseases and disorders
*[[OCI:Main_Page]] - Ontology of Clinical Investigation

Advanced Administration

2016-01-28T17:15:54Z

Vemonet: Created page with "Category:NCBO Virtual Appliance On this page you will find some guidelines on how Advanced administration of the BioPortal Virtual Appliance. To proceed you will need a s..."

[[Category:NCBO Virtual Appliance]]

On this page you will find some guidelines on how Advanced administration of the BioPortal Virtual Appliance. To proceed you will need a ssh connexion to your Appliance.<br/>
To simplify the examples we will consider we are using a Virtual Appliance deployed at '''http://my-bioportal-vm/'''

= Solr search index =

== Access to solr ==

It is hosted by the tomcat on the port 8082
http://my-bioportal-vm:8082/solr

There are 2 cores:

* core1 is used for search
* core2 is used when doing a full reindexing (it allows the search to work fine when reindexing the whole bioportal)

To swap the 2 cores:

* Go to core admin tab
* Select core1
* Hit the "Swap" button

== Reindex all ontologies in core2 ==

Re-indexing in the core2 allows the BioPortal Appliance to not stop working during the reindexing (you just have to swap the cores when the reindexing is over)

bin/ncbo_ontology_index -a -l logs/reindexing_all.log -c http://my-bioportal-vm:8082/solr/core2

== Reindex only the specified ontologies in core2 ==

bin/ncbo_ontology_index -o STY,SNOMED -l logs/reindexing_STY_SNOMED.log -c http://my-bioportal-vm:8082/solr/core2

= Annotator cache and dictionary update =

See the [http://www.bioontology.org/wiki/index.php/Annotator_Documentation#Update_cache_and_dictionary Annotator Documentation]

= Common troubles =

== ontologies_report.json is missing ==

When running some job you can face the following error:
ontologies_report.json : No such file or directory

To avoid it you have to create a ''ontologies_report.json'' file at the following location:
/srv/reports/ontologies_report.json

It has to be owned by ncbobp:ncbobp and to have permission 666

Category:NCBO Virtual Appliance

2016-01-28T16:59:28Z

Vemonet: /* Basic System Administration */

== NCBO VIRTUAL APPLIANCE v2.4 IS NOW AVAILABLE ==
The NCBO Virtual Appliance has been updated to use NCBO's new v4.0 software infrastructure, including the use of an RDF triplestore as the primary data storage mechanism. Please read below as much of the Appliance workflow has changed.

For documentation of v1.0 Virtual Appliances, please see our [http://www.bioontology.org/wiki/index.php?title=Category:NCBO_Virtual_Appliance&oldid=12610 archive].

== Virtual Appliance Basics ==

The NCBO Virtual Appliance image contains a pre-installed, pre-configured version of commonly-used open source NCBO software running on a Linux operating system.

It is available as a VMWare Virtual Appliance OVF, as well as an Amazon Web Service AMI, and can be obtained by contacting us following the instructions under '''Getting Started''' below.

The following software is included on the image as of v2.4:
* Ontologies API (REST service)
* Annotator
* Recommender
* BioPortal Web User Interface (including ontology visualization, widgets, biomixer, and Annotator UI)

Please see below for how-to documentation for managing the software and running data population for Annotator.

== Getting Started ==
=== VMWare Virtual Appliance ===
* To obtain the VMWare Virtual Appliance, contact [mailto:support@bioontology.org NCBO Support] to initiate your request. You'll then be asked privately for your BioPortal account username. organizational goals, and reason for preferring the local installation. (If you don't have a BioPortal account, you can create one at: http://bit.ly/bioportal-account.) The overall transaction can take a few working days, depending on resource availability.
* The download is provided as a zip archive containing several files. One of these is an Open Virtualization Format (OVF) file that may need to be [http://www.bioontology.org/wiki/index.php/Virtual_Appliance_FAQ#How_can_I_use_the_OVF_image_with_my_virtualization_software.3F_.28VMWare.2C_VirtualBox.2C_KVM.2C_Xen.2C_etc.29 converted to work in your virtualization environment].
* You can supply the hostname (machine name) for the virtual machine during the deployment process. Documentation will refer to this hostname as 'example'.
* '''Change default passwords'''
** Operating System
*** Username: <code>root</code>
*** Password: password is prompted on the first boot
** BioPortal Admin User
*** Username: <code>admin</code>
*** Password: <code>changeme</code>

=== Amazon AWS AMI ===
Amazon AMI is available on the [https://aws.amazon.com/marketplace/seller-profile/ref=dtl_pcp_sold_by?ie=UTF8&id=76948a46-8f8a-4a68-9a5c-3e3ff6b82d10 AWS Market Place]. Please contact [mailto:support@bioontology.org NCBO Support] for more information.
Once the instance is running, enter the public DNS provided by Amazon into your browser to access BioPortal web interface. The default application administrator is 'admin' and the initial password is the Instance ID. You can also SSH to the machine using the username 'ec2-user' and your Amazon private key.

=== General Instruction ===
* Add an ontology using the BioPortal Admin User here: http://{YourDomainName}/ontologies/new
** The ncbo_cron project is configured to automatically process new ontologies every 5 minutes (see [http://www.bioontology.org/wiki/index.php/Virtual_Appliance_FAQ#When_are_new_ontologies_parsed.3F documentation for enabling the scheduler]). This processing includes:
*** Parsing any new, unparsed ontologies
*** Calculating a set of metrics for these ontologies
*** Indexing these ontologies for use with search
*** Processing the ontology for use with the annotator
* REST services are available at the following location:
** http://example:8080
** http://example:8080/documentation

== System Requirements ==
The following requirements are for the resources that you devote to your Appliance instance, not for the machine running your host environment. For example, if you are using a system with 4GB of RAM, then you will need to devote all of that RAM to your guest Appliance.

'''Note: these requirements are for basic usage'''. System requirements will vary greatly depending on the size of the ontologies you work with, the number of ontologies in the system, and the number of concurrent requests that the system needs to respond to. It can also vary depending on how the ontologies are used. For example, the search index can be RAM-intensive but parsing ontologies can be CPU-intensive. You will need to experiment with your Appliance resource settings to find what works for your scenario.

* Minimum
** 2 CPU (2 GHz)
** 4GB RAM
** Hard disk space: 20GB

* Recommended for heavier usage
** 4 CPU (3 GHz)
** 8GB RAM (or more depending on the size/number of ontologies)
** Hard disk space: 20GB (or more depending on number/size of ontologies)

== Image Format and Operating System Details ==
The NCBO Appliance image was created using the [http://www.vmware.com/appliances/getting-started/learn/ovf.html Open Virtualization Format], which should allow the machine to be used in a variety of environments.

The operating system is CentOS 6.6 64-bit running:
*Tomcat 6.0.26
*Solr 4.10.4
*Java 7
*MySQL 5.1.x
*Rails 3.2.x
*Ruby 2.1.5
*memcached
*redis 2.8.18
*nginx 1.6.3
*4store
*Passenger/Apache

The following applications use these services/runtime environments:
* Sinatra, Ruby, 4store, redis
** Ontologies API
** Annotator
** Recommender
* Rails, Ruby, memcached, mysql
** BioPortal Web User Interface (including ontology visualization, Flex widgets, biomixer, Annotator)
* Tomcat, Solr
** Search index

== Basic System Administration ==
* Most of our administration scripts and build environment assume that you will be running as the root user.
* We assume familiarity with basic CentOS Linux administration
* 2 text editors are originally installed: nano and vim
* All services required for BioPortal Stack are automatically started on boot. For troubleshooting purposes you can manually stop/start services:
** <code>bprestart</code>: manually restart services required for BioPortal Stack.
** <code>bpstart</code>: manually start services.
** <code>bpstop</code>: manually stop services
* Start individual services:
** <code>/sbin/service httpd start</code>
** <code>/sbin/service memcached start</code>
** <code>/sbin/service tomcat6 start</code>
** <code>/sbin/service mgrep start</code>
** <code>/sbin/service 4s-httpd-ontologies_api start</code>
** <code>/sbin/service unicorn start</code>
* Stop individual services:
** <code>/sbin/service httpd stop</code>
** <code>/sbin/service memcached stop</code>
** <code>/sbin/service tomcat6 stop</code>
** <code>/sbin/service mgrep stop</code>
** <code>/sbin/service 4s-httpd-ontologies_api stop</code>
** <code>/sbin/service unicorn stop</code>

== License ==
All NCBO software is released with the 2-clause BSD license. Source code is included on the Virtual Appliance.

Annotator Documentation

2016-01-28T16:34:58Z

Vemonet: Created page with "Category:NCBO Virtual Appliance = General workflow informations = * Cache and dictionary file updated at each ontology submission parsing * The dictionary.txt file is ge..."

[[Category:NCBO Virtual Appliance]]

= General workflow informations =

* Cache and dictionary file updated at each ontology submission parsing
* The dictionary.txt file is generated from Redis data (annotator cache at localhost:6379)
* Annotator ruby lib (https://github.com/ncbo/ncbo_annotator)
** The Annotator calls mgrep on port 5555 which uses ''/srv/mgrep/dictionary/dictionary.txt'' to annotate the text.
** Mgrep uses /srv/mgrep/dictionary.txt to retrieve the mgrep ID (1st column of dictionary.txt)
** Redis use the mgrep ID to retrieve the class URI and other data (PREF or SYN, ontology URI)

= Update cache and dictionary =

== Redis prefix ==
When caching data about a concept Redis use a prefix (it allows BioPortal to update the cache on another prefix without interrupting the Annotator service)

To get the prefix used by the annotator in Redis:

<code>
redis-cli -h localhost -p 6379 <br/>
6379> get current_instance
</code>

To change it:
<code>
6379> set current_instance "c1:"
</code>

== Totally clean the cache (optional) ==
Not always necessary, but can be useful if you have old annotations (for concept from deleted submissions) that won't go away.<br/>
Be careful it also delete the "current_instance", so you will have to set it.

<code>
redis-cli -h localhost -p 6379 <br/>
6379> flushall
6879> set current_instance "c1:"
</code>

== Setting up of config files ==

Running the generate cache CRON job will create new entries in Redis with the prefix defined in the config file with "annotator_redis_alt_prefix"

Config files location in the VM:<br/>
* /srv/ncbo/ncbo_cron/config/appliance.rb<br/>
* /srv/ncbo/ontologies_api/current/config/environments/appliance.rb

To generate the cache on a new "alt" prefix and continue to annotate with the one defined in Redis current_instance
<code>
Annotator.config do |config|
config.annotator_redis_prefix = ""
config.annotator_redis_alt_prefix = "c1:"
</code>

== Generate the Redis cache and dictionary for all ontologies ==

The following CRON job will generate the annotator cache in Redis, then generate the dictionary used by mgrep (in ''/srv/mgrep/dictionary/dictionary.txt'')<br/>
'''Be careful!''' It is using the _annotator_redis_alt_prefix_ to generate the cache, but the Redis _current_instance_ to generate the dictionary.<br/>
So if you are doing a total refresh of the cache and dictionary (after a flushall for example) you will have to define the same prefix in Redis current_instance and in config.annotator_redis_alt_prefix

<br/>
Then run the following command from the ncbo_cron project (''/srv/ncbo/ncbo_cron''):
<code>
bin/ncbo_ontology_annotate_generate_cache -a -r -d -l logs/all_cache.log
</code>

It will generates entry in Redis like this : "c1:term:-1762481778059933889"
Check for it in Redis:

<code>
redis-cli
6379> keys c1:term:*
</code>

Restart mgrep : ''service mgrep restart'' or do a ''bprestart''

The annotator should be updated!

== More details on the generate cache options ==

The CRON job ''ncbo_ontology_annotate_generate_cache'' can be run with different options:

* -r or --remove-cache
Reset cache to scratch
DO NOT use the -r attribute when generating cache for individual ontologies, as it will delete the entire cache used by the Annotator and replace it with a new one that will only contain the given ontologies.

* -a or --all-ontologies
To generate the cache for all ontologies : run the process on the annotator_redis_alt_prefix

* -o STY,SNMI
Generate the cache only for the specified ontologies : run the process on the annotator_redis_prefix

* -d or --generate-dictionary
Also re-generate the dictionary for the terms prefixed by current_instance. Does the same job as the ncbo_ontology_annotate_generate_dictionary CRON job.

== Generate the dictionary only ==

To generate the dictionary from the Redis cache (terms prefixed with Redis ''current_instance''):

/srv/ncbo/ncbo_cron/bin/ncbo_ontology_annotate_generate_dictionary

= Troubleshooting =

== Check which dictionary mgrep is using ==

Restart mgrep and run it:
ps -ef | grep mgrep

It should give something like:
mgrep 27924 1 0 Jan27 ? 00:00:00 /usr/local/mgrep-4.0.2/mgrep --port=55555 -f /srv/mgrep/mgrep-55555/dict -w /usr/local/mgrep-4.0.2/word_divider.txt -c /usr/local/mgrep-4.0.2/CaseFolding.txt

== Run queries on mgrep ==

Run queries against the mgrep directly, using a telnet connection to the mgrep server

<code>
telnet my-bioportal-vm 55555
ANY entity
</code>

It should return something like this (if entity is cached for the annotator, which should be since it is in STY)

1882193402596741431 2 7 entity