Difference between revisions of "Virtual Appliance FAQ"

From NCBO Wiki
Jump to: navigation, search
(improving formatting emphasis)
 
(67 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 +
[[Category:NCBO Virtual Appliance]]
 +
[[Category:Migrated to GitHub]]
 +
 +
This content has been moved! Please find the new content for the 3.0 version of the Virtual Appliance at our new [https://ontoportal.github.io/administration/ OntoPortal Virtual Appliance Administration pages].
 +
 +
In particular '''this content''' is mostly at the '''[https://https://ontoportal.github.io/administration/general/faq/ Frequently Asked Questions (FAQ) page]'''.
 +
 +
 +
= '''ARCHIVAL''' ONTOPORTAL VIRTUAL APPLIANCE v2.5 FAQ '''ARCHIVAL''' =
 +
 +
This page is the archival FAQ for the the Ontoportal Virtual Appliance v2.5.
 +
 +
The FAQ for v1.0 is available in the [http://www.bioontology.org/wiki/index.php?title=Virtual_Appliance_FAQ&oldid=12621 archive]
 +
 +
= General =
 +
== How do I access Virtual Appliance? ==
 +
 +
Virtual Appliance Web UI can be accessed by going to http://{your_appliance_ip_or_domain_name} on your web browser.<br>
 +
API can be accessed at http://{your_appliance_ip_or_domain_name}:8080
 +
 +
 +
You can get IP address of the Appliance with the following methods:
 +
* Local IP Address from terminal:
 +
<code>ip address show eth0 | awk '/inet / {print $2}' | cut -d/ -f1'</code>
 +
* External IP address from the terminal:
 +
<code>curl http://ipecho.net/plain; echo</code>
 +
* Vmware Appliance:
 +
Check 'IP Addresses' on the summary page of the appliance.  Please note that Virtual Appliance does not come with vmware tools installed so IP Address would not be displayed until vmware tools are installed. 
 +
* Amazon AWS:
 +
use Public IP Addresses or Public DNS listed in your EC2 management console.
 +
 +
== How can I enable emails for lost passwords, notes, and ontology processing? ==
 +
 +
Emails are sent via the ontologies_api project on the Appliance. You need to provide a valid mail server (smtp) configuration. The configuration should be provided in the /srv/ncbo/ontologies_api/current/config/environments/production.rb file.
 +
 +
Here are the available settings:
 +
<code>
 +
  config.enable_notifications  = true # Set to 'true' to send emails
 +
  config.email_sender          = "admin@example.org" # Default sender for emails
 +
  config.email_disable_override = true # If this is set to 'false', all emails will be sent to the email configured in the 'email_override' setting
 +
  config.email_override        = "admin@example.org"
 +
  config.smtp_host              = "smtp.example.org"
 +
  config.smtp_port              = 25
 +
  config.smtp_auth_type        = :none # :none, :plain, :login, :cram_md5
 +
  config.smtp_user              = "username" # only used if auth_type is not :none
 +
  config.smtp_password          = "password" # only used if auth_type is not :none
 +
  config.smtp_domain            = "example.org"
 +
</code>
 +
 +
Once you have changed your settings, you will need to restart the server by running the command <code>/sbin/service unicorn restart</code>
 +
 +
== Can I use the Annotator and Recommender on their own? ==
 +
 +
Because services like the Annotator and Recommender depend on the Ontologies API in BioPortal, you can not usefully install those services in a stand-alone way without also installing the core BioPortal software and ontologies.
 +
 
= Ontology Management =
 
= Ontology Management =
  
== How do I identify version numbers for ontologies stored in the system? ==
+
== How do I add or change categories or groups? ==
NCBO Ontology Web Services can be used with either the ontology identifier (sometimes referred to as the virtual identifier) or the ontology version identifier (sometimes referred to as the version id). The ontology identifier is used to identify and ontology and all of its versions. The ontology version identifier represents a single ontology version. The following screenshot shows how to identify these version numbers using the BioPortal Web UI.
+
There is currently no UI administrator interface. Categories and groups can be added using a console after logging into the Appliance as the ontoportal user.
<br>[[Image:Bioportal_internal_ontology_ids.png|200px|x]]
+
<pre>
 +
# from the bash shell:
 +
cd /srv/ncbo/ncbo_cron
 +
bin/ncbo_cron --console
 +
# once in the ruby console:
 +
category = LinkedData::Models::Category.new
 +
category.name = "My Category"
 +
category.acronym = "MY_CAT"
 +
category.save
 +
group = LinkedData::Models::Group.new
 +
group.name = "My Group"
 +
group.acronym = "MY_GRP"
 +
group.save
 +
</pre>
  
== How do I add or change categories? ==
+
== How do I add or change slices? ==
<ol>
+
There is currently no UI administrator interface. Slices can be added using a console after logging into the Appliance as the ontoportal user.
  <li>Download the Protégé client '''(Version Protégé 3.4.4)'''</li>
+
<pre>
  <ol>
+
# from the bash shell:
    <li>[http://protege.stanford.edu/download/download.html http://protege.stanford.edu/download/download.html]<br>
+
cd /srv/ncbo/ncbo_cron
    </li>
+
bin/ncbo_cron --console
  </ol>
+
# once in the ruby console:
  <li>Download a copy of the [http://www.bioontology.org/wiki/images/e/ef/Bioportal_metadata.pprj.zip BioPortal Metadata Protégé project file] and modify as follows</li>
+
ont1 = LinkedData::Models::Ontology.find("ONT1").first
  <ol>
+
ont2 = LinkedData::Models::Ontology.find("ONT2").first
    <li>Line 639: make sure this matches your BioPortal Ontology Services database (ip address or domain of the virtual appliance)</li>
+
slice = LinkedData::Models::Slice.new
    <li>Line 649: this should be your MySQL username (bp_protege)</li>
+
slice.name = "My Slice",
    <li>Line 654: this should be your MySQL password (bioportalprotege)</li>
+
slice.description = "This is my custom slice",
    <li>'''Note:''' These are the default username/password which should be changed for security purposes. If you have already changed them, please use the new username/password.</li>
+
slice.acronym = "my_slice",
  </ol>
+
slice.ontologies = [ont1, ont2]
  <li>Open the project file using Protégé</li>
+
slice.save
  <li>Go to the "Individuals" tab</li>
+
</pre>
  <li>Look for OMV:OntologyDomain and click it</li>
+
 
  <li>A list of "Asserted Instances" should show up, these are the defaults</li>
+
== How do I delete an ontology? ==
  <li>You can create or delete instances by clicking on the buttons</li>
+
Deleting can be done using a console after logging into the Appliance as the ontoportal user.
  <li>Instances should have the following properties:</li>
+
<pre>
  <ol>
+
# from the bash shell:
    <li>id = unique integer value</li>
+
cd /srv/ncbo/ncbo_cron
    <li>omv:name = display name for the category</li>
+
bin/ncbo_cron --console
    <li>omv:isSubDomainOf = you can create hierarchies by making
+
# once in the ruby console:
      categories subDomains of one another</li>
+
ontology = LinkedData::Models::Ontology.find("MY_ONTOLOGY_ACRONYM").first
  </ol>
+
ontology.delete
</ol>
+
</pre>
  
 
== How can I migrate ontologies from BioPortal or previous NCBO Virtual Appliance versions into a new Appliance? ==
 
== How can I migrate ontologies from BioPortal or previous NCBO Virtual Appliance versions into a new Appliance? ==
NCBO developers have created a script that can be used to automatically import ontologies from BioPortal or previous versions of the Virtual Appliance. The script was written in Ruby and is included on the Appliance. To use the script, do the following (requires knowledge of Linux command-line tools):
 
  
* Edit the script, located here: <code>/var/rails/BioPortal/current/util/import_ontologies/import_ontologies.rb</code>
+
Programmatic migrations are not officially supported ("Ontologies can be manually downloaded and added using the Web UI") but we included a script for importing ontologies.
** Make sure you have valid API keys for the FROM and TO systems.
+
** Assign the $FROM and $TO variables and ports as appropriate. The default configuration points to NCBO's BioPortal but can be changed to the location of your Virtual Appliance instance.
+
** The $ONTOLOGY_OWNER is the user id of the user who should own the ontologies on the TO system.
+
** $INCLUDE_ONTOLOGIES can be uncommented and used to import specific ontologies only. Otherwise all ontologies from the FROM system will be imported.
+
** Save your edits and continue below.
+
* Run the script using this command: <code>sh /var/rails/BioPortal/current/util/import_ontologies/import_ontologies.rb</code>
+
* The script will take some time to complete, updating the console as it runs. If automatic parsing is enabled, the ontologies should get parsed when the process next runs.
+
  
'''Please Note:''' The import script will attempt to use category and group IDs assigned to ontologies on the FROM system if they exist on the TO system as well. Ontology artifacts, such as notes, views, and mappings, are not imported. Only the most recent version of ontologies are imported; there is no method to import multiple versions of the same ontology using a common virtual id.
+
/srv/ncbo/virtual_appliance/deployment/utils/bioportal_ontologies_import.rb,  you will need to modify SOURCE_API, SOURCE_APIKEY and ONTOLOGIES_TO_IMPORT variables.
  
= Ontology Parsing =
+
== How can I restrict ontology downloads via the UI? ==
== When are new ontologies parsed? ==
+
The BioPortal Ontology Services application uses a scheduler to run a process that collects newly submitted ontologies and parses them, adds them to the search index, and calculates metrics. You can also parse ontologies manually, but you will need to index them for search and calculate metrics manually to have that information available.
+
  
The default schedule can be disabled by doing the following:
+
This line of code should be modified to include the acronyms of the ontologies for which you want to restrict downloads via the UI:
* Open /ncbo/sources/bioportal/tags/1030/build.properties
+
* Look for the section that starts with "# Ontology Parse Scheduler properties"
+
* Set <code>ontology.parse.scheduler.enabled</code> to 'false'
+
* From the /ncbo/sources/bioportal/tags/1030/ directory, run 'ant clean deploywar'
+
  
== How do I manually parse an ontology? ==
+
https://github.com/ncbo/bioportal_web_ui/blob/master/app/views/ontologies/_submissions.html.haml#L32
To manually parse an ontology, visit the admin interface at http://example:8080/bioportal_admin and select "Parse Ontologies" from the list at the right. You must enter the version id, or a comma-separated list of ids, and then click "Run". The process will return with errors if it encounters any.
+
  
== How do I manually add an ontology to the search index? ==
+
This will prevent the “Downloads” column in the Submissions table from appearing on ontology summary pages.
To manually add an ontology to the search index, go to the admin interface and select "Index Ontologies". You can provide a single version id for an ontology or a comma-separated list of ids. You can also choose to index all ontologies currently stored in the Appliance (this may take some time if you have many terms in the system).
+
  
== How do I know if an ontology has parsed? ==
+
If you want to restrict download via the REST endpoint, you need to modify your environment configuration files (e.g., production.rb, staging.rb, etc.) in your ontologies_api project. The production.rb file should contain a “config.restrict_download” property -- there’s an example of what it should look like in the sample configuration file in GitHub:
Because the BioPortal Web UI uses aggressive caching, it may not immediately reflect the status of an ontology once it's been submitted. By default the ontology status on the Web UI is updated once every four or 12 hours (depending where you are viewing the information). You can change these default options by doing the following:
+
* Open /var/rails/BioPortal/current/app/models/data_access.rb
+
* Change the following in the getOntologyList method:
+
** <code>return self.cache_pull("ont_list", "getOntologyList", nil, MEDIUM_CACHE_EXPIRE_TIME)</code>
+
** <code>return self.cache_pull("ont_list", "getOntologyList", nil, 60*15)</code>
+
* Change the following in the getOntology method:
+
** <code>return self.cache_pull("#{ontology_id}::_details", "getOntology", { :ontology_id => ontology_id })</code>
+
** <code>return self.cache_pull("#{ontology_id}::_details", "getOntology", { :ontology_id => ontology_id }, 60*15)</code>
+
* This will change the Web UI so that it refreshed information about ontologies every 15 minutes
+
* You will need to run /sbin/service httpd restart to have the change take affect
+
* '''NOTE:''' You will need to make this change again if you update the Web UI code. In addition, if you have a large list of ontologies this could slow down the Web UI for users as it will be required to retrieve more information more often.
+
  
In addition, you can look at the REST service directly, which will always give you the most updated information. To do this, visit the following URL:
+
https://github.com/ncbo/ontologies_api/blob/master/config/environments/config.rb.sample#L38
* http://example:8080/bioportal/ontologies/{ontology_version_id}
+
* You can look for the statusId element to get the status, the codes are as follows:
+
** 1 - Waiting
+
** 2 - Parsing
+
** 3 - Ready
+
** 4 - Error
+
** 5 - Not Applicable (for metadata-only ontologies)
+
  
== Is there a log file for parsing? ==
+
= Ontology Parsing =
Parsing progress is logged in the BioPortal Ontology Services log files.
+
== When are new ontologies parsed? ==
 +
The ncbo_cron project uses a scheduler to run a process that collects new ontology submissions and parses them, adds them to the search index, calculates metrics, and processes them for use with the annotator. You can also parse ontologies manually.
  
There is a separate log file for OBO ontologies that can be monitored:  
+
== How do I manually parse an ontology? ==
<code>tail -f /var/log/tomcat6/lexgrid/LexBIG_load_log.txt</code>
+
To manually parse an ontology, you will need to interact with the code using the console after switching to ontoportal user:
 +
<pre>
 +
# from the bash shell:
 +
cd /srv/ncbo/ncbo_cron
 +
bin/ncbo_cron --console
 +
# once in the ruby console:
 +
ontology = LinkedData::Models::Ontology.find("MY_ACRONYM").first
 +
submission = ontology.latest_submission(status: :any)
 +
logger = Logger.new(STDOUT)
 +
submission.process_submission(logger)
 +
# make available in annotator
 +
annotator = Annotator::Models::NcboAnnotator.new
 +
annotator.create_term_cache_for_submission(logger, submission)
 +
annotator.generate_dictionary_file()
 +
</pre>
  
OWL and Protege-based ontologies are logged into the general log file:
+
== How can I process a UMLS ontology? ==
<code>tail -f /var/log/tomcat6/bioportal.log</code>
+
  
== How can I manually remove the hierarchy information from an ontology like NCI Metathesaurus? ==
+
UMLS ontologies can be processed to work with our system by converting them to RDF. There is no automated way to do this and you must have your own UMLS MySQL installation and an OSX/Linux/Unix machine with 8GB+ of RAM in order for the conversion process to work. The scripts to convert UMLS to RDF are available on [https://github.com/ncbo/umls2rdf Github].
* Find the database prefix of the ontology by looking up the ontology in the LexEVS registry.xml file.
+
* Connect to the database that stores the LexEVS tables.
+
* Find the set of tables that stores the ontology using the table prefix.
+
* The table called codingSchemeSupportedAttrib needs to be updated to disable the hierarchy. To do so, find the row that has the column supportedAttributeTab that has a value called Hierarchy. Update that column to value HierarchyDisabled so that the loaded hierarchy is ignored.
+
* Restart Tomcat so that the new values are read in and the cache gets cleared.
+
  
= Systems Administration =
+
Once you have converted UMLS to RDF, you will get Turtle (.ttl) files that can be uploaded using the BioPortal Web UI. Please select UMLS as the format for these ontologies.
  
== How do I increase the Java heap size that Tomcat uses? ==
+
== How do I know if an ontology has parsed? ==
* By default we use a 512MB initial heap with a 2GB maximum. You can change these values by editing this file:
+
The BioPortal Web UI will cache information about ontologies for 60 seconds. After parsing is complete, just refresh the ontology summary page to see the status for the most recent submission listed under the "Submissions" table.
** <code>/usr/local/tomcat6/conf/tomcat6.conf</code>
+
* Change the following line to match your requirements. -Xms is initial size, -Xmx is maximum size.
+
** <code>JAVA_OPTS="${DEBUG} -Xms512m -Xmx2G -Djava.awt.headless=true -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true"</code>
+
* Restart Tomcat by running the following command:
+
** <code>/sbin/service tomcat6 restart</code>
+
  
[[Category:NCBO Virtual Appliance]]
+
In addition, you can look at the REST service directly, which will always give you the most updated information. To do this, visit the following URL:
 +
* http://{your_appliance_ip_or_domain_name}:8080/ontologies/{ontology_acronym}/latest_submission?include=all
 +
* You can look for the submissionStatus attribute to get the status
 +
 
 +
== Is there a log file for parsing? ==
 +
Parsing progress is logged in the ontology submission repository folder:
 +
/srv/ncbo/repository/{ontology acronym}/{submission id}
  
 
= Web User Interface =
 
= Web User Interface =
 
== How can I clear the memcached-based UI cache? ==
 
== How can I clear the memcached-based UI cache? ==
* If you are logged in as the admin user, simply visit http://example/admin and click the "Flush Memcache" button. There should be a response indicating success or failure.
+
* If you are logged in as the admin user, simply visit http://{your_appliance_ip_or_domain_name}/admin and click the "Flush Memcache" button. There should be a response indicating success or failure.
  
 
== How can I use widgets with my Virtual Appliance? ==
 
== How can I use widgets with my Virtual Appliance? ==
 
=== Working with Javascript-based widgets (Form Autocomplete, Quick Jump/Jump To) ===
 
 
In addition to the [http://www.bioontology.org/wiki/index.php/NCBO_Widgets#How_to_use_NCBO_Widgets existing instructions], you must define an additional Javascript variable in order to have the widgets communicate with your instance of the Virtual Appliance.
 
In addition to the [http://www.bioontology.org/wiki/index.php/NCBO_Widgets#How_to_use_NCBO_Widgets existing instructions], you must define an additional Javascript variable in order to have the widgets communicate with your instance of the Virtual Appliance.
 
<code>
 
<code>
Line 126: Line 168:
 
Replace the '{your_appliance_ip_or_domain_name}' text with the IP address or domain name that's assigned to your Virtual Appliance.
 
Replace the '{your_appliance_ip_or_domain_name}' text with the IP address or domain name that's assigned to your Virtual Appliance.
  
=== Working with Flex/Flash-based widgets (FlexViz, Ontology Tree Widget) ===
+
= Virtualization Environments =
  
When working with Flash widgets, you'll need to modify the [http://www.bioontology.org/wiki/index.php/NCBO_Widgets#How_to_use_NCBO_Widgets existing instructions] to change and add your ip address or domain name. See below and look for '{your_appliance_ip_or_domain_name}' and '{ontology_id}'. The '{ontology_id}' is the numerical id for the ontology you want to use with the widget, sometimes called a virtual id. You can find this via the REST service by looking for the <ontologyId> element, or looking in the Web UI on the Browse page. The links for ontologies contain a number that should be the virtual id.
+
== How can I use the OVF image with my virtualization software? <br/><span style="font-size: smaller;">(VMware, VirtualBox, KVM, Xen, etc)</span> ==
  
This example is for the Tree Browser.
+
=== VMware ===
 +
You can use [http://communities.vmware.com/community/vmtn/vsphere/automationtools/ovf VMware's ovftool] to convert the appliance to work with your VMware product. For example, to convert the appliance for use in VMware Player or Workstation, you would run the command:
  
 
<pre>
 
<pre>
<object classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000"
+
ovftool ncbo-appliance.ovf ncbo-appliance.vmx
  id="OntologyTree" width="300" height="100%"
+
  codebase="http://fpdownload.macromedia.com/get/flashplayer/current/swflash.cab">
+
  <param name="movie" value="http://{your_appliance_ip_or_domain_name}/flex/OntologyTree.swf" />
+
  <param name="quality" value="high" />
+
  <param name="bgcolor" value="#ffffff" />
+
  <param name="allowScriptAccess" value="always" />
+
  <param name="flashVars" value="ontology=<%= @ontology.ontologyId%>&alerterrors=false&canchangeontology=false&virtual=true&server=http://{your_appliance_ip_or_domain_name}:8080/bioportal" />
+
  <embed src="http://{your_appliance_ip_or_domain_name}/flex/OntologyTree.swf" quality="high" bgcolor="#ffffff"
+
    width="300" height="100%" name="OntologyTree" align="middle"
+
    play="true"
+
    loop="false"
+
    allowScriptAccess="always"
+
    type="application/x-shockwave-flash"
+
    flashVars="ontology={ontology_id}&alerterrors=false&canchangeontology=false&virtual=true&server=http://{your_appliance_ip_or_domain_name}:8080/bioportal"
+
    pluginspage="http://www.adobe.com/go/getflashplayer">
+
  </embed>
+
</object>
+
 
</pre>
 
</pre>
  
In addition, there is a bug in existing versions of the Virtual Appliance that is scheduled to be fixed for the .5 version release in March 2012. Until then, you will need to run the following SQL commands to insert records into the 'bioportal' database:
+
=== VirtualBox ===
 
+
<pre>
+
INSERT INTO `ncbo_user` (`id`, `username`, `api_key`, `open_id`, `password`, `email`, `firstname`, `lastname`, `phone`, `date_created`)
+
VALUES
+
(39108, 'annotator_flex', X'32626535656235352D383665622D343164362D386238312D653238653532356163633063', NULL, '8qQ8v8cjUZBCs99voQ2+i4mR1VhyFuj70ZCQUV9caXrPQv4K9FuDIfsfrn+ICoXp\r\n', 'example@domain.org', X'', X'', NULL, '2011-06-28 12:09:32'),
+
(39109, 'search_flex', X'39626437366335382D323230642D346337662D386164302D306135363731313634313939', NULL, '/jj4oWOmI1SXbfczyReBZI9z4DSzRoZP/iX9WJe9lb/7vm1owez/yAh6lIRpNOyj\r\n', 'example@domain.org', X'', X'', NULL, '2011-06-28 12:10:18'),
+
(39110, 'recommender_flex', X'35636564383837322D646636342D343838372D613338652D376338653736333935633830', NULL, 'RBmX8e1X2oUEEcqdp73/X7P7JkrfA7lbK8+svGPEojYmcmxDoHui3gC011Pg1cRF\r\n', 'example@domain.org', X'', X'', NULL, '2011-06-28 12:11:16'),
+
(39111, 'flexviz', X'34363832396130352D386331632D343232622D393530642D313836373633616530663763', NULL, 'uivHblEQJII216fGfG0No4RGELzBtY7Bo14IQulDsQOJY4cLUItYuTtU+TkPs1ln\r\n', 'example@domain.org', X'', X'', NULL, '2011-06-28 12:13:18');
+
</pre>
+
 
+
= How can I use the OVF image with my virtualization software? <br/><span style="font-size: smaller;">(VMware, VirtualBox, KVM, Xen, etc)</span> =
+
 
+
== VMware ==
+
You can use [http://communities.vmware.com/community/vmtn/vsphere/automationtools/ovf VMware's ovftool] to convert the appliance to work with your VMware product. For example, to convert the appliance for use in VMware Player or Workstation, you would run the command: <code>ovftool ncbo-appliance.ovf ncbo-appliance.vmx</code>
+
 
+
== VirtualBox ==
+
 
VirtualBox supports importing OVF images directly. Simply start your VirtualBox software, then select '''File->Import Appliance''' and select the OVF file included in the NCBO Virtual Appliance download.
 
VirtualBox supports importing OVF images directly. Simply start your VirtualBox software, then select '''File->Import Appliance''' and select the OVF file included in the NCBO Virtual Appliance download.
  
== KVM ==
+
=== KVM ===
 +
First, convert the OVF to VMX format as mentioned in the VMware section above.
  
=== HVM ===
+
Next, ensure that the <code>kvm-qemu-img</code> RPM (or <code>qemu-kvm</code> DEB) is installed. Then, convert the [new] VMDKs (from the VMX conversion step) to raw disk images via the following command:
You will need to convert the OVF to VMX format as mentioned in the VMware section above.
+
  
You must also ensure that the <code>kvm-qemu-img</code> RPM (or <code>qemu-kvm</code> DEB) is installed. Convert the [new] VMDKs (from the VMX conversion step) to raw disk images via the following command:
+
<pre>for disk in `ls -1 *.vmdk`; do diskbase=`basename $disk .vmdk`; qemu-img convert -O raw ${diskbase}.vmdk ${diskbase}.img; done</pre>
<code>for disk in `ls -1 *.vmdk`; do diskbase=`basename $disk .vmdk`; qemu-img convert -O raw ${diskbase}.vmdk${diskbase}.img; done</code>
+
  
 
Create <code>/etc/libvirt/qemu/ncbo-appliance.xml</code> with the following contents:
 
Create <code>/etc/libvirt/qemu/ncbo-appliance.xml</code> with the following contents:
<code>
+
 
 +
<pre>
 
<domain type='kvm'>
 
<domain type='kvm'>
 
   <name>ncbo-appliance</name>
 
   <name>ncbo-appliance</name>
Line 205: Line 215:
 
     <disk type='file' device='disk'>
 
     <disk type='file' device='disk'>
 
       <driver name='qemu' type='raw'/>
 
       <driver name='qemu' type='raw'/>
       <source file='/var/lib/libvirt/images/ncbo-appliance-0.5/ncbo-appliance-0.5-disk1.img'/>
+
       <source file='/var/lib/libvirt/images/ncbo-appliance/ncbo-appliance-disk1.img'/>
 
       <target dev='hda' bus='ide'/>
 
       <target dev='hda' bus='ide'/>
 
       <address type='drive' controller='0' bus='0' unit='0'/>
 
       <address type='drive' controller='0' bus='0' unit='0'/>
Line 211: Line 221:
 
     <disk type='file' device='disk'>
 
     <disk type='file' device='disk'>
 
       <driver name='qemu' type='raw'/>
 
       <driver name='qemu' type='raw'/>
       <source file='/var/lib/libvirt/images/ncbo-appliance-0.5/ncbo-appliance-0.5-disk2.img'/>
+
       <source file='/var/lib/libvirt/images/ncbo-appliance/ncbo-appliance-disk2.img'/>
 
       <target dev='hdb' bus='ide'/>
 
       <target dev='hdb' bus='ide'/>
 
       <address type='drive' controller='0' bus='0' unit='1'/>
 
       <address type='drive' controller='0' bus='0' unit='1'/>
Line 233: Line 243:
 
   </devices>
 
   </devices>
 
</domain>
 
</domain>
</code>
+
</pre>
  
Make any necessary edits to the above file, and run <code>virsh start ncbo-appliance</code>.
+
Finally, make any necessary edits to the above file, and run:
  
=== PVM ===
+
<pre>
Xen PVMs (Paravirtual Machines) are not supported.
+
virsh start ncbo-appliance
 +
</pre>
  
== Xen ==
+
=== Xen ===
 +
First, convert the VMDKs to raw disk images as mentioned in the KVM section above.
  
=== HVM ===
+
Create <code>/etc/xen/ncbo-appliance.cfg</code> with the following contents:
You will need to convert the VMDKs to raw disk images as mentioned in the KVM section above.
+
  
Create <code>/etc/libvirt/qemu/ncbo-appliance.xml</code> with the following contents:
+
<pre>
<code>
+
 
name = "ncbo-appliance"
 
name = "ncbo-appliance"
 
memory = 4096
 
memory = 4096
Line 265: Line 275:
 
vncunused = 1
 
vncunused = 1
 
keymap = "en-us"
 
keymap = "en-us"
disk = [ "file:/var/lib/xen/images/ncbo-appliance-0.5/ncbo-appliance-0.5-disk1.img,hda,w", "file:/var/lib/xen/images/ncbo-appliance-0.5/ncbo-appliance-0.5-disk2.img,hdb,w" ]
+
disk = [ "file:/var/lib/xen/images/ncbo-appliance/ncbo-appliance-disk1.img,hda,w", "file:/var/lib/xen/images/ncbo-appliance/ncbo-appliance-disk2.img,hdb,w" ]
 
vif = [ "bridge=xenbr0,script=vif-bridge,vifname=vif41.0" ]
 
vif = [ "bridge=xenbr0,script=vif-bridge,vifname=vif41.0" ]
 
parallel = "none"
 
parallel = "none"
 
serial = "pty"
 
serial = "pty"
</code>
+
</pre>
 +
 
 +
Finally, make any necessary edits to the above file, and run:
 +
 
 +
<pre>
 +
xm create ncbo-appliance
 +
</pre>
 +
 
 +
== How can I use the Appliance on Amazon EC2? ==
 +
 
 +
Appliance is available on [https://aws.amazon.com/marketplace/pp/B00MX5YKVU Amazon Marketplace]
  
Make any necessary edits to the above file, and run <code>xm create ncbo-appliance</code>.
+
=== What is the admin password for the AWS Appliance? ===
  
=== PVM ===
+
The default application administrator is 'admin' and the initial password is the Instance ID (i-xxxxxxxx). Please change this as soon as you install the virtual appliance.
KVM PVMs (Paravirtual Machines) are not supported.
+

Latest revision as of 20:26, 12 June 2020


This content has been moved! Please find the new content for the 3.0 version of the Virtual Appliance at our new OntoPortal Virtual Appliance Administration pages.

In particular this content is mostly at the Frequently Asked Questions (FAQ) page.


Contents

[edit] ARCHIVAL ONTOPORTAL VIRTUAL APPLIANCE v2.5 FAQ ARCHIVAL

This page is the archival FAQ for the the Ontoportal Virtual Appliance v2.5.

The FAQ for v1.0 is available in the archive

[edit] General

[edit] How do I access Virtual Appliance?

Virtual Appliance Web UI can be accessed by going to http://{your_appliance_ip_or_domain_name} on your web browser.
API can be accessed at http://{your_appliance_ip_or_domain_name}:8080


You can get IP address of the Appliance with the following methods:

  • Local IP Address from terminal:

ip address show eth0 | awk '/inet / {print $2}' | cut -d/ -f1'

  • External IP address from the terminal:

curl http://ipecho.net/plain; echo

  • Vmware Appliance:

Check 'IP Addresses' on the summary page of the appliance. Please note that Virtual Appliance does not come with vmware tools installed so IP Address would not be displayed until vmware tools are installed.

  • Amazon AWS:

use Public IP Addresses or Public DNS listed in your EC2 management console.

[edit] How can I enable emails for lost passwords, notes, and ontology processing?

Emails are sent via the ontologies_api project on the Appliance. You need to provide a valid mail server (smtp) configuration. The configuration should be provided in the /srv/ncbo/ontologies_api/current/config/environments/production.rb file.

Here are the available settings:

 config.enable_notifications   = true # Set to 'true' to send emails
 config.email_sender           = "admin@example.org" # Default sender for emails
 config.email_disable_override = true # If this is set to 'false', all emails will be sent to the email configured in the 'email_override' setting
 config.email_override         = "admin@example.org"
 config.smtp_host              = "smtp.example.org"
 config.smtp_port              = 25
 config.smtp_auth_type         = :none # :none, :plain, :login, :cram_md5
 config.smtp_user              = "username" # only used if auth_type is not :none
 config.smtp_password          = "password" # only used if auth_type is not :none
 config.smtp_domain            = "example.org"

Once you have changed your settings, you will need to restart the server by running the command /sbin/service unicorn restart

[edit] Can I use the Annotator and Recommender on their own?

Because services like the Annotator and Recommender depend on the Ontologies API in BioPortal, you can not usefully install those services in a stand-alone way without also installing the core BioPortal software and ontologies.

[edit] Ontology Management

[edit] How do I add or change categories or groups?

There is currently no UI administrator interface. Categories and groups can be added using a console after logging into the Appliance as the ontoportal user.

# from the bash shell:
cd /srv/ncbo/ncbo_cron
bin/ncbo_cron --console
# once in the ruby console:
category = LinkedData::Models::Category.new
category.name = "My Category"
category.acronym = "MY_CAT"
category.save
group = LinkedData::Models::Group.new
group.name = "My Group"
group.acronym = "MY_GRP"
group.save

[edit] How do I add or change slices?

There is currently no UI administrator interface. Slices can be added using a console after logging into the Appliance as the ontoportal user.

# from the bash shell:
cd /srv/ncbo/ncbo_cron
bin/ncbo_cron --console
# once in the ruby console:
ont1 = LinkedData::Models::Ontology.find("ONT1").first
ont2 = LinkedData::Models::Ontology.find("ONT2").first
slice = LinkedData::Models::Slice.new
slice.name = "My Slice",
slice.description = "This is my custom slice",
slice.acronym = "my_slice",
slice.ontologies = [ont1, ont2]
slice.save

[edit] How do I delete an ontology?

Deleting can be done using a console after logging into the Appliance as the ontoportal user.

# from the bash shell:
cd /srv/ncbo/ncbo_cron
bin/ncbo_cron --console
# once in the ruby console:
ontology = LinkedData::Models::Ontology.find("MY_ONTOLOGY_ACRONYM").first
ontology.delete

[edit] How can I migrate ontologies from BioPortal or previous NCBO Virtual Appliance versions into a new Appliance?

Programmatic migrations are not officially supported ("Ontologies can be manually downloaded and added using the Web UI") but we included a script for importing ontologies.

/srv/ncbo/virtual_appliance/deployment/utils/bioportal_ontologies_import.rb, you will need to modify SOURCE_API, SOURCE_APIKEY and ONTOLOGIES_TO_IMPORT variables.

[edit] How can I restrict ontology downloads via the UI?

This line of code should be modified to include the acronyms of the ontologies for which you want to restrict downloads via the UI:

https://github.com/ncbo/bioportal_web_ui/blob/master/app/views/ontologies/_submissions.html.haml#L32

This will prevent the “Downloads” column in the Submissions table from appearing on ontology summary pages.

If you want to restrict download via the REST endpoint, you need to modify your environment configuration files (e.g., production.rb, staging.rb, etc.) in your ontologies_api project. The production.rb file should contain a “config.restrict_download” property -- there’s an example of what it should look like in the sample configuration file in GitHub:

https://github.com/ncbo/ontologies_api/blob/master/config/environments/config.rb.sample#L38

[edit] Ontology Parsing

[edit] When are new ontologies parsed?

The ncbo_cron project uses a scheduler to run a process that collects new ontology submissions and parses them, adds them to the search index, calculates metrics, and processes them for use with the annotator. You can also parse ontologies manually.

[edit] How do I manually parse an ontology?

To manually parse an ontology, you will need to interact with the code using the console after switching to ontoportal user:

# from the bash shell:
cd /srv/ncbo/ncbo_cron
bin/ncbo_cron --console
# once in the ruby console:
ontology = LinkedData::Models::Ontology.find("MY_ACRONYM").first
submission = ontology.latest_submission(status: :any)
logger = Logger.new(STDOUT)
submission.process_submission(logger)
# make available in annotator
annotator = Annotator::Models::NcboAnnotator.new
annotator.create_term_cache_for_submission(logger, submission)
annotator.generate_dictionary_file()

[edit] How can I process a UMLS ontology?

UMLS ontologies can be processed to work with our system by converting them to RDF. There is no automated way to do this and you must have your own UMLS MySQL installation and an OSX/Linux/Unix machine with 8GB+ of RAM in order for the conversion process to work. The scripts to convert UMLS to RDF are available on Github.

Once you have converted UMLS to RDF, you will get Turtle (.ttl) files that can be uploaded using the BioPortal Web UI. Please select UMLS as the format for these ontologies.

[edit] How do I know if an ontology has parsed?

The BioPortal Web UI will cache information about ontologies for 60 seconds. After parsing is complete, just refresh the ontology summary page to see the status for the most recent submission listed under the "Submissions" table.

In addition, you can look at the REST service directly, which will always give you the most updated information. To do this, visit the following URL:

[edit] Is there a log file for parsing?

Parsing progress is logged in the ontology submission repository folder: /srv/ncbo/repository/{ontology acronym}/{submission id}

[edit] Web User Interface

[edit] How can I clear the memcached-based UI cache?

[edit] How can I use widgets with my Virtual Appliance?

In addition to the existing instructions, you must define an additional Javascript variable in order to have the widgets communicate with your instance of the Virtual Appliance.

 var BP_SEARCH_SERVER = "http://{your_appliance_ip_or_domain_name}";

Replace the '{your_appliance_ip_or_domain_name}' text with the IP address or domain name that's assigned to your Virtual Appliance.

[edit] Virtualization Environments

[edit] How can I use the OVF image with my virtualization software?
(VMware, VirtualBox, KVM, Xen, etc)

[edit] VMware

You can use VMware's ovftool to convert the appliance to work with your VMware product. For example, to convert the appliance for use in VMware Player or Workstation, you would run the command:

ovftool ncbo-appliance.ovf ncbo-appliance.vmx

[edit] VirtualBox

VirtualBox supports importing OVF images directly. Simply start your VirtualBox software, then select File->Import Appliance and select the OVF file included in the NCBO Virtual Appliance download.

[edit] KVM

First, convert the OVF to VMX format as mentioned in the VMware section above.

Next, ensure that the kvm-qemu-img RPM (or qemu-kvm DEB) is installed. Then, convert the [new] VMDKs (from the VMX conversion step) to raw disk images via the following command:

for disk in `ls -1 *.vmdk`; do diskbase=`basename $disk .vmdk`; qemu-img convert -O raw ${diskbase}.vmdk ${diskbase}.img; done

Create /etc/libvirt/qemu/ncbo-appliance.xml with the following contents:

<domain type='kvm'>
  <name>ncbo-appliance</name>
  <memory>4194304</memory>
  <vcpu>2</vcpu>
  <os>
    <type arch='x86_64' machine='rhel5.4.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/ncbo-appliance/ncbo-appliance-disk1.img'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/ncbo-appliance/ncbo-appliance-disk2.img'/>
      <target dev='hdb' bus='ide'/>
      <address type='drive' controller='0' bus='0' unit='1'/>
    </disk>
    <controller type='ide' index='0'/>
    <interface type='network'>
      <source network='default'/>
      <model type='virtio'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
    </video>
  </devices>
</domain>

Finally, make any necessary edits to the above file, and run:

virsh start ncbo-appliance

[edit] Xen

First, convert the VMDKs to raw disk images as mentioned in the KVM section above.

Create /etc/xen/ncbo-appliance.cfg with the following contents:

name = "ncbo-appliance"
memory = 4096
vcpus = 2
builder = "hvm"
kernel = "/usr/lib/xen/boot/hvmloader"
boot = "c"
pae = 1
acpi = 1
apic = 1
localtime = 0
on_poweroff = "destroy"
on_reboot = "destroy"
on_crash = "destroy"
device_model = "/usr/lib64/xen/bin/qemu-dm"
sdl = 0
vnc = 1
vncunused = 1
keymap = "en-us"
disk = [ "file:/var/lib/xen/images/ncbo-appliance/ncbo-appliance-disk1.img,hda,w", "file:/var/lib/xen/images/ncbo-appliance/ncbo-appliance-disk2.img,hdb,w" ]
vif = [ "bridge=xenbr0,script=vif-bridge,vifname=vif41.0" ]
parallel = "none"
serial = "pty"

Finally, make any necessary edits to the above file, and run:

xm create ncbo-appliance

[edit] How can I use the Appliance on Amazon EC2?

Appliance is available on Amazon Marketplace

[edit] What is the admin password for the AWS Appliance?

The default application administrator is 'admin' and the initial password is the Instance ID (i-xxxxxxxx). Please change this as soon as you install the virtual appliance.

Personal tools
Namespaces

Variants
Actions
Navigation
Toolbox