Importing UMLS To Virtual Appliance - Revision history

Alexskr at 22:52, 23 March 2023

2023-03-23T22:52:28Z

Graybeal: added Migrated category

2020-06-13T01:26:43Z

added Migrated category

Graybeal: forwarded to new content

2020-06-13T01:16:43Z

forwarded to new content

Vemonet at 17:12, 1 February 2016

2016-02-01T17:12:36Z

Graybeal: add context from Vincent

2015-12-05T00:29:55Z

add context from Vincent

Graybeal: changes to add an example workflow

2015-12-05T00:24:31Z

changes to add an example workflow

Palexand: Created page with "Category:NCBO Virtual Appliance
The NCBO Virtual Appliance supports [http://www.geneontology.org/GO.format.obo-1_2.shtml OBO] and [http://www.w3.org/TR/owl-features/ O..."

2014-03-05T19:05:22Z

Created page with "Category:NCBO Virtual Appliance <p>The NCBO Virtual Appliance supports [http://www.geneontology.org/GO.format.obo-1_2.shtml OBO] and [http://www.w3.org/TR/owl-features/ O..."

New page

[[Category:NCBO Virtual Appliance]]

<p>The NCBO Virtual Appliance supports [http://www.geneontology.org/GO.format.obo-1_2.shtml OBO] and [http://www.w3.org/TR/owl-features/ OWL] ontology formats but not UMLS in its native form. To bridge this gap, we have developed a project called [https://github.com/ncbo/umls2rdf/ UMLS2RDF] that transforms UMLS ontologies into OWL/RDF.</p>

<p>UMLS2RDF is a Python script that connects to a UMLS MySQL installation and extracts the UMLS ontologies in a format that the Appliance can work with.</p>

<h3>Install UMLS MySQL</h3>

<p>To import UMLS ontologies, a local installation of the [http://www.nlm.nih.gov/research/umls/implementation_resources/scripts/index.html UMLS MySQL release] needs to be available. Please refer to the [http://www.nlm.nih.gov/research/umls/new_users/index.html UMLS documentation] for instructions on how to install the UMLS MySQL distribution.</p>

<h3>Install UMLS2RDF</h3>

<ol>
<li>First clone the github project:<br/><code>git clone https://github.com/ncbo/umls2rdf/</code></li>
<li>Install the MySQL Python driver. We recommend to use <code>pip</code> for this:<br/><code>pip install MySQL-python</code></li>
</ol>

<h3>Configure UMLS2RDF</h3>

<p>UMLS2RDF has two configuration files:</p>

<ol>
<li><strong>conf.py</strong> where the database configuration (host,name,user and password) needs to be specified. Also the output folder.</li>
<li><strong>umls.conf</strong> where one can specified the UMLS ontologies to be extracted. This is a comma separated file with the following 4 fields:
<ol type="a">
<li>SAB</li>
<li>This is legacy. Any value works.</li>
<li>Output file name.</li>
<li>Conversion strategy. Accepted values (load_on_codes, load_on_cuis).</li>
</ol>
</li>
</ol>

<p>With <em>load_on_codes</em> the original source of the ontology will be used as strategy. The Class IDs will be constructed with the MRCONSO.CODE field. If <em>load_on_cuis</em> is selected then the strategy to transform the ontology will use CUIs to construct the Class IDs.</p>

<p>In our [https://github.com/ncbo/umls2rdf/blob/master/umls.conf configuration file], you can see the settings used by our production system. These are all the UMLS ontologies that are publicly available in BioPortal.</p>

<h3>Run UMLS2RDF</h3>

<p>Once the configuration files have the settings run the command:</p>

<p><code>python umls2rdf.py</code></p>

<p>Depending on how many ontologies are extracted the run time can range from a few minutes to four hours. This process is memory intensive and to transform the largest UMLS ontologies (i.e: SNOMED) one needs at least 16G RAM available.</p>

<h3>Upload files to the NCBO Virtual Appliance</h3>

<p>The output files will be located in the folder specified in <strong>conf.py</strong>. Use the BioPortal Web form available in your appliance to submit the extracted ontologies. <strong>IMPORTANT:</strong> The ontology format in the submission form should be UMLS.</p>

<h3>Hardware Considerations</h3>

<p>NCBO dedicates a fair amount of resources (powerful servers) to handle a good portion of UMLS ontologies. Some of the UMLS ontologies contain millions of classes. To import the largest UMLS ontologies (i.e: RXNORM or SNOMEDCT) Users will have to run the Appliance in a powerful dedicated environment with 8GB RAM and 5GB hard disk space available.</p>

@@ Line 2: / Line 2: @@
 [[Category:Migrated to GitHub]]
-This content has been moved! Please find the new content for the 3.0 version of the Virtual Appliance at our new [https://ontoportal.github.io/administration/ OntoPortal Virtual Appliance Administration pages].
+This content has been moved! Please find the new content for the 3.0 version of the Virtual Appliance at our new [https://ontoportal.github.io/documentation/administration OntoPortal Virtual Appliance Administration pages].
-In particular '''this content''' is mostly at the '''[https://ontoportal.github.io/administration/ontologies/handling_umls/ Submitting UMLS Content page]'''.
+In particular '''this content''' is mostly at the '''[https://ontoportal.github.io/documentation/administration/ontologies/handling_umls Submitting UMLS Content page]'''.

@@ Line 1: / Line 1: @@
 [[Category:NCBO Virtual Appliance]]
 This content has been moved! Please find the new content for the 3.0 version of the Virtual Appliance at our new [https://ontoportal.github.io/administration/ OntoPortal Virtual Appliance Administration pages].
 In particular '''this content''' is mostly at the '''[https://ontoportal.github.io/administration/ontologies/handling_umls/ Submitting UMLS Content page]'''.

@@ Line 58: / Line 58: @@
 <blockquote>This offers more details on how to generate UMLS turtle files, It is simplified to provide the essentials needed to generate the RDF files you want. Note that I am using linux and the 2015AB release for this tutorial.</blockquote>
-<h3>1. Install UMLS using mmsys</h3>
+<h3>Install UMLS using mmsys</h3>
 <ul>

@@ Line 55: / Line 55: @@
 <h2>Example Workflow</h2>
-<p>This workflow for importing UMLS data has been provided by Vincent Emonet, and has been provided here without testing by the NCBO team.</p>
+<p>This workflow for importing UMLS data has been provided by Vincent Emonet, and has been provided here without testing by the NCBO team. Vincent comments:</p>
 <h3>1. Install UMLS using mmsys</h3>

← Older revision		Revision as of 00:24, 5 December 2015
Line 52:		Line 52:

	<p>NCBO dedicates a fair amount of resources (powerful servers) to handle a good portion of UMLS ontologies. Some of the UMLS ontologies contain millions of classes. To import the largest UMLS ontologies (i.e: RXNORM or SNOMEDCT) Users will have to run the Appliance in a powerful dedicated environment with 8GB RAM and 5GB hard disk space available.</p>		<p>NCBO dedicates a fair amount of resources (powerful servers) to handle a good portion of UMLS ontologies. Some of the UMLS ontologies contain millions of classes. To import the largest UMLS ontologies (i.e: RXNORM or SNOMEDCT) Users will have to run the Appliance in a powerful dedicated environment with 8GB RAM and 5GB hard disk space available.</p>
		+
		+	<h2>Example Workflow</h2>
		+
		+	<p>This workflow for importing UMLS data has been provided by Vincent Emonet, and has been provided here without testing by the NCBO team.</p>
		+
		+	<h3>1. Install UMLS using mmsys</h3>
		+
		+	<ul>
		+	<li>Download everything from https://www.nlm.nih.gov/research/umls/licensedcontent/umlsknowledgesources.html</li>
		+	<li>Unzip mmsys.zip</li>
		+	<li>Put the following files in the now unzipped mmsys directory : </li>
		+	<ul>
		+	<li>2015AB-1-meta.nlm</li>
		+	<li>2015AB-2-meta.nlm</li>
		+	<li>2015AB-otherks.nlm</li>
		+	<li>mmsys.zip (why not?)</li>
		+	<li>2015AB.CHK</li>
		+	</ul>
		+	<li>./run_linux.sh (or run.bat or run_mac.sh)</li>
		+	<code>Install UMLS</code>
		+	<li>Source: path to mmsys directory and Destination: path of a directory where the subset file will be generated</li>
		+	<li>Semantic Network -> Choose Database Load Scripts: Mysql 5.6 (to generate the mysql load script for the semantic network subset, aka STY in bioportal)</li>
		+	<li>Select "New configuration.."</li>
		+	<li>Select Default Subset: select the default subset you want (it is not really important, you can specificaly choose each thesaurus in the next step)</li>
		+	<li>Go to the "Output Options" tab > Write Database Load Scripts</li>
		+	<ul><li>Select database > MySQL 5.6</li></ul>
		+	<li>Go to the "Source List" and select the sources (aka ontologies in bioportal) you want</li>
		+	<ul>
		+	<li>hold ctrl to select many</li>
		+	<li>careful there is an option to define if you want the selected source excluded or included in the subset</li>
		+	</ul>
		+	<li>Then click on "Done" in the top command on the window</li>
		+	<li>And wait for UMLS to install</li>
		+	<li>Note: mmsys will generate RRF files. You can use the mmsys software to browse it, but we will just use it to populate a SQL database</li>
		+	</ul>
		+	<h3>Load subset in MySQL</h3>
		+	<ul>
		+	<li>Go to mysql</li>
		+
		+	<code>>create database umls2015ab;</code></li>
		+	<li>Configure database character encoding to UTF-8</li>
		+	<code>>ALTER DATABASE umls2015ab CHARACTER SET utf8 COLLATE utf8_unicode_ci;</code></li>
		+	<li>Go to the 2015AB directory (where UMLS has been installed) in the META directory</li>
		+	<li>Open the populate_mysql_db.sh script and change the first lines with your MySQL credentials:</li>
		+	<ul><li>(note: for MYSQL_HOME it adds "/bin/mysql" to find the mysql bin)</li></ul>
		+	<code>
		+	MYSQL_HOME=/usr
		+	user=<username>
		+	password=<password>
		+	db_name=umls2015ab
		+	</code>
		+
		+	<li>Then run <code>./populate_mysql_db.sh</code></li>
		+	</ul>
		+	<h3>Generate RDF from MySQL</h3>
		+	<ul>
		+	<li>Clone https://github.com/ncbo/umls2rdf</li>
		+	<li>Rename conf_sample.py as conf.py and configure the access to your database (example when database in local)</li>
		+	<pre>
		+	#Folder to dump the RDF files.
		+	OUTPUT_FOLDER = "output"
		+
		+	#DB Config
		+	DB_HOST = "localhost"
		+	DB_NAME = "umls2015ab"
		+	DB_USER = "root"
		+	DB_PASS = "<password>"
		+
		+	UMLS_VERSION = "2015ab"
		+	UMLS_BASE_URI = "http://purl.bioontology.org/ontology/"
		+	</pre>
		+
		+	<li>Define the ontology you want to generate in umls.conf</li>
		+	<ul>
		+	<li>Example for LNC-RU-RU: "LNC-RU-RU,LNC-RU-RU.ttl,load_on_codes"</li>
		+	<li>If you want to get the name used for you ontology in the database you can get them by going to mySQL, select the umls database and get it from the following query: "select distinct SAB from MRCONSO;"</li>
		+	</ul>
		+	<li>Run ./umls2rdf.py</li>
		+	<li>Get the ttl files in the output directory</li>
		+	</ul>