Difference between revisions of "Importing UMLS To Virtual Appliance"
(changes to add an example workflow)
m (add context from Vincent)
|Line 55:||Line 55:|
<p>This workflow for importing UMLS data has been provided by Vincent Emonet, and has been provided here without testing by the NCBO team.</p>
<p>This workflow for importing UMLS data has been provided by Vincent Emonet, and has been provided here without testing by the NCBO team. </p
<h3>1. Install UMLS using mmsys</h3>
<h3>1. Install UMLS using mmsys</h3>
Revision as of 17:29, 4 December 2015
The NCBO Virtual Appliance supports OBO and OWL ontology formats but not UMLS in its native form. To bridge this gap, we have developed a project called UMLS2RDF that transforms UMLS ontologies into OWL/RDF.
UMLS2RDF is a Python script that connects to a UMLS MySQL installation and extracts the UMLS ontologies in a format that the Appliance can work with.
Install UMLS MySQL
- First clone the github project:
git clone https://github.com/ncbo/umls2rdf/
- Install the MySQL Python driver. We recommend to use
pip install MySQL-python
UMLS2RDF has two configuration files:
- conf.py where the database configuration (host,name,user and password) needs to be specified. Also the output folder.
- umls.conf where one can specified the UMLS ontologies to be extracted. This is a comma separated file with the following 4 fields:
- This is legacy. Any value works.
- Output file name.
- Conversion strategy. Accepted values (load_on_codes, load_on_cuis).
With load_on_codes the original source of the ontology will be used as strategy. The Class IDs will be constructed with the MRCONSO.CODE field. If load_on_cuis is selected then the strategy to transform the ontology will use CUIs to construct the Class IDs.
In our configuration file, you can see the settings used by our production system. These are all the UMLS ontologies that are publicly available in BioPortal.
Once the configuration files have the settings run the command:
Depending on how many ontologies are extracted the run time can range from a few minutes to four hours. This process is memory intensive and to transform the largest UMLS ontologies (i.e: SNOMED) one needs at least 16G RAM available.
Upload files to the NCBO Virtual Appliance
The output files will be located in the folder specified in conf.py. Use the BioPortal Web form available in your appliance to submit the extracted ontologies. IMPORTANT: The ontology format in the submission form should be UMLS.
NCBO dedicates a fair amount of resources (powerful servers) to handle a good portion of UMLS ontologies. Some of the UMLS ontologies contain millions of classes. To import the largest UMLS ontologies (i.e: RXNORM or SNOMEDCT) Users will have to run the Appliance in a powerful dedicated environment with 8GB RAM and 5GB hard disk space available.
This workflow for importing UMLS data has been provided by Vincent Emonet, and has been provided here without testing by the NCBO team. Vincent comments:
This offers more details on how to generate UMLS turtle files, It is simplified to provide the essentials needed to generate the RDF files you want. Note that I am using linux and the 2015AB release for this tutorial.
1. Install UMLS using mmsys
- Download everything from https://www.nlm.nih.gov/research/umls/licensedcontent/umlsknowledgesources.html
- Unzip mmsys.zip
- Put the following files in the now unzipped mmsys directory :
- mmsys.zip (why not?)
- ./run_linux.sh (or run.bat or run_mac.sh)
- Source: path to mmsys directory and Destination: path of a directory where the subset file will be generated
- Semantic Network -> Choose Database Load Scripts: Mysql 5.6 (to generate the mysql load script for the semantic network subset, aka STY in bioportal)
- Select "New configuration.."
- Select Default Subset: select the default subset you want (it is not really important, you can specificaly choose each thesaurus in the next step)
- Go to the "Output Options" tab > Write Database Load Scripts
- Select database > MySQL 5.6
- Go to the "Source List" and select the sources (aka ontologies in bioportal) you want
- hold ctrl to select many
- careful there is an option to define if you want the selected source excluded or included in the subset
- Then click on "Done" in the top command on the window
- And wait for UMLS to install
- Note: mmsys will generate RRF files. You can use the mmsys software to browse it, but we will just use it to populate a SQL database
Load subset in MySQL
- Go to mysql
- Configure database character encoding to UTF-8
- Go to the 2015AB directory (where UMLS has been installed) in the META directory
- Open the populate_mysql_db.sh script and change the first lines with your MySQL credentials:
- (note: for MYSQL_HOME it adds "/bin/mysql" to find the mysql bin)
- Then run
>create database umls2015ab;
>ALTER DATABASE umls2015ab CHARACTER SET utf8 COLLATE utf8_unicode_ci;
Generate RDF from MySQL
- Clone https://github.com/ncbo/umls2rdf
- Rename conf_sample.py as conf.py and configure the access to your database (example when database in local)
#Folder to dump the RDF files. OUTPUT_FOLDER = "output" #DB Config DB_HOST = "localhost" DB_NAME = "umls2015ab" DB_USER = "root" DB_PASS = "<password>" UMLS_VERSION = "2015ab" UMLS_BASE_URI = "http://purl.bioontology.org/ontology/"
- Example for LNC-RU-RU: "LNC-RU-RU,LNC-RU-RU.ttl,load_on_codes"
- If you want to get the name used for you ontology in the database you can get them by going to mySQL, select the umls database and get it from the following query: "select distinct SAB from MRCONSO;"