https://www.bioontology.org//mediawiki/api.php?action=feedcontributions&feedformat=atom&user=Tejaswit 2026-06-06T05:52:29Z User contributions MediaWiki 1.35.9 https://www.bioontology.org//mediawiki/index.php?title=MetaMap_Installation&diff=9708 2010-01-28T04:42:28Z

<p>Tejaswit: /* Versions (prototypes &amp; releases) */</p> <hr /> <div>== Presentation ==<br /> MetaMap [http://mmtx.nlm.nih.gov/] is a highly powerful tool to map biomedical text to the UMLS [http://www.nlm.nih.gov/research/umls/] Metathesaurus or a custom Metathesaurus, equivalently, to discover Metathesaurus concepts referred to in text. The following article describes the way of using the Data File Builder utility inside MetaMap to build a concept recognizer for text containing NCBO terms.<br /> <br /> NCBO [http://www.bioontology.org/] is developing a system [http://www.bioontology.org/annotator-service] to annotate large numbers of data resources automatically, and have developed a prototype system for ontology-based annotation and indexing of biomedical data [http://www.biomedcentral.com/1471-2105/10/S9/S14]. As a part of the workflow a concept recognizer is used to recognize a given ontology concept in text. The goal of the project is to embed MetaMap into the annotator workflow. MetaMap is a very powerful for this task due to the use of complex NLP techniques in its concept recognition algorithm. <br /> <br /> Below listed is some of the useful processing done by MetaMap:<br /> <br /> 1. Removal of problematic terms from data [numbers, single alphabets etc..]<br /> <br /> 2. Lexical and Syntactic filtering to recognize long phrases accurately<br /> <br /> 3. Identifying linguistic variants of base words and terms that have been computed using a variety of generation methods. This handles various forms like inflectional (heart → hearts), derivational (treat → treatment), synonyms (pyrosis ↔ heartburn), acronyms (vt ↔ ventricular tachycardia) and spellings (apnea ↔ apnoea)<br /> <br /> Such kind of rich semantic processing can drastically increase the number of concepts that can be recognized from a text. <br /> <br /> Although Metamap was originally built to work on UMLS concepts, its Data File builder utility (DFB) lets us create our own custom dictionary for NCBO concepts.<br /> <br /> == Contacts ==<br /> <br /> * For questions or feature requests, contact Clement Jonquet [mailto:jonquet@stanford.edu?subject=MetaMap%Installation] and Tejaswi Tenneti [mailto:tejaswit@stanford.edu?subject=MetaMap%Installation]<br /> <br /> == Versions (prototypes &amp; releases) ==<br /> <br /> * '''December 2009 - Second Annotator prototype including MetaMap: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html]''' <br /> <br /> The release consists of an integrated MetaMap 2009 installation. The Data File builder utility was used build a custom dictionary containing the ontologies '''CST, OMS, 39857, 39046, 40187 and 39885'''. The data was obtained from the '''obs_v1_sept09''' schema. However, some ontologies like 39857 is not present in '''obs_v1_sept09'''<br /> <br /> * '''November 2009 - Second Annotator prototype including Mmtx: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html]''' <br /> <br /> The release consists of an integrated MMTx installation. The Data File builder utility was used build a custom dictionary containing all the ontologies in the '''obs''' schema. A defect with this installation is that the Data File builder scripts processed only CUIs of 8 digits length. This is a bug in the MMTx documentation.<br /> <br /> == References ==<br /> * gforge code location https://bmir-gforge.stanford.edu/gf/project/obs/<br /> <br /> == Collaboration &amp; Acknowledgment ==<br /> <br /> We thank the researchers at National Library of Medicine [http://www.nlm.nih.gov/] for their constant support and help at various stages of this project.<br /> <br /> == MetaMap Installation documentation ==<br /> <br /> The installation of MetaMap primarily involves 3 stages. Each of them are explained below followed by instructions to run the newly installed custom MetaMap.<br /> <br /> === Preparation of data ===<br /> <br /> There are primarily 4 files to be generated for the installation. <br /> <br /> '''1.MRCON:''' This file is primarily generated from the table OBS_TT. Below is a mapping between the columns in MRCON and OBS_TT. The columns listed in OBS_TT are used to fill their corresponding columns in MRCON file.<br /> <br /> MRCON OBS_TT<br /> <br /> CUI (Concept Unique ID) conceptID<br /> <br /> Language- <br /> Status isPreferred <br /> LUI(Lexical Unique ID) termID <br /> String type -<br /> SUI(String Unique ID) termID <br /> String termName <br /> LRL - <br /> <br /> The sql command would fill correct values into MRCON table in database. <br /> <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|ENG|P|L0027051|PF| S0064638|Myocardial Infarction|0|<br /> <br /> <br /> '''2.MRSO:''' This file is primarily generated from the tables OBS_TT, OBS_OT and OBS_CT. Below is a mapping between the columns in MRSO and the above OBS tables. The columns listed in the tables are used to fill their corresponding columns in MRSO file.<br /> <br /> MRSO OBS tables<br /> CUI (Concept Unique ID) OBS_TT.conceptID<br /> LUI(Lexical Unique ID) OBS_TT.termID<br /> SUI(String Unique ID) OBS_TT.termID <br /> SABSourceAbbrev OBS_OT.localOntologyID<br /> TermType OBS_TT.isPreferred<br /> SourceID OBS_OT.ontologyID<br /> Restrictionlevel -<br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|L0027051|S0064638| MEDLINEPLUS|ET|T5|0|<br /> <br /> <br /> '''3.MRSTY:''' This file is primarily generated from the table OBS_STT. Below is a mapping between the columns in MRSTY and OBS_STT. The columns listed in OBS_STT are used to fill their corresponding columns in MRSTY file.<br /> <br /> MRSTY OBS_STT<br /> CUI (Concept Unique ID) OBS_STT.conceptID<br /> TUI (term Unique ID) OBS_STT.localSemanticTypeID<br /> STY OBS_STT.semanticTypeName <br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|T047|Disease or Syndrome|<br /> <br /> <br /> '''4.st.raw:''' This file is primarily generated from the column semanticTypeName in the table OBS_STT. This file contains row of semantic type names and a short name for each of them. <br /> <br /> Example row:<br /> semantic Type Name|semantictypename<br /> <br /> <br /> <br /> === Setting up the workspace ===<br /> <br /> 1. '''Install the Lexical Variant Generator (LVG)''' before running<br /> MetaMap's install program. LVG is part of the Lexical Tools distribution and is available from the Lexical Systems Group (http://lexsrv3.nlm.nih.gov/SPECIALIST/Projects/lvg/current/index.html).<br /> <br /> 2. Before using MetaMap install program to install data file builder, you also need to '''add LVG's bin directory''' {LVG_DIR}/bin to your program path:<br /> <br /> export PATH=$PATH:&lt;LVG_DIR&gt;/bin<br /> <br /> 3. '''Install MetaMap''' using instructions at their site <br /> <br /> 4. Unzip the DFB installation files in the same directory as MetaMap<br /> <br /> 5. Connect to the new directory created by extracting the distribution and invoke the install program:<br /> <br /> cd &lt;distribution directory&gt;<br /> <br /> ./bin/install.sh<br /> <br /> '''A sample run of the installation script follows:'''<br /> <br /> Enter basedir of installation [/nfsvol/nlsaux15/public_mm] <br /> <br /> Basedir is set to /nfsvol/nlsaux15/public_mm.<br /> <br /> The WSD Server requires Sun's Java Runtime Environment (JRE)<br /> Sun's Java Developer Kit (JDK) will work as well. if the<br /> command: &quot;which&quot; java returns /usr/local/jre1.4.2/bin/java, then the<br /> JRE resides in /usr/local/jre1.4.2/.<br /> <br /> Where does your distribution of Sun's JRE reside?<br /> Enter home path of JRE (JDK) [/usr]: /nfsvol/nls/tools/Linux-i686/java1.4.2<br /> Using /nfsvol/nls/tools/Linux-i686/java1.4.2 for JAVA_HOME.<br /> <br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/disambServer.cfg generated<br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/log4j.properties generated<br /> /nfsvol/nlsaux15/public_mm/bin/SKRrun generated.<br /> /nfsvol/nlsaux15/public_mm/bin/metamap07 generated.<br /> /nfsvol/nlsaux15/public_mm/bin/wsdserverctl generated.<br /> /nfsvol/nlsaux15/public_mm/bin/skrmedpostctl generated.<br /> Install complete.<br /> Would like to use a custom data set with MetaMap (use data file builder)? [yN]: <br /> <br /> running Data File Builder Install...<br /> Is LVG installed? [yN] &lt;The user types y and return&gt;<br /> <br /> running Data File Builder Install...<br /> Enter home path of LVG [/nfsvol/nls/tools/Linux-i686/lvg2009]:<br /> Using /nfsvol/nls/tools/Linux-i686/lvg2009 for LVG_DIR.<br /> <br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.lvglab generated.<br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.xwords generated.<br /> Datafile Builder Setup is complete.<br /> <br /> <br /> 6. '''Make sure that the SKR/MedPOST tagger is running;''' to run the tagger, move to the public_mm directory present inside the working directory and invoke<br /> <br /> ./bin/skrmedpostctl start<br /> <br /> '''Incase there is an error''' due to a port number, change the port number in the following files and try to run the tagger again:<br /> <br /> /bin/skrmedpostctl<br /> <br /> If you change the port number skrmedpostctl uses you need to change public_mm/bin/SKRrun (or SKRrun.in and re-run install.sh) to match the port you set in skrmedpostctl. The environment variables that should be changed in SKRrun are:<br /> <br /> TAGGER_SERVER_DEFAULT_TCP_PORT<br /> <br /> TAGGER_SERVER_TCP_PORT_0<br /> <br /> TAGGER_SERVER_TCP_PORT_1<br /> <br /> <br /> 7. Inside the directory public_mm/sourceData create a directory for workspace<br /> <br /> mkdir sourceData/09_custom<br /> <br /> Finally, Create a directory to store the knowledge sources:<br /> <br /> mkdir sourceData/09_custom/umls<br /> <br /> === Installation of Metamap ===<br /> <br /> <br /> 1. First, run the BuildDataFiles program as follows:<br /> <br /> $fullpath/public_mm/bin/BuildDataFiles<br /> <br /> 2. The file st.raw, located in $fullpath/public_mm/data/dfbuilder/2009 will need to be modified. Append the file generated in point 4 of Preparation of Data section to this file.<br /> <br /> 3. Move to $fullpath/public_mm/sourceData/09_custom/01metawordindex and execute the following scripts in order<br /> <br /> ./01CreateWorkFiles<br /> <br /> ./02Suppress<br /> <br /> ./03FilterPrep<br /> <br /> ./04FilterStrict<br /> <br /> ./05GenerateMWIFiles<br /> <br /> 4. Move to 02treecodes directory and run 01GenerateTreecodes<br /> <br /> cd ../02treecodes<br /> <br /> ./01GenerateTreecodes<br /> <br /> 5. Move to 03Variants directory and run 01GenerateVariants<br /> <br /> cd ../03Variants<br /> <br /> ./01GenerateVariants<br /> <br /> 6. cd ../04synonyms <br /> <br /> ./01GenerateSynonyms<br /> <br /> 7. cd ../05abbrAcronyms<br /> <br /> ./01GenerateAbbrAcronyms<br /> <br /> 8. move to $fullpath/public_mm and run<br /> <br /> ./bin/LoadDataFiles<br /> <br /> === Running MetaMap ===<br /> <br /> To run the newly installed MetaMap dfb, move to the main workspace folder (public_mm) and run the command below<br /> <br /> bin/SKRrun -L 2009 -M /DATA/XDR -B /BDB4 -w ./lexicon ./bin/metamap09.BINARY -Z 09_custom ./resources/input -I</div>

Tejaswit https://www.bioontology.org//mediawiki/index.php?title=MetaMap_Installation&diff=9707 2010-01-28T04:39:25Z

<p>Tejaswit: /* Versions (prototypes &amp; releases) */</p> <hr /> <div>== Presentation ==<br /> MetaMap [http://mmtx.nlm.nih.gov/] is a highly powerful tool to map biomedical text to the UMLS [http://www.nlm.nih.gov/research/umls/] Metathesaurus or a custom Metathesaurus, equivalently, to discover Metathesaurus concepts referred to in text. The following article describes the way of using the Data File Builder utility inside MetaMap to build a concept recognizer for text containing NCBO terms.<br /> <br /> NCBO [http://www.bioontology.org/] is developing a system [http://www.bioontology.org/annotator-service] to annotate large numbers of data resources automatically, and have developed a prototype system for ontology-based annotation and indexing of biomedical data [http://www.biomedcentral.com/1471-2105/10/S9/S14]. As a part of the workflow a concept recognizer is used to recognize a given ontology concept in text. The goal of the project is to embed MetaMap into the annotator workflow. MetaMap is a very powerful for this task due to the use of complex NLP techniques in its concept recognition algorithm. <br /> <br /> Below listed is some of the useful processing done by MetaMap:<br /> <br /> 1. Removal of problematic terms from data [numbers, single alphabets etc..]<br /> <br /> 2. Lexical and Syntactic filtering to recognize long phrases accurately<br /> <br /> 3. Identifying linguistic variants of base words and terms that have been computed using a variety of generation methods. This handles various forms like inflectional (heart → hearts), derivational (treat → treatment), synonyms (pyrosis ↔ heartburn), acronyms (vt ↔ ventricular tachycardia) and spellings (apnea ↔ apnoea)<br /> <br /> Such kind of rich semantic processing can drastically increase the number of concepts that can be recognized from a text. <br /> <br /> Although Metamap was originally built to work on UMLS concepts, its Data File builder utility (DFB) lets us create our own custom dictionary for NCBO concepts.<br /> <br /> == Contacts ==<br /> <br /> * For questions or feature requests, contact Clement Jonquet [mailto:jonquet@stanford.edu?subject=MetaMap%Installation] and Tejaswi Tenneti [mailto:tejaswit@stanford.edu?subject=MetaMap%Installation]<br /> <br /> == Versions (prototypes &amp; releases) ==<br /> <br /> * December 2009 - Second Annotator prototype including MetaMap: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> <br /> The release consists of an integrated MetaMap 2009 installation. The Data File builder utility was used build a custom dictionary containing the ontologies '''CST, OMS, 39857, 39046, 40187 and 39885'''. The data was obtained from the '''obs_v1_sept09''' schema. However, some ontologies like 39857 is not present in '''obs_v1_sept09'''<br /> <br /> * November 2009 - Second Annotator prototype including Mmtx: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> <br /> The release consists of an integrated MMTx installation. The Data File builder utility was used build a custom dictionary containing all the ontologies in the '''obs''' schema. A defect with this installation is that the Data File builder scripts processed only CUIs of 8 digits length. This is a bug in the MMTx documentation.<br /> <br /> == References ==<br /> * gforge code location https://bmir-gforge.stanford.edu/gf/project/obs/<br /> <br /> == Collaboration &amp; Acknowledgment ==<br /> <br /> We thank the researchers at National Library of Medicine [http://www.nlm.nih.gov/] for their constant support and help at various stages of this project.<br /> <br /> == MetaMap Installation documentation ==<br /> <br /> The installation of MetaMap primarily involves 3 stages. Each of them are explained below followed by instructions to run the newly installed custom MetaMap.<br /> <br /> === Preparation of data ===<br /> <br /> There are primarily 4 files to be generated for the installation. <br /> <br /> '''1.MRCON:''' This file is primarily generated from the table OBS_TT. Below is a mapping between the columns in MRCON and OBS_TT. The columns listed in OBS_TT are used to fill their corresponding columns in MRCON file.<br /> <br /> MRCON OBS_TT<br /> <br /> CUI (Concept Unique ID) conceptID<br /> <br /> Language- <br /> Status isPreferred <br /> LUI(Lexical Unique ID) termID <br /> String type -<br /> SUI(String Unique ID) termID <br /> String termName <br /> LRL - <br /> <br /> The sql command would fill correct values into MRCON table in database. <br /> <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|ENG|P|L0027051|PF| S0064638|Myocardial Infarction|0|<br /> <br /> <br /> '''2.MRSO:''' This file is primarily generated from the tables OBS_TT, OBS_OT and OBS_CT. Below is a mapping between the columns in MRSO and the above OBS tables. The columns listed in the tables are used to fill their corresponding columns in MRSO file.<br /> <br /> MRSO OBS tables<br /> CUI (Concept Unique ID) OBS_TT.conceptID<br /> LUI(Lexical Unique ID) OBS_TT.termID<br /> SUI(String Unique ID) OBS_TT.termID <br /> SABSourceAbbrev OBS_OT.localOntologyID<br /> TermType OBS_TT.isPreferred<br /> SourceID OBS_OT.ontologyID<br /> Restrictionlevel -<br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|L0027051|S0064638| MEDLINEPLUS|ET|T5|0|<br /> <br /> <br /> '''3.MRSTY:''' This file is primarily generated from the table OBS_STT. Below is a mapping between the columns in MRSTY and OBS_STT. The columns listed in OBS_STT are used to fill their corresponding columns in MRSTY file.<br /> <br /> MRSTY OBS_STT<br /> CUI (Concept Unique ID) OBS_STT.conceptID<br /> TUI (term Unique ID) OBS_STT.localSemanticTypeID<br /> STY OBS_STT.semanticTypeName <br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|T047|Disease or Syndrome|<br /> <br /> <br /> '''4.st.raw:''' This file is primarily generated from the column semanticTypeName in the table OBS_STT. This file contains row of semantic type names and a short name for each of them. <br /> <br /> Example row:<br /> semantic Type Name|semantictypename<br /> <br /> <br /> <br /> === Setting up the workspace ===<br /> <br /> 1. '''Install the Lexical Variant Generator (LVG)''' before running<br /> MetaMap's install program. LVG is part of the Lexical Tools distribution and is available from the Lexical Systems Group (http://lexsrv3.nlm.nih.gov/SPECIALIST/Projects/lvg/current/index.html).<br /> <br /> 2. Before using MetaMap install program to install data file builder, you also need to '''add LVG's bin directory''' {LVG_DIR}/bin to your program path:<br /> <br /> export PATH=$PATH:&lt;LVG_DIR&gt;/bin<br /> <br /> 3. '''Install MetaMap''' using instructions at their site <br /> <br /> 4. Unzip the DFB installation files in the same directory as MetaMap<br /> <br /> 5. Connect to the new directory created by extracting the distribution and invoke the install program:<br /> <br /> cd &lt;distribution directory&gt;<br /> <br /> ./bin/install.sh<br /> <br /> '''A sample run of the installation script follows:'''<br /> <br /> Enter basedir of installation [/nfsvol/nlsaux15/public_mm] <br /> <br /> Basedir is set to /nfsvol/nlsaux15/public_mm.<br /> <br /> The WSD Server requires Sun's Java Runtime Environment (JRE)<br /> Sun's Java Developer Kit (JDK) will work as well. if the<br /> command: &quot;which&quot; java returns /usr/local/jre1.4.2/bin/java, then the<br /> JRE resides in /usr/local/jre1.4.2/.<br /> <br /> Where does your distribution of Sun's JRE reside?<br /> Enter home path of JRE (JDK) [/usr]: /nfsvol/nls/tools/Linux-i686/java1.4.2<br /> Using /nfsvol/nls/tools/Linux-i686/java1.4.2 for JAVA_HOME.<br /> <br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/disambServer.cfg generated<br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/log4j.properties generated<br /> /nfsvol/nlsaux15/public_mm/bin/SKRrun generated.<br /> /nfsvol/nlsaux15/public_mm/bin/metamap07 generated.<br /> /nfsvol/nlsaux15/public_mm/bin/wsdserverctl generated.<br /> /nfsvol/nlsaux15/public_mm/bin/skrmedpostctl generated.<br /> Install complete.<br /> Would like to use a custom data set with MetaMap (use data file builder)? [yN]: <br /> <br /> running Data File Builder Install...<br /> Is LVG installed? [yN] &lt;The user types y and return&gt;<br /> <br /> running Data File Builder Install...<br /> Enter home path of LVG [/nfsvol/nls/tools/Linux-i686/lvg2009]:<br /> Using /nfsvol/nls/tools/Linux-i686/lvg2009 for LVG_DIR.<br /> <br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.lvglab generated.<br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.xwords generated.<br /> Datafile Builder Setup is complete.<br /> <br /> <br /> 6. '''Make sure that the SKR/MedPOST tagger is running;''' to run the tagger, move to the public_mm directory present inside the working directory and invoke<br /> <br /> ./bin/skrmedpostctl start<br /> <br /> '''Incase there is an error''' due to a port number, change the port number in the following files and try to run the tagger again:<br /> <br /> /bin/skrmedpostctl<br /> <br /> If you change the port number skrmedpostctl uses you need to change public_mm/bin/SKRrun (or SKRrun.in and re-run install.sh) to match the port you set in skrmedpostctl. The environment variables that should be changed in SKRrun are:<br /> <br /> TAGGER_SERVER_DEFAULT_TCP_PORT<br /> <br /> TAGGER_SERVER_TCP_PORT_0<br /> <br /> TAGGER_SERVER_TCP_PORT_1<br /> <br /> <br /> 7. Inside the directory public_mm/sourceData create a directory for workspace<br /> <br /> mkdir sourceData/09_custom<br /> <br /> Finally, Create a directory to store the knowledge sources:<br /> <br /> mkdir sourceData/09_custom/umls<br /> <br /> === Installation of Metamap ===<br /> <br /> <br /> 1. First, run the BuildDataFiles program as follows:<br /> <br /> $fullpath/public_mm/bin/BuildDataFiles<br /> <br /> 2. The file st.raw, located in $fullpath/public_mm/data/dfbuilder/2009 will need to be modified. Append the file generated in point 4 of Preparation of Data section to this file.<br /> <br /> 3. Move to $fullpath/public_mm/sourceData/09_custom/01metawordindex and execute the following scripts in order<br /> <br /> ./01CreateWorkFiles<br /> <br /> ./02Suppress<br /> <br /> ./03FilterPrep<br /> <br /> ./04FilterStrict<br /> <br /> ./05GenerateMWIFiles<br /> <br /> 4. Move to 02treecodes directory and run 01GenerateTreecodes<br /> <br /> cd ../02treecodes<br /> <br /> ./01GenerateTreecodes<br /> <br /> 5. Move to 03Variants directory and run 01GenerateVariants<br /> <br /> cd ../03Variants<br /> <br /> ./01GenerateVariants<br /> <br /> 6. cd ../04synonyms <br /> <br /> ./01GenerateSynonyms<br /> <br /> 7. cd ../05abbrAcronyms<br /> <br /> ./01GenerateAbbrAcronyms<br /> <br /> 8. move to $fullpath/public_mm and run<br /> <br /> ./bin/LoadDataFiles<br /> <br /> === Running MetaMap ===<br /> <br /> To run the newly installed MetaMap dfb, move to the main workspace folder (public_mm) and run the command below<br /> <br /> bin/SKRrun -L 2009 -M /DATA/XDR -B /BDB4 -w ./lexicon ./bin/metamap09.BINARY -Z 09_custom ./resources/input -I</div>

Tejaswit https://www.bioontology.org//mediawiki/index.php?title=MetaMap_Installation&diff=9706 2010-01-27T10:27:28Z

<p>Tejaswit: /* Collaboration &amp; Acknowledgment */</p> <hr /> <div>== Presentation ==<br /> MetaMap [http://mmtx.nlm.nih.gov/] is a highly powerful tool to map biomedical text to the UMLS [http://www.nlm.nih.gov/research/umls/] Metathesaurus or a custom Metathesaurus, equivalently, to discover Metathesaurus concepts referred to in text. The following article describes the way of using the Data File Builder utility inside MetaMap to build a concept recognizer for text containing NCBO terms.<br /> <br /> NCBO [http://www.bioontology.org/] is developing a system [http://www.bioontology.org/annotator-service] to annotate large numbers of data resources automatically, and have developed a prototype system for ontology-based annotation and indexing of biomedical data [http://www.biomedcentral.com/1471-2105/10/S9/S14]. As a part of the workflow a concept recognizer is used to recognize a given ontology concept in text. The goal of the project is to embed MetaMap into the annotator workflow. MetaMap is a very powerful for this task due to the use of complex NLP techniques in its concept recognition algorithm. <br /> <br /> Below listed is some of the useful processing done by MetaMap:<br /> <br /> 1. Removal of problematic terms from data [numbers, single alphabets etc..]<br /> <br /> 2. Lexical and Syntactic filtering to recognize long phrases accurately<br /> <br /> 3. Identifying linguistic variants of base words and terms that have been computed using a variety of generation methods. This handles various forms like inflectional (heart → hearts), derivational (treat → treatment), synonyms (pyrosis ↔ heartburn), acronyms (vt ↔ ventricular tachycardia) and spellings (apnea ↔ apnoea)<br /> <br /> Such kind of rich semantic processing can drastically increase the number of concepts that can be recognized from a text. <br /> <br /> Although Metamap was originally built to work on UMLS concepts, its Data File builder utility (DFB) lets us create our own custom dictionary for NCBO concepts.<br /> <br /> == Contacts ==<br /> <br /> * For questions or feature requests, contact Clement Jonquet [mailto:jonquet@stanford.edu?subject=MetaMap%Installation] and Tejaswi Tenneti [mailto:tejaswit@stanford.edu?subject=MetaMap%Installation]<br /> <br /> == Versions (prototypes &amp; releases) ==<br /> <br /> * December 2009 - Second Annotator prototype including MetaMap: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> * November 2009 - Second Annotator prototype including Mmtx: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> <br /> == References ==<br /> * gforge code location https://bmir-gforge.stanford.edu/gf/project/obs/<br /> <br /> == Collaboration &amp; Acknowledgment ==<br /> <br /> We thank the researchers at National Library of Medicine [http://www.nlm.nih.gov/] for their constant support and help at various stages of this project.<br /> <br /> == MetaMap Installation documentation ==<br /> <br /> The installation of MetaMap primarily involves 3 stages. Each of them are explained below followed by instructions to run the newly installed custom MetaMap.<br /> <br /> === Preparation of data ===<br /> <br /> There are primarily 4 files to be generated for the installation. <br /> <br /> '''1.MRCON:''' This file is primarily generated from the table OBS_TT. Below is a mapping between the columns in MRCON and OBS_TT. The columns listed in OBS_TT are used to fill their corresponding columns in MRCON file.<br /> <br /> MRCON OBS_TT<br /> <br /> CUI (Concept Unique ID) conceptID<br /> <br /> Language- <br /> Status isPreferred <br /> LUI(Lexical Unique ID) termID <br /> String type -<br /> SUI(String Unique ID) termID <br /> String termName <br /> LRL - <br /> <br /> The sql command would fill correct values into MRCON table in database. <br /> <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|ENG|P|L0027051|PF| S0064638|Myocardial Infarction|0|<br /> <br /> <br /> '''2.MRSO:''' This file is primarily generated from the tables OBS_TT, OBS_OT and OBS_CT. Below is a mapping between the columns in MRSO and the above OBS tables. The columns listed in the tables are used to fill their corresponding columns in MRSO file.<br /> <br /> MRSO OBS tables<br /> CUI (Concept Unique ID) OBS_TT.conceptID<br /> LUI(Lexical Unique ID) OBS_TT.termID<br /> SUI(String Unique ID) OBS_TT.termID <br /> SABSourceAbbrev OBS_OT.localOntologyID<br /> TermType OBS_TT.isPreferred<br /> SourceID OBS_OT.ontologyID<br /> Restrictionlevel -<br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|L0027051|S0064638| MEDLINEPLUS|ET|T5|0|<br /> <br /> <br /> '''3.MRSTY:''' This file is primarily generated from the table OBS_STT. Below is a mapping between the columns in MRSTY and OBS_STT. The columns listed in OBS_STT are used to fill their corresponding columns in MRSTY file.<br /> <br /> MRSTY OBS_STT<br /> CUI (Concept Unique ID) OBS_STT.conceptID<br /> TUI (term Unique ID) OBS_STT.localSemanticTypeID<br /> STY OBS_STT.semanticTypeName <br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|T047|Disease or Syndrome|<br /> <br /> <br /> '''4.st.raw:''' This file is primarily generated from the column semanticTypeName in the table OBS_STT. This file contains row of semantic type names and a short name for each of them. <br /> <br /> Example row:<br /> semantic Type Name|semantictypename<br /> <br /> <br /> <br /> === Setting up the workspace ===<br /> <br /> 1. '''Install the Lexical Variant Generator (LVG)''' before running<br /> MetaMap's install program. LVG is part of the Lexical Tools distribution and is available from the Lexical Systems Group (http://lexsrv3.nlm.nih.gov/SPECIALIST/Projects/lvg/current/index.html).<br /> <br /> 2. Before using MetaMap install program to install data file builder, you also need to '''add LVG's bin directory''' {LVG_DIR}/bin to your program path:<br /> <br /> export PATH=$PATH:&lt;LVG_DIR&gt;/bin<br /> <br /> 3. '''Install MetaMap''' using instructions at their site <br /> <br /> 4. Unzip the DFB installation files in the same directory as MetaMap<br /> <br /> 5. Connect to the new directory created by extracting the distribution and invoke the install program:<br /> <br /> cd &lt;distribution directory&gt;<br /> <br /> ./bin/install.sh<br /> <br /> '''A sample run of the installation script follows:'''<br /> <br /> Enter basedir of installation [/nfsvol/nlsaux15/public_mm] <br /> <br /> Basedir is set to /nfsvol/nlsaux15/public_mm.<br /> <br /> The WSD Server requires Sun's Java Runtime Environment (JRE)<br /> Sun's Java Developer Kit (JDK) will work as well. if the<br /> command: &quot;which&quot; java returns /usr/local/jre1.4.2/bin/java, then the<br /> JRE resides in /usr/local/jre1.4.2/.<br /> <br /> Where does your distribution of Sun's JRE reside?<br /> Enter home path of JRE (JDK) [/usr]: /nfsvol/nls/tools/Linux-i686/java1.4.2<br /> Using /nfsvol/nls/tools/Linux-i686/java1.4.2 for JAVA_HOME.<br /> <br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/disambServer.cfg generated<br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/log4j.properties generated<br /> /nfsvol/nlsaux15/public_mm/bin/SKRrun generated.<br /> /nfsvol/nlsaux15/public_mm/bin/metamap07 generated.<br /> /nfsvol/nlsaux15/public_mm/bin/wsdserverctl generated.<br /> /nfsvol/nlsaux15/public_mm/bin/skrmedpostctl generated.<br /> Install complete.<br /> Would like to use a custom data set with MetaMap (use data file builder)? [yN]: <br /> <br /> running Data File Builder Install...<br /> Is LVG installed? [yN] &lt;The user types y and return&gt;<br /> <br /> running Data File Builder Install...<br /> Enter home path of LVG [/nfsvol/nls/tools/Linux-i686/lvg2009]:<br /> Using /nfsvol/nls/tools/Linux-i686/lvg2009 for LVG_DIR.<br /> <br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.lvglab generated.<br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.xwords generated.<br /> Datafile Builder Setup is complete.<br /> <br /> <br /> 6. '''Make sure that the SKR/MedPOST tagger is running;''' to run the tagger, move to the public_mm directory present inside the working directory and invoke<br /> <br /> ./bin/skrmedpostctl start<br /> <br /> '''Incase there is an error''' due to a port number, change the port number in the following files and try to run the tagger again:<br /> <br /> /bin/skrmedpostctl<br /> <br /> If you change the port number skrmedpostctl uses you need to change public_mm/bin/SKRrun (or SKRrun.in and re-run install.sh) to match the port you set in skrmedpostctl. The environment variables that should be changed in SKRrun are:<br /> <br /> TAGGER_SERVER_DEFAULT_TCP_PORT<br /> <br /> TAGGER_SERVER_TCP_PORT_0<br /> <br /> TAGGER_SERVER_TCP_PORT_1<br /> <br /> <br /> 7. Inside the directory public_mm/sourceData create a directory for workspace<br /> <br /> mkdir sourceData/09_custom<br /> <br /> Finally, Create a directory to store the knowledge sources:<br /> <br /> mkdir sourceData/09_custom/umls<br /> <br /> === Installation of Metamap ===<br /> <br /> <br /> 1. First, run the BuildDataFiles program as follows:<br /> <br /> $fullpath/public_mm/bin/BuildDataFiles<br /> <br /> 2. The file st.raw, located in $fullpath/public_mm/data/dfbuilder/2009 will need to be modified. Append the file generated in point 4 of Preparation of Data section to this file.<br /> <br /> 3. Move to $fullpath/public_mm/sourceData/09_custom/01metawordindex and execute the following scripts in order<br /> <br /> ./01CreateWorkFiles<br /> <br /> ./02Suppress<br /> <br /> ./03FilterPrep<br /> <br /> ./04FilterStrict<br /> <br /> ./05GenerateMWIFiles<br /> <br /> 4. Move to 02treecodes directory and run 01GenerateTreecodes<br /> <br /> cd ../02treecodes<br /> <br /> ./01GenerateTreecodes<br /> <br /> 5. Move to 03Variants directory and run 01GenerateVariants<br /> <br /> cd ../03Variants<br /> <br /> ./01GenerateVariants<br /> <br /> 6. cd ../04synonyms <br /> <br /> ./01GenerateSynonyms<br /> <br /> 7. cd ../05abbrAcronyms<br /> <br /> ./01GenerateAbbrAcronyms<br /> <br /> 8. move to $fullpath/public_mm and run<br /> <br /> ./bin/LoadDataFiles<br /> <br /> === Running MetaMap ===<br /> <br /> To run the newly installed MetaMap dfb, move to the main workspace folder (public_mm) and run the command below<br /> <br /> bin/SKRrun -L 2009 -M /DATA/XDR -B /BDB4 -w ./lexicon ./bin/metamap09.BINARY -Z 09_custom ./resources/input -I</div>

Tejaswit https://www.bioontology.org//mediawiki/index.php?title=MetaMap_Installation&diff=9705 2010-01-27T10:26:37Z

<p>Tejaswit: /* Collaboration &amp; Acknowledgment */</p> <hr /> <div>== Presentation ==<br /> MetaMap [http://mmtx.nlm.nih.gov/] is a highly powerful tool to map biomedical text to the UMLS [http://www.nlm.nih.gov/research/umls/] Metathesaurus or a custom Metathesaurus, equivalently, to discover Metathesaurus concepts referred to in text. The following article describes the way of using the Data File Builder utility inside MetaMap to build a concept recognizer for text containing NCBO terms.<br /> <br /> NCBO [http://www.bioontology.org/] is developing a system [http://www.bioontology.org/annotator-service] to annotate large numbers of data resources automatically, and have developed a prototype system for ontology-based annotation and indexing of biomedical data [http://www.biomedcentral.com/1471-2105/10/S9/S14]. As a part of the workflow a concept recognizer is used to recognize a given ontology concept in text. The goal of the project is to embed MetaMap into the annotator workflow. MetaMap is a very powerful for this task due to the use of complex NLP techniques in its concept recognition algorithm. <br /> <br /> Below listed is some of the useful processing done by MetaMap:<br /> <br /> 1. Removal of problematic terms from data [numbers, single alphabets etc..]<br /> <br /> 2. Lexical and Syntactic filtering to recognize long phrases accurately<br /> <br /> 3. Identifying linguistic variants of base words and terms that have been computed using a variety of generation methods. This handles various forms like inflectional (heart → hearts), derivational (treat → treatment), synonyms (pyrosis ↔ heartburn), acronyms (vt ↔ ventricular tachycardia) and spellings (apnea ↔ apnoea)<br /> <br /> Such kind of rich semantic processing can drastically increase the number of concepts that can be recognized from a text. <br /> <br /> Although Metamap was originally built to work on UMLS concepts, its Data File builder utility (DFB) lets us create our own custom dictionary for NCBO concepts.<br /> <br /> == Contacts ==<br /> <br /> * For questions or feature requests, contact Clement Jonquet [mailto:jonquet@stanford.edu?subject=MetaMap%Installation] and Tejaswi Tenneti [mailto:tejaswit@stanford.edu?subject=MetaMap%Installation]<br /> <br /> == Versions (prototypes &amp; releases) ==<br /> <br /> * December 2009 - Second Annotator prototype including MetaMap: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> * November 2009 - Second Annotator prototype including Mmtx: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> <br /> == References ==<br /> * gforge code location https://bmir-gforge.stanford.edu/gf/project/obs/<br /> <br /> == Collaboration &amp; Acknowledgment ==<br /> <br /> We thank the guys at NLM [http://www.nlm.nih.gov/] for their constant support and help at various stages of this project.<br /> <br /> == MetaMap Installation documentation ==<br /> <br /> The installation of MetaMap primarily involves 3 stages. Each of them are explained below followed by instructions to run the newly installed custom MetaMap.<br /> <br /> === Preparation of data ===<br /> <br /> There are primarily 4 files to be generated for the installation. <br /> <br /> '''1.MRCON:''' This file is primarily generated from the table OBS_TT. Below is a mapping between the columns in MRCON and OBS_TT. The columns listed in OBS_TT are used to fill their corresponding columns in MRCON file.<br /> <br /> MRCON OBS_TT<br /> <br /> CUI (Concept Unique ID) conceptID<br /> <br /> Language- <br /> Status isPreferred <br /> LUI(Lexical Unique ID) termID <br /> String type -<br /> SUI(String Unique ID) termID <br /> String termName <br /> LRL - <br /> <br /> The sql command would fill correct values into MRCON table in database. <br /> <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|ENG|P|L0027051|PF| S0064638|Myocardial Infarction|0|<br /> <br /> <br /> '''2.MRSO:''' This file is primarily generated from the tables OBS_TT, OBS_OT and OBS_CT. Below is a mapping between the columns in MRSO and the above OBS tables. The columns listed in the tables are used to fill their corresponding columns in MRSO file.<br /> <br /> MRSO OBS tables<br /> CUI (Concept Unique ID) OBS_TT.conceptID<br /> LUI(Lexical Unique ID) OBS_TT.termID<br /> SUI(String Unique ID) OBS_TT.termID <br /> SABSourceAbbrev OBS_OT.localOntologyID<br /> TermType OBS_TT.isPreferred<br /> SourceID OBS_OT.ontologyID<br /> Restrictionlevel -<br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|L0027051|S0064638| MEDLINEPLUS|ET|T5|0|<br /> <br /> <br /> '''3.MRSTY:''' This file is primarily generated from the table OBS_STT. Below is a mapping between the columns in MRSTY and OBS_STT. The columns listed in OBS_STT are used to fill their corresponding columns in MRSTY file.<br /> <br /> MRSTY OBS_STT<br /> CUI (Concept Unique ID) OBS_STT.conceptID<br /> TUI (term Unique ID) OBS_STT.localSemanticTypeID<br /> STY OBS_STT.semanticTypeName <br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|T047|Disease or Syndrome|<br /> <br /> <br /> '''4.st.raw:''' This file is primarily generated from the column semanticTypeName in the table OBS_STT. This file contains row of semantic type names and a short name for each of them. <br /> <br /> Example row:<br /> semantic Type Name|semantictypename<br /> <br /> <br /> <br /> === Setting up the workspace ===<br /> <br /> 1. '''Install the Lexical Variant Generator (LVG)''' before running<br /> MetaMap's install program. LVG is part of the Lexical Tools distribution and is available from the Lexical Systems Group (http://lexsrv3.nlm.nih.gov/SPECIALIST/Projects/lvg/current/index.html).<br /> <br /> 2. Before using MetaMap install program to install data file builder, you also need to '''add LVG's bin directory''' {LVG_DIR}/bin to your program path:<br /> <br /> export PATH=$PATH:&lt;LVG_DIR&gt;/bin<br /> <br /> 3. '''Install MetaMap''' using instructions at their site <br /> <br /> 4. Unzip the DFB installation files in the same directory as MetaMap<br /> <br /> 5. Connect to the new directory created by extracting the distribution and invoke the install program:<br /> <br /> cd &lt;distribution directory&gt;<br /> <br /> ./bin/install.sh<br /> <br /> '''A sample run of the installation script follows:'''<br /> <br /> Enter basedir of installation [/nfsvol/nlsaux15/public_mm] <br /> <br /> Basedir is set to /nfsvol/nlsaux15/public_mm.<br /> <br /> The WSD Server requires Sun's Java Runtime Environment (JRE)<br /> Sun's Java Developer Kit (JDK) will work as well. if the<br /> command: &quot;which&quot; java returns /usr/local/jre1.4.2/bin/java, then the<br /> JRE resides in /usr/local/jre1.4.2/.<br /> <br /> Where does your distribution of Sun's JRE reside?<br /> Enter home path of JRE (JDK) [/usr]: /nfsvol/nls/tools/Linux-i686/java1.4.2<br /> Using /nfsvol/nls/tools/Linux-i686/java1.4.2 for JAVA_HOME.<br /> <br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/disambServer.cfg generated<br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/log4j.properties generated<br /> /nfsvol/nlsaux15/public_mm/bin/SKRrun generated.<br /> /nfsvol/nlsaux15/public_mm/bin/metamap07 generated.<br /> /nfsvol/nlsaux15/public_mm/bin/wsdserverctl generated.<br /> /nfsvol/nlsaux15/public_mm/bin/skrmedpostctl generated.<br /> Install complete.<br /> Would like to use a custom data set with MetaMap (use data file builder)? [yN]: <br /> <br /> running Data File Builder Install...<br /> Is LVG installed? [yN] &lt;The user types y and return&gt;<br /> <br /> running Data File Builder Install...<br /> Enter home path of LVG [/nfsvol/nls/tools/Linux-i686/lvg2009]:<br /> Using /nfsvol/nls/tools/Linux-i686/lvg2009 for LVG_DIR.<br /> <br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.lvglab generated.<br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.xwords generated.<br /> Datafile Builder Setup is complete.<br /> <br /> <br /> 6. '''Make sure that the SKR/MedPOST tagger is running;''' to run the tagger, move to the public_mm directory present inside the working directory and invoke<br /> <br /> ./bin/skrmedpostctl start<br /> <br /> '''Incase there is an error''' due to a port number, change the port number in the following files and try to run the tagger again:<br /> <br /> /bin/skrmedpostctl<br /> <br /> If you change the port number skrmedpostctl uses you need to change public_mm/bin/SKRrun (or SKRrun.in and re-run install.sh) to match the port you set in skrmedpostctl. The environment variables that should be changed in SKRrun are:<br /> <br /> TAGGER_SERVER_DEFAULT_TCP_PORT<br /> <br /> TAGGER_SERVER_TCP_PORT_0<br /> <br /> TAGGER_SERVER_TCP_PORT_1<br /> <br /> <br /> 7. Inside the directory public_mm/sourceData create a directory for workspace<br /> <br /> mkdir sourceData/09_custom<br /> <br /> Finally, Create a directory to store the knowledge sources:<br /> <br /> mkdir sourceData/09_custom/umls<br /> <br /> === Installation of Metamap ===<br /> <br /> <br /> 1. First, run the BuildDataFiles program as follows:<br /> <br /> $fullpath/public_mm/bin/BuildDataFiles<br /> <br /> 2. The file st.raw, located in $fullpath/public_mm/data/dfbuilder/2009 will need to be modified. Append the file generated in point 4 of Preparation of Data section to this file.<br /> <br /> 3. Move to $fullpath/public_mm/sourceData/09_custom/01metawordindex and execute the following scripts in order<br /> <br /> ./01CreateWorkFiles<br /> <br /> ./02Suppress<br /> <br /> ./03FilterPrep<br /> <br /> ./04FilterStrict<br /> <br /> ./05GenerateMWIFiles<br /> <br /> 4. Move to 02treecodes directory and run 01GenerateTreecodes<br /> <br /> cd ../02treecodes<br /> <br /> ./01GenerateTreecodes<br /> <br /> 5. Move to 03Variants directory and run 01GenerateVariants<br /> <br /> cd ../03Variants<br /> <br /> ./01GenerateVariants<br /> <br /> 6. cd ../04synonyms <br /> <br /> ./01GenerateSynonyms<br /> <br /> 7. cd ../05abbrAcronyms<br /> <br /> ./01GenerateAbbrAcronyms<br /> <br /> 8. move to $fullpath/public_mm and run<br /> <br /> ./bin/LoadDataFiles<br /> <br /> === Running MetaMap ===<br /> <br /> To run the newly installed MetaMap dfb, move to the main workspace folder (public_mm) and run the command below<br /> <br /> bin/SKRrun -L 2009 -M /DATA/XDR -B /BDB4 -w ./lexicon ./bin/metamap09.BINARY -Z 09_custom ./resources/input -I</div>

Tejaswit https://www.bioontology.org//mediawiki/index.php?title=MetaMap_Installation&diff=9704 2010-01-27T10:25:01Z

<p>Tejaswit: /* References */</p> <hr /> <div>== Presentation ==<br /> MetaMap [http://mmtx.nlm.nih.gov/] is a highly powerful tool to map biomedical text to the UMLS [http://www.nlm.nih.gov/research/umls/] Metathesaurus or a custom Metathesaurus, equivalently, to discover Metathesaurus concepts referred to in text. The following article describes the way of using the Data File Builder utility inside MetaMap to build a concept recognizer for text containing NCBO terms.<br /> <br /> NCBO [http://www.bioontology.org/] is developing a system [http://www.bioontology.org/annotator-service] to annotate large numbers of data resources automatically, and have developed a prototype system for ontology-based annotation and indexing of biomedical data [http://www.biomedcentral.com/1471-2105/10/S9/S14]. As a part of the workflow a concept recognizer is used to recognize a given ontology concept in text. The goal of the project is to embed MetaMap into the annotator workflow. MetaMap is a very powerful for this task due to the use of complex NLP techniques in its concept recognition algorithm. <br /> <br /> Below listed is some of the useful processing done by MetaMap:<br /> <br /> 1. Removal of problematic terms from data [numbers, single alphabets etc..]<br /> <br /> 2. Lexical and Syntactic filtering to recognize long phrases accurately<br /> <br /> 3. Identifying linguistic variants of base words and terms that have been computed using a variety of generation methods. This handles various forms like inflectional (heart → hearts), derivational (treat → treatment), synonyms (pyrosis ↔ heartburn), acronyms (vt ↔ ventricular tachycardia) and spellings (apnea ↔ apnoea)<br /> <br /> Such kind of rich semantic processing can drastically increase the number of concepts that can be recognized from a text. <br /> <br /> Although Metamap was originally built to work on UMLS concepts, its Data File builder utility (DFB) lets us create our own custom dictionary for NCBO concepts.<br /> <br /> == Contacts ==<br /> <br /> * For questions or feature requests, contact Clement Jonquet [mailto:jonquet@stanford.edu?subject=MetaMap%Installation] and Tejaswi Tenneti [mailto:tejaswit@stanford.edu?subject=MetaMap%Installation]<br /> <br /> == Versions (prototypes &amp; releases) ==<br /> <br /> * December 2009 - Second Annotator prototype including MetaMap: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> * November 2009 - Second Annotator prototype including Mmtx: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> <br /> == References ==<br /> * gforge code location https://bmir-gforge.stanford.edu/gf/project/obs/<br /> <br /> == Collaboration &amp; Acknowledgment ==<br /> <br /> [a sentence to acknowledge the support of NLM]<br /> <br /> == MetaMap Installation documentation ==<br /> <br /> The installation of MetaMap primarily involves 3 stages. Each of them are explained below followed by instructions to run the newly installed custom MetaMap.<br /> <br /> === Preparation of data ===<br /> <br /> There are primarily 4 files to be generated for the installation. <br /> <br /> '''1.MRCON:''' This file is primarily generated from the table OBS_TT. Below is a mapping between the columns in MRCON and OBS_TT. The columns listed in OBS_TT are used to fill their corresponding columns in MRCON file.<br /> <br /> MRCON OBS_TT<br /> <br /> CUI (Concept Unique ID) conceptID<br /> <br /> Language- <br /> Status isPreferred <br /> LUI(Lexical Unique ID) termID <br /> String type -<br /> SUI(String Unique ID) termID <br /> String termName <br /> LRL - <br /> <br /> The sql command would fill correct values into MRCON table in database. <br /> <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|ENG|P|L0027051|PF| S0064638|Myocardial Infarction|0|<br /> <br /> <br /> '''2.MRSO:''' This file is primarily generated from the tables OBS_TT, OBS_OT and OBS_CT. Below is a mapping between the columns in MRSO and the above OBS tables. The columns listed in the tables are used to fill their corresponding columns in MRSO file.<br /> <br /> MRSO OBS tables<br /> CUI (Concept Unique ID) OBS_TT.conceptID<br /> LUI(Lexical Unique ID) OBS_TT.termID<br /> SUI(String Unique ID) OBS_TT.termID <br /> SABSourceAbbrev OBS_OT.localOntologyID<br /> TermType OBS_TT.isPreferred<br /> SourceID OBS_OT.ontologyID<br /> Restrictionlevel -<br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|L0027051|S0064638| MEDLINEPLUS|ET|T5|0|<br /> <br /> <br /> '''3.MRSTY:''' This file is primarily generated from the table OBS_STT. Below is a mapping between the columns in MRSTY and OBS_STT. The columns listed in OBS_STT are used to fill their corresponding columns in MRSTY file.<br /> <br /> MRSTY OBS_STT<br /> CUI (Concept Unique ID) OBS_STT.conceptID<br /> TUI (term Unique ID) OBS_STT.localSemanticTypeID<br /> STY OBS_STT.semanticTypeName <br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|T047|Disease or Syndrome|<br /> <br /> <br /> '''4.st.raw:''' This file is primarily generated from the column semanticTypeName in the table OBS_STT. This file contains row of semantic type names and a short name for each of them. <br /> <br /> Example row:<br /> semantic Type Name|semantictypename<br /> <br /> <br /> <br /> === Setting up the workspace ===<br /> <br /> 1. '''Install the Lexical Variant Generator (LVG)''' before running<br /> MetaMap's install program. LVG is part of the Lexical Tools distribution and is available from the Lexical Systems Group (http://lexsrv3.nlm.nih.gov/SPECIALIST/Projects/lvg/current/index.html).<br /> <br /> 2. Before using MetaMap install program to install data file builder, you also need to '''add LVG's bin directory''' {LVG_DIR}/bin to your program path:<br /> <br /> export PATH=$PATH:&lt;LVG_DIR&gt;/bin<br /> <br /> 3. '''Install MetaMap''' using instructions at their site <br /> <br /> 4. Unzip the DFB installation files in the same directory as MetaMap<br /> <br /> 5. Connect to the new directory created by extracting the distribution and invoke the install program:<br /> <br /> cd &lt;distribution directory&gt;<br /> <br /> ./bin/install.sh<br /> <br /> '''A sample run of the installation script follows:'''<br /> <br /> Enter basedir of installation [/nfsvol/nlsaux15/public_mm] <br /> <br /> Basedir is set to /nfsvol/nlsaux15/public_mm.<br /> <br /> The WSD Server requires Sun's Java Runtime Environment (JRE)<br /> Sun's Java Developer Kit (JDK) will work as well. if the<br /> command: &quot;which&quot; java returns /usr/local/jre1.4.2/bin/java, then the<br /> JRE resides in /usr/local/jre1.4.2/.<br /> <br /> Where does your distribution of Sun's JRE reside?<br /> Enter home path of JRE (JDK) [/usr]: /nfsvol/nls/tools/Linux-i686/java1.4.2<br /> Using /nfsvol/nls/tools/Linux-i686/java1.4.2 for JAVA_HOME.<br /> <br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/disambServer.cfg generated<br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/log4j.properties generated<br /> /nfsvol/nlsaux15/public_mm/bin/SKRrun generated.<br /> /nfsvol/nlsaux15/public_mm/bin/metamap07 generated.<br /> /nfsvol/nlsaux15/public_mm/bin/wsdserverctl generated.<br /> /nfsvol/nlsaux15/public_mm/bin/skrmedpostctl generated.<br /> Install complete.<br /> Would like to use a custom data set with MetaMap (use data file builder)? [yN]: <br /> <br /> running Data File Builder Install...<br /> Is LVG installed? [yN] &lt;The user types y and return&gt;<br /> <br /> running Data File Builder Install...<br /> Enter home path of LVG [/nfsvol/nls/tools/Linux-i686/lvg2009]:<br /> Using /nfsvol/nls/tools/Linux-i686/lvg2009 for LVG_DIR.<br /> <br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.lvglab generated.<br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.xwords generated.<br /> Datafile Builder Setup is complete.<br /> <br /> <br /> 6. '''Make sure that the SKR/MedPOST tagger is running;''' to run the tagger, move to the public_mm directory present inside the working directory and invoke<br /> <br /> ./bin/skrmedpostctl start<br /> <br /> '''Incase there is an error''' due to a port number, change the port number in the following files and try to run the tagger again:<br /> <br /> /bin/skrmedpostctl<br /> <br /> If you change the port number skrmedpostctl uses you need to change public_mm/bin/SKRrun (or SKRrun.in and re-run install.sh) to match the port you set in skrmedpostctl. The environment variables that should be changed in SKRrun are:<br /> <br /> TAGGER_SERVER_DEFAULT_TCP_PORT<br /> <br /> TAGGER_SERVER_TCP_PORT_0<br /> <br /> TAGGER_SERVER_TCP_PORT_1<br /> <br /> <br /> 7. Inside the directory public_mm/sourceData create a directory for workspace<br /> <br /> mkdir sourceData/09_custom<br /> <br /> Finally, Create a directory to store the knowledge sources:<br /> <br /> mkdir sourceData/09_custom/umls<br /> <br /> === Installation of Metamap ===<br /> <br /> <br /> 1. First, run the BuildDataFiles program as follows:<br /> <br /> $fullpath/public_mm/bin/BuildDataFiles<br /> <br /> 2. The file st.raw, located in $fullpath/public_mm/data/dfbuilder/2009 will need to be modified. Append the file generated in point 4 of Preparation of Data section to this file.<br /> <br /> 3. Move to $fullpath/public_mm/sourceData/09_custom/01metawordindex and execute the following scripts in order<br /> <br /> ./01CreateWorkFiles<br /> <br /> ./02Suppress<br /> <br /> ./03FilterPrep<br /> <br /> ./04FilterStrict<br /> <br /> ./05GenerateMWIFiles<br /> <br /> 4. Move to 02treecodes directory and run 01GenerateTreecodes<br /> <br /> cd ../02treecodes<br /> <br /> ./01GenerateTreecodes<br /> <br /> 5. Move to 03Variants directory and run 01GenerateVariants<br /> <br /> cd ../03Variants<br /> <br /> ./01GenerateVariants<br /> <br /> 6. cd ../04synonyms <br /> <br /> ./01GenerateSynonyms<br /> <br /> 7. cd ../05abbrAcronyms<br /> <br /> ./01GenerateAbbrAcronyms<br /> <br /> 8. move to $fullpath/public_mm and run<br /> <br /> ./bin/LoadDataFiles<br /> <br /> === Running MetaMap ===<br /> <br /> To run the newly installed MetaMap dfb, move to the main workspace folder (public_mm) and run the command below<br /> <br /> bin/SKRrun -L 2009 -M /DATA/XDR -B /BDB4 -w ./lexicon ./bin/metamap09.BINARY -Z 09_custom ./resources/input -I</div>

Tejaswit https://www.bioontology.org//mediawiki/index.php?title=MetaMap_Installation&diff=9703 2010-01-27T10:24:41Z

<p>Tejaswit: /* References */</p> <hr /> <div>== Presentation ==<br /> MetaMap [http://mmtx.nlm.nih.gov/] is a highly powerful tool to map biomedical text to the UMLS [http://www.nlm.nih.gov/research/umls/] Metathesaurus or a custom Metathesaurus, equivalently, to discover Metathesaurus concepts referred to in text. The following article describes the way of using the Data File Builder utility inside MetaMap to build a concept recognizer for text containing NCBO terms.<br /> <br /> NCBO [http://www.bioontology.org/] is developing a system [http://www.bioontology.org/annotator-service] to annotate large numbers of data resources automatically, and have developed a prototype system for ontology-based annotation and indexing of biomedical data [http://www.biomedcentral.com/1471-2105/10/S9/S14]. As a part of the workflow a concept recognizer is used to recognize a given ontology concept in text. The goal of the project is to embed MetaMap into the annotator workflow. MetaMap is a very powerful for this task due to the use of complex NLP techniques in its concept recognition algorithm. <br /> <br /> Below listed is some of the useful processing done by MetaMap:<br /> <br /> 1. Removal of problematic terms from data [numbers, single alphabets etc..]<br /> <br /> 2. Lexical and Syntactic filtering to recognize long phrases accurately<br /> <br /> 3. Identifying linguistic variants of base words and terms that have been computed using a variety of generation methods. This handles various forms like inflectional (heart → hearts), derivational (treat → treatment), synonyms (pyrosis ↔ heartburn), acronyms (vt ↔ ventricular tachycardia) and spellings (apnea ↔ apnoea)<br /> <br /> Such kind of rich semantic processing can drastically increase the number of concepts that can be recognized from a text. <br /> <br /> Although Metamap was originally built to work on UMLS concepts, its Data File builder utility (DFB) lets us create our own custom dictionary for NCBO concepts.<br /> <br /> == Contacts ==<br /> <br /> * For questions or feature requests, contact Clement Jonquet [mailto:jonquet@stanford.edu?subject=MetaMap%Installation] and Tejaswi Tenneti [mailto:tejaswit@stanford.edu?subject=MetaMap%Installation]<br /> <br /> == Versions (prototypes &amp; releases) ==<br /> <br /> * December 2009 - Second Annotator prototype including MetaMap: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> * November 2009 - Second Annotator prototype including Mmtx: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> <br /> == References ==<br /> gforge code location<br /> <br /> * https://bmir-gforge.stanford.edu/gf/project/obs/<br /> <br /> == Collaboration &amp; Acknowledgment ==<br /> <br /> [a sentence to acknowledge the support of NLM]<br /> <br /> == MetaMap Installation documentation ==<br /> <br /> The installation of MetaMap primarily involves 3 stages. Each of them are explained below followed by instructions to run the newly installed custom MetaMap.<br /> <br /> === Preparation of data ===<br /> <br /> There are primarily 4 files to be generated for the installation. <br /> <br /> '''1.MRCON:''' This file is primarily generated from the table OBS_TT. Below is a mapping between the columns in MRCON and OBS_TT. The columns listed in OBS_TT are used to fill their corresponding columns in MRCON file.<br /> <br /> MRCON OBS_TT<br /> <br /> CUI (Concept Unique ID) conceptID<br /> <br /> Language- <br /> Status isPreferred <br /> LUI(Lexical Unique ID) termID <br /> String type -<br /> SUI(String Unique ID) termID <br /> String termName <br /> LRL - <br /> <br /> The sql command would fill correct values into MRCON table in database. <br /> <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|ENG|P|L0027051|PF| S0064638|Myocardial Infarction|0|<br /> <br /> <br /> '''2.MRSO:''' This file is primarily generated from the tables OBS_TT, OBS_OT and OBS_CT. Below is a mapping between the columns in MRSO and the above OBS tables. The columns listed in the tables are used to fill their corresponding columns in MRSO file.<br /> <br /> MRSO OBS tables<br /> CUI (Concept Unique ID) OBS_TT.conceptID<br /> LUI(Lexical Unique ID) OBS_TT.termID<br /> SUI(String Unique ID) OBS_TT.termID <br /> SABSourceAbbrev OBS_OT.localOntologyID<br /> TermType OBS_TT.isPreferred<br /> SourceID OBS_OT.ontologyID<br /> Restrictionlevel -<br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|L0027051|S0064638| MEDLINEPLUS|ET|T5|0|<br /> <br /> <br /> '''3.MRSTY:''' This file is primarily generated from the table OBS_STT. Below is a mapping between the columns in MRSTY and OBS_STT. The columns listed in OBS_STT are used to fill their corresponding columns in MRSTY file.<br /> <br /> MRSTY OBS_STT<br /> CUI (Concept Unique ID) OBS_STT.conceptID<br /> TUI (term Unique ID) OBS_STT.localSemanticTypeID<br /> STY OBS_STT.semanticTypeName <br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|T047|Disease or Syndrome|<br /> <br /> <br /> '''4.st.raw:''' This file is primarily generated from the column semanticTypeName in the table OBS_STT. This file contains row of semantic type names and a short name for each of them. <br /> <br /> Example row:<br /> semantic Type Name|semantictypename<br /> <br /> <br /> <br /> === Setting up the workspace ===<br /> <br /> 1. '''Install the Lexical Variant Generator (LVG)''' before running<br /> MetaMap's install program. LVG is part of the Lexical Tools distribution and is available from the Lexical Systems Group (http://lexsrv3.nlm.nih.gov/SPECIALIST/Projects/lvg/current/index.html).<br /> <br /> 2. Before using MetaMap install program to install data file builder, you also need to '''add LVG's bin directory''' {LVG_DIR}/bin to your program path:<br /> <br /> export PATH=$PATH:&lt;LVG_DIR&gt;/bin<br /> <br /> 3. '''Install MetaMap''' using instructions at their site <br /> <br /> 4. Unzip the DFB installation files in the same directory as MetaMap<br /> <br /> 5. Connect to the new directory created by extracting the distribution and invoke the install program:<br /> <br /> cd &lt;distribution directory&gt;<br /> <br /> ./bin/install.sh<br /> <br /> '''A sample run of the installation script follows:'''<br /> <br /> Enter basedir of installation [/nfsvol/nlsaux15/public_mm] <br /> <br /> Basedir is set to /nfsvol/nlsaux15/public_mm.<br /> <br /> The WSD Server requires Sun's Java Runtime Environment (JRE)<br /> Sun's Java Developer Kit (JDK) will work as well. if the<br /> command: &quot;which&quot; java returns /usr/local/jre1.4.2/bin/java, then the<br /> JRE resides in /usr/local/jre1.4.2/.<br /> <br /> Where does your distribution of Sun's JRE reside?<br /> Enter home path of JRE (JDK) [/usr]: /nfsvol/nls/tools/Linux-i686/java1.4.2<br /> Using /nfsvol/nls/tools/Linux-i686/java1.4.2 for JAVA_HOME.<br /> <br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/disambServer.cfg generated<br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/log4j.properties generated<br /> /nfsvol/nlsaux15/public_mm/bin/SKRrun generated.<br /> /nfsvol/nlsaux15/public_mm/bin/metamap07 generated.<br /> /nfsvol/nlsaux15/public_mm/bin/wsdserverctl generated.<br /> /nfsvol/nlsaux15/public_mm/bin/skrmedpostctl generated.<br /> Install complete.<br /> Would like to use a custom data set with MetaMap (use data file builder)? [yN]: <br /> <br /> running Data File Builder Install...<br /> Is LVG installed? [yN] &lt;The user types y and return&gt;<br /> <br /> running Data File Builder Install...<br /> Enter home path of LVG [/nfsvol/nls/tools/Linux-i686/lvg2009]:<br /> Using /nfsvol/nls/tools/Linux-i686/lvg2009 for LVG_DIR.<br /> <br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.lvglab generated.<br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.xwords generated.<br /> Datafile Builder Setup is complete.<br /> <br /> <br /> 6. '''Make sure that the SKR/MedPOST tagger is running;''' to run the tagger, move to the public_mm directory present inside the working directory and invoke<br /> <br /> ./bin/skrmedpostctl start<br /> <br /> '''Incase there is an error''' due to a port number, change the port number in the following files and try to run the tagger again:<br /> <br /> /bin/skrmedpostctl<br /> <br /> If you change the port number skrmedpostctl uses you need to change public_mm/bin/SKRrun (or SKRrun.in and re-run install.sh) to match the port you set in skrmedpostctl. The environment variables that should be changed in SKRrun are:<br /> <br /> TAGGER_SERVER_DEFAULT_TCP_PORT<br /> <br /> TAGGER_SERVER_TCP_PORT_0<br /> <br /> TAGGER_SERVER_TCP_PORT_1<br /> <br /> <br /> 7. Inside the directory public_mm/sourceData create a directory for workspace<br /> <br /> mkdir sourceData/09_custom<br /> <br /> Finally, Create a directory to store the knowledge sources:<br /> <br /> mkdir sourceData/09_custom/umls<br /> <br /> === Installation of Metamap ===<br /> <br /> <br /> 1. First, run the BuildDataFiles program as follows:<br /> <br /> $fullpath/public_mm/bin/BuildDataFiles<br /> <br /> 2. The file st.raw, located in $fullpath/public_mm/data/dfbuilder/2009 will need to be modified. Append the file generated in point 4 of Preparation of Data section to this file.<br /> <br /> 3. Move to $fullpath/public_mm/sourceData/09_custom/01metawordindex and execute the following scripts in order<br /> <br /> ./01CreateWorkFiles<br /> <br /> ./02Suppress<br /> <br /> ./03FilterPrep<br /> <br /> ./04FilterStrict<br /> <br /> ./05GenerateMWIFiles<br /> <br /> 4. Move to 02treecodes directory and run 01GenerateTreecodes<br /> <br /> cd ../02treecodes<br /> <br /> ./01GenerateTreecodes<br /> <br /> 5. Move to 03Variants directory and run 01GenerateVariants<br /> <br /> cd ../03Variants<br /> <br /> ./01GenerateVariants<br /> <br /> 6. cd ../04synonyms <br /> <br /> ./01GenerateSynonyms<br /> <br /> 7. cd ../05abbrAcronyms<br /> <br /> ./01GenerateAbbrAcronyms<br /> <br /> 8. move to $fullpath/public_mm and run<br /> <br /> ./bin/LoadDataFiles<br /> <br /> === Running MetaMap ===<br /> <br /> To run the newly installed MetaMap dfb, move to the main workspace folder (public_mm) and run the command below<br /> <br /> bin/SKRrun -L 2009 -M /DATA/XDR -B /BDB4 -w ./lexicon ./bin/metamap09.BINARY -Z 09_custom ./resources/input -I</div>

Tejaswit https://www.bioontology.org//mediawiki/index.php?title=MetaMap_Installation&diff=9702 2010-01-27T10:23:53Z

<p>Tejaswit: /* References */</p> <hr /> <div>== Presentation ==<br /> MetaMap [http://mmtx.nlm.nih.gov/] is a highly powerful tool to map biomedical text to the UMLS [http://www.nlm.nih.gov/research/umls/] Metathesaurus or a custom Metathesaurus, equivalently, to discover Metathesaurus concepts referred to in text. The following article describes the way of using the Data File Builder utility inside MetaMap to build a concept recognizer for text containing NCBO terms.<br /> <br /> NCBO [http://www.bioontology.org/] is developing a system [http://www.bioontology.org/annotator-service] to annotate large numbers of data resources automatically, and have developed a prototype system for ontology-based annotation and indexing of biomedical data [http://www.biomedcentral.com/1471-2105/10/S9/S14]. As a part of the workflow a concept recognizer is used to recognize a given ontology concept in text. The goal of the project is to embed MetaMap into the annotator workflow. MetaMap is a very powerful for this task due to the use of complex NLP techniques in its concept recognition algorithm. <br /> <br /> Below listed is some of the useful processing done by MetaMap:<br /> <br /> 1. Removal of problematic terms from data [numbers, single alphabets etc..]<br /> <br /> 2. Lexical and Syntactic filtering to recognize long phrases accurately<br /> <br /> 3. Identifying linguistic variants of base words and terms that have been computed using a variety of generation methods. This handles various forms like inflectional (heart → hearts), derivational (treat → treatment), synonyms (pyrosis ↔ heartburn), acronyms (vt ↔ ventricular tachycardia) and spellings (apnea ↔ apnoea)<br /> <br /> Such kind of rich semantic processing can drastically increase the number of concepts that can be recognized from a text. <br /> <br /> Although Metamap was originally built to work on UMLS concepts, its Data File builder utility (DFB) lets us create our own custom dictionary for NCBO concepts.<br /> <br /> == Contacts ==<br /> <br /> * For questions or feature requests, contact Clement Jonquet [mailto:jonquet@stanford.edu?subject=MetaMap%Installation] and Tejaswi Tenneti [mailto:tejaswit@stanford.edu?subject=MetaMap%Installation]<br /> <br /> == Versions (prototypes &amp; releases) ==<br /> <br /> * December 2009 - Second Annotator prototype including MetaMap: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> * November 2009 - Second Annotator prototype including Mmtx: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> <br /> == References ==<br /> <br /> * https://bmir-gforge.stanford.edu/gf/project/obs/<br /> <br /> == Collaboration &amp; Acknowledgment ==<br /> <br /> [a sentence to acknowledge the support of NLM]<br /> <br /> == MetaMap Installation documentation ==<br /> <br /> The installation of MetaMap primarily involves 3 stages. Each of them are explained below followed by instructions to run the newly installed custom MetaMap.<br /> <br /> === Preparation of data ===<br /> <br /> There are primarily 4 files to be generated for the installation. <br /> <br /> '''1.MRCON:''' This file is primarily generated from the table OBS_TT. Below is a mapping between the columns in MRCON and OBS_TT. The columns listed in OBS_TT are used to fill their corresponding columns in MRCON file.<br /> <br /> MRCON OBS_TT<br /> <br /> CUI (Concept Unique ID) conceptID<br /> <br /> Language- <br /> Status isPreferred <br /> LUI(Lexical Unique ID) termID <br /> String type -<br /> SUI(String Unique ID) termID <br /> String termName <br /> LRL - <br /> <br /> The sql command would fill correct values into MRCON table in database. <br /> <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|ENG|P|L0027051|PF| S0064638|Myocardial Infarction|0|<br /> <br /> <br /> '''2.MRSO:''' This file is primarily generated from the tables OBS_TT, OBS_OT and OBS_CT. Below is a mapping between the columns in MRSO and the above OBS tables. The columns listed in the tables are used to fill their corresponding columns in MRSO file.<br /> <br /> MRSO OBS tables<br /> CUI (Concept Unique ID) OBS_TT.conceptID<br /> LUI(Lexical Unique ID) OBS_TT.termID<br /> SUI(String Unique ID) OBS_TT.termID <br /> SABSourceAbbrev OBS_OT.localOntologyID<br /> TermType OBS_TT.isPreferred<br /> SourceID OBS_OT.ontologyID<br /> Restrictionlevel -<br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|L0027051|S0064638| MEDLINEPLUS|ET|T5|0|<br /> <br /> <br /> '''3.MRSTY:''' This file is primarily generated from the table OBS_STT. Below is a mapping between the columns in MRSTY and OBS_STT. The columns listed in OBS_STT are used to fill their corresponding columns in MRSTY file.<br /> <br /> MRSTY OBS_STT<br /> CUI (Concept Unique ID) OBS_STT.conceptID<br /> TUI (term Unique ID) OBS_STT.localSemanticTypeID<br /> STY OBS_STT.semanticTypeName <br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|T047|Disease or Syndrome|<br /> <br /> <br /> '''4.st.raw:''' This file is primarily generated from the column semanticTypeName in the table OBS_STT. This file contains row of semantic type names and a short name for each of them. <br /> <br /> Example row:<br /> semantic Type Name|semantictypename<br /> <br /> <br /> <br /> === Setting up the workspace ===<br /> <br /> 1. '''Install the Lexical Variant Generator (LVG)''' before running<br /> MetaMap's install program. LVG is part of the Lexical Tools distribution and is available from the Lexical Systems Group (http://lexsrv3.nlm.nih.gov/SPECIALIST/Projects/lvg/current/index.html).<br /> <br /> 2. Before using MetaMap install program to install data file builder, you also need to '''add LVG's bin directory''' {LVG_DIR}/bin to your program path:<br /> <br /> export PATH=$PATH:&lt;LVG_DIR&gt;/bin<br /> <br /> 3. '''Install MetaMap''' using instructions at their site <br /> <br /> 4. Unzip the DFB installation files in the same directory as MetaMap<br /> <br /> 5. Connect to the new directory created by extracting the distribution and invoke the install program:<br /> <br /> cd &lt;distribution directory&gt;<br /> <br /> ./bin/install.sh<br /> <br /> '''A sample run of the installation script follows:'''<br /> <br /> Enter basedir of installation [/nfsvol/nlsaux15/public_mm] <br /> <br /> Basedir is set to /nfsvol/nlsaux15/public_mm.<br /> <br /> The WSD Server requires Sun's Java Runtime Environment (JRE)<br /> Sun's Java Developer Kit (JDK) will work as well. if the<br /> command: &quot;which&quot; java returns /usr/local/jre1.4.2/bin/java, then the<br /> JRE resides in /usr/local/jre1.4.2/.<br /> <br /> Where does your distribution of Sun's JRE reside?<br /> Enter home path of JRE (JDK) [/usr]: /nfsvol/nls/tools/Linux-i686/java1.4.2<br /> Using /nfsvol/nls/tools/Linux-i686/java1.4.2 for JAVA_HOME.<br /> <br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/disambServer.cfg generated<br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/log4j.properties generated<br /> /nfsvol/nlsaux15/public_mm/bin/SKRrun generated.<br /> /nfsvol/nlsaux15/public_mm/bin/metamap07 generated.<br /> /nfsvol/nlsaux15/public_mm/bin/wsdserverctl generated.<br /> /nfsvol/nlsaux15/public_mm/bin/skrmedpostctl generated.<br /> Install complete.<br /> Would like to use a custom data set with MetaMap (use data file builder)? [yN]: <br /> <br /> running Data File Builder Install...<br /> Is LVG installed? [yN] &lt;The user types y and return&gt;<br /> <br /> running Data File Builder Install...<br /> Enter home path of LVG [/nfsvol/nls/tools/Linux-i686/lvg2009]:<br /> Using /nfsvol/nls/tools/Linux-i686/lvg2009 for LVG_DIR.<br /> <br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.lvglab generated.<br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.xwords generated.<br /> Datafile Builder Setup is complete.<br /> <br /> <br /> 6. '''Make sure that the SKR/MedPOST tagger is running;''' to run the tagger, move to the public_mm directory present inside the working directory and invoke<br /> <br /> ./bin/skrmedpostctl start<br /> <br /> '''Incase there is an error''' due to a port number, change the port number in the following files and try to run the tagger again:<br /> <br /> /bin/skrmedpostctl<br /> <br /> If you change the port number skrmedpostctl uses you need to change public_mm/bin/SKRrun (or SKRrun.in and re-run install.sh) to match the port you set in skrmedpostctl. The environment variables that should be changed in SKRrun are:<br /> <br /> TAGGER_SERVER_DEFAULT_TCP_PORT<br /> <br /> TAGGER_SERVER_TCP_PORT_0<br /> <br /> TAGGER_SERVER_TCP_PORT_1<br /> <br /> <br /> 7. Inside the directory public_mm/sourceData create a directory for workspace<br /> <br /> mkdir sourceData/09_custom<br /> <br /> Finally, Create a directory to store the knowledge sources:<br /> <br /> mkdir sourceData/09_custom/umls<br /> <br /> === Installation of Metamap ===<br /> <br /> <br /> 1. First, run the BuildDataFiles program as follows:<br /> <br /> $fullpath/public_mm/bin/BuildDataFiles<br /> <br /> 2. The file st.raw, located in $fullpath/public_mm/data/dfbuilder/2009 will need to be modified. Append the file generated in point 4 of Preparation of Data section to this file.<br /> <br /> 3. Move to $fullpath/public_mm/sourceData/09_custom/01metawordindex and execute the following scripts in order<br /> <br /> ./01CreateWorkFiles<br /> <br /> ./02Suppress<br /> <br /> ./03FilterPrep<br /> <br /> ./04FilterStrict<br /> <br /> ./05GenerateMWIFiles<br /> <br /> 4. Move to 02treecodes directory and run 01GenerateTreecodes<br /> <br /> cd ../02treecodes<br /> <br /> ./01GenerateTreecodes<br /> <br /> 5. Move to 03Variants directory and run 01GenerateVariants<br /> <br /> cd ../03Variants<br /> <br /> ./01GenerateVariants<br /> <br /> 6. cd ../04synonyms <br /> <br /> ./01GenerateSynonyms<br /> <br /> 7. cd ../05abbrAcronyms<br /> <br /> ./01GenerateAbbrAcronyms<br /> <br /> 8. move to $fullpath/public_mm and run<br /> <br /> ./bin/LoadDataFiles<br /> <br /> === Running MetaMap ===<br /> <br /> To run the newly installed MetaMap dfb, move to the main workspace folder (public_mm) and run the command below<br /> <br /> bin/SKRrun -L 2009 -M /DATA/XDR -B /BDB4 -w ./lexicon ./bin/metamap09.BINARY -Z 09_custom ./resources/input -I</div>

Tejaswit https://www.bioontology.org//mediawiki/index.php?title=MetaMap_Installation&diff=9701 2010-01-27T10:23:39Z

<p>Tejaswit: /* References */</p> <hr /> <div>== Presentation ==<br /> MetaMap [http://mmtx.nlm.nih.gov/] is a highly powerful tool to map biomedical text to the UMLS [http://www.nlm.nih.gov/research/umls/] Metathesaurus or a custom Metathesaurus, equivalently, to discover Metathesaurus concepts referred to in text. The following article describes the way of using the Data File Builder utility inside MetaMap to build a concept recognizer for text containing NCBO terms.<br /> <br /> NCBO [http://www.bioontology.org/] is developing a system [http://www.bioontology.org/annotator-service] to annotate large numbers of data resources automatically, and have developed a prototype system for ontology-based annotation and indexing of biomedical data [http://www.biomedcentral.com/1471-2105/10/S9/S14]. As a part of the workflow a concept recognizer is used to recognize a given ontology concept in text. The goal of the project is to embed MetaMap into the annotator workflow. MetaMap is a very powerful for this task due to the use of complex NLP techniques in its concept recognition algorithm. <br /> <br /> Below listed is some of the useful processing done by MetaMap:<br /> <br /> 1. Removal of problematic terms from data [numbers, single alphabets etc..]<br /> <br /> 2. Lexical and Syntactic filtering to recognize long phrases accurately<br /> <br /> 3. Identifying linguistic variants of base words and terms that have been computed using a variety of generation methods. This handles various forms like inflectional (heart → hearts), derivational (treat → treatment), synonyms (pyrosis ↔ heartburn), acronyms (vt ↔ ventricular tachycardia) and spellings (apnea ↔ apnoea)<br /> <br /> Such kind of rich semantic processing can drastically increase the number of concepts that can be recognized from a text. <br /> <br /> Although Metamap was originally built to work on UMLS concepts, its Data File builder utility (DFB) lets us create our own custom dictionary for NCBO concepts.<br /> <br /> == Contacts ==<br /> <br /> * For questions or feature requests, contact Clement Jonquet [mailto:jonquet@stanford.edu?subject=MetaMap%Installation] and Tejaswi Tenneti [mailto:tejaswit@stanford.edu?subject=MetaMap%Installation]<br /> <br /> == Versions (prototypes &amp; releases) ==<br /> <br /> * December 2009 - Second Annotator prototype including MetaMap: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> * November 2009 - Second Annotator prototype including Mmtx: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> <br /> == References ==<br /> <br /> * [https://bmir-gforge.stanford.edu/gf/project/obs/]<br /> <br /> == Collaboration &amp; Acknowledgment ==<br /> <br /> [a sentence to acknowledge the support of NLM]<br /> <br /> == MetaMap Installation documentation ==<br /> <br /> The installation of MetaMap primarily involves 3 stages. Each of them are explained below followed by instructions to run the newly installed custom MetaMap.<br /> <br /> === Preparation of data ===<br /> <br /> There are primarily 4 files to be generated for the installation. <br /> <br /> '''1.MRCON:''' This file is primarily generated from the table OBS_TT. Below is a mapping between the columns in MRCON and OBS_TT. The columns listed in OBS_TT are used to fill their corresponding columns in MRCON file.<br /> <br /> MRCON OBS_TT<br /> <br /> CUI (Concept Unique ID) conceptID<br /> <br /> Language- <br /> Status isPreferred <br /> LUI(Lexical Unique ID) termID <br /> String type -<br /> SUI(String Unique ID) termID <br /> String termName <br /> LRL - <br /> <br /> The sql command would fill correct values into MRCON table in database. <br /> <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|ENG|P|L0027051|PF| S0064638|Myocardial Infarction|0|<br /> <br /> <br /> '''2.MRSO:''' This file is primarily generated from the tables OBS_TT, OBS_OT and OBS_CT. Below is a mapping between the columns in MRSO and the above OBS tables. The columns listed in the tables are used to fill their corresponding columns in MRSO file.<br /> <br /> MRSO OBS tables<br /> CUI (Concept Unique ID) OBS_TT.conceptID<br /> LUI(Lexical Unique ID) OBS_TT.termID<br /> SUI(String Unique ID) OBS_TT.termID <br /> SABSourceAbbrev OBS_OT.localOntologyID<br /> TermType OBS_TT.isPreferred<br /> SourceID OBS_OT.ontologyID<br /> Restrictionlevel -<br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|L0027051|S0064638| MEDLINEPLUS|ET|T5|0|<br /> <br /> <br /> '''3.MRSTY:''' This file is primarily generated from the table OBS_STT. Below is a mapping between the columns in MRSTY and OBS_STT. The columns listed in OBS_STT are used to fill their corresponding columns in MRSTY file.<br /> <br /> MRSTY OBS_STT<br /> CUI (Concept Unique ID) OBS_STT.conceptID<br /> TUI (term Unique ID) OBS_STT.localSemanticTypeID<br /> STY OBS_STT.semanticTypeName <br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|T047|Disease or Syndrome|<br /> <br /> <br /> '''4.st.raw:''' This file is primarily generated from the column semanticTypeName in the table OBS_STT. This file contains row of semantic type names and a short name for each of them. <br /> <br /> Example row:<br /> semantic Type Name|semantictypename<br /> <br /> <br /> <br /> === Setting up the workspace ===<br /> <br /> 1. '''Install the Lexical Variant Generator (LVG)''' before running<br /> MetaMap's install program. LVG is part of the Lexical Tools distribution and is available from the Lexical Systems Group (http://lexsrv3.nlm.nih.gov/SPECIALIST/Projects/lvg/current/index.html).<br /> <br /> 2. Before using MetaMap install program to install data file builder, you also need to '''add LVG's bin directory''' {LVG_DIR}/bin to your program path:<br /> <br /> export PATH=$PATH:&lt;LVG_DIR&gt;/bin<br /> <br /> 3. '''Install MetaMap''' using instructions at their site <br /> <br /> 4. Unzip the DFB installation files in the same directory as MetaMap<br /> <br /> 5. Connect to the new directory created by extracting the distribution and invoke the install program:<br /> <br /> cd &lt;distribution directory&gt;<br /> <br /> ./bin/install.sh<br /> <br /> '''A sample run of the installation script follows:'''<br /> <br /> Enter basedir of installation [/nfsvol/nlsaux15/public_mm] <br /> <br /> Basedir is set to /nfsvol/nlsaux15/public_mm.<br /> <br /> The WSD Server requires Sun's Java Runtime Environment (JRE)<br /> Sun's Java Developer Kit (JDK) will work as well. if the<br /> command: &quot;which&quot; java returns /usr/local/jre1.4.2/bin/java, then the<br /> JRE resides in /usr/local/jre1.4.2/.<br /> <br /> Where does your distribution of Sun's JRE reside?<br /> Enter home path of JRE (JDK) [/usr]: /nfsvol/nls/tools/Linux-i686/java1.4.2<br /> Using /nfsvol/nls/tools/Linux-i686/java1.4.2 for JAVA_HOME.<br /> <br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/disambServer.cfg generated<br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/log4j.properties generated<br /> /nfsvol/nlsaux15/public_mm/bin/SKRrun generated.<br /> /nfsvol/nlsaux15/public_mm/bin/metamap07 generated.<br /> /nfsvol/nlsaux15/public_mm/bin/wsdserverctl generated.<br /> /nfsvol/nlsaux15/public_mm/bin/skrmedpostctl generated.<br /> Install complete.<br /> Would like to use a custom data set with MetaMap (use data file builder)? [yN]: <br /> <br /> running Data File Builder Install...<br /> Is LVG installed? [yN] &lt;The user types y and return&gt;<br /> <br /> running Data File Builder Install...<br /> Enter home path of LVG [/nfsvol/nls/tools/Linux-i686/lvg2009]:<br /> Using /nfsvol/nls/tools/Linux-i686/lvg2009 for LVG_DIR.<br /> <br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.lvglab generated.<br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.xwords generated.<br /> Datafile Builder Setup is complete.<br /> <br /> <br /> 6. '''Make sure that the SKR/MedPOST tagger is running;''' to run the tagger, move to the public_mm directory present inside the working directory and invoke<br /> <br /> ./bin/skrmedpostctl start<br /> <br /> '''Incase there is an error''' due to a port number, change the port number in the following files and try to run the tagger again:<br /> <br /> /bin/skrmedpostctl<br /> <br /> If you change the port number skrmedpostctl uses you need to change public_mm/bin/SKRrun (or SKRrun.in and re-run install.sh) to match the port you set in skrmedpostctl. The environment variables that should be changed in SKRrun are:<br /> <br /> TAGGER_SERVER_DEFAULT_TCP_PORT<br /> <br /> TAGGER_SERVER_TCP_PORT_0<br /> <br /> TAGGER_SERVER_TCP_PORT_1<br /> <br /> <br /> 7. Inside the directory public_mm/sourceData create a directory for workspace<br /> <br /> mkdir sourceData/09_custom<br /> <br /> Finally, Create a directory to store the knowledge sources:<br /> <br /> mkdir sourceData/09_custom/umls<br /> <br /> === Installation of Metamap ===<br /> <br /> <br /> 1. First, run the BuildDataFiles program as follows:<br /> <br /> $fullpath/public_mm/bin/BuildDataFiles<br /> <br /> 2. The file st.raw, located in $fullpath/public_mm/data/dfbuilder/2009 will need to be modified. Append the file generated in point 4 of Preparation of Data section to this file.<br /> <br /> 3. Move to $fullpath/public_mm/sourceData/09_custom/01metawordindex and execute the following scripts in order<br /> <br /> ./01CreateWorkFiles<br /> <br /> ./02Suppress<br /> <br /> ./03FilterPrep<br /> <br /> ./04FilterStrict<br /> <br /> ./05GenerateMWIFiles<br /> <br /> 4. Move to 02treecodes directory and run 01GenerateTreecodes<br /> <br /> cd ../02treecodes<br /> <br /> ./01GenerateTreecodes<br /> <br /> 5. Move to 03Variants directory and run 01GenerateVariants<br /> <br /> cd ../03Variants<br /> <br /> ./01GenerateVariants<br /> <br /> 6. cd ../04synonyms <br /> <br /> ./01GenerateSynonyms<br /> <br /> 7. cd ../05abbrAcronyms<br /> <br /> ./01GenerateAbbrAcronyms<br /> <br /> 8. move to $fullpath/public_mm and run<br /> <br /> ./bin/LoadDataFiles<br /> <br /> === Running MetaMap ===<br /> <br /> To run the newly installed MetaMap dfb, move to the main workspace folder (public_mm) and run the command below<br /> <br /> bin/SKRrun -L 2009 -M /DATA/XDR -B /BDB4 -w ./lexicon ./bin/metamap09.BINARY -Z 09_custom ./resources/input -I</div>

Tejaswit https://www.bioontology.org//mediawiki/index.php?title=MetaMap_Installation&diff=9700 2010-01-27T10:16:59Z

<p>Tejaswit: /* Presentation */</p> <hr /> <div>== Presentation ==<br /> MetaMap [http://mmtx.nlm.nih.gov/] is a highly powerful tool to map biomedical text to the UMLS [http://www.nlm.nih.gov/research/umls/] Metathesaurus or a custom Metathesaurus, equivalently, to discover Metathesaurus concepts referred to in text. The following article describes the way of using the Data File Builder utility inside MetaMap to build a concept recognizer for text containing NCBO terms.<br /> <br /> NCBO [http://www.bioontology.org/] is developing a system [http://www.bioontology.org/annotator-service] to annotate large numbers of data resources automatically, and have developed a prototype system for ontology-based annotation and indexing of biomedical data [http://www.biomedcentral.com/1471-2105/10/S9/S14]. As a part of the workflow a concept recognizer is used to recognize a given ontology concept in text. The goal of the project is to embed MetaMap into the annotator workflow. MetaMap is a very powerful for this task due to the use of complex NLP techniques in its concept recognition algorithm. <br /> <br /> Below listed is some of the useful processing done by MetaMap:<br /> <br /> 1. Removal of problematic terms from data [numbers, single alphabets etc..]<br /> <br /> 2. Lexical and Syntactic filtering to recognize long phrases accurately<br /> <br /> 3. Identifying linguistic variants of base words and terms that have been computed using a variety of generation methods. This handles various forms like inflectional (heart → hearts), derivational (treat → treatment), synonyms (pyrosis ↔ heartburn), acronyms (vt ↔ ventricular tachycardia) and spellings (apnea ↔ apnoea)<br /> <br /> Such kind of rich semantic processing can drastically increase the number of concepts that can be recognized from a text. <br /> <br /> Although Metamap was originally built to work on UMLS concepts, its Data File builder utility (DFB) lets us create our own custom dictionary for NCBO concepts.<br /> <br /> == Contacts ==<br /> <br /> * For questions or feature requests, contact Clement Jonquet [mailto:jonquet@stanford.edu?subject=MetaMap%Installation] and Tejaswi Tenneti [mailto:tejaswit@stanford.edu?subject=MetaMap%Installation]<br /> <br /> == Versions (prototypes &amp; releases) ==<br /> <br /> * December 2009 - Second Annotator prototype including MetaMap: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> * November 2009 - Second Annotator prototype including Mmtx: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> <br /> == References ==<br /> <br /> * [Gforge code location]<br /> <br /> == Collaboration &amp; Acknowledgment ==<br /> <br /> [a sentence to acknowledge the support of NLM]<br /> <br /> == MetaMap Installation documentation ==<br /> <br /> The installation of MetaMap primarily involves 3 stages. Each of them are explained below followed by instructions to run the newly installed custom MetaMap.<br /> <br /> === Preparation of data ===<br /> <br /> There are primarily 4 files to be generated for the installation. <br /> <br /> '''1.MRCON:''' This file is primarily generated from the table OBS_TT. Below is a mapping between the columns in MRCON and OBS_TT. The columns listed in OBS_TT are used to fill their corresponding columns in MRCON file.<br /> <br /> MRCON OBS_TT<br /> <br /> CUI (Concept Unique ID) conceptID<br /> <br /> Language- <br /> Status isPreferred <br /> LUI(Lexical Unique ID) termID <br /> String type -<br /> SUI(String Unique ID) termID <br /> String termName <br /> LRL - <br /> <br /> The sql command would fill correct values into MRCON table in database. <br /> <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|ENG|P|L0027051|PF| S0064638|Myocardial Infarction|0|<br /> <br /> <br /> '''2.MRSO:''' This file is primarily generated from the tables OBS_TT, OBS_OT and OBS_CT. Below is a mapping between the columns in MRSO and the above OBS tables. The columns listed in the tables are used to fill their corresponding columns in MRSO file.<br /> <br /> MRSO OBS tables<br /> CUI (Concept Unique ID) OBS_TT.conceptID<br /> LUI(Lexical Unique ID) OBS_TT.termID<br /> SUI(String Unique ID) OBS_TT.termID <br /> SABSourceAbbrev OBS_OT.localOntologyID<br /> TermType OBS_TT.isPreferred<br /> SourceID OBS_OT.ontologyID<br /> Restrictionlevel -<br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|L0027051|S0064638| MEDLINEPLUS|ET|T5|0|<br /> <br /> <br /> '''3.MRSTY:''' This file is primarily generated from the table OBS_STT. Below is a mapping between the columns in MRSTY and OBS_STT. The columns listed in OBS_STT are used to fill their corresponding columns in MRSTY file.<br /> <br /> MRSTY OBS_STT<br /> CUI (Concept Unique ID) OBS_STT.conceptID<br /> TUI (term Unique ID) OBS_STT.localSemanticTypeID<br /> STY OBS_STT.semanticTypeName <br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|T047|Disease or Syndrome|<br /> <br /> <br /> '''4.st.raw:''' This file is primarily generated from the column semanticTypeName in the table OBS_STT. This file contains row of semantic type names and a short name for each of them. <br /> <br /> Example row:<br /> semantic Type Name|semantictypename<br /> <br /> <br /> <br /> === Setting up the workspace ===<br /> <br /> 1. '''Install the Lexical Variant Generator (LVG)''' before running<br /> MetaMap's install program. LVG is part of the Lexical Tools distribution and is available from the Lexical Systems Group (http://lexsrv3.nlm.nih.gov/SPECIALIST/Projects/lvg/current/index.html).<br /> <br /> 2. Before using MetaMap install program to install data file builder, you also need to '''add LVG's bin directory''' {LVG_DIR}/bin to your program path:<br /> <br /> export PATH=$PATH:&lt;LVG_DIR&gt;/bin<br /> <br /> 3. '''Install MetaMap''' using instructions at their site <br /> <br /> 4. Unzip the DFB installation files in the same directory as MetaMap<br /> <br /> 5. Connect to the new directory created by extracting the distribution and invoke the install program:<br /> <br /> cd &lt;distribution directory&gt;<br /> <br /> ./bin/install.sh<br /> <br /> '''A sample run of the installation script follows:'''<br /> <br /> Enter basedir of installation [/nfsvol/nlsaux15/public_mm] <br /> <br /> Basedir is set to /nfsvol/nlsaux15/public_mm.<br /> <br /> The WSD Server requires Sun's Java Runtime Environment (JRE)<br /> Sun's Java Developer Kit (JDK) will work as well. if the<br /> command: &quot;which&quot; java returns /usr/local/jre1.4.2/bin/java, then the<br /> JRE resides in /usr/local/jre1.4.2/.<br /> <br /> Where does your distribution of Sun's JRE reside?<br /> Enter home path of JRE (JDK) [/usr]: /nfsvol/nls/tools/Linux-i686/java1.4.2<br /> Using /nfsvol/nls/tools/Linux-i686/java1.4.2 for JAVA_HOME.<br /> <br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/disambServer.cfg generated<br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/log4j.properties generated<br /> /nfsvol/nlsaux15/public_mm/bin/SKRrun generated.<br /> /nfsvol/nlsaux15/public_mm/bin/metamap07 generated.<br /> /nfsvol/nlsaux15/public_mm/bin/wsdserverctl generated.<br /> /nfsvol/nlsaux15/public_mm/bin/skrmedpostctl generated.<br /> Install complete.<br /> Would like to use a custom data set with MetaMap (use data file builder)? [yN]: <br /> <br /> running Data File Builder Install...<br /> Is LVG installed? [yN] &lt;The user types y and return&gt;<br /> <br /> running Data File Builder Install...<br /> Enter home path of LVG [/nfsvol/nls/tools/Linux-i686/lvg2009]:<br /> Using /nfsvol/nls/tools/Linux-i686/lvg2009 for LVG_DIR.<br /> <br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.lvglab generated.<br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.xwords generated.<br /> Datafile Builder Setup is complete.<br /> <br /> <br /> 6. '''Make sure that the SKR/MedPOST tagger is running;''' to run the tagger, move to the public_mm directory present inside the working directory and invoke<br /> <br /> ./bin/skrmedpostctl start<br /> <br /> '''Incase there is an error''' due to a port number, change the port number in the following files and try to run the tagger again:<br /> <br /> /bin/skrmedpostctl<br /> <br /> If you change the port number skrmedpostctl uses you need to change public_mm/bin/SKRrun (or SKRrun.in and re-run install.sh) to match the port you set in skrmedpostctl. The environment variables that should be changed in SKRrun are:<br /> <br /> TAGGER_SERVER_DEFAULT_TCP_PORT<br /> <br /> TAGGER_SERVER_TCP_PORT_0<br /> <br /> TAGGER_SERVER_TCP_PORT_1<br /> <br /> <br /> 7. Inside the directory public_mm/sourceData create a directory for workspace<br /> <br /> mkdir sourceData/09_custom<br /> <br /> Finally, Create a directory to store the knowledge sources:<br /> <br /> mkdir sourceData/09_custom/umls<br /> <br /> === Installation of Metamap ===<br /> <br /> <br /> 1. First, run the BuildDataFiles program as follows:<br /> <br /> $fullpath/public_mm/bin/BuildDataFiles<br /> <br /> 2. The file st.raw, located in $fullpath/public_mm/data/dfbuilder/2009 will need to be modified. Append the file generated in point 4 of Preparation of Data section to this file.<br /> <br /> 3. Move to $fullpath/public_mm/sourceData/09_custom/01metawordindex and execute the following scripts in order<br /> <br /> ./01CreateWorkFiles<br /> <br /> ./02Suppress<br /> <br /> ./03FilterPrep<br /> <br /> ./04FilterStrict<br /> <br /> ./05GenerateMWIFiles<br /> <br /> 4. Move to 02treecodes directory and run 01GenerateTreecodes<br /> <br /> cd ../02treecodes<br /> <br /> ./01GenerateTreecodes<br /> <br /> 5. Move to 03Variants directory and run 01GenerateVariants<br /> <br /> cd ../03Variants<br /> <br /> ./01GenerateVariants<br /> <br /> 6. cd ../04synonyms <br /> <br /> ./01GenerateSynonyms<br /> <br /> 7. cd ../05abbrAcronyms<br /> <br /> ./01GenerateAbbrAcronyms<br /> <br /> 8. move to $fullpath/public_mm and run<br /> <br /> ./bin/LoadDataFiles<br /> <br /> === Running MetaMap ===<br /> <br /> To run the newly installed MetaMap dfb, move to the main workspace folder (public_mm) and run the command below<br /> <br /> bin/SKRrun -L 2009 -M /DATA/XDR -B /BDB4 -w ./lexicon ./bin/metamap09.BINARY -Z 09_custom ./resources/input -I</div>

Tejaswit https://www.bioontology.org//mediawiki/index.php?title=MetaMap_Installation&diff=9699 2010-01-27T10:15:53Z

<p>Tejaswit: /* Presentation */</p> <hr /> <div>== Presentation ==<br /> MetaMap [http://mmtx.nlm.nih.gov/] is a highly powerful tool to map biomedical text to the UMLS [http://www.nlm.nih.gov/research/umls/] Metathesaurus or a custom Metathesaurus, equivalently, to discover Metathesaurus concepts referred to in text. The following article describes the way of using the Data File Builder utility inside MetaMap to build a concept recognizer for text containing NCBO terms.<br /> <br /> NCBO [http://www.bioontology.org/] is developing a system [http://www.bioontology.org/annotator-service] to annotate large numbers of data resources automatically, and have developed a prototype system for ontology-based annotation and indexing of biomedical data [http://www.biomedcentral.com/1471-2105/10/S9/S14]. As a part of the workflow a concept recognizer is used to recognize a given ontology concept in text. The goal of the project is to embed MetaMap into the annotator workflow. MetaMap is a very powerful for this task due to the use of complex NLP techniques in its concept recognition algorithm. Below listed is some of the useful processing done by MetaMap:<br /> <br /> 1. Removal of problematic terms from data [numbers, single alphabets etc..]<br /> <br /> 2. Lexical and Syntactic filtering to recognize long phrases accurately<br /> <br /> 3. Identifying linguistic variants of base words and terms that have been computed using a variety of generation methods. This handles various forms like inflectional (heart → hearts), derivational (treat → treatment), synonyms (pyrosis ↔ heartburn), acronyms (vt ↔ ventricular tachycardia) and spellings (apnea ↔ apnoea)<br /> <br /> Such kind of rich semantic processing can drastically increase the number of concepts that can be recognized from a text. <br /> <br /> Although Metamap was originally built to work on UMLS concepts, its Data File builder utility (DFB) lets us create our own custom dictionary for NCBO concepts.<br /> <br /> == Contacts ==<br /> <br /> * For questions or feature requests, contact Clement Jonquet [mailto:jonquet@stanford.edu?subject=MetaMap%Installation] and Tejaswi Tenneti [mailto:tejaswit@stanford.edu?subject=MetaMap%Installation]<br /> <br /> == Versions (prototypes &amp; releases) ==<br /> <br /> * December 2009 - Second Annotator prototype including MetaMap: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> * November 2009 - Second Annotator prototype including Mmtx: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> <br /> == References ==<br /> <br /> * [Gforge code location]<br /> <br /> == Collaboration &amp; Acknowledgment ==<br /> <br /> [a sentence to acknowledge the support of NLM]<br /> <br /> == MetaMap Installation documentation ==<br /> <br /> The installation of MetaMap primarily involves 3 stages. Each of them are explained below followed by instructions to run the newly installed custom MetaMap.<br /> <br /> === Preparation of data ===<br /> <br /> There are primarily 4 files to be generated for the installation. <br /> <br /> '''1.MRCON:''' This file is primarily generated from the table OBS_TT. Below is a mapping between the columns in MRCON and OBS_TT. The columns listed in OBS_TT are used to fill their corresponding columns in MRCON file.<br /> <br /> MRCON OBS_TT<br /> <br /> CUI (Concept Unique ID) conceptID<br /> <br /> Language- <br /> Status isPreferred <br /> LUI(Lexical Unique ID) termID <br /> String type -<br /> SUI(String Unique ID) termID <br /> String termName <br /> LRL - <br /> <br /> The sql command would fill correct values into MRCON table in database. <br /> <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|ENG|P|L0027051|PF| S0064638|Myocardial Infarction|0|<br /> <br /> <br /> '''2.MRSO:''' This file is primarily generated from the tables OBS_TT, OBS_OT and OBS_CT. Below is a mapping between the columns in MRSO and the above OBS tables. The columns listed in the tables are used to fill their corresponding columns in MRSO file.<br /> <br /> MRSO OBS tables<br /> CUI (Concept Unique ID) OBS_TT.conceptID<br /> LUI(Lexical Unique ID) OBS_TT.termID<br /> SUI(String Unique ID) OBS_TT.termID <br /> SABSourceAbbrev OBS_OT.localOntologyID<br /> TermType OBS_TT.isPreferred<br /> SourceID OBS_OT.ontologyID<br /> Restrictionlevel -<br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|L0027051|S0064638| MEDLINEPLUS|ET|T5|0|<br /> <br /> <br /> '''3.MRSTY:''' This file is primarily generated from the table OBS_STT. Below is a mapping between the columns in MRSTY and OBS_STT. The columns listed in OBS_STT are used to fill their corresponding columns in MRSTY file.<br /> <br /> MRSTY OBS_STT<br /> CUI (Concept Unique ID) OBS_STT.conceptID<br /> TUI (term Unique ID) OBS_STT.localSemanticTypeID<br /> STY OBS_STT.semanticTypeName <br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|T047|Disease or Syndrome|<br /> <br /> <br /> '''4.st.raw:''' This file is primarily generated from the column semanticTypeName in the table OBS_STT. This file contains row of semantic type names and a short name for each of them. <br /> <br /> Example row:<br /> semantic Type Name|semantictypename<br /> <br /> <br /> <br /> === Setting up the workspace ===<br /> <br /> 1. '''Install the Lexical Variant Generator (LVG)''' before running<br /> MetaMap's install program. LVG is part of the Lexical Tools distribution and is available from the Lexical Systems Group (http://lexsrv3.nlm.nih.gov/SPECIALIST/Projects/lvg/current/index.html).<br /> <br /> 2. Before using MetaMap install program to install data file builder, you also need to '''add LVG's bin directory''' {LVG_DIR}/bin to your program path:<br /> <br /> export PATH=$PATH:&lt;LVG_DIR&gt;/bin<br /> <br /> 3. '''Install MetaMap''' using instructions at their site <br /> <br /> 4. Unzip the DFB installation files in the same directory as MetaMap<br /> <br /> 5. Connect to the new directory created by extracting the distribution and invoke the install program:<br /> <br /> cd &lt;distribution directory&gt;<br /> <br /> ./bin/install.sh<br /> <br /> '''A sample run of the installation script follows:'''<br /> <br /> Enter basedir of installation [/nfsvol/nlsaux15/public_mm] <br /> <br /> Basedir is set to /nfsvol/nlsaux15/public_mm.<br /> <br /> The WSD Server requires Sun's Java Runtime Environment (JRE)<br /> Sun's Java Developer Kit (JDK) will work as well. if the<br /> command: &quot;which&quot; java returns /usr/local/jre1.4.2/bin/java, then the<br /> JRE resides in /usr/local/jre1.4.2/.<br /> <br /> Where does your distribution of Sun's JRE reside?<br /> Enter home path of JRE (JDK) [/usr]: /nfsvol/nls/tools/Linux-i686/java1.4.2<br /> Using /nfsvol/nls/tools/Linux-i686/java1.4.2 for JAVA_HOME.<br /> <br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/disambServer.cfg generated<br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/log4j.properties generated<br /> /nfsvol/nlsaux15/public_mm/bin/SKRrun generated.<br /> /nfsvol/nlsaux15/public_mm/bin/metamap07 generated.<br /> /nfsvol/nlsaux15/public_mm/bin/wsdserverctl generated.<br /> /nfsvol/nlsaux15/public_mm/bin/skrmedpostctl generated.<br /> Install complete.<br /> Would like to use a custom data set with MetaMap (use data file builder)? [yN]: <br /> <br /> running Data File Builder Install...<br /> Is LVG installed? [yN] &lt;The user types y and return&gt;<br /> <br /> running Data File Builder Install...<br /> Enter home path of LVG [/nfsvol/nls/tools/Linux-i686/lvg2009]:<br /> Using /nfsvol/nls/tools/Linux-i686/lvg2009 for LVG_DIR.<br /> <br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.lvglab generated.<br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.xwords generated.<br /> Datafile Builder Setup is complete.<br /> <br /> <br /> 6. '''Make sure that the SKR/MedPOST tagger is running;''' to run the tagger, move to the public_mm directory present inside the working directory and invoke<br /> <br /> ./bin/skrmedpostctl start<br /> <br /> '''Incase there is an error''' due to a port number, change the port number in the following files and try to run the tagger again:<br /> <br /> /bin/skrmedpostctl<br /> <br /> If you change the port number skrmedpostctl uses you need to change public_mm/bin/SKRrun (or SKRrun.in and re-run install.sh) to match the port you set in skrmedpostctl. The environment variables that should be changed in SKRrun are:<br /> <br /> TAGGER_SERVER_DEFAULT_TCP_PORT<br /> <br /> TAGGER_SERVER_TCP_PORT_0<br /> <br /> TAGGER_SERVER_TCP_PORT_1<br /> <br /> <br /> 7. Inside the directory public_mm/sourceData create a directory for workspace<br /> <br /> mkdir sourceData/09_custom<br /> <br /> Finally, Create a directory to store the knowledge sources:<br /> <br /> mkdir sourceData/09_custom/umls<br /> <br /> === Installation of Metamap ===<br /> <br /> <br /> 1. First, run the BuildDataFiles program as follows:<br /> <br /> $fullpath/public_mm/bin/BuildDataFiles<br /> <br /> 2. The file st.raw, located in $fullpath/public_mm/data/dfbuilder/2009 will need to be modified. Append the file generated in point 4 of Preparation of Data section to this file.<br /> <br /> 3. Move to $fullpath/public_mm/sourceData/09_custom/01metawordindex and execute the following scripts in order<br /> <br /> ./01CreateWorkFiles<br /> <br /> ./02Suppress<br /> <br /> ./03FilterPrep<br /> <br /> ./04FilterStrict<br /> <br /> ./05GenerateMWIFiles<br /> <br /> 4. Move to 02treecodes directory and run 01GenerateTreecodes<br /> <br /> cd ../02treecodes<br /> <br /> ./01GenerateTreecodes<br /> <br /> 5. Move to 03Variants directory and run 01GenerateVariants<br /> <br /> cd ../03Variants<br /> <br /> ./01GenerateVariants<br /> <br /> 6. cd ../04synonyms <br /> <br /> ./01GenerateSynonyms<br /> <br /> 7. cd ../05abbrAcronyms<br /> <br /> ./01GenerateAbbrAcronyms<br /> <br /> 8. move to $fullpath/public_mm and run<br /> <br /> ./bin/LoadDataFiles<br /> <br /> === Running MetaMap ===<br /> <br /> To run the newly installed MetaMap dfb, move to the main workspace folder (public_mm) and run the command below<br /> <br /> bin/SKRrun -L 2009 -M /DATA/XDR -B /BDB4 -w ./lexicon ./bin/metamap09.BINARY -Z 09_custom ./resources/input -I</div>

Tejaswit https://www.bioontology.org//mediawiki/index.php?title=MetaMap_Installation&diff=9698 2010-01-27T10:09:51Z

<p>Tejaswit: /* Contacts */</p> <hr /> <div>== Presentation ==<br /> MetaMap is a highly powerful tool to map biomedical text to the UMLS Metathesaurus or a custom Metathesaurus, equivalently, to discover Metathesaurus concepts referred to in text. The following article describes the way of using the Data File Builder utility inside MetaMap to build a concept recognizer for text containing NCBO terms.<br /> <br /> NCBO is developing a system to annotate large numbers of data resources automatically, and have developed a prototype system for ontology-based annotation and indexing of biomedical data [quote Nipun's paper]. As a part of the workflow a concept recognizer is used to recognize a given ontology concept in text. The goal of the project is to embed MetaMap into the annotator workflow. MetaMap is a very powerful for this task due to the use of complex NLP techniques in its concept recognition algorithm. Below listed is some of the useful processing done by MetaMap:<br /> <br /> 1. Removal of problematic terms from data [numbers, single alphabets etc..]<br /> <br /> 2. Lexical and Syntactic filtering to recognize long phrases accurately<br /> <br /> 3. Identifying linguistic variants of base words and terms that have been computed using a variety of generation methods. This handles various forms like inflectional (heart → hearts), derivational (treat → treatment), synonyms (pyrosis ↔ heartburn), acronyms (vt ↔ ventricular tachycardia) and spellings (apnea ↔ apnoea)<br /> <br /> Such kind of rich semantic processing can drastically increase the number of concepts that can be recognized from a text. <br /> <br /> Although Metamap was originally built to work on UMLS concepts, its Data File builder utility (DFB) lets us create our own custom dictionary for NCBO concepts.<br /> <br /> == Contacts ==<br /> <br /> * For questions or feature requests, contact Clement Jonquet [mailto:jonquet@stanford.edu?subject=MetaMap%Installation] and Tejaswi Tenneti [mailto:tejaswit@stanford.edu?subject=MetaMap%Installation]<br /> <br /> == Versions (prototypes &amp; releases) ==<br /> <br /> * December 2009 - Second Annotator prototype including MetaMap: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> * November 2009 - Second Annotator prototype including Mmtx: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> <br /> == References ==<br /> <br /> * [Gforge code location]<br /> <br /> == Collaboration &amp; Acknowledgment ==<br /> <br /> [a sentence to acknowledge the support of NLM]<br /> <br /> == MetaMap Installation documentation ==<br /> <br /> The installation of MetaMap primarily involves 3 stages. Each of them are explained below followed by instructions to run the newly installed custom MetaMap.<br /> <br /> === Preparation of data ===<br /> <br /> There are primarily 4 files to be generated for the installation. <br /> <br /> '''1.MRCON:''' This file is primarily generated from the table OBS_TT. Below is a mapping between the columns in MRCON and OBS_TT. The columns listed in OBS_TT are used to fill their corresponding columns in MRCON file.<br /> <br /> MRCON OBS_TT<br /> <br /> CUI (Concept Unique ID) conceptID<br /> <br /> Language- <br /> Status isPreferred <br /> LUI(Lexical Unique ID) termID <br /> String type -<br /> SUI(String Unique ID) termID <br /> String termName <br /> LRL - <br /> <br /> The sql command would fill correct values into MRCON table in database. <br /> <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|ENG|P|L0027051|PF| S0064638|Myocardial Infarction|0|<br /> <br /> <br /> '''2.MRSO:''' This file is primarily generated from the tables OBS_TT, OBS_OT and OBS_CT. Below is a mapping between the columns in MRSO and the above OBS tables. The columns listed in the tables are used to fill their corresponding columns in MRSO file.<br /> <br /> MRSO OBS tables<br /> CUI (Concept Unique ID) OBS_TT.conceptID<br /> LUI(Lexical Unique ID) OBS_TT.termID<br /> SUI(String Unique ID) OBS_TT.termID <br /> SABSourceAbbrev OBS_OT.localOntologyID<br /> TermType OBS_TT.isPreferred<br /> SourceID OBS_OT.ontologyID<br /> Restrictionlevel -<br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|L0027051|S0064638| MEDLINEPLUS|ET|T5|0|<br /> <br /> <br /> '''3.MRSTY:''' This file is primarily generated from the table OBS_STT. Below is a mapping between the columns in MRSTY and OBS_STT. The columns listed in OBS_STT are used to fill their corresponding columns in MRSTY file.<br /> <br /> MRSTY OBS_STT<br /> CUI (Concept Unique ID) OBS_STT.conceptID<br /> TUI (term Unique ID) OBS_STT.localSemanticTypeID<br /> STY OBS_STT.semanticTypeName <br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|T047|Disease or Syndrome|<br /> <br /> <br /> '''4.st.raw:''' This file is primarily generated from the column semanticTypeName in the table OBS_STT. This file contains row of semantic type names and a short name for each of them. <br /> <br /> Example row:<br /> semantic Type Name|semantictypename<br /> <br /> <br /> <br /> === Setting up the workspace ===<br /> <br /> 1. '''Install the Lexical Variant Generator (LVG)''' before running<br /> MetaMap's install program. LVG is part of the Lexical Tools distribution and is available from the Lexical Systems Group (http://lexsrv3.nlm.nih.gov/SPECIALIST/Projects/lvg/current/index.html).<br /> <br /> 2. Before using MetaMap install program to install data file builder, you also need to '''add LVG's bin directory''' {LVG_DIR}/bin to your program path:<br /> <br /> export PATH=$PATH:&lt;LVG_DIR&gt;/bin<br /> <br /> 3. '''Install MetaMap''' using instructions at their site <br /> <br /> 4. Unzip the DFB installation files in the same directory as MetaMap<br /> <br /> 5. Connect to the new directory created by extracting the distribution and invoke the install program:<br /> <br /> cd &lt;distribution directory&gt;<br /> <br /> ./bin/install.sh<br /> <br /> '''A sample run of the installation script follows:'''<br /> <br /> Enter basedir of installation [/nfsvol/nlsaux15/public_mm] <br /> <br /> Basedir is set to /nfsvol/nlsaux15/public_mm.<br /> <br /> The WSD Server requires Sun's Java Runtime Environment (JRE)<br /> Sun's Java Developer Kit (JDK) will work as well. if the<br /> command: &quot;which&quot; java returns /usr/local/jre1.4.2/bin/java, then the<br /> JRE resides in /usr/local/jre1.4.2/.<br /> <br /> Where does your distribution of Sun's JRE reside?<br /> Enter home path of JRE (JDK) [/usr]: /nfsvol/nls/tools/Linux-i686/java1.4.2<br /> Using /nfsvol/nls/tools/Linux-i686/java1.4.2 for JAVA_HOME.<br /> <br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/disambServer.cfg generated<br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/log4j.properties generated<br /> /nfsvol/nlsaux15/public_mm/bin/SKRrun generated.<br /> /nfsvol/nlsaux15/public_mm/bin/metamap07 generated.<br /> /nfsvol/nlsaux15/public_mm/bin/wsdserverctl generated.<br /> /nfsvol/nlsaux15/public_mm/bin/skrmedpostctl generated.<br /> Install complete.<br /> Would like to use a custom data set with MetaMap (use data file builder)? [yN]: <br /> <br /> running Data File Builder Install...<br /> Is LVG installed? [yN] &lt;The user types y and return&gt;<br /> <br /> running Data File Builder Install...<br /> Enter home path of LVG [/nfsvol/nls/tools/Linux-i686/lvg2009]:<br /> Using /nfsvol/nls/tools/Linux-i686/lvg2009 for LVG_DIR.<br /> <br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.lvglab generated.<br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.xwords generated.<br /> Datafile Builder Setup is complete.<br /> <br /> <br /> 6. '''Make sure that the SKR/MedPOST tagger is running;''' to run the tagger, move to the public_mm directory present inside the working directory and invoke<br /> <br /> ./bin/skrmedpostctl start<br /> <br /> '''Incase there is an error''' due to a port number, change the port number in the following files and try to run the tagger again:<br /> <br /> /bin/skrmedpostctl<br /> <br /> If you change the port number skrmedpostctl uses you need to change public_mm/bin/SKRrun (or SKRrun.in and re-run install.sh) to match the port you set in skrmedpostctl. The environment variables that should be changed in SKRrun are:<br /> <br /> TAGGER_SERVER_DEFAULT_TCP_PORT<br /> <br /> TAGGER_SERVER_TCP_PORT_0<br /> <br /> TAGGER_SERVER_TCP_PORT_1<br /> <br /> <br /> 7. Inside the directory public_mm/sourceData create a directory for workspace<br /> <br /> mkdir sourceData/09_custom<br /> <br /> Finally, Create a directory to store the knowledge sources:<br /> <br /> mkdir sourceData/09_custom/umls<br /> <br /> === Installation of Metamap ===<br /> <br /> <br /> 1. First, run the BuildDataFiles program as follows:<br /> <br /> $fullpath/public_mm/bin/BuildDataFiles<br /> <br /> 2. The file st.raw, located in $fullpath/public_mm/data/dfbuilder/2009 will need to be modified. Append the file generated in point 4 of Preparation of Data section to this file.<br /> <br /> 3. Move to $fullpath/public_mm/sourceData/09_custom/01metawordindex and execute the following scripts in order<br /> <br /> ./01CreateWorkFiles<br /> <br /> ./02Suppress<br /> <br /> ./03FilterPrep<br /> <br /> ./04FilterStrict<br /> <br /> ./05GenerateMWIFiles<br /> <br /> 4. Move to 02treecodes directory and run 01GenerateTreecodes<br /> <br /> cd ../02treecodes<br /> <br /> ./01GenerateTreecodes<br /> <br /> 5. Move to 03Variants directory and run 01GenerateVariants<br /> <br /> cd ../03Variants<br /> <br /> ./01GenerateVariants<br /> <br /> 6. cd ../04synonyms <br /> <br /> ./01GenerateSynonyms<br /> <br /> 7. cd ../05abbrAcronyms<br /> <br /> ./01GenerateAbbrAcronyms<br /> <br /> 8. move to $fullpath/public_mm and run<br /> <br /> ./bin/LoadDataFiles<br /> <br /> === Running MetaMap ===<br /> <br /> To run the newly installed MetaMap dfb, move to the main workspace folder (public_mm) and run the command below<br /> <br /> bin/SKRrun -L 2009 -M /DATA/XDR -B /BDB4 -w ./lexicon ./bin/metamap09.BINARY -Z 09_custom ./resources/input -I</div>

Tejaswit https://www.bioontology.org//mediawiki/index.php?title=MetaMap_Installation&diff=9697 2010-01-27T10:03:04Z

<p>Tejaswit: /* Presentation */</p> <hr /> <div>== Presentation ==<br /> MetaMap is a highly powerful tool to map biomedical text to the UMLS Metathesaurus or a custom Metathesaurus, equivalently, to discover Metathesaurus concepts referred to in text. The following article describes the way of using the Data File Builder utility inside MetaMap to build a concept recognizer for text containing NCBO terms.<br /> <br /> NCBO is developing a system to annotate large numbers of data resources automatically, and have developed a prototype system for ontology-based annotation and indexing of biomedical data [quote Nipun's paper]. As a part of the workflow a concept recognizer is used to recognize a given ontology concept in text. The goal of the project is to embed MetaMap into the annotator workflow. MetaMap is a very powerful for this task due to the use of complex NLP techniques in its concept recognition algorithm. Below listed is some of the useful processing done by MetaMap:<br /> <br /> 1. Removal of problematic terms from data [numbers, single alphabets etc..]<br /> <br /> 2. Lexical and Syntactic filtering to recognize long phrases accurately<br /> <br /> 3. Identifying linguistic variants of base words and terms that have been computed using a variety of generation methods. This handles various forms like inflectional (heart → hearts), derivational (treat → treatment), synonyms (pyrosis ↔ heartburn), acronyms (vt ↔ ventricular tachycardia) and spellings (apnea ↔ apnoea)<br /> <br /> Such kind of rich semantic processing can drastically increase the number of concepts that can be recognized from a text. <br /> <br /> Although Metamap was originally built to work on UMLS concepts, its Data File builder utility (DFB) lets us create our own custom dictionary for NCBO concepts.<br /> <br /> == Contacts ==<br /> <br /> * For questions or feature requests, contact XX and YY [mailto:XXX?subject=YYY]<br /> <br /> == Versions (prototypes &amp; releases) ==<br /> <br /> * December 2009 - Second Annotator prototype including MetaMap: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> * November 2009 - Second Annotator prototype including Mmtx: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> <br /> == References ==<br /> <br /> * [Gforge code location]<br /> <br /> == Collaboration &amp; Acknowledgment ==<br /> <br /> [a sentence to acknowledge the support of NLM]<br /> <br /> == MetaMap Installation documentation ==<br /> <br /> The installation of MetaMap primarily involves 3 stages. Each of them are explained below followed by instructions to run the newly installed custom MetaMap.<br /> <br /> === Preparation of data ===<br /> <br /> There are primarily 4 files to be generated for the installation. <br /> <br /> '''1.MRCON:''' This file is primarily generated from the table OBS_TT. Below is a mapping between the columns in MRCON and OBS_TT. The columns listed in OBS_TT are used to fill their corresponding columns in MRCON file.<br /> <br /> MRCON OBS_TT<br /> <br /> CUI (Concept Unique ID) conceptID<br /> <br /> Language- <br /> Status isPreferred <br /> LUI(Lexical Unique ID) termID <br /> String type -<br /> SUI(String Unique ID) termID <br /> String termName <br /> LRL - <br /> <br /> The sql command would fill correct values into MRCON table in database. <br /> <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|ENG|P|L0027051|PF| S0064638|Myocardial Infarction|0|<br /> <br /> <br /> '''2.MRSO:''' This file is primarily generated from the tables OBS_TT, OBS_OT and OBS_CT. Below is a mapping between the columns in MRSO and the above OBS tables. The columns listed in the tables are used to fill their corresponding columns in MRSO file.<br /> <br /> MRSO OBS tables<br /> CUI (Concept Unique ID) OBS_TT.conceptID<br /> LUI(Lexical Unique ID) OBS_TT.termID<br /> SUI(String Unique ID) OBS_TT.termID <br /> SABSourceAbbrev OBS_OT.localOntologyID<br /> TermType OBS_TT.isPreferred<br /> SourceID OBS_OT.ontologyID<br /> Restrictionlevel -<br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|L0027051|S0064638| MEDLINEPLUS|ET|T5|0|<br /> <br /> <br /> '''3.MRSTY:''' This file is primarily generated from the table OBS_STT. Below is a mapping between the columns in MRSTY and OBS_STT. The columns listed in OBS_STT are used to fill their corresponding columns in MRSTY file.<br /> <br /> MRSTY OBS_STT<br /> CUI (Concept Unique ID) OBS_STT.conceptID<br /> TUI (term Unique ID) OBS_STT.localSemanticTypeID<br /> STY OBS_STT.semanticTypeName <br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|T047|Disease or Syndrome|<br /> <br /> <br /> '''4.st.raw:''' This file is primarily generated from the column semanticTypeName in the table OBS_STT. This file contains row of semantic type names and a short name for each of them. <br /> <br /> Example row:<br /> semantic Type Name|semantictypename<br /> <br /> <br /> <br /> === Setting up the workspace ===<br /> <br /> 1. '''Install the Lexical Variant Generator (LVG)''' before running<br /> MetaMap's install program. LVG is part of the Lexical Tools distribution and is available from the Lexical Systems Group (http://lexsrv3.nlm.nih.gov/SPECIALIST/Projects/lvg/current/index.html).<br /> <br /> 2. Before using MetaMap install program to install data file builder, you also need to '''add LVG's bin directory''' {LVG_DIR}/bin to your program path:<br /> <br /> export PATH=$PATH:&lt;LVG_DIR&gt;/bin<br /> <br /> 3. '''Install MetaMap''' using instructions at their site <br /> <br /> 4. Unzip the DFB installation files in the same directory as MetaMap<br /> <br /> 5. Connect to the new directory created by extracting the distribution and invoke the install program:<br /> <br /> cd &lt;distribution directory&gt;<br /> <br /> ./bin/install.sh<br /> <br /> '''A sample run of the installation script follows:'''<br /> <br /> Enter basedir of installation [/nfsvol/nlsaux15/public_mm] <br /> <br /> Basedir is set to /nfsvol/nlsaux15/public_mm.<br /> <br /> The WSD Server requires Sun's Java Runtime Environment (JRE)<br /> Sun's Java Developer Kit (JDK) will work as well. if the<br /> command: &quot;which&quot; java returns /usr/local/jre1.4.2/bin/java, then the<br /> JRE resides in /usr/local/jre1.4.2/.<br /> <br /> Where does your distribution of Sun's JRE reside?<br /> Enter home path of JRE (JDK) [/usr]: /nfsvol/nls/tools/Linux-i686/java1.4.2<br /> Using /nfsvol/nls/tools/Linux-i686/java1.4.2 for JAVA_HOME.<br /> <br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/disambServer.cfg generated<br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/log4j.properties generated<br /> /nfsvol/nlsaux15/public_mm/bin/SKRrun generated.<br /> /nfsvol/nlsaux15/public_mm/bin/metamap07 generated.<br /> /nfsvol/nlsaux15/public_mm/bin/wsdserverctl generated.<br /> /nfsvol/nlsaux15/public_mm/bin/skrmedpostctl generated.<br /> Install complete.<br /> Would like to use a custom data set with MetaMap (use data file builder)? [yN]: <br /> <br /> running Data File Builder Install...<br /> Is LVG installed? [yN] &lt;The user types y and return&gt;<br /> <br /> running Data File Builder Install...<br /> Enter home path of LVG [/nfsvol/nls/tools/Linux-i686/lvg2009]:<br /> Using /nfsvol/nls/tools/Linux-i686/lvg2009 for LVG_DIR.<br /> <br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.lvglab generated.<br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.xwords generated.<br /> Datafile Builder Setup is complete.<br /> <br /> <br /> 6. '''Make sure that the SKR/MedPOST tagger is running;''' to run the tagger, move to the public_mm directory present inside the working directory and invoke<br /> <br /> ./bin/skrmedpostctl start<br /> <br /> '''Incase there is an error''' due to a port number, change the port number in the following files and try to run the tagger again:<br /> <br /> /bin/skrmedpostctl<br /> <br /> If you change the port number skrmedpostctl uses you need to change public_mm/bin/SKRrun (or SKRrun.in and re-run install.sh) to match the port you set in skrmedpostctl. The environment variables that should be changed in SKRrun are:<br /> <br /> TAGGER_SERVER_DEFAULT_TCP_PORT<br /> <br /> TAGGER_SERVER_TCP_PORT_0<br /> <br /> TAGGER_SERVER_TCP_PORT_1<br /> <br /> <br /> 7. Inside the directory public_mm/sourceData create a directory for workspace<br /> <br /> mkdir sourceData/09_custom<br /> <br /> Finally, Create a directory to store the knowledge sources:<br /> <br /> mkdir sourceData/09_custom/umls<br /> <br /> === Installation of Metamap ===<br /> <br /> <br /> 1. First, run the BuildDataFiles program as follows:<br /> <br /> $fullpath/public_mm/bin/BuildDataFiles<br /> <br /> 2. The file st.raw, located in $fullpath/public_mm/data/dfbuilder/2009 will need to be modified. Append the file generated in point 4 of Preparation of Data section to this file.<br /> <br /> 3. Move to $fullpath/public_mm/sourceData/09_custom/01metawordindex and execute the following scripts in order<br /> <br /> ./01CreateWorkFiles<br /> <br /> ./02Suppress<br /> <br /> ./03FilterPrep<br /> <br /> ./04FilterStrict<br /> <br /> ./05GenerateMWIFiles<br /> <br /> 4. Move to 02treecodes directory and run 01GenerateTreecodes<br /> <br /> cd ../02treecodes<br /> <br /> ./01GenerateTreecodes<br /> <br /> 5. Move to 03Variants directory and run 01GenerateVariants<br /> <br /> cd ../03Variants<br /> <br /> ./01GenerateVariants<br /> <br /> 6. cd ../04synonyms <br /> <br /> ./01GenerateSynonyms<br /> <br /> 7. cd ../05abbrAcronyms<br /> <br /> ./01GenerateAbbrAcronyms<br /> <br /> 8. move to $fullpath/public_mm and run<br /> <br /> ./bin/LoadDataFiles<br /> <br /> === Running MetaMap ===<br /> <br /> To run the newly installed MetaMap dfb, move to the main workspace folder (public_mm) and run the command below<br /> <br /> bin/SKRrun -L 2009 -M /DATA/XDR -B /BDB4 -w ./lexicon ./bin/metamap09.BINARY -Z 09_custom ./resources/input -I</div>

Tejaswit https://www.bioontology.org//mediawiki/index.php?title=MetaMap_Installation&diff=9696 2010-01-27T09:55:51Z

<p>Tejaswit: /* Presentation */</p> <hr /> <div>== Presentation ==<br /> MetaMap is a highly powerful tool to map biomedical text to the UMLS Metathesaurus or a custom Metathesaurus, equivalently, to discover Metathesaurus concepts referred to in text. The following article describes the way of using the Data File Builder utility inside MetaMap to build a concept recognizer for text containing NCBO terms.<br /> <br /> NCBO is developing a system for automated, ontology-based access to online biomedical resources. As a part of the workflow a concept recognizer is used to recognize a given ontology concept in text. MetaMap is very powerful for this task due to the use of complex NLP techniques. Below listed is some of the useful processing done by MetaMap:<br /> <br /> 1. Removal of problematic terms from data [numbers, single alphabets etc..]<br /> <br /> 2. Lexical and Syntactic filtering to recognize long phrases accurately<br /> <br /> 3. Identifying linguistic variants of base words and terms that have been computed using a variety of generation methods. This handles various forms like inflectional (heart → hearts), derivational (treat → treatment), synonyms (pyrosis ↔ heartburn), acronyms (vt ↔ ventricular tachycardia) and spellings (apnea ↔ apnoea)<br /> <br /> Such kind of rich semantic processing can drastically increase the number of concepts that can be recognized from a text. <br /> <br /> Although Metamap was originally built to work on UMLS concepts, its Data File builder utility (DFB) lets us create our own custom dictionary out of NCBO concepts.<br /> <br /> == Contacts ==<br /> <br /> * For questions or feature requests, contact XX and YY [mailto:XXX?subject=YYY]<br /> <br /> == Versions (prototypes &amp; releases) ==<br /> <br /> * December 2009 - Second Annotator prototype including MetaMap: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> * November 2009 - Second Annotator prototype including Mmtx: [http://obs.bioontology.org/oba/OBA_v1.2_rest.html] <br /> [Tejaswi: give detail about the release, and the problems of the installation e.g., the list of ontologies used]<br /> <br /> == References ==<br /> <br /> * [Gforge code location]<br /> <br /> == Collaboration &amp; Acknowledgment ==<br /> <br /> [a sentence to acknowledge the support of NLM]<br /> <br /> == MetaMap Installation documentation ==<br /> <br /> The installation of MetaMap primarily involves 3 stages. Each of them are explained below followed by instructions to run the newly installed custom MetaMap.<br /> <br /> === Preparation of data ===<br /> <br /> There are primarily 4 files to be generated for the installation. <br /> <br /> '''1.MRCON:''' This file is primarily generated from the table OBS_TT. Below is a mapping between the columns in MRCON and OBS_TT. The columns listed in OBS_TT are used to fill their corresponding columns in MRCON file.<br /> <br /> MRCON OBS_TT<br /> <br /> CUI (Concept Unique ID) conceptID<br /> <br /> Language- <br /> Status isPreferred <br /> LUI(Lexical Unique ID) termID <br /> String type -<br /> SUI(String Unique ID) termID <br /> String termName <br /> LRL - <br /> <br /> The sql command would fill correct values into MRCON table in database. <br /> <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|ENG|P|L0027051|PF| S0064638|Myocardial Infarction|0|<br /> <br /> <br /> '''2.MRSO:''' This file is primarily generated from the tables OBS_TT, OBS_OT and OBS_CT. Below is a mapping between the columns in MRSO and the above OBS tables. The columns listed in the tables are used to fill their corresponding columns in MRSO file.<br /> <br /> MRSO OBS tables<br /> CUI (Concept Unique ID) OBS_TT.conceptID<br /> LUI(Lexical Unique ID) OBS_TT.termID<br /> SUI(String Unique ID) OBS_TT.termID <br /> SABSourceAbbrev OBS_OT.localOntologyID<br /> TermType OBS_TT.isPreferred<br /> SourceID OBS_OT.ontologyID<br /> Restrictionlevel -<br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|L0027051|S0064638| MEDLINEPLUS|ET|T5|0|<br /> <br /> <br /> '''3.MRSTY:''' This file is primarily generated from the table OBS_STT. Below is a mapping between the columns in MRSTY and OBS_STT. The columns listed in OBS_STT are used to fill their corresponding columns in MRSTY file.<br /> <br /> MRSTY OBS_STT<br /> CUI (Concept Unique ID) OBS_STT.conceptID<br /> TUI (term Unique ID) OBS_STT.localSemanticTypeID<br /> STY OBS_STT.semanticTypeName <br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|T047|Disease or Syndrome|<br /> <br /> <br /> '''4.st.raw:''' This file is primarily generated from the column semanticTypeName in the table OBS_STT. This file contains row of semantic type names and a short name for each of them. <br /> <br /> Example row:<br /> semantic Type Name|semantictypename<br /> <br /> <br /> <br /> === Setting up the workspace ===<br /> <br /> 1. '''Install the Lexical Variant Generator (LVG)''' before running<br /> MetaMap's install program. LVG is part of the Lexical Tools distribution and is available from the Lexical Systems Group (http://lexsrv3.nlm.nih.gov/SPECIALIST/Projects/lvg/current/index.html).<br /> <br /> 2. Before using MetaMap install program to install data file builder, you also need to '''add LVG's bin directory''' {LVG_DIR}/bin to your program path:<br /> <br /> export PATH=$PATH:&lt;LVG_DIR&gt;/bin<br /> <br /> 3. '''Install MetaMap''' using instructions at their site <br /> <br /> 4. Unzip the DFB installation files in the same directory as MetaMap<br /> <br /> 5. Connect to the new directory created by extracting the distribution and invoke the install program:<br /> <br /> cd &lt;distribution directory&gt;<br /> <br /> ./bin/install.sh<br /> <br /> '''A sample run of the installation script follows:'''<br /> <br /> Enter basedir of installation [/nfsvol/nlsaux15/public_mm] <br /> <br /> Basedir is set to /nfsvol/nlsaux15/public_mm.<br /> <br /> The WSD Server requires Sun's Java Runtime Environment (JRE)<br /> Sun's Java Developer Kit (JDK) will work as well. if the<br /> command: &quot;which&quot; java returns /usr/local/jre1.4.2/bin/java, then the<br /> JRE resides in /usr/local/jre1.4.2/.<br /> <br /> Where does your distribution of Sun's JRE reside?<br /> Enter home path of JRE (JDK) [/usr]: /nfsvol/nls/tools/Linux-i686/java1.4.2<br /> Using /nfsvol/nls/tools/Linux-i686/java1.4.2 for JAVA_HOME.<br /> <br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/disambServer.cfg generated<br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/log4j.properties generated<br /> /nfsvol/nlsaux15/public_mm/bin/SKRrun generated.<br /> /nfsvol/nlsaux15/public_mm/bin/metamap07 generated.<br /> /nfsvol/nlsaux15/public_mm/bin/wsdserverctl generated.<br /> /nfsvol/nlsaux15/public_mm/bin/skrmedpostctl generated.<br /> Install complete.<br /> Would like to use a custom data set with MetaMap (use data file builder)? [yN]: <br /> <br /> running Data File Builder Install...<br /> Is LVG installed? [yN] &lt;The user types y and return&gt;<br /> <br /> running Data File Builder Install...<br /> Enter home path of LVG [/nfsvol/nls/tools/Linux-i686/lvg2009]:<br /> Using /nfsvol/nls/tools/Linux-i686/lvg2009 for LVG_DIR.<br /> <br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.lvglab generated.<br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.xwords generated.<br /> Datafile Builder Setup is complete.<br /> <br /> <br /> 6. '''Make sure that the SKR/MedPOST tagger is running;''' to run the tagger, move to the public_mm directory present inside the working directory and invoke<br /> <br /> ./bin/skrmedpostctl start<br /> <br /> '''Incase there is an error''' due to a port number, change the port number in the following files and try to run the tagger again:<br /> <br /> /bin/skrmedpostctl<br /> <br /> If you change the port number skrmedpostctl uses you need to change public_mm/bin/SKRrun (or SKRrun.in and re-run install.sh) to match the port you set in skrmedpostctl. The environment variables that should be changed in SKRrun are:<br /> <br /> TAGGER_SERVER_DEFAULT_TCP_PORT<br /> <br /> TAGGER_SERVER_TCP_PORT_0<br /> <br /> TAGGER_SERVER_TCP_PORT_1<br /> <br /> <br /> 7. Inside the directory public_mm/sourceData create a directory for workspace<br /> <br /> mkdir sourceData/09_custom<br /> <br /> Finally, Create a directory to store the knowledge sources:<br /> <br /> mkdir sourceData/09_custom/umls<br /> <br /> === Installation of Metamap ===<br /> <br /> <br /> 1. First, run the BuildDataFiles program as follows:<br /> <br /> $fullpath/public_mm/bin/BuildDataFiles<br /> <br /> 2. The file st.raw, located in $fullpath/public_mm/data/dfbuilder/2009 will need to be modified. Append the file generated in point 4 of Preparation of Data section to this file.<br /> <br /> 3. Move to $fullpath/public_mm/sourceData/09_custom/01metawordindex and execute the following scripts in order<br /> <br /> ./01CreateWorkFiles<br /> <br /> ./02Suppress<br /> <br /> ./03FilterPrep<br /> <br /> ./04FilterStrict<br /> <br /> ./05GenerateMWIFiles<br /> <br /> 4. Move to 02treecodes directory and run 01GenerateTreecodes<br /> <br /> cd ../02treecodes<br /> <br /> ./01GenerateTreecodes<br /> <br /> 5. Move to 03Variants directory and run 01GenerateVariants<br /> <br /> cd ../03Variants<br /> <br /> ./01GenerateVariants<br /> <br /> 6. cd ../04synonyms <br /> <br /> ./01GenerateSynonyms<br /> <br /> 7. cd ../05abbrAcronyms<br /> <br /> ./01GenerateAbbrAcronyms<br /> <br /> 8. move to $fullpath/public_mm and run<br /> <br /> ./bin/LoadDataFiles<br /> <br /> === Running MetaMap ===<br /> <br /> To run the newly installed MetaMap dfb, move to the main workspace folder (public_mm) and run the command below<br /> <br /> bin/SKRrun -L 2009 -M /DATA/XDR -B /BDB4 -w ./lexicon ./bin/metamap09.BINARY -Z 09_custom ./resources/input -I</div>

Tejaswit https://www.bioontology.org//mediawiki/index.php?title=MetaMap_Installation&diff=9682 2010-01-22T10:35:48Z

<p>Tejaswit: /* Setting up the workspace */</p> <hr /> <div>== Introduction ==<br /> MetaMap is a highly powerful tool to map biomedical text to the UMLS Metathesaurus or a custom Metathesaurus, equivalently, to discover Metathesaurus concepts referred to in text. The following article describes the way of using the Data File Builder utility inside MetaMap to build a concept recognizer for text containing NCBO terms.<br /> <br /> Installation of MetaMap primarily involves 3 stages. <br /> <br /> 1.Preparation of data<br /> <br /> 2.Setting up the workspace<br /> <br /> 3.Installation of Metamap<br /> <br /> Each of the step is explained below followed by instructions to run the newly installed custom MetaMap.<br /> <br /> == Preparation of data ==<br /> <br /> There are primarily 4 files to be generated for the installation. <br /> <br /> '''1.MRCON:''' This file is primarily generated from the table OBS_TT. Below is a mapping between the columns in MRCON and OBS_TT. The columns listed in OBS_TT are used to fill their corresponding columns in MRCON file.<br /> <br /> MRCON OBS_TT<br /> <br /> CUI (Concept Unique ID) conceptID<br /> <br /> Language- <br /> Status isPreferred <br /> LUI(Lexical Unique ID) termID <br /> String type -<br /> SUI(String Unique ID) termID <br /> String termName <br /> LRL - <br /> <br /> The sql command would fill correct values into MRCON table in database. <br /> <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|ENG|P|L0027051|PF| S0064638|Myocardial Infarction|0|<br /> <br /> <br /> '''2.MRSO:''' This file is primarily generated from the tables OBS_TT, OBS_OT and OBS_CT. Below is a mapping between the columns in MRSO and the above OBS tables. The columns listed in the tables are used to fill their corresponding columns in MRSO file.<br /> <br /> MRSO OBS tables<br /> CUI (Concept Unique ID) OBS_TT.conceptID<br /> LUI(Lexical Unique ID) OBS_TT.termID<br /> SUI(String Unique ID) OBS_TT.termID <br /> SABSourceAbbrev OBS_OT.localOntologyID<br /> TermType OBS_TT.isPreferred<br /> SourceID OBS_OT.ontologyID<br /> Restrictionlevel -<br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|L0027051|S0064638| MEDLINEPLUS|ET|T5|0|<br /> <br /> <br /> '''3.MRSTY:''' This file is primarily generated from the table OBS_STT. Below is a mapping between the columns in MRSTY and OBS_STT. The columns listed in OBS_STT are used to fill their corresponding columns in MRSTY file.<br /> <br /> MRSTY OBS_STT<br /> CUI (Concept Unique ID) OBS_STT.conceptID<br /> TUI (term Unique ID) OBS_STT.localSemanticTypeID<br /> STY OBS_STT.semanticTypeName <br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|T047|Disease or Syndrome|<br /> <br /> <br /> '''4.st.raw:''' This file is primarily generated from the column semanticTypeName in the table OBS_STT. This file contains row of semantic type names and a short name for each of them. <br /> <br /> Example row:<br /> semantic Type Name|semantictypename<br /> <br /> <br /> <br /> == Setting up the workspace ==<br /> <br /> 1. '''Install the Lexical Variant Generator (LVG)''' before running<br /> MetaMap's install program. LVG is part of the Lexical Tools distribution and is available from the Lexical Systems Group (http://lexsrv3.nlm.nih.gov/SPECIALIST/Projects/lvg/current/index.html).<br /> <br /> 2. Before using MetaMap install program to install data file builder, you also need to '''add LVG's bin directory''' {LVG_DIR}/bin to your program path:<br /> <br /> export PATH=$PATH:&lt;LVG_DIR&gt;/bin<br /> <br /> 3. '''Install MetaMap''' using instructions at their site <br /> <br /> 4. Unzip the DFB installation files in the same directory as MetaMap<br /> <br /> 5. Connect to the new directory created by extracting the distribution and invoke the install program:<br /> <br /> cd &lt;distribution directory&gt;<br /> <br /> ./bin/install.sh<br /> <br /> '''A sample run of the installation script follows:'''<br /> <br /> Enter basedir of installation [/nfsvol/nlsaux15/public_mm] <br /> <br /> Basedir is set to /nfsvol/nlsaux15/public_mm.<br /> <br /> The WSD Server requires Sun's Java Runtime Environment (JRE)<br /> Sun's Java Developer Kit (JDK) will work as well. if the<br /> command: &quot;which&quot; java returns /usr/local/jre1.4.2/bin/java, then the<br /> JRE resides in /usr/local/jre1.4.2/.<br /> <br /> Where does your distribution of Sun's JRE reside?<br /> Enter home path of JRE (JDK) [/usr]: /nfsvol/nls/tools/Linux-i686/java1.4.2<br /> Using /nfsvol/nls/tools/Linux-i686/java1.4.2 for JAVA_HOME.<br /> <br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/disambServer.cfg generated<br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/log4j.properties generated<br /> /nfsvol/nlsaux15/public_mm/bin/SKRrun generated.<br /> /nfsvol/nlsaux15/public_mm/bin/metamap07 generated.<br /> /nfsvol/nlsaux15/public_mm/bin/wsdserverctl generated.<br /> /nfsvol/nlsaux15/public_mm/bin/skrmedpostctl generated.<br /> Install complete.<br /> Would like to use a custom data set with MetaMap (use data file builder)? [yN]: <br /> <br /> running Data File Builder Install...<br /> Is LVG installed? [yN] &lt;The user types y and return&gt;<br /> <br /> running Data File Builder Install...<br /> Enter home path of LVG [/nfsvol/nls/tools/Linux-i686/lvg2009]:<br /> Using /nfsvol/nls/tools/Linux-i686/lvg2009 for LVG_DIR.<br /> <br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.lvglab generated.<br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.xwords generated.<br /> Datafile Builder Setup is complete.<br /> <br /> <br /> 6. '''Make sure that the SKR/MedPOST tagger is running;''' to run the tagger, move to the public_mm directory present inside the working directory and invoke<br /> <br /> ./bin/skrmedpostctl start<br /> <br /> '''Incase there is an error''' due to a port number, change the port number in the following files and try to run the tagger again:<br /> <br /> /bin/skrmedpostctl<br /> <br /> If you change the port number skrmedpostctl uses you need to change public_mm/bin/SKRrun (or SKRrun.in and re-run install.sh) to match the port you set in skrmedpostctl. The environment variables that should be changed in SKRrun are:<br /> <br /> TAGGER_SERVER_DEFAULT_TCP_PORT<br /> <br /> TAGGER_SERVER_TCP_PORT_0<br /> <br /> TAGGER_SERVER_TCP_PORT_1<br /> <br /> <br /> 7. Inside the directory public_mm/sourceData create a directory for workspace<br /> <br /> mkdir sourceData/09_custom<br /> <br /> Finally, Create a directory to store the knowledge sources:<br /> <br /> mkdir sourceData/09_custom/umls<br /> <br /> == Installation of Metamap ==<br /> <br /> <br /> 1. First, run the BuildDataFiles program as follows:<br /> <br /> $fullpath/public_mm/bin/BuildDataFiles<br /> <br /> 2. The file st.raw, located in $fullpath/public_mm/data/dfbuilder/2009 will need to be modified. Append the file generated in point 4 of Preparation of Data section to this file.<br /> <br /> 3. Move to $fullpath/public_mm/sourceData/09_custom/01metawordindex and execute the following scripts in order<br /> <br /> ./01CreateWorkFiles<br /> <br /> ./02Suppress<br /> <br /> ./03FilterPrep<br /> <br /> ./04FilterStrict<br /> <br /> ./05GenerateMWIFiles<br /> <br /> 4. Move to 02treecodes directory and run 01GenerateTreecodes<br /> <br /> cd ../02treecodes<br /> <br /> ./01GenerateTreecodes<br /> <br /> 5. Move to 03Variants directory and run 01GenerateVariants<br /> <br /> cd ../03Variants<br /> <br /> ./01GenerateVariants<br /> <br /> 6. cd ../04synonyms <br /> <br /> ./01GenerateSynonyms<br /> <br /> 7. cd ../05abbrAcronyms<br /> <br /> ./01GenerateAbbrAcronyms<br /> <br /> 8. move to $fullpath/public_mm and run<br /> <br /> ./bin/LoadDataFiles<br /> <br /> <br /> == Running MetaMap ==<br /> <br /> To run the newly installed MetaMap dfb, move to the main workspace folder (public_mm) and run the command below<br /> <br /> bin/SKRrun -L 2009 -M /DATA/XDR -B /BDB4 -w ./lexicon ./bin/metamap09.BINARY -Z 09_custom ./resources/input -I</div>

Tejaswit https://www.bioontology.org//mediawiki/index.php?title=MetaMap_Installation&diff=9681 2010-01-22T10:28:24Z

<p>Tejaswit: New page: == Introduction == MetaMap is a highly powerful tool to map biomedical text to the UMLS Metathesaurus or a custom Metathesaurus, equivalently, to discover Metathesaurus concepts referred t...</p> <hr /> <div>== Introduction ==<br /> MetaMap is a highly powerful tool to map biomedical text to the UMLS Metathesaurus or a custom Metathesaurus, equivalently, to discover Metathesaurus concepts referred to in text. The following article describes the way of using the Data File Builder utility inside MetaMap to build a concept recognizer for text containing NCBO terms.<br /> <br /> Installation of MetaMap primarily involves 3 stages. <br /> <br /> 1.Preparation of data<br /> <br /> 2.Setting up the workspace<br /> <br /> 3.Installation of Metamap<br /> <br /> Each of the step is explained below followed by instructions to run the newly installed custom MetaMap.<br /> <br /> == Preparation of data ==<br /> <br /> There are primarily 4 files to be generated for the installation. <br /> <br /> '''1.MRCON:''' This file is primarily generated from the table OBS_TT. Below is a mapping between the columns in MRCON and OBS_TT. The columns listed in OBS_TT are used to fill their corresponding columns in MRCON file.<br /> <br /> MRCON OBS_TT<br /> <br /> CUI (Concept Unique ID) conceptID<br /> <br /> Language- <br /> Status isPreferred <br /> LUI(Lexical Unique ID) termID <br /> String type -<br /> SUI(String Unique ID) termID <br /> String termName <br /> LRL - <br /> <br /> The sql command would fill correct values into MRCON table in database. <br /> <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|ENG|P|L0027051|PF| S0064638|Myocardial Infarction|0|<br /> <br /> <br /> '''2.MRSO:''' This file is primarily generated from the tables OBS_TT, OBS_OT and OBS_CT. Below is a mapping between the columns in MRSO and the above OBS tables. The columns listed in the tables are used to fill their corresponding columns in MRSO file.<br /> <br /> MRSO OBS tables<br /> CUI (Concept Unique ID) OBS_TT.conceptID<br /> LUI(Lexical Unique ID) OBS_TT.termID<br /> SUI(String Unique ID) OBS_TT.termID <br /> SABSourceAbbrev OBS_OT.localOntologyID<br /> TermType OBS_TT.isPreferred<br /> SourceID OBS_OT.ontologyID<br /> Restrictionlevel -<br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|L0027051|S0064638| MEDLINEPLUS|ET|T5|0|<br /> <br /> <br /> '''3.MRSTY:''' This file is primarily generated from the table OBS_STT. Below is a mapping between the columns in MRSTY and OBS_STT. The columns listed in OBS_STT are used to fill their corresponding columns in MRSTY file.<br /> <br /> MRSTY OBS_STT<br /> CUI (Concept Unique ID) OBS_STT.conceptID<br /> TUI (term Unique ID) OBS_STT.localSemanticTypeID<br /> STY OBS_STT.semanticTypeName <br /> <br /> The sql command would fill correct values into MRSO table in database. <br /> <br /> '''Example entry row:'''<br /> <br /> C0027051|T047|Disease or Syndrome|<br /> <br /> <br /> '''4.st.raw:''' This file is primarily generated from the column semanticTypeName in the table OBS_STT. This file contains row of semantic type names and a short name for each of them. <br /> <br /> Example row:<br /> semantic Type Name|semantictypename<br /> <br /> <br /> <br /> == Setting up the workspace ==<br /> <br /> 1. Install the Lexical Variant Generator (LVG) before running<br /> MetaMap's install program. LVG is part of the Lexical Tools distribution and is available from the Lexical Systems Group (http://lexsrv3.nlm.nih.gov/SPECIALIST/Projects/lvg/current/index.html).<br /> <br /> 2. Before using MetaMap install program to install data file builder,<br /> You also need to add LVG's bin directory {LVG_DIR}/bin to your<br /> program path:<br /> # in Bourne Again Shell (bash)<br /> export PATH=$PATH:&lt;LVG_DIR&gt;/bin<br /> <br /> 3. Install MetaMap using instructions at their site <br /> <br /> 4. Unzip the DFB installation files in the same directory as MetaMap<br /> <br /> 5. Connect to the new directory created by extracting the distribution<br /> and invoke the install program:<br /> cd &lt;distribution directory&gt;<br /> ./bin/install.sh<br /> <br /> A sample run of the installation script follows:<br /> <br /> ^&lt;&lt;<br /> Enter basedir of installation [/nfsvol/nlsaux15/public_mm] &lt;user hits<br /> return to get the default&gt;<br /> Basedir is set to /nfsvol/nlsaux15/public_mm.<br /> <br /> The WSD Server requires Sun's Java Runtime Environment (JRE)<br /> Sun's Java Developer Kit (JDK) will work as well. if the<br /> command: &quot;which&quot; java returns /usr/local/jre1.4.2/bin/java, then the<br /> JRE resides in /usr/local/jre1.4.2/.<br /> <br /> Where does your distribution of Sun's JRE reside?<br /> Enter home path of JRE (JDK) [/usr]: /nfsvol/nls/tools/Linux-i686/java1.4.2<br /> Using /nfsvol/nls/tools/Linux-i686/java1.4.2 for JAVA_HOME.<br /> <br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/disambServer.cfg generated<br /> /nfsvol/nlsaux15/public_mm/WSD_Server/config/log4j.properties generated<br /> /nfsvol/nlsaux15/public_mm/bin/SKRrun generated.<br /> /nfsvol/nlsaux15/public_mm/bin/metamap07 generated.<br /> /nfsvol/nlsaux15/public_mm/bin/wsdserverctl generated.<br /> /nfsvol/nlsaux15/public_mm/bin/skrmedpostctl generated.<br /> Install complete.<br /> Would like to use a custom data set with MetaMap (use data file builder)? [yN]: &lt;user types y and return&gt;<br /> <br /> running Data File Builder Install...<br /> Is LVG installed? [yN] &lt;The user types y and return&gt;<br /> <br /> running Data File Builder Install...<br /> Enter home path of LVG [/nfsvol/nls/tools/Linux-i686/lvg2009]: &lt;user hits<br /> return to get the default&gt;<br /> <br /> Using /nfsvol/nls/tools/Linux-i686/lvg2009 for LVG_DIR.<br /> <br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.lvglab generated.<br /> /nfsvol/nlsaux15/public_mm/scripts/dfbuilder/mm_variants/0doit.xwords generated.<br /> Datafile Builder Setup is complete.<br /> % <br /> ^&gt;&gt;<br /> <br /> 6. Make sure that the SKR/MedPOST tagger is running; to run the tagger, move to the public_mm directory present inside the working directory and invoke<br /> ./bin/skrmedpostctl start<br /> <br /> Incase there is an error due to a port number, change the port number in the following files and try to run the tagger again:<br /> /bin/skrmedpostctl<br /> <br /> If you change the port number skrmedpostctl uses you need to change<br /> public_mm/bin/SKRrun (or SKRrun.in and re-run install.sh) to match the<br /> port you set in skrmedpostctl. The environment variables that should<br /> be changed in SKRrun are:<br /> TAGGER_SERVER_DEFAULT_TCP_PORT<br /> TAGGER_SERVER_TCP_PORT_0<br /> TAGGER_SERVER_TCP_PORT_1<br /> <br /> <br /> 7. Inside the directory public_mm/sourceData create a directory for workspace<br /> mkdir sourceData/09_custom<br /> <br /> Finally, Create a directory to store the knowledge sources:<br /> mkdir sourceData/09_custom/umls<br /> <br /> <br /> == Installation of Metamap ==<br /> <br /> <br /> 1. First, run the BuildDataFiles program as follows:<br /> <br /> $fullpath/public_mm/bin/BuildDataFiles<br /> <br /> 2. The file st.raw, located in $fullpath/public_mm/data/dfbuilder/2009 will need to be modified. Append the file generated in point 4 of Preparation of Data section to this file.<br /> <br /> 3. Move to $fullpath/public_mm/sourceData/09_custom/01metawordindex and execute the following scripts in order<br /> <br /> ./01CreateWorkFiles<br /> <br /> ./02Suppress<br /> <br /> ./03FilterPrep<br /> <br /> ./04FilterStrict<br /> <br /> ./05GenerateMWIFiles<br /> <br /> 4. Move to 02treecodes directory and run 01GenerateTreecodes<br /> <br /> cd ../02treecodes<br /> <br /> ./01GenerateTreecodes<br /> <br /> 5. Move to 03Variants directory and run 01GenerateVariants<br /> <br /> cd ../03Variants<br /> <br /> ./01GenerateVariants<br /> <br /> 6. cd ../04synonyms <br /> <br /> ./01GenerateSynonyms<br /> <br /> 7. cd ../05abbrAcronyms<br /> <br /> ./01GenerateAbbrAcronyms<br /> <br /> 8. move to $fullpath/public_mm and run<br /> <br /> ./bin/LoadDataFiles<br /> <br /> <br /> == Running MetaMap ==<br /> <br /> To run the newly installed MetaMap dfb, move to the main workspace folder (public_mm) and run the command below<br /> <br /> bin/SKRrun -L 2009 -M /DATA/XDR -B /BDB4 -w ./lexicon ./bin/metamap09.BINARY -Z 09_custom ./resources/input -I</div>

Tejaswit