MacQIIME Installation

MacQIIME Is Outdated & No-Longer Needed!

Since the QIIME pipeline was updated to version 2.0, MacQIIME is now outdated and is no-longer needed! Thanks to the QIIME developers, QIIME 2.x will now install on a Mac using the built-in automated installation via miniconda. For more information and installation instructions, check out qiime2.org!  If you're used to the MacQIIME 1.x command-line interface, I'd suggest following the q2cli > Installing QIIME 2 > Native Installation instructions on qiime2.org. You are still welcome to install MacQIIME if you like, but note that it is the old 1.9.1 version of QIIME from 2015.

It is also possible to have MacQIIME 1.9.1 and QIIME 2 installed at the same time, if you really want to. They shouldn't fight with each other, as long as you only have one of them sourced at any given time.

Backing up your old version

If you have an old/previous version of macqiime installed, you don't really have to back it up, because you can always just install it again from the archives.  But, backing things up instead of deleting them is always a safe way to go.  Anything you leave in the /macqiime directory will be deleted by the new installation, so if you've added any files to the old /macqiime directory you will want to make some copies of them or move them!  To move the whole directory to a new location, you can just use the mv command in the Terminal.  You will have to be a sudoer to do this:

# Backing up your old MacQIIME 1.9.0 installation

sudo mv /macqiime /macqiime-1.9.0-backup

Then, in the future, if you ever want to revert back to your old install of MacQIIME 1.9.0, for any reason, you can move things back like this:

# Reverting back to your old install of MacQIIME 1.9.0 for some reason

# Move the new macqiime 1.9.1 out of the way:

sudo mv /macqiime /macqiime-1.9.1-backup

# Rename your old backup of macqiime 1.9.0 to /macqiime:

sudo mv /macqiime-1.9.0-backup /macqiime

Installing the new MacQIIME 1.9.1

Installing MacQIIME should hopefully be pretty simple in any Mac with OS 10.7 and up. The only catch is that you have to be an administrative user with access to the root of the filesystem (i.e., you have to be an sudoer). Download the tgz archive and save to disk. Assuming it was saved to the Downloads folder on your Mac, open a Terminal and cd to that directory:

cd ~/Downloads

Check the MD5 sum of the file, to make sure the download wasn't corrupted. Assuming it's the latest version, the file should be named as in the following example and thus the command you would use is:

md5sum MacQIIME_1.9.1-20150604_OS10.7.tgz

  or

md5 MacQIIME_1.9.1-20150604_OS10.7.tgz

  (which ever one works, depends on which version of OS X you have.)

That command should return an output something like this:

  6634cb226d71ecf17d60483e6039f66a  MacQIIME_1.9.1-20150604_OS10.7.tgz

Specifically, please check to make sure the code is the same as the code returned on your system. If the code returned is different, that would indicate that your download was incomplete or corrupted.

Unarchive the file using the tar command.  

tar -xvf MacQIIME_1.9.1-20150604_OS10.7.tgz

You'll see a long list of the file contents scroll by as they are extracted from the archive.  Then, there should be a new directory called "MacQIIME_1.9.1-20150604_OS10.7" that you can cd into like this:

cd MacQIIME_1.9.1-20150604_OS10.7/

Once you are in that directory, you should find a script file called "install.s" -- all this file does, really, is copy the contents of the macqiime folder to your filesystem root, and then copy the "macqiime" script to /usr/bin/.  (If you know what you're doing, you can always just do this stuff yourself. However, the "macqiime" folder has to be at the root "/" of the filesystem; sorry for the inconvenience!)  You can run the install.s script like this:

./install.s

Follow the script's instructions.  Once it finishes, you should have a working install of QIIME!

One change as of 1.9.1: In older versions, the /macqiime folder used to belong to root. I'm now leaving the folder contents with ownership by the user who installed MacQIIME. This owner should then be able to use "conda" or "pip" to add or update python libraries. MacQIIME uses Anaconda python, so please try "conda" first instead of pip for updates or installations. If the package you want isn't available through conda, then go with pip. The Anaconda versions of software through conda are more likely than pip installs to work without problems.

Getting MacQIIME to work in El Capitan (OS 10.11)

This section is required only if you are running OS 10.11 (El Capitan) or higher. Apple added a new security feature in El Capitan that makes it impossible to install software into /usr/bin/ folder (that was where I was putting the “macqiime” sourcing script.  I'll fix this in the next release of MacQIIME. In the meantime, we can work around it. The quick solution is to use this command, instead of the “macqiime” script:

source /macqiime/configs/bash_profile.txt

That should give you a *working* terminal with MacQIIME enabled and everything should work.

For a more permanent fix, you can put the “macqiime” script somewhere else that’s in your PATH.  The /usr/local/bin folder works in OS 10.11.  To do this fix, assuming the unzipped macqiime installation folder is still in your Downloads folder:

1. In a terminal go into the MacQIIME installation folder

cd ~/Downloads/MacQIIME_1.9.1-20150604_OS10.7/

2. Make a copy of the macqiime script in the /usr/local/bin folder (requires administrative password)

sudo cp scripts/macqiime /usr/local/bin/macqiime

3. And make sure that new script is executable

sudo chmod a+x /usr/local/bin/macqiime

Assuming that all worked, you should now be able to run the “macqiime” script to source macqiime.

Delete the installation folder

Once you're finished installing MacQIIME as described above, hopefully without any errors (!!!), then you can delete the "MacQIIME_1.9.1-20150604_OS10.7" folder full of installation files. They've all been copied to the /macqiime folder on your computer.

Installing requirements and add-ons not included in MacQIIME

Important Requirement: XQuartz X11 libraries.  You probably need to install XQuartz.  If you get an error about not being able to load an X11 font library called something like libfont.so, that's due to missing X11 libraries on your system. It's entirely possible that even if you've installed this before, you may have to install it again. Sometimes folks have an outdated version of XQuartz, and sometimes an OS X update breaks the previous installation of XQuartz. Just download and install the latest one, and these problems should be fixed.

Optional Add-Ons

Due to licensing, logistics, and the fact that many people already have some of these items, the following are not included in the MacQIIME download. To gain the functionality of these additional programs/data in QIIME, all you have to do is install them. They can be installed anywhere on your system, as long as you make sure the binaries are in your PATH. (It is best not to put them in your /macqiime folder, because they may get deleted when you upgrade macqiime.) Please note that all of these add-ons are optional. They are also totally independent, meaning that if you only want one of them, you can just install the one you want.

Java. OS X no-longer includes Java, and when you try to use the RDP classifier, OS X may direct you to a Java download page which is the wrong download. For some reason OS X may think you need the Java web browser plugin (the JRE download), but that does not include the command-line "java" program. If you're missing Java on your system, you can get it here. To use the RDP Classifier, you need the "Java Development Kit" or "JDK." You do not need the JRE download. 

R  - Installing R on your system and adding the following libraries has become much more important in the latest QIIME versions 1.9+. There are now many neat features in QIIME that require R.  If you don't have R installed, you can get it from here. Please note that even if you installed R and these libraries previously for MacQIIME 1.8.0, you should still upgrade to/install the latest version of R, at least version 3.1.2, and re-install all these R packages to get everything working. Make sure the R executable is in your PATH.  You will have to open R and install additional packages: in the R command line, do the following, in the following order, one line at a time.

install.packages('randomForest')

install.packages('ape')

install.packages('vegan')

install.packages('optparse')

install.packages('gtools')

install.packages('klaR')

install.packages('RColorBrewer')

install.packages("biom")

source("http://bioconductor.org/biocLite.R")

biocLite("metagenomeSeq")

biocLite("DESeq2")

What it's used for in QIIME: There are now many statistical functions in QIIME that use these R libraries

BLAST - If you don't have legacy BLAST 2.2.22 installed, you can follow these instructions.

What it's used for in QIIME: you can use BLAST instead of the RDP Classifier 2.2 for taxonomy assignments. It's also required to run the ChimeraSlayer or "blast-fragments" methods of chimera identification, and a bunch of other random extras here-and-there.

USEARCH - If you don't have USEARCH installed, you can get it here. There are two required versions and you will need both in QIIME 1.9.0: USEARCH v5.2.236 and USEARCH 6.1, each required for DIFFERENT scripts, unfortunately. Name the 5.2.236 executable "usearch" and the 6.1 executable "usearch61" and make sure they're in your PATH.

What it's used for in QIIME: You can use USEARCH in pick_otus.py as an alternative clustering method. This includes chimera detection via the UCHIME methodology. Also needed for the rtax method in assign_taxonomy.py.

AmpliconNoise - If you don't have AmpliconNoise installed, you can get it here. To compile it, you will first need the GNU Science Library. Follow the instructions for compiling each of these packages, and put them somewhere convenient.  You will also have to add AmpliconNoise to your PATH and set the environment variables PYRO_LOOKUP_FILE and SEQ_LOOKUP_FILE as described in the QIIME instructions.  Note that you do not need AmpliconNoise to denoise 454 reads; there *is* a built-in alternative method called Denoiser. However, if you prefer to use AmpliconNoise instead of Denoiser, then you can install it in order to do so. If you've never compiled anything before in the terminal, this may be a learning experience.

What it's used for in QIIME: You can use AmpliconNoise instead of the default Denoiser for denoising 454 reads. I.e., if you want to use ampliconnoise.py instead of the default denoising method with the already built-in denoise_wrapper.py.

TopiaryExplorer - Available here, can be used to visualize some of your QIIME results files in the context of a phylogenetic tree.

Cytoscape - Available here, is needed to visualize OTU-sample networks created by make_otu_network.py. You can generate an example network in the QIIME Overview Tutorial.

Testing and using MacQIIME

Normally, when you run a command in the terminal, all the packages in MacQIIME will be invisible to your computer.  In order to access macqiime, after starting a new terminal session you have to source the environment variables with the "macqiime" command:

macqiime

Then you can run QIIME commands normally, like this (get help info for align_seqs.py):

align_seqs.py -h

Or, you can use print_qiime_config.py like this:

print_qiime_config.py -t

This should print out the results of a number of tests. The following may fail, if you haven't installed some of the additional software:

FAIL: test_ampliconnoise_install (if AmpliconNoise is not installed)

FAIL: test_blast_supported_version (if BLAST is not installed)

If you'd like to run the QIIME tutorial, you can make a copy of the tutorial data in your home directory like this:

cp -R /macqiime/QIIME/qiime_tutorial ~/

cd ~/qiime_tutorial/

Now you can look at the contents of the directory using the ls command:

ls -lh

This should show a number of files.

drwxr-xr-x+ 3 me  staff   102B Dec 15 13:48 18S_tutorial_files

-rw-r--r--+ 1 me  staff   454K Dec 15 13:48 Fasting_Example.fna

-rw-r--r--+ 1 me  staff   1.1M Dec 15 13:48 Fasting_Example.qual

-rw-r--r--+ 1 me  staff   2.1M Dec 15 13:48 Fasting_Example.sff

-rw-r--r--+ 1 me  staff   6.0M Dec 15 13:48 Fasting_Example.sff.txt

-rw-r--r--+ 1 me  staff   982B Dec 15 13:48 Fasting_Map.txt

These files, and the steps we'll take to analyze them, are described in the QIIME Overview Tutorial.  

If you want to get back to your normal environment variables, you can exit the macqiime subshell by typing "exit" and pressing return.

Learn more about the QIIME pipeline

You may be interested in this tutorial page, which includes a modification of the official QIIME Overview Tutorial. It goes through each of the pipeline steps separately, with Unix command-line help along the way, to help students understand the overall process in more detail.

Advanced MacQIIME hacking

This pertains to people who know what they're doing and want to incorporate MacQIIME into automated pipelines, or get it to work on a cluster, etc.

There is nothing special about the "macqiime" command - it just starts a BASH subshell with this file sourced:

/macqiime/configs/bash_profile.txt

and, in that bash_profile.txt file, among other things there's a statement that adds the environment variable 

export QIIME_CONFIG_FP=/macqiime/configs/qiime_config.txt

So, if you want to avoid having to use the "macqiime" command in order to run QIIME stuff through automated scripts, it's easy.  For example, here is a shell script that would incorporate a MacQIIME script without having to source it with "macqiime": let's call it AlignThese16SSeqs.script

#!/bin/bash

source /macqiime/configs/bash_profile.txt

align_seqs.py -i $1

That's it.  The $1 variable is the first argument given to the script.  So, if you had a file named My_16S_Seqs.fa, you could align them to the greengenes core using this script as follows:

AlignThese16SSeqs.script My_16S_Seqs.fa

You don't need to have MacQIIME sourced - the "source" line in the script does the same thing as the "macqiime" command.  And, remember that a bash script runs in its own subshell, which means that anything that got sourced in that script will be forgotten when the script ends.

Alternatively, if you want to have a user named "qiime" and make all the qiime software instantly available to anyone who logs in as "qiime," then just use these files for the .profile and .qiime_config files in that user's home directory:

/macqiime/configs/bash_profile.txt

/macqiime/configs/qiime_config.txt

If you put the macqiime bash profile data into a user's .profile file, you may want to edit one thing at least: either change or remove the lines for the QIIME_CONFIG_FP variable, and give the user their own qiime parameters file.

If you want to add anything to the MacQIIME "bottle" custom-style you can just edit the bash profile file accordingly. The case where this makes most sense would be if you wanted to add some kind of python library. If you want to add python libraries to macqiime, this is completely separate from any python libraries you have otherwise installed on your Mac. To add a python library to MacQIIME, just make sure you have MacQIIME sourced first before using "sudo setup.py install" etc.  I.e., if you type "which python" it should tell you that it's using the python in the /macqiime/anaconda/bin/ directory. The python site-packages folder in MacQIIME is located here:

/macqiime/anaconda/lib/python2.7/site-packages/

That's where all the scientific python libraries are located within MacQIIME. Notice that these are completely separate and different from your regular python install in OS X, and they're invisible when you don't have macqiime sourced in the command line.

Adding or Updating Python Stuff Within MacQIIME. If you'd like to add python libraries or update python libraries within MacQIIME, as of version 1.9.1 you'll be working with a built-in version of Anaconda Python.  Anaconda has a built-in package manager called "conda." While it is possible to use pip to install anything in Anaconda, the first choice (if available) is to first try conda. If the package is not available in conda, then go for pip. The anaconda folder within MacQIIME, while available to all users for running QIIME scripts etc, is owned by the user who installed MacQIIME. To install any new Python functionality within MacQIIME, it will be easiest to do so as that same user. Alternatively, you can change the ownership recursively of all the contents of the /macqiime folder so that they belong to another user via "chown." If you'd like the security of everything belonging to root only, you can do this:

sudo chown -R root /macqiime

As far as I know, that shouldn't break anything, but it might be inconvenient sometimes.

The one thing you cannot do, because it will break macqiime, is you cannot move it to be located anywhere else than the location it was installed to by the install.s script. It has to be available at root, in a folder called /macqiime/ However, if you want to physically store the software elsewhere, i.e. in a central location mounted by multiple machines, that's fine, as long as you have a symlink to its mount point from root on each machine, and the symlink needs to be named /macqiime

SourceTracker is available in the /macqiime/SourceTracker/ folder which is set as the SOURCETRACKER_PATH in /macqiime/configs/bash_profile.txt and you can update SourceTracker either by replacing the existing SourceTracker folder or by editing the /macqiime/configs/bash_profile.txt file to point to your new location for a more recent SourceTracker package.