Install Hugging Face Transformers on Apple M1

Along with Tensorflow and Tokenizers Package

Photo by david latorre romero on Unsplash

Not all things comes easy. The transition to Apple M1 has a similar story to tell. Even though I love the speed, I hate going to have to find non-traditional ways to install traditional libraries that otherwise would have been just one line on command prompt.

If you are an Apple M1 user, and work closely with NLP, there are chances that you’ve encountered this before or even found a solution, but if not, or you recently joined the M1 party, this article provide a way you can install Hugging Face transformer on your MacBook with M1 chip. And no, it is not pip install transformers.

There are three steps to get transformers up and running.

1. Install Tensorflow
2. Install Tokenizers Package (with Rust Compilers)
3. Install Transformers Package

Although it works, please consider researching for more reliable ways to install transformers — written on 2021.10.25

1. Tensorflow

Installing tensorflow is easy, you just have to point to your existing virtual env, or you can create a new one to play around, or it will create one for you if you are just not feeling over the moon to create a new one.

1. Create and Enable Virtual Environment

Install the virtualenv package if you don’t already have one

pip install virtualenv

Create virtualenv by typing the following command

python virtualenv venv --python=python3

To enable the virtual env

source venv/bin/activate

2. Download

Download the latest release from here. Make sure you download the tar.gz file which contains the installation script. At the time of writing this article, the latest release is tensorflow-macos 0.1 alpha 3. Yes, it is tensorflow-macos, and not tensorflow.

3. Install

Once you have the script, go to the download directory and run the following command ./install_venv.sh --prompt. It will ask you for the path to your virtual environment folder, provide the one that we created. You can also let it create another one, but in my opinion it is always better to create the one we intend to use, or the one that we have already.

Enter the path for virtual environment

Successfully installed tensorflow-macos

Note — You can leverage M1 to accelerate training of your machine learning model with tensorflow-metal. If that’s the case for you, learn more about it here.

2. Tokenizers

We will be building Tokenizers from source to avoid any interruptions, which I am sure will be there if we decide to go otherwise. There are a few steps here so be precise when you follow along.

2.1 Install Rust language

We need to have Rust installed to install tokenizers from source. You can do so by typing the following command

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

You can configure your current shell as per the instruction on the command prompt after installation, or you could just open another session to see it in effect.

2.2 Instal Tokenizers

Once we have Rust, we can download the source for tokenizers

git clone https://github.com/huggingface/tokenizers

Go to the python bindings folder cd tokenizers/bindings/python

Make sure you have virtual environment installed and activated, and then type the following command to compile tokenizers

pip install setuptools_rust

And finally, install tokenizers

python setup.py install

3. Transformers

And now finally, you can install transformers

pip install git+https://github.com/huggingface/transformers

Voila, successful installation of transformers

End Notes

Lately I’ve been trying to figure out how to install many things on Apple M1, and while I figure out, I am trying to jot it down to make it easier for other who might have the same problem. Hope this helped you, if you have any feedback or are struggling with any other installation, feel free to reach out, maybe I have a solution but haven’t got a chance to write it down. Happy Learning.