-Trial and error to run "Mac-optimized TensorFlow and TensorFlow Addons" (https://github.com/apple/tensorflow_macos) on MacBook Air with M1 --Migrating your environment from the Intel architecture MacBook Pro (Early 2015) and looking for ways to adapt to the ARM-based architecture -(Mac-optimized) TensorFlow execution result is not included, up to the point of importing for the time being
Apple's new chip "M1" whose details were released at the Apple Event held from 2020/11/11 03:00 Japan time.
In the announcement, I was curious because "TensorFlow" was mentioned as a specific software name, but after that, Apple's GitHub Organization gave "Mac-optimized TensorFlow and TensorFlow Addons" (https:: //github.com/apple/tensorflow_macos) has been released. It is a pre-release driver that uses Apple's ML Compute framework.
I was just thinking about replacing it with a MacBook Pro (Early 2015), so I decided to buy a MacBook Air with an M1 chip and experiment with it. [^ 1]
[^ 1]: It was impressive that the announcement and reservation started on 2020/11/11 (JST), and I also decided to purchase it. If you are not good at games, animation, etc. and have some time to spare, please search for "THE IDOLM @ STER Hobby Programming Birthday". I haven't done much about "development" this time, but I was influenced by "Idolmaster-driven development" (reference: https://www.slideshare.net/treby/imas-driven-developemnt) in this respect. It may be said that it was.
There were some things that got stuck in moving tensorflow_macos from a Mac with an Intel architecture to a Mac with an ARM-based architecture, so I'd like to keep a record of it. [^ 2]
I'm late, but my name is @ forest1988. This is the first time I have posted to Qiita. I think that there are many parts that cannot be reached, but I would be grateful if you could take a look and find some useful parts.
There may be a part that is uselessly detoured because I wrote the process of trial and error, or an error due to my misunderstanding. If you have any questions, I would be very grateful if you could point them out.
[^ 2]: Regarding the operation of Intel architecture on Mac, Qiita article "Apple seems to have issued tensorflow for mac, so try installing it on intel's MacBook air" (https://qiita.com/notfolder/items) / b27cc00bd77eb1587832), which I used as a reference. Thank you.
Due to the migration of the environment, there are some differences from the initial state of macOS 11 (Big Sur) (for example, the shell of the terminal is not zsh but the bash used before macOS Catalina). ..
Therefore, those who use Big Sur with M1 in a clean state without migrating data, those who migrate from Windows machines, etc., and those who have created a completely different environment in the first place are different. Please note that there is a good chance that different results will be obtained if you do the same.
I was able to copy the python environment created on the MacBook Pro as it was, so I first tried to run it. Specifically, it is the state where anaconda3-2020.07 was installed in pyenv. The Python version was 3.8.3.
Follow the link and download tensorflow_macos-0.1alpha0.tar.gz at https://github.com/apple/tensorflow_macos/releases.
There are download and installation scripts in the GitHub repository, but this pre-release compressed file contains optimized TensorFlow etc. (which can be downloaded by script), so it's about 331.6MB in size. It has become.
First, create a virtual environment for tensorflow_macos using Python 3.8 venv.
I was worried that it would be strange to add more venv to anaconda of pyenv, but I decided to give it a try (as mentioned later, it was a problem before that).
$ /bin/bash install_venv.sh --prompt --python /path/to/anaconda3-2020.7/bin/python
Was executed, and the prompt on the way proceeded with the default settings. [^ 3]
A tensorflow_macos_venv directory will be created at the location specified at the prompt, and you can enter the created virtual environment by using / bin / activate here. [^ 4]
[^ 3]: zsh should be the default from macOS Catalina, so I'm curious if there's any reason to dare to use bash.
[^ 4]: The host name and user name that can be displayed in the setting of $ PS1 are replaced with primary_prompt_string, and the place where tensorflow_macos_venv is placed is replaced with / path / to /. We hope that you will read it according to your own environment. Especially for the former, I referred to the following article. Thank you: "Customize the output before $ in the [Mac] terminal" (https://www.yoheim.net/blog.php?q=20140309), "Prompt confirmation and settings --Pocketstudio.jp Linux Wiki "(http://pocketstudio.jp/linux/?%A5%D7%A5%ED%A5%F3%A5%D7%A5%C8%A4%CE%B3%CE%C7%A7%A4%E4" % C0% DF% C4% EA)
$ . /path/to/tensorflow_macos_venv/bin/activate
(tensorflow_macos_venv) (base) primary_prompt_string$ python
Python 3.8.3 (default, Jul 2 2020, 11:26:31)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
The README.md in the tensorflow_macos repository contains code snippets for selecting CPU and GPU, so I decided to give it a try first. Did.
# Import mlcompute module to use the optional set_mlc_device API for device selection with ML Compute.
from tensorflow.python.compiler.mlcompute import mlcompute
Let's try this with python, which we just launched in interactive mode.
>>> # Import mlcompute module to use the optional set_mlc_device API for device selection with ML Compute.
>>> from tensorflow.python.compiler.mlcompute import mlcompute
Traceback (most recent call last):
File "/path/to/tensorflow_macos_venv/lib/python3.8/site-packages/tensorflow/python/pywrap_tensorflow.py", line 64, in <module>
from tensorflow.python._pywrap_tensorflow_internal import *
ImportError: dlopen(/path/to/tensorflow_macos_venv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so, 6): no suitable image found. Did find:
/path/to/tensorflow_macos_venv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so: mach-o, but wrong architecture
/path/to/tensorflow_macos_venv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so: mach-o, but wrong architecture
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/path/to/tensorflow_macos_venv/lib/python3.8/site-packages/tensorflow/__init__.py", line 41, in <module>
from tensorflow.python.tools import module_util as _module_util
File "/path/to/tensorflow_macos_venv/lib/python3.8/site-packages/tensorflow/python/__init__.py", line 39, in <module>
from tensorflow.python import pywrap_tensorflow as _pywrap_tensorflow
File "/path/to/tensorflow_macos_venv/lib/python3.8/site-packages/tensorflow/python/pywrap_tensorflow.py", line 83, in <module>
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "/path/to/tensorflow_macos_venv/lib/python3.8/site-packages/tensorflow/python/pywrap_tensorflow.py", line 64, in <module>
from tensorflow.python._pywrap_tensorflow_internal import *
ImportError: dlopen(/path/to/tensorflow_macos_venv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so, 6): no suitable image found. Did find:
/path/to/tensorflow_macos_venv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so: mach-o, but wrong architecture
/path/to/tensorflow_macos_venv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so: mach-o, but wrong architecture
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.
With that in mind, I decided to try the other installation method. See INSTALLATION item in README.md of repository I ran the following script.
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/apple/tensorflow_macos/master/scripts/download_and_install.sh)"
The same error occurred again, and I checked the error message again and noticed the place called wrong architecture.
I suspected that the cause was that the anaconda I was using was compatible with Intel's architecture x86-64, so I decided to reorganize the Python environment.
First, I removed the PATH settings for pyenv and anaconda from ~ / .bash_profile and restarted terminal. You should no longer refer to pyenv, anaconda that you copied from the source machine.
After that, when I tried to re-install Python with Homebrew that was already included, I encountered the following error.
Error: Cannot install in Homebrew on ARM processor in Intel default prefix (/usr/local)!
Please create a new installation in /opt/homebrew using one of the
"Alternative Installs" from:
https://docs.brew.sh/Installation
You can migrate your previously installed formula list with:
brew bundle dump
Read "Alternative HCl" and re-install Homebrew as follows:
$ mkdir homebrew && curl -L https://github.com/Homebrew/brew/tarball/master | tar xz --strip 1 -C homebrew
$ mv homebrew/ /opt/
Add the following to ~ / .bashrc and
export PATH=/opt/homebrew/bin/:$PATH
Reflect using source.
$ source ~/.bashrc
Then use brew bundle dump to create a list of installed packages. [^ 5]
[^ 5]: Since it is necessary to do it with Homebrew that has already been installed, I think it would have been better to execute it before installing a new Homebrew without the hassle of specifying the full path.
$ /usr/local/Homebrew/bin/brew bundle dump
As a result, the following file was generated as Brewfile.
tap "cartr/qt4"
tap "homebrew/bundle"
...
<Abbreviation>
...
tap "sanemat/font"
brew "automake"
brew "[email protected]"
brew "[email protected]"
...
<Abbreviation>
...
brew "sanemat/font/ricty"
This is reflected as follows.
$ brew tap Homebrew/bundle
$ touch Brewfile
$ brew bundle --file Brewfile
However, I got the following warning and failed to install some packages.
Warning: You are running macOS on a arm64 CPU architecture.
We do not provide support for this (yet).
Reinstall Homebrew under Rosetta 2 until we support it.
You will encounter build failures with some formulae.
Please create pull requests instead of asking for help on Homebrew's GitHub,
Twitter or any other official channels. You are responsible for resolving
any issues you experience while you are running this
unsupported configuration.
I was able to install Python, but it was likely to cause the same problem as before, so I'll uninstall it and move on.
REQUIREMENTS mentions the use of Xcode Command Line Tools.
$ xcode-select --install
I put in Xcode Command Line Tools in. [^ 6]
I wondered if this would solve the Python problem ... but maybe because of the environment migration, Python 3.9 works for some reason.
With this, I was impatient that tensorflow_macos, which requires Python 3.8, does not work ... but there were two causes.
--Uninstallation with Homebrew failed.
--Python 3.9 remained in / usr / local / bin / python3, probably because of the environment migration.
Apparently, one of the causes was that Python installed with Homebrew couldn't be uninstalled successfully.
When I checked with which python, / opt / homebrew / bin // python3 was referenced. [^ 7]
Even from the date displayed when it is executed, it seems to be the one at the time of the above installation.
If you try to uninstall again,
$ brew uninstall python3
Error: Refusing to uninstall /opt/homebrew/Cellar/[email protected]/3.9.0_2
because it is required by itstool, libxml2 and sphinx-doc, which are currently installed.
You can override this and force removal with:
brew uninstall --ignore-dependencies python3
Error is displayed. Follow this instruction to uninstall.
$ brew uninstall --ignore-dependencies python3
Uninstalling /opt/homebrew/Cellar/[email protected]/3.9.0_2... (9,224 files, 137.5MB)
Now which python3 points to / usr / local / bin / python3.
However, this is also Python 3.9.0.
$ /usr/local/bin/python3
Python 3.9.0 (default, Nov 4 2020, 22:18:28)
[Clang 12.0.0 (clang-1200.0.32.21)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
Now, looking back at the Homebrew warning,
Error: Cannot install in Homebrew on ARM processor in Intel default prefix (/usr/local)!
Was written.
I used to use it without thinking in detail, but the fact that / usr / local is Intel's default prefix and does not support ARM processor means that
Where did the python in this / usr / local / bin come from?
Perhaps the Python that was included in Homebrew before the environment migration remains. Therefore, try uninstalling using Homebrew that was installed before the migration.
$ /usr/local/Homebrew/bin/brew uninstall python3
You will get the same dependency error as before, so add --ignore-dependencies and try again.
$ /usr/local/Homebrew/bin/brew uninstall --ignore-dependencies python3
This time the uninstall was successful and which python3 now points to / usr / bin / python3.
$ which python3
/usr/bin/python3
$ python3
Python 3.8.2 (default, Oct 2 2020, 10:45:41)
[Clang 12.0.0 (clang-1200.0.32.27)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
Now you're finally ready for your Python environment!
[^ 6]: For installing Xcode Command Line Tools, please refer to Qiita article: "Installing python3 with Xcode on Mac" (https://qiita.com/todotani/items/73877deea8773c316694). It was. Thank you.
[^ 7]: The "//" is not a clerical error, it was actually displayed as such. It seems that it was not good to add / after bin in export PATH = / opt / homebrew / bin /: $ PATH.
$ /bin/bash install_venv.sh --prompt --python=/usr/bin/python3
I was able to install it neatly without any problems.
$ . "/path/to/tensorflow_macos_venv/bin/activate"
(tensorflow_macos_venv) primary_prompt_string$ which python
/path/to/tensorflow_macos_venv/bin/python
(tensorflow_macos_venv) primary_prompt_string$ python
Python 3.8.2 (default, Oct 2 2020, 10:45:41)
[Clang 12.0.0 (clang-1200.0.32.27)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> # Import mlcompute module to use the optional set_mlc_device API for device selection with ML Compute.
>>> from tensorflow.python.compiler.mlcompute import mlcompute
>>>
This time I was able to import without error! But what about CPU and GPU usage?
>>> # Select CPU device.
>>> mlcompute.set_mlc_device(device_name='cpu')
>>> # Select GPU device.
>>> mlcompute.set_mlc_device(device_name='gpu')
WARNING:tensorflow:Eager mode on GPU is extremely slow. Consider to use CPU instead
There is a Warning about GPU, but it seems good to think that it is possible to specify CPU and GPU for the time being.
Since it is Eager mode on GPU is extremely slow, it is assumed that it will be executed in Define-and-Run of TensorFlow 1 system instead of Eager mode (Define-by-Run) which is the default in TensorFlow 2.0. Is it?
If you look at the README.md in the repository again, it says Please note that in eager mode, ML Compute will use the CPU., So in the above, it may be forcibly switched to CPU with Warning.
To test the true value of "Mac-optimized TensorFlow and TensorFlow Addons", it seems that you need to prepare code that is not in Eager mode.
With the goal of running "Mac-optimized TensorFlow and TensorFlow Addons" (https://github.com/apple/tensorflow_macos) on a MacBook Air with M1 Here is a summary of the parts that got caught in building a Python environment when migrating data from a Mac with an Intel processor to a Mac with an M1.
The bottom line is that you need to erase or reconfigure the Python environment that you created in the source environment so that it is not referenced. Since the architecture is different, it is natural to think about it, but I was wondering if I could use the environment that I had created at the migration source as much as possible, so I had a hard time unexpectedly.
Homebrew may not currently support ARM-based architectures, but given the notation We do not provide support for this (yet) ., this is a future. I think it will be resolved.
It is expected that M1 support will be promoted for other software as well, so it is expected that the transition from Macs with Intel processors to Macs with M1 will become easier and easier.
This article is just a record of attempts in the situation as of November 24, 2020, and the situation may change in the near future (perhaps even today).
In the first place, tensorflow_macos is also a pre-release, and I would like to look forward to what the official release version will look like.
(that's all)
Recommended Posts