[PYTHON] Use cartopy without bugs in Google Colaboratory

If you want to do data science with Python (pandas or NumPy), the current de facto standard is to use matplotlib as the drawing library, and [for drawing geospatial data and maps It's best to use cartopy. The cartopy is 2020- when I try to use it with Google Colaboratory which is Jupyter notebook on the Google cloud. 04 Currently, the official method is plagued by bugs and is virtually unusable. Here is a summary of workarounds and their explanations.

Conclusion

Install with either of the following.

pattern 1

!grep '^deb ' /etc/apt/sources.list | \
  sed 's/^deb /deb-src /g' | \
  tee /etc/apt/sources.list.d/deb-src.list
!apt-get -qq update

!apt-get -qq build-dep python3-cartopy
!pip uninstall -y shapely

!pip install --no-binary cartopy cartopy==0.17.0

Pattern 2

!grep '^deb ' /etc/apt/sources.list | \
  sed 's/^deb /deb-src /g' | \
  tee /etc/apt/sources.list.d/deb-src.list
!apt-get update

!apt-get -qq build-dep python3-cartopy
!apt-get -qq remove python-shapely python3-shapely

!pip install --no-binary shapely shapely --force
!pip install --no-binary cartopy cartopy==0.17.0

I'm worried about bugs if it's the official method

The official Google Colaboratory notebook also how to install cartopy, OS packaging as follows: It is supposed to be possible using the system apt.

!apt-get -qq install python-cartopy python3-cartopy
import cartopy

However, the cartopy installed this way has a bug and it crashes with a simple code like this:

Problem 1: `gridlines ()` gives an error

I get an error with gridlines (), so I can't draw the grid.

Sample code:

import cartopy.crs as ccrs
import matplotlib.pyplot as plt

ax = plt.axes(projection=ccrs.PlateCarree())
ax.gridlines()
ax.coastlines()

error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-3e1ebed7d096> in <module>()
      3 
      4 ax = plt.axes(projection=ccrs.PlateCarree())
----> 5 ax.gridlines()
      6 ax.coastlines()

4 frames
/usr/local/lib/python3.6/dist-packages/matplotlib/ticker.py in _validate_steps(steps)
   2108         steps = np.asarray(steps)
   2109         if np.any(np.diff(steps) <= 0) or steps[-1] > 10 or steps[0] < 1:
-> 2110             raise ValueError('steps argument must be an increasing sequence '
   2111                              'of numbers between 1 and 10 inclusive')
   2112         if steps[0] != 1:

ValueError: steps argument must be an increasing sequence of numbers between 1 and 10 inclusive

Problem 2: Fall with `coastlines ()`

Drawing a coastline using coastlines () will result in "session crashed for unknown reason" with the following runtime log:

Sample code:

import cartopy.crs as ccrs
import matplotlib.pyplot as plt

ax = plt.axes(projection=ccrs.PlateCarree())
ax.coastlines()

Error log (runtime log):

Apr 14, 2020, 6:18:57 PM	WARNING	warn("IPython.utils.traitlets has moved to a top-level traitlets package.")
Apr 14, 2020, 6:18:57 PM	WARNING	/usr/local/lib/python3.6/dist-packages/IPython/utils/traitlets.py:5: UserWarning: IPython.utils.traitlets has moved to a top-level traitlets package.
Apr 14, 2020, 6:18:57 PM	WARNING	WARNING:root:kernel e5263554-c566-4983-aea5-418e4cb6441a restarted
Apr 14, 2020, 6:18:57 PM	INFO	KernelRestarter: restarting kernel (1/5), keep random ports
Apr 14, 2020, 6:18:54 PM	WARNING	python3: geos_ts_c.cpp:3991: int GEOSCoordSeq_getSize_r(GEOSContextHandle_t, const geos::geom::CoordinateSequence*, unsigned int*): Assertion `0 != cs' failed.

Cause of each problem

Question 1

The cause of problem 1 is that the installed cartopy is as old as v0.14.2. Although it has been reported in cartopy's Issue # 1374, the bug itself is PR # 773. It has been fixed in / cartopy / pull / 773) and has been fixed since v0.16.0 in February 2018.

By the way, v0.14.2 is the version released in April 2016, and the reason why such an old one is installed is because the OS is Ubuntu 18.04.3 LTS. As LTS says, it is a long-term supported version, and the first release, 18.04, was released in April 2018. Since it is a long-term support version, you can use it with confidence for a long time, but if it is minor software or software that is not stable yet, it is a little difficult that the version with the bug is packaged and used all the time. (See Package Information).

In [1]: cartopy.__version__

Out[1]: '0.14.2'

In [2]: !cat /etc/os-release

Out[2]:
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Problem 2

Problem 2 is that cartopy and the Shapely used by cartopy are using different versions of GEOS, causing incompatibilities. Reported as Issue # 871 and Issue # 1490 of cartopy Issue # 805 discusses the fundamental response, but it has not been resolved yet.

In Debian and Ubuntu, as the version of a library package goes up, the packages that depend on it will be rebuilt and should not be linked to a different version of the library. In fact, if you put cartopy with apt, the deb package version Shapely 1.6.4 provided by the OS will be put together with the cartopy linked to it, as shown below. If that Shapely is used, no problem will occur.

Deb packages entered with ʻapt-get install` at the beginning:

python-pkg-resources (39.0.1-2 Ubuntu:18.04/bionic [all])
python-pyshp (1.2.12+ds-1 Ubuntu:18.04/bionic [all])
python-shapely (1.6.4-1 Ubuntu:18.04/bionic [amd64])
python-six (1.11.0-2 Ubuntu:18.04/bionic [all])
python-cartopy (0.14.2+dfsg1-2build3 Ubuntu:18.04/bionic [amd64])
python3-pkg-resources (39.0.1-2 Ubuntu:18.04/bionic [all])
python3-pyshp (1.2.12+ds-1 Ubuntu:18.04/bionic [all])
python3-shapely (1.6.4-1 Ubuntu:18.04/bionic [amd64])
python3-six (1.11.0-2 Ubuntu:18.04/bionic [all])
python3-cartopy (0.14.2+dfsg1-2build3 Ubuntu:18.04/bionic [amd64])

However, as shown below, it seems that the latest version v1.7.0 of Shapely has been included in Google Colaboratory with pip from the beginning.

In [3]: !pip list | grep Shapely

Out[3]: Shapely                  1.7.0

If you simply import in this state, cartopy will be apt and Shapely will be pip. This is the cause of the incompatibility. It's a hassle.

In [4]:
import shapely
shapely.__version__

Out[4]: '1.7.0'

Digression: If only problem 2 is, the countermeasure is actually easy

Regarding problem 2, the combination of pip and apt is bad, so this is actually easy. If you execute the following before or after ʻapt-get install at the beginning, Shapely will use the one entered by apt (deb version) (ʻimport cartopy before pip uninstall And, note that it will be ʻimport shapely` in it).

!pip uninstall -y shapely

counter-measure

Since the cause is the above, the policy of countermeasures is as follows.

Install cartopy v0.16.0 or later (preferably the latest version v0.17.0)
Use Shapely and cartopy linked to the same GEOS

Specifically, do this.

Delete the Shapely entered with pip
Build the latest version of cartopy from source with pip using GEOS with Shapely linked

apt-get build-dep

When importing from source, dependent libraries such as GEOS must be included. It's a hassle to put them one by one, so this time I'll use ʻapt-get build-dep`.

ʻApt-get build-dep` is a command that installs the deb package needed to build the source package for the specified deb package. In this case, the environment required to build cartopy v0.17.0 will be prepared based on the build information of the source package of the deb package of cartopy v0.14.2. Of course, if there are changes in the dependencies between v0.14.2 and v0.17.0, this won't work, but if there aren't any major changes, it's okay (in fact, this time it worked).

Edit apt sources.list

Then, I want to go to ʻapt-get build-dep` immediately, but even if I run it suddenly, an error occurs.

In [5]: !apt-get build-dep cartopy

Out[5]:
Reading package lists... Done
E: Unable to find a source package for cartopy

ʻThe source package information used by apt-get build-depis/etc/apt/sources.list (and /etc/apt/sources.list.d/*`) which is the package acquisition source setting of apt. That's because it's commented out by default.

In [6]: !grep deb /etc/apt/sources.list

Out[6]:
deb http://archive.ubuntu.com/ubuntu/ bionic main restricted
# deb-src http://archive.ubuntu.com/ubuntu/ bionic main restricted
deb http://archive.ubuntu.com/ubuntu/ bionic-updates main restricted
# deb-src http://archive.ubuntu.com/ubuntu/ bionic-updates main restricted
deb http://archive.ubuntu.com/ubuntu/ bionic universe
# deb-src http://archive.ubuntu.com/ubuntu/ bionic universe
deb http://archive.ubuntu.com/ubuntu/ bionic-updates universe
# deb-src http://archive.ubuntu.com/ubuntu/ bionic-updates universe
deb http://archive.ubuntu.com/ubuntu/ bionic multiverse
# deb-src http://archive.ubuntu.com/ubuntu/ bionic multiverse
deb http://archive.ubuntu.com/ubuntu/ bionic-updates multiverse
# deb-src http://archive.ubuntu.com/ubuntu/ bionic-updates multiverse
deb http://archive.ubuntu.com/ubuntu/ bionic-backports main restricted universe multiverse
# deb-src http://archive.ubuntu.com/ubuntu/ bionic-backports main restricted universe multiverse
# deb http://archive.canonical.com/ubuntu bionic partner
# deb-src http://archive.canonical.com/ubuntu bionic partner
deb http://security.ubuntu.com/ubuntu/ bionic-security main restricted
# deb-src http://security.ubuntu.com/ubuntu/ bionic-security main restricted
deb http://security.ubuntu.com/ubuntu/ bionic-security universe
# deb-src http://security.ubuntu.com/ubuntu/ bionic-security universe
deb http://security.ubuntu.com/ubuntu/ bionic-security multiverse
# deb-src http://security.ubuntu.com/ubuntu/ bionic-security multiverse
deb https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/

That's the only cause, so you can solve it by simply preparing a file with the line starting with deb replaced with deb-src and updating the package information with ʻapt-get update`.

In [7]:
!grep '^deb ' /etc/apt/sources.list | \
  sed 's/^deb /deb-src /g' | \
  tee /etc/apt/sources.list.d/deb-src.list
!apt-get -qq update

This will also pass ʻapt-get build-dep` mentioned above.

Installation

After that, if you use pip install --no-binary to build without using the binary package, the procedure described at the beginning will be used. I think there are both cases where there is no problem with Shapely of apt and cases where you want to use the latest version originally included in Google Colaboratory, so I will show two patterns. You can change it depending on which Shapey you want to use.

Pattern 1: When using apt's deb package version Shapely (v1.6.4)

!grep '^deb ' /etc/apt/sources.list | \
  sed 's/^deb /deb-src /g' | \
  tee /etc/apt/sources.list.d/deb-src.list
!apt-get -qq update

!apt-get -qq build-dep python3-cartopy
!pip uninstall -y shapely

!pip install --no-binary cartopy cartopy==0.17.0

Pattern 2: When using the latest version of Shapely (v1.7.0) from the source

!grep '^deb ' /etc/apt/sources.list | \
  sed 's/^deb /deb-src /g' | \
  tee /etc/apt/sources.list.d/deb-src.list
!apt-get update

!apt-get -qq build-dep python3-cartopy
!apt-get -qq remove python-shapely python3-shapely

!pip install --no-binary shapely shapely --force
!pip install --no-binary cartopy cartopy==0.17.0

Operation check

Both Python programs in question 1 and 2 work fine. As shown below, Official gallery waves also works properly.

import matplotlib.pyplot as plt
import numpy as np

import cartopy.crs as ccrs


def sample_data(shape=(73, 145)):
    """Return ``lons``, ``lats`` and ``data`` of some fake data."""
    nlats, nlons = shape
    lats = np.linspace(-np.pi / 2, np.pi / 2, nlats)
    lons = np.linspace(0, 2 * np.pi, nlons)
    lons, lats = np.meshgrid(lons, lats)
    wave = 0.75 * (np.sin(2 * lats) ** 8) * np.cos(4 * lons)
    mean = 0.5 * np.cos(2 * lats) * ((np.sin(2 * lats)) ** 2 + 2)

    lats = np.rad2deg(lats)
    lons = np.rad2deg(lons)
    data = wave + mean

    return lons, lats, data


def main():
    fig = plt.figure(figsize=(10, 5))
    ax = fig.add_subplot(1, 1, 1, projection=ccrs.Mollweide())

    lons, lats, data = sample_data()

    ax.contourf(lons, lats, data,
                transform=ccrs.PlateCarree(),
                cmap='nipy_spectral')
    ax.coastlines()
    ax.set_global()
    plt.show()


main()

Postscript (2020-04-15): Return to the initial state

Since the contents (state) of the OS file system changes during installation, it is better to know how to return to the initial state when trying and erroring the installation. You can reset it by the following method.

Select "Reset Runtime to Factory Settings" from the "Runtime" menu

With "Restart Runtime", the code execution state is in the initial state (nothing is in memory), but the file system remains changed.