[PYTHON] How to install your own (root) CA

I have to install my own root CA certificate and have summarized it. If you are using various tools and applications, you cannot say "it is safe to set it in the OS", and there are many things that need to be set individually in addition to the OS settings.

This article describes how to set your own root CA certificate for the following environments and tools:

1. Ubuntu (18.04)
2. Red Hat / CentOS (7)
3. Python - certifi/requests/ssl
4. Mozilla Firefox
5. Mozilla Thunderbird
6. Windows 10
7. Splunk

Ubuntu (18.04)

1. Create a suitable (not sloppy) directory under / usr / share / ca-certificates /. Here, it is * mylocal *. (Hereafter, it is referred to as / usr / share / ca-certificates / * mylocal * /) The directory is for organizing by management organization unit. (Probably there is a directory called mozilla /, but I won't touch it) You don't have to have a directory, but I think it's easier to manage local ones later if you organize them in a directory.

2. Place the certificate file under the created folder * mylocal *. Here, it is * mylocal-root-cacert.crt *. (/usr/share/ca-certificates/mylocal/mylocal-root-cacert.crt)

3. In /etc/ca-certificates.conf, add the relative path under/ usr / share / ca-certificates /of the added file. That is, in the above example, add "* mylocal * / * mylocal-root-cacert.crt *".

4. Execute ʻupdate-ca-certificates (/ usr / sbin / update-ca-certificates`).

5. Make sure you have a symbolic link to / usr / share / ca-certificates / * mylocal * / * mylocal-root-cacert.crt * under / etc / ssl / certs / Please give me.

Ubuntu18.CA update procedure in 04

# cd /usr/share/ca-certificates
# mkdir mylocal
# cp somewhere/mylocal-root-cacert.crt mylocal/
# cp -p /etc/ca-certificates.conf /etc/ca-certificates.conf.bak
# echo "mylocal/mylocal-root-cacert.crt" >> /etc/ca-certificates.conf
# update-ca-certificates
# ls -l /etc/ssl/certs/ | grep mylocal-root-cacert

As an aside, if this is done correctly, a CA bundle will be created with /etc/ssl/certs/ca-certificates.crt that contains all the CA certificates, so refer to this if necessary Then it would be good. For example, if you want to specify the path of the CA Bundle in the argument of requests () in python.

ʻUpdate-ca-certificates will automatically update /etc/ssl/certs/ca-certificates.crt`, so it is strictly forbidden to add it yourself.

Red Hat / CentOS (7)

See also: [Red Hat Enterprise Linux 7 | Red Hat Customer Portal-Using Shared System Certificates](https://access.redhat.com/documentation/ja-jp/red_hat_enterprise_linux/7/html/security_guide/sec-shared] -system-certificates)

Place the certificate file under / usr / share / pki / ca-trust-source / anchors / and run ʻupdate-ca-trust`.

# cp somewhere/mylocal-root-cacert.crt /usr/share/pki/ca-trust-source/anchors/
# update-ca-trust
# ls -l /etc/pki/ca-trust/extracted/openssl/

The same is true under / etc / pki / ca-trust / source / anchors /. / usr / share / pki is for backward compatibility and the setting of / etc / share / pki takes precedence. The certificate is placed under / usr / share / pki / ca-trust-source / anchors / so that it can be set for both.

Currently, giving an argument to ʻupdate-ca-trust is ignored, but if you look inside the script, the comment states that it will support" ʻupdate-ca-trust extract" in the future. ..

(CentOS7) /bin/update-ca-trust

#!/bin/sh

#set -vx

# At this time, while this script is trivial, we ignore any parameters given.
# However, for backwards compatibility reasons, future versions of this script must
# support the syntax "update-ca-trust extract" trigger the generation of output
# files in $DEST.

DEST=/etc/pki/ca-trust/extracted

# OpenSSL PEM bundle that includes trust flags
# (BEGIN TRUSTED CERTIFICATE)
/usr/bin/p11-kit extract --comment --format=openssl-bundle --filter=certificates --overwrite $DEST/openssl/ca-bundle.trust.crt
/usr/bin/p11-kit extract --comment --format=pem-bundle --filter=ca-anchors --overwrite --purpose server-auth $DEST/pem/tls-ca-bundle.pem
/usr/bin/p11-kit extract --comment --format=pem-bundle --filter=ca-anchors --overwrite --purpose email $DEST/pem/email-ca-bundle.pem
/usr/bin/p11-kit extract --comment --format=pem-bundle --filter=ca-anchors --overwrite --purpose code-signing $DEST/pem/objsign-ca-bundle.pem
/usr/bin/p11-kit extract --format=java-cacerts --filter=ca-anchors --overwrite --purpose server-auth $DEST/java/cacerts

Use the trust command to manage the configured trusted CA certificate.

trust command

$ trust
usage: trust command <args>...

Common trust commands are:
  list             List trust or certificates
  extract          Extract certificates and trust
  extract-compat   Extract trust compatibility bundles
  anchor           Add, remove, change trust anchors
  dump             Dump trust objects in internal format

See 'trust <command> --help' for more information

You can delete or change the configured CA by using the trust command, but it is omitted here.

Python - certifi/requests/ssl

**Caution! !! : Check the OS settings before adding the certificate for the Python environment. ** **

Summarize the settings of the certifi / requests / ssl module used by python.

python --CA settings for certifi / requests

If you are using requests, it depends on the certifi package (since requests 2.4, you are using certifi. See: CA Certificates /en/v2.7.0/user/advanced/#ca-certificates)), so if it is different from the OS reference, add it to the CA bundle of certifi.

From Requests version 2.4.0 onwards, Requests will attempt to use certificates from certifi if it is present on the system. This allows for users to update their trusted certificates without having to change the code that runs on their system.

You can check the reference destination as follows.

certifi/Confirmation of certificate bundle for requests

$ python3 -c "import certifi; print(certifi.where())"
/etc/ssl/certs/ca-certificates.crt

$ python3 -c "import requests;print(requests.__version__)"
2.22.0

$ python3 -c "import requests; print(requests.certs.where())"
/etc/ssl/certs/ca-certificates.crt

If you are referencing the same OS settings (for example, /etc/ssl/certs/ca-certificates.crt for Ubuntu 18.04 openssl), you do not need to add your own.

If you are referencing a file different from the OS (for example, in a virtual environment), add a text format certificate to the file in the displayed path and you're done.

I'm doing the same with the python script below. Place the installation certificate in the current directory as mylocal-root-cacert.crt and run the Python script.

local_ca_install.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import certifi

cabundle = certifi.where()
local_rootCA = 'mylocal-root-cacert.crt'

print( 'read from {}'.format( local_rootCA ) )
with open( local_rootCA, 'rb' ) as infile:
    myrootca = infile.read()

print( 'append to {}'.format( cabundle ) )
with open( cabundle, 'ab' ) as outfile:
    outfile.write( myrootca )

print( '{} has been imported.'.format( local_rootCA ) )

Execution result

(pythonsample) $ python local_ca_install.py
read from mylocal-root-cacert.crt
append to /home/.../envs/pythonsample/lib/python3.6/site-packages/certifi/cacert.pem
mylocal-root-cacert.crt has been imported.
(pythonsample) $

python --ssl settings

Check the reference destination of the ssl module. Even if the referenced directory is different, the symbolic link may refer to the directory / file of the same entity as requests / certifi, so check the referenced directory as well.

Confirmation of certificate bundle for ssl

$ python3 -c "import ssl; print(ssl.get_default_verify_paths())"; (set -x; ls -l /usr/lib/ssl)
DefaultVerifyPaths(cafile=None, capath='/usr/lib/ssl/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/lib/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/lib/ssl/certs')
+ ls -l /usr/lib/ssl
total 4
lrwxrwxrwx 1 root root   14 Apr 23  2018 certs -> /etc/ssl/certs
drwxr-xr-x 2 root root 4096 Nov 20 06:52 misc
lrwxrwxrwx 1 root root   20 Nov 13 01:58 openssl.cnf -> /etc/ssl/openssl.cnf
lrwxrwxrwx 1 root root   16 Apr 23  2018 private -> /etc/ssl/private

If the referenced entity is different, configure it to refer to the same configuration as the openssl directory or CA bundle.

python --Change by environment variable

Environment variables allow you to change the reference destination.

Environment variable REQUESTS_CA_BUNDLE --requests

See also: SSL Cert Verification (https://2.python-requests.org/en/v2.7.0/user/advanced/?highlight=requests_ca_bundle#ssl-cert-verification)

If you use * requests, the environment variable REQUESTS_CA_BUNDLE is set to CA Bundle (that is,/home/.../envs/pythonsample/lib/python3.6/site-packages/certifi/cacert.pem in the above example. By setting, requests () refers to that CA.

• Although it is not an environment variable, you can specify the path of the CA Bundle file as an argument of requests.get / post (requests.get (..., verify ='* path to the CA Bundle file *). ' ) is). However, be aware that if you hardcode the path, it will be less versatile.

• Setting verify to False is a last resort. We recommend that you keep it for a temporary test.

Environment variables SSL_CERT_DIR and SSL_CERT_FILE --ssl

• The environment variables referenced by the ssl module are SSL_CERT_DIR and SSL_CERT_FILE. (Although there is a lot of information on the net, I couldn't find the primary source for SSL_CERT_DIR and SSL_CERT_FILE)

Conda / Anaconda environment

The same is true for Conda / Anaconda.

If you are using Anaconda, add the certificate to the cabundle of certifi in the conda environment. On Windows etc., it may be necessary to display hidden files (folders) such as under AppData depending on the installation location.

python virtual environment

If you are using a Python virtual environment, you need to configure it in each virtual environment.

Mozilla Firefox

Firefox on Microsoft Windows is managed in a certificate store separate from Microsoft Windows (unconfirmed on Linux and MacOS), so it needs to be managed separately from the certificate settings for Microfoft Windows. (Unconfirmed, but it seems that it needs to be set for each profile)

[Options] --[Privacy and Security] --"Certificates"-[View Certificates ... (C)] --[Certificate Authority Certificates]--[Import (M). ..]is the certificate read.

If you're using Firefox, this explanation should suffice.

Firefox: How to audit & reset the list of trusted servers/CAs - Red Hat Customer Portal

Mozilla Thunderbird

Thunderbird is managed in a certificate store separate from Microsoft Windows (unconfirmed on Linux and MacOS), so it needs to be managed separately from setting the certificate on Microfoft Windows. (Unconfirmed, but it seems that it needs to be set for each profile)

[Option] --[Details] -- [Certificate] (Tab) --[Manage Certificate (M)]--[Certificate Authority Certificate]--[Import (M). ..]is the certificate read.

If you're using Thunderbird, this explanation should be sufficient.

Windows 10

Hold down the Windows key and press the " R "key. The "Run" dialog will appear. Enter certmgr.msc to execute it.

(If the UAC (User Access Control) dialog is displayed, select "Yes" to execute it.)

"Certificate-Current User" is displayed.

Right-click "Trusted Root Certification Authorities"-"Certificates" and click "All Tasks"-"Import ...".

The Certificate Import Wizard dialog is displayed. Follow the wizard to import the certificate.

Click Next and select the certificate to read under Browse.

Select the certificate and click Open. You will be returned to the original window, so click [Next].

In the "Certificate Store" selection, select "Place all certificates in the following store" and select "Trusted Root Certification Authorities" from the pull-down menu. Click [Next].

A confirmation screen will appear. If there are no mistakes, click [Finish].

When the message "Imported successfully" is displayed, the process is complete.

Close certmgr when you no longer need it.

As an aside, if you get a warning when using Remote Desktop in a local environment, instead of displaying it and importing it as it is, you can export it to a file and set it by the above method. , The warning will disappear. (I don't know the reason, but sometimes the warning did not disappear even if I displayed the certificate from the warning dialog and imported it as it was)

Splunk

In Splunk, you need to make the following settings for SSL communication.

1. Splunk Apps (python) preferences
2. Settings when Splunk becomes a client and connects to an external server
3. SSL settings provided by Splunk

The third setting is omitted here.

Splunk Apps (python) preferences

In the example below, it is SPLUNK_HOMOE = / opt / splunk.

$ /opt/splunk/bin/splunk cmd python -c "import requests; print(requests.certs.where())"
/opt/splunk/lib/python2.7/site-packages/certifi/cacert.pem
$ /opt/splunk/bin/splunk cmd python -c "import certifi; print(certifi.where())"
/opt/splunk/lib/python2.7/site-packages/certifi/cacert.pem
$ 
$ /opt/splunk/bin/splunk cmd python3 -c "import requests; print(requests.certs.where())"
/opt/splunk/lib/python3.7/site-packages/certifi/cacert.pem
$ /opt/splunk/bin/splunk cmd python3 -c "import certifi; print(certifi.where())"
/opt/splunk/lib/python3.7/site-packages/certifi/cacert.pem
$

I think we'll move to Python3 in the future, but for the time being, existing apps depend on python2, so we need to consider python2 as well.

Some Apps have their own cacert.pem in site-packages and refer to it, so you need to support these as well.

splunk Under the home cacert.pem Search example

$ (cd /opt/splunk; sudo find . -name cacert.pem -ls)
 49024396    276 -r--r--r--1 splunk splunk 282085 June 16 2019./lib/python2.7/site-packages/certifi/cacert.pem
 48893114    268 -r--r--r--1 splunk splunk 271088 August 9 03:32 ./lib/python2.7/site-packages/botocore/cacert.pem
 49024315    268 -r--r--r--1 splunk splunk 271088 August 9 03:32 ./lib/python2.7/site-packages/botocore/vendored/requests/cacert.pem
 49155480    276 -r--r--r--1 splunk splunk 282085 June 16 2019./lib/python3.7/site-packages/certifi/cacert.pem
 49026412    268 -r--r--r--1 splunk splunk 271088 August 9 03:32 ./lib/python3.7/site-packages/botocore/cacert.pem
 49155400    268 -r--r--r--1 splunk splunk 271088 August 9 03:32 ./lib/python3.7/site-packages/botocore/vendored/requests/cacert.pem
 39325129      4 -rw-------1 splunk splunk 1265 August 9 2018./etc/auth/cacert.pem
 41816513    344 -rw-r--r--1 splunk splunk 348799 February 20 2019./etc/apps/splunk_app_addon-builder/bin/splunk_app_add_on_builder/requests/cacert.pem
 41943696    344 -rw-r--r--1 splunk splunk 348799 February 20 2019./etc/apps/splunk_app_addon-builder/bin/ta_generator/resources_lib/requests/cacert.pem
 40370693    304 -rw-r--r--1 splunk splunk 308434 September 18 2018./etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages/pip/_vendor/requests/cacert.pem
 40371155    340 -rw-r--r--1 splunk splunk 344712 September 18 2018./etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/lib/python2.7/site-packages/requests/cacert.pem
$

Settings when Splunk becomes a client and goes to connect to an external server

If you want to match the settings to your OS, it's a good idea to look at /etc/ssl/certs/ca-certificates.crt (for Ubuntu).

${SPLUNK_HOME}/etc/system/local/server.conf

[sslConfig]
sslRootCAPath = /etc/ssl/certs/ca-certificates.crt

I couldn't find any description about how to set it officially. (Re-investigation required)