[PYTHON] Summary of Proxy connection settings

I always forget how to set up Proxy, so I'll summarize-my memo. As a side note, supplementary items regarding settings are also described.

1. CheatSheet
2. Details (Supplementary information)
3. Python
   1. Python - requests
   2. conda/Anaconda
4. apt / apt-get (Ubuntu / Debian series)
5. yum (CentOS / Red Hat series)
6. curl
7. wget
8. Splunk
9. docker

CheatSheet

What can be specified by environment variables

Although it differs between uppercase environment variables and lowercase environment variables, if you mainly specify the following four, you can cover some commands.

export HTTP_PROXY=http://username:[email protected]:8080
export http_proxy=${HTTP_PROXY}
export HTTPS_PROXY=https://username:[email protected]:8080
export https_proxy=${HTTPS_PROXY}

Various configuration files

conda/Anaconda

~/.condarc

proxy_servers:
    http: http://proxy.example.com:8080
    https: https://proxy.example.com:8080

apt/apt-get

/etc/apt/apt.conf

Acquire::http::Proxy "http://proxy.example.com:8080/";
Acquire::https::Proxy "https://proxy.example.com:8080/";
Acquire::ftp::Proxy "https://proxy.example.com:8080/";

yum

/etc/yum.conf

[main]
...
proxy = https://proxy.example.com:8080

curl

~/.curlrc

proxy = protocol://username:[email protected]:port

wget

~/.wgetrc or /etc/wgetrc

http_proxy = http://proxy.example.com:8080/
https_proxy = https://proxy.example.com:8080/
ftp_proxy = http://proxy.example.com:8080/

Splunk

${SPLUNK_HOME}/etc/system/local/server.conf

[proxyConfig]
http_proxy =  http://proxy.example.com:8080
https_proxy = https://proxy.example.com:8080

docker

json:~/.docker/config.json

{
 "proxies":
 {
   "default":
   {
     "httpProxy": "http://proxy.example.com:8080",
     "httpsProxy": "http://proxy.example.com:8080",
     "noProxy": "*.test.example.com,.example2.com"
   }
 }
}

The docker daemon is set on the systemctl side. See also: Control Docker with systemd | Docker Documentation

ini:/etc/systemd/system/docker.service.d/http-proxy.conf (Create New)

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:8080/"
Environment="HTTPS_PROXY=https://proxy.example.com:8080/"

ini:/etc/systemd/system/docker.service.d/http-proxy.conf (Create New)

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:8080/" "NO_PROXY=localhost,127.0.0.1,docker-registry.example.com,.corp"
Environment="HTTPS_PROXY=https://proxy.example.com:8080/" "NO_PROXY=localhost,127.0.0.1,docker-registry.example.com,.corp"

Details (Supplementary information)

Python

Python - requests See also: Proxies --Advanced Usage — requests-docs-ja 1.0.4 documentation

• Specify the dictionary type in the argument proxies of requests.get () / post ().
• Either 'http' or 'https' must be specified.
• Currently, there is no'no_proxy'(exception specification that does not use Proxy).
• Use HTTP_PROXY and HTTPS_PROXY when specifying from outside with environment variables.

Proxies parameter specification(Example 1)

my_proxies = {
    'http':  '10.0.0.1:8080',
    'https': '10.0.0.1:8080',
}

requests.get(..., proxies = my_proxies )

Proxies parameter specification(Example 2)

my_proxies = {
    'http':  'https://user:[email protected]:8080',
    'https': 'https://user:[email protected]:8080',
}

requests.get(..., proxies = my_proxies )

Below is a sample program. A case where an original CA is used is also assumed.

requests_sample.py

import requests
import os

#For cases where all TLS connections are decrypted by the Proxy as a TLS Forward Proxy
ca_verify_file = '/some/where/local_ca.crt'
ca_verify = True

#Default Proxy to set
my_http_proxy  = 'https://proxy.example.com:8080'
my_https_proxy = 'https://proxy.example.com:8080'

#See environment variables
http_proxy  = os.getenv( 'HTTP_PROXY', default=my_http_proxy )
https_proxy = os.getenv( 'HTTPS_PROXY', default=my_https_proxy )

#Generate Proxy parameters to pass to requests
proxies = {
    'http':  http_proxy,
    'https': https_proxy,
}

#Target URL
target_url = 'https://www.google.co.jp/search'

# Payload (Here is the query)
payload = {
    'q': 'python requests proxy'
}

try:
    #Proxy is proxies=Pass dictionary type
    r = requests.get(target_url, params=payload, proxies=proxies, verify=ca_verify)
except requests.ConnectionError as e:
    if type(e) == requests.exceptions.SSLError:
        #Specify a local root CA certificate file
        # ca_verify_Do not specify file directly later ca_By inspecting verify
        #To be able to check which mode you called
        ca_verify = ca_verify_file
        r = requests.get(target_url, params=payload, proxies=proxies, verify=ca_verify)
    else:
        raise e

#Display of acquired contents
print(r.text)

In the sample program above, the environment variables are explicitly referenced and set, but if the proxies parameter is not specified, the environment variables will be referenced in requests.get () / post ().

Conda / Anaconda

There are two ways to set conda, one is to specify the proxy in the configuration file, and the other is to set it with the conda command.

When editing .condarc directly

See also: Configure conda for use behind a proxy server (proxy_servers) --Using the .condarc conda configuration file — conda documentation /use-condarc.html#config-proxy)

The configuration file for conda is ~ / .conda. .condarc is written in YAML format.

~/.condarc Setting example 1

proxy_servers:
    http: https://proxy.example.com:8080
    https: https://proxy.example.com:8080

To change the Proxy server depending on the connection destination, write as follows.

~/.condarc Setting example 2

proxy_servers:
  'http://10.20.1.128': 'http://10.10.1.10:5323'

Setting by conda command

Reference: conda config --Command Reference

When configuring with the conda command, use the conda config command, but you cannot specify a YAML structured setting because the --set option can only specify a Boolean or a string. pattern.

In this case, use the --stdin option to input from standard input in YAML format. See proxy_servers with the --show option for confirmation.

MS On Windows conda Setting example by config command 1

(base) C:\Users\localuser>conda config --stdin
proxy_servers:
  http: https://proxy.example.com:8080
  https: https://proxy.example.com:8080
^Z

(base) C:\Users\localuser>conda config --show proxy_servers
proxy_servers:
  http: https://proxy.example.com:8080
  https: https://proxy.example.com:8080

(base) C:\Users\localuser>

However, direct input is not realistic because it causes input errors, so it is better to prepare a file in advance and pass it to standard input.

inputfile.txt

proxy_servers:
  http: https://proxy.example.com:8080
  https: https://proxy.example.com:8080

MS On Windows conda Setting example 2 by config command

(base) C:\Users\localuser>type inputfile.txt
proxy_servers:
  http: https://proxy.example.com:8080
  https: https://proxy.example.com:8080

(base) C:\Users\localuser>type inputfile.txt | conda config --stdin

(base) C:\Users\localuser>conda config --show proxy_servers
proxy_servers:
  http: https://proxy.example.com:8080
  https: https://proxy.example.com:8080

(base) C:\Users\localuser>

If you prepare a file, it seems better to edit ~ / .condarc directly, but when managing by batch processing or Ansible, prepare a file for the setting part in advance and use it. Can be considered.

Specify the --system option if you want to configure system settings instead of personal settings.

apt / apt-get (Ubuntu / Debian series)

See also: Configuration Options --6.2. aptitude, apt-get, and apt Commands

The apt / apt-get configuration file can be found on the man page (ʻapt.conf (5) `).

• The default configuration file for apt / apt-get is /etc/apt/apt.conf
• The configuration file can be changed with the environment variable ʻAPT_CONFIG`.
• The setting file name includes alphanumeric characters, hyphens (-), underscores (\ _), and periods (.).
• Http_proxy and https_proxy when specified by environment variables

Each line

group::tool::directive "value";

It is in the form of. Quotation marks (") and final semicolon (;) are required. The following three writing styles have the same settings.

group {
   tool {
     directive1 "value1";
     directive2 "value2";
   };
};

group::tool {
  directive1 "value1";
  directive2 "value2";
};

group::tool::drective1 "value1";
group::tool::drective2 "value2";

The proxy settings are set in http, https, and ftp in the ʻAcquiregroup for downloading packages. Instructions forhttp and httpscan be found in the ʻapt-transport-http (1)and ʻapt-transport-https (1)` man pages.

man apt.conf(5)

       ftp
           ftp::Proxy sets the default proxy to use for FTP URIs. It is
           in the standard form of ftp://[[user][:pass]@]host[:port]/.
           Per host proxies can also be specified by using the form
           ftp::Proxy::<host> with the special keyword DIRECT meaning
           to use no proxies. If no one of the above settings is
           specified, ftp_proxy environment variable will be used. To
           use an FTP proxy you will have to set the ftp::ProxyLogin
           script in the configuration file. This entry specifies the
           commands to send to tell the proxy server what to connect
           to.The following is omitted

• If it is not set, it refers to the environment variable ftp_proxy.

man apt-transport-http(1)

   Proxy Configuration
       The environment variable http_proxy is supported for system wide
       configuration. Proxies specific to APT can be configured via the
       option Acquire::http::Proxy. Proxies which should be used only
       for certain hosts can be specified via
       Acquire::http::Proxy::host. Even more finegrained control can be
       achieved via proxy autodetection, detailed further below. All
       these options use the URI format
       scheme://[[user][:pass]@]host[:port]/. Supported URI schemes are
       socks5h (SOCKS5 with remote DNS resolution), http and https.
       Authentication details can be supplied via apt_auth.conf(5)
       instead of including it in the URI directly.

       The various APT configuration options support the special value
       DIRECT meaning that no proxy should be used. The environment
       variable no_proxy is also supported for the same purpose.

The following is omitted

man apt-transport-https(1)

OPTIONS
       The HTTPS protocol is based on the HTTP protocol, so all options
       supported by apt-transport-http(1) are also available via
       Acquire::https and will default to the same values specified for
       Acquire::http. This manpage will only document the options
       unique to https.

• ʻAcquire :: http :: Proxy` to set Proxy.
• Use ʻAcquire :: http :: Proxy :: host` to set for each connection destination (* host * is the connection destination).
• The http option is the same for https.
• Environment variable is http_proxy (https_proxy is not described in any man page, but it is empirically valid)

Environment variable setting example

$ export ftp_proxy="https://proxy.example.com:8080"
$ export http_proxy="https://proxy.example.com:8080"
$ export https_proxy="https://proxy.example.com:8080"
$ export no_proxy="192.168.0.1,172.17.0.3,10.0.0.5"

For system-wide permanent settings, add the following to /etc/apt/apt.conf.

/etc/apt/apt.conf

Acquire::ftp::Proxy "https://proxy.example.com:8080/";
Acquire::http::Proxy "https://proxy.example.com:8080/";
Acquire::https::Proxy "https://proxy.example.com:8080/";

yum (CentOS / Red Hat series)

See also: yum --Trac

• How to specify with command line options
• How to specify in the configuration file

There is.

When specified as a command line option

• Use --setopt if specified as an option.

--setopt=proxy=https://proxy.example.com:8080

man yum(8)

       --setopt=option=value
              Set  any  config option in yum config or repo files. For options
              in the global config just use:  --setopt=option=value  for  repo
              options use: --setopt=repoid.option=value

• --setopt specifies proxy or repositoryid.proxy of[main]. The description on the man page is as follows.

man yum.conf(5)

DESCRIPTION
       Yum uses a configuration file at /etc/yum.conf.

       Additional  configuration  files are also read from the directories set
       by the reposdir option (default is `/etc/yum.repos.d').  See the repos-
       dir option below for further details.
<Omitted>
[main] OPTIONS
       The  [main]  section  must exist for yum to do anything. It consists of
       the following options:
<Omitted>
        proxy URL to the proxy server that yum should use.  Set this  to
              `libproxy'  to  enable  proxy  auto  configuration via libproxy.
              Defaults to direct connection.

        proxy_username username to use for proxy

        proxy_password password for this proxy
<Omitted>

[repository] OPTIONS
       The repository section(s) take the following form:

              Example: [repositoryid]
              name=Some name for this repository
              baseurl=url://path/to/repository/

              repositoryid  Must  be  a  unique  name for each repository, one
              word.
<Omitted>

       proxy  URL  to  the  proxy  server  for  this repository. Set to
              '_none_' to disable the global proxy setting  for  this  reposi-
              tory. If this is unset it inherits it from the global setting

       proxy_username  username  to use for proxy.  If this is unset it
              inherits it from the global setting

       proxy_password password for this proxy.  If  this  is  unset  it
              inherits it from the global setting

(The man page output has been slightly modified for readability)

When specifying in the configuration file

Permanent settings are described in /etc/yum.conf.

/etc/yum.conf

[main]
...
proxy = https://proxy.example.com:8080

curl See also man page

• How to specify with command line arguments
• How to specify in the configuration file
• How to specify with environment variables

When specified by command line argument

The specification in the argument is as follows.

curl -x https://proxy.example.com:8080 ...

The explanation on the man page is as follows.

man curl(1)

   -x, --proxy [protocol://]host[:port]
              Use the specified proxy.

              The proxy string can be specified with a protocol:// pre‐
              fix.  No protocol specified or http:// will be treated as
              HTTP  proxy.  Use  socks4://,  socks4a://,  socks5://  or
              socks5h://  to  request  a  specific  SOCKS version to be
              used.  (The protocol support was added in curl 7.21.7)

              HTTPS proxy support  via  https://  protocol  prefix  was
              added in 7.52.0 for OpenSSL, GnuTLS and NSS.

              Unrecognized  and  unsupported  proxy  protocols cause an
              error since 7.52.0.  Prior versions may ignore the proto‐
              col and use http:// instead.

              If  the port number is not specified in the proxy string,
              it is assumed to be 1080.

              This option overrides existing environment variables that
              set  the proxy to use. If there's an environment variable
              setting a proxy, you can set proxy to "" to override it.

              All operations that are performed over an HTTP proxy will
              transparently be converted to HTTP. It means that certain
              protocol specific operations might not be available. This
              is  not  the case if you can tunnel through the proxy, as
              one with the -p, --proxytunnel option.

              User and password that might be  provided  in  the  proxy
              string  are  URL decoded by curl. This allows you to pass
              in special characters such as @ by using %40 or pass in a
              colon with %3a.

              The proxy host can be specified the exact same way as the
              proxy environment variables, including the protocol  pre‐
              fix (http://) and the embedded user + password.

              If  this  option is used several times, the last one will
              be used.

You can see that you can specify it with -x or --proxy.

When specifying in the configuration file

If you want to set it constantly, put the following in .curlrc in your home directory. (_Curlrc on Windows)

~/.curlrc

proxy = protocol://username:[email protected]:port

The description on the man page is as follows.

man curl(1)

              The default  config  file  is checked for in the following
              places in this order:

              1) curl tries to find the "home dir": It first  checks  for  the
              CURL_HOME and then the HOME environment variables. Failing that,
              it uses getpwuid() on Unix-like systems (which returns the  home
              dir  given the current user in your system). On Windows, it then
              checks for the APPDATA variable, or as a last resort the '%USER‐
              PROFILE%\Application Data'.

              2)  On  windows, if there is no _curlrc file in the home dir, it
              checks for one in the same dir the curl executable is placed. On
              Unix-like  systems,  it will simply try to load .curlrc from the
              determined home dir.

When specifying with an environment variable

Use http_proxy, HTTPS_PROXY, url-protocol_PROXY (such as FTP_PROXY), ʻALL_PROXY, and NO_PROXYto set in environment variables. Note that onlyhttp_proxy` is in lowercase.

man curl(1)

ENVIRONMENT
       The environment variables can be  specified  in  lower  case  or
       upper case. The lower case version has precedence. http_proxy is
       an exception as it is only available in lower case.

       Using an environment variable to set  the  proxy  has  the  same
       effect as using the -x, --proxy option.

       http_proxy [protocol://]<host>[:port]
              Sets the proxy server to use for HTTP.

       HTTPS_PROXY [protocol://]<host>[:port]
              Sets the proxy server to use for HTTPS.

       [url-protocol]_PROXY [protocol://]<host>[:port]
              Sets  the  proxy  server to use for [url-protocol], where
              the protocol is a protocol  that  curl  supports  and  as
              specified  in  a  URL.  FTP, FTPS, POP3, IMAP, SMTP, LDAP
              etc.

       ALL_PROXY [protocol://]<host>[:port]
              Sets the proxy server  to  use  if  no  protocol-specific
              proxy is set.

       NO_PROXY <comma-separated list of hosts>
              list  of  host names that shouldn't go through any proxy.
              If set to a asterisk '*' only, it matches all hosts.

              Since 7.53.0, this environment variable disable the proxy
              even   if   specify   -x,   --proxy   option.   That   is
              NO_PROXY=direct.example.com  curl  -x  http://proxy.exam‐
              ple.com http://direct.example.com accesses the target URL
              directly,   and   NO_PROXY=direct.example.com   curl   -x
              http://proxy.example.com     http://somewhere.example.com
              accesses the target URL through proxy.

wget See also man page

• How to specify with command line options
• How to specify with environment variables
• How to specify in the configuration file

There is.

When specified as a command line option

Specify using the -e option

wget-e option

$ wget -e http_proxy=http://proxy.example.com:8080 ...
Connecting to proxy.example.com:8080... connected.
Proxy request sent, awaiting response... 200 OK
...

The description on the man page is as follows.

man wget(1)

OPTIONS
<Omitted>
   Basic Startup Options
<Omitted>
       -e command
       --execute command
           Execute command as if it were a part of .wgetrc.  A command thus
           invoked will be executed after the commands in .wgetrc, thus taking
           precedence over them.  If you need to specify more than one wgetrc
           command, use multiple instances of -e.

If you need basic authentication, --proxy-user=user --proxy-pasword=password Is specified as an option.

When specifying with an environment variable

• Specified by the environment variables * _proxy (http_proxy, https_proxy, ftp_proxy, no_proxy).

man wget(1)

ENVIRONMENT
       Wget supports proxies for both HTTP and FTP retrievals.  The standard
       way to specify proxy location, which Wget recognizes, is using the
       following environment variables:

       http_proxy
       https_proxy
           If set, the http_proxy and https_proxy variables should contain the
           URLs of the proxies for HTTP and HTTPS connections respectively.

       ftp_proxy
           This variable should contain the URL of the proxy for FTP
           connections.  It is quite common that http_proxy and ftp_proxy are
           set to the same URL.

       no_proxy
           This variable should contain a comma-separated list of domain
           extensions proxy should not be used for.  For instance, if the
           value of no_proxy is .mit.edu, proxy will not be used to retrieve
           documents from MIT.

When specifying in the configuration file

Permanently listed in ~ / .wgetrc (individual), / etc / wgetrc (system).

~/.wgetrc or /etc/wgetrc

https_proxy = http://proxy.example.com:8080/
http_proxy = http://proxy.example.com:8080/
ftp_proxy = http://proxy.example.com:8080/

man wget(1)

FILES
       /etc/wgetrc
           Default location of the global startup file.

       .wgetrc
           User startup file.

The man page doesn't say ~ / .wgetrc, but it can be read by putting .wgetrc in your home directory.

Splunk See also: Configure splunkd to use your HTTP Proxy Server --Splunk Documentation](https://docs.splunk.com/Documentation/Splunk/latest/Admin/ConfigureSplunkforproxy)

Set in server.conf

${SPLUNK_HOME}/etc/system/local/server.conf

[proxyConfig]
http_proxy = <string that identifies the server proxy. When set, splunkd sends all HTTP requests through this proxy server. The default value is unset.> 
https_proxy = <string that identifies the server proxy. When set, splunkd sends all HTTPS requests through the proxy server defined here. If not set, splunkd uses the proxy defined in http_proxy. The default value is unset.>  
no_proxy = <string that identifies the no proxy rules. When set, splunkd uses the [no_proxy] rules to decide whether the proxy server needs to be bypassed for matching hosts and IP Addresses. Requests going to localhost/loopback address are not proxied. Default is "localhost, 127.0.0.1, ::1">

docker See also: Configure Docker to use a proxy server | Docker Documentation See also: Control Docker with systemd | Docker Documentation

The proxy setting of the docker client is set to ~ / .docker / config.json.

json:~/.docker/config.json

{
 "proxies":
 {
   "default":
   {
     "httpProxy": "http://127.0.0.1:3001",
     "httpsProxy": "http://127.0.0.1:3001",
     "ftpProxy": "http://127.0.0.1:3001",
     "noProxy": "*.test.example.com,.example2.com"
   }
 }
}

(Quoted from Configure Docker to use a proxy server | Docker Documentation)

The proxy setting of dockerd is done on the systemctl side. (Not set in daemon.json)

ini:/etc/systemd/system/docker.service.d/http-proxy.conf (Create New)

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:8080/" "NO_PROXY=localhost,127.0.0.1,docker-registry.example.com,.corp"
Environment="HTTPS_PROXY=https://proxy.example.com:8080/" "NO_PROXY=localhost,127.0.0.1,docker-registry.example.com,.corp"

After setting, restart dockerd.

$ sudo systemctl daemon-reload
$ sudo systemctl restart docker

Check the settings.

$ systemctl show --property=Environment docker
Environment=HTTP_PROXY=http://proxy.example.com:8080/