[PYTHON] From "drawing" to "writing" the configuration diagram: Try drawing the AWS configuration diagram with Diagrams

things to do

Try drawing an AWS configuration diagram with code in Diagrams, also known as "Diagram as Code"

What are Diagrams

--Diagrams is a Python module --Graphviz is also required to use Diagrams --You can draw cloud system architecture with Python code --The site is here - https://diagrams.mingrammer.com/ ――The explanation of Diagrams on the site is in English, but the amount is small, so please read it! (Automatic translation is enough to understand)

In this way, you can write a configuration diagram with Python code. When this code is executed as a Python script, the configuration diagram is output as an image.

-** Good points ** --You can code the configuration diagram ――Since you can edit the drawing by code editing, it is easier than image correction --Since it is a code, it can be managed with Git together with the CloudFormation code. --Non-AWS icons are also available --The output configuration diagram can be used as a rough sketch or better.

• Task --Not all AWS icons are available --You can't specify where to place the icon (that's why it's a simple code)

Diagrams installation environment

I installed Diagrams in two environments, " Raspberry Pi 4 Model B " and " Cloud9 ".

Environment (1) Raspberry Pi 4 Model B

$ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

Python comes in two versions, 2.7 and 3.7. Install Diagrams on Python 3.7

$ python --version
Python 2.7.16
$ python3 --version
Python 3.7.3

Environment (2) Cloud9

$ cat /etc/os-release
NAME="Amazon Linux AMI"
VERSION="2018.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2018.03"
PRETTY_NAME="Amazon Linux AMI 2018.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2018.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"

Cloud9 is Python 3.x only, not Python 2.x

$ python --version
Python 3.6.8

Diagrams installation

Installation (1) Raspberry Pi

Just install diagrams and Graphviz on Python3.

Install diagram on python3

$ python3 -m pip install diagrams

Graphviz installation

$ sudo apt-get install graphviz

Installation (2) Cloud9

$ sudo pip install --upgrade pip
$ sudo python3 -m pip install diagrams
$ sudo yum install graphviz

(Verification) Try drawing the AWS configuration diagram

• "Draw" → "Write" the configuration diagram

(Verification 1) Draw a configuration diagram of the Web server

Here's what I was able to do, a decent degree of perfection

I could see it like this on Cloud9 (convenient!)

--Outline of configuration --Build a WebApp server on EC2 --Prepare two WebApp servers for redundancy --Distribute the load on the WebApp server with ALB --Domain registration with Rote53 --Prepare WebApp server database with RDS

This is the code I made.

Web server configuration diagram

from diagrams import Cluster, Diagram, Edge
from diagrams.aws.network import VPC
from diagrams.aws.compute import EC2
from diagrams.aws.database import RDS
from diagrams.aws.network import Route53
from diagrams.aws.network import NATGateway
from diagrams.aws.network import InternetGateway
from diagrams.aws.network import ElasticLoadBalancing
from diagrams.onprem.network import Internet
from diagrams.onprem.client import Client
from diagrams.onprem.client import Users

with Diagram("Fig-001-WebSite", show=False):
    with Cluster("Fig-001-WebSite"):
        internet_01 = Internet("Internet")
        users_01 = Users("user")
        with Cluster("AWS/Develop/ap-northeast-1"):
            r53_01 = Route53("hoge.fuga.local")
            with Cluster("VPC/192.168.0.0/HOGE-SYSTEM"):
                alb_01 = ElasticLoadBalancing("ALB")
                igw_01 = InternetGateway("InternetGateway")
                with Cluster("subnet/192.168.1.0/public"):
                    natgw_01 = NATGateway("NATGW")
                    ec2_01 = EC2("WebApp")
                with Cluster("subnet/192.168.101.0/private"):
                    rds_01 = RDS("postgres-Master")
                with Cluster("subnet/192.168.2.0/public"):
                    natgw_02 = NATGateway("NATGW")
                    ec2_02 = EC2("WebApp")
                with Cluster("subnet/192.168.102.0/private"):
                    rds_02 = RDS("postgres-Slave")

    #Configuration diagram Web server
    users_01 - internet_01 - igw_01 - alb_01 #User access route
    alb_01 - ec2_01 - rds_01 #WebApp redundancy
    alb_01 - ec2_02 - rds_01 #WebApp redundancy
    rds_01 - Edge(style="dotted") - rds_02 #DB redundancy

(Verification 2) Draw a block diagram of WorkSpaces

--Amazon WorkSpaces is a virtual desktop service provided by AWS. --Diagrams does not have a WorkSpaces icon --This is the configuration diagram you want to create ↓

This is what I wrote in Diagrams (I think it's reasonable)

--Icons that are not in Diagrams such as Storage Gateway and WorkSpaces are replaced with different icons.

Client icons are used instead of icons that are not provided in Diagrams, such as WorkSpaces and ENI.

WorkSpace configuration diagram

from diagrams import Cluster, Diagram, Edge
from diagrams.aws.network import VPC
from diagrams.aws.compute import EC2
from diagrams.aws.database import RDS
from diagrams.aws.network import Route53
from diagrams.aws.network import NATGateway
from diagrams.aws.network import InternetGateway
from diagrams.onprem.network import Internet
from diagrams.onprem.client import Client
from diagrams.aws.security import DirectoryService
from diagrams.aws.network import DirectConnect
from diagrams.onprem.compute import Server
from diagrams.onprem.client import User
from diagrams.onprem.compute import Nomad

with Diagram("Fig-002-WorkSpaces", show=False):
    with Cluster("Fig-002-WorkSpaces"):
        Internet_01 = Internet("Internet")
        User_01 = User("User 1")
        User_02 = User("User 2")
        UserDevice_01 = Client("device")
        UserDevice_02 = Client("device")

        with Cluster("VPC/AWS management"):
            StorageGw_01 = Client("StorageGateway")
            WorkSpace_01 = Client("WorkSpacesForUser1")
            WorkSpace_02 = Client("WorkSpacesForUser2")

        with Cluster("VPC/192.168.0.0/HOGE"):
            with Cluster("subnet/192.168.1.0/public"):
                eni_01 = Client("ENI User1")
                eni_02 = Client("ENI User2")

        DirectConnect_01 = DirectConnect("DirectConnect")
        ADConnector_01 = DirectoryService("AD Connector")

        with Cluster("Base"):
            with Cluster("Internal network"):
                ActiveDirectory_01 = Server("AD")

    #Configuration diagram WorkSpace
    User_01 - UserDevice_01 - Internet_01 - StorageGw_01 - WorkSpace_01 - eni_01 #device-Communication path between WorkSpaces
    User_02 - UserDevice_02 - Internet_01
    StorageGw_01 - WorkSpace_02 - eni_02
    ADConnector_01 - DirectConnect_01 - ActiveDirectory_01 #In-base AD-Communication path between AD Connector
    ADConnector_01 - Edge(style="dotted") - WorkSpace_01
    ADConnector_01 - Edge(style="dotted") - WorkSpace_02

• Digression * I had a hard time because the icon of the base was on the upper left and I couldn't place the icon where I expected.

Impressions: What I thought about using Diagrams

• The entire --Icons that are not in Diagrams such as Storage Gateway and WorkSpaces are replaced with different icons. ――If you decide on a substitute icon, you can use it as a configuration diagram to share within the team. -(I'd be happy if Diagrams provided a substitute icon, please!) ――A level that can be used as a rough sketch, it seems to be usable when asking for a clean copy --Learning costs are low and it helps --How to write the code --Variable naming conventions are important (not limited to this) --It is recommended to aggregate the connection information of each icon at the end of the code. --Share the rules with the team so that you can see the configuration diagram by looking at the connection information (* 1)

(* 1) If you decide the rules, you can create a composition diagram in your brain by looking at it.

bonus

Put the icon output and Python code of onPrem (information as of May 2020)

onPrem icon output

from diagrams import Cluster, Diagram, Edge
# onprem.database
from diagrams.onprem.database import Cassandra
from diagrams.onprem.database import Clickhouse, ClickHouse
from diagrams.onprem.database import Cockroachdb, CockroachDB
from diagrams.onprem.database import Couchdb, CouchDB
from diagrams.onprem.database import Dgraph
from diagrams.onprem.database import Hbase, HBase
from diagrams.onprem.database import Influxdb, InfluxDB
from diagrams.onprem.database import Janusgraph, JanusGraph
from diagrams.onprem.database import Mariadb, MariaDB
from diagrams.onprem.database import Mongodb, MongoDB
from diagrams.onprem.database import Mssql, MSSQL
from diagrams.onprem.database import Mysql, MySQL
from diagrams.onprem.database import Neo4J
from diagrams.onprem.database import Oracle
from diagrams.onprem.database import Postgresql, PostgreSQL
from diagrams.onprem.database import Scylla
# onprem.security
from diagrams.onprem.security import Trivy
from diagrams.onprem.security import Vault
# onprem.ci
from diagrams.onprem.ci import Circleci, CircleCI
from diagrams.onprem.ci import Jenkins
from diagrams.onprem.ci import Teamcity, TC
from diagrams.onprem.ci import Travisci, TravisCI
# onprem.etl
from diagrams.onprem.etl import Embulk
# onprem.mlops
from diagrams.onprem.mlops import Polyaxon
# onprem.network
from diagrams.onprem.network import Apache
from diagrams.onprem.network import Caddy
from diagrams.onprem.network import Consul
from diagrams.onprem.network import Envoy
from diagrams.onprem.network import Etcd, ETCD
from diagrams.onprem.network import Haproxy, HAProxy
from diagrams.onprem.network import Internet
from diagrams.onprem.network import Istio
from diagrams.onprem.network import Kong
from diagrams.onprem.network import Linkerd
from diagrams.onprem.network import Nginx
from diagrams.onprem.network import Pfsense, PFSense
from diagrams.onprem.network import Pomerium
from diagrams.onprem.network import Tomcat
from diagrams.onprem.network import Traefik
from diagrams.onprem.network import Vyos, VyOS
from diagrams.onprem.network import Zookeeper
# onprem.inmemory
from diagrams.onprem.inmemory import Aerospike
from diagrams.onprem.inmemory import Hazelcast
from diagrams.onprem.inmemory import Memcached
from diagrams.onprem.inmemory import Redis
# onprem.search
from diagrams.onprem.search import Elasticsearch
from diagrams.onprem.search import Solr
# onprem.container
from diagrams.onprem.container import Docker
from diagrams.onprem.container import Rkt, RKT
# onprem.iac
from diagrams.onprem.iac import Ansible
from diagrams.onprem.iac import Awx
from diagrams.onprem.iac import Terraform
# onprem.compute
from diagrams.onprem.compute import Nomad
from diagrams.onprem.compute import Server
# onprem.vcs
from diagrams.onprem.vcs import Git
from diagrams.onprem.vcs import Github
from diagrams.onprem.vcs import Gitlab
# onprem.workflow
from diagrams.onprem.workflow import Airflow
from diagrams.onprem.workflow import Digdag
from diagrams.onprem.workflow import Kubeflow, KubeFlow
from diagrams.onprem.workflow import Nifi, NiFi
# onprem.queue
from diagrams.onprem.queue import Activemq, ActiveMQ
from diagrams.onprem.queue import Celery
from diagrams.onprem.queue import Kafka
from diagrams.onprem.queue import Rabbitmq, RabbitMQ
from diagrams.onprem.queue import Zeromq, ZeroMQ
# onprem.cd
from diagrams.onprem.cd import Spinnaker
# onprem.gitops
from diagrams.onprem.gitops import Argocd, ArgoCD
# onprem.monitoring
from diagrams.onprem.monitoring import Datadog
from diagrams.onprem.monitoring import Grafana
from diagrams.onprem.monitoring import Kibana
from diagrams.onprem.monitoring import Prometheus
from diagrams.onprem.monitoring import Splunk
from diagrams.onprem.monitoring import Thanos
# onprem.client
from diagrams.onprem.client import Client
from diagrams.onprem.client import User
from diagrams.onprem.client import Users
# onprem.logging
from diagrams.onprem.logging import Fluentd
from diagrams.onprem.logging import Logstash, LogStash
from diagrams.onprem.logging import Loki
# onprem.analytics
from diagrams.onprem.analytics import Beam
from diagrams.onprem.analytics import Flink
from diagrams.onprem.analytics import Hadoop
from diagrams.onprem.analytics import Hive
from diagrams.onprem.analytics import Metabase
from diagrams.onprem.analytics import Norikra
from diagrams.onprem.analytics import Spark
from diagrams.onprem.analytics import Storm
from diagrams.onprem.analytics import Tableau



with Diagram("OnPrem-Aicons", show=False):
    with Cluster("OnPrem"):
        # onprem.database("# onprem.database")
        Cassandra("Cassandra")
        Clickhouse, ClickHouse("Clickhouse, ClickHouse")
        Cockroachdb, CockroachDB("Cockroachdb, CockroachDB")
        Couchdb, CouchDB("Couchdb, CouchDB")
        Dgraph("Dgraph")
        Hbase, HBase("Hbase, HBase")
        Influxdb, InfluxDB("Influxdb, InfluxDB")
        Janusgraph, JanusGraph("Janusgraph, JanusGraph")
        Mariadb, MariaDB("Mariadb, MariaDB")
        Mongodb, MongoDB("Mongodb, MongoDB")
        Mssql, MSSQL("Mssql, MSSQL")
        Mysql, MySQL("Mysql, MySQL")
        Neo4J("Neo4J")
        Oracle("Oracle")
        Postgresql, PostgreSQL("Postgresql, PostgreSQL")
        Scylla("Scylla")
        # onprem.security("# onprem.security")
        Trivy("Trivy")
        Vault("Vault")
        # onprem.ci("# onprem.ci")
        Circleci, CircleCI("Circleci, CircleCI")
        Jenkins("Jenkins")
        Teamcity, TC("Teamcity, TC")
        Travisci, TravisCI("Travisci, TravisCI")
        # onprem.etl("# onprem.etl")
        Embulk("Embulk")
        # onprem.mlops("# onprem.mlops")
        Polyaxon("Polyaxon")
        # onprem.network("# onprem.network")
        Apache("Apache")
        Caddy("Caddy")
        Consul("Consul")
        Envoy("Envoy")
        Etcd, ETCD("Etcd, ETCD")
        Haproxy, HAProxy("Haproxy, HAProxy")
        Internet("Internet")
        Istio("Istio")
        Kong("Kong")
        Linkerd("Linkerd")
        Nginx("Nginx")
        Pfsense, PFSense("Pfsense, PFSense")
        Pomerium("Pomerium")
        Tomcat("Tomcat")
        Traefik("Traefik")
        Vyos, VyOS("Vyos, VyOS")
        Zookeeper("Zookeeper")
        # onprem.inmemory("# onprem.inmemory")
        Aerospike("Aerospike")
        Hazelcast("Hazelcast")
        Memcached("Memcached")
        Redis("Redis")
        # onprem.search("# onprem.search")
        Elasticsearch("Elasticsearch")
        Solr("Solr")
        # onprem.container("# onprem.container")
        Docker("Docker")
        Rkt, RKT("Rkt, RKT")
        # onprem.iac("# onprem.iac")
        Ansible("Ansible")
        Awx("Awx")
        Terraform("Terraform")
        # onprem.compute("# onprem.compute")
        Nomad("Nomad")
        Server("Server")
        # onprem.vcs("# onprem.vcs")
        Git("Git")
        Github("Github")
        Gitlab("Gitlab")
        # onprem.workflow("# onprem.workflow")
        Airflow("Airflow")
        Digdag("Digdag")
        Kubeflow, KubeFlow("Kubeflow, KubeFlow")
        Nifi, NiFi("Nifi, NiFi")
        # onprem.queue("# onprem.queue")
        Activemq, ActiveMQ("Activemq, ActiveMQ")
        Celery("Celery")
        Kafka("Kafka")
        Rabbitmq, RabbitMQ("Rabbitmq, RabbitMQ")
        Zeromq, ZeroMQ("Zeromq, ZeroMQ")
        # onprem.cd("# onprem.cd")
        Spinnaker("Spinnaker")
        # onprem.gitops("# onprem.gitops")
        Argocd, ArgoCD("Argocd, ArgoCD")
        # onprem.monitoring("# onprem.monitoring")
        Datadog("Datadog")
        Grafana("Grafana")
        Kibana("Kibana")
        Prometheus("Prometheus")
        Splunk("Splunk")
        Thanos("Thanos")
        # onprem.client("# onprem.client")
        Client("Client")
        User("User")
        Users("Users")
        # onprem.logging("# onprem.logging")
        Fluentd("Fluentd")
        Logstash, LogStash("Logstash, LogStash")
        Loki("Loki")
        # onprem.analytics("# onprem.analytics")
        Beam("Beam")
        Flink("Flink")
        Hadoop("Hadoop")
        Hive("Hive")
        Metabase("Metabase")
        Norikra("Norikra")
        Spark("Spark")
        Storm("Storm")
        Tableau("Tableau")