Learning history for participating in team application development in Python ~ Supplement of basic items and construction of jupyterLab environment ~

Previous article

Learning history for participating in team application development in Python ~ Finishing "Introduction to Python3" of paiza learning ~

Supplement to the previous article

The concept of slices

python


word = 'hello'
print(word[0:2]) #Extract from 0th element to 2nd element

##Processing result
he

Specifying the position of an element with a so-called index seems to be called slicing in Python.

Of course there is a slice notation in the list

python



wordlist = ['apple','bike','cake','drum','electric']
print(wordlist[2:]) #Extract from the second element to the end
print(wordlist[:2]) #Extract from the first element to the second element

##Processing result

['cake', 'drum', 'electric']
['apple', 'bike']

About the list

Add an element to the end of the list .append()

Delete the element specified as an argument from the list .remove()

Delete all list elements .clear()

Add x to the i-th in the list .insert(i,x)

python



wordlist = ['apple','bike','cake','drum','electric']
wordlist.insert(5,'falcon')
print(wordlist)

##Output result
['apple', 'bike', 'cake', 'drum', 'electric', 'falcon']

Returns the index of the first element with the value set in the argument .index()

python



lst = [1, 2, 3, 4, 5]
idx = lst.index(4)
print(idx)

##Output result
3

The list also has the concept of id, so a function to make a duplicate .copy()

python



#Example of making a copy of the list and editing the copy but failing because the ids are the same

wordlist = ['apple','bike','cake','drum','electric']
copylist = wordlist

copylist[0] = 'admin'
print(wordlist)
print(copylist) 

##Output result
['admin', 'bike', 'cake', 'drum', 'electric']
['admin', 'bike', 'cake', 'drum', 'electric']

#Success story

wordlist = ['apple','bike','cake','drum','electric']
copylist = wordlist.copy()
print(wordlist is not copylist) #Verify that wordlist is not copylist

copylist[0] = 'admin'
print(wordlist)
print(copylist) 


##Output result
True
['apple', 'bike', 'cake', 'drum', 'electric']
['admin', 'bike', 'cake', 'drum', 'electric']

mata Loops for more than one list can be represented using zip.

For example, the following example

python



list1 = list(range(1,5))
list2 = list(range(5,9))

for a,b in zip(list1,list2):
    print(f"a = {a}, b = {b}", a, b )

Prepare two counter variables and repeat the assignment from list1 and list2 respectively. Then, it becomes as follows.

##Output result

a = 1, b = 5 1 5
a = 2, b = 6 2 6
a = 3, b = 7 3 7
a = 4, b = 8 4 8

Statement delimiter (semicolon)

In Python, you don't have to add a semicolon at the end of a sentence, but you can add it to delimit instructions. In other words

a = 10; b=20; c=30;

That means that variable definitions can be grouped together in a row. Of course, if you put them all on one line, readability will be significantly impaired, so be careful.

break and continue

Python also has break and continue.

python



i = 0

while True:
    i += 1

   if i > 10:
        break
    if i % 2 == 1:
        continue
    print(i)

print('End the process')

If i is above 10, processing is interrupted, and if i is odd, processing is returned to i + = 1. Then, what is printed is an even number of 10 or less.

Tuple type and list type

Lists are like so-called arrays, but Python has a similar concept of tuple types.

The difference from the list is that ** the stored elements cannot be updated / added / deleted **. However, conversely, it can be understood that it stores a constant value.

You can convert other types to lists or tuples with list () and tuple ().

python



a =list("sample"); b=tuple("sample2"); c=list(range(1,11,2)); d=tuple(range(1,11,2))
print(type(d))

Values generated by strings and ranges (acquiring values starting from 1 and incrementing by 2 in the range less than 11) are stored as list types and tuples, respectively. By the way, if you convert a character string to a list or tuple, it will be stored separately for each character.

Unpacking can be used if you want each variable to be assigned an element stored in a list or tuple.

python



words = ['apple','bike','corn']
a,b,c = words
print(a)

##Output result

apple

At this time, note that an error will occur if words = a, b, c.

If you want to combine multiple lists into a two-dimensional list, use zip ().

python



a = ['apple','bike','corn']
b = ['docter','egg','falcon']
c = ['game','heart','icon']
d = ['jack','king','love']
array_list = zip(a,b,c,d)
result = list(array_list)
print(result)

##Output result
[('apple', 'docter', 'game', 'jack'), ('bike', 'egg', 'heart', 'king'), ('corn', 'falcon', 'icon', 'love')]

At the stage of executing zip (), it is just an object, so I will convert it to a list with the list function. And as you can see from the output result, when using zip (), You can see that the elements of each list are restored for each index. In other words

python



print(result[0])

##Output result
('apple', 'docter', 'game', 'jack')

It turns out that. Note that this is a different process than creating a list with a for statement.

Set type

Think of it as a set of mathematical terms, or a group. In other words, it is a collection of non-overlapping elements. For example

fruit = {'apple','banana','lemon'}

It's like that. Lists are represented by [] and tuples by (), while sets are represented by {}. However, it takes the form of x = {}, and when no element is registered in it or when the registered element is a key: value, it becomes a dictionary instead of a set. Also, tuples can be registered for sets, but lists cannot be registered.

Summary of lists, tuples, sets and dictionaries

list

-A so-called array. -Elements can be updated, added, and deleted. -Addition is ʻappend (), deletion is remove (). -Conversion from other types to lists is list ()`.

Tuple

-An image of an array that stores constant values. -Elements cannot be updated, added, or deleted. -Conversion from other types to tuples is tuple ().

set

-A so-called set array (group). -Elements with the same content cannot be registered more than once. -Tuples can be taken as elements, but lists cannot. -Conversion from other types to sets is set (). -Unlike lists and tuples, there is no well-ordered order, so there is no ** index. **. -Add ʻadd () and delete remove () `. -Although the set can update, add, and delete elements, there are some types that cannot do so even in the same set.

dictionary

-So-called associative array. The element must be in the form of key: value. -Use keys () to retrieve keys, value () to retrieve values, and ʻitems ()` to retrieve both at the same time.

It turns out that.

Random module supplement

The random module can be used not only for numerical values but also for sequences such as tuples and lists.

Get one element randomly from the sequence with choice (). Gets k elements with duplicates from the sequence set in the argument with choices (seq, k = 1), and returns them as a list. sample (seq, k) gets k elements from the sequence and returns them as a list without duplication. You can sort the sequence with shuffle ().

In other words

python



import random

a = ['apple','basket','color','dictionary','error']

wordlist =list(a)

print(random.choice(wordlist))
print(random.choices(wordlist,k=3))
print(random.sample(wordlist,3))

print(wordlist)
random.shuffle(wordlist)
print(wordlist)

##Output result(Since it is a random module, the result changes every time it is executed.)

dictionary
['error', 'apple', 'color']
['dictionary', 'basket', 'color']
['apple', 'basket', 'color', 'dictionary', 'error']
['color', 'apple', 'dictionary', 'error', 'basket']

Comprehension notation

python



squares = [v ** 2 for v in range(10)]
print(squares)

As I did last time, if you create a for statement in the list as described above and put the expression of the element you want to store repeatedly in the list before for, you can easily create a sequence. For example, in the above example, the squared values of 0 to 9 will be stored in the list. It can also be used as a set, in which case a set can be generated.

python



sample = {x for x in range(30) if x %2 == 0}
print(sample)

##Output result
{0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28}

From the point of view of generating a set, there should be regularity in what it produces to distinguish it from a simple list (the above is an even set in the range 0≤x <30).

In the case of a dictionary, this is the case.


languages = [
    "apple",
    "bike",
    "cow",
    "decade",
    "egostic"
]

dictionary = {lang.upper():len(lang) for lang in languages}
print(dictionary)

##Output result
{'APPLE': 5, 'BIKE': 4, 'COW': 3, 'DECADE': 6, 'EGOSTIC': 7}

First, make a key with a list. This is fine with tuples. And define the form of key: value before for. This time, the created list is assigned to the counter variable lang. Then, based on that lang, the property is created as the capital letter of the character string stored in languages: the value of the number of characters, and the dictionary is created repeatedly.

Creating a Jupyter Lab environment with Docker

Construction procedure

1. Creating a Dockerfile

FROM jupyter/datascience-notebook:7a0c7325e470 #Specifying the image

COPY requirements.txt tmp/requirements.txt #A file that lists the packages to install.
RUN pip install -r tmp/requirements.txt #pip is a command to install Python packages. Here requirements.Install the packages listed in txt.

EXPOSE 8888

As a review, FROM selects the base image, that is, specifies the image you want to use. EXPOSE opens the specified port. Of course, if you haven't built a Docker environment on your Mac or Windows 10 Pro, you'll also need to set up port forwarding on the Virtual Box side.

2. Make docker-compose.yml


version: "3.7" # docker-compose.Specify the version of the yml file format
services:
  jupyterlab: #Services to handle
    build: .
    container_name: jupyterlab #Container name
    environment:
      - JUPYTER_ENABLE_LAB=yes #Enable jupyterLab
    ports:
      - "8888:8888" #Specifying the port handled by the service
    volumes:
      - ./JupyterLab:/home/jovyan/work #Directory settings to share / synchronize. Directory on the host side:Directory on the virtual space side
    command: start.sh jupyter lab --NotebookApp.token='' #Omit login token input when starting container


I'm still unfamiliar with Docker, so I tried various things while googled, and the meaning of creating docker-compose.yml is that if you try to do it with docker run, it will be easier to understand where you have to make long options later. There seems to be a meaning. And the quickest way to build an environment is to pull the image from the official, create a container with options with docker run, and start it there, but after all this got used to Docker I felt that it was the cause of getting stuck in the swamp if I was not a person, and managing the container with two files, Dockerfile (I heard that I may not write this recently) and docker-compose.yml I thought it would be good in terms of ease.

Read and write files in Python while using JupyterLab.

Character code confirmation

python



import sys
print(sys.getdefaulttencoding())

Read file

python



with open('jinko.txt','rt')as f:
    print(f.read())

Reading and writing files in Python starts with ʻopen ()and ends withclose (), but to prevent forgetting close (), use with open ()`.

Read the data in the file as a list

python



with open('jinko.txt') as f:
    l = f.readlines()
print(l)

#Or

with open('jinko.txt') as f:
    while True:
        l = f.readline()
        print(l)
        if not l:
            break

Assign the file to the variable f, make it an object, and use the readlines () method. Or use the readline () method in a loop to read line by line.

Limit the number of lines read with while

python



count = 0
with open('jinko.txt') as f:
    l = f.readline()
    while True:
        l = f.readline()
        m = l.strip()
        print(m)
        count += 1
        if count > 5:
            break

#When listing line by line, separated by commas

import csv

count = 0
with open('jinko.txt') as f:
    l = f.readline()
    reader = csv.reader(f)
    for l in reader:
        print(l)
        count += 1
        if count > 5:
            break

#Take out numerical values from data and try to calculate

import csv

s = 0
with open('jinko.txt') as f:
    reader = csv.reader(f)
    for l in reader:
        if l[0] == "Hokkaido":
            s = s + int(l[2])
print("Hokkaido population", s ,"Man")

##Partial excerpt of the data read above

['Hokkaido', 'Sapporo', '1913545']
['Hokkaido', 'Hakodate', '279127']
['Hokkaido', 'Otaru City', '131928']
['Hokkaido', 'Asahikawa', '347095']
['Hokkaido', 'Muroran City', '94535']
['Hokkaido', 'Kushiro City', '181169']


csv.reader () works likesplit (). It is recommended to use this if the data is in CSV format. Regarding the calculation part, first from the excerpt of the data to be read, l [0] indicates the prefecture, that is, ʻif l [0] == "Hokkaido": determines the range of data to be read. And it turns out that l [2]contains data on population, that is, numbers. However, since it is stored as a character string, it is used after being digitized with ʻint (). Therefore, if you write the above sentence, you can calculate the population of Hokkaido in the data.

Read a file with a different encoding

python



with open('jinko_sjis.txt', encoding='shift-jis', newline='\r\n') as f:
    while True:
        l = f.readline()
        print(l)
        if not l:
            break

When reading a file other than utf-8, optionally specify ʻencoding ='file format you want to read'. Also, specify the type of line break with newline`. This time, the line feed is specified in Windows.

Writing a file

python



with open("write_example.txt", "w") as f:
    f.write("abc\r\n")
    f.write("defgh")

#Writing multi-line data

data = ["abc","defgh"]
with open("write_example2.txt","w") as f:
    f.writelines(data)

#Write CSV data

import csv

alldata = [
    [1,2,3],[2,3,5]
]

with open("write_example3","w") as f:
    writer =csv.writer(f)
    writer.writerows(alldata)


##for statement can also be used


import csv

alldata = [
   [1, 2, 3],
   [2, 3, 5]
]

with open("writeexample5.txt", "w") as f:
    writer = csv.writer(f)
    for data in alldata:
            writer.writerow(data)


Reference site

A story about trying to create a machine learning environment using Docker

Create a container environment for JupyterLab

Recommended Posts

Learning history for participating in team application development in Python ~ Supplement of basic items and construction of jupyterLab environment ~
Learning history for participating in team application development in Python ~ Index page ~
Learning history for participating in team app development in Python ~ Django Tutorial 5 ~
Learning history for participating in team app development in Python ~ Django Tutorial 4 ~
Learning history for participating in team app development in Python ~ Django Tutorial 1, 2, 3 ~
Learning history for participating in team app development in Python ~ Django Tutorial 6 ~
Learning history for participating in team app development in Python ~ Django Tutorial 7 ~
Learning history for participating in team application development in Python ~ Think a little about requirement definition ~
Learning history to participate in team application development in Python ~ After finishing "Introduction to Python 3" of paiza learning ~
Python environment construction and SQL execution example to DB and memo of basic processing for statistics 2019
Python and machine learning environment construction (macOS)
Construction of development environment for Choreonoid class
Get started with Python in 30 minutes! Development environment construction & learn basic grammar
Easy-to-understand explanation of Python Web application (Django) even for beginners (1) [Environment construction]
[Basic] Unify everything from tabulation to dashboard creation! Data visualization application development with Python and Dash ~ Overview of Dash, environment construction, sample execution ~
Installation of Python3 and Flask [Environment construction summary]
python development environment -use of pyenv and virtualenv-
Basic story of inheritance in Python (for beginners)
Build an interactive environment for machine learning in Python
Build a machine learning application development environment with Python
[Python] Chapter 01-02 About Python (Execution and installation of development environment)
Python development environment construction
python2.7 development environment construction
Development environment in Python
Check the operation of Python for .NET in each environment
Construction of Python local development environment Part 2 (pyenv-virtualenv, pip usage)
[Note] List of basic commands for building python / conda environment
Building a development environment for Android apps-creating Android apps in Python
[Tips] Problems and solutions in the development of python + kivy
Construction of Cortex-M development environment for TOPPERS using Raspberry Pi
Python environment construction For Mac
Python environment construction and TensorFlow
[MEMO] [Development environment construction] Python
[For organizing] Python development environment
Environment construction of python2 & 3 (OSX)
Build and test a CI environment for multiple versions of Python
How about Anaconda for building a machine learning environment in Python?
Construction of Python local development environment Part 1 (pyenv, pyenv-virtualenv, pip installation)
Python + Unity Reinforcement learning environment construction
Python: Application of supervised learning (regression)
Web application development memo in python
Python development environment options for May 2020
Python development environment construction on macOS
Emacs settings for Python development environment
Environment construction of python3.8 on mac
Function synthesis and application in Python
Python3 TensorFlow for Mac environment construction
Emacs Python development environment construction memo
Prepare Python development environment for each project in Windows environment (VSCode + virtualEnvWrapper + Pylint)
Graph of the history of the number of layers of deep learning and the change in accuracy
Comparing the basic grammar of Python and Go in an easy-to-understand manner
Realize environment construction for "Deep Learning from scratch" with docker and Vagrant
Development and deployment of REST API in Python using Falcon Web Framework
Procedure from AWS CDK (Python) development to AWS resource construction * For beginners of development
Implementation of particle filters in Python and application to state space models
Explanation of creating an application for displaying images and drawing with Python
Python: Preprocessing in machine learning: Handling of missing, outlier, and imbalanced data
[Python] Reason for dtype "int32" in Numpy (Windows environment) (as of September 2020)