I made a pet camera that is always connected with WebRTC (Nuxt.js + Python + Firebase + SkyWay + Raspberry Pi)
Introduction
A fun trip of 2 days and 1 night! But I can't help but be worried about the cat in the answering machine. ..
I made a "traveling pet watching camera" for my beloved cat: camera:
** Click here for GitHub repository **
Actually, I refer to the excellent article here as a source of material. SkyWay WebRTC Gateway Hands-on
The source is Ruby. I'm very sorry this time, but I made many improvements such as organizing functions and files in Python and making it possible to reconnect to the camera many times. : bow:
Feature
You can do this!
――You can watch your home camera image anytime, anywhere from outside through the WEB app. --Safe with login authentication --LED on / off and operation (I want to move the angle of the camera in the future) --You can shut down the program from the WEB app (chanting as Barus!)
Technology used
--Nuxt.js (Vue.js): Creating a web app --Python: Home camera control --SkyWay (WebRTC): Send camera video / message --Firebase: Web app deployment, login function, SkyWay API Key storage
RaspberryPi Used for video distribution on the home camera side. Connect the USB camera and set the SkyWay module webrtc-gateway and the control Python program. As a rough mechanism, the camera image is acquired by a Python program, accessed by web-rtc-gateway, and streamed to the connected WEB application side.
In addition, we have made it possible to turn on / off by connecting an LED with a message from the web application side. In the future, I would like to control the servo motor through GPIO to control the orientation and angle of the USB camera.
SkyWay It is an SDK that can easily handle WebRTC of NTT Communications. I connected my home camera to a web app and used it to send videos and messages. For details, SkyWay official website
Follow the instructions to register for free and get an API KEY.
On the web application side, SkyWay is operated as a Node.js module.
The Raspberry Pi on the camera side does not launch a web browser, it is headless, so it uses the skyway-webrtc-gateway module.
skyway-webrtc-gateway A compiled module that makes it easy to run SkyWay on IOT devices. You can send and receive video, audio, and messages by realizing WebRTC without a web browser. Official github
All you have to do is give permission to the Raspberry Pi on the camera side and execute it.
Firebase I used Firebase this time because I wanted to implement troublesome functions quickly.
--Authentification: Login function --Firestore: Save SkyWay API Key --Hosting: WEB application deployment destination
Nuxt.js (WEB application side)
I built the WEB application side using Nuxt.js which is the front end framework of Vue.js. Since it is a super private video at home, the login function is implemented using Firebase Authentication. By the way, I also used Firebase as the deployment destination. It's an API Key for connecting to SkyWay, but I didn't want it on the front end. So I saved it in the Firestore so that only logged-in users can get the API Key.
Directory structure
#Partially omitted
frontend
├── components
│ ├── Header.vue
│ ├── Login.vue
│ └── ProjectTitle.vue
├── layouts
│ └── default.vue
├── middleware
│ └── authenticated.js
├── pages
│ ├── index.vue
│ └── operate.vue
├── static
│ └── favicon.ico
├── store
│ └── index.js
└── nuxt.config.js
Rough processing flow
- Users access pages/index.vue Due to middleware/authenticated.js, you cannot browse other than index if you are not logged in.
- Login through componets/login.vue
- Get SkyWay API Key from firestore with store/index.js at the same time you log in
- Users access pages/operator.vue
- Connect to SkyWay using the obtained API Key
- Establish a connection with the already connected Raspberry Pi on the home camera side Watch camera footage from WebRTC-gateway on the Raspberry Pi side of your home
The first point is that ** the troublesome part is left to Firebase **. : thumbsup_tone3: I want to make a pet camera system, not a good website, so I threw the functions that I entrusted to Firebase. Especially, it is very convenient to keep the API Key in the Firestore.
Python (home camera / Raspberry Pi side)
Main & most fun part. Once WebRTC-gateway is run, it is controlled by Python through the REST API.
Directory structure
backend
├── __init__.py
├── config.py
├── data.py
├── media.py
├── peer.py
├── robot.py
├── util.py
└── webrtc_control.py
The executable file is webrtc_control.py.
Other files are modules by function.
webrtc_control.py This is the main program that controls WebRTC-gateway and USB cameras. By running this program next to the gateway, the pet camera will be put on standby for remote use.
webrtc_control.py
# ~abridgement~
def main():
queue = que.Queue()
robot = Robot()
peer_id = config.PEER_ID
media_connection_id = ''
data_connection_id = ''
process_gst = None
#Establish Peer(Connect to SkyWay)
peer_token = Peer.create_peer(config.API_KEY, peer_id)
if peer_token is None:
count_create = 1
retry_peer_id = ''
#If Peer cannot be established, change the ID and challenge every 5 seconds
while peer_token is None:
time.sleep(5)
count_create += 1
retry_peer_id = peer_id + str(count_create)
peer_token = Peer.create_peer(config.API_KEY, retry_peer_id)
peer_id = retry_peer_id
peer_id, peer_token = Peer.listen_open_event(peer_id, peer_token)
#Thread to always listen for events to Peer.
th_listen = threading.Thread(target=Peer.listen_event,
args=(peer_id, peer_token, queue, robot))
th_listen.setDaemon(True)
th_listen.start()
#Always listen and thread messages from web apps.
th_socket = threading.Thread(target=robot.socket_loop, args=(queue,))
th_socket.setDaemon(True)
th_socket.start()
try:
while True:
#Receive a queue from a thread.
results = queue.get()
#Words in List in queue("Bals")If is included, exit while
if 'data' in results.keys():
if results['data'] in config.SHUTDOWN_LIST:
break
#Listen to the video event once the video connection is established
elif 'media_connection_id' in results.keys():
media_connection_id = results['media_connection_id']
th_media = threading.Thread(target=Media.listen_media_event,
args=(queue, media_connection_id))
th_media.setDaemon(True)
th_media.start()
#Once you start a video stream with Gstreamer, get that process.
elif 'process_gst' in results.keys():
process_gst = results['process_gst']
#Data connection(Message exchange)Once established, store that ID.
elif 'data_connection_id' in results.keys():
data_connection_id = results['data_connection_id']
#Applicable content in video event(close、error)Then you can connect to the next one
#The video stream I was using, MediaConnection,Discard DataConnection
#In this case, it refers to the case where the connected Web application side closes or an error occurs.
elif 'media_event' in results.keys():
if results['media_event'] in ['CLOSE', 'ERROR']:
process_gst.kill()
Media.close_media_connections(media_connection_id)
Data.close_data_connections(data_connection_id)
except KeyboardInterrupt:
pass
robot.close()
Peer.close_peer(peer_id, peer_token)
process_gst.kill()
print('all shutdown!')
if __name__ == '__main__':
main()
Multithread
In order to always listen to event calls to Peer, it is daemonized with th_listen and thread parallelized.
th_socket is also threaded and parallel, but this is a Listen of DataConnection (message from web application).
The while statement separates the processing when the status of each connection changes.
Each thread is prepared to skip the queue according to the received event, and this while statement runs the process according to the contents of the queue.
Listen for events from MediaConnection with th_media in while.
** By opening and re-preparing the Connection according to the status from each thread, it is possible to reconnect with the Raspberry Pi on the camera side even if the Web application side is disconnected many times in the middle. **: tada:
Originally, it should be possible to easily open and establish a media connection immediately, but is it a specification? Or I don't know if I'm immature, If you try to create a new media connection from the second time onward, WebRTC-gateway will result in a 400 error, so it is a brute force implementation to create a new media connection.
- Originally, it's a smarter process, but it doesn't work as officially, so it's a bit redundant. is. *
util.py
Since WebRTC-gateway is controlled by REST API, you will send requests frequently. I will put them together in a method so that they can be used extensively.
util.py
import requests
import config
def request(method_name, _uri, *args):
response = None
uri = config.HOST + _uri
if method_name == 'get':
response = requests.get(uri, *args)
elif method_name == 'post':
response = requests.post(uri, *args)
elif method_name == 'put':
response = requests.put(uri, *args)
elif method_name == 'delete':
response = requests.delete(uri)
else:
print('There is no method called it')
return response
robot.py In the future, I want to move the camera with a robot arm, so I have a class with this name. This file has the following functions.
--GPIO control (L chica and motor) --Establishment of socket communication and data reception
Receive instructions to GPIO through the web app and data connection.
(Currently, you can only turn on/off the LED w)
Below are the methods threaded in webrtc_control.py.
robot.py
# ~abridgement~
class Robot:
# ~abridgement~
#A threaded method.
#It listens for messages from web apps.
def socket_loop(self, queue):
"""Wait for socket communication
"""
#Create socket communication
self.make_socket()
while True:
#data(message)To receive
data = self.recv_data()
data = data.decode(encoding="utf8", errors='ignore')
#Send message content to queue
queue.put({'data': data})
#Operate GPIO according to the message content(In this case, the LED turns on and off)
self.pin(data)
# ~abridgement~
peer.py
Establish a Peer connection with SkyWay and make SkyWay ready for use.
Below are the methods threaded in webrtc_control.py.
peer.py
# ~abridgement~
class Peer:
"""Connect to SkyWay and manage sessions
"""
# ~abridgement~
#Thread it. Listen for events to Peer.
@classmethod
def listen_event(cls, peer_id, peer_token, queue, robot):
"""Wait for Peer object events
"""
gst_cmd = "gst-launch-1.0 -e v4l2src ! video/x-raw,width=640,height=480,framerate=30/1 ! videoconvert ! vp8enc deadline=1 ! rtpvp8pay pt=96 ! udpsink port={} host={} sync=false"
uri = "/peers/{}/events?token={}".format(peer_id, peer_token)
while True:
_res = request('get', uri)
res = json.loads(_res.text)
if 'event' not in res.keys():
# print('No peer event')
pass
#When a video connection is requested by the web application
elif res['event'] == 'CALL':
print('CALL!')
media_connection_id = res["call_params"]["media_connection_id"]
queue.put({'media_connection_id': media_connection_id})
#Create media
(video_id, video_ip, video_port) = Media.create_media()
#Create a video stream from a USB camera with Gstreamer
cmd = gst_cmd.format(video_port, video_ip)
process_gst = subprocess.Popen(cmd.split())
queue.put({'process_gst': process_gst})
#Connect the web app and mediaConnection
Media.answer(media_connection_id, video_id)
#Data connection from web app(Message exchange)If requested
elif res['event'] == 'CONNECTION':
print('CONNECT!')
data_connection_id = res["data_params"]["data_connection_id"]
queue.put({'data_connection_id': data_connection_id})
#Create Data
(data_id, data_ip, data_port) = Data.create_data()
#data(Redirect the message skip destination to a port that GPIO can easily handle
Data.set_data_redirect(data_connection_id, data_id, "127.0.0.1",
robot.port)
elif res['event'] == 'OPEN':
print('OPEN!')
time.sleep(1)
# ~abridgement~
media.py
After the connection with SkyWay is established in Peer, ** video / audio ** exchange with other Peer users will start.
Below are the methods threaded in webrtc_control.py.
media.py
# ~abridgement~
class Media:
"""Use MediaStream
Specify how to send and receive the MediaConnection object and the media to be transferred
"""
# ~abridgement~
#Threaded method
#Wait for media events
#Simply:Notify when the connected web application closes or an error occurs
@classmethod
def listen_media_event(cls, queue, media_connection_id):
"""Wait for an event on a MediaConnection object
"""
uri = "/media/connections/{}/events".format(media_connection_id)
while True:
_res = request('get', uri)
res = json.loads(_res.text)
if 'event' in res.keys():
#Throw an event to a queue
queue.put({'media_event': res['event']})
# close,error terminates this thread because the original media of this liten is released in another thread
if res['event'] in ['CLOSE', 'ERROR']:
break
else:
# print('No media_connection event')
pass
time.sleep(1)
# ~abridgement~
data.py After the connection with SkyWay is established in Peer, it will start exchanging ** data ** with other Peer users. This time, it will be the exchange of characters.
It's similar to Media, and there's nothing special to mention, so I'll skip it.
config.py This is a configuration file.
config.py
API_KEY = "Enter skyway API Key"
PEER_ID = "Enter any peer id"
HOST = "http://localhost:8000"
SHUTDOWN_LIST = ['Barth', 'Barusu', 'balus', 'balusu', 'barusu', 'barus']
--API_KEY: SkyWay API Key. --PEER_ID: PEER_ID on the home camera side. You can change it to any name you like. --HOST: It will be the HOST destination when you start the gateway. --SHUZTDOWN_LIST: Casting this word in a web app will shut down the program. (Bals: skull_crossbones :)
Recommendation of remote debugging
I used GPIO to get the video stream from L-Chika and USB camera, executed WebRTC-gateway, and because there was a lot of hardware dependence on Raspberry Pi, I developed it by remote debugging. In this case, I'm debugging with Intellij on my Mac, but actually running with code on the Raspberry Pi on the same network. This was very easy to develop because you can check machine-dependent functions while coding and debugging as usual on the Mac.
IntelliJ remote debugging method
Jetbrains IDE: IntelliJ method. Remote debugging is possible with PyCharm in the same way.
Select Python Debug Server from the Run/Debug configuration
Enter the IDE host name. A local IP address is also acceptable. (In this case, Mac)
Then enter any free port (also Mac)
Pathpapping is specified when you want to map directories between Mac and Raspberry Pi.
Next, perform 1.2 in the explanation column. It was easier to install with pip than to add eggs.
#Install on a running machine(In this case Raspberry Pi)
pip install pydevd-pycharm~=203.5981.155
#The version depends on the IDE.
Add command to code
In this case, add it to the beginning of the executable file webrtc_control.py.
import pydevd_pycharm
pydevd_pycharm.settrace(<IDE host name>, port=<port number>, stdoutToServer=True, stderrToServer=True, suspend=False)
You are ready to go. Debug execution
- First, debug and execute
Python Debug Serverwith IntelliJ on the Mac side. - In Raspberry Pi, execute the additionally coded code you want to debug
You can now debug your remote environment. You can see breakpoints and transitions of variables in the middle like normal debugging, which is convenient! !! : smiley:
It is recommended to keep the Mac on the development environment side with a fixed IP. Otherwise, you'll have to rewrite the IP part of the additional code every time ...
in conclusion
This can still be used as a pet camera, If I have a chance, I would like to attach a camera to the robot arm and move it remotely.
Very useful & useful to see your cat at home while traveling: cat: Thank you for reading this far!