Author: hugtech

Let’s make a simple observing with Notification powered by AWS deeplens and Amazon Alexa.

Today’s Goal

The goal of this post is that you can feel fun and you feel that using the AI and Alexa is easier.

Background Story

  • I’m a Japanese. I’m really missing Japanese foods.
  • Fortunately, I can get the really fresh fish every weekend at the Market.
  • I want to store these fishes longer. So I decided to make a dried fishes.

I tried to dry fishes at our balcony. And They came from the sea to pick up my lovely fishes.

Thus, I decided to observe our enemies to save my fish.

Designing of Observation

Details

1. AWS Deeplens

The deeplens provides the power of AI easier. We can deploy pre build training model to the deeplens for several simple steps.

https://aws.amazon.com/deeplens/

2. Amazon Echo

For notifying to me, I could use Amazon Echo.

Let’s go to cooking.

Setting and Deploying the Object Detection Model to the deeplens

Left : selecting Project template on the deeplens console
Middle: MQTT topic filter on the deeplens console
Right: testing dialog on the AWS IoT console

The deeplens provides the Project Template to implement models easier.

Select the Object Detection in Project template.

MQTT

MQTT is the lightweight M2M protocol. When the model deployed to the deeplens, MQTT topic which sends the detecting status is deployed too. You can see on the deeplens console.

Also, you can test that the handling messages in AWS IoT Core console. You can access it from Project output column.

As you can see, Simple message is receiving on AWS Iot Core console like this.

{ "chair": <percentage of confidence> }

when some bird coming, the deeplens will send message as follows.

{ "bird": <percentage of confidence> }

Make the Alexa skill with notification

OK. Now We could make Object detection part. Let’s make Alexa Skill for accepting notification. To do this, We have to configure the Manifest file for using the Alexa proactive API. The API provides a capability to send notification which Alexa defined schemas.

Defined Schemas
https://developer.amazon.com/ja/docs/smapi/schemas-for-proactive-events.html

Unfortunately, We can only use a defined schema. So In this demo, I alternatively use the WheatherAlert Schema for notifying as assuming the bird to storm. 🙂

AMAZON.WeatherAlert.Activated

If you want to add some schemas, You can send a request to add a new schema in the Alexa Developer Forum.
https://alexa.uservoice.com/forums/906892-alexa-skills-developer-voice-and-vote

To use proactive API is a really simple modification in Alexa Skill Manifest (skill.json). You only add the permission block and the events block

Then, You can deploy by using ASK-CLI.

ask new
git clone https://github.com/haruharuharuby/server-room
ask deploy

Source:
https://github.com/haruharuharuby/server-room

After deploying succeeded, Let see Alexa Developer Console. And check the ClientId and ClientSecret in the permission dialog. (these are used later)

Deploy Lambda function

Until now, You deployed two front interfaces individually. (deeplens, alexa skill). So Let’s connect each other!! This lambda code does two things.

  • Filtering the message to pick up the specific word ‘bird’
  • alling Alexa proactive API

To deploy this function, You need to pass 2 steps.

Step1: Add clientId and clientSecret to the parameter store on AWS System Manager

If you can use AWS CLI. Run the script. (Of course, You can set t on AWS Management Console)

aws ssm put-parameter --type String --name bird-detection-client-id --value <your-client-id>
aws ssm put-parameter --type String --name bird-detection-client-secret --value <your-client-secret>
aws ssm put-parameter --type String --name bird-detection-topic-filter --value <your-mqtt-topic-filter>

Step2: deploy lambda function by the serverless framework

The serverless framework is really useful to deploy function and around resources. This script deploy Lambda function and set the trigger from AWS IoT Rules.

git clone https://github.com/haruharuharuby/bird-detect-message-handler
cd bird-detect-message-handler
serverless deploy

Source:
https://github.com/haruharuharuby/bird-detect-message-handler

After deploying, you can see lambda function and set the trigger on the AWS IoT Rules.

Now current configuration assumes the message has 80% confidence of the bird.

Note: AWS IoT Rules

AWS IoT Rules is a feature for filtering the message by (like) SQL query.

If you want to change the topic filter, and notify to another skill, You just modify serverless.yml

:
functions:
  hello:
    handler: handler.handler
    events:
      - iot:
          name: 'birdDetection'
          sql: "select bird from '${self:custom.iot.topicFilter}' where bird > 0.800"
    environment:
      STAGE: ${self:custom.stage}
      CLIENT_ID: ${self:custom.alexa.clientId}
      CLIENT_SECRET: ${self:custom.alexa.clientSecret}
      PROACTIVE_AUTH_ENDPOINT: https://api.amazon.com
      PROACTIVE_EVENT_ENDPOINT: https://api.amazonalexa.com
      ALEXA_NOTIFICATION_EXPIRY_MINUTES: ${self:custom.alexa.notificationExpiryMinutes}

If you are in EU, or Asia Pacific region, You should be changed the PROACTIVE_EVENT_ENDPOINT to appropriate one.

https://developer.amazon.com/docs/smapi/proactive-events-api.html#call-proactive

Let’s take a check!

This film is for testing. I was setting the AWS IoT Rules as follows.

select 'person' from <<topic-filter>> where 'person' > 0.600

All Done! We could use AI power and Alexa without deeply knowledge of Machine Learning 🙂

Let’s enjoy Alexa style 🙂

https://www.npmjs.com/package/ask-cli

https://serverless.com/framework/docs/providers/aws/guide/variables/#reference-variables-using-the-ssm-parameter-store

https://docs.aws.amazon.com/deeplens/latest/dg/deeplens-getting-started.html

https://docs.aws.amazon.com/iot/latest/developerguide/iot-rules-tutorial.html