Integrate your Raspberry Pi with Amazon Web Services IoT (AWS IoT) for Real-Time Data Collection and Analysis

Learn how to connect a Raspberry Pi to the AWS IoT platform, collect real-time data from sensors, and analyze it in near-real time using AWS services like Lambda, Kinesis Firehose, and QuickSight. …


Updated October 17, 2023

Need help with your Raspberry Pi?
Contact Me!

Do you love silly Raspberry Pi Projects?
Check out my this YouTube Channel!


Learn how to connect a Raspberry Pi to the AWS IoT platform, collect real-time data from sensors, and analyze it in near-real time using AWS services like Lambda, Kinesis Firehose, and QuickSight.

Raspberry Pi is a popular single-board computer used for various applications including robotics, automation, and IoT projects. Amazon Web Services (AWS) is a cloud computing platform that provides a wide range of services to build, deploy, and manage applications. AWS IoT Core is a fully managed service that makes it easy to securely connect devices to the AWS Cloud and enables real-time data collection and analysis. In this article, we will show you how to connect your Raspberry Pi to AWS IoT Core, collect sensor data using Python, and process the data in real time using AWS services like Lambda and Kinesis Firehose. We will also use Amazon QuickSight for visualizing the data in near-real time.

Prerequisites:

Before you start, make sure you have the following:

  1. Raspberry Pi with an internet connection (either WiFi or Ethernet).
  2. AWS account - You can sign up for a free tier account here.
  3. Python 3 installed on your Raspberry Pi - You can install it using sudo apt-get update && sudo apt-get upgrade && sudo apt-get install python3
  4. Breadboard, jumper wires, and a few sensors (e.g., temperature sensor, light sensor).

Step 1: Setup Raspberry Pi and Sensors

First, connect your sensors to the Raspberry Pi using a breadboard and jumper wires. You can follow any online tutorial for this step. Make sure you have connected power supply (either via USB or a power source) to your Raspberry Pi.

Step 2: Install AWS CLI on Raspberry Pi

We will use the AWS Command Line Interface (CLI) to communicate with AWS IoT Core. To install it, run the following commands:

sudo apt-get update && sudo apt-get upgrade -y
curl "https://awscli.amazonaws.com/awscli-exe-linux-arm.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Step 3: Create an AWS IoT Thing and Certificates

  1. Sign in to the AWS Management Console and navigate to AWS IoT Core.
  2. Click on “Manage” -> “Things” in the left-hand menu, then click “Create” to create a new thing. Give it a name (e.g., “RaspberryPi”) and click “Next”.
  3. Skip the “Add Thing Type” step by clicking “Next”.
  4. In the “Add Certificates for This Thing” page, select “Create certificate” and download the certificate.pem.crt, private.pem.key, and public.pem.key files. Keep these files safe as you will need them later.
  5. Click “Activate” to finish creating the thing.

Step 4: Configure AWS CLI on Raspberry Pi

  1. Copy the downloaded certificates (certificate.pem.crt, private.pem.key, and public.pem.key) to your Raspberry Pi.
  2. Run the following command to configure the AWS CLI with your AWS credentials:
    aws configure

    Enter your access key ID, secret access key, default region (e.g., us-east-1), and output format (json) when prompted.

  3. Configure AWS IoT using the following command:
    aws iot describe-endpoint --endpoint-type iot:Data-ATS

    This will return an endpoint URL (e.g., abc123xyz456.iot.us-east-1.amazonaws.com). Save it for later use.

Step 5: Install MQTT Client on Raspberry Pi

We will use the Eclipse Paho MQTT client to connect your Raspberry Pi to AWS IoT Core. Run the following command to install it:

sudo pip3 install paho-mqtt

Step 6: Write Python Script to Collect Sensor Data and Publish to AWS IoT

Create a new Python script (e.g., publish_sensor_data.py) and add the following code:

import time
import json
import random
from AWSIoTPythonSDK.MQTTLib import AWSIoTMQTTClient
import Adafruit_DHT # Replace with your sensor library

# Set your AWS IoT Core endpoint URL (obtained in Step 4) and certificates path
aws_iot_endpoint = "abc123xyz456.iot.us-east-1.amazonaws.com"
certificate_path = "/home/pi/certificates/" # Update this path to match your setup

# Replace with your sensor type and GPIO pin
sensor = Adafruit_DHT.DHT222
gpio_pin = 4

# Initialize MQTT client
mqtt_client = AWSIoTMQTTClient("RaspberryPi")
mqtt_client.configureEndpoint(aws_iot_endpoint, 8883)
mqtt_client.configureCredentials(certificate_path + "certificate.pem.crt", certificate_path + "private.pem.key", certificate_path + "public.pem.key")
mqtt_client.configureOfflinePublishQueueing(-1)  # Infinite offline Publish queueing
mqtt_client.configureDrainingFrequency(2)  # Draining: 2 Hz
mqtt_client.configureConnectDisconnectTimeout(10)  # 10 sec
mqtt_client.configureMQTTOperationTimeout(5)  # 5 sec
mqtt_client.connect()

# Function to read sensor data and publish it to AWS IoT
def collect_and_publish_sensor_data():
    while True:
        humidity, temperature = Adafruit_DHT.read_retry(sensor, gpio_pin) # Replace with your sensor reading function
        
        if humidity is not None and temperature is not None:
            data = {"temperature": temperature, "humidity": humidity}
            mqtt_client.publish("$aws/things/RaspberryPi/shadow/update", json.dumps(data), 0) # Replace "RaspberryPi" with your thing name
        else:
            print("Failed to read sensor data")
        
        time.sleep(10) # Adjust the sleep interval as needed

collect_and_publish_sensor_data()

This script reads the sensor data (in this case, temperature and humidity), formats it in JSON format, and publishes it to AWS IoT Core using MQTT. Replace Adafruit_DHT with your specific sensor library and update the GPIO pin number accordingly.

Step 7: Run Python Script on Raspberry Pi

Run the script using the following command:

python3 publish_sensor_data.py

You should see the sensor data being published to AWS IoT Core in real time.

Step 8: Create an AWS Lambda Function for Data Processing

  1. In the AWS Management Console, navigate to AWS Lambda and click “Create function”.
  2. Choose “Author from scratch” and give your function a name (e.g., “RaspberryPiDataProcessor”).
  3. Select Python 3.8 as runtime and create a new role with basic Lambda execution permissions.
  4. In the “Function code” section, enter the following code:
import json

def lambda_handler(event, context):
    # Extract sensor data from MQTT message
    message = event["Records"][0]["Sns"]["Message"]
    data = json.loads(message)
    
    # Process the data (e.g., store it in a database or trigger an alert)
    print("Temperature:", data["temperature"])
    print("Humidity:", data["humidity"])
  1. Under “Designer”, add an SNS topic as a trigger for your Lambda function (create one if needed). The topic should match the one subscribed to by your Lambda function (e.g., raspberrypi).
  2. Click “Deploy” and test your function using the sample MQTT message from earlier:
{
  "temperature": 25,
  "humidity": 40
}

If everything works correctly, you should see the temperature and humidity values printed in the Lambda function’s logs.

Step 9: Subscribe AWS IoT to SNS Topic

  1. In the AWS Management Console, navigate to AWS IoT Core and click “Manage”.
  2. Click on “Act” and select “Subscribe to SNS topic”. Choose the SNS topic created earlier (e.g., raspberrypi) and click “Subscribe”.
  3. Confirm the subscription in your email.

Now, every time sensor data is published from Raspberry Pi, it will be forwarded to the SNS topic, which triggers your Lambda function for processing. You can add more logic to this Lambda function to store the data in a database or trigger alerts based on certain conditions.