• Presentation
• 2

Hands-on Workshop on Combining Snap! with the NVIDIA Jetson AI Embedded Systems

Together we will create a Snap! game based on fruits and vegetables detection (image classification).
Local and remote attendance possible!

Trainers: Harley, Ilgar, Ali and Rolf (in Heidelberg)

We will use the NVIDIA Jetson Nano computers we brought to the Heidelberg conference location. The reservation links for the Jetsons are further down.

We are able to perform object detection ('DetectNet') and segmantic image segmentation, too, but for this workshop we decided to concentrate on image classification for the sake of simplicity.

Reserve your NVIDIA Jetson Nano for the Let's plAIy! Workshop!

In the Let's plAIy! workshop at Snap!Con 2022 we will use AI in Snap! to create simple games based on image classification.
The image classification (an AI algorithm) is running on a NVIDIA Jetson.
We have 14 Jetsons available! First come, first served!

Participants in Heidelberg

• We have 7 Jetsons in total for attendees in HD
• Register under section Heidelberg to reserve one.
• You will connect through the local Wifi to the Jetson allocated to you.
• You will get your personal Jetson hostname after registration in the workshop.

Participants Abroad / Remote

• We have 7 Jetsons in total for attendees not in HD
• Register under section Abroad to reserve one.
• You will connect through VPN to the Jetson allocated to you.
• This requires to install VPN. See our VPN installation instructions
• You need additional personal username / password. We will send them by E-Mail to you after registration.

Installation Instructions

Please follow the INSTALLATION INSTRUCTIONS ON EOLAB WIKI

Motivation

AI at Schools

Artificial intelligence is a prevailing technology that poses opportunities and risks for society and the environment. Building knowledge and skills in schools is critical to enable students early on to master the technology instead of being mastered, to demystify AI, to acquire critical reflection competencies, to assess the opportunities and limitations of AI for problem solving. The usability of many AI toolkits continues to improve. Less and less expert knowledge is required to use them. This Simplification can be further advanced.

Combining Snap! with AI on NVIDIA Jetson

Artificial intelligence (AI) and machine learning (ML) are being integrated more and more into school curricula. This raises the interesting question of how AI and ML can be combined with Snap! Several groups are working on such integrations. Our approach presented and practiced in this workshop is using the AI power of NVIDIA Jetson Nano computers for image classification in Snap! The software is open source and no cloud is needed. Jetsons running Ubuntu are interesting for schools, since you can do many more things with them, e.g. build your own mobile robots (aka JetBots). Snap! could even run on the Jetsons but we decided to demonstrate a distributed system architecture.

Technical Background: Snap! and NVIDIA Jetson

We use SNAP! to communicate with image classifiers and object detectors based on convolutional neural networks (CNN). For many activities we are using NVIDIA Jetson Embedded AI computers with GPU. The Jetsons are extremely versatile and NVIDIA provides very good AI software optimized for the 'little' machines.

Snap! is used to take video frames from the local user PC webcam. These frames (pictures) showing up on the Snap! stage are transmitted via a JavaScript websocket to the AI server.

Image Classification

We use ImageNet. To be continued ...

Object Detection

In the current setup we utilize the object detection software DetectNet running on a NVIDIA Jetson Nano embedded computer as a backend server.

This backend object detector reports the bounding box, the object type, and the confidence level for each identified object on the image back to Snap! This information is overlaid with the picture in Snap! This object metadata can be used to control Snap! programs such as games.