Snapchat AR Creator Residency – MyTomato

Pitch

For the 2020 Snapchat AR Creator Residency, I brought together a team of three and submitted the concept for the SnapChat Lens MyTomato, an AR Tomato growing lens which showed a growing tomato plant mixed in with the user’s environment.

The initial pitch was made through a Google Form, which made customizing difficult, however I was able to link to an Imgur album which allowed for more information on what was intended. In the Imgur album, I showed the life stages of the tomato plant, how it would look in the real world, as well as some of the decisions and complications caused by player action.

Screenshot of an Imgur album from the original pitch for MyTomato. The image shows a cartoon drawing of a tomato plant just starting to flower projected onto a table next to a real plant.

Lens Studio

For the residency itself, the three of us had to learn to use LensStudio and deliver the final product all within the month. I focused on the engine and programming, while Tyler focused on Shaders, SFX and VFX and Megan focused on modeling, texturing and animation.

Surprisingly, given the constraints of the Lens, the final product ended up resembling my initial scribbles more than expected. SnapChat Lenses have a total of 4 MB of available storage, requiring us to have very limited textures. Megan managed to make the textures quite small through using gradients whenever possible.

Screenshot of Lens Studio, showing both the scene view and the preview of what the app would look like on a phone.

Templates and Methods

For the project itself, I started with the DepthToybox template, which allowed for a 3d object placable in the world, taking into account the estimated volume of the real world space. While there were other templates that would have mapped out the world space better, they were more limited in the hardware that they worked on.

I set up a game controller that would track and store variables of the plant age, whether or not it was infested with bugs, as well as whether or not it was altered, storing those remotely with SnapChat. During the initial pitch, I had planned to also access RL location and weather data, so the user wouldn’t have to water their plant if it was raining that day and the like.

Rescoping

The original pitch involved having the plants grow in real time over the course of 10 weeks, checking to see if the users had consistently watered the plants (if necessary) and showing the growth of the tomato plant over time.

One of my mentors suggested that this assumed a lot of player buyin, and that if players didn’t get the full experience of the tomato growth at first, they wouldn’t come back to try it out later. While I do think they are right, I do think the project lost something special in switching away from the real-time and real world interactions.

In the future, I do want to experiment more with integrating real life weather information into VR and AR experiences.

 

 

 

Animation and Material Switching

For the actual growth of the tomato plants, we actually went with a larger animation, where the growth of the plant would push it towards specific key frames in the animation and then stay paused on that frame, instead of switching between models.

Depending on whether or not the plant was dry or infected with bugs, I would swap out the materials and turn on or off various VFX. This was made more complicated by LensStudio having different categories of materials, effectively one that was standard materials, and one that were custom shadered materials, with different commands for applying each.

The last major feature added was dragging the watering can, or the gardening gloves onto the plant to either water it, or remove the bugs. While this worked, it wasn’t as flashy as I had hoped. If we had more time, I’d likely want something a bit more interactive, like an actual modeled watering can that you would tilt over the plant.

Even with these limits, it was an interesting experience, and one that showed a lot of potential in the future, especially with building off some of the other templates, and the features of the SnapChat API.