Quick Stats:

Trick or Treaters Served: 323 🧛
Candy Distributed: 63.25lbs (~1650 pieces) 🍬

After our DMV themed Halloween costume last year, our Halloween crew was searching for a new costume idea that would be equally frightening for both children and adults. In March 2023, we had our answer.

What’s Silicandy Valley Bank?

In the months before Halloween, we began planning our costume with a rather ambitious objective: construct a fully functioning bank that utilized Halloween treats as currency, and have it fail in a spectacular fashion. Since it’s rather difficult to invest liquid treat capital into long-term Treasury bills, we needed a different way for our bank to collapse. We settled on using a combination of aggressive deposit incentives (to get large numbers of kids to open an account), ludicrous interest rates, and poor fiscal controls to try getting the bank to collapse in an obvious manner, with the hope of inciting a bank run as the obviously floundering bank obviously ran out of money (treats). As it turns out, building a fully functional bank that opens hundreds of accounts, tracks interest on said accounts, and then collapses spectacularly in a few hours is pretty complicated. The fully realized “Silicandy Valley Bank” had a lot of moving parts, and succeeded in its task of pleasing a large number of children while failing spectacularly, just not in the manner that was originally intended.

How was SVB supposed to work?

Getting kids to open a “bank account” with treats is actually a bit more difficult than doing it with cash. Candy is very much unlike fiat currency; in addition to being unable to pay your taxes with it, each piece of candy is rather unique. Besides the obvious flavor and size differences, kids care a lot about other details, like “has this chocolate bar been melted in somebody’s pocket” or “did someone lick all the sour flavor coating off of these Skittles”. We also wanted to avoid the potential liability associated with handing out treats originally gathered by one kid to a different kid (e.g. the aforementioned “pre-licked candy” issue). These issues could be avoided by sequestering treats by account number, so each kid would never receive treats that were originally deposited by a different kid, but this would be a huge inventory headache

In the end, we decided the easiest solution was to only give out treats that were roughly equivalent in value, and not accept actual deposits from trick or treaters. This was accomplished via an “incentive deposit” system.

The workflow was supposed to go something like this:

A Trick or Treater (TOT) shows up to the bank, and is provided a form to fill out in order to open their account.
The TOT waits in line then goes up to the next available banker to open their account.
The banker reviews the TOT’s form and uses their computer to open a new account. The TOT is provided with an initial incentive deposit of two pieces of candy. The TOT can withdraw one or both pieces of candy immediately, or leave the candy in their account with a ludicrously high interest rate, to come back and claim a higher balance later in the night.
The banker prints out a debit card for the TOT linked to their new account. Printed on the debit card is a web URL that can be used to view the account balance, as well as a QR code that can be scanned by a smartphone to open the same account page. When scanned by the banker, the same QR code is used to pull up the TOT’s account in order to make deposits or withdrawals.
TOT goes and does TOT things, while occasionally checking their account balance.
Just as the TOT’s balance begins to get really big looking, they see notifications on their account page (and hear from other kids) that the bank might be running out of candy soon, since so many other kids have opened high interest rate accounts, and the bank is only distributing candy (via new account incentive deposits + interest) and never actually increasing its reserves.
Chaos.
The Federal Treat Insurance Corporation steps in to save the day, disbursing candy up to the Insured Amount(TM) of each account.

For an added twist (since this somehow didn’t seem complicated enough), we added a referral system that would allow TOTs to open an account with another TOT’S referral code, thereby providing both TOTs with an additional piece of candy added to their account balance. We even got partway through implementing peer to peer account balance transfers on the day of Halloween, before things exploded in a rather spectacular fashion in a manner that we had not anticipated.

But first, let’s talk about the pieces of the costume that made it possible!

Piece 1: The Team

The most important part of this project was the excellent collection of friends who made it happen! As our most ambitious Halloween project yet, it wouldn’t have been possible without hundreds of person hours from 14+ contributors.

Riding off the success of 2022’s DMV costume, we had a dedicated crew of volunteers who were excited to do something crazy for Halloween once again. Our volunteers had a good spread of knowledge and engineering capabilities, ranging from data science to mechanical engineering and frontend web design. We did our best to tailor the costume to the skilled labor that we had available, while also creating something that could be accessible to TOTs. Many late nights were spent cutting vinyl, fiddling with ESC-POS receipt printers, rebuilding and ID card printer from the early 2000’s, resolving merge conflicts, and designing costumes and sets. When some of the more critical parts of our IT infrastructure tossed their cookies on the night of the big event, our crew pulled together to cover the gaps and keep the show running for hundreds of kids. I couldn’t have asked for a better team!

Costume Participants:

Kenneth McNelly, Noah Kjos, Caitlin Hogan, Paul Walter, Matthew Trost, Anika Hanson, Jillian MacGregor, Nicolas Weininger, Jeff Barratt, Scott Kwong, Alexander Zhang, Jason Kmec, David Gonzalez, Michal Adamkiewicz.

Special Acknowledgements:

Website and IT systems contributors: Caitlin Hogan, Morgan Tenney, Paul Walter, Jason Kmec, Cale Lester
Props: Jason Kmec, David Gonzalez, Alexander Zhang
Operations: Kenneth McNelly, Matthew Trost
Skit Actors: Matthew Trost, Paul Walter
Video Editing: Scott Kwong, David Gonzalez

Piece 2: The Website

As originally designed, the core of Silicandy Valley Bank’s IT infrastructure is its website, which contains both a customer-facing account interface as well as an internal website for creating customer accounts, updating account balances, and printing debit cards. There’s also a special “god mode” view, which allows an admin to adjust bank reserves and release a series of increasingly panicked pre-prepared “don’t worry everything is fine” blog posts on the website. The website was intended to serve as the glue between all the other puzzle pieces: bank employees would use it to process TOTs through the costume, TOTs and parents would use it to track the progress of their accounts and see the bank “run” in real time, and its various drivers would run all the required IT infrastructure on site, including barcode readers, receipt printers, ID card printers, and infotainment displays.

The public-facing homepage of the website was registered on a subdomain of my project website, pantsforbirds.com, with a proper HTTPS reverse proxy at https://svb.pantsforbirds.com (the server is no longer live, since the website is held together with spit and tape and only intended to run for a few hours–compound interest rates are high enough that if left running for a few weeks with nonzero interest rates, the website database has numerical overflow issues). This homepage displayed a live-updating chart of the bank’s current reserve balance, as well as the outstanding liabilities it had (in the form of treat deposits + interest). During the course of trick-or-treating, we were expecting to see the reserve balance plummet as TOTs withdrew the candy associated with their original incentive deposit and accrued interest. Simultaneously, we expected the liabilities balance to skyrocket as new accounts were created and interest accrued, since the bank was never receiving real deposits, and was strictly giving away candy with the free “incentive deposits” it added to every new account. When these two lines crossed, the bank would fold, as its liabilities would exceed its available reserves.

While bank reserves and liabilities are admittedly a bit of a complicated concept for trick or treaters to grasp, we hoped that parents might be able to clue their kids in that they might want to withdraw the candy sooner rather than later, leading to a desired “bank run” effect. Each TOT was intended to receive a debit card with a QR code that they could scan in order to visit their personalized account page, at https://svb.pantsforbirds.com/c/<customer_id>. This customer page would display their current account balance, current interest rate, as well as the bank’s overall reserves (so they could tell if a run was starting).

On the back end, an internal site was accessible via login for Silicandy Valley Bank employees. Interfaces were available for creating new customers, withdrawing account balances, and referrals. 2D barcode scanners connected via USB to each employee’s laptop allowed easy scanning of debit cards to quickly open a TOT’s bank account and withdraw treats as necessary.

Banker interface for creating new customer accounts and printing debig cards.

In addition, the internal website had a nifty interface for writing blog posts that would get posted to the website. Blog posts were categorized by level of panic, so they could be written en masse ahead of time and then rolled out in stages as the bank began to fail.

Interest rates and overall bank state were controlled from the “god mode” interface, allowing the bank to be initialized with an arbitrary amount of candy and tuned to collapse at a specific time by manually modifying interest rates to accommodate the rate of new account creation. A number of simulations were conducted to try predicting likely outcomes (time for the bank to run with various interest rates, incentive deposit amounts, and new account creation rates), which informed our initial incentive deposit amount and interest rate settings.

Interest accrual was calculated using a series of database entries we called “anchor events, which contained an account balance and timestamp. Each account would have an “anchor event” for when it was created (with its starting balance and interest rate), and for each subsequent modification to the account, be it an interest rate change, deposit, or withdrawal. Calculating an account’s current balance was done by pulling its latest anchor event from the database, and applying the anchor event’s interest rate to its anchor event balance using the continuously compounding interest formula.

Using this anchor event system allowed interest displays to be shown “live” with actively increasing numbers on the account view’s web frontend, with minimal load for the web server. Each live account view would poll the database for the latest account anchor event every 10 seconds or so, then use the anchor event parameters and the continuously compounding interest formula to create a lively scrolling numbers display. The hope was that seeing interest accrued in “real time” would encourage TOTs to keep their treats in the bank as long as possible to get the largest possible payout, thereby creating the correct incentive structure for a bank run when treats would begin to run out.

The website was run on an old Intel NUC as a series of docker containers, including containers for the Django webserver, PostgreSQL database, NginX reverse-proxy server, and SSL certbot (for installing HTTPS certificates). Once a DNS entry for the svb.pantsforbirds.com subdomain was set up with my domain registrar, getting the webserver port-forwarded through the network at my parents’ house was relatively straightforward.

Regrettably, the NUC needed to run Windows in order to make use of the available printer drivers for the ID card printer, which was running off a direct USB connection (using its integrated 10 BASE-T Ethernet connection was excruciatingly slow). This prevented SSHing into the web server and made me miss a number of other Linux utilities, but was still the lowest-effort path to getting things up and running in a limited amount of time. Remote work on the server was conducted using Chrome Remote Desktop and Git Bash, which worked quite well for getting things deployed in the days before Halloween when we weren’t on site.

Since getting USB interfaced with a docker container is a royal pain in the butt, especially on Windows, ID card print jobs were run from a local poetry environment, which ran a python print script that pulled ID card print jobs from the PostgreSQL database and forwarded them over USB to the ID card printer using system calls to PDF reader software installed on the system. In theory, the PostgreSQL database could have been exposed to the local network and the ID card printer script could have been run on a separate Windows device, allowing the primary server to be a Linux machine, but the added complexity of this alternate configuration was undesirable.

Remote view of the webserver via Chrome Remote Desktop.

Dockerizing the website containers allowed volunteers to easily develop on their own machines without needing to install dependencies separately, which was hugely helpful for allowing features to be built in parallel. A custom backup utility was written to archive copies of the website database and static files, allowing developers to easily share copies of the website in various “states” (open accounts, blog posts, etc). This made parallel development much easier, and we’ll definitely be copying it for future web projects!

Piece 3: The Set

Just like for our DMV costume, the event took place in my parents’ driveway, which we did our best to tranform into a bank interior with potted plants, service counters, stanchions, and even a lighted “exit” sign. Two of our volunteers lugged over their gigantic ~100lb infotainment display, which we used to show bank statistics and B-roll promoting Silicandy Valley Bank.

We also set up a pop-up tent at the front of the driveway with a banner and some tables, as a good spot to explain the workings of the bank and help kids fill out their account application forms before getting in line. A significant amount of thought was put into optimizing the layout for organized queuing and adequate flow of trick-or-treaters, since we wanted everyone to be able to visit the bank in a timely manner while also being able to see the action. Special attention was paid to securing large or heavy props (like the infotainment screen) to make sure they wouldn’t pose a risk to kids in a potentially crowded situation.

The videos shown on the infotainment displays were made from a combination of stock video footage and B-roll shot in our garage the weekend before. We aimed for maximum cheese / cringe factor via an amalgamation of generic corporate video montages and some more ridiculous “disruptive advertising” flavored shots.

Bank employees were issued uniform polos that were constructed with many painstaking hours of cutting designs out of heat transfer vinyl and using a sublimation press to adhere them to polos. Trick-or-treating bags with the bank motto and logo were constructed with the same techniques. FBI-style windbreakers for the FTIC officers were lettered using adhesive-backed vinyl, since we wanted to reuse the windbreakers in a future year, and we thought it was pretty unlikely that we’d have another use for “FTIC” branded apparel.

Hundreds of debit cards were printed during the week before halloween, since we’d previously found that the color printing process for ID cards was the bottleneck of our DMV operation (printing a single color ID card requires five consecutive passes with the ID card machine’s print head, as it prints through Cyan, Magenta, Yellow, Black, and Overlay panels of its ribbon). In order to speed things up this year, we separated the customized information on the ID card to the backside, which was printed in monochrome black, and left only standardized decorative elements on the front, which could be printed in color in advance of the event. Naturally, in the course of printing our debit cards, we ran into all manner of issues with the ID card printer, which first needed to be completely disassembled to have its aging drive belt replaced, and then later began melting ribbons and printing inconsistently (apparently it’s really hard to do a uniform solid color fill as the background to an ID card, since it requires intense and even heat across the full length of the ribbon). With some struggles and fine-tuning the print parameters for the ID cards, we managed to get just under 400 full-color debit cards printed before the event.

Last but not least, we purchased a lot of candy. Silicandy Valley Bank consumed 11x 5.75lb bags of candy. In the spirit of making treats as fungible as possible, all candies were non-chocolate “Funhouse Treats” with approximately equivalent size, but varying flavors. Trick or treaters were allowed to pick the number of treats corresponding with their withdrawal amount from a bowl filled with candy, ensuring that candy flavors were depleted relatively uniformly, while children would still receive the flavors they generally preferred.

Halloween

On the day of Halloween, things were coming together, but just barely in time! I pulled an all-nighter debugging and integrating a few last modules into the website, and we spent most of the day on Halloween assembling the set and testing IT infrastructure.

Around 5pm, trick or treaters began to gather at the top of the driveway to fill out their account forms, and shortly after that we started registering new accounts on the internal site. This worked flawlessly for approximately two minutes, after which a primary key error in the database prevented any additional accounts from being registered.

What followed was approximately 20 minutes of frantic debugging, while trick or treaters waited patiently in an increasingly lengthy line. Fried all-nighter brain does not a smart programmer make, and despite a number of repair efforts, we were unable to resolve the primary key issue. Eventually, the good decision was made to fall back to a completely manual banking protocol, where children would be issued a debit card with their name and account creation timestamp scribbled on the back in Sharpie. Children could choose to withdraw their full balance upon account creation, or could return to collect their balance at a later time, at which point interest would be manually calculated from the timestamp on their card and a fixed interest rate.

SVB Bank Manager (bottom right) views the unfolding chaos.

Without the core bank infrastructure online, there wasn’t an easy way to track deposits or change interest rates dynamically, so we had to drop the plans of inciting a real bank run and instead settled for operating as a normal bank. We proceeded with pre-planned skits of the SVB “bank manager” having some choice interactions with officers from the Federal Treat Insurance Corporation (FTIC), and did our best to reward trick or treaters who kept their candy in the bank with large quantities of plastic-wrapped sugar when they came to make their withdrawals.

FTIC agents shutting down the bank after taking control from the bank manager.

Despite the technical difficulties, the event was a smashing success. 323 trick-or-treaters opened accounts with the bank, and amazingly, the majority of them chose to leave their candy savings in the bank to accrue interest! The core mechanics of our costume could be boiled down to the basic structure of a marshmallow experiment, and I was entertained to note that our number of participants was much larger than the participant groups of many of the largest studies combined.

Trick-or-treaters and parents seemed to quite enjoy the SVB customer experience, and I was greatly entertained by young trick or treaters lecturing their siblings and friends about the power of compound interest and the importance of opening a savings account (little kids are smart)! Some trick or treaters even brought their DTV Treat Licenses that they had been issued the previous year, and attempted to use them as identification when opening their accounts!

Software RCA: The Importance of Integration Tests, and the Spirit of Halloween

Weeks after Halloween, once the dust had settled, I fired up the docker containers one more time to figure out why our database had pooped itself during the event. I knew that it was most likely related to our custom primary key generation functions; generating primary keys manually for a Django database is generally considered a Bad Idea™, but our previous DTV costume had done so without issue.

The cause of the database failure turned out to be a combination of factors.

I re-used some code from the DTV costume for generating primary keys, but didn’t pay close attention to the fact that the DTV primary keys were being generated with a different structure. DTV Treat License keys were generated with a fixed prefix based on the date and branch number, followed by a strictly incrementing series of numbers. SVB debit card IDs were generated with a prefix based on the trick-or-treater’s information (first name and costume), followed by an incrementing number. Some of the code I had reused from the DTV project was saving Customer models before their primary key was customized, leading to duplicate entries in the database in the case that multiple bankers were trying to create a new Customer model at the same time.
Test cases never tested the scenario where multiple Customer creation events were interleaved, and manual testing generally involved single users testing against their local website, so there weren’t opportunities for the primary key race condition to occur.

Oops! This one line of code borked our whole IT system.

The primary key issue has since been fixed (in case anyone wants to reuse our codebase for something else), and we’ve learned a very valuable lesson about the value of integration testing and thorough test coverage!

Perhaps more importantly, the biggest lesson that we learned was the importance of structuring an experience-centered project like a large-scale Halloween costume around people, and not technology. Seeing our volunteers adapt to keep the show running despite the near-complete failure of our IT system was extremely heartening, and made me realize that the while the IT system provided some additional interactivity and fun features, it was by no means the heart of the costume. The real core of our Halloween project was the team of friends who decided to dedicate many tens of hours of their time towards building a memorable experience for hundreds of trick-or-treaters, and I’m very excited to see what we do next, with or without technology!