AI Parental Controls: Using LLMs for Context-Dependent Self-Moderation (w/ Transformer.js)

I recently hit a roadblock on my odyssey towards social media abstinence. My slash-and-burn technique of all-out blocking YouTube turned out to be more harmful than helpful to my productivity. As it turns out, perhaps unsurprisingly, interspersed between Mr Beast spectacles and Minecraft play-alongs is a lot of genuinely useful educational content. As a visual learner, I found depriving myself of that content to be particular disadvantagous for my learning. Thus, I began searching for an alternative solution.

My first instinct was to white-list specific channels that I knew made educational-leaning content (3blue1brown, Jack Herrington, etc). I hacked together a quick Tapermonkey script and began white-listing channels. If I encountered a video I wanted to watch from a channel I hadn't white-listed, I'd add it... Within a week, I had a whitelist long enough to no longer make it useful. Next, I tried downloading the occasional video I needed via yt-dlp. This worked for a while since it meant I could keep YT blocked, but I found it required far too much self-constraint and was too time consuming and I quickly abandoned it.

What I needed was something that would vet a given YouTube video automatically, without manual intervention or maintenance. As a baseline, the tool would need to be able to distinguish educational content from entertainment, but I also wanted the tool to take my particular learning goals into consideration so I wouldn't inadvertently veer off into tangential educational content (e.g., The Roman Empire).

In theory, this sounded like something an LLM should be capable of. I was skeptical, since the line between education and entertainment is invariably blurry on YouTube, but curious enough to build something out.

Experiment

Version 1 - Chat

My first instinct was to load an LLM with some context about the video (title, description, etc), and then block the video behind a chat interface if the LLM deemed it "unsuitable".

I started with a simple prompt that defined my learning goals and asked the LLM to only allow ‘relevant’ educational content:

You are responsible for ensuring that I stay productive and only watch videos that would be beneficial to my learning.

Right now I am learning about:
- Transformers, LLMs, Linear Algebra, and other related topics.
- React, TypeScript, and other related topics.
- WebGl, Three.js, and other related topics.

If you think the video should be allowed, please respond with "I'll allow it".

For each blocked video, I would then have to plead my case for why I thought the video was appropriate to watch. The LLM, acting like a benevolent parental guardian, would either accept or deny my request.

I threw together a small script, loaded it into Tapermonkey, and hit the ground running.

With more sophisticated models, like the GPT-4 series, this worked surprisingly well. More often than not, I agreed with the decision the LLM made to block or unblock a given video. However, there were two major challenges with this approach:

1. The chat interface was unbearably clunky and frustrating. The overhead of going back and forth with a model to plead my case, only to get denied (even if rightly so) was annoying enough for me to not want to use the extension.
2. Conversations with smaller, but cheaper, models like GPT-3.5-turbo quickly devolved. Sometimes the model would lose the context and the chat would derail. Other times, the model thought I was the one determining if the video could be viewed and would start pleading the case for why it should be allowed to view the video.

Version 2 - Direct

For those reasons, I abandoned the chat interface relatively quickly in favor of just letting the LLM directly approve or deny the video without user input. This came with the trade-off of not being able to reason with the LLM, or see why it decided a video should be blocked (which was helpful for improving the prompt), but the speed and ease of use more than made up for the lack of visibility.

I removed the chat interface in favor of a floating ‘status’ pill:

And make a few small tweaks to the end of the prompt:

You may respond with one of the following **and nothing else**:
"Yes" - if the video is allowed
"No" - if the video is not allowed

This time around, I wanted to be slightly more rigorous in my testing so I compiled two lists of videos that I thought should pass and fail the filter.

Unsurprisingly, the results were nearly identical to the chat version except now I didn’t have to chat with the LLM first. In hindsight, I’m not actually sure how often I convinced the LLM to change its mind in the chat interface version. Unscientifically, I failed to record those instances when I did convince it to let me view a video but they were rare.

At this point, I was mostly satisfied with the extension except for two major flaws:

1. For smaller models, ‘educational’ content that had clickbaity titles wouldn’t pass since it didn’t have the context about the actual video content. On the other hand, videos that seemed relevant but weren’t (like this one on “AI taking over”) did pass.
2. I had indirectly turned YouTube into a paid service and I hated the idea of paying (however small) to view a video.

To mitigate the first issue, I tried supplying the video transcript. This had a small, but noticeable, improvement in the LLM's ability to vet a given video for the smaller models with larger windows like GPT-3.5-Turbo. For GPT-4 there was no difference except now the inference cost more with the added tokens. If I’d had more time, I would have experimented with different transcript sampling techniques (e.g., random snippets) or run the transcript through a different LLM to summarize it first.

To help with the second issue, I used transformers.js to load models I could run for free directly in the browser. I’m sure you can guess that this made the extension significantly slower and, since the models weren’t as good, all around less useful. I iterated through a few different models (Xenova/LaMini-Flan-T5-783M seemed to work the best) but in all honesty, I never quite got this to be good enough to replace even GPT-3.5. If I had more time, I would have experimented with using better models or tweaking the prompt.

Results

Here are the results of running each of the models in the final variation (no-transcript, no-chat, temperature at 0) through the two test lists of videos:

Keep in mind this is by no means a rigorous study considering there were only 8 videos in the list. Even so, GPT-4 was actually quite good at filtering the videos purely from a description, a title, and a list of topics I’m learning about.

In the end, I was quite pleased with how well using an LLM as a personalized content filter worked. But, after a week of daily usage on the GPT-4 version, I ended up disabling the extension and went back to an all-out weekday YT ban. The reason was primarily because:

1. YouTube is free and using this extension was not. Actually, it was pretty expensive (especially when I had transcripts turned on). A week of usage cost me about $2 ($104 a year)! As LLMs become more cost-effective, I imagine this will be less of an issue.
2. The line between ‘educational’ and ‘entertaining’ content is far too blurry to achieve my real goal of improving my personal productivity. I found myself binging ‘educational’ content that met my learning goals in the same way I would with more entertainment-focused content.

Conclusion

There are many forms of higher-level, non-monotonous, cognitive thinking that we have yet to offload to LLMs. In this particular exercise, I was concerned with the idea of willpower. That is, can a reasonably sophisticated LLM be a suitable substitute for my own willpower and self-control. Is there a world where I no longer need self-accountability and self-monitoring because, with a well-tailored prompt, an LLM can make those decisions for me? Can an LLM, with my best interests in mind, alleviate the ‘burden’ of staying on task in a noisy digital world?

I don’t know. But, I hope this exercise illuminates for you, as it did for me, that it is a possibility. I see a world where, after understanding and absorbing your life goals, an LLM ensures you’re always on task and is constantly molding your digital world to ensure you stay ‘on track’.