Coding Test for Llama 3: Implementing JSON Persistence

While building a decentralized Twitter in a previous post, I included some code that implemented JSON persistence. With the launch of Meta’s Llama 3 this month, I thought it’d be a good opportunity to explore how a new LLM can help with coding.

Meta.AI is the chat frontend for Llama 3, which we will use after developing our persistence solution. As I write this (late April), Meta.AI is not available globally, so you may need to use a VPN to get to it. You may need to use a Facebook login for some things, like image generation, but that doesn’t seem to be a necessity.

First, I checked that I was actually talking to Llama 3, although this was not totally conclusive:

When looking at a previous LLM (Llama 2), I asked it to do some long multiplication that it failed to manage. It did however “fail like a human.” Let us see how this model has progressed:

A quick, and much better-looking answer. Still wrong; the answer is 1,223,834,880. However, we aren’t here to worry about this known lacuna. Let’s return to the main topic: persistence with JSON.

Saving with JSON

JSON is a portable data format, and while less efficient than an arbitrary native binary format, it can at least be checked by humans and implemented anywhere.

Because JSON can handle basic value types and some collections, we can in general map our instance data to JSON format. This is known as marshalling and is a lot less of an issue today than it used to be. Whether you want to persist (i.e., save and load) user preferences, a turn in a game or just some plain old useful data, JSON has your back.

In this post I’ll use .NET and C#. While we can do just fine with the inbuilt System.Text.Json, the Newtonsoft package is a bit nicer, so I’ll call that.

First I’ll set up a console project that will use the new style top-level statements to cut out the need for main statements and other boilerplate. So in a terminal:

I started with a quick Car class, which I wrote directly in Program.cs. We will use this as our target object:

The console output after running this is:

To use the Newtonsoft.JSON package from the command line, we can just run this in the terminal:

You could also use the palette within VS Code, though I find that a little unreliable sometimes. So now that we have pushed the code for the car object to its own file, Car.cs, and namespace, CarClass, let’s do a simple conversion in Program.cs:

The output to this in the terminal is:

Oh, it didn’t work? But this is fine. The enum used for CarColor needs to be handled separately as it isn’t a type recognized by JSON. We could write the conversion ourselves or just leave it to the existing converter by decorating the property in Cars.cs:

The output result is now correct:

What we want to work toward for a round trip is to persist a couple of cars down to a JSON file, restore, and then update one aspect (for instance, a garage service) and save back, finally checking the JSON on file. First, let’s check that we can save a list down to a file:

The file produced is fine, containing two JSON objects:

Let’s generalize the persistence part and put it in its own project file:

Note that the two methods that actually call JsonConvert do not need to know anything about the type being converted; they work with any generic list of types.

Here’s the round trip, which amends the second car’s service mileage:

The resulting file is even properly formatted:.

OK, so we created some persistence to a simple file using JSON, using a specific package. We ended up with these files in our project workspace:

Now let’s try to see how far we can get with Llama 3. Let’s break down what we did.

• Created a simple class to represent an owned car.
• Turned it into json.
• Used the json format to save and load into a local file.

So first let’s ask for a car class. The query is “Create a C# class that represents an owned car with these attributes: brand, make, miles since last service and color.”

Great. It also gave examples of using the class and setting the properties. Let us see if we can persuade it to make the color property an enum:

It did! Even with my spelling mistake in the request!

Now let’s see if it can use JSON to convert. Let’s force it to use Newtonsoft.json too.

It thinks this will be the output:

Of course, we already know this won’t work as it is. We need to fix the enum. We could try to tell Llama this, but we only know it isn’t correct because we’ve been here before. However, let’s take the win.

The good news is that Llama 3 was quick, well formatted and good with basic requirements.

• It allowed us to work on the code in a continuous fashion over time in one thread.
• It understood how to use an arbitrary named package.
• It correctly created example instances of Car, as well as some nice car colors.
• The answers are nicely formatted.

Where it went wrong was a detail that I happened to have worked through in advance. This leaves us in roughly the same place we’ve been in all the time with LLM coding: It can quickly create a working template that we can tailor, but it will have inaccuracies that will require fixing. This means it is a useful bootstrapping tool for a working developer but not yet a sort of magic solution maker.

SUBSCRIBE

David has been a London-based professional software developer with Oracle Corp. and British Telecom, and a consultant helping teams work in a more agile fashion. He wrote a book on UI design and has been writing technical articles ever since....

Read more from David Eastman