Running Python micro-benchmarks using the ChatGPT Code Interpreter alpha

This is an incredible way to use ChatGPT. It requires some alpha plug-in access most people don't have.

Here's the initial prompt I sent it:

In Python create an in-memory SQLite database with 100 tables each with 10 columns.

Time how long it takes to execute PRAGMA schema_version against that database 100 times.

Then add another 100 tables and time PRAGMA schema_version 100 times again.

Now run the whole benchmark a second time, but instead of PRAGMA schema_version time how long it takes to run hashlib.md5(db.execute("select group_concat(sql) from sqlite_master").fetchall()[0]).hexdigest() instead

Okay, that is a wild set of requirements that would take a significant amount of time to write a testing harness for.

From start to finish, this entire benchmarking exercise took me less than five minutes—because ChatGPT did almost all of the work for me.

Willison goes on to tweak and modify this benchmark to make it more useful.

This whole process is remarkable. 🤯