I was playing around with this for use in my video games but ended up not using any of it, at least for now. It's annoying when you learn a bunch of stuff and it ends up not being useful! So I figured I might as well write a blog post about it.
The goal here is to move data from IndexedDB to a file without reading all of the data into memory at once. If you are able to read your data into memory, you can create a
blob and use
URL.createObjectURL to download it to a file - but that's old news. This is about streaming.
The building blocks of this are two fairly new web APIs: the Streams API and the File System Access API. The File System Access API is currently only supported in recent versions of Chrome, but it's the only way to stream data to a file.
What about getting data out of IndexedDB? The IndexedDB API predates streams, so it has no built-in support for that. But it does have cursors, which allow you to iterate over data in your database, which is basically the same thing.
I got a fun email earlier today - a support request for literally the second piece of software I ever wrote, back in 2001 when I was a kid with a couple months of programming under my belt.
It's a click tracker that I called Click Manager. Pretty simple stuff - a Perl CGI script that counts how many times a link was clicked, storing the data in a flat file database.
Eventually I even added a nifty UI to view the stats. Check it out, in all its early 2000s glory:
I made another mini project for no apparent reason. You can load one or many CSV files into a SQL database within your web browser (client side) and run SQL queries on them. In most situations it's not that useful (most databases already support importing CSVs without too much trouble), but it might save somebody some time in a pinch. And it's cool that things like this are even possible, purely client side.
Code is on GitHub.
Since this is my blog, it makes sense to talk about the greatest technical accomplishment of my life.
...except I already wrote a long blog post about it. Just not here, over on the Basketball GM blog.
TLDR: I got an order of magnitude performance improvement out of a mature piece of software, with no negative tradeoffs from an end user perspective.
Over at Basketball GM, when I upgraded Prebid.js from 0.34 to 1.x, most of the bid adapters could just be easily copied over. Index Exchange was the one exception. For a while it wasn't availble at all on 1.x, but that changed with version 1.10.0, released a few weeks ago. However it requires some changes to your configuration to upgrade from 0.34 to 1.10+, and I found the documentation to be a bit lacking, so I wrote a blog post!
God damn, I've been neglecting this blog. It's sad really, because I do have a lot to say. I guess I've just been talking to myself instead of blogging lately, which maybe says something about my mental stability, but whatever. I'm blogging now, and I'm going to blog the fuck out of this blog.
It's 2016. My job is pretty cool in some ways, but in some other ways it bothers me a great deal. It's probably not in my best interest to go into that in great detail here (as if anyone is reading this, right?) so I will leave the rest unsaid, and just get to the broader point. I am someone who cares a great deal about science, engineering, creating cool things, doing things the right way, etc. And when I say "cares a great deal" I mean probably more than I can adequately articulate with my pedestrian writing skills. It's almost like a spiritual thing. So for someone like me, what do you do when your day job is preventing you from attaining your desired level of spiritual satisfaction?
Chris Stucchio wrote an article about the differences between basic income and basic job policies, based on relatively straightforward math. Briefly, basic income says give everyone money with no strings attached and get rid of other forms of welfare. Basic job is the same, except anyone who can work is mandated to work, either in a normal job like today or in a New Deal-style government works program.
Chris's main conclusion was that basic job came out looking way better than basic income. Additionally, a major purpose of his post was to encourage other people to play around with the math as well rather than just bloviating. Since I'm a big basic income proponent and have some quibbles with how he came to conclude that basic income doesn't look too good, I will follow his lead and play around with the math.
Despite miraculously recovering from ACL surgery and successfully leading his team for the playoffs, Adrian Peterson tragically missed the all time rushing record by 9 yards.
...or did he?
Let's think about how the NFL measures yardage. They take the difference between where the ball was before the play and where the ball is after the play, and then they round to the nearest integer. What happens if you rush for half a yard? It'll get recorded as either 0 yards or 1 yard. Spread out over an entire season, and this kind of rounding error can have a big impact.