<![CDATA[dumbmatter.com]]>http://dumbmatter.com/metalsmith-feedFri, 27 Aug 2021 05:26:15 GMT<![CDATA[I think the pandemic is about over]]>Why make these Covid posts? Isn't the Internet saturated with hot takes already? Am I really adding anything here?

I think the only reason for me to write about Covid is so I have a record to look back on what I thought at the time, which is kind of interesting for me, but maybe not so interesting for you :)

And what I think now is that the pandemic in the US is about over.

I know, there are pretty bad outbreaks in a lot of places right now. Some hospitals are overflowing with patients. That is very bad.

However those situations seem to mostly be peaking, or approaching a peak. I don't think they'll get substantially worse. And then, at some point in the somewhat near future, they will hopefully start to get better.

And when that happens, I predict that we'll never again face as big of a Covid outbreak as we just have. It'll mostly fade into the background of other normal diseases people get.

That was mostly the idea behind vaccination, that if we could vaccinate everyone, then it would either prevent infections or result in mild infections. That still seems to be true, despite the Delta variant. The vaccines are holding up pretty well, despite what you may hear from unreliable sources like the media and the government.

Of course, not everybody is vaccinated. That's a problem, sure. But there's a couple big factors that I think will still lead to the end of the pandemic, despite a significant unvaccinated popuation.

  1. Vaccination rates are not constant with age. A lot more older people are vaccinated than younger people. And older people are at much higher risk from Covid, so younger people being unvaccinated is less of a concern.

  2. There is at least some imperfect evidence suggesting that natural infection provides even better immunity than vaccination.

At this point, the vast majority of the vulnerable population is vaccinated, previously infected, or dead. I know there is heterogeneity all over the place and that matters a lot, but how many places have a large population of vulnerable people with no protection against Covid? Places like that are the ones being hit hard by Delta now. But the harder they are hit, the more people move into that "previously infected" category. And Delta is so infectious that I wonder how many places like that can really be hiding from it still.

Anyway, I'm not here to do any rigorous modeling to actually make a convincing case of anything. It just seems like it's going to be increasingly difficult for Covid to cause huge outbreaks like this in the US anymore. So maybe the pandemic is about over?

]]>
http://dumbmatter.com/posts/pandemic-over.mdhttp://dumbmatter.com/posts/pandemic-over.mdFri, 27 Aug 2021 00:00:00 GMT
<![CDATA[Streaming data from IndexedDB to a file with the File System Access API]]>I was playing around with this for use in my video games but ended up not using any of it, at least for now. It's annoying when you learn a bunch of stuff and it ends up not being useful! So I figured I might as well write a blog post about it.

The goal here is to move data from IndexedDB to a file without reading all of the data into memory at once. If you are able to read your data into memory, you can create a blob and use URL.createObjectURL to download it to a file - but that's old news. This is about streaming.

The building blocks of this are two fairly new web APIs: the Streams API and the File System Access API. The File System Access API is currently only supported in recent versions of Chrome, but it's the only way to stream data to a file.

What about getting data out of IndexedDB? The IndexedDB API predates streams, so it has no built-in support for that. But it does have cursors, which allow you to iterate over data in your database, which is basically the same thing.

That gives the general idea... somehow turn an IndexedDB cursor into a stream and send that to a file with the File System Access API.

That "somehow" in the previous sentence is doing a lot of work though! IndexedDB is a notoriously difficult API to work with. In this case, the sticking point is that IndexedDB transactions automatically close whenever they are not active, and "not active" includes things like "waiting for some other async activity to resolve". And you know what involves a lot of asynchronous stuff? Streams. So if you build a naive readable stream on IndexedDB, you run into inactive transaction errors.

A solution is to do something like this:

<button id="button">Stream</button>

<script type="module">
import { openDB } from "https://unpkg.com/idb@^6?module";

const STORE_NAME = "test";

const connectDB = async () => {
  const db = await openDB("streamingTestDB", 1, {
    upgrade(db) {
      const store = db.createObjectStore(STORE_NAME, {
        keyPath: "id"
      });

      for (let i = 0; i < 1000; i++) {
        store.add({
          id: i,
          random: Math.random(),
        });
      }
    },
    blocked() {
      throw new Error("blocked");
    },
    blocking() {
      throw new Error("blocking");
    },
    terminated() {
      throw new Error("terminated");
    },
  });

  return db;
};

const makeReadableStream = (db, store) => {
  let prevKey;

  return new ReadableStream({
    async pull(controller) {
      const range = prevKey !== undefined
        ? IDBKeyRange.lowerBound(prevKey, true)
        : undefined;

      let batchCount = 0;

      let cursor = await db.transaction(store).store.openCursor(range);
      while (cursor) {
        controller.enqueue(`${JSON.stringify(cursor.value)}\n`);
        prevKey = cursor.key
        batchCount += 1;

        if (controller.desiredSize > 0) {
          cursor = await cursor.continue();
        } else {
          break;
        }
      }

      console.log(`Done batch of ${batchCount} object`);

      if (!cursor) {
        // Actually done with this store, not just paused
        console.log("Completely done");
        controller.close();
      }
    },
  }, {
    highWaterMark: 100,
  });
};

const getNewFileHandle = async () => {
  const handle = await window.showSaveFilePicker({
    suggestedName: "foo.txt",
    types: [
      {
        description: "Text Files",
        accept: {
          "text/plain": [".txt"],
        },
      },
    ],
  });
  return handle;
};

document.getElementById("button").addEventListener("click", async () => {
  const fileHandle = await getNewFileHandle();
  const writableStream = await fileHandle.createWritable();

  const db = await connectDB();
  const readableStream = makeReadableStream(db, STORE_NAME);

  await readableStream.pipeTo(writableStream);
});
</script>

makeReadableStream returns a ReadableStream that pulls data from IndexedDB. The highWaterMark of 100 means it will read up to 100 records into memory as a buffer before pausing. Since our test data has 1000 records and reading data from IndexedDB is faster than writing to disk, this ensures we see streaming behavior. It will load 100 (and a few more) records in the first batch, and then pause until it's ready to load more, while storing the place it left off in prevKey. Then each time more data is requested from our readable stream, it creates a brand new IndexedDB transaction, starting from prevKey.

There's a problem with this code though! pull gets called each time the buffer falls under highWaterMark. That often means it's called when there are 99 records in the buffer, resulting in a new transaction being created just to pull a single additional record, which shows up on the console as a bunch of "Done batch of 1 object" messages. That's kind of slow because there is some cost associated with creating a transaction.

To work around that, I introduced another variable MIN_BATCH_SIZE. I'm using that in addition to the built-in controller.desiredSize to ensure that if we're going to go through the trouble of creating a transaction, we're at least going to use that transaction for multiple records. Here's the final code:

<button id="button">Stream</button>

<script type="module">
import { openDB } from "https://unpkg.com/idb@^6?module";

const STORE_NAME = "test";

const connectDB = async () => {
  const db = await openDB("streamingTestDB", 1, {
    upgrade(db) {
      const store = db.createObjectStore(STORE_NAME, {
        keyPath: "id"
      });

      for (let i = 0; i < 1000; i++) {
        store.add({
          id: i,
          random: Math.random(),
        });
      }
    },
    blocked() {
      throw new Error("blocked");
    },
    blocking() {
      throw new Error("blocking");
    },
    terminated() {
      throw new Error("terminated");
    },
  });

  return db;
};

const makeReadableStream = (db, store) => {
  let prevKey;

  return new ReadableStream({
    async pull(controller) {
      const range = prevKey !== undefined
        ? IDBKeyRange.lowerBound(prevKey, true)
        : undefined;

      const MIN_BATCH_SIZE = 100;
      let batchCount = 0;

      let cursor = await db.transaction(store).store.openCursor(range);
      while (cursor) {
        controller.enqueue(`${JSON.stringify(cursor.value)}\n`);
        prevKey = cursor.key
        batchCount += 1;

        if (controller.desiredSize > 0 || batchCount < MIN_BATCH_SIZE) {
          cursor = await cursor.continue();
        } else {
          break;
        }
      }

      console.log(`Done batch of ${batchCount} object`);

      if (!cursor) {
        // Actually done with this store, not just paused
        console.log("Completely done");
        controller.close();
      }
    },
  }, {
    highWaterMark: 100,
  });
};

const getNewFileHandle = async () => {
  const handle = await window.showSaveFilePicker({
    suggestedName: "foo.txt",
    types: [
      {
        description: "Text Files",
        accept: {
          "text/plain": [".txt"],
        },
      },
    ],
  });
  return handle;
};

document.getElementById("button").addEventListener("click", async () => {
  const fileHandle = await getNewFileHandle();
  const writableStream = await fileHandle.createWritable();

  const db = await connectDB();
  const readableStream = makeReadableStream(db, STORE_NAME);

  await readableStream.pipeTo(writableStream);
});
</script>

And here's a runnable version of it (only works in recent versions of Chrome, at the time of writing):

There's a lot more functionality in all of the APIs used here, but hopefully this is a useful minimal example of how to put them all together. And yet, none of this thrills me. A simpler or more efficient way to stream data out of IndexedDB would be pretty neat. If you can think of some way to improve this, please let me know!

]]>
http://dumbmatter.com/posts/streaming-data-from-indexeddb.mdhttp://dumbmatter.com/posts/streaming-data-from-indexeddb.mdThu, 03 Jun 2021 00:00:00 GMT
<![CDATA[An 18 year old bug]]>I got a fun email earlier today - a support request for literally the second piece of software I ever wrote, back in 2001 when I was a kid with a couple months of programming under my belt.

It's a click tracker that I called Click Manager. Pretty simple stuff - a Perl CGI script that counts how many times a link was clicked, storing the data in a flat file database.

Eventually I even added a nifty UI to view the stats. Check it out, in all its early 2000s glory:

Anyway, point is, someone emailed me about a bug in Click Manager. That absolutely made my day, to learn that somebody was still using my old script.

I took a look at it. Turns out he was using version 2.2.5, from 2003. That is not the latest version though! The latest version is version 2.2.6, from 2005. (You can actually still download it from my website, but I like linking to those nice old layouts that archive.org has saved.)

After looking at a diff between version 2.2.5 and 2.2.6 (this was back before I used version control) it became clear that the only thing version 2.2.6 did was fix the exact bug he emailed me about!

Moral of the story: please update your software at least once per decade, or this might happen to you :)

]]>
http://dumbmatter.com/posts/an-18-year-old-bug.mdhttp://dumbmatter.com/posts/an-18-year-old-bug.mdTue, 04 May 2021 00:00:00 GMT
<![CDATA[Do Covid lockdowns still make sense in the US?]]>There are two possible goals that a government might have when imposing lockdown. The first goal is to eradicate the disease. The second goal is to prevent overloading hospitals with tons of sick patients at the same time. This is the "flatten the curve" strategy, where the idea isn't really to prevent people from getting infected, but to spread out the infections over time.

Those two goals are pretty different. Eradicating the disease is much harder. It requires a much stricter lockdown, and it is much more difficult to achieve when the disease is widespread in the population. Flattening the curve is easier (not "easy", just "easier") because it requires a less strict lockdown.

The problem is, like I mentioned a couple months ago, flattening the curve does not give us a good solution to the pandemic. If the disease spreads until we achieve natural herd immunity or develop a vaccine, the death toll will be high. And we might have to flatten the curve for a very long time, which would have huge negative impacts on many aspects of life.

Based on these two possible goals of lockdowns, where are we now and where do we go from here? Most US states, including my home state of NJ, have implemented lockdowns. I believe this was necessary at the time because there were too many unknowns, mostly because the pitiful state of testing meant we didn't know where the disease had spread. Due to the high number of asymptomatic infections and the incubation period between infection and symptoms, there was concern that hospitals could become overloaded in many parts of the country.

It has since turned out that some parts of the country had very high infection levels, but most didn't. Hospitals were overrun in parts of New York, and basically nowhere else. Changes in behavior have reduced the reproduction rate of the virus sufficiently that there is no imminent risk of hospitals being overloaded. And improvements in testing capacity mean that in the future, we will likely be able to identify a rapidly-growing outbreak early enough to deal with it. A stricter lockdown could be imposed to stop an outbreak from growing further, and medical resources could be diverted to the area to prepare for an increase in hospitalizations.

Basically what I'm saying is, we have flattened the curve, and hospitals are unlikely to become overloaded.

What about eradication? That does not seem to be the goal of the federal government or any state or local government. Even if we were trying for eradication, I'm not sure if we could reasonably achieve it, given how widespread the virus is and how Americans tend not to like the government telling them what to do. So there's not much value in talking about eradication, until we have natural herd immunity or a vaccine. Which could be years off.

As I said, flattening the curve is not a good solution. But it's what we're doing now, and I don't see a feasible alternative. We're not going for eradication. A vaccine is too distant and uncertain. We have no alternative but many deaths and natural herd immunity. The only question is how long it will take to get there. To most efficiently reach this end state, we should open the country as much as possible, while also doing the type of monitoring described above to prevent overloading hospitals. Any lockdown more severe than that will only prolong the pain.

And yes, I am aware that natural herd immunity may not be possible for COVID-19. I wrote about that a couple months ago. But even if there is only an X% chance that natural herd immunity works, it's still the best option we have. Eradication is still completely infeasible. A vaccine is still too distant and uncertain. I wish I had a better answer.

]]>
http://dumbmatter.com/posts/do-covid-lockdowns-still-make-sense.mdhttp://dumbmatter.com/posts/do-covid-lockdowns-still-make-sense.mdFri, 22 May 2020 00:00:00 GMT
<![CDATA[A simple explanation for why modeling COVID-19 is hard]]>Over at FiveThirtyEight there is a great article about why it's so hard to model the effects COVID-19. Basically their answer is that there are many factors that go into a model, but many of them are very uncertain, and many of them are also dynamic. For instance, what is the probability of transmission when an infected person interacts with a non-infected person? There's a lot of uncertainty in that estimate. But also, it's going to change over time. Particularly, as the pandemic worsens, people will likely do more social distancing and other mitigation strategies, resulting in a lower transmission rate.

Tricky stuff to predict precisely! But I think that's not quite the complete picture, and there's an even simpler and clearer explanation.

I'm thinking about this issue more because of the IHME COVID-19 model. They are trying to predict hospital resources needed to treat COVID-19 patients, which is even harder than modeling the spread of COVID-19. As new data comes in and they tweak their model, sometimes the results change a lot. This has led to articles like HUGE! Official IHME Model for Coronavirus Used by CDC Just Cut Their Numbers by Half!... They're Making It Up As they Go Along! getting shared a lot on social media.

And that interpretation of the IHME model is understandable. This is supposed to be the gold standard synthesis of all expert opinion that goverments use to set policy. Sure there's a lot of uncertainty in it, but to cut their predictions in half in a single day seems beyond the pale.

But consider exponential growth, since infectious diseases tend to spread exponentially, not linearly.

Imagine a hypothetical disease that starts in 1 person, and then it spreads to 2 new people every day. After a month, it will have spread to over 2 billion people. Exponential growth is wild!

Now imagine a slightly different scenario. It doesn't spread to 2 new people every day, it just spreads to an average of 1.8 new people every day. There's not a big difference between 2 and 1.8, right? Just a 10% difference. Well, in the 1.8 case it will only spread to 100 million people in a month. That's 95% fewer cases, just by decreasing the transmission rate by 10%.

Models involving exponential growth are incredibly sensitive to their parameters. This is not the fault of the people who make the models, it's the fault of math and reality. So if you have a model with many uncertain parameters that govern exponential growth, expect your projections to be wrong. Very wrong.

Does that mean modeling should not be done in cases like this? Definitely not. Models can still help us understand the range of possible outcomes, even if that range is wide. We just need to be careful about how we interpret the results.

Update, September 2020: At this point it's pretty clear that the IHME model kind of sucks. What I wrote above is still true about the fundamental challenge of modeling, particularly in the early days of a pandemic. But now you're probably better off looking at more competent models like covid19-projections.com. Their projections have been much more accurate than the IHME's.

]]>
http://dumbmatter.com/posts/why-modeling-covid-19-is-hard.mdhttp://dumbmatter.com/posts/why-modeling-covid-19-is-hard.mdTue, 07 Apr 2020 00:00:00 GMT
<![CDATA[My take on COVID-19]]>I have a bit of time on my hands right now, so I figured I'd write up my current COVID-19 take. Not really because I think anyone cares. I mean, there are many better informed takes out there. I'm mostly writing this for myself, so I can look back on it and see how my perception has changed.

I've been worried about COVID-19 for a while, but there were ways to imagine it not getting too bad. Like maybe it would be contained by the early response in China. Or maybe there was some genetic or environmental factor that would limit its spread or its deadliness.

I've been very worried ever since Italy took a turn for the worse around February 23. Their number of known cases jumped from ~10 to ~100, and things progressed rapidly from there.

On February 24 I started stocking up on nonperishables, figuring that there was a good chance that a time would come when I'd have to lay low for a few weeks. I also started telling my friends and family to similarly prepare, which resulted in multiple people telling me they were a little freaked out by my concern, since I tend not to get worked up about whatever "crisis" dominates the news cycle.

As the news trickled in from the US and Europe, it gradually became clear that many countries were indeed on the same trajectory as Italy. Western countries did not act as swiftly and strongly as East Asian countries like South Korea, Japan, and Singapore. Most troubling, particularly in the US, was the lack of testing. All we knew about the spread of the disease came from severe cases, but for each severe case there are many asymptomatic cases, some of which will soon be severe. This lack of information continues to impede decisionmaking.

Today, I believe there are two big outstanding questions.

First, how many people have already been infected by COVID-19 and experienced no or mild symptoms? Without knowing how many people will catch COVID-19 and ultimately be fine, it's really hard to say what the appropriate political response is. This article by John Ioannidis really hammers the point home, while questioning if our response might be too strong. While I do hope Ioannidis's skepticism turns out to be correct, I wouldn't bet on it. I agree with Nicholas Christakis when he points out that there is at least some evidence about the fatality rate of people with COVID-19, and it doesn't very look good; and a huge number of cases happening all at once is a big deal.

Regarding the huge number of cases happening all at once, there is much talk of "flattening the curve". Unfortunately, if the pandemic is bad enough, flattening the curve becomes kind of infeasible. If you need to flatten the curve for 10 years, that's never going to work. But like I mentioned above, we really lack the data to conclusively say much here.

My second big outstanding question is, what is the long term prognosis of people after infection? Do they retain some level of immunity? How good is that immunity? And how long does it last? Already there have been isolated reports of individuals catching COVID-19 twice, suggesting some very pessimistic answers to these questions. And I have read that evidence from similar viruses suggests immunity may not last forever. However we shouldn't jump to conclusions without more data. And this is not just a matter of testing, like the first question. It is also a matter of time.

These questions are important because they will determine the success or failure of government responses to COVID-19. Of course, ideally the outbreaks would have been limited by more testing and more careful social practices. But in the US and Europe, it seems like the cat is out of the bag. So what? Strict lockdown like Wuhan? Let it burn through the country and develop herd immunity? Something in between?

To my first outstanding question, since we don't really know how bad COVID-19 will be if it just burns through the population, it's hard to evaluate those options. I believe that the available data suggests that lockdowns are appropriate, but there's much uncertainty.

More troublingly, to my second outstanding question, we don't know if either of those strategies will actually work! If people can be reinfected with COVID-19 multiple times, without experiencing a substantial period of immunity, then herd immunity will never happen. So we lockdown... and then what? As soon as the lockdown ends, the pandemic will resume. So we let it burn through the country... and then what? Next year it just burns through again, since nobody is immune?

Scary scenarios. We simply don't know enough right now. We absolutely should be doing as much testing and monitoring as possible to help answer these questions. And hopefully we find that it's not as severe as originally thought, and that catching it once confers long term immunity. Time will tell. But right now, it seems like there's a distinct possibility that Earth is now simply a worse place for humans than it was before, and we'll just have to learn to live with that.

]]>
http://dumbmatter.com/posts/covid-19-take.mdhttp://dumbmatter.com/posts/covid-19-take.mdTue, 17 Mar 2020 00:00:00 GMT
<![CDATA[What happened to the Pete Buttigieg "High Hopes" dance?]]>In late 2019, a few videos appeared online showing some Pete Buttigieg supporters doing a corny dance to Panic! at the Disco's "High Hopes". Here's one, and another, yet another, and still one more.

The rapid release of all those different videos suggested to me that this was not an isolated incident. Crazed Mayor Pete fans must be doing that lame dance all over the place!

I was excited to see more videos. But sadly that never happened. Those are still the only four Pete Buttigieg dance videos I have ever seen.

So what happened? Did Mayor Pete decide to crush the high hopes of his supporters by banning their fun little dance? Or are they still doing it, but with extreme levels of security to prevent further online ridicule?

]]>
http://dumbmatter.com/posts/what-happened-to-the-pete-buttigieg-high-hopes-dance.mdhttp://dumbmatter.com/posts/what-happened-to-the-pete-buttigieg-high-hopes-dance.mdMon, 24 Feb 2020 00:00:00 GMT
<![CDATA[Porting Basketball GM to TypeScript]]>I'm gonna do that thing again where I link to a post on my Basketball GM blog that possbily is of broader interest.

]]>
http://dumbmatter.com/posts/typescript.mdhttp://dumbmatter.com/posts/typescript.mdMon, 20 Jan 2020 00:00:00 GMT
<![CDATA[Moving from Browserify to Rollup]]>I'm gonna do that thing again where I link to a post on my Basketball GM blog that possbily is of broader interest.

]]>
http://dumbmatter.com/posts/browserify-to-rollup.mdhttp://dumbmatter.com/posts/browserify-to-rollup.mdTue, 17 Sep 2019 00:00:00 GMT
<![CDATA[Why do NCAA tournament brackets lack symmetry?]]>Very important stuff here!

Take a look at a portion of a normal NCAA tournament bracket. This is one region, cut off at the sweet 16, with all the favorites filled in:

Let me highlight the first round games that lead to the sweet 16 teams, assuming all the favorites win:

Isn't that literally the worst thing you've ever seen? No symmetry. I can see why you put the 2 seed at the bottom, so that the 1 and 2 approach each other from opposite ends. I can see why you do the same for the 4 seed.

But why leave the 3 seed in no man's land? This looks much better, with the 6-11 and 3-14 games swapped:

This doesn't change any of the games played, it just makes for a more elegant and symmetric bracket.

Am I missing something here? If not, does anyone know how we wound up in this horrible situation? It's hard to Google for information, because I just find articles about ranking teams and placing teams in the bracket, not about the actual structure of the bracket itself!

]]>
http://dumbmatter.com/posts/ncaa-bracket-symmetry.mdhttp://dumbmatter.com/posts/ncaa-bracket-symmetry.mdTue, 27 Aug 2019 00:00:00 GMT