Jekyll2024-01-20T00:16:27+00:00https://haacked.com/atom.xmlYou’ve Been HaackedYou've been Haacked is a blog about Technology, Software, Management, and Open Source. It's full of good stuff.
Phil HaackFailure suuuuucks2023-11-13T00:00:00+00:002023-11-13T00:00:00+00:00https://haacked.com/archive/2023/11/13/failure<p>When you fail, many people will tell you how failure is a great teacher. And they’re not wrong. But you know what else is a great teacher? Success! And success is a lot less expensive than failure.</p>
<p><img src="https://github.com/haacked/haacked.com/assets/19977/4ca7b2c4-d353-4282-bb17-1713e74b9fbe" alt="Fallen ice cream cone" /></p>
<p>About a month ago, my co-founder and I decided to shut down our startup, A Serious Business, Inc., the makers of Abbot. He wrote some <a href="https://www.linkedin.com/feed/update/urn:li:activity:7114700034398425088/">beautiful words about it on LinkedIn</a>. Now it’s my turn to write some less than beautiful words about the experience.</p>
<p>Before I get all maudlin about failure, let me say that the experience of building a company from scratch with a close friend and amazing team was one of the most rewarding experiences of my career. We built a great company, team, and product. The only thing we failed to do was the only thing that mattered for a startup — obtain product market fit.</p>
<p>I’ve been very fortunate in my career. I’ve encountered so little failure. Not because I’m so great, but because I haven’t taken huge risks until now. The biggest risk I remember taking was leaving my cush high-paying job at Microsoft in order to join a scrappy little startup for much less pay. It so happens that startup was GitHub. In retrospect, not that much of a risk, though it felt like it back then. So yeah, I’ve been lucky. Very lucky.</p>
<p>Back to the main topic, why didn’t we achieve product market fit? I’ve been reflecting on that question a lot, but I keep running into a stumbling block.</p>
<p>By now, most of us are familiar with the idea of <a href="https://en.wikipedia.org/wiki/Survivorship_bias">survivorship bias</a> as exemplified by the famous airplane image:</p>
<p><img src="https://github.com/haacked/haacked.com/assets/19977/4f69f803-2f8c-4d7a-aa7d-2eabec54cf25" alt="image of returning airplane showing bullet holes" /></p>
<p>For those who don’t know, survivorship bias is the logical error of looking at the survivors (or successes) of a process and drawing conclusions without also considering the failures.</p>
<p>During World War II, military researchers studied the distribution of bullet holes from returning aircraft and wanted to add armor to the areas where bullet holes were concentrated. A Hungarian mathematician (Abraham Wald) suggested differently. He noted that the planes that did not return were not being considered. He suggested adding armor to the areas without bullet holes as it’s likely the reason those areas were sparse in bullet holes was because the planes that were hit there did not return.</p>
<p>I think the same bias occurs when examining failures. Perhaps we should call it Failureship Bias. If that term catches on, you’ve heard it here first.</p>
<p>For example, one question I’ve pondered is whether our tech stack held us back. I’ve said many times in the past that the tech stack is the least interesting part of a company. The product market fit is all that matters in the beginning and later on, the company culture, the ability to sell, etc.</p>
<p>But to reach product market fit, you have to be able to shotgun features at the wall and see what sticks. Fast experimentation is really key. Chris Wanstrath (aka defunkt) <a href="https://twitter.com/defunkt/status/1724186412738826499">tweeted the following today</a>:</p>
<blockquote>
<p>I started learning Rails in 2005 and doing it professionally in 2006. By 2007, when we started GitHub, I had already worked on or made dozens of sites. The velocity was a huge part of the appeal - we could create new features fast!</p>
</blockquote>
<p>At A Serious Business, Inc. I chose ASP.NET Core and C# because I knew <em>I</em> would be faster with it than any other stack. I helped build that stack. Even so, there is still much ceremony and paper cuts when it comes to the inner loop of development. It may not seem like much, but that shit adds up. For example, compilation and startup time when making changes compounds. I would love to have ASP.NET Core interpreted while in local development. Or interpreted while it’s background compiled.</p>
<p>So did the stack hold us back? Again, going back to Failurship Bias, I can’t run a double blind experiment where another team with the same exact circumstances builds the same exact product using Rails and see if they survive. Maybe some day we can peek into parallel universes and I can see how Bill Maack, the Rubyist, fares.</p>
<p>Having said that, there was another team who built a product very similar to ours and seems to be doing well. They also went through the YCombinator program like we did. Is it their stack that helped them? Or did they benefit from the <a href="https://insight.kellogg.northwestern.edu/article/the_second_mover_advantage">second-mover advantage</a>? Or is it the fact that all three of the co-founders live and work in the same apartment. In their own words, this is all they do.</p>
<p>Perhaps all of those are reasons why they succeeded and we did not. Perhaps not. I hope that’s not what it takes because I’m not willing to move into an apartment with my co-founder. I love him, but not that much. And his family and my family probably would object.</p>
<p>So what <em>is</em> the lesson I’ve learned from this failure? Well as I said in the title. It really suuuuuucks. But don’t cry for me Argentina. The experience of building a product with wonderful people was its own reward.</p>
<p>And I did gain some ideas that I want to experiment with the next time I start a company. I’m just sober enough to understand that if my next company succeeds, it’s just as likely that it was luck in-the-moment as it was the lessons I learned from this failure. But hey, I’ll take it.</p>Phil HaackWhen you fail, many people will tell you how failure is a great teacher. And they’re not wrong. But you know what else is a great teacher? Success! And success is a lot less expensive than failure.Calling internal ctors in your unit tests2023-05-01T00:00:00+00:002023-05-01T00:00:00+00:00https://haacked.com/archive/2023/05/01/calling-internal-ctors<p>One of my pet peeves is when I’m using a .NET client library that uses internal constructors for its return type. For example, let’s take a look at the <a href="https://www.nuget.org/packages/Azure.AI.OpenAI"><code class="language-plaintext highlighter-rouge">Azure.AI.OpenAI</code> nuget package</a>. Now, I don’t mean to single out this package, as this is a common practice. It just happens to be the one I’m using at the moment. It’s an otherwise lovely package. I’m sure the authors are lovely people.</p>
<p><img src="https://user-images.githubusercontent.com/19977/235283338-9c406f6d-77b6-4669-9273-4c90bf821487.png" alt="Inside a room looking outside at a construction site" /></p>
<p>Here’s a method that calls the Azure Open AI service to get completions. Note that this is a simplified version of the actual method for demonstration purposes:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">Completions</span><span class="p">></span> <span class="nf">GetCompletionsAsync</span><span class="p">()</span> <span class="p">{</span>
<span class="kt">var</span> <span class="n">endpoint</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">Uri</span><span class="p">(</span><span class="s">"https://wouldn't-you-like-to-know.openai.azure.com/"</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">client</span> <span class="p">=</span> <span class="k">new</span> <span class="n">Azure</span><span class="p">.</span><span class="n">AI</span><span class="p">.</span><span class="n">OpenAI</span><span class="p">.</span><span class="nf">OpenAIClient</span><span class="p">(</span><span class="n">endpoint</span><span class="p">,</span> <span class="k">new</span> <span class="nf">DefaultAzureCredential</span><span class="p">());</span>
<span class="kt">var</span> <span class="n">response</span> <span class="p">=</span> <span class="k">await</span> <span class="n">client</span><span class="p">.</span><span class="nf">GetCompletionsAsync</span><span class="p">(</span><span class="s">"text-davinci-003"</span><span class="p">,</span> <span class="k">new</span> <span class="n">CompletionsOptions</span>
<span class="p">{</span>
<span class="n">Temperature</span> <span class="p">=</span> <span class="p">(</span><span class="kt">float</span><span class="p">)</span><span class="m">1.0</span><span class="p">,</span>
<span class="n">Prompts</span> <span class="p">=</span> <span class="p">{</span> <span class="s">"Some prompt"</span> <span class="p">},</span>
<span class="n">MaxTokens</span> <span class="p">=</span> <span class="m">2048</span><span class="p">,</span>
<span class="p">});</span>
<span class="k">return</span> <span class="n">response</span><span class="p">?.</span><span class="n">Value</span> <span class="p">??</span> <span class="k">throw</span> <span class="k">new</span> <span class="nf">Exception</span><span class="p">(</span><span class="s">"We'll handle this situation later"</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This code works fine. But I have existing code that calls Open AI directly using the <a href="https://www.nuget.org/packages/OpenAI"><code class="language-plaintext highlighter-rouge">OpenAI</code></a> library. While I work to transition over to Azure, I need to be able to easily switch between the two libraries. So what I really want to do is change this method to return a <code class="language-plaintext highlighter-rouge">CompletionResult</code> from the <code class="language-plaintext highlighter-rouge">OpenAI</code> library. This is easy enough to do with an extension method to convert a <code class="language-plaintext highlighter-rouge">Completions</code> into a <code class="language-plaintext highlighter-rouge">CompletionResult</code>.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">static</span> <span class="n">CompletionResult</span> <span class="nf">ToCompletionResult</span><span class="p">(</span><span class="k">this</span> <span class="n">Completions</span> <span class="n">completions</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="n">CompletionResult</span>
<span class="p">{</span>
<span class="n">Completions</span> <span class="p">=</span> <span class="n">completions</span><span class="p">.</span><span class="n">Choices</span><span class="p">.</span><span class="nf">Select</span><span class="p">(</span><span class="n">c</span> <span class="p">=></span> <span class="k">new</span> <span class="n">Choice</span>
<span class="p">{</span>
<span class="n">Text</span> <span class="p">=</span> <span class="n">c</span><span class="p">.</span><span class="n">Text</span><span class="p">,</span>
<span class="n">Index</span> <span class="p">=</span> <span class="n">c</span><span class="p">.</span><span class="n">Index</span><span class="p">.</span><span class="nf">GetValueOrDefault</span><span class="p">(),</span>
<span class="p">}).</span><span class="nf">ToList</span><span class="p">(),</span>
<span class="n">Usage</span> <span class="p">=</span> <span class="k">new</span> <span class="n">CompletionUsage</span>
<span class="p">{</span>
<span class="n">PromptTokens</span> <span class="p">=</span> <span class="n">completions</span><span class="p">.</span><span class="n">Usage</span><span class="p">.</span><span class="n">PromptTokens</span><span class="p">,</span>
<span class="n">CompletionTokens</span> <span class="p">=</span> <span class="p">(</span><span class="kt">short</span><span class="p">)</span><span class="n">completions</span><span class="p">.</span><span class="n">Usage</span><span class="p">.</span><span class="n">CompletionTokens</span><span class="p">,</span>
<span class="n">TotalTokens</span> <span class="p">=</span> <span class="n">completions</span><span class="p">.</span><span class="n">Usage</span><span class="p">.</span><span class="n">TotalTokens</span><span class="p">,</span>
<span class="p">},</span>
<span class="n">Model</span> <span class="p">=</span> <span class="n">completions</span><span class="p">.</span><span class="n">Model</span><span class="p">,</span>
<span class="n">Id</span> <span class="p">=</span> <span class="n">completions</span><span class="p">.</span><span class="n">Id</span><span class="p">,</span>
<span class="n">CreatedUnixTime</span> <span class="p">=</span> <span class="n">completions</span><span class="p">.</span><span class="n">Created</span><span class="p">,</span>
<span class="p">};</span>
<span class="p">}</span>
</code></pre></div></div>
<p>But how do I test this? Well, it’d be nice to just “new” up a <code class="language-plaintext highlighter-rouge">Completions</code>, call this method on it, and make sure all the properties match up. But you see where this is going. As the beginning of this post foreshadowed, the <code class="language-plaintext highlighter-rouge">Completions</code> type only has <code class="language-plaintext highlighter-rouge">internal</code> constructors for no good reason I can see. So I can’t easily create a <code class="language-plaintext highlighter-rouge">Completions</code> object in my unit tests. Instead, I have to use one of my handy-dandy helper methods for dealing with this sort of paper cut.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">static</span> <span class="n">T</span> <span class="n">Instantiate</span><span class="p"><</span><span class="n">T</span><span class="p">>(</span><span class="k">params</span> <span class="kt">object</span><span class="p">[]</span> <span class="n">args</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">type</span> <span class="p">=</span> <span class="k">typeof</span><span class="p">(</span><span class="n">T</span><span class="p">);</span>
<span class="n">Type</span><span class="p">[]</span> <span class="n">parameterTypes</span> <span class="p">=</span> <span class="n">args</span><span class="p">.</span><span class="nf">Select</span><span class="p">(</span><span class="n">p</span> <span class="p">=></span> <span class="n">p</span><span class="p">.</span><span class="nf">GetType</span><span class="p">()).</span><span class="nf">ToArray</span><span class="p">();</span>
<span class="kt">var</span> <span class="n">constructor</span> <span class="p">=</span> <span class="n">type</span><span class="p">.</span><span class="nf">GetConstructor</span><span class="p">(</span><span class="n">BindingFlags</span><span class="p">.</span><span class="n">NonPublic</span> <span class="p">|</span> <span class="n">BindingFlags</span><span class="p">.</span><span class="n">Instance</span><span class="p">,</span> <span class="k">null</span><span class="p">,</span> <span class="n">parameterTypes</span><span class="p">,</span> <span class="k">null</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">constructor</span> <span class="k">is</span> <span class="k">null</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">ArgumentException</span><span class="p">(</span><span class="s">"The args don't match any ctor"</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="p">(</span><span class="n">T</span><span class="p">)</span><span class="n">constructor</span><span class="p">.</span><span class="nf">Invoke</span><span class="p">(</span><span class="n">args</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>With this method, I can now write a unit test for my extension method.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="n">Fact</span><span class="p">]</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">CreatesCompletionResultFromCompletions</span><span class="p">()</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">choices</span> <span class="p">=</span> <span class="k">new</span><span class="p">[]</span>
<span class="p">{</span>
<span class="n">Instantiate</span><span class="p"><</span><span class="n">Choice</span><span class="p">>(</span>
<span class="s">"the resulting text"</span><span class="p">,</span>
<span class="p">(</span><span class="kt">int</span><span class="p">?)</span><span class="m">0.7</span><span class="p">,</span>
<span class="n">Instantiate</span><span class="p"><</span><span class="n">CompletionsLogProbability</span><span class="p">>(),</span>
<span class="s">"stop"</span><span class="p">)</span>
<span class="p">};</span>
<span class="kt">var</span> <span class="n">usage</span> <span class="p">=</span> <span class="n">Instantiate</span><span class="p"><</span><span class="n">CompletionsUsage</span><span class="p">>(</span><span class="m">200</span><span class="p">,</span> <span class="m">123</span><span class="p">,</span> <span class="m">323</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">completion</span> <span class="p">=</span> <span class="n">Instantiate</span><span class="p"><</span><span class="n">Completions</span><span class="p">>(</span>
<span class="s">"some-id"</span><span class="p">,</span>
<span class="p">(</span><span class="kt">int</span><span class="p">?)</span><span class="m">123245</span><span class="p">,</span>
<span class="s">"text-davinci-003"</span><span class="p">,</span>
<span class="n">choices</span><span class="p">,</span>
<span class="n">usage</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">result</span> <span class="p">=</span> <span class="n">completion</span><span class="p">.</span><span class="nf">ToCompletionResult</span><span class="p">();</span>
<span class="n">Assert</span><span class="p">.</span><span class="nf">Equal</span><span class="p">(</span><span class="s">"the resulting text"</span><span class="p">,</span> <span class="n">result</span><span class="p">.</span><span class="n">Completions</span><span class="p">[</span><span class="m">0</span><span class="p">].</span><span class="n">Text</span><span class="p">);</span>
<span class="n">Assert</span><span class="p">.</span><span class="nf">Equal</span><span class="p">(</span><span class="s">"text-davinci-003"</span><span class="p">,</span> <span class="n">result</span><span class="p">.</span><span class="n">Model</span><span class="p">);</span>
<span class="n">Assert</span><span class="p">.</span><span class="nf">Equal</span><span class="p">(</span><span class="s">"some-id"</span><span class="p">,</span> <span class="n">result</span><span class="p">.</span><span class="n">Id</span><span class="p">);</span>
<span class="n">Assert</span><span class="p">.</span><span class="nf">Equal</span><span class="p">(</span><span class="m">200</span><span class="p">,</span> <span class="n">result</span><span class="p">.</span><span class="n">Usage</span><span class="p">.</span><span class="n">CompletionTokens</span><span class="p">);</span>
<span class="n">Assert</span><span class="p">.</span><span class="nf">Equal</span><span class="p">(</span><span class="m">123</span><span class="p">,</span> <span class="n">result</span><span class="p">.</span><span class="n">Usage</span><span class="p">.</span><span class="n">PromptTokens</span><span class="p">);</span>
<span class="n">Assert</span><span class="p">.</span><span class="nf">Equal</span><span class="p">(</span><span class="m">323</span><span class="p">,</span> <span class="n">result</span><span class="p">.</span><span class="n">Usage</span><span class="p">.</span><span class="n">TotalTokens</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>If you’re wondering how I call the method without having to declare the type the method belongs to, recall that you can import methods with the <code class="language-plaintext highlighter-rouge">using static</code> declaration. So this method is part of my <code class="language-plaintext highlighter-rouge">ReflectionExtensions</code> class (so original, I know), so I have a <code class="language-plaintext highlighter-rouge">using static Serious.ReflectionExtensions;</code> at the top of my unit tests.</p>
<p>With this all in place, I can update my original method now:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">CompletionResult</span><span class="p">></span> <span class="nf">GetCompletionsAsync</span><span class="p">()</span> <span class="p">{</span>
<span class="kt">var</span> <span class="n">endpoint</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">Uri</span><span class="p">(</span><span class="s">"https://wouldn't-you-like-to-know.openai.azure.com/"</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">client</span> <span class="p">=</span> <span class="k">new</span> <span class="n">Azure</span><span class="p">.</span><span class="n">AI</span><span class="p">.</span><span class="n">OpenAI</span><span class="p">.</span><span class="nf">OpenAIClient</span><span class="p">(</span><span class="n">endpoint</span><span class="p">,</span> <span class="k">new</span> <span class="nf">DefaultAzureCredential</span><span class="p">());</span>
<span class="kt">var</span> <span class="n">response</span> <span class="p">=</span> <span class="k">await</span> <span class="n">client</span><span class="p">.</span><span class="nf">GetCompletionsAsync</span><span class="p">(</span><span class="s">"text-davinci-003"</span><span class="p">,</span> <span class="k">new</span> <span class="n">CompletionsOptions</span>
<span class="p">{</span>
<span class="n">Temperature</span> <span class="p">=</span> <span class="p">(</span><span class="kt">float</span><span class="p">)</span><span class="m">1.0</span><span class="p">,</span>
<span class="n">Prompts</span> <span class="p">=</span> <span class="p">{</span> <span class="s">"Some prompt"</span> <span class="p">},</span>
<span class="n">MaxTokens</span> <span class="p">=</span> <span class="m">2048</span><span class="p">,</span>
<span class="p">});</span>
<span class="k">return</span> <span class="n">response</span><span class="p">?.</span><span class="n">Value</span><span class="p">.</span><span class="nf">ToCompletionResult</span><span class="p">()</span>
<span class="p">??</span> <span class="k">throw</span> <span class="k">new</span> <span class="nf">Exception</span><span class="p">(</span><span class="s">"We'll handle this situation later"</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>So yeah, I can work around the internal constructor pretty easily, but in my mind it’s unnecessary friction. Also, I know a lot of folks are going to tell me I should wrap the entire API with my own data types. Sure, but that doesn’t change the fact that I’m going to want to test the translation from the API’s types to my own types. Not to mention, I wouldn’t have to do this if the data types returned by the API were simple constructable DTOs. For my needs, this is also unnecessary friction.</p>
<p>I hope this code helps you work around it the next time you run into this situation.</p>Phil HaackOne of my pet peeves is when I’m using a .NET client library that uses internal constructors for its return type. For example, let’s take a look at the Azure.AI.OpenAI nuget package. Now, I don’t mean to single out this package, as this is a common practice. It just happens to be the one I’m using at the moment. It’s an otherwise lovely package. I’m sure the authors are lovely people.When Your DbContext Has The Wrong Scope2023-01-09T00:00:00+00:002023-01-09T00:00:00+00:00https://haacked.com/archive/2023/01/09/scoping-db-context%20copy<p>This is the final installment of the adventures of Bill Maack the Hapless Developer (any similarity to me is purely coincidental and a result of pure random chance in an infinite universe). Follow along as Bill continues to improve the reliability of his ASP.NET Core and Entity Framework Core code. If you haven’t read the previous installments, you can find them here:</p>
<ol>
<li><a href="https://haacked.com/archive/2022/12/05/recover-from-dbupdate-exception/">How to Recover from a DbUpdateException With EF Core</a></li>
<li><a href="https://haacked.com/archive/2022/12/12/specific-db-exception/">Why Did That Database Throw That Exception?</a></li>
</ol>
<p>In the first post, we looked at a background <a href="https://www.hangfire.io/">Hangfire</a> job that processed incoming Slack event and it raised some questions such as:</p>
<blockquote>
<p>DbContext is not supposed to be thread safe. Why are allowing your repository method to be executed concurrently from multiple threads?</p>
</blockquote>
<p>This post addresses that question and more!</p>
<p><img src="https://user-images.githubusercontent.com/19977/211218857-0acd9f7d-e9a2-474a-9789-79785f9ca7f3.png" alt="Looking through a scope at an island in the middle of the ocean" title="Is that DbContext scoped properly?" /></p>
<p>Part of the confusion lies in the fact that the original example didn’t provide enough context. Let’s take a deeper look at the scenario.</p>
<p>Bill works on the team that builds <a href="https://abbot.app/">Abbot</a>, a Slack app that helps customer success/support teams keep track of conversations within Slack and support more customers with less effort. The app is built on ASP.NET Core and Entity Framework Core.</p>
<p>As a Slack App, it receives events from Slack in the form of HTTP POST requests. A simple ASP.NET MVC controller can handle that. Note that the following code is a paraphrase of the actual code as it leaves out some details such as verifying the Slack request signature. Bill would never skimp on security and definitely validates those Slack signatures.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">SlackController</span> <span class="p">:</span> <span class="n">Controller</span> <span class="p">{</span>
<span class="k">readonly</span> <span class="n">AbbotDbContext</span> <span class="n">_db</span><span class="p">;</span>
<span class="k">readonly</span> <span class="n">ISlackEventParser</span> <span class="n">_slackEventParser</span><span class="p">;</span>
<span class="k">readonly</span> <span class="n">IBackgroundJobClient</span> <span class="n">_backgroundJobClient</span><span class="p">;</span> <span class="c1">// Hangfire</span>
<span class="k">public</span> <span class="nf">SlackController</span><span class="p">(</span><span class="n">AbbotContext</span> <span class="n">db</span><span class="p">,</span> <span class="n">ISlackEventParser</span> <span class="n">slackEventParser</span><span class="p">,</span> <span class="n">IBackgroundJobClient</span> <span class="n">backgroundJobClient</span><span class="p">)</span> <span class="p">{</span>
<span class="n">_db</span> <span class="p">=</span> <span class="n">db</span><span class="p">;</span>
<span class="n">_slackEventParser</span> <span class="p">=</span> <span class="n">slackEventParser</span><span class="p">;</span>
<span class="n">_backgroundJobClient</span> <span class="p">=</span> <span class="n">backgroundJobClient</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">[</span><span class="n">HttpPost</span><span class="p">]</span>
<span class="k">public</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">IActionResult</span><span class="p">></span> <span class="nf">PostAsync</span><span class="p">()</span> <span class="p">{</span>
<span class="kt">var</span> <span class="n">slackEvent</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_slackEventParser</span><span class="p">.</span><span class="nf">ParseAsync</span><span class="p">(</span><span class="n">Request</span><span class="p">);</span>
<span class="n">_db</span><span class="p">.</span><span class="n">SlackEvents</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="n">slackEvent</span><span class="p">);</span>
<span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="nf">SaveChangesAsync</span><span class="p">();</span>
<span class="n">_backgroundJobClient</span><span class="p">.</span><span class="n">Enqueue</span><span class="p"><</span><span class="n">SlackEventProcessor</span><span class="p">>(</span><span class="n">x</span> <span class="p">=></span> <span class="n">x</span><span class="p">.</span><span class="nf">ProcessEventAsync</span><span class="p">(</span><span class="n">id</span><span class="p">));</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This code is pretty straightforward. Bill parses the incoming Slack event, saves it to the database, and then enqueues it for background processing using Hangfire.</p>
<p>When Hangfire is ready to process that event, it uses the ASP.NET Core dependency injection container to create an instance of <code class="language-plaintext highlighter-rouge">SlackEventProcessor</code> and calls the <code class="language-plaintext highlighter-rouge">ProcessEventAsync</code> method. What’s nice about this generic method approach is that <code class="language-plaintext highlighter-rouge">SlackEventProcessor</code> itself doesn’t even need to be registered in the container, only all of its dependencies need to be registered.</p>
<p>Here’s the <code class="language-plaintext highlighter-rouge">SlackEventProcessor</code> class that handles the background processing.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">SlackEventProcessor</span> <span class="p">{</span>
<span class="k">readonly</span> <span class="n">AbbotContext</span> <span class="n">_db</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">SlackEventProcessor</span><span class="p">(</span><span class="n">AbbotContext</span> <span class="n">db</span><span class="p">)</span> <span class="p">{</span>
<span class="n">_db</span> <span class="p">=</span> <span class="n">db</span><span class="p">;</span> <span class="c1">// AbbotContext derives from DbContext</span>
<span class="p">}</span>
<span class="c1">// This code runs in a background Hangfire job.</span>
<span class="k">public</span> <span class="k">async</span> <span class="n">Task</span> <span class="nf">ProcessEventAsync</span><span class="p">(</span><span class="kt">int</span> <span class="n">id</span><span class="p">)</span> <span class="p">{</span>
<span class="kt">var</span> <span class="n">nextEvent</span> <span class="p">=</span> <span class="p">(</span><span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="n">SlackEvents</span><span class="p">.</span><span class="nf">FindAsync</span><span class="p">(</span><span class="n">id</span><span class="p">))</span>
<span class="p">??</span> <span class="k">throw</span> <span class="k">new</span> <span class="nf">InvalidOperationException</span><span class="p">(</span><span class="s">$"Event not found: </span><span class="p">{</span><span class="n">id</span><span class="p">}</span><span class="s">"</span><span class="p">);</span>
<span class="k">try</span> <span class="p">{</span>
<span class="c1">// This does the actual processing of the Slack event.</span>
<span class="k">await</span> <span class="nf">RunPipelineAsync</span><span class="p">(</span><span class="n">nextEvent</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">catch</span> <span class="p">(</span><span class="n">Exception</span> <span class="n">e</span><span class="p">)</span> <span class="p">{</span>
<span class="n">nextEvent</span><span class="p">.</span><span class="n">Error</span> <span class="p">=</span> <span class="n">e</span><span class="p">.</span><span class="nf">ToString</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">finally</span> <span class="p">{</span>
<span class="n">nextEvent</span><span class="p">.</span><span class="n">Completed</span> <span class="p">=</span> <span class="n">DateTime</span><span class="p">.</span><span class="n">UtcNow</span><span class="p">;</span>
<span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="nf">SaveChangesAsync</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The key thing to note here is that in the case of Hangfire, every time Hangfire processes a job, it creates a unit of work (aka a scope) for that job. The end result is that as long as your <code class="language-plaintext highlighter-rouge">DbContext</code> derived instance (in this case <code class="language-plaintext highlighter-rouge">AbbotContext</code>) is registered with a lifetime of <code class="language-plaintext highlighter-rouge">ServiceLifetime.Scoped</code>, Hangfire will inject a new instance of your <code class="language-plaintext highlighter-rouge">DbContext</code> when invoking a job. So the code here doesn’t call any <code class="language-plaintext highlighter-rouge">DbContext</code> methods on multiple threads concurrently. We’re Ok here in that regard.</p>
<p>However, there <em>is</em> an issue with Bill’s code here. I glossed over it before, but the <code class="language-plaintext highlighter-rouge">RunPipelineAsync</code> method internally uses dependency injection to resolve a service to handle the Slack event processing. That service depends on <code class="language-plaintext highlighter-rouge">AbbotContext</code>. Since this is all running as part of a Hangfire job, it’s all in the same Lifetime scope. What that means is that the <code class="language-plaintext highlighter-rouge">AbbotContext</code> instance that is used to retrieve the <code class="language-plaintext highlighter-rouge">SlackEvent</code> instance is the same instance that is used to process the event. That’s not good.</p>
<p>The <code class="language-plaintext highlighter-rouge">AbbotContext</code> instance in <code class="language-plaintext highlighter-rouge">SlackEventProcessor</code> should only be responsible for retrieving and updating the <code class="language-plaintext highlighter-rouge">SlackEvent</code> instance that it needs to process. It should not be the same instance that is used when running the Slack event processing pipeline.</p>
<p>The solution is to create a separate <code class="language-plaintext highlighter-rouge">AbbotContext</code> instance for the outer scope. To do that, Bill needs to inject an <code class="language-plaintext highlighter-rouge">IDbContextFactory</code> into <code class="language-plaintext highlighter-rouge">SlackEventProcessor</code> and use that to create a new <code class="language-plaintext highlighter-rouge">AbbotContext</code> instance for the outer scope, resulting in:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">SlackEventProcessor</span> <span class="p">{</span>
<span class="k">readonly</span> <span class="n">IDbContextFactory</span><span class="p"><</span><span class="n">AbbotContext</span><span class="p">></span> <span class="n">_dbContextFactory</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">SlackEventProcessor</span><span class="p">(</span><span class="n">IDbContextFactory</span><span class="p"><</span><span class="n">AbbotContext</span><span class="p">></span> <span class="n">dbContextFactory</span><span class="p">)</span> <span class="p">{</span>
<span class="n">_dbContextFactory</span> <span class="p">=</span> <span class="n">dbContextFactory</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// This code runs in a background Hangfire job.</span>
<span class="k">public</span> <span class="k">async</span> <span class="n">Task</span> <span class="nf">ProcessEventAsync</span><span class="p">(</span><span class="kt">int</span> <span class="n">id</span><span class="p">)</span> <span class="p">{</span>
<span class="k">await</span> <span class="k">using</span> <span class="nn">var</span> <span class="n">db</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_dbContextFactory</span><span class="p">.</span><span class="nf">CreateDbContextAsync</span><span class="p">();</span>
<span class="kt">var</span> <span class="n">nextEvent</span> <span class="p">=</span> <span class="p">(</span><span class="k">await</span> <span class="n">db</span><span class="p">.</span><span class="n">SlackEvents</span><span class="p">.</span><span class="nf">FindAsync</span><span class="p">(</span><span class="n">id</span><span class="p">))</span>
<span class="p">??</span> <span class="k">throw</span> <span class="k">new</span> <span class="nf">InvalidOperationException</span><span class="p">(</span><span class="s">$"Event not found: </span><span class="p">{</span><span class="n">id</span><span class="p">}</span><span class="s">"</span><span class="p">);</span>
<span class="k">try</span> <span class="p">{</span>
<span class="c1">// This does the actual processing of the Slack event.</span>
<span class="c1">// The AbbotContext is injected into the pipeline and is not shared with `SlackEventProcessor`.</span>
<span class="k">await</span> <span class="nf">RunPipelineAsync</span><span class="p">(</span><span class="n">nextEvent</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">catch</span> <span class="p">(</span><span class="n">Exception</span> <span class="n">e</span><span class="p">)</span> <span class="p">{</span>
<span class="n">nextEvent</span><span class="p">.</span><span class="n">Error</span> <span class="p">=</span> <span class="n">e</span><span class="p">.</span><span class="nf">ToString</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">finally</span> <span class="p">{</span>
<span class="n">nextEvent</span><span class="p">.</span><span class="n">Completed</span> <span class="p">=</span> <span class="n">DateTime</span><span class="p">.</span><span class="n">UtcNow</span><span class="p">;</span>
<span class="k">await</span> <span class="n">db</span><span class="p">.</span><span class="nf">SaveChangesAsync</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The instance of <code class="language-plaintext highlighter-rouge">AbbotContext</code> created by the factory will always be a new instance. It won’t be the same instance injected into any dependencies that are resolved by the DI container.</p>
<p>This is a pretty straightforward fix, except the first time Bill tried it, it didn’t work.</p>
<h2 id="registering-the-dbcontextfactory-correctly">Registering the DbContextFactory Correctly</h2>
<p>Let’s take a step back and look at how Bill registered the <code class="language-plaintext highlighter-rouge">DbContext</code> instance with the DI container. Since Bill is working on an ASP.NET Core application, the recommended way to register the <code class="language-plaintext highlighter-rouge">DbContext</code> is to use the <code class="language-plaintext highlighter-rouge">AddDbContext</code> extension method on <code class="language-plaintext highlighter-rouge">IServiceCollection</code>.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">services</span><span class="p">.</span><span class="n">AddDbContext</span><span class="p"><</span><span class="n">AbbotContext</span><span class="p">>(</span><span class="n">options</span> <span class="p">=></span> <span class="p">{...});</span>
</code></pre></div></div>
<p>This sets the <code class="language-plaintext highlighter-rouge">ServiceLifetime</code> for the <code class="language-plaintext highlighter-rouge">DbContext</code> to <code class="language-plaintext highlighter-rouge">ServiceLifetime.Scoped</code>. This means that the <code class="language-plaintext highlighter-rouge">DbContext</code> instance is scoped to the current HTTP request. This is the default and recommended behavior for ASP.NET Core applications.</p>
<p>We wouldn’t want this to be a <code class="language-plaintext highlighter-rouge">ServiceLifetime.Singleton</code> as that would cause issues with concurrent calls to the <code class="language-plaintext highlighter-rouge">DbContext</code> which is a big no no.</p>
<p>You’ll never guess the name of the method to register a <code class="language-plaintext highlighter-rouge">DbContextFactory</code> with the DI container. Yep, it’s <code class="language-plaintext highlighter-rouge">AddDbContextFactory</code>.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">services</span><span class="p">.</span><span class="n">AddDbContextFactory</span><span class="p"><</span><span class="n">AbbotContext</span><span class="p">>(</span><span class="n">options</span> <span class="p">=></span> <span class="p">{...});</span>
</code></pre></div></div>
<p>Now here’s where it gets tricky. When Bill ran this code, he ran into an exception that looked something like:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Cannot consume scoped service 'Microsoft.EntityFrameworkCore.DbContextOptions1[AbbotContext]' from singleton 'Microsoft.EntityFrameworkCore.IDbContextFactory1[AbbotContext]'.
</code></pre></div></div>
<p>What’s happening here is that <code class="language-plaintext highlighter-rouge">AddDbContext</code> is not just registering our <code class="language-plaintext highlighter-rouge">DbContext</code> instance, it’s also registering the <code class="language-plaintext highlighter-rouge">DbContextOptions</code> instance used to create the <code class="language-plaintext highlighter-rouge">DbContext</code> instance. The lifetime of <code class="language-plaintext highlighter-rouge">DbContextOptions</code> is the same as <code class="language-plaintext highlighter-rouge">DbContext</code>, aka <code class="language-plaintext highlighter-rouge">ServiceLifetime.Scoped</code>.</p>
<p>However, <code class="language-plaintext highlighter-rouge">DbContextFactory</code> <em>also</em> needs to consume the <code class="language-plaintext highlighter-rouge">DbContextOptions</code> instance, but <code class="language-plaintext highlighter-rouge">DbContextFactory</code> has a lifetime of <code class="language-plaintext highlighter-rouge">ServiceLifetime.Singleton</code>. As a Singleton, it can’t consume a <code class="language-plaintext highlighter-rouge">Scoped</code> service because the <code class="language-plaintext highlighter-rouge">Scoped</code> service has a shorter lifetime than the <code class="language-plaintext highlighter-rouge">Singleton</code> service.</p>
<p>To summarize, <code class="language-plaintext highlighter-rouge">DbContext</code> is <code class="language-plaintext highlighter-rouge">Scoped</code> while <code class="language-plaintext highlighter-rouge">DbContextFactory</code> is <code class="language-plaintext highlighter-rouge">Singleton</code> and they both need a <code class="language-plaintext highlighter-rouge">DbContextOptions</code> which is <code class="language-plaintext highlighter-rouge">Scoped</code> by default.</p>
<p>Fortunately, there’s a simple solution. Well, it’s simple when you know it, otherwise it’s the kind of thing that makes a Bill want to pull his hair out. The solution is to make <code class="language-plaintext highlighter-rouge">DbContextOptions</code> a <code class="language-plaintext highlighter-rouge">Singleton</code> as well. Then both <code class="language-plaintext highlighter-rouge">DbContext</code> and <code class="language-plaintext highlighter-rouge">DbContextFactory</code> could both use it.</p>
<p>There’s an overload to <code class="language-plaintext highlighter-rouge">AddDbContext</code> that accepts a <code class="language-plaintext highlighter-rouge">ServiceLifetime</code> specifically for the <code class="language-plaintext highlighter-rouge">DbContextOptions</code> and you can set <em>that</em> to <code class="language-plaintext highlighter-rouge">Singleton</code>. So Bill’s final registration code looks like:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">services</span><span class="p">.</span><span class="n">AddDbContextFactory</span><span class="p"><</span><span class="n">AbbotContext</span><span class="p">>(</span><span class="n">options</span> <span class="p">=></span> <span class="p">{...});</span>
<span class="n">services</span><span class="p">.</span><span class="n">AddDbContext</span><span class="p"><</span><span class="n">AbbotContext</span><span class="p">>(</span><span class="n">options</span> <span class="p">=></span> <span class="p">{...},</span> <span class="n">optionsLifetime</span><span class="p">:</span> <span class="n">ServiceLifetime</span><span class="p">.</span><span class="n">Singleton</span><span class="p">);</span>
</code></pre></div></div>
<p>Bill used a named parameter to make it clear what the lifetime is for. So to summarize, <code class="language-plaintext highlighter-rouge">DbContext</code> still has a lifetime of <code class="language-plaintext highlighter-rouge">Scoped</code> while <code class="language-plaintext highlighter-rouge">DbContextFactory</code> and <code class="language-plaintext highlighter-rouge">DbContextOptions</code> have a <code class="language-plaintext highlighter-rouge">Singleton</code> lifetime. And EF Core is happy and Bill’s code works and is more robust. The End!</p>Phil HaackThis is the final installment of the adventures of Bill Maack the Hapless Developer (any similarity to me is purely coincidental and a result of pure random chance in an infinite universe). Follow along as Bill continues to improve the reliability of his ASP.NET Core and Entity Framework Core code. If you haven’t read the previous installments, you can find them here:Why Did That Database Throw That Exception?2022-12-12T00:00:00+00:002022-12-12T00:00:00+00:00https://haacked.com/archive/2022/12/12/specific-db-exception<p>In the <a href="https://haacked.com/archive/2022/12/05/recover-from-dbupdate-exception/">previous installment</a> of the adventures of the hapless developer, Bill Maack, Bill faced some code that tries to recover from a race condition when creating a <code class="language-plaintext highlighter-rouge">User</code> if the <code class="language-plaintext highlighter-rouge">User</code> entity doesn’t already exist.</p>
<p>As a reminder, these events are based on real events with real production code, but with names, locations, and code changed to protect the guilty. All code samples have been simplified for brevity.</p>
<p>At the end of the last post, Bill pondered the following question:</p>
<blockquote>
<p>There’s a problem with the exception handling. A unique constraint violation is not the only reason EF might throw a <code class="language-plaintext highlighter-rouge">DbUpdateException</code>. And what if it’s a violation for another table?</p>
</blockquote>
<p><img src="https://user-images.githubusercontent.com/19977/206928216-e465caff-5f86-4449-bf91-0f1fbe4a2da6.png" alt="Robot at a computer that's on fire" title="I should have caught that exception." /></p>
<p>“What if” indeed! Bill decided to dig into that. Here’s the section of the relevant code:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">catch</span> <span class="p">(</span><span class="n">DbUpdateException</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Maybe the user already exists? If so, return that user.</span>
<span class="n">user</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="n">Users</span><span class="p">.</span><span class="nf">SingleOrDefaultAsync</span><span class="p">(</span><span class="n">u</span> <span class="p">=></span> <span class="n">u</span><span class="p">.</span><span class="n">SlackId</span> <span class="p">==</span> <span class="n">slackId</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">user</span> <span class="k">is</span> <span class="k">null</span><span class="p">)</span> <span class="p">{</span>
<span class="k">throw</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">_db</span><span class="p">.</span><span class="nf">Entry</span><span class="p">(</span><span class="n">user</span><span class="p">).</span><span class="n">State</span> <span class="p">=</span> <span class="n">EntityState</span><span class="p">.</span><span class="n">Detached</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>How can Bill be certain that this <code class="language-plaintext highlighter-rouge">DbUpdateException</code> really corresponds to a unique constraint violation and not some other random database exception. He could try and parse the exception message, but that’s fragile and error prone. A total Bill Maack thing to do, but Bill is trying to be better. Instead, let’s look at the underlying database provider error.</p>
<p>Abbot, the application Bill works on, uses PostgreSQL as the database. The code accesses the database via the <a href="https://www.npgsql.org/efcore/">Entity Framework Core provider for Npgsql</a>. Npgsql is an open source ADO.NET Data Provider for PostgreSQL. That’s a mouthful, isn’t it?</p>
<p>When running into a database error, <code class="language-plaintext highlighter-rouge">DbUpdateException</code> exposes the underlying provider specific exception via the <code class="language-plaintext highlighter-rouge">InnerException</code> property. In the case of Npgsql, this is a <code class="language-plaintext highlighter-rouge">PostgresException</code> which exposes the <code class="language-plaintext highlighter-rouge">TableName</code> and <code class="language-plaintext highlighter-rouge">ConstraintName</code> along with the underlying PostgreSQL error code. The <a href="https://www.postgresql.org/docs/current/static/errcodes-appendix.html">error codes are documented here</a>.</p>
<p>Bill could define a custom exception type for unique constraint violations, and there’s nothing wrong with that if you’re into that sort of thing. Bill decided to go another way. He also didn’t want the calling code to have to know what the underlying database provider in cases he uses this code on other projects.</p>
<p>First, he defined a base <code class="language-plaintext highlighter-rouge">DatabaseError</code> record.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">/// <summary></span>
<span class="c1">/// Provides additional Database specific information about </span>
<span class="c1">/// a <see cref="DbUpdateException"/> thrown by EF Core.</span>
<span class="c1">/// </summary></span>
<span class="c1">/// <param name="TableName">The table involved, if any.</param></span>
<span class="c1">/// <param name="ConstraintName">The constraint involved, if any.</param></span>
<span class="c1">/// <param name="Exception">The unwrapped database provider specific exception.</param></span>
<span class="k">public</span> <span class="n">record</span> <span class="nf">DatabaseError</span><span class="p">(</span><span class="kt">string</span><span class="p">?</span> <span class="n">TableName</span><span class="p">,</span> <span class="kt">string</span><span class="p">?</span> <span class="n">ConstraintName</span><span class="p">,</span> <span class="n">Exception</span> <span class="n">Exception</span><span class="p">);</span>
</code></pre></div></div>
<p>And then defined a specific one for unique constraints.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">/// <summary></span>
<span class="c1">/// Provides additional Postgres specific information about a </span>
<span class="c1">/// <see cref="DbUpdateException"/> thrown by EF Core.This describes </span>
<span class="c1">/// the case where the exception is a unique constraint violation.</span>
<span class="c1">/// </summary></span>
<span class="c1">/// <param name="ColumnNames">The column names parsed from the constraint </span>
<span class="c1">/// name assuming the constraint follows the "IX_{Table}_{Column1}_..._{ColumnN}" naming convention.</param></span>
<span class="c1">/// <param name="TableName">The table involved, if any.</param></span>
<span class="c1">/// <param name="ConstraintName">The constraint involved, if any.</param></span>
<span class="c1">/// <param name="Exception">The unwrapped database provider specific exception.</param></span>
<span class="k">public</span> <span class="n">record</span> <span class="nf">UniqueConstraintError</span><span class="p">(</span>
<span class="n">IReadOnlyList</span><span class="p"><</span><span class="kt">string</span><span class="p">></span> <span class="n">ColumnNames</span><span class="p">,</span>
<span class="kt">string</span><span class="p">?</span> <span class="n">TableName</span><span class="p">,</span>
<span class="kt">string</span><span class="p">?</span> <span class="n">ConstraintName</span><span class="p">,</span>
<span class="n">Exception</span> <span class="n">Exception</span><span class="p">)</span> <span class="p">:</span> <span class="nf">DatabaseError</span><span class="p">(</span><span class="n">TableName</span><span class="p">,</span> <span class="n">ConstraintName</span><span class="p">,</span> <span class="n">Exception</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">/// <summary></span>
<span class="c1">/// Creates a <see cref="UniqueConstraintError"/> from a <see cref="PostgresException"/>.</span>
<span class="c1">/// </summary></span>
<span class="c1">/// <param name="postgresException">The <see cref="PostgresException"/>.</param></span>
<span class="c1">/// <returns>A <see cref="UniqueConstraintError"/> with extra information about the unique constraint violation.</returns></span>
<span class="k">public</span> <span class="k">static</span> <span class="n">UniqueConstraintError</span> <span class="nf">FromPostgresException</span><span class="p">(</span><span class="n">PostgresException</span> <span class="n">postgresException</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">constraintName</span> <span class="p">=</span> <span class="n">postgresException</span><span class="p">.</span><span class="n">ConstraintName</span><span class="p">;</span>
<span class="kt">var</span> <span class="n">tableName</span> <span class="p">=</span> <span class="n">postgresException</span><span class="p">.</span><span class="n">TableName</span><span class="p">;</span>
<span class="kt">var</span> <span class="n">constrainPrefix</span> <span class="p">=</span> <span class="n">tableName</span> <span class="k">is</span> <span class="n">not</span> <span class="k">null</span>
<span class="p">?</span> <span class="s">$"IX_</span><span class="p">{</span><span class="n">tableName</span><span class="p">}</span><span class="s">_"</span>
<span class="p">:</span> <span class="k">null</span><span class="p">;</span>
<span class="kt">var</span> <span class="n">columnNames</span> <span class="p">=</span> <span class="n">constrainPrefix</span> <span class="k">is</span> <span class="n">not</span> <span class="k">null</span>
<span class="p">&&</span> <span class="n">constraintName</span> <span class="k">is</span> <span class="n">not</span> <span class="k">null</span>
<span class="p">&&</span> <span class="n">constraintName</span><span class="p">.</span><span class="nf">StartsWith</span><span class="p">(</span><span class="n">constrainPrefix</span><span class="p">,</span> <span class="n">StringComparison</span><span class="p">.</span><span class="n">Ordinal</span><span class="p">)</span>
<span class="p">?</span> <span class="n">constraintName</span><span class="p">[</span><span class="n">constrainPrefix</span><span class="p">.</span><span class="n">Length</span><span class="p">..].</span><span class="nf">Split</span><span class="p">(</span><span class="sc">'_'</span><span class="p">)</span>
<span class="p">:</span> <span class="n">Array</span><span class="p">.</span><span class="n">Empty</span><span class="p"><</span><span class="kt">string</span><span class="p">>();</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">UniqueConstraintError</span><span class="p">(</span><span class="n">columnNames</span><span class="p">,</span> <span class="n">tableName</span><span class="p">,</span> <span class="n">constraintName</span><span class="p">,</span> <span class="n">postgresException</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>And finally, to connect it all together, add an extension method to map PostgreSQL error codes to these new error record types.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">/// <summary></span>
<span class="c1">/// Extensions to <see cref="DbUpdateException"/> used to retrieve more </span>
<span class="c1">/// database specific information about the thrown exception.</span>
<span class="c1">/// </summary></span>
<span class="k">public</span> <span class="k">static</span> <span class="k">class</span> <span class="nc">DbUpdateExceptionExtensions</span>
<span class="p">{</span>
<span class="c1">/// <summary></span>
<span class="c1">/// Retrieves a <see cref="DatabaseError"/> with database specific error </span>
<span class="c1">/// information from the <see cref="DbUpdateException"/> thrown by EF Core. </span>
<span class="c1">/// </summary></span>
<span class="c1">/// <param name="exception">The <see cref="DbUpdateException"/> thrown.</param></span>
<span class="c1">/// <returns>A <see cref="DatabaseError"/> or derived class if the inner </span>
<span class="c1">/// exception matches one of the supported types. Otherwise returns null.</returns></span>
<span class="k">public</span> <span class="k">static</span> <span class="n">DatabaseError</span><span class="p">?</span> <span class="nf">GetDatabaseError</span><span class="p">(</span><span class="k">this</span> <span class="n">DbUpdateException</span> <span class="n">exception</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">exception</span><span class="p">.</span><span class="n">InnerException</span> <span class="k">is</span> <span class="n">PostgresException</span> <span class="n">postgresException</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">postgresException</span><span class="p">.</span><span class="n">SqlState</span> <span class="k">switch</span>
<span class="p">{</span>
<span class="n">PostgresErrorCodes</span><span class="p">.</span><span class="n">UniqueViolation</span> <span class="p">=></span> <span class="n">UniqueConstraintError</span>
<span class="p">.</span><span class="nf">FromPostgresException</span><span class="p">(</span><span class="n">postgresException</span><span class="p">),</span>
<span class="c1">//... Other error codes mapped to other error types.</span>
<span class="n">_</span> <span class="p">=></span> <span class="k">new</span> <span class="nf">DatabaseError</span><span class="p">(</span>
<span class="n">postgresException</span><span class="p">.</span><span class="n">TableName</span><span class="p">,</span>
<span class="n">postgresException</span><span class="p">.</span><span class="n">ConstraintName</span><span class="p">,</span>
<span class="n">postgresException</span><span class="p">)</span>
<span class="p">};</span>
<span class="p">}</span>
<span class="k">return</span> <span class="k">null</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Putting it all together, Bill made the following changes to the original code. Hold your britches here, because he leans heavily on recent C# pattern matching features!</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">catch</span> <span class="p">(</span><span class="n">DbUpdateException</span> <span class="n">e</span><span class="p">)</span> <span class="nf">when</span> <span class="p">(</span><span class="n">e</span><span class="p">.</span><span class="nf">GetDatabaseError</span><span class="p">()</span>
<span class="k">is</span> <span class="n">UniqueConstraintError</span> <span class="p">{</span> <span class="n">TableName</span><span class="p">:</span> <span class="s">"Users"</span><span class="p">,</span> <span class="n">ColumnNames</span><span class="p">:</span> <span class="p">[</span><span class="k">nameof</span><span class="p">(</span><span class="n">SlackId</span><span class="p">)]</span> <span class="p">}</span> <span class="n">constraintError</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">existing</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="n">Users</span><span class="p">.</span><span class="nf">SingleOrDefaultAsync</span><span class="p">(</span><span class="n">u</span> <span class="p">=></span> <span class="n">u</span><span class="p">.</span><span class="n">SlackId</span> <span class="p">==</span> <span class="n">slackId</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">existing</span> <span class="k">is</span> <span class="k">null</span><span class="p">)</span> <span class="p">{</span>
<span class="k">throw</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">_db</span><span class="p">.</span><span class="nf">Entry</span><span class="p">(</span><span class="n">user</span><span class="p">).</span><span class="n">State</span> <span class="p">=</span> <span class="n">EntityState</span><span class="p">.</span><span class="n">Detached</span><span class="p">;</span>
<span class="n">user</span> <span class="p">=</span> <span class="n">existing</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Let me break down that <code class="language-plaintext highlighter-rouge">catch</code> expression as a refresher for those unfamiliar with some of the new pattern matching expressions.</p>
<p>When catching an exception, we can use a <code class="language-plaintext highlighter-rouge">when</code> expression to apply a filter to which exceptions we catch. In our case, we only want to catch exceptions where <code class="language-plaintext highlighter-rouge">e.GetDatabaseError() is UniqueConstraintError</code>. However, that’s not enough, we only want <code class="language-plaintext highlighter-rouge">UniqueConstraintError</code> where the <code class="language-plaintext highlighter-rouge">TableName</code> is <code class="language-plaintext highlighter-rouge">Users</code> and the <code class="language-plaintext highlighter-rouge">ColumnNames</code> is a list with a single element, “SlackId”. The <code class="language-plaintext highlighter-rouge">ColumnNames: [nameof(SlackId) ]</code> is an example of a list pattern. This is useful in cases where the unique constraint encompasses multiple columns. We could easily match on the set of columns like so:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">is</span> <span class="n">UniqueConstraintError</span> <span class="p">{</span> <span class="n">TableName</span><span class="p">:</span> <span class="s">"TableName"</span><span class="p">,</span> <span class="n">ColumnNames</span><span class="p">:</span> <span class="p">[</span><span class="s">"Column1"</span><span class="p">,</span> <span class="s">"Column2"</span><span class="p">,</span> <span class="p">...,</span> <span class="s">"ColumnN"</span><span class="p">]}</span>
</code></pre></div></div>
<p>Here’s what this code would be expanded out in old school C#.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">catch</span> <span class="p">(</span><span class="n">DbUpdateException</span> <span class="n">e</span><span class="p">)</span> <span class="p">{</span>
<span class="kt">var</span> <span class="n">uniqueConstraintError</span> <span class="p">=</span> <span class="n">e</span><span class="p">.</span><span class="nf">GetDatabaseError</span><span class="p">()</span> <span class="k">as</span> <span class="n">UniqueConstraintError</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">uniqueConstraintError</span> <span class="p">!=</span> <span class="k">null</span>
<span class="p">&&</span> <span class="n">uniqueConstraintError</span><span class="p">.</span><span class="n">TableName</span> <span class="p">==</span> <span class="s">"Users"</span>
<span class="p">&&</span> <span class="n">uniqueConstraintError</span><span class="p">.</span><span class="n">ColumnNames</span><span class="p">.</span><span class="n">Length</span> <span class="p">==</span> <span class="m">1</span>
<span class="p">&&</span> <span class="n">uniqueConstraintError</span><span class="p">.</span><span class="n">ColumnNames</span><span class="p">[</span><span class="m">0</span><span class="p">]</span> <span class="p">==</span> <span class="s">"SlackId"</span><span class="p">)</span> <span class="p">{</span>
<span class="n">user</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="n">Users</span><span class="p">.</span><span class="nf">SingleOrDefaultAsync</span><span class="p">(</span><span class="n">u</span> <span class="p">=></span> <span class="n">u</span><span class="p">.</span><span class="n">SlackId</span> <span class="p">==</span> <span class="n">slackId</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">user</span> <span class="p">!=</span> <span class="k">null</span><span class="p">)</span> <span class="p">{</span>
<span class="n">_db</span><span class="p">.</span><span class="nf">Entry</span><span class="p">(</span><span class="n">user</span><span class="p">).</span><span class="n">State</span> <span class="p">=</span> <span class="n">EntityState</span><span class="p">.</span><span class="n">Detached</span><span class="p">;</span>
<span class="k">return</span> <span class="n">user</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">throw</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The point being, Bill now has full confidence that the code is not trying to recover from an error it shouldn’t be. The only thing he doesn’t like about this code is how the column names are parsed from the unique constraint name. We haven’t yet found a better approach there.</p>
<p>To mitigate the risk, we review all our migrations so we know that this pattern is always in effect. This gives us reasonable confidence in this code.</p>
<p>The code is now way more robust than it was before, but our hapless protagonist is not done yet. At the end of the first post, we asked another question.</p>
<blockquote>
<p>Also, isn’t it a bit fragile that our top-level processing code could throw because the <code class="language-plaintext highlighter-rouge">DbContext</code> is in a weird state? Yes. Yes it is fragile.</p>
</blockquote>
<p>In the next installment of the adventures of Bill Maack the hapless Developer, we’ll cover how to make this code more robust.</p>Phil HaackIn the previous installment of the adventures of the hapless developer, Bill Maack, Bill faced some code that tries to recover from a race condition when creating a User if the User entity doesn’t already exist.How to Recover from a DbUpdateException With EF Core2022-12-05T00:00:00+00:002022-12-05T00:00:00+00:00https://haacked.com/archive/2022/12/05/recover-from-dbupdate-exception<p>There are cases where recovery from an Entity Framework Core (EF Core) <code class="language-plaintext highlighter-rouge">DbUpdateException</code> is possible if you play your cards right. Play them wrong and the result is heartbreak and tears as every call to <code class="language-plaintext highlighter-rouge">SaveChangesAsync</code> rethrows the same exception.</p>
<p>The following story examines one example of heartache and tears. This story is based on actual events. Only the names, locations, and code have been changed to protect the guilty. All code samples have been simplified for brevity.</p>
<p><img src="https://user-images.githubusercontent.com/19977/205463714-68148077-0539-45c9-955d-5c687058cfa8.png" alt="Robot recovering in an ice bath" title="Even robots need to recover" /></p>
<p>My company, <a href="https://www.aseriousbusiness.com/">A Serious Business, Inc.</a> (the real name), has a product called <a href="https://ab.bot/">Abbot</a> (name is also real), a Slack app that helps customer success/support teams keep track of conversations within Slack and support more customers with less effort.</p>
<p>As a Slack App, it needs to ingest a lot of Slack events. At a high level, we have a <code class="language-plaintext highlighter-rouge">Controller</code> that receives Slack events over HTTP, saves a <code class="language-plaintext highlighter-rouge">SlackEvent</code> entity to the database, and then sends a message to a background service (Hangfire) to process the event.</p>
<p>A while ago, one of our developers, we’ll call him Bill Maack (any similarity to persons real or imagined is entirely “coincidental”), wrote code to run our bot pipeline on incoming Slack events. The code looks something like this (<code class="language-plaintext highlighter-rouge">_db</code> is an instance of <code class="language-plaintext highlighter-rouge">AbbotContext</code>, our <code class="language-plaintext highlighter-rouge">DbContext</code> derived class).</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// This code runs in a background Hangfire job.</span>
<span class="k">public</span> <span class="k">async</span> <span class="n">Task</span> <span class="nf">ProcessEventAsync</span><span class="p">(</span><span class="kt">int</span> <span class="n">id</span><span class="p">)</span> <span class="p">{</span>
<span class="kt">var</span> <span class="n">nextEvent</span> <span class="p">=</span> <span class="p">(</span><span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="n">SlackEvents</span><span class="p">.</span><span class="nf">FindAsync</span><span class="p">(</span><span class="n">id</span><span class="p">))</span>
<span class="p">??</span> <span class="k">throw</span> <span class="k">new</span> <span class="nf">InvalidOperationException</span><span class="p">(</span><span class="s">$"Event not found: </span><span class="p">{</span><span class="n">id</span><span class="p">}</span><span class="s">"</span><span class="p">);</span>
<span class="k">try</span> <span class="p">{</span>
<span class="k">await</span> <span class="nf">RunPipelineAsync</span><span class="p">(</span><span class="n">nextEvent</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">catch</span> <span class="p">(</span><span class="n">Exception</span> <span class="n">e</span><span class="p">)</span> <span class="p">{</span>
<span class="n">nextEvent</span><span class="p">.</span><span class="n">Error</span> <span class="p">=</span> <span class="n">e</span><span class="p">.</span><span class="nf">ToString</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">finally</span> <span class="p">{</span>
<span class="n">nextEvent</span><span class="p">.</span><span class="n">Completed</span> <span class="p">=</span> <span class="n">DateTime</span><span class="p">.</span><span class="n">UtcNow</span><span class="p">;</span>
<span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="nf">SaveChangesAsync</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This code worked well enough for a long time, but we noticed that our logs would have the occasional uncaught <code class="language-plaintext highlighter-rouge">DbUpdateException</code>. More odd was it was thrown in the <code class="language-plaintext highlighter-rouge">finally</code> block at the line where <code class="language-plaintext highlighter-rouge">_db.SaveChangesAsync()</code> is called. The exception message indicated a unique constraint violation for a user record. This was confusing because we’re trying to save a <code class="language-plaintext highlighter-rouge">SlackEvent</code> which is totally unrelated to saving a user. So Phil…I mean Bill, <del>ignored it for a while in a confused stupor</del>immediately jumped into action. Our Application Insights logs indicated the exception was a result of the following SQL query:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">UPDATE</span> <span class="nv">"SlackEvents"</span> <span class="k">SET</span> <span class="nv">"Completed"</span> <span class="o">=</span> <span class="o">@</span><span class="n">p0</span>
<span class="k">WHERE</span> <span class="nv">"Id"</span> <span class="o">=</span> <span class="o">@</span><span class="n">p1</span><span class="p">;</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="nv">"Users"</span> <span class="p">(</span><span class="nv">"SlackId"</span><span class="p">)</span>
<span class="k">VALUES</span> <span class="p">(</span><span class="o">@</span><span class="n">p2</span><span class="p">)</span>
<span class="n">RETURNING</span> <span class="nv">"Id"</span><span class="p">;</span>
</code></pre></div></div>
<p>The plot thickens! Why are there two queries? Well we need a little more background about our app to understand that.</p>
<p>Here’s our <code class="language-plaintext highlighter-rouge">User</code> entity, stripped down to its skivvies for brevity.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">User</span> <span class="p">{</span>
<span class="k">public</span> <span class="kt">int</span> <span class="n">Id</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="c1">// Slack User Id looks something like U012BILJMAK</span>
<span class="k">public</span> <span class="kt">string</span> <span class="n">SlackId</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span> <span class="p">=</span> <span class="k">null</span><span class="p">!;</span>
<span class="c1">//...</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">OnModelCreating</code> method of <code class="language-plaintext highlighter-rouge">AbbotContext</code> configures a unique constraint on the <code class="language-plaintext highlighter-rouge">SlackId</code> column of <code class="language-plaintext highlighter-rouge">User</code>:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">protected</span> <span class="k">override</span> <span class="k">void</span> <span class="nf">OnModelCreating</span><span class="p">(</span><span class="n">ModelBuilder</span> <span class="n">modelBuilder</span><span class="p">)</span> <span class="p">{</span>
<span class="k">base</span><span class="p">.</span><span class="nf">OnModelCreating</span><span class="p">(</span><span class="n">modelBuilder</span><span class="p">);</span>
<span class="n">modelBuilder</span><span class="p">.</span><span class="n">Entity</span><span class="p"><</span><span class="n">User</span><span class="p">>()</span>
<span class="p">.</span><span class="nf">HasIndex</span><span class="p">(</span><span class="n">i</span> <span class="p">=></span> <span class="n">i</span><span class="p">.</span><span class="n">SlackId</span><span class="p">)</span>
<span class="p">.</span><span class="nf">IsUnique</span><span class="p">();</span>
<span class="c1">// ...</span>
<span class="p">}</span>
</code></pre></div></div>
<p>At some point in our pipeline, we try to retrieve a user by their slack id, and if we don’t find one, we create a new one. The code looks something like this:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">User</span><span class="p">></span> <span class="nf">GetUserBySlackIdAsync</span><span class="p">(</span><span class="kt">string</span> <span class="n">slackId</span><span class="p">)</span> <span class="p">{</span>
<span class="kt">var</span> <span class="n">user</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="n">Users</span><span class="p">.</span><span class="nf">SingleOrDefaultAsync</span><span class="p">(</span><span class="n">u</span> <span class="p">=></span> <span class="n">u</span><span class="p">.</span><span class="n">SlackId</span> <span class="p">==</span> <span class="n">slackId</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">user</span> <span class="k">is</span> <span class="k">null</span><span class="p">)</span> <span class="p">{</span>
<span class="n">user</span> <span class="p">=</span> <span class="k">new</span> <span class="n">User</span> <span class="p">{</span>
<span class="n">SlackId</span> <span class="p">=</span> <span class="n">slackId</span><span class="p">,</span>
<span class="p">};</span>
<span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="n">Users</span><span class="p">.</span><span class="nf">AddAsync</span><span class="p">(</span><span class="n">user</span><span class="p">);</span>
<span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="nf">SaveChangesAsync</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">user</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>If you squint, you can see a race condition lying in wait to prey on the hapless developer in this code. After the code checks if the <code class="language-plaintext highlighter-rouge">User</code> exists, another thread could have created the <code class="language-plaintext highlighter-rouge">User</code>. So when we try to save the <code class="language-plaintext highlighter-rouge">User</code>, we get a <code class="language-plaintext highlighter-rouge">DbUpdateException</code> because the unique constraint is violated.</p>
<p>Here’s Bill’s first naive attempt to fix this:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">User</span><span class="p">></span> <span class="nf">GetUserBySlackIdAsync</span><span class="p">(</span><span class="kt">string</span> <span class="n">slackId</span><span class="p">)</span> <span class="p">{</span>
<span class="kt">var</span> <span class="n">user</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="n">Users</span><span class="p">.</span><span class="nf">SingleOrDefaultAsync</span><span class="p">(</span><span class="n">u</span> <span class="p">=></span> <span class="n">u</span><span class="p">.</span><span class="n">SlackId</span> <span class="p">==</span> <span class="n">slackId</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">user</span> <span class="k">is</span> <span class="k">null</span><span class="p">)</span> <span class="p">{</span>
<span class="n">user</span> <span class="p">=</span> <span class="k">new</span> <span class="n">User</span> <span class="p">{</span>
<span class="n">SlackId</span> <span class="p">=</span> <span class="n">slackId</span><span class="p">,</span>
<span class="p">};</span>
<span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="n">Users</span><span class="p">.</span><span class="nf">AddAsync</span><span class="p">(</span><span class="n">user</span><span class="p">);</span>
<span class="k">try</span> <span class="p">{</span>
<span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="nf">SaveChangesAsync</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">catch</span> <span class="p">(</span><span class="n">DbUpdateException</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Maybe the user already exists? If so, return that user.</span>
<span class="n">user</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="n">Users</span><span class="p">.</span><span class="nf">SingleOrDefaultAsync</span><span class="p">(</span><span class="n">u</span> <span class="p">=></span> <span class="n">u</span><span class="p">.</span><span class="n">SlackId</span> <span class="p">==</span> <span class="n">slackId</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">user</span> <span class="k">is</span> <span class="k">null</span><span class="p">)</span> <span class="p">{</span>
<span class="k">throw</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">user</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>After testing it a bit, Bill called it a day. A job well done. How little did Silly Billy know. As it turns out, this was the source of the weird <code class="language-plaintext highlighter-rouge">DbUpdateException</code>s at the top of our callstack. Do you see the issue?</p>
<p>The problem is that when the <code class="language-plaintext highlighter-rouge">DbUpdateException</code> is thrown, the attempt to insert the new <code class="language-plaintext highlighter-rouge">User</code> record is still in the EF change tracker. So any subsequent calls to <code class="language-plaintext highlighter-rouge">SaveChangesAsync</code> will continue to try and save the <code class="language-plaintext highlighter-rouge">User</code> instance and throw the same exception.</p>
<p>The solution is to detach the <code class="language-plaintext highlighter-rouge">User</code> instance from the change tracker after the exception is thrown. We can do this in a generic way by detaching all the relevant entries reported by the exception like so:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">catch</span> <span class="p">(</span><span class="n">DbUpdateException</span> <span class="n">e</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="k">foreach</span> <span class="p">(</span><span class="kt">var</span> <span class="n">entry</span> <span class="k">in</span> <span class="n">e</span><span class="p">.</span><span class="n">Entries</span><span class="p">)</span> <span class="p">{</span>
<span class="n">entry</span><span class="p">.</span><span class="n">State</span> <span class="p">=</span> <span class="n">EntityState</span><span class="p">.</span><span class="n">Detached</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>However, in a rare flash of wisdom, Bill opted to be very specific in this case. It’s unlikely but possible that some other entity caused this exception. That would violate our expectations so we’d want this to throw in that case. Since we have a specific entity we’re trying to create, we should only detach that element.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">_db</span><span class="p">.</span><span class="nf">Entry</span><span class="p">(</span><span class="n">user</span><span class="p">).</span><span class="n">State</span> <span class="p">=</span> <span class="n">EntityState</span><span class="p">.</span><span class="n">Detached</span><span class="p">;</span>
</code></pre></div></div>
<p>Here’s that code in context.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">User</span><span class="p">></span> <span class="nf">GetUserBySlackIdAsync</span><span class="p">(</span><span class="kt">string</span> <span class="n">slackId</span><span class="p">)</span> <span class="p">{</span>
<span class="kt">var</span> <span class="n">user</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="n">Users</span><span class="p">.</span><span class="nf">SingleOrDefaultAsync</span><span class="p">(</span><span class="n">u</span> <span class="p">=></span> <span class="n">u</span><span class="p">.</span><span class="n">SlackId</span> <span class="p">==</span> <span class="n">slackId</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">user</span> <span class="k">is</span> <span class="k">null</span><span class="p">)</span> <span class="p">{</span>
<span class="n">user</span> <span class="p">=</span> <span class="k">new</span> <span class="n">User</span> <span class="p">{</span>
<span class="n">SlackId</span> <span class="p">=</span> <span class="n">slackId</span><span class="p">,</span>
<span class="p">};</span>
<span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="n">Users</span><span class="p">.</span><span class="nf">AddAsync</span><span class="p">(</span><span class="n">user</span><span class="p">);</span>
<span class="k">try</span> <span class="p">{</span>
<span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="nf">SaveChangesAsync</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">catch</span> <span class="p">(</span><span class="n">DbUpdateException</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Maybe the user already exists? If so, return that user.</span>
<span class="kt">var</span> <span class="n">existing</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_db</span><span class="p">.</span><span class="n">Users</span><span class="p">.</span><span class="nf">SingleOrDefaultAsync</span><span class="p">(</span><span class="n">u</span> <span class="p">=></span> <span class="n">u</span><span class="p">.</span><span class="n">SlackId</span> <span class="p">==</span> <span class="n">slackId</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">existing</span> <span class="k">is</span> <span class="k">null</span><span class="p">)</span> <span class="p">{</span>
<span class="k">throw</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">_db</span><span class="p">.</span><span class="nf">Entry</span><span class="p">(</span><span class="n">user</span><span class="p">).</span><span class="n">State</span> <span class="p">=</span> <span class="n">EntityState</span><span class="p">.</span><span class="n">Detached</span><span class="p">;</span>
<span class="n">user</span> <span class="p">=</span> <span class="n">existing</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">user</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>And that solved the immediate problem. But we’re not done yet. There’s a problem with the exception handling. A unique constraint violation is not the only reason EF might throw a <code class="language-plaintext highlighter-rouge">DbUpdateException</code>. And what if it’s a violation for another table? Also, isn’t it a bit fragile that our top-level processing code could throw because the <code class="language-plaintext highlighter-rouge">DbContext</code> is in a weird state? Yes. Yes it is fragile.</p>
<p>Stay tuned for the further adventures of our intrepid developer, Bill Maack, as he continues to iterate on this code to make it more robust and tries to be less of a pain in the ass to his team.</p>Phil HaackThere are cases where recovery from an Entity Framework Core (EF Core) DbUpdateException is possible if you play your cards right. Play them wrong and the result is heartbreak and tears as every call to SaveChangesAsync rethrows the same exception.C# List Pattern Examples2022-11-22T00:00:00+00:002022-11-22T00:00:00+00:00https://haacked.com/archive/2022/11/22/csharp-list-pattern<p>We recently upgraded <a href="https://ab.bot/">Abbot</a> to .NET 7 and C# 11 and I’m just loving the new language features in C#. In this post, I’ll give a couple examples of list patterns.</p>
<p><img src="https://user-images.githubusercontent.com/19977/203370937-0793aeb9-4d2a-444e-bc44-dd81124105f7.png" alt="A shopping list in front of a treasure hoard" title="Ancient scroll with my shopping list" /></p>
<h2 id="single-item-list">Single Item List</h2>
<p>There are cases where I expect up to one item in a list. Any more and I want to throw an exception. Here’s one way you can deal with it:</p>
<p><strong>BEFORE</strong></p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">List</span><span class="p"><</span><span class="kt">int</span><span class="p">></span> <span class="n">list</span> <span class="p">=</span> <span class="nf">SomeMethodThatReturnsAList</span><span class="p">(</span><span class="n">someId</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">formattedItem</span> <span class="p">=</span> <span class="n">list</span><span class="p">.</span><span class="nf">SingleOrDefault</span><span class="p">()</span> <span class="k">is</span> <span class="p">{}</span> <span class="n">singleItem</span>
<span class="p">?</span> <span class="s">$"Formatted: </span><span class="p">{</span><span class="n">singleItem</span><span class="p">}</span><span class="s">"</span>
<span class="p">:</span> <span class="s">"No items found"</span><span class="p">;</span>
</code></pre></div></div>
<p>Note that if there’s no items, nothing happens. That’s fine in this case. If there’s two or more items, it throws an exception, but not exactly the most helpful one. Here’s how I might handle this with a list pattern.</p>
<p><strong>AFTER</strong></p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">List</span><span class="p"><</span><span class="kt">int</span><span class="p">></span> <span class="n">list</span> <span class="p">=</span> <span class="nf">SomeMethodThatReturnsAList</span><span class="p">(</span><span class="n">someId</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">formattedItem</span> <span class="p">=</span> <span class="n">list</span> <span class="k">switch</span> <span class="p">{</span>
<span class="p">[]</span> <span class="p">=></span> <span class="s">"No items found"</span><span class="p">,</span>
<span class="p">[</span><span class="kt">var</span> <span class="n">singleItem</span><span class="p">]</span> <span class="p">=></span> <span class="s">$"Formatted: </span><span class="p">{</span><span class="n">singleItem</span><span class="p">}</span><span class="s">"</span><span class="p">,</span>
<span class="n">_</span> <span class="p">=></span> <span class="k">throw</span> <span class="k">new</span> <span class="nf">InvalidOperationException</span><span class="p">(</span><span class="s">$"Expected 0 or 1 items, but got </span><span class="p">{</span><span class="n">list</span><span class="p">.</span><span class="n">Count</span><span class="p">}</span><span class="s"> items for Id: </span><span class="p">{</span><span class="n">someId</span><span class="p">}</span><span class="s">."</span><span class="p">)</span>
<span class="p">};</span>
</code></pre></div></div>
<p>This is a bit more verbose, but it’s very clear that I’m handling every possible case I care about for the list, rather than relying on the behavior of <code class="language-plaintext highlighter-rouge">SingleOrDefault()</code>. Not only that, the exception I get is much more helpful.</p>
<h2 id="multiple-items">Multiple Items</h2>
<p>Here’s something that comes up a lot in my code. I have a formatted string I want to split into parts. Suppose it’s fine to have two or three parts, but no less and no more.</p>
<p><strong>BEFORE</strong></p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="n">record</span> <span class="nf">Parts</span><span class="p">(</span><span class="kt">string</span> <span class="n">Part1</span><span class="p">,</span> <span class="kt">string</span> <span class="n">Part2</span><span class="p">,</span> <span class="kt">string</span><span class="p">?</span> <span class="n">Part3</span> <span class="p">=</span> <span class="k">null</span><span class="p">)</span> <span class="p">{</span>
<span class="k">public</span> <span class="k">static</span> <span class="n">Parts</span> <span class="nf">Parse</span><span class="p">(</span><span class="kt">string</span> <span class="n">formatted</span><span class="p">)</span> <span class="p">{</span>
<span class="kt">var</span> <span class="n">parts</span> <span class="p">=</span> <span class="n">formatted</span><span class="p">.</span><span class="nf">Split</span><span class="p">(</span><span class="sc">'|'</span><span class="p">);</span>
<span class="k">return</span> <span class="n">parts</span><span class="p">.</span><span class="n">Length</span> <span class="k">switch</span> <span class="p">{</span>
<span class="m">2</span> <span class="p">=></span> <span class="k">new</span> <span class="nf">Parts</span><span class="p">(</span><span class="n">parts</span><span class="p">[</span><span class="m">0</span><span class="p">],</span> <span class="n">parts</span><span class="p">[</span><span class="m">1</span><span class="p">]),</span>
<span class="m">3</span> <span class="p">=></span> <span class="k">new</span> <span class="nf">Parts</span><span class="p">(</span><span class="n">parts</span><span class="p">[</span><span class="m">0</span><span class="p">],</span> <span class="n">parts</span><span class="p">[</span><span class="m">1</span><span class="p">],</span> <span class="n">parts</span><span class="p">[</span><span class="m">2</span><span class="p">]),</span>
<span class="kt">var</span> <span class="n">length</span> <span class="p">=></span> <span class="k">throw</span> <span class="k">new</span> <span class="nf">InvalidOperationException</span><span class="p">(</span><span class="s">$"Expected 3 parts, but got </span><span class="p">{</span><span class="n">length</span><span class="p">}</span><span class="s"> parts for formatted string: </span><span class="p">{</span><span class="n">formatted</span><span class="p">}</span><span class="s">."</span><span class="p">),</span>
<span class="p">};</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">override</span> <span class="kt">string</span> <span class="nf">ToString</span><span class="p">()</span> <span class="p">=></span> <span class="n">Part3</span> <span class="k">is</span> <span class="k">null</span>
<span class="p">?</span> <span class="s">$"</span><span class="p">{</span><span class="n">Part1</span><span class="p">}</span><span class="s">|</span><span class="p">{</span><span class="n">Part2</span><span class="p">}</span><span class="s">"</span>
<span class="p">:</span> <span class="s">$"</span><span class="p">{</span><span class="n">Part1</span><span class="p">}</span><span class="s">|</span><span class="p">{</span><span class="n">Part2</span><span class="p">}</span><span class="s">|</span><span class="p">{</span><span class="n">Part3</span><span class="p">}</span><span class="s">"</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p><strong>AFTER</strong></p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="n">record</span> <span class="nf">Parts</span><span class="p">(</span><span class="kt">string</span> <span class="n">Part1</span><span class="p">,</span> <span class="kt">string</span> <span class="n">Part2</span><span class="p">,</span> <span class="kt">string</span><span class="p">?</span> <span class="n">Part3</span> <span class="p">=</span> <span class="k">null</span><span class="p">)</span> <span class="p">{</span>
<span class="k">public</span> <span class="k">static</span> <span class="n">Parts</span> <span class="nf">Parse</span><span class="p">(</span><span class="kt">string</span> <span class="n">formatted</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">formatted</span><span class="p">.</span><span class="nf">Split</span><span class="p">(</span><span class="sc">'|'</span><span class="p">)</span> <span class="k">switch</span> <span class="p">{</span>
<span class="p">[</span><span class="kt">var</span> <span class="n">part1</span><span class="p">,</span> <span class="kt">var</span> <span class="n">part2</span><span class="p">]</span> <span class="p">=></span> <span class="k">new</span> <span class="nf">Parts</span><span class="p">(</span><span class="n">part1</span><span class="p">,</span> <span class="n">part2</span><span class="p">),</span>
<span class="p">[</span><span class="kt">var</span> <span class="n">part1</span><span class="p">,</span> <span class="kt">var</span> <span class="n">part2</span><span class="p">,</span> <span class="kt">var</span> <span class="n">part3</span><span class="p">]</span> <span class="p">=></span> <span class="k">new</span> <span class="nf">Parts</span><span class="p">(</span><span class="n">part1</span><span class="p">,</span> <span class="n">part2</span><span class="p">,</span> <span class="n">part3</span><span class="p">),</span>
<span class="kt">var</span> <span class="n">parts</span> <span class="p">=></span> <span class="k">throw</span> <span class="k">new</span> <span class="nf">InvalidOperationException</span><span class="p">(</span><span class="s">$"Expected 3 parts, but got </span><span class="p">{</span><span class="n">parts</span><span class="p">.</span><span class="n">Length</span><span class="p">}</span><span class="s"> parts for formatted string: </span><span class="p">{</span><span class="n">formatted</span><span class="p">}</span><span class="s">."</span><span class="p">),</span>
<span class="p">};</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">override</span> <span class="kt">string</span> <span class="nf">ToString</span><span class="p">()</span> <span class="p">=></span> <span class="n">Part3</span> <span class="k">is</span> <span class="k">null</span>
<span class="p">?</span> <span class="s">$"</span><span class="p">{</span><span class="n">Part1</span><span class="p">}</span><span class="s">|</span><span class="p">{</span><span class="n">Part2</span><span class="p">}</span><span class="s">"</span>
<span class="p">:</span> <span class="s">$"</span><span class="p">{</span><span class="n">Part1</span><span class="p">}</span><span class="s">|</span><span class="p">{</span><span class="n">Part2</span><span class="p">}</span><span class="s">|</span><span class="p">{</span><span class="n">Part3</span><span class="p">}</span><span class="s">"</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="conclusion">Conclusion</h2>
<p>Working with lists just got easier with C# 11. It’s taking all my restraint not to go through our entire codebase and refactor all the code that would be improved with list patterns. I’m sure I’ll get there eventually.</p>
<p>For more on list patterns, check <a href="https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/operators/patterns#list-patterns">out the docs</a>!</p>Phil HaackWe recently upgraded Abbot to .NET 7 and C# 11 and I’m just loving the new language features in C#. In this post, I’ll give a couple examples of list patterns.So you want to speak at conferences2022-11-07T00:00:00+00:002022-11-07T00:00:00+00:00https://haacked.com/archive/2022/11/07/so-you-want-to-speak-at-conferences<p>I just finished speaking at my favorite conference, the <a href="https://cdc.dev/">Caribbean Developer’s Conference</a>. Held in a wonderful resort in Punta Cana, Dominican Republic, it brings together a local and international crowd of speakers and attendees. I’ve <a href="https://haacked.com/archive/2018/11/02/caribbean-developer-conf/">gushed about it before</a>.</p>
<p>Wonderful locations are certainly a perk of some conferences, but what really gets me jazzed is the so-called “hallway track” where we get into interesting conversations with attendees and other speakers. Especially when I hear from developers just starting out where this is one of their first conferences. Inevitably, some will ask how did I get started speaking.</p>
<p>For the most part, I lucked into it. My experience may not be the best template for others to follow. For example, it really helped that I was hired by Microsoft to work on a wildly popular web platform that millions of developers use. So yeah, by virtue of my job, I had a built in audience. “Become Program Manager of ASP.NET MVC” isn’t a scalable approach to becoming a speaker. But there were a few other things that helped me get started.</p>
<h2 id="start-small">Start small</h2>
<p>When I got the job, I knew that I would be invited to speak at conferences here and there. I had attended a few, but never saw myself speaking at them. So I started by contacting the local .NET User Group in Los Angeles (where I was living at the time) and asked if I could speak at the next meeting. They were happy to have me as they are always looking for new speakers.</p>
<p>Speaking at local user groups or meetups is a great way to get started. One way to find such groups is through <a href="https://www.meetup.com/">Meetup.com</a>. For example, if you’re specifically interested in .NET, the .NET Foundation has a list of <a href="https://www.meetup.com/pro/dotnet/">.NET meetups here</a>.</p>
<p>Not only is speaking at a meetup good practice, but it puts you in touch with other like-minded developers. It’s a great way to start building your network. And you don’t have to travel far.</p>
<h2 id="have-something-to-say">Have Something to Say</h2>
<p>You may be thinking, why would anyone want to hear from me? And what would I talk about? If you’re writing code every day, building real software, there’s undoubtedly some interesting perspective you can share if you look hard enough. And don’t discount the fact that your inexperience can be an advantage in some ways.</p>
<p>You often see the same speakers crop up again and again at conferences. This is not a bad thing. These folks have a wealth of experience and wisdom to share. On the other hand, this can to the same type of talk being repeated over and over.</p>
<p>Experience is a double edged sword. It can further detach a person from the beginner’s mind. The beginner’s mind is a concept from Zen Buddhism that describes how you approach a concept without preconceptions through the eyes of a beginner. Most of the technologies and APIs we use are built by very experienced developers. We often need to be reminded that what’s intuitive to us old codgers may not be so obvious to a beginner. Especially a beginner who grew up with an entirely different cultural context.</p>
<p>For example, I dabbled a bit in coding before my first job out of college, but I didn’t really learn to code until my first job where I was thrust into the frying pan and learned on the job on customer projects. I don’t know what it’s like to go through a coding bootcamp, or how they teach coding. That would be an interesting topic.</p>
<p>Another source of inspiration is to look at the session list for existing conferences. Read the titles and abstracts. And then consider if you might have an interesting perspective on that same topic or a way to add more context. Don’t worry too much if you see others talking about the same topic you want to talk about. There’s always a new take to be had. I’m not the only one to ever give a talk on Social Coding, ASP.NET MVC, or Building a Startup, but I hope my talks at least offered one or two new perspectives to consider on these well trodden subjects.</p>
<h2 id="have-somewhere-to-say-it">Have Somewhere to Say It</h2>
<p>At some point, you’re ready to speak at a larger conference. How do you get started? If you’ve been speaking at local user groups, you hopefully have a network of people you can reach out to get a sense of what conferences are looking for speakers.</p>
<p>Another great resource is to fill out a speaker profile on <a href="https://sessionize.com/">Sessionize</a> and then use their <a href="https://sessionize.com/app/speaker/discover">Discover events</a>. I learned that several speakers discovered Caribbean Developers Conference through Sessionize.</p>
<p>Follow folks on Twitter who are well-known speakers in the type of conferences you’re interested in. They often tweet about upcoming conferences and call for speakers. Get to know the set of conferences and conference organizers in your areas of interest and pay attention to them.</p>
<h2 id="some-other-resources">Some other resources</h2>
<p>After I drafted this post, I googled the title and found this great post by <a href="https://katemats.com/blog/public-speaking-at-a-conference">Kate Matsudaira</a>. We have some ideas in common, but she makes a great point about preparing a strong pitch.</p>
<p>Another great resource is of course, Scott Hanselman who is a fantastic speaker. In particular, his video on <a href="https://www.hanselman.com/blog/video-the-art-of-speaking-with-scott-hanselman">The Art of Speaking</a> as well as his <a href="https://www.hanselman.com/blog/tips-for-preparing-for-a-technical-presentation">specific tips for technical presentations</a>.</p>
<p>I hope this is helpful and I see you on the speaker’s list at the next conference!</p>Phil HaackI just finished speaking at my favorite conference, the Caribbean Developer’s Conference. Held in a wonderful resort in Punta Cana, Dominican Republic, it brings together a local and international crowd of speakers and attendees. I’ve gushed about it before.Calculating MRR with Stripe and C#2022-10-12T00:00:00+00:002022-10-12T00:00:00+00:00https://haacked.com/archive/2022/10/12/calculating-mrr-with-stripe-and-csharp<p>Over here at <a href="https://www.aseriousbusiness.com/">A Serious Business, Inc.</a> we’re very <a href="https://ab.bot/blog/abbot-is-soc2-compliant">serious about security</a>. One principle that’s important to us is what we call the <em>Principle of Least Exposure</em> (not to be confused with the similar <a href="https://en.wikipedia.org/wiki/Principle_of_least_privilege">Principle of Least Privilege</a>).</p>
<p>In simple terms, the principle is:</p>
<blockquote>
<p>You can’t expose what you don’t have.</p>
</blockquote>
<p>For example, our product <a href="https://ab.bot/">Abbot</a> doesn’t store or accept credit card payment info directly. Instead, we use a trusted third party, <a href="https://stripe.com/">Stripe</a>, to handle that for us.</p>
<p><a href="https://www.flickr.com/photos/pictures-of-money/17123251389/"><img src="https://user-images.githubusercontent.com/19977/195433434-7d3bd771-e32a-4630-b12f-08980bf5abc2.jpg" alt="Rolls of $100 bills" title="Dolla dolla bills, y'all! - CC BY 2.0 by Pictures of Money" /></a></p>
<p>Stripe can be a bit tricky to get started with, but that complexity reflects the inherent complexity of accepting payments. Fortunately, they supply some great tools and <a href="https://stripe.com/docs">documentation</a> to help you get started. Also, their support is top quality. Every time I contact them, they’re quick to respond and helpful. And before you ask, no, they’re not paying me to say that. But in the interest of full disclosure, they did send me a Stripe t-shirt. I’m wearing it right now. It’s very comfortable.</p>
<h2 id="calculating-mrr">Calculating MRR</h2>
<p>One of the most important metrics for any SaaS business is Monthly Recurring Revenue (MRR). It’s a simple concept: how much money are you making every month? It’s a great indicator of the health of your business. If your MRR is growing, you’re doing well. If it’s shrinking, you’re not.</p>
<p>MRR is forward looking, which is why you don’t want to sum up invoices. Instead, you want to look at all the subscriptions you have and sum up the amount of money you’re expecting to receive from them in the next month. For yearly subscriptions, you’ll want to pro-rate the yearly amount to get the monthly amount.</p>
<p>Fortunately, Stripe has a nice dashboard where you can see your MRR. I’m not supposed to share ours, but we’re all friends here right?!</p>
<p><img src="https://user-images.githubusercontent.com/19977/195445796-618ccc09-e369-46a8-b680-f4f60f6deb9f.png" alt="Example of MRR on the Stripe Dashboard" title="We're RICH!...friend." /></p>
<p>Unfortunately, it can take some time to reflect recent purchases. In our case, we wanted to display the MRR in an internal dashboard and I could not find a way to grab the MRR that Stripe calculates from the Stripe API. If there is a way and I just missed it, I hope someone will let me know in the comments.</p>
<p>So naturally I wrote some code to do this. As with most .NET code, it starts with installing a NuGet package.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">></span> dotnet add package Stripe.net
</code></pre></div></div>
<p>Then I wrote some methods to calculate the MRR. If you want to jump to the <a href="https://gist.github.com/haacked/0a34391bfc2fddda192a082cfe5867af">full code listing, check out this gist</a>. The following walks through each step.</p>
<p>The first step is to calculate the MRR for a single subscription. There’s two important things to consider here. First, we need to normalize subscriptions to monthly. For example, for an annual subscription, you need to divide by 12 to get the monthly revenue amount. For days and weeks, I just picked 30 and 4 respectively. I’m not sure what the standard value should be here, but we don’t offer daily or weekly subscriptions so it’s not important to me. I just include it here for completeness.</p>
<p>The second important thing is to consider the item quantity. My first attempt at this code neglected the item quantity and I ended up underreporting our MRR.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">static</span> <span class="kt">decimal</span> <span class="nf">CalculateSubscriptionMonthlyRevenue</span><span class="p">(</span><span class="n">Subscription</span> <span class="n">subscription</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">decimal</span> <span class="n">revenue</span> <span class="p">=</span> <span class="m">0</span><span class="p">;</span>
<span class="k">foreach</span> <span class="p">(</span><span class="kt">var</span> <span class="n">item</span> <span class="k">in</span> <span class="n">subscription</span><span class="p">.</span><span class="n">Items</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">multiplier</span> <span class="p">=</span> <span class="n">item</span><span class="p">.</span><span class="n">Plan</span><span class="p">.</span><span class="n">Interval</span> <span class="k">switch</span>
<span class="p">{</span>
<span class="s">"day"</span> <span class="p">=></span> <span class="m">30</span><span class="n">M</span><span class="p">,</span>
<span class="s">"week"</span> <span class="p">=></span> <span class="m">4</span><span class="n">M</span><span class="p">,</span>
<span class="s">"month"</span> <span class="p">=></span> <span class="m">1</span><span class="n">M</span><span class="p">,</span>
<span class="s">"year"</span> <span class="p">=></span> <span class="m">1</span><span class="n">M</span> <span class="p">/</span> <span class="m">12</span><span class="n">M</span><span class="p">,</span>
<span class="n">_</span> <span class="p">=></span> <span class="k">throw</span> <span class="k">new</span> <span class="nf">UnreachableException</span><span class="p">(</span><span class="s">$"Unexpected plan interval: </span><span class="p">{</span><span class="n">item</span><span class="p">.</span><span class="n">Plan</span><span class="p">.</span><span class="n">Interval</span><span class="p">}</span><span class="s">."</span><span class="p">)</span>
<span class="p">};</span>
<span class="n">revenue</span> <span class="p">+=</span> <span class="n">multiplier</span> <span class="p">*</span> <span class="n">item</span><span class="p">.</span><span class="n">Quantity</span> <span class="p">*</span> <span class="n">item</span><span class="p">.</span><span class="n">Price</span><span class="p">.</span><span class="n">UnitAmountDecimal</span><span class="p">.</span><span class="nf">GetValueOrDefault</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">revenue</span> <span class="p">/</span> <span class="m">100</span><span class="n">M</span><span class="p">;</span> <span class="c1">// The UnitAmount is in cents.</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Next we calculate the MRR for a customer. In theory, a customer can have multiple subscriptions, so we handle that. More importantly, if your customer has a discount due to a coupon, you want to apply that discount to the calculation. However, for a one-time amount off discount, you can pretty much ignore it for the purposes of MRR as it’s forward looking.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">static</span> <span class="kt">decimal</span> <span class="nf">CalculateCustomerMonthlyRevenue</span><span class="p">(</span><span class="n">Customer</span> <span class="n">customer</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">subscriptions</span> <span class="p">=</span> <span class="n">customer</span><span class="p">.</span><span class="n">Subscriptions</span><span class="p">;</span>
<span class="kt">var</span> <span class="n">revenue</span> <span class="p">=</span> <span class="m">0</span><span class="n">M</span><span class="p">;</span>
<span class="k">foreach</span> <span class="p">(</span><span class="kt">var</span> <span class="n">subscription</span> <span class="k">in</span> <span class="n">subscriptions</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">revenue</span> <span class="p">+=</span> <span class="nf">CalculateSubscriptionMonthlyRevenue</span><span class="p">(</span><span class="n">subscription</span><span class="p">);</span>
<span class="p">}</span>
<span class="c1">// Apply the coupon, if any. We only look at % off coupons.</span>
<span class="c1">// We can ignore the amount off discount. That's a one time discount and doesn't affect ongoing MRR.</span>
<span class="k">if</span> <span class="p">(</span><span class="n">customer</span><span class="p">.</span><span class="n">Discount</span> <span class="k">is</span> <span class="p">{</span> <span class="n">Coupon</span><span class="p">.</span><span class="n">PercentOff</span><span class="p">:</span> <span class="p">{</span> <span class="p">}</span> <span class="n">percentOff</span> <span class="p">})</span>
<span class="p">{</span>
<span class="n">revenue</span> <span class="p">*=</span> <span class="m">1</span> <span class="p">-</span> <span class="n">percentOff</span> <span class="p">/</span> <span class="m">100</span><span class="n">M</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">revenue</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>And finally, we put it all together by grabbing all of the active subscriptions.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">static</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="kt">decimal</span><span class="p">></span> <span class="nf">CalculateMonthlyRecurringRevenue</span><span class="p">()</span>
<span class="p">{</span>
<span class="kt">string</span><span class="p">?</span> <span class="n">lastId</span> <span class="p">=</span> <span class="k">null</span><span class="p">;</span>
<span class="kt">var</span> <span class="n">customerClient</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">CustomerService</span><span class="p">();</span>
<span class="kt">decimal</span> <span class="n">revenue</span> <span class="p">=</span> <span class="m">0</span><span class="n">M</span><span class="p">;</span>
<span class="kt">bool</span> <span class="n">hasMore</span> <span class="p">=</span> <span class="k">true</span><span class="p">;</span>
<span class="k">while</span> <span class="p">(</span><span class="n">hasMore</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">customers</span> <span class="p">=</span> <span class="k">await</span> <span class="n">customerClient</span><span class="p">.</span><span class="nf">ListAsync</span><span class="p">(</span>
<span class="k">new</span> <span class="n">CustomerListOptions</span>
<span class="p">{</span>
<span class="n">Limit</span> <span class="p">=</span> <span class="m">100</span><span class="p">,</span> <span class="cm">/* Max Limit is 100 */</span>
<span class="n">Expand</span> <span class="p">=</span> <span class="k">new</span> <span class="n">List</span><span class="p"><</span><span class="kt">string</span><span class="p">></span> <span class="p">{</span> <span class="s">"data.subscriptions"</span> <span class="p">},</span>
<span class="n">StartingAfter</span> <span class="p">=</span> <span class="n">lastId</span>
<span class="p">});</span>
<span class="n">revenue</span> <span class="p">+=</span> <span class="n">customers</span><span class="p">.</span><span class="nf">Sum</span><span class="p">(</span><span class="n">CalculateCustomerMonthlyRevenue</span><span class="p">);</span>
<span class="n">hasMore</span> <span class="p">=</span> <span class="n">customers</span><span class="p">.</span><span class="n">HasMore</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">hasMore</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">lastId</span> <span class="p">=</span> <span class="n">customers</span><span class="p">.</span><span class="nf">LastOrDefault</span><span class="p">()?.</span><span class="n">Id</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">lastId</span> <span class="k">is</span> <span class="k">null</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">InvalidOperationException</span><span class="p">(</span><span class="s">"API reports more customers but no last id was returned."</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">revenue</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The code here is a bit tricky because the Stripe APIs only let you grab 100 customers at a time. And since we always want to prepare for success, we need to handle the case where there are more than 100 customers. So we loop through the customers, grabbing 100 at a time, until we get to the end.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I hope this helps you use Stripe to calculate your MRR. If you have improvements to the code, feel free to comment on the gist with the <a href="https://gist.github.com/haacked/0a34391bfc2fddda192a082cfe5867af">full code listing</a>.</p>Phil HaackOver here at A Serious Business, Inc. we’re very serious about security. One principle that’s important to us is what we call the Principle of Least Exposure (not to be confused with the similar Principle of Least Privilege).Pitfalls with eager loading of collections in EF Core2022-09-30T00:00:00+00:002022-09-30T00:00:00+00:00https://haacked.com/archive/2022/09/30/ef-core-collection-pitfalls<p>When using an ORM with a web app, lazy loading will almost certainly result in hidden <a href="https://medium.com/doctolib/understanding-and-fixing-n-1-query-30623109fe89">N+1 queries</a>. Eager loading is a great way to avoid this, but has its own pitfalls. In particular, for each query, you need to be careful about what you include in the query. If you include too much, you can end up with a lot of data that you don’t need. If you include too little, you can end up with confusing logic. For example, deep in your application code, it may not be clear if a navigation collection has been loaded yet or not. This can lead to unexpected behavior.</p>
<p><img src="https://user-images.githubusercontent.com/19977/193374485-45a55426-a73c-4971-b6f8-b67e81f91d0b.jpg" alt="Drawing of Atari 2600 Pitfall game - CC BY-NC 2.0 by Doctor Popular on Flickr" title="Pitfall - CC BY-NC 2.0 by Doctor Popular" /></p>
<p>I know it’s boring lazy to use the example of a blog post to illustrate this concept in a blog post, but it’s a well understood domain and it’s what’s often used in EF Core’s own documentation. So bear with me.</p>
<p>Here’s one approach that you might take (with some properties omitted for brevity):</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">Post</span> <span class="p">{</span>
<span class="k">public</span> <span class="kt">int</span> <span class="n">Id</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="k">public</span> <span class="n">IList</span><span class="p"><</span><span class="n">Author</span><span class="p">></span> <span class="n">Authors</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="k">public</span> <span class="n">IList</span><span class="p"><</span><span class="n">Comment</span><span class="p">></span> <span class="n">Comments</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>In our hypothetical web app, every page that displays a blog post will need to display the post’s authors, but only some of them will display comments.</p>
<p>One approach we could take is to always load both collections:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// For the individual post page.</span>
<span class="k">public</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">Post</span><span class="p">?></span> <span class="nf">GetPostAsync</span><span class="p">(</span><span class="kt">int</span> <span class="n">id</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">await</span> <span class="n">_context</span><span class="p">.</span><span class="n">Posts</span>
<span class="p">.</span><span class="nf">Include</span><span class="p">(</span><span class="n">p</span> <span class="p">=></span> <span class="n">p</span><span class="p">.</span><span class="n">Authors</span><span class="p">)</span>
<span class="p">.</span><span class="nf">Include</span><span class="p">(</span><span class="n">p</span> <span class="p">=></span> <span class="n">p</span><span class="p">.</span><span class="n">Comments</span><span class="p">)</span>
<span class="p">.</span><span class="nf">FirstOrDefaultAsync</span><span class="p">(</span><span class="n">p</span> <span class="p">=></span> <span class="n">p</span><span class="p">.</span><span class="n">Id</span> <span class="p">==</span> <span class="n">id</span><span class="p">);</span>
<span class="p">}</span>
<span class="c1">// For the home page.</span>
<span class="k">public</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">List</span><span class="p"><</span><span class="n">Post</span><span class="p">>></span> <span class="nf">GetPostsAsync</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">await</span> <span class="n">_context</span><span class="p">.</span><span class="n">Posts</span>
<span class="p">.</span><span class="nf">Include</span><span class="p">(</span><span class="n">p</span> <span class="p">=></span> <span class="n">p</span><span class="p">.</span><span class="n">Authors</span><span class="p">)</span>
<span class="p">.</span><span class="nf">Include</span><span class="p">(</span><span class="n">p</span> <span class="p">=></span> <span class="n">p</span><span class="p">.</span><span class="n">Comments</span><span class="p">)</span>
<span class="p">.</span><span class="nf">ToListAsync</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This is a simple approach, but not really scalable. The method to get all posts is loading all the comments for every post even though they’re not needed. On my blog, this wouldn’t be a problem. But this can get expensive if the blog is very popular and millions of people post long-winded comments on it. Let’s improve this:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// For the home page.</span>
<span class="k">public</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">List</span><span class="p"><</span><span class="n">Post</span><span class="p">?>></span> <span class="nf">GetPostsAsync</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">await</span> <span class="n">_context</span><span class="p">.</span><span class="n">Posts</span>
<span class="p">.</span><span class="nf">Include</span><span class="p">(</span><span class="n">p</span> <span class="p">=></span> <span class="n">p</span><span class="p">.</span><span class="n">Authors</span><span class="p">)</span>
<span class="p">.</span><span class="nf">ToListAsync</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Better, but now we expose another problem. Suppose deep in our app logic, we get passed a <code class="language-plaintext highlighter-rouge">Post</code> instance and we want to show the comment count, but we’re not sure which query it came from. Notice that <code class="language-plaintext highlighter-rouge">post.Comments</code> is non-nullable. So I should safely be able to reference <code class="language-plaintext highlighter-rouge">post.Comments.Count</code> as far as the compiler is concerned. But that’ll throw a <code class="language-plaintext highlighter-rouge">NullReferenceException</code> if the query that loaded the <code class="language-plaintext highlighter-rouge">Post</code> didn’t include comments.</p>
<p>One solution is to make the collection nullable.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">Post</span> <span class="p">{</span>
<span class="k">public</span> <span class="kt">int</span> <span class="n">Id</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="k">public</span> <span class="n">IList</span><span class="p"><</span><span class="n">Author</span><span class="p">></span> <span class="n">Authors</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span> <span class="c1">// We always load this, so it's non-nullable.</span>
<span class="k">public</span> <span class="n">IList</span><span class="p"><</span><span class="n">Comment</span><span class="p">>?</span> <span class="n">Comments</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span> <span class="c1">// We don't always load this, so it's nullable.</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Then we’d check for null before accessing the collection. Something like this:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="kt">int</span><span class="p">></span> <span class="nf">GetCommentCountAsync</span><span class="p">(</span><span class="n">Post</span> <span class="n">post</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">post</span><span class="p">.</span><span class="n">Comments</span> <span class="k">is</span> <span class="k">null</span><span class="p">)</span> <span class="p">{</span>
<span class="k">await</span> <span class="n">_dbContext</span><span class="p">.</span><span class="nf">Entry</span><span class="p">(</span><span class="n">post</span><span class="p">).</span><span class="nf">Collection</span><span class="p">(</span><span class="n">p</span> <span class="p">=></span> <span class="n">p</span><span class="p">.</span><span class="n">Comments</span><span class="p">).</span><span class="nf">LoadAsync</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">post</span><span class="p">.</span><span class="n">Comments</span><span class="p">!.</span><span class="n">Count</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Yes, nullable collections suck, but in this case, it makes sense because it communicates an important distinction between the collection being empty vs the collection not being loaded.</p>
<p>So we’re in the clear, right? Well, no. It’s possible for <code class="language-plaintext highlighter-rouge">post.Comments</code> to be non-null, but not be fully loaded. Suppose, for some reason, earlier in the same request with the same <code class="language-plaintext highlighter-rouge">DbContext</code> we load a comment like this:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">var</span> <span class="n">comment</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_dbContext</span><span class="p">.</span><span class="n">Comments</span><span class="p">.</span><span class="nf">FirstOrDefaultAsync</span><span class="p">(</span><span class="n">c</span> <span class="p">=></span> <span class="n">c</span><span class="p">.</span><span class="n">Id</span> <span class="p">==</span> <span class="n">id</span><span class="p">);</span>
<span class="c1">// comment.PostId just happens to be the same as post.Id.</span>
</code></pre></div></div>
<p>And this comment belongs to the same post that we’re trying to get the comment count for. It turns out that even though we haven’t explicitely included or loaded <code class="language-plaintext highlighter-rouge">post.Comments</code>, it will be a non-null collection with one entry, the comment. Why?</p>
<p>When using eager-loading with EF Core, it <a href="https://learn.microsoft.com/en-us/ef/core/querying/related-data/eager">has an automatic-fixup feature</a>:</p>
<blockquote>
<p>Entity Framework Core will automatically fix-up navigation properties to any other entities that were previously loaded into the context instance. So even if you don’t explicitly include the data for a navigation property, the property may still be populated if some or all of the related entities were previously loaded.</p>
</blockquote>
<p>Since a <code class="language-plaintext highlighter-rouge">Comment</code> associated with the <code class="language-plaintext highlighter-rouge">Post</code> is already loaded in the <code class="language-plaintext highlighter-rouge">DbContext</code>, the <code class="language-plaintext highlighter-rouge">Post</code>’s <code class="language-plaintext highlighter-rouge">Comments</code> collection will be non-null and contain that comment. This is a bit of a gotcha, that we ran into with <a href="https://ab.bot/">Abbot</a> in local development recently, so it’s not just a hypothetical case. Here’s how I ended up fixing it:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="kt">int</span><span class="p">></span> <span class="nf">GetCommentCountAsync</span><span class="p">(</span><span class="n">Post</span> <span class="n">post</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">post</span><span class="p">.</span><span class="n">Comments</span> <span class="k">is</span> <span class="k">null</span> <span class="p">||</span> <span class="p">!</span><span class="n">_dbContext</span><span class="p">.</span><span class="nf">Entry</span><span class="p">(</span><span class="n">post</span><span class="p">).</span><span class="nf">Collection</span><span class="p">(</span><span class="n">p</span> <span class="p">=></span> <span class="n">p</span><span class="p">.</span><span class="n">Comments</span><span class="p">).</span><span class="n">IsLoaded</span><span class="p">)</span> <span class="p">{</span>
<span class="k">await</span> <span class="n">_dbContext</span><span class="p">.</span><span class="nf">Entry</span><span class="p">(</span><span class="n">post</span><span class="p">).</span><span class="nf">Collection</span><span class="p">(</span><span class="n">p</span> <span class="p">=></span> <span class="n">p</span><span class="p">.</span><span class="n">Comments</span><span class="p">).</span><span class="nf">LoadAsync</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">post</span><span class="p">.</span><span class="n">Comments</span><span class="p">!.</span><span class="n">Count</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The call to <code class="language-plaintext highlighter-rouge">_dbContext.Entry(post).Collection(p => p.Comments).IsLoaded</code> tells us if the collection is fully loaded or not. Automatic fix-up does not set <code class="language-plaintext highlighter-rouge">IsLoaded</code> to true for the collection. I left the null check as an optimization, but it’s not strictly necessary.</p>
<p><strong>UPDATE:</strong> On Twitter, https://twitter.com/RichardDeeming noted that <code class="language-plaintext highlighter-rouge">LoadAsync</code> checks <code class="language-plaintext highlighter-rouge">IsLoaded</code> internally, so the null check <em>and</em> the <code class="language-plaintext highlighter-rouge">IsLoaded</code> check is unnecessary. Thus we’re left with:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="kt">int</span><span class="p">></span> <span class="nf">GetCommentCountAsync</span><span class="p">(</span><span class="n">Post</span> <span class="n">post</span><span class="p">)</span> <span class="p">{</span>
<span class="k">await</span> <span class="n">_dbContext</span><span class="p">.</span><span class="nf">Entry</span><span class="p">(</span><span class="n">post</span><span class="p">).</span><span class="nf">Collection</span><span class="p">(</span><span class="n">p</span> <span class="p">=></span> <span class="n">p</span><span class="p">.</span><span class="n">Comments</span><span class="p">).</span><span class="nf">LoadAsync</span><span class="p">();</span>
<span class="k">return</span> <span class="n">post</span><span class="p">.</span><span class="n">Comments</span><span class="p">!.</span><span class="n">Count</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Since we can’t rely on the nullability of the collection to tell us if it’s fully loaded or not, I don’t think it makes sense to even use nullable collections. Which brings us back to the first approach:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">Post</span> <span class="p">{</span>
<span class="k">public</span> <span class="kt">int</span> <span class="n">Id</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="k">public</span> <span class="n">IList</span><span class="p"><</span><span class="n">Author</span><span class="p">></span> <span class="n">Authors</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="k">public</span> <span class="n">IList</span><span class="p"><</span><span class="n">Comment</span><span class="p">></span> <span class="n">Comments</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>I think it would be interesting if the type system could somehow express this distinction. For example, if the entity is returned from a query that includes the collection, then the collection is non-nullable and fully loaded. If the entity is returned from a query that doesn’t include the collection, then the collection is nullable and not fully loaded. I’m not sure how to do this in C#, but it would be interesting to explore.</p>Phil HaackWhen using an ORM with a web app, lazy loading will almost certainly result in hidden N+1 queries. Eager loading is a great way to avoid this, but has its own pitfalls. In particular, for each query, you need to be careful about what you include in the query. If you include too much, you can end up with a lot of data that you don’t need. If you include too little, you can end up with confusing logic. For example, deep in your application code, it may not be clear if a navigation collection has been loaded yet or not. This can lead to unexpected behavior.Lessons From a Startup Pivot2022-07-25T00:00:00+00:002022-07-25T00:00:00+00:00https://haacked.com/archive/2022/07/25/lessons-from-the-pivot<p>Building a startup is easy. You file some paperwork and bam! You’re a startup!</p>
<p>Building a start-up that’s sustainable and can pay everyone a nice salary, on the other hand, is very tough. We are launching <a href="https://www.producthunt.com/posts/abbot">Abbot on Product Hunt</a> today, and preparing last week made me reflect on some of the lessons we’ve learned since starting our business.</p>
<p><a href="https://ab.bot/">Abbot</a> is a Slack App (and bot) that helps teams scale their customer conversations in Slack.</p>
<p>Some of you might be thinking, “Wait! I thought Abbot was <a href="https://news.ycombinator.com/item?id=27974077">ChatOps as a service</a>?” Indeed, that was our original goal, but we found it tough to sell. Conversations often would go like this:</p>
<blockquote>
<p>them: What problem do you solve?
us: What problems do you have? Abbot can help with them all!
them: <em>snoring</em></p>
</blockquote>
<p>Yeah, not too effective as a sales pitch.</p>
<h2 id="lesson-start-with-selling-a-product-not-a-platform">Lesson: Start with selling a Product not a Platform</h2>
<p>Abbot started off as an automation platform. With Abbot, we could quickly write and host new capabilities of the bot from within Abbot. We used it internally to solve whole classes of problems and run experiments. Platforms give people the tools to solve whole classes of problems, but they don’t solve any specific problems on their own. And this is why platforms are hard to sell. For example, Amazon didn’t begin with selling their Amazon Web Services (AWS) platform. They started with an online bookstore. A concrete product. And over time, they reached the point where they could sell the underlying platform (AWS) the bookstore was built on.</p>
<h2 id="lesson-a-platform-does-make-it-easy-to-experiment-and-pivot">Lesson: A Platform does make it easy to experiment and pivot</h2>
<p>One thing we did do well during this time was to listen to our current and prospective customers. We noticed a theme. Many of them struggled to support their own customers in Slack. Slack has a neat feature called Slack Connect that lets you invite other companies into a shared Slack channel. We discovered that a lot of companies used this feature to provide support for their own customers. Their customers liked the immediacy of being in a shared chat room. It’s more responsive than getting on a phone or dealing with email. And it makes hand-off of discussions easy.</p>
<p>But supporting customers in chat brings its own set of problems. With so many conversations coming in, it can be tough to keep track of them all. Did we respond to every conversation? Is there any conversations waiting on a response? What’s our average response time? There are a ton of tools to answer those questions for email support, but not so much for chat.</p>
<p>This sounds like a very specific problem to solve! Fortunately, we had a flexible platform we could leverage and build upon to solve this specific problem. So we pivoted.</p>
<p>But as we did so, it didn’t take us long to run into other problems.</p>
<h2 id="lesson-dont-completely-neglect-engineering-and-infrastructure-in-order-to-achieve-product-market-fit">Lesson: Don’t completely neglect engineering and infrastructure in order to achieve Product Market Fit</h2>
<p>Before I get into the problems we ran into, it’s important to understand some context, lest you think we’re running a clown shoes operation. When we joined the YCombinator startup program, one of the key lessons drilled into us is that Product Market Fit is our top priority. And the reasoning is simple, if you don’t achieve product market fit, then you don’t have a business and your company runs out of money and dies. And it’s hard to build great software with a dead company. So while good engineering and infrastructure is important, if you spend too much time on that, but don’t achieve product market fit, then all that exquisite infrastructure goes to waste.</p>
<p>At the same time, if you wait too long to address your infrastructure needs, your product may fail under the weight of all that juicy product market fit. It’s definitely a delicate balancing act and one that you have to feel out for yourself. I hope that sharing some of our failures can inpspire some ideas on how to strike that balance for your own product and company. That balance is going to be different for every product and company.</p>
<h2 id="lesson-all-services-are-leaky">Lesson: All Services Are Leaky</h2>
<p>Around the time dinosaurs roamed the world hosting their own server racks, I was a fresh-faced college graduate starting my first job at a custom development shop. We had a full server rack in the closet where we hosted all our clients’ web sites. This put us in control of our own hardware, software, and destiny. Which as it turns out, isn’t always a good thing. I remember the time my boss wanted to demonstrate how our new hard-drive array was hot-swappable by pulling out a drive. He pulled out the drive and everything continued to work flawlessly. Just kidding. It all came crashing down like a soccer player who feels a gentle tug on the shirt in the penalty box (if the sport metaphor doesn’t work for you, just trust me on this one).</p>
<p>At the time, this was the way you built web software. You hosted everything.</p>
<p>Fast forward a bunch of years, and the pendulum swung to the other side. I didn’t want to host a damn thing. And with tools like Heroku, Azure Web Sites, AWS, Google Cloud, etc. I really didn’t need to. Much of my later career at Microsoft and GitHub was spent building developer tools and client applications. Occasionally, I’d build a website on the side for fun. I’d just deploy it to a cloud provider, take advantage of all their services, and not worry about it too much.</p>
<p>And this approach worked well! It worked so well, when I started a company with my colleague, Paul, we were able to <a href="https://ab.bot/">get a site up</a> and running in no time. And this all worked great, everyone was happy, and it all just worked out…Until <a href="https://www.dotnetrocks.com/default.aspx?ShowNum=1791">we needed to pivot</a>.</p>
<p><img src="https://user-images.githubusercontent.com/19977/180076597-6fbdb672-6539-43fd-b10d-1a1e667c8d8b.png" alt="Driver telling a police office he should have turned left at Albuquerque" title="Free Public Domain image from https://freesvg.org/1535673080" /></p>
<p>You may have heard that <a href="https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/">all abstractions are leaky</a>. In a similar manner, all services are leaky. Whether it be Software as a Service (SAAS), Platform as a Service (PAAS), Infrastructure as a Service (IAAS), or whatever the next thing is.</p>
<p>The first “abstraction” to break was relying on Azure Bot Service for our bot. The service provides a single API to make it possible to write a bot once for multiple chat services. Where have we heard that promise before, right?</p>
<p>As you might expect, it meant our bot had access to the lowest common denominator of what chat platforms have to offer. This was fine for our initial product idea, <a href="https://news.ycombinator.com/item?id=27974077">ChatOps as a service</a>. With ChatOps, the interface is primarily textual, so we didn’t need all the bells and whistles. But as we pivoted to a Customer Success product, it became imperative that we supported all the rich features that Slack has to offer.</p>
<p>This meant bypassing the bot service and writing our code to directly interact with the Slack API. I don’t regret that we started with the bot service as it got us far fast, but I do regret when we configured our Slack Events Endpoints (the URLs that Slack sends its events to), with the bot service URLs. Something like <code class="language-plaintext highlighter-rouge">https://slack.botframework.com/api/messages/events</code> (or whatever it is). The reason this was a problem is that we submitted our Slack App to the Slack App Directory. So every change we make to the app’s configuration can take up to six weeks to be approved by Slack. This meant that even though we were ready to go with our new approach, we had to wait for the Slack approval process to start serving Slack requests directly.</p>
<h2 id="lesson-own-your-traffic-with-a-level-of-indirection-built-in">Lesson: Own Your Traffic With A Level of Indirection built in</h2>
<p>A little sprinkling of indirection would have helped a lot here. We should have set up a DNS entry under our control, pointed it to Azure Bot Service, and configured Slack with that URL. It would have allowed us to point Slack to our own web server when we needed to switch. It just wasn’t a consideration in the beginning. At the time, we felt we wouldn’t need it. But it’s cheap to set up a DNS entry to provide a bit of indirection just in case.</p>
<p>As a corollary to this, I recommend managing DNS as code. We have several DNS providers and managing DNS across them can be a pain. But we now use <a href="https://github.com/octodns">OctoDNS</a> to manage all our DNS in some YAML files that are versioned within a GitHub repository.</p>
<p>Alright, back to the story. So what do you think happened once Slack approved our change to direct Slack traffic to our own server at https://ab.bot? We found a bug that broke everything.</p>
<p>Now a minute ago I was talking about all the benefits of indirection that using our own Domain Name afforded us. Well, I made another mistake here. I pointed Slack directly to https://ab.bot/, which the rest of our site also ran on. So we couldn’t just point this DNS entry back to Bot Service. And to revert the change might take another six weeks to go through Slack’s approval process. So we just knuckled down to fix it.</p>
<p>Today, we have a separation where https://ab.bot/ points to our marketing site and login page, but the app runs on https://app.ab.bot/. And for each service that needs to be publicly exposed, we use a unique host name in case we need to redirect traffic.</p>
<h2 id="lesson-balance-convenience-with-the-right-tools">Lesson: Balance Convenience With The Right Tools</h2>
<p>Having been a Microsoft employee in the past, most of my experience is with Azure. When in doubt, I tend to choose Azure tools because they’re familiar to me and it’s convenient to have everything in one place with the Azure Portal. However, this sometimes means I end up choosing tools that aren’t up to snuff. For example, we looked at Azure Front Door and Azure Application Gateway to provide a proxy that would let us better control how traffic reaches our site. Together, they supported the features we needed, but each one was missing something. And using them together seemed complex and overkill. So we ended up using Cloudflare which did everything we need.</p>
<h2 id="lesson-observability-and-monitoring-with-alerts-is-essential">Lesson: Observability and Monitoring With Alerts is Essential</h2>
<p>As we started to build out our new product offering, we started to notice connection timeouts. Our database was getting hammered periodically and we had no understanding of why. Fortunately, we were able to figure it out by adding better instrumentation.</p>
<p>See, we had a lot of logging, but not enough. Even worse, we didn’t have enough alerts set up early on. So some errors would occur and we wouldn’t notice them. This wasn’t really a problem at first, but as our pivot started getting traction with real customers, being able to diagnose problems quickly rose up in priority.</p>
<h2 id="lesson-code-your-infrastructure">Lesson: Code Your Infrastructure</h2>
<p>We ended up figuring out that some problematic code was the cause of some of our error spikes. But we were also on a pretty low-tier plan for the database and web server. As we got closer and closer to shipping our 1.0, we realized we needed to scale up. Now <em>this</em> is where the cloud shines, right?</p>
<p>Well no, we happened to choose a plan that didn’t allow us to automatically scale up. We had to migrate to a new plan. This meant a lot of manual work and scripting to migrate our web servers and data to new servers and databases in the cloud.</p>
<p>And that helped a lot! So now we’re on a plan that will allow us to automatically scale up our infrastructure! Except we happened to choose a region that’s at capacity due to recent supply chain issues. So in order to scale up our hardware even more, we need to do yet another migration.</p>
<p>It wouldn’t be so bad if all our infrastructure was scripted using something like Terraform or Pulumi, but we’re not quite there yet. We’ve made a lot of progress with automating our infrastructure lately, but we’re still not at the point where we can just migrate to a new data center with a few commands.</p>
<p>I used to believe that being Cloud Provider agnostic was a waste of engineering effort, but as we build out our product, we can see that even if you stick with a single cloud provider, it has benefits.</p>
<h2 id="lesson-hire-great-people">Lesson: Hire great people</h2>
<p>One of our saving graces is we’ve been very fortunate in our early hiring. In the areas where I’m weak, we have folks who are superstars. At this stage of our company, every person we hire has a huge impact on the company. If we hadn’t brought in these amazing people, I think we’d have failed already.</p>
<p>If you are setting out to build your own startup, I hope there’s at least something in this post that resonates with you. And if you’re giving Abbot a try, I hope it give you some insight into the sweat, tears, and love we’ve put in to build it. Take care!</p>Phil HaackBuilding a startup is easy. You file some paperwork and bam! You’re a startup!