Filtering by Category: tech

What kind of software business can you build today?

I took a couple of months “off” before joining Honeycomb — spent mostly doing go to market consulting work for pre-launch startups and talking to people about what’s interesting, clocking 2–5 meetings a day.

Exhausting. Interesting.

TL;DR

You can either build a marketplace or be vertically integrated.

What you cannot do is build a platform, unless it’s for some completely un[der]served market.

Platform plays are hard

This should be a truism. How do you compete with Amazon, Google, or even Twilio? If we take them on head on, we lose. There’s nothing you’re going to build that they can’t copy. Witness the snap-ification of all Facebook properties.

Users who come to you first, an entirely new generation, may stay with you. But the vast population using an existing platform will prefer to use that platform’s copy of your feature because the value to them generated by the network effects (and their own personal investment) of the incumbent platform is not worth abandoning to start from scratch on your thing.

Platforms tend to be subject to winner-take-most-if-not-all dynamics and you have to find somewhere where a winner has not already taken the most.

Thus un[der]served markets. It seems there’s plenty of space to build non-generic platforms for specific markets with their own needs, language, culture, and particularities. Of course that’s TAM limiting — and unless there are premiums to be charged (high relative margins) — maybe not fit for venture capital.

Be vertically integrated

Instead of building wide, build deep. Although I think this applies more to existing software businesses rather than someone building something new. Unless it’s in an unserved niche, in which case maybe you can build deep before attracting too much attention from outside the market you choose to operate in.

If you look at Salesforce’s acquisitions, ignoring the usual failed forays, what stands out is that they are going deep in go to market. They’re buying up bits of the funnel, including customer success/support for the world of MRR/ARR driven GTM machines.

Build (or modernize) marketplaces

Connect supply and demand that haven’t been connected before, or electro-internet-connectify a marketplace that’s still run off paper-faxes-and-phone-calls. Usually in the process you’ll either become a new (hopefully) value-adding intermediary or disintermediate incumbent rent-seekers.

See Haven and Grand Rounds.

What is the critical differentiator for incumbents, and can some aspect of that differentiator be digitized? If that differentiator is digitized, competition shifts to the user experience, which gives a significant advantage to new entrants built around the proper incentives Companies that win the user experience can generate a virtuous cycle where their ownership of consumers/users attracts suppliers which improves the user experience— Aggregation Theory by Ben Thompson

Promises, Microservices, and Intent

Last year, after a bit of wrangling and lots of editing by the fantastic Jenn Webb, O’Reilly published a discussion Mark Burgess and I had on one of his trips through the Valley as a podcast.

The audio is iffy, which is entirely my fault. I have zero experience recording for any purpose other than capturing notes from interviews for my own use later.

I’ve referenced Mark’s work before in talks and presentations like this one at O’Reilly Velocity two years ago.

The interesting thing about Mark’s view is that he approaches systems as a physicist, looking at systems as phenomena, as things that happen, as something that behaves with behaviors that can be described. My highlights from our conversation below.

Measuring Intent

It all goes back to scales, the order of magnitude we look at. When you measure, observe, and characterize the world you have to find something to measure it by. The descriptions of systems are often qualitatively different at different scales. This is not something we tend to understand in a clear way in CS.

In CS, you’re mostly focused on intent — what we call semantics. There’s no numerical measurement. We simply write code. In physical phenomena, changes in measurable things have scales. Part of my work has been to figure out how we could invent scales and measuring sticks for semantics. This is how Promise Theory came about.

A promise is a description of intent that you share with someone. Until you share it, it’s just an intent. As soon as you share it, it becomes a promise. In a promise-oriented view, you look at what are the agencies capable of bringing about state (making promises). In the most abstract sense, an agency is anything with a purpose or function with some semantics and behavior — machines, people, tools.

Microservices’ Promises

The purpose of a promise is to describe that behavior and be able to break it down into constituent parts. It provides a language for systems that allows us to describe the elements of a system and what happens when you put them together, like atoms into molecules, etc.

Microservices are a modern incarnation of this idea.

There are two ends of a promise: the offer of a service and the acceptance of it. You may choose to accept less than is offered. This is another way of talking about autonomy of the actors in a system. This is a good representation of how the world actually works. You simply cannot force your will onto another actor, or device, without attacking it in some way. Ultimately, it’s up to that agent to act the way you want.

This drives you to understand all the communications pathways in a system and exposes all the failure modes that can manifest.

Humans Feed Back

They’re also a great example of where humans enter the picture.

We cannot understand computer systems without taking into account humans, because they are an integral part providing not only input, but feedback (and modification). It’s a symbiosis. Microservices are an example of how you can break things down to smaller, more manageable pieces in order to scale the human.

If this kind of thing interest you at all, read Mark’s books: In Search of Certainty andThinking in Promises.

Theory in Practice — OODA, Mapping, Antifragility

Based on a talk presented at Velocity 2016 in Santa Clara, this post tries to show the practical application of concepts like OODA, Wardley Maps, and Antifragility with examples from my day-to-day work at a startup.

Theory

OODA — Observe Orient Decide Act

Observe the situation, i.e. acquire data. Orient to the data, the universe of discourse, the operating environment, what is and isn’t possible, and other actors and their actions. Decide on a course of action. Act on it.

Typically, you hear people saying that we’re supposed to go through the loop [O -> O -> D -> A -> O -> O…] faster than others. Let’s break that down.

  • If we traverse the loop before an adversary acts, then whatever they are acting to achieve may not matter because we have changed the environment in some way that nullifies or dulls the effectiveness of their action. They are acting on a outdated model of the world.
  • If we traverse the loop before they decides, we may short circuit their process and cause them to jump to the start because new data has come in suggesting the model is wrong.
  • If we traverse the loop at this faster tempo continuously, we frustrate their attempt to orient — causing disorientation — changing the environment faster than they can apprehend, much rather act.
  • We move further ahead in time. Or to be more exact, they’re falling further backwards: unable to match observations to models, change orientation, have confidence in decisions, or act meaningfully.

This is what Boyd called operating inside someone’s time scale.

Our main means of connecting the components of the loop is via models (and projections of cause/effect based on those models). Observations are tied into and given context via models.

Another way to think of models is as maps.

Mapping

This is a Wardley, or Value Chain, Map. It’s the most useful model I’ve encountered for building products or businesses. Watch Simon’s OSCONkeynotes or read his blog to really dig in to the concept.

It starts with a user need at the top. What problem are we solving? How are we going to make someone’s life better.

Then it goes deep, laying out the supply (dependency) chain of components needed to service that need. The further down, the less visible and exposed the component is to our end user. For example, if we’re building a SaaS product, users are never (or should never) be exposed to the systems running the code. This is the Y axis.

The X axis is where it gets interesting. It provides stages of development that components map into. Nearly everything naturally moves from the left to the right over time as invented or discovered things become standardized, well understood, built by more producers competing for market share, until some eventually become absolute commodities or provided as utilities. It’s a kind of natural evolution.

  • Genesis [stage 1]: something that’s being discovered/built from scratch
  • Custom Built [stage 2]: built out of existing technologies but highly customized for a specific use case and not generalized to broad use
  • Product [stage 3]: COTS software, something bought from someone else vs self-built
  • Commodity [stage 4]: something that’s effectively fungible, for which there are multiple equivalent providers, that may be provided as a utility

Individual components, regardless of their stage, can be expanded into finer grained production pipelines, marked as something that’s either provided or consumed, and aligned with methodologies like in-house-agile-developed vs outsourced-to-cloud-provider.

Finally, each component can be treaded as a piece on the field and moving them around as functions of product strategy, attempts at changing the competitive landscape.

For example, open sourcing something to try to commoditize it or create a de facto standard. Or providing something as a utility / platform / API in order to build a moat (that you can also consume) out of the ecosystem that you engender around it

[Anti]Fragility

But everything is constantly changing. Which means our map can become stale fast. Which makes us fragile — exposed and unaware — to ever more risks. Black swans.

 

A black swan is only a black swan if you can’t predict it (or assign it a probability). They’re inevitable. As our maps become out of synch with the real world, non-black swans become black swans. It’s possible to be fragile to one kind of black swan but not another. There are activities or patterns that will make us fragile with respect to something. And those that will make us antifragile.

 

There’s no such thing as absolute antifragility. It’s contextual. A severe enough stressor over a short enough time period will destroy anything.

Maps can be made robust (to some scale) through adaptive mechanisms, learning and correcting to match for change in the world.

But beyond some scale, every map is fragile. The world can change faster than, or so severely that, any attempt to update the map fails. Events can get inside a map’s timescale.

Systems can be antifragile (to some scale) through constant stress, breakage, refactoring, rebuilding, adaptation and evolution. This is basically how Netflix’s chaos army + the system-evolution mechanism that is their army of brains iterating on the construction and operation of their systems works.

pragmatism_5.png

For example, here’s our model of the APIs or services we rely on — smooth and reliable, with clearly defined boundaries and expected behavior. This is also the model that those things have of the APIs and services they rely on. All the way down

 

But this is how most things actually look. Eventually in the course of operation, the gaps line up in such a way that a minor fault event becomes magnified into systemic failure.

Systems, software, teams, societies — everything eventually crumbles under the weight of it’s own technical debt.

Which is why we should be refactoring, paying down technical debt, or what I just call “doing maintenance”, all the time at every layer.

Practice

Caveats: My views don’t represent those of my employer or anyone else and a great deal of detail is left out.

Example: mapping at work

I’ll build a map for a new feature SignalFx just released in beta.

Starting with the user need which I describe as “discovering known and unknown unknowns.”

A lot is left out, but generally speaking: on the top left we have the need, immediately connected to that is how that need is served and proceeding out from there is a generalized view of the supply chain of components needed to make it so.

Some things worth noting:

  • We rely on utility or commodity technology and services for all our infrastructure hardware and software, like operating systems, and also middleware — using things like AWS, Linux, Kafka, Cassandra, Elasticsearch, etc. This is standard behavior for a software as a service company.
  • We rely on relatively standard means of getting data into the system, in our case collectd, StatsD, Dropwizard metrics, etc., and a host of plugins and libraries that conform to open APIs and use well known open, or public, protocols.
  • We can see that there’s a lynchpin without which the map would fall apart, the streaming real-time analytics engine.
  • In order to build what was needed to serve that user need we started with, we needed to build many other things: a specialized quantization service, lossless + real-time message bus, specialized timeseries database, a high-performance metadata store, real-time streaming analytics engine, and an interactive real-time web-based visualization for streaming data, etc.

Many of the components we built are, if they were generalized, standalone products that others build entire companies on. In this specific case those are all the open source technologies — Kafka, Cassandra, Elasticsearch, etc — that we built our highly customized components out of.

Given all of that, I have one important positive question each day: Given the amount of time I’m going to spend working today, what one thing can I do to move the needle in serving this user need through what we do?

And one negative one, seeking invalidation: Is there any evidence that our map, our hypothesis, our approach, have been invalidated?

  • Is our projected user need real? Will people pay for it? Is it the problem they actually want solved? Do people really not want leverage? Do they not want to be given more power and time through tools? Do they want thinking to be replaced, instead of force-multiplied?
  • Is our lynchpin really the point of leverage and differentiation we believe it to be? Has it become a commodity and we’re just fooling ourselves into thinking we’ve built something novel?
  • Has the territory changed in any way, through macro trends or the actions of players in the ecosystem, such that we need to rework our model?

Example: knowing what’s possible

Imagine we want to build a personal relationship management [PRM] system to meet some a need for people to manage their complicated and ever-growing network of contacts.

The top left is where we’re starting from. The y-axis is basically features or sub-capabilities that add up to something. The x-axis is what they add up to: products or capabilities that are in and of themselves valuable. The bar for something belonging in the leading row is being a viable sub-product. Everything in the column below are the features needed for it. Where the line is for being able to declare that we’ve built a minimum usable product may be different per column, as may the line for what constitutes an MVP.

We have limited time, people, and money. So we can only build so many things at once. Let’s say we can only build one column at a time. We have to get to usability and viability in each column to be able to expand users and business sufficiently to build the next column.

But every single thing we build limits our options for the next thing we build. We can go down and we can go to the right.

We can scrap everything further below and further to the right of the point we’re at today and figure out something else to build from where we are. This is effectively a pivot.

But what we can’t do is go from 3 steps down in the 2nd column [a graph of your contacts that’s auto generated based on your communications with them over Gmail, Twitter, LinkedIn, Facebook, and Outlook email that shows degrees of separation] to, say, a restaurant reservations and point of sales SaaS product. There’s no getting from here to there. But you can get from here to a product referral network.

Seeking invalidation:

  • Is a personal relationship management service still the best way to serve the user need? Is there a better way?
  • Can we build to that better way from where we are?
  • Have we built an minimum usable product? Is it viable? Can we generate enough business (or funding) from what we’ve built to build the next thing?

Example: hiring for antifragility

The core principle of antifragility, as I see it, is to arrange things such that we get stronger through stress. More or less how muscle growth works.

How do you build that into an organization? How do you decrease brittleness? The only way I’ve ever found is through diversity. Inclusion, and forced exposure, to different points of views is absolutely necessary if we don’t want to get stuck — stuck in a way of thinking, stuck in a way of dealing with issues, stuck with a pattern of response, stuck in a point of view that makes us blind to threats and opportunities.

Think of it this way. We have to stir the pot in order to not get trapped in a local maximum. Not once, but constantly. Even a hint of homogeneity — whether it’s of people or ideas or practices or anything — is a clear signal that we are fragilizing and becoming brittle.

For my team, here’s what that looks like: no one in my org has a background in tech except for me. My background is both deep and wide, but it’s 90% tech. There’re just a large swath of things I’m blind to. What I’ve got is people who’ve studied art, biology, english lit, who don’t look like me or think like me. Things that I’m fragile to, they’re not. As a group, we’re way stronger than if I was hiring copies of myself.

Seeking invalidation:

  • Is this the right team? Can they do what we need to do right now? Can they do what we need to do in a quarter, in a year, in 5 years?
  • Are they the wrong team, or am I failing at
  • Helping them get from where they are to where I need them to be?
  • Getting the most out of their perspectives?
  • Creating a safe environment for them to bring their best to the table?

Questions

John Allspaw asked if it’s possible for a person to be anti fragile. I don’t think so. I don’t think any given person or component of a system can be antifragile. I think groups and systems can be made antifragile. Complexity can be a symptom of the build up of antifragility in a system. Beyond some envelope, it’s also a harbinger of collapse.

Peter van Hardenberg asked where I set the bar. Assuming baseline functional competence (can do the job at hand), the next thing I look for is differentness. What do you bring to the table that’s unique from what we already have?


Wrapping up, here are the daily operating principles arising from this study:

Always be refactoring

Diversity has intrinsic value

Territory > Map

Seek invalidation


The above builds on ideas in these previous talks and posts:

The original abstract was way too ambitious for a 40 min time slot. The presentation suffered quite a bit from me erratically moving through the material, trying to pack in too many ideas.

product rails

I’ve written about not forgetting the future you dreamed of to settle for the present you’ve made.

On the way to building something great, we inevitably build other (hopefully useful) things. We’re swayed by what customers claim to want, what engineers say can or cannot be done, what we can figure out how to market and sell, what investors think will make money, what the press gets excited by, etc.

It’s challenging to keep in mind where you intended to go when you’re working hard to just take the next step. Here's a way to think about it.

The top left is where we’re starting. Z is the vision. The y-axis is basically features or sub-capabilities that add up to something. The x-axis is what they add up to: products or capabilities that are in and of themselves valuable. The bar for something belonging in the leading row is being a minimum usable product subset of Z. Everything in the column below an MUP is what's needed for it. Where the line is for being able to declare that we've built an MUP (the depth needed in a column) may be different per column and needs to be called out. Where the line is for something that has go-to-market viability per column may be different still. All of which is different from the depth we want to go to. Z is realized when the whole matrix has been built.

Everything’s a hypothesis:

  • Is each of the “products” sufficiently valuable that someone would pay for them on their own?
  • Is the minimum usable depth we project actually sufficient for someone to experience value?
  • Is the minimum usable depth sufficient to get the product to get the product to the point of go-to-market viability — we can market, sell, and close business against it? If not, how much further?
  • Is going any deeper than that worthwhile for the customer or the business?
  • Are these the right capabilities in the best order?

To some degree, the order doesn't matter. The ideal case is to get to something in each column that provides enough tangible value and positive experience that someone would pay for it before moving on. But as long as we don’t leave the matrix, we’re still progressing towards the vision.

At every step, we need metrics for success and failure. In my view, it’s more important to know what constitutes disconfirmatory evidence than confirmatory—so we know when it’s time to cut our losses and move on.

There's also an existential question: is Z the right thing. Are we building it for its own sake, or to solve some specific problem in the world? Assuming we’re driven to fix something, to make someone’s life better — what happens if there’s a better way to do it than this one? How do we even know? This is impossible without actively seeking disconfirmation.

This is where going to market matters the most. It’s the sensing mechanism to discover how the map compares to reality.

A final note: what we build today limits what we can build tomorrow. We can go deeper and broader. And we can stop where we are; discard everything that might follow, build a new vision to a new place. But that new place has to be reachable from where we are right now. Every thing we build closes some doors and opens others. It’s near impossible to do something completely discontinuous.

minimum usable product

A lesson from 9ish years in and adjacent to product work.

Minimum viability is very much a product-outwards perspective: what’s the least amount of work we can do to find out whether going down this line of thinking is a business idea that’s worth being invested in. It has nothing to do with viability for users.

It’s a well worn notion that the right way to build a product is to iterate through stages of development, where at each stage you deliver something that, on it’s own, provides real incremental value by accomplishing the user's goal appreciably faster/cheaper/better than was possible before. A functional approach.

What makes a product viable for use is something that’s more usable at each stage of creation; that creates experiences of greater efficacy at every turn; that provides incremental wins that add up to something much greater—a sense of joy. [Something I’ve seen enough times to say it with a straight face.] This is distinctly not a product-out orientation—but instead a user-in orientation.

We have to make up for the pain we put users through--our stumbling attempts at building something useful, the suffering of (re)learning how to do something, breaking their workflows--with some pleasure on the other side. 

Leading to questions that should be answered (see the HEART framework for thoroughness):

  • What is the qualitative, subjective improvement from the perspective of the user? Does it feel better? Does it yield results of higher quality?
  • What is the quantitative, objective improvement from the perspective of the user? Does it get the task done faster? Does it yield more results?
  • What is the quantitative, objective improvement from the perspective of the product? Is it faster? Does it do more of what users want?

Make Minimum Usable Products.

the dangers of models

All models are wrong; some are useful.

Disconfirmatory evidence is more important than confirmatory evidence.

Actively seek model invalidation.

Every thing was built in some context, or scale. Reading primary sources, or learning how/why a thing was made, is essential to understanding  the conditions that held and knowing bounding scales beyond which something may become unsafe.

This is something I think about a lot. It's true in software, distributed systems, and organizations. Which is the world I breathe in every day at SignalFx.

It began to knit together around OODA:

  • ooda x cloud-- positing how it OODA relates to our operating models
  • change the game-- the difference between O--A and -OD- and what we can achieve
  • pacing-- the problem with tunneling on "fast" as a uniform good
  • deliver better-- the real benefit of being faster at the right things
  • ooda redux-- bringing it all together

OODA is just a vehicle for the larger issue of models, biases, and model-based blindness--Taleb's Procrustean Bed. Where we chop off the disconfirmatory evidence that suggests our models are wrong AND manipulate [or manufacture] confirmatory evidence. 

Because if we allowed the wrongness to be true, or if we allowed ourselves to see that differentness works, we'd want/have to change. That hurts.

Our attachment [and self-identification] to particular models and ideas about how things are in the face of evidence to the contrary--even about how we ourselves are--is the source of avoidable disasters like the derivatives driven financial crisis. Black Swans.

  • Black swans are precisely those events that lie outside our models
  • Data that proves the model wrong is more important than data that proves it right 
  • Black swans are inevitable, because models are, at best, approximations

Antifragility is possible, to some scale. But I don’t believe models can be made antifragile. Systems, however, can.

  • Models that do not change when the thing modeled (turtles all the way down) change become less approximate approximations
  • Models can be made robust [to some scale] through adaptive mechanisms [or, learning] 
  • Systems can be antifragile [to some scale] through constant stress, breakage, refactoring, rebuilding, adaptation and evolution— chaos army + the system-evolution mechanism that is an army of brains iterating on the construction and operation of a system

The way we structure our world is by building models on models. All tables are of shape x and all objects y made to go on tables rely on x being the shape of tables. Some change in x can destroy the property of can-rest-on-table for all y in an instant.

  • Higher level models assume lower level models 
  • Invalidation of a lower level model might invalidate the entire chain of downstream (higher level) models—higher level models can experience catastrophic failures that are unforeseen 
  • Every model is subject to invalidation at the boundaries of a specific scale [proportional to its level of abstraction or below]

Even models that are accurate in one context or a particular scale become invalid or risky in a different context or scale. What is certain for this minute may not be certain for this year. What is certain for this year may not be certain for this minute. It’s turtles all the way down. If there are enough turtles that we can’t grasp the entire depth of our models, we have been fragilized and are [over]exposed to black swans.

This suggests that we should resist abstractions. Only use them when necessary, and remove [layers of] them whenever possible.

We should resist abstractions.

Rather than relying on models as sources of truth, we should rely on principles or systems of behavior like giving more weight to disconfirmatory evidence and actively seeking model invalidation. 

OODA, like grasping and unlocking affordances, is a process of continuous checking and evaluation of the model of the world with the experience of the world. And seeking invalidation is getting to the faults before the faults are exploited [or blow up]. 

Bringing it all back around to code--I posit that the value of making as many things programmable as possible is the effect on scales.

  • Observation can be instrumented > scaled beyond human capacity
  • Action can be automated > scaled beyond human capacity
  • Orientation and decision can be short-circuited [for known models] > scaled beyond human capacity
  • Time can be reallocated to orienting and deciding in novel contexts > scaling to human capacity

That last part is what matters. We are the best, amongst our many technologies, at understanding and successfully adapting to novel contexts. So we should be optimizing for making sure that's where our time is spent when necessary.

Scale problems to human capacity.

aws lambda - some words

To get these out of my head so I can stop thinking about them...

At re:Invent last year, Ben Golub was up on stage singing the praises of Docker. The masterminds at AWS had arranged for a solid 20-30min of Docker love-in before making the day 2 technical announcements. Ben said that [one of] Docker's goals was to free developers from having to worry about production and delivery (or something like that, see his keynote). Then Werner comes on stage, describes Lambda, and more or less says that while others are trying to free developers--Lambda actually does that. Pretty amusing.

Lambda will drive some usage away from other AWS services. I've already seen experimentation and real usage start amongst high end AWS users (not just Netflix). You could view it as cannibalization, but it's much smarter. Presumably AWS has figured out how to price Lambda in an accurate way such that the cost of all the underlying and adjacent services consumed is priced in.

Lambda might be a "true" PaaS in the sense of being a pure runtime where you don't have to understand the underlying mechanics or implementation of compute, storage, database, etc etc at all. There are no buildpacks, runtime plugins, etc etc like you have in most PaaSes.

Like Jeff Barr said in his blog post: "You don't have to configure, launch, or monitor EC2 instances. You don't have to install any operating systems or language environments. You don't need to think about scale or fault tolerance and you don't need to request or reserve capacity. A freshly created function is ready and able to handle tens of thousands of requests per hour with absolutely no incremental effort on your part, and on a very cost-effective basis."

Although it has constraints, like only being Node and only allowing up to 1GB of memory consumption per function (last I checked), etc--it's a completely abstracted runtime environment. You give it code and a few variables. It does the rest.

It completely removes Ops. Why DevOps when you can just Dev? It's more like Google App Engine than anything else out there. But GAE won't let you have long running functions (more than a few secs, last I checked), so in its limited way it's already a step ahead.

Where a Docker container gives you theoretical portability because your entire app is packaged in a way that's independent of what it's running on (but not really). Lambda locks you in because you have no idea how your code is running or what it takes to run your code. The only thing you could conceivably move to is GAE, but you'd have to rewrite bits and metadata in order to do it. Oh, except that GAE doesn't do Node. So nevermind.

It's brilliant. 

It's also dangerous. If you never learn how the thing below what you are doing--what you are downstream of and rely on--works, then you become intrinsically dependent on the provider of that service. Great when that service is an actual commoditized utility with multiple providers in a competitive marketplace. Miserable when it's a monopoly. Creating that dependence is good gameplay on AWS's part. Not providing equivalent alternatives that conform to the same interfaces is bad gameplay on everyone else's. Becoming hooked is a poor decision on our part, unless we do it with eyes wide open and willingness to do the work of unhooking ourselves in the future.


Or, as Nick Weaver puts it: