Most people learning audio programming aren't making a standalone audio app where they do all the processing, or at least not an interesting one. They're usually either making something like a plugin that ends up in somebody else's bus/graph, or something like a game or application that creates a bus/graph and shoves a bunch of different stuff into it.

It depends on your appetite for risk and the cost of failure.

A big part of the problem is that general purpose computing systems (operating systems and hardware) are not engineered as real-time systems and there are rarely vendor guarantees with respect to real-time behavior. Under such circumstances, my position is that you need to code defensively. For example, if your operating system memory allocator does not guarantee a worst-case bound on execution time, do not use it in a real-time context.

I tend to agree, but...

From my recollection of using Zoom-- it has this bizarre but workable recovery method for network interruptions. Either the server or the client keeps some amount of the last input audio in a buffer. Then if the server detects connection problems at time 't', it grabs the buffer from t - 1 seconds all the way until the server detects better connectivity. Then it starts a race condition, playing back that amount of the buffer to all clients at something like 1.5 speed. From what I remember, this algo typically wins the race and saves the client from having to repeat themselves.

That's not happening inside a DSP routine. But my point is that some clever engineer(s) at Zoom realized that missing deadlines in audio delivery does not necessarily mean "hosed." I'm also going to rankly speculate that every other video conferencing tool hard-coupled missing deadlines with "hosed," and that's why Zoom is the only one where I've ever experienced the benefit of that feature.

Indeed, like all real-time systems you need to think in terms of worst-case time complexity, not amortized complexity.

An audio glitch is very annoying by comparison, especially if the application is a live musical instrument or something like that. Even the choppy rocket motor sounds of Kerbal Space Program (caused by garbage collector pauses) are infuriating.

It's kind of the difference between soft and hard real time systems. Although most audio applications don't strictly qualify as hard real time (missing a deadline is as bad as a total failure) but failing a deadline is much worse than in graphics.

You might sometimes build an app where (through your operating system) you connect directly with an input device and/or output device and then do all the audio processing yourself. In this case, you'd more or less control the whole bus and all the code processing samples on it and have a fairly true sense of your deadline. (The OS and drivers would still be introducing some overhead for mixing or resampling, etc, but that's generally of small concern and hard to avoid)

Often, though, you're either going to be building a bus and applying your own effects and some others (from your OS, from team members, from third party plugins/libraries, etc) or you're going to be writing some kind of effect/generator that gets inserted into somebody else's bus in something like a DAW or game. In all these cases, you need to assume that all processing code that isn't yours needs all the time that you can leave for it and just make your own code as efficient as is reasonable.

A bus is a shared medium of communication[1]. Often, busses are time-division multiplexed[2], so if you want to use the bus, but another module is already using it, you need to wait.

For example, if your audio buffers are ultimately submitted to a sound card over a PCI bus, the submission may need to wait for any ongoing transactions on the PCI bus, such as messages to a graphics card.

[1]: https://en.wikipedia.org/wiki/Bus_(computing)

[2]: https://en.wikipedia.org/wiki/Time-division_multiplexing

Modern computers are ridiculously fast, relatively speaking you don’t need much resources to calculate a missile trajectory, so “simply” 100% sure doing some calculations at a fixed rate, with even a GC cycle that has a deterministic higher bound (e.g. it will go throw the whole, non-resizable heap, but it will surely always take n seconds), you can pass the requirements. Though a desktop computer pretty much already begets the hard part of hard realtime, due to all the stuff that makes it fast - memory caching, CPU pipelining, branch prediction, normal OSs scheduling, etc.

I suppose it's hard to make guarantees with different environments and hardware, but I realized when we (non-realtime people) ship software we don't really have guarantees for when our functions run.

- You are at the mercy of the browser. If browser engineers mess up the audio thread or garbage collection, even the most resilient web audio app breaks. It happens.

- Security mitigations prevent or restrict use of some useful APIs. For example, SharedArrayBuffer and high resolution clocks.

[1] https://bungee.parabolaresearch.com/bungee-web-demo

Musicians have keenly trained ears. I would imagine their much more sensitive to audio latency than even a pro gamer, nevermind average Joe off the street.

Where latency really matters is when you have a musical instrument that plays a sound and it's connected to a monitor. If those sounds are separated by more than 8ms or so the difference will be super noticeable to anyone, including Joe off the street.

I'd be interested for someone to run a user study on MIDI keyboard latency. I'd bet $3.50 that anything under 40 milliseconds would be sufficient. Maybe 30 milliseconds. I'd be utterly shocked if it needed to be 8 milliseconds. And I'd be extremely shocked if every popular MIDI keyboard on the market actually hit that level of latency.

I think so. TBH I'm quite new to the world of DSPs so I don't know the right terminology. The purpose of the DSP (which I should've mentioned in my original post now that I think of it) is to tweak the speakers on my laptop - there are for example ways to "fake" bass (through missing harmonics), or have dynamically changing bass. I'll have a look at VirtualAudioStream, thanks for the recommendation.

What are you even talking about, man.