How to flip
an audio bit

Hello!

Overview

Part I: Understanding the goal
Part II: Implementing a bit crushing effect
Part III: Actual bit flipping
(Part IV: Revisiting Oto Biscuit)

Part I

Understanding our goal

## Prologue: Different kind of nerds

<aside class='notes'>
							<li>This might be an unusual topic for day-to-day web developers. Meet David</li>
						</aside>

---

## Prologue: Different kind of nerds

<aside class='notes'>
							<li>In this image you see various machines connected through cables in a modular way.</li>
							<li>We'll come back to this analogy later when looking at code.</li>
						</aside>

---

## Could you help me with this code?

> I need to disable or invert specific bits of an audio stream

<aside class='notes'>
							<li>How it all began. We'll come back to this question, but need to cover some grounds first</li>
						</aside>

---

## Oto Biscuit

<aside class='notes'>
	<li>This is a discontinoud product with a distinct sound. It uses very expensive 8-bit converters, only 2000 pieces where produced</li>
	<li>By using <b>8-bit converters</b>, digital processing and analog resonant filters, BISCUIT opens up a wide range of sounds from harsh distortion, digital and aliasing artifacts to warm and fat 8-bit sounds.</li>
	<li>You can mute or invert each of the 8 bits of the digital signal by pressing one of the 8 rectangular switches, to generate strong distortions with an intense digital character</li>
	<li>https://www.otomachines.com/product/biscuit/</li>
</aside>

<!-- --- Skipped for brevity
            ## Digitizing Audio waves

![Sampling and quantization](images/4-bit-linear-PCM.svg)

Note:

Sampling is the conversion of a continuous signal to a discrete signal

Put another way, a continuous sound wave, such as a band playing live, is converted into a sequence of digital samples (a discrete-time signal) that allows a computer to handle the audio in distinct blocks.

Sampling: how often we "listen" for values. Default sample rate for a audio context is 44,1 kHz according to devtools

Humans hear ~20kHz, we have to sample at least double of that (https://en.wikipedia.org/wiki/44,100_Hz)

4-bit Quantization: 16 possible values

Values are represented as floats, not ints..
-->
						---
						## Audio terminology

<div class='flush-left'>
    <ul>
      <li class='fragment'><strong>Input</strong>: Anything that makes a sound (microphone, audio file or synthetic sound)</li>
      <li class='fragment'><strong>Output</strong>: Speakers or another audio processing device</li>
      <li class='fragment'><strong>Effect</strong>: Anything that processes an audio stream</li>
      <li class='fragment'><strong>Audio channel</strong>: Mono (one), Stereo (two), 5.1 etc.</li>
      
      <li class='fragment'><strong>Sample value</strong>: A float representing a single value for the audio stream of a channel at a given time ([-1 .. 1])</li>
      
    </ul>
  </div>

Note:

**Input**: A microphone, audio file, oscillator or synthesizer

**Effect** ~= DSP = [Digital signal processors](https://en.wikipedia.org/wiki/Digital_signal_processor)
    Any filter or modularization ("the fun part")

**Sample Size**: The number of samples per channel (quantization)

**Frame**: All samples over all channels at a given time

---

## The plan
						![bit crushing visualization](images/bitrate-to-samplerate.png)

We want to "crush" each _sample_ from the float value space to only 2<sup>N</sup> distinct values, then modify those

E.g. <a href="https://stackoverflow.com/questions/17949796/how-many-unique-values-are-there-between-0-and-1-of-a-standard-float">1,065,353,215</a> -> 256

<aside class="notes">
							<li>This creates a <strong>bit crushing</strong> effect and will allows us to modify those "bits" later on.</li>
							<li>Disclaimer: This is the opposite of auto tune. The sound will get "worse", more distorted</li>
							<li>Now for bit depth as I said with sample rate our computers record the audio one step at a time at a very fast rate and each step switches on and off or in binary terms 1 and 0. Our computers do this for anything a computer does, even we talk about text, images or audio. These 1’s and 0’s are the bits and our computers can handle many strings of these bits at a time. As we all should know there are 8 bits in a byte and 16 bits in two bytes and so on, one or two bytes compose a word, for sixteen bits (two bytes) this means that there are 16 digits in a word for 24 bit 24 digits and so on._</li>
							<li>Taken from https://www.producerspot.com/review-decimort2-bit-crusher-plugin-by-d16-group :</li>
						</aside>

Part II

Implementing a bit crushing effect

### Presenting
            ## Web Audio API

![W3C Audio worklet events](images/audioworklet-instantiation-sequence.png)

Note:

* **AudioContext**: Pipes together **AudioNodes** and gives access to the **Web Audio API**
              * Reference to modular synthesizer: API just like that (Inputs <> Outputs)
              * Nodes must be connected. Multiple source s -> multiple effects -> one destination

* **Web Audio API**:
              * Good documentation, implementation close to the modular hardware

* **AudioWorklet Node** is part of the Main Thread

* **AudioWorklet** is used to supply custom audio processing scripts that execute in a separate thread to provide very low latency audio processing.
              * This was a breakthrough in 2018, as it first allowed active modification of concurrent audio streams

* **AudioWorklet Processor** is part of the WebAudio Render Thread

* Inter-Process **communication** via parameters (once per sample rate) or async port messages

* Legacy: single thread _ScriptProcessorNode_

* Code must be **fast**, we're having a real time app

---
            ## Base audio pipeline

![input output](https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API/audio-context_.png)

```html [1-5]
<button onclick="playDrumloop()">Connect Input</button>
<button onclick="pause()">Pause</button>
<button onclick="resume()">Resume</button>
```

```js
const audioContext = new AudioContext()
```

<aside class="notes">
	<li>We now focus on the <strong>left</strong> part of the MDN graphic and create a basic audio context playground</li>
	<li>Basics: Audio Context, user interaction, secure context, audioContext.state === 'running'</li>
</aside>

---

## Base audio pipeline: Audio Input

```js [1-2,18-19|3-10,15-17|11-14]
function playDrumloop() {
  const drumLoopStream = audioContext.createBufferSource()
  const request = new XMLHttpRequest()

request.open('GET', '/drumloop.wav', true)
  request.withCredentials = false
  request.responseType = 'arraybuffer'
  request.onload = () => {
    const audioData = request.response
    audioContext.decodeAudioData(audioData, function(buffer) {
      drumLoopStream.buffer = buffer
      drumLoopStream.playbackRate.value = 1.0
      drumLoopStream.loop = true
      drumLoopStream.start(0)
    })
  }
  request.send()
  drumLoopStream.connect(audioContext.destination)
}
```

---

## Base audio pipeline: Pausing the signal

```js [1-3|4-7]
  function pause() {
    audioContext.suspend()
  }

function resume() {
    audioContext.resume()
  }
```

---

## Base audio pipeline: Demo

---

## Repeat: Audio Worklet Scopes

![W3C Audio worklet events](images/audioworklet-instantiation-sequence.png)

Note:

* **AudioContext**: Pipes together **AudioNodes** and gives access to the **Web Audio API**
  * Reference to modular synthesizer: API just like that (Inputs <> Outputs)
  * Nodes must be connected. Multiple source s -> multiple effects -> one destination

* **Web Audio API**:
  * Good documentation, implementation close to the modular hardware

* **AudioWorklet Node** is part of the Main Thread

* **AudioWorklet** is used to supply custom audio processing scripts that execute in a separate thread to provide very low latency audio processing.
  * This was a breakthrough in 2018, as it first allowed active modification of concurrent audio streams

* **AudioWorklet Processor** is part of the WebAudio Render Thread

* Inter-Process **communication** via parameters (once per sample rate) or async port messages

* Legacy: single thread _ScriptProcessorNode_

* Code must be **fast**, we're having a real time app

---

## Bit crushing effect: <br> AudioWorkletNode

```html [1|2]
<button onclick="connectCustomProcessor()">Connect Processor</button>
<input type="range" min="1" max="8" oninput="changeBitCount(this)"/>
```

```js [2,13|3-7|8-11|1,12,15-17]
let processorBitCount
async function connectCustomProcessor() {
  await audioContext.audioWorklet.addModule(
    'bit-crusher-processor.js',
  )
  const bitCrusher = new AudioWorkletNode(audioContext,
    'bit-crusher-processor')
  drumLoopStream.disconnect(audioContext.destination)
  drumLoopStream
    .connect(bitCrusher)
    .connect(audioContext.destination)
  processorBitCount = bitCrusher.parameters.get('bits')
}

function changeBitCount(slider) {
  processorBitCount.value = slider.value
}
```

<aside class="notes">
	<aside class="notes">
		<li>We now shift over to the <strong>right</strong> part of the MDN graphic and create the custom filter that reduces the value space</li>
		<li>we need a representation of the bit crushing processor in our main process (AWNode) and separate process (module)</li>
		<li>there are two ways to communicate with a processor, this slide explains it with the `bits` parameter.</li>
</aside>

---

## Bit crushing effect: <br> AudioWorkletProcessor

```js [2,18|20|3-12|14-17]
// bit-crusher-processor.js
class BitCrusherProcessor extends AudioWorkletProcessor {
  static get parameterDescriptors() {
    return [
      {
        name: 'bits',
        defaultValue: 8,
        minValue: 1,
        maxValue: 8,
      },
    ]
  }

process(inputs, outputs, parameters) {
    this.totalBits = parameters.bits[0]
    // value modification is done on the next slide
  }
}

registerProcessor('bit-crusher-processor', BitCrusherProcessor)
```
<aside class="notes">
  The string for registerProcess (bit-crusher-processor) must match the string defined from the AudioWorkletNode
</aside>

---

## Bit crushing effect: <br> AudioWorkletProcessor
```js [1-2,18-19|3-10|12,16|13,15|14]
process(inputs, outputs, parameters) {
  this.totalBits = parameters.bits[0]
  // Example `inputs` and `outputs` values for the drumloop:
  // Array(connected nodes, e.g. a microphone)[
  //   Array(channels, e.g. stereo)[
  //    Float32Array(128 samples) [0.20704758167266846, 0.205214858055..]
  //    Float32Array(128 samples) [0.20704758167266846, 0.205214858055..]
  //   ]
  // ]
  // Each sample value is in range of [-1 .. 1]

inputs[0].forEach((inputChannel, channelIndex) => {
    inputChannel.forEach((channelSample, sampleIndex) => {
      outputs[0][channelIndex][sampleIndex] = this.crushBits(channelSample)
    })
  })

return true // keep running
}
```

---

## Bit crushing effect: <br> AudioWorkletProcessor
```js [1-5|7,15|8-9|10-12|13-14]
process(inputs, outputs, parameters) {
  // inputs[0].forEach ...
  outputs[channelIndex][sampleIndex] = this.crushBits(channelSample)
  // ... return true
}

crushBits(sampleValue) {
  // Determine how many *different* values we allow, e.g. 256 for 8 bits
  const possibleValuesCount = 2 ** this.totalBits 
  // Round each sample value to its closest value (zero-indexed)
  const sign = Math.sign(sampleValue)
  const intSample = Math.abs(Math.floor(sampleValue * possibleValuesCount))
  // Scale it back to [-1 .. 1]
  return (intSample * sign) / possibleValuesCount
}
```

---

## The plan: succeeded?
![bit crushing visualization](images/bitrate-to-samplerate.png)

We want to "crush" each _sample_ from the float value space to only 2<sup>N</sup> distinct values, then modify those

Note:

_Now for bit depth as I said with sample rate our computers record the audio one step at a time at a very fast rate and each step switches on and off or in binary terms 1 and 0. Our computers do this for anything a computer does, even we talk about text, images or audio. These 1’s and 0’s are the bits and our computers can handle many strings of these bits at a time. As we all should know there are 8 bits in a byte and 16 bits in two bytes and so on, one or two bytes compose a word, for sixteen bits (two bytes) this means that there are 16 digits in a word for 24 bit 24 digits and so on._

Taken from https://www.producerspot.com/review-decimort2-bit-crusher-plugin-by-d16-group :
---

## Demo time: Bit crushing effect

Part III

Part III: Actual bit flipping

## How to actually flip a bit

Example: Inverting the "fourth" bit of 55

```plain
       0000 0001 (1 in decimal)
    <<         4 (left shift 4 times)
-----------------
       0001 0000
     ^ 0011 0111 (55 in decimal)
-----------------
    => 0010 0111 (final result)
```

```js
let sample = 55 // 32 + 16 + 4 + 2 + 1
const bitmask = 1 << 4 // 16
sample ^= bitmask
sample === 39 // 55 - 16
```

---
## How to actually flip a bit

```js [1-8,20|9,10-12,18-19|13-17]
// AudioWorkletProcessor: previous code is omitted
process(inputs, outputs, parameters) {
  this.totalBits = parameters.bits[0]
  this.bitToFlip = parameters.flippedBit[0]
	outputs[channelIndex][sampleIndex] = this.flipBit(channelSample)
}

flipBit(sampleValue) {
  const possibleValuesCount = 2 ** this.totalBits
  // Round each sample value to its closest value (see previous example)
  const sign = Math.sign(sampleValue)
  let intSample = Math.abs(Math.floor(sampleValue * possibleValuesCount))

// Modify the int value by inverting/flipping the nth-bit
  const bitmask = 1 << (this.totalBits - this.bitToFlip)
  intSample ^= bitmask

// Scale it back to [-1 .. 1]
  return (intSample * sign) / this.totalBits
}
```

---
						## Bit flipping effect: Demo

Part IV

Revisiting Oto Biscuit

## Oto Biscuit

![oto biscuit](images/oto-biscuit.png)

---

## Showcase: Bitflipper

Note:

* Output device auf sof-hda-dsp
						* Input device auf Microphone USB Audio device
							* Handy mit Aux->USB anschließen + Spotify
						* Einzug der Gladiatoren
						  * Redux auf 25%
							* Cutoff 2000 Hz
							* Q 10
						* Links bits deaktivieren

<!--

## Bitflipper explained: Volume control

It reduces the float value of each sample

-->
						
						<!--

## Bitflipper explained: Float Range

Similar effect to bit crushing but not limited to <code>2^N</code>
						-->
						
						<!--

## Bitflipper explained: Tech stack

* No preprocessor, just one dependency (Unpoly).
            * Chainable with other devices, MIDI controllers etc.
						-->

---

## Sources

```plain
						https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API
						https://developer.mozilla.org/en-US/docs/Web/API/AudioWorklet
						https://docs.google.com/presentation/d/11OZyHyWRTWOCETW4x7m5gCPNSdSl9HevCQ1aiR_0xFE/htmlpresent
						https://en.wikipedia.org/wiki/Digital_audio#/media/File:4-bit-linear-PCM.svg
						https://www.otomachines.com/wp-content/uploads/2015/07/biscuit_cam02_V02.jpg
						https://i.kym-cdn.com/entries/icons/original/000/001/007/WAT.jpg
						https://www.w3.org/TR/webaudio/images/audioworklet-instantiation-sequence.png
						https://www.producerspot.com/wp-content/uploads/2016/01/decimort2-sample-bit-rates.jpg
						https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API/audio-context_.png
						https://s7d1.scene7.com/is/image/izotope/Reconstructing%20the%20original%20signal?fmt=webp-alpha&resMode=sharp2&wid=576&
						https://s7d1.scene7.com/is/image/izotope/Increasing%20bit-depth%20resolution?fmt=webp-alpha&resMode=sharp2&wid=590&
						https://upload.wikimedia.org/wikipedia/commons/f/f5/Byte45.png
						```

<small><a href="https://talk.bitflipper.leimstaedtner.it/">talk.bitflipper.leimstaedtner.it</a> ·</small>
						<small><a href="https://bitflipper.leimstaedtner.it/">bitflipper.leimstaedtner.it</a> ·</small>
						<small><a href="https://github.com/Ichaelus/bitflipper/">github.com/Ichaelus/bitflipper</a></small>

Thank you for your time

Bonus Slides

Listening to MIDI signals
Effect theory: Frequency Reduction & Filters
Implementing an oscilloscope
Tooling
Production readiness

## Bonus: Listening to MIDI signals

* A MIDI signal consists of a <code>status</code> and two <code>data</code> values
						* It might be signal to turn on/off a note, control code or similar.

```js [1|2,4|3,5-8]
						const midiAccess = await navigator.requestMIDIAccess({ sysex: true })
						for (const input of midiAccess.inputs) {
							input[1].onmidimessage = (e) => console.table(e.data, 'value');
						}
						// (index) | Value
						// 0       | 176
						// 1       | 4
						// 2       | 22
						```

---

## Bonus: MIDI debugging on linux

```sh [1-2|4-6|8-11]
						$ ls -ll /dev/snd/midi*
						# crw-rw----+ 1 root audio 116, 11 Jan  3 09:29 /dev/snd/midiC1D0
						
						$ aseqdump -l
						# Port    Client name                      Port name
						# 20:0    nanoKONTROL                      nanoKONTROL nanoKONTROL _ CTRL

$ aseqdump -p 20:0
						# Waiting for data. Press Ctrl+C to end.
						# Source  Event                  Ch  Data
						# 20:0   Control change          0, controller 2, value 125
						```

---

## Effect theory: Frequency Reduction

---

## Effect theory: Filter

Filters allow gradual removing of different frequencies:
            * **lowpass** ignores high frequencies
            * **highpass** ignores low frequencies
            * **bandpass** ignores frequencies around the <code>cutoff</code> value

The <code>Q</code> or <code>resonance</code> value adds a sinus wave at the cutoff value.

---

## Bonus: Implementing an oscilloscope
```js [1-4,21|6-9|10-11,20|12-19]
function connectOscilloscope() {
  const oscilloscope = audioContext.createAnalyser()
  drumLoopStream.connect(oscilloscope)
  const freqData = new Uint8Array(oscilloscope.frequencyBinCount)

const exampleOutput = document.querySelector('#firstFrequency')
  const minOutput = document.querySelector('#minFrequency')
  const maxOutput = document.querySelector('#maxFrequency')
  const averageOutput = document.querySelector('#averageFrequency')

setInterval(() => {
    oscilloscope.getByteTimeDomainData(freqData)
    const averageFrequency = freqData.reduce((f1, f2) => {
      return f1 + f2
    }, 0) / freqData.length
    exampleOutput.innerText = freqData[0]
    minOutput.innerText = Math.min(...freqData)
    maxOutput.innerText = Math.max(...freqData)
    averageOutput.innerText = averageFrequency
  }, 100)
}
```
						---

## Bonus slide: Tooling

* Chrome devtools: <code>WebAudio</code> tab (sample rate, audio context state,)
						* Firefox Devtools: Better but deprecated. Maybe part of Firefox developer edition.

---

## Bonus: Production readiness

* You need SSL, User permission, a click interaction and probably some input/output mangling
						* Good Browser support since 2021

How to flipan audio bit

Hello!

Overview

Part I

Understanding our goal

Part II

Implementing a bit crushing effect

Part III

Part III: Actual bit flipping

Part IV

Revisiting Oto Biscuit

Thank you for your time

Bonus Slides

How to flip
an audio bit