jamieonkeys.dev

Building an animated piano keyboard with JavaScript and MIDI

How I built a simple on-screen keyboard for my jazz piano tutorial website JazzKeys.fyi.

Posted by Jamie on .

I’m currently working on JazzKeys.fyi, a website of jazz piano tutorials. I wanted each musical example to include a simple on-screen keyboard that would animate as notes are being played.

There didn’t seem to a solution that matched quite what I was looking for, so I decided to roll my own, a demo of which you interact with below.

A working demo of the keyboard in action. Drag the vertical resizer bar to the right to reveal the (editable) code.

Approach

I wanted the solution to be no more complex than it needed to be, so I started with a piano keyboard SVG I found on Wikimedia Commons. I extended the graphic to cover all 88 keys, rounded the key edges, and added some gradients.

My idea was to have the program play an audio file and update the fill property of whichever <rect>s in the SVG represented the keys that were being played.

MIDI to JSON

MIDI was the obvious way to get the music (which I played and recorded in Logic) in the form of note-on, note-off and velocity data. I used Tone.js (‘a Web Audio framework for creating interactive music in the browser’) and Tone.js Midi to covert MIDI file data into a JSON format that could be understood by Tone.js.

When the user clicks ‘Play’, the MIDI events are scheduled by Tone.js; then, when the audio file has loaded, the audio and MIDI files are played simultaneously.

There isn’t a synth hooked up to the Tone.js instance, so the MIDI file is, in effect, silent. Using Tone.js’s Draw class, you can schedule code to run on each MIDI event, and I use this to add and remove note-on class names to the relevant UI elements.

Audio

I use Howler.js, a library built on the Web Audio API, to handle the playing of audio. Here it plays the examples in either M4A or WebM formats, depending on the browser.

UX

The colour red is used to show what the right hand is playing, and blue the left. The brighter the colour, the higher the velocity (i.e. the speed with which the key is being depressed).1

Additionally, a circle appears above each key as it’s played: a filled-in circle for the right hand, and a circle outline for the left. The circles have a ~1s fade-out to help orient the user. (By orient I mean that when a given note starts playing, there is an indication to the user as to which note or notes were played immediately before.)

The first couple of bars, in score form, of the funk groove used in this blog post. Explanatory annotations are in pink.

The tutorials feature a musical chart (score), often with explanatory annotations, for each example. The site is aimed mostly at players who already have at least some experience of playing piano, and such folk can probably already read Western music notation. To these users the animated keyboards may have limited value.

Even if you can read music, however, seeing the velocities indicated visually might he helpful if you’re looking to improve your command of the jazz or funk ‘feel’. For this funk groove, for example, the colours could help users see which notes are being ‘ghosted’.

Technical details

Playing audio when an iOS device is muted

On iOS, the Web Audio API can’t produce sound if the device is muted. This is probably a feature (rather than a bug) to prevent websites from playing sound when the user doesn’t expect it.

In the case of JazzKeys.fyi, users are explicitly tapping a ‘Play’ button and so very likely want to hear something even if their mute switch is active.

Unlike the Web Audio API, the HTML5 <audio> element plays sound no matter a device’s mute state, and so we can work around the no-sound-on-mute issue by simultaneously playing a silent audio file via an <audio> element.

Code structure

In a typical JazzKeys.fyi tutorial, there will be several musical examples. I structured this in code by creating the class ItemAudioState, from which a new object per example is instantiated on page load:

class ItemAudioState {
  constructor(
    name,
    buttonPlay,
    loop,
    tempo,
    filePath,
    midi,
    numberOfKeys
  ) {
    this.name = name; // e.g. lick-blues-1. Taken from `data-name` on .button-play
    this.buttonPlay = buttonPlay;
    this.loop = loop; // true/false
    this.tempo = tempo; // integer/undefined
    this.filePath = filePath; // e.g. /audio/lick-blues-1/lick-blues-1-straight-120 (file extensions added later)
    this.midi = midi; // Path to MIDI file
    this.numberOfKeys = numberOfKeys; // Number of keys on keyboard
  }
}

State management

Each instantiated instance of ItemAudioState is stored in the state object, in the itemsState property:

const state = {
  init: false,
  itemsState: {}, // Instantiated `ItemAudioState` objects
  activeItemState: undefined, // Reference to active item's object in `itemsState`. Active item == currently playing item (set on Play button click)
  audio: undefined, // Main howler object
  playing: false,
  loading: false,
  playingItemObjectName: undefined, // Name of the item currently being played
  playCount: 0, // Incremented after end of each play of main audio file
  updateAudio: false, // Was a relevant parameter (e.g. tempo or rhythm) changed during playing that requires the main audio file playing to be updated?
  isiOS: undefined, // true/false
  htmlAudio: undefined, // For iOS
  htmlAudioSetup: false, // Flag. Background HTML5 element created
  htmlAudioPlaying: false, // Flag. Background HTML5 has started playing
  tonejs: {
    itemsMidi: {}, // MIDI files as JS objects
    visuals: []
  },
  timeouts: {
    // Store timeouts in state object so they can be cleared (and we can prevent multiple instances thereof being fired). All timeouts in this object are cleared by playStopHowl()
    loading: undefined // If audio taking more than a brief moment to load, show 'loading' indicator
  }
};

Listeners are added to various UI elements, such as the ‘Loop’ switch and ‘Tempo’ dropdown menu. When there is a change in value to one of those UI elements, a matching property in the relevant object in state.itemsState is updated.

A listener is also added to the body to capture any change that‘s made in the UI. This updates the audio file path based on changes to tempo and rhythm (whose values themselves are updated by the element-specific listeners described in the previous paragraph). It also captures changes to elements added dynamically, such as the ‘Bar dimming’ switch2.

Querying the state

The state can be updated at any time, including while audio is being played and the keyboard is animating. The updated state will then be queried at the relevant point in the code, such as when the program decides whether the audio should loop and/or load new audio.

The state management also handles the scenario in which the user interrupts the playing of one example by clicking ‘Play’ on another example. Here, the object in state.itemsState which corresponds to the newly activated example will become the source of what audio is to play and what keyboard is to animate.

HTML data attributes

The id and various data attributes in the HTML associate the ‘Play’ button with its accompanying UI elements:

<div class="flex controls">
  <button class="button button-play" data-name="figure-bebop-6" data-tempo="#tempo6" data-chord_name="#chord6" data-loop="#loop6" data-annotations="#annotations6" data-midi="88|49">Play</button>
</div>
<div class="flex controls">
  <div>
    <label for="tempo6">Tempo (speed)</label>
    <select id="tempo6" class="select tempo" data-name="figure-bebop-6">
      <option value="140">140bpm (1x)</option>
      <option value="105">105bpm (0.75x)</option>
      <option value="70">70bpm (0.5x)</option>
    </select>
  </div>
  <div>
    <label for="chord6">Chords</label>
    <select id="chord6" class="select chord-name" data-name="figure-bebop-6">
      <option value="">None</option>
      <option value="C7|F-rh-140">C7 | Fmaj7</option>
      <option value="Gm7-C7|F-rh-140">Gm7 C7 | Fmaj7</option>
    </select>
  </div>
  <div>
    <label class="form-switch">
      Loop
      <input type="checkbox" class="loop" id="loop6" data-name="figure-bebop-6">
      <i></i>
    </label>
  </div>
  <div>
    <label class="form-switch">
      Annotations
      <input type="checkbox" class="annotations" id="annotations6" checked data-name="figure-bebop-6">
      <i></i>
    </label>
  </div>
</div>

That all elements for each example share the same name data attribute allows the JavaScript to target the relevant object in state.3

Get notified on launch

And there we have it: a simple animated piano keyboard built in JavaScript. You can view the demo and edit its code at this CodeSandbox.

If you’d like to be notified when JazzKeys.fyi launches, feel free to enter your email.

  1. Different hues of the same colour alone may not be enough to make velocity clear to the user. I’m considering adding a line similar to that featured in Logic Pro’s piano roll editor. Logic also uses colour to describe velocity, but following the Logic approach of using different colours would, in my case, confuse which hand is playing. I’ve also experimented with modifying the radius of the circles above the keys to indicate velocity. I may finesse this feature and activate it in the future. ↩︎

  2. The ‘bar dimming’ feature highlights the currently playing bar in the musical score to help the user follow along. ↩︎

  3. data-midi="88|49" means that a 49-key version of the SVG keyboard can be shown on narrower devices (such as phones) if the example in question contains no notes outwith the 49-key range. This improves the viewing experience for mobile users. ↩︎