Skip to main content

PianoSheetia: Converting Synthesia Piano Videos to MIDI with Computer Vision

· 2 min read
Yi-Chen Zhang
Yi-Chen Zhang
Lead Engineer, AI and Autonomous

Over the past few weeks, I've been working on a Python tool that converts Synthesia-style piano videos into MIDI files using computer vision — and I'm making it public today.

The Motivation​

The idea came from a Synthesia piano video I found on YouTube that I really wanted to learn and play. Unfortunately, after searching everywhere, I couldn't find the corresponding sheet music.

Disclaimer: I would gladly pay for the sheet music if it were available, but it simply doesn't seem to exist.

I tried several existing Python packages for this purpose, but none worked reliably — some failed to detect the keyboard, others produced incorrect notes or tempos, and some couldn't even download the YouTube videos. So I decided to build my own solution.

Introducing PianoSheetia​

The name combines Synthesia + Piano + Music Sheet. The tool uses computer vision to:

  • Detect the piano keyboard in a Synthesia-style video
  • Track which keys are pressed frame by frame
  • Map key presses to correct notes and timing
  • Output a valid MIDI file

The repository is now public — feel free to try it out and contribute:

👉 github.com/Chris7462/piano-sheetia

Limitations of MIDI-to-Sheet Conversion​

Raw MIDI data only contains note on/off events with timing and velocity. It does not contain musical interpretation such as phrasing, pedaling, or proper voice leading that makes sheet music readable. For best results, you'll need to manually edit the exported MIDI after importing it into notation software to add proper musical conventions.

Note: If you enjoy the music, please support the composer by purchasing the official sheet music whenever it is available.