# Simple experiments with speech detection

## Tags: programming, projects

« Previous post: Volume rendering for DICOM images — Next post: Creating heat maps of git commits »One of my pet projects involves detecting *speech* in audio signals. I am not interested in actual
*voice recognition* but rather want to detect whether a signal is *probably* a person speaking or
something else. Since this is a pet project and I have no knowledge of digital signal processing, I
wanted to discover things on my own. This posts contains my very preliminary results.

# Processing audio signals with Python

For simplicity, I only tackle uncompressed `WAV`

files for now. SciPy is my
friend here:

```
data = scipy.io.wavfile.read( "Test.wav" )[1]
```

`data`

now contains the amplitude information of the signal. To make any calculations on the signal
more robust, I chunk the signal without overlaps into portions of equal length, which I will refer
to as *frames*. Each frame describes a short amount (say 10ms) of audio. This makes it possible to
calculate statistics per frame and obtain distributions of said statistics for the complete
recording. For proper signal processing, I should probably use some sort of windowing approach.

# Time-domain statistics

For this post, I shall only use some features in the *time domain* of the signal. I am well aware
that Fourier analysis exists, but my knowledge about these techniques is too sparse at the
moment—I know about high- and low-pass filters and the theory behind algorithms such as FFT,
but that’s about it.

By browsing some books about audio analysis, I stumbled over some statistics that seemed worthwhile (because they are easy to implement):

- Short-term energy
- Zero-crossing rate
- Entropy of energy

I only have a cursory understanding of them (for now) so I hope that I got the terminology right.

## Short-term energy

The short-term energy of a frame is defined as the sum of the squared absolute values of the amplitudes, normalized by the frame length:

```
def shortTermEnergy(frame):
return sum( [ abs(x)**2 for x in frame ] ) / len(frame)
```

## Zero-crossing rate

The zero-crossing rate counts how often the amplitude values of the frame change their sign. This value is then normalized by the length of the frame:

```
def zeroCrossingRate(frame):
signs = numpy.sign(frame)
signs[signs == 0] = -1
return len(numpy.where(numpy.diff(signs))[0])/len(frame)
```

## Entropy of energy

By further subdividing each frame into a set of sub-frames, we can calculate their respective
short-term energies and treat them as *probabilities*, thus permitting us to calculate their
entropy:

```
def chunks(l, k):
for i in range(0, len(l), k):
yield l[i:i+k]
def entropyOfEnergy(frame, numSubFrames):
lenSubFrame = int(numpy.floor(len(frame) / numSubFrames))
shortFrames = list(chunks(frame, lenSubFrame))
energy = [ shortTermEnergy(s) for s in shortFrames ]
totalEnergy = sum(energy)
energy = [ e / totalEnergy for e in energy ]
entropy = 0.0
for e in energy:
if e != 0:
entropy = entropy - e * numpy.log2(e)
return entropy
```

# Combining everything

My brief literature survey suggested that these three statistics can be rather simply used to determine speech. More precisely, we need to take a look at the following quantities:

- The
*coefficient of variation*of the short-term energy. Speech tends to have higher values here than non-speech. Some experimentation with a few speech/music files shows that a threshold of`1.0`

is OK for discriminating between both classes. - The standard deviation of the zero-crossing rate. Again, speech tends to have higher values here.
A threshold of
`0.05`

seems to work. - The minimum entropy of energy. This is where speech usually has
*lower*values than music. Also, different genres of music seem to exhibit different distributions here—this seems pretty cool to me. Here, I used a threshold of`2.5`

to decide whether an audio file contains speech.

This is in no way backed by anything other than me trying and fiddling with some audio files. I hope this is interesting and useful to someone. I also hope that I will find the time to do more with it! Thus, no special git repository for this one (at least for now), but rather the complete source code. Enjoy:

```
#!/usr/bin/env python3
#
# Released under the HOT-BEVERAGE-OF-MY-CHOICE LICENSE: Bastian Rieck wrote
# this script. As long you retain this notice, you can do whatever you want
# with it. If we meet some day, and you feel like it, you can buy me a hot
# beverage of my choice in return.
import numpy
import scipy.io.wavfile
import scipy.stats
import sys
def chunks(l, k):
"""
Yields chunks of size k from a given list.
"""
for i in range(0, len(l), k):
yield l[i:i+k]
def shortTermEnergy(frame):
"""
Calculates the short-term energy of an audio frame. The energy value is
normalized using the length of the frame to make it independent of said
quantity.
"""
return sum( [ abs(x)**2 for x in frame ] ) / len(frame)
def rateSampleByVariation(chunks):
"""
Rates an audio sample using the coefficient of variation of its short-term
energy.
"""
energy = [ shortTermEnergy(chunk) for chunk in chunks ]
return scipy.stats.variation(energy)
def zeroCrossingRate(frame):
"""
Calculates the zero-crossing rate of an audio frame.
"""
signs = numpy.sign(frame)
signs[signs == 0] = -1
return len(numpy.where(numpy.diff(signs))[0])/len(frame)
def rateSampleByCrossingRate(chunks):
"""
Rates an audio sample using the standard deviation of its zero-crossing rate.
"""
zcr = [ zeroCrossingRate(chunk) for chunk in chunks ]
return numpy.std(zcr)
def entropyOfEnergy(frame, numSubFrames):
"""
Calculates the entropy of energy of an audio frame. For this, the frame is
partitioned into a number of sub-frames.
"""
lenSubFrame = int(numpy.floor(len(frame) / numSubFrames))
shortFrames = list(chunks(frame, lenSubFrame))
energy = [ shortTermEnergy(s) for s in shortFrames ]
totalEnergy = sum(energy)
energy = [ e / totalEnergy for e in energy ]
entropy = 0.0
for e in energy:
if e != 0:
entropy = entropy - e * numpy.log2(e)
return entropy
def rateSampleByEntropy(chunks):
"""
Rates an audio sample using its minimum entropy.
"""
entropy = [ entropyOfEnergy(chunk, 20) for chunk in chunks ]
return numpy.min(entropy)
#
# main
#
# Frame size in ms. Will use this quantity to collate the raw samples
# accordingly.
frameSizeInMs = 0.01
frequency = 44100 # Frequency of the input data
numSamplesPerFrame = int(frequency * frameSizeInMs)
data = scipy.io.wavfile.read( sys.argv[1] )
chunkedData = list(chunks(list(data[1]), numSamplesPerFrame))
variation = rateSampleByVariation(chunkedData)
zcr = rateSampleByCrossingRate(chunkedData)
entropy = rateSampleByEntropy(chunkedData)
print("Coefficient of variation = %f\n"
"Standard deviation of ZCR = %f\n"
"Minimum entropy = %f" % (variation, zcr, entropy) )
if variation >= 1.0:
print("Coefficient of variation suggests that the sample contains speech")
if zcr >= 0.05:
print("Standard deviation of ZCR suggests that the sample contains speech")
if entropy < 2.5:
print("Minimum entropy suggests that the sample contains speech")
```