Attacking Audio "reCaptcha" using Google's Web Speech API

I had a fun project months back, Where I had to deal with digital signal processing and low level audio processing. I was never interested in DSP and all other control system stuffs, But when question arises about breaking things, every thing becomes interesting :) . In this post i'm going to share one technique to fully/ partially bypass reCaptcha test. This is not actually a vulnerability but its better if we call it "Abuse of functionality".

Disclaimer : Please remember this information is for Educational Purpose only and should not be used for malicious purpose. I will not assume any liability or responsibility to any person or entity with respect to loss or damages incurred from information contained in this article.

1. What is Captcha

A CAPTCHA is a program that protects websites against bots by generating and grading tests that humans can pass but current computer programs cannot. The term CAPTCHA (for Completely Automated Public Turing Test To Tell Computers and Humans Apart) was coined in 2000 by Luis von Ahn, Manuel Blum, Nicholas Hopper and John Langford of Carnegie Mellon University.


2. What is Re-captcha

reCAPTCHA is a free CAPTCHA service by Google, that helps to digitize books, newspapers and old time radio shows. More details can be found here.


3. Audio reCaptcha

reCAPTCHA also comes with an audio test to ensure that blind users can freely navigate.

4. Main Idea: Attacking Audio reCaptcha using Google's Web Speech API Service





5. Google Web Speech API

Chrome has a really interesting new feature for HTML5 speech input API. Using this user can talk to computer using microphone and Chrome will interpret it. This feature is also available for Android devices. If you are not aware of this feature you can find a live demo here.

https://www.google.com/intl/en/chrome/demos/speech.html

I was always very curious about the Speech recognition API of chrome. I tried sniff the api/voice traffic using Wireshirk but this API uses SSL. :(.

So finally I started browsing the Chromium source code repo. Finally I found exactly what I wanted.

http://src.chromium.org/viewvc/chrome/trunk/src/content/browser/speech/

It pretty simple, First the audio is collected from the mic, and then it posts it to Google web service, which responds with a JSON object with the results.  The URL which handles the request is :

https://www.google.com/speech-api/v1/recognize

Another important thing is this api only accepts flac audio format.

6. Programatically Accessing Google Web Speech API(Python)

Below python script was written to send a flac audio file to Google Web Speech API and print out the JSON response.

./google_speech.py hello.flac


'''
Accessing Google Web Speech API using Pyhon
Author : Debasish Mandal

'''

import httplib
import sys

print '[+] Sending clean file to Google voice API'
f = open(sys.argv[1])
data = f.read()
f.close()
google_speech = httplib.HTTPConnection('www.google.com')
google_speech.request('POST','/speech-api/v1/recognize?xjerr=1&client=chromium&lang=en-US',data,{'Content-type': 'audio/x-flac; rate=16000'})
print google_speech.getresponse().read()
google_speech.close()



7. Thoughts on complexity of reCaptcha Audio Challenges

While dealing with audio reCaptcha, you may know , it basically gives two types of audio challenges. One is pretty clean and simple (Example : https://dl.dropboxusercontent.com/u/107519001/easy.wav) . percentage of noise is very less in this type of challenges. 

Another one is very very noisy and its very difficult for even human to guess (Example : https://dl.dropboxusercontent.com/u/107519001/difficult.wav). Constant hissss noise and overlapping voice makes it really difficult to crack human. You may wanna read this discussion on complexity of audio reCapctha.

In this post I will mainly cover the technique / tricks to solve the easier one using Google Speech API. Although I've tried several approaches to solve the complex one, but as I've already said, its very very had to guess digits even for human :( .

8. Cracking the Easy Captcha Manually Using Audacity and Google Speech API

Google Re-captcha allows user to download audio challenges in mp3 format. And Google web speech API accepts audio in flac format. So if we just normally convert the mp3 audio challenge to flac format of frame rate 16000 its does not work :( .  Google Chrome Speech to text api does not respond to this sound.

But after some experiment and head scratching, it was found that we can actually make Google web speech api convert the easy captcha challenge to text for us, if we can process the audio challenge little bit. In this section i will show how this audio manipulation can be done using Audacity.

To manually verify that first I'm going to use a tool called Audacity to do necessary changes to the downloaded mp3 file. 

Step 1: Download the challenge as mp3 file.
Step 2: Open the challenge audio in Audacity.



Step 3: Copy the first digit speaking sound from main window and paste it in a new window. So here we will only have a one digit speaking sound.

Step 4: From effect option make it repetitive once. (Now It should speak the same digit twice).

Lets say for example if the main challenge is  7 6 2 4 6, Now we have only first digit challenge in wav format which having the digit 7 twice.





Step 5: Export the updated audio in WAV format.
Step 6: Now convert the wav file to flac format using sox tool and send it to Google speech server using the python script posted in section 6. And we will see something like this.

Note: In some cases little bit amplification might be required if voice strength is too low.

debasish@debasish ~/Desktop/audio/heart attack/final $ sox cut_0.wav -r 16000 -b 16 -c 1 cut_0.flac lowpass -2 2500
debasish@debasish ~/Desktop/audio/heart attack/final $ python send.py cut_0.flac 


Great! As you can see first digit of the audio challenge has been resolved by Google Speech. :) :) :) Now in same manner we can solve the entire challenge. In next section we will automate the same thing using python and it's wave module. 

9. Automation using Python and it's WAVE Module

Before we jump into processing of raw WAV audio using low level python API, its important to have some idea of how digital audio actually works. In above process we've extracted the most louder voices using audacity but to do it automatically using python, we must have some understanding of how digital audio is actually represented in numbers.

9.1. How is audio represented with numbers

There is an excellent stackoverflow post which explains the same. In short ,we can say audio is nothing but a vibration. Typically, when we're talking about vibrations of air between approximately 20Hz and 20,000Hz. Which means the air is moving back and forth 20 to 20,000 times per second. If somehow we can measure that vibration and convert it to an electrical signal using a microphone, we'll get an electrical signal with the voltage varying in the same waveform as the sound. In our pure-tone hypothetical, that waveform will match that of the sine function.

Now, we have an analogue signal, the voltage. Still not digital. But, we know this voltage varies between (for example) -1V and +1V. We can, of course, attach a volt meter to the wires and read the voltage.  Arbitrarily, we'll change the scale on our volt meter. We'll multiple the volts by 32767. It now calls -1V -32767 and +1V 32767. Oh, and it'll round to the nearest integer.

Now after having a set of signed integers we can easily draw an waveform using the data sets.

X axis -> Time
Y axis -> Amplitude (signed integers)



Now, if we attach our volt meter to a computer, and instruct the computer to read the meter 44,100 times per second. Add a second volt meter (for the other stereo channel), and we now have the data that goes on an audio CD. This format is called stereo 44,100 Hz, 16-bit linear PCM. And it really is just a bunch of voltage measurements.

9.2. WAVE File Format walk through using Python

As an example lets open up a very small wav file with a hex editor.

  

9.3. Parsing the same WAV file using Python

The wave module provides a convenient interface to the WAV sound format. It does not support compression/decompression, but it does support mono/stereo. Now we are going to parse the same wav file using python wave module and try to relate what we have just seen in hex editor.

Let's write a python script:

import wave 
f = wave.open('sample.wav', 'r') 
print '[+] WAV parameters ',f.getparams() 
print '[+] No. of Frames ',f.getnframes() 
for i in range(f.getnframes()): 
    single_frame = f.readframes(1) 
    print single_frame.encode('hex') 
f.close()

Line 1 imports python wav module.
Line 2: Opens up the sample.wav file.
Line 3: getparams() routine returns a tuple (nchannels, sampwidth, framerate, nframes, comptype, compname), equivalent to output of the get*() methods.
Line 4: getnframes() returns number of audio frames.
Line 5,6,7: Now we are iterating through all the frames present in the sample.wav file and printing them one by one.
Line 8: Closes the opened file

Now if we run the script we will find something like this:

[+] WAV parameters (1, 2, 44100, 937, 'NONE', 'not compressed')
[+] No. of Frames 937
[+] Sample 0 = 62fe    <- Sample 1
[+] Sample 1 = 99fe   <- Sample 2
[+] Sample 2 = c1ff    <- Sample 3
[+] Sample 3 = 9000
[+] Sample 4 = 8700
[+] Sample 5 = b9ff
[+] Sample 6 = 5cfe
[+] Sample 7 = 35fd
[+] Sample 8 = b1fc
[+] Sample 9 = f5fc
[+] Sample 10 = 9afd
[+] Sample 11 = 3cfe
[+] Sample 12 = 83fe
[+] ....
and so on,

It should make sense now. In first line we get number of channels, sample width , frame/sample rate,total number of frames etc etc. Which is exact same what we saw in the hex editor (Section 9.2). From second line it stars printing the frames/sample which is also same as what we have seen in hex editor. Each channel is 2 bytes long because the audio is 16 bit. Each channel will only be one byte. We can use the getsampwidth() method to determine this. Also, getchannels() will determine if its mono or stereo.

Now its time to decode each and every frames of that file. They're actually little-endian. So we will now modify the python script little bit so that we can get the exact value of each frame. We can use python struct module to decode the frame values to signed integers.

import wave 
import struct 

f = wave.open('sample.wav', 'r') 
print '[+] WAV parameters ',f.getparams() 
print '[+] No. of Frames ',f.getnframes() 
for i in range(f.getnframes()): 
    single_frame = f.readframes(1) 
    sint = struct.unpack('<h', single_frame) [0]
    print "[+] Sample ",i," = ",single_frame.encode('hex')," -> ",sint[0] 
f.close()

This script will print something like this:

[+] WAV parameters (1, 2, 44100, 937, 'NONE', 'not compressed')
[+] No. of Frames 937
[+] Sample 0 = 62fe -> -414
[+] Sample 1 = 99fe -> -359
[+] Sample 2 = c1ff -> -63
[+] Sample 3 = 9000 -> 144
[+] Sample 4 = 8700 -> 135
[+] Sample 5 = b9ff -> -71
[+] Sample 6 = 5cfe -> -420
[+] Sample 7 = 35fd -> -715
[+] Sample 8 = b1fc -> -847
[+] Sample 9 = f5fc -> -779
[+] Sample 10 = 9afd -> -614
[+] Sample 11 = 3cfe -> -452
[+] Sample 12 = 83fe -> -381
[+] Sample 13 = 52fe -> -430
[+] Sample 14 = e2fd -> -542

Now what we can see we have a set of positive and negative integers. Now you should be able to connect the dots. What I have explained in section 9.1. 

So now if we plot the same positive and negative values in a graph will find complete wave form. Lets do it using python matlab module.

import wave 
import struct 
import matplotlib.pyplot as plt 

data_set = [] 
f = wave.open('sample.wav', 'r') 
print '[+] WAV parameters ',f.getparams() 
print '[+] No. of Frames ',f.getnframes() 
for i in range(f.getnframes()): 
    single_frame = f.readframes(1)
    sint = struct.unpack('<h', single_frame)[0]
    data_set.append(sint) 
f.close() 
plt.plot(data_set) 
plt.ylabel('Amplitude')
plt.xlabel('Time') 
plt.show()

This should form following graph

Now you must be familiar with this type of graph. This is what you see in SoundCloud, But definitely more complex one.

So now we have clear understanding of how audio represented in numbers. Now it will be easier for readers to understand how the python script ( shared in section 9.3 ) actually works.

9.3. Python Script

In this section we will develop a script which automate the steps we did using Audacity in Section 8. Below python script will try extract loud voices from input wav file and generate separate wav files.



Once the main challenge is broken into parts we can easily convert it to flac format and send each parts of the challenge to Google speech API using the Python script shared in section 6.

9.4. Demo:



10. Attempt to Crack the Difficult(noisy) audio challenge

So we have successfully broken down the easy challenge.Now its time to give the difficult one a try. So I started with one noisy captcha challenge. You can see the matlab plot of the same noisy audio challenge below.

In above figure we can understand presence of a constant hisss noise. One of the standard ways to analyze sound is to look at the frequencies that are present in a sample. The standard way of doing that is with a discrete Fourier transform using the fast Fourier transform or FFT algorithm. What these basically in this case is to take a sound signal isolate the frequencies of sine waves that make up that sound.

10.1. Signal Filtering using Fourier Transform

Lets get started with a  simple example. Consider a signal consisting of a single sine wave, s(t)=sin(w∗t). Let the signal be subject to white noise which is added in during measurement, Smeasured(t)=s(t)+n. Let F be the Fourier transform of S. Now by setting the value of F to zero for frequencies above and below w, the noise can be reduced. Let Ffiltered be the filtered Fourier transform. Taking the inverse Fourier transform of Ffiltered yields Sfiltered(t). 

The way to filter that sound is to set the amplitudes of the fft values around X Hz to 0. In addition to filtering this peak, It's better to remove the frequencies below the human hearing range and above the normal human voice range. Then we recreate the original signal via an inverse FFT.

I have written couple of scripts which successfully removes the constant hiss noise from the audio file but main challenge is the overlapping voice. Over lapping voice makes it very very difficult even for human to guess digits. Although I was not able to successfully crack any of given difficult challenges using Google Speech API still I've shared few noise removal scrips (using Fourier Transform). 

These scripts can be found in the GitHub project page. There is tons of room for improvement of all this scripts.

11. Code Download

Every code I've written during this project is hosted here:  

12. Conclusion

When I reported this issue to Google security team, they've confirmed that, this mechanism is working as intended. The more difficult audio patterns are only triggered only when abuse/non-human interaction is suspected. So as per the email communication noting is going to be changed to stop this.

Thanks for reading. I hope you have enjoyed. Please drop me an email/comment in case of any doubt and confusion.

13. References

http://rsmith.home.xs4all.nl/miscellaneous/filtering-a-sound-recording.html
http://www.topherlee.com/software/pcm-tut-wavformat.html
http://exnumerus.blogspot.in/2011/12/how-to-remove-noise-from-signal-using.html
http://www.swharden.com/blog/2009-01-21-signal-filtering-with-python/

Comments

  1. Replies
    1. This is sound that seems like it was done in an expert account studio, and nobody needs to realize that you've done it on your PC. علی خدابنده

      Delete
  2. This is sound that seems like it was done in an expert account studio, and nobody needs to realize that you've done it on your PC. audacity for mac

    ReplyDelete
  3. Customers find the audio, find value in the tip, and hit the feed button. Each time the audio is updated, comfortable closed back headphones they are alerted, reminding them to re-visit the site for more great information.

    ReplyDelete
  4. Machine Learning is a practice of studying algorithms and statistics and training the computer to perform a specific task for the recognition of specific data. data science course syllabus

    ReplyDelete
  5. Great article Lot's of information to Read...Great Man Keep Posting and update to People..Thanks trafficize reviews

    ReplyDelete
  6. This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea.

    data science course in India

    ReplyDelete
  7. Took me time to read all the comments, but I really enjoyed the article. It proved to be Very helpful to me and I am sure to all the commenters here! It’s always nice when you can not only be informed, but also entertained! technocom email extractor pro download crack

    ReplyDelete
  8. Everything has its value. Thanks for sharing this informative information with us. GOOD works! how to send a mail merge in gmail

    ReplyDelete
  9. I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!
    Artificial Intelligence Course

    ReplyDelete
  10. I don t have the time at the moment to fully read your site but I have bookmarked it and also add your RSS feeds. I will be back in a day or two. thanks for a great site.

    Best Course for Digital Marketing In Hyderabad

    ReplyDelete
  11. This is a wonderful article, Given so much info in it, Thanks for sharing. CodeGnan offers courses in new technologies and makes sure students understand the flow of work from each and every perspective in a Real-Time environment AR Training in Vijayawada. ,

    ReplyDelete
  12. Seo company in Varanasi, India : Best SEO Companies in Varanasi, India: Hire Kashi Digital Agency, best SEO Agency in varanasi, india, who Can Boost Your SEO Ranking, guaranteed SEO Services; Free SEO Analysis.

    Best Website Designing company in Varanasi, India : Web Design Companies in varanasi We design amazing website designing, development and maintenance services running from start-ups to the huge players


    Wordpress Development Company Varanasi, India : Wordpress development Company In varanasi, india: Kashi Digital Agency is one of the Best wordpress developer companies in varanasi, india. Ranked among the Top website designing agencies in varanasi, india. wordpress website designing Company.

    E-commerce Website designing company varanasi, India : Ecommerce website designing company in Varanasi, India: Kashi Digital Agency is one of the Best Shopping Ecommerce website designing agency in Varanasi, India, which provides you the right services.

    ReplyDelete
  13. Fabulous post, you have denoted out some fantastic points, I likewise think this s a very wonderful website. I will visit again for more quality contents and also, recommend this site to all. Thanks. dotcomsecrets

    ReplyDelete
  14. Audio books have advantages that have hardly been exploited in education, training and many areas of self-help, as a result of the tyranny of the printed word. Let's end that tyranny NOW! Best Car Audio

    ReplyDelete
  15. Nice blog here! Also your website loads up very fast! What web host are you using? Can I get your affiliate link to your host? I wish my website loaded up as quickly as yours lol Mega888 game client download

    ReplyDelete
  16. We are well established IT and outsourcing firm working in the market since 2013. We are providing training to the people ,
    like- Web Design , Graphics Design , SEO, CPA Marketing & YouTube Marketing.Call us Now whatsapp: +(88) 01537587949
    :Freelancing training in Bangladesh
    Free bangla sex video:careful
    good post outsourcing institute in bangladesh

    ReplyDelete
  17. Get the digital marketing services in hyderabad digital marketing agency in hyderabad and adevertising agency in hyderabad solutions you need. Our online marketing services include PPC, SEO, social, and more". https://goviralll.com/packages.html

    ReplyDelete
  18. Thank you very much for this useful article. I like it. buy google reviews

    ReplyDelete
  19. The great content I have seen will be very useful to us, I am grateful.

    Our Service Here:
    Buy Google Business Reviews

    ReplyDelete
  20. Great job for publishing such a nice article. Your article isn’t only useful but it is additionally really informative. Thank you because you have been willing to share information with us. This is an extraordinary motivating article.I am basically satisfied with your great work.You put truly exceptionally accommodating data
    gmbboostup

    ReplyDelete
  21. This is really interesting, You’re a very skilled blogger. I have joined your feed and look forward to seeking more of your great post. Also, I’ve shared your site in my social networks!
    bizodesk

    ReplyDelete
  22. learn digital marketing https://www.digitalbrolly.com/digital-marketing-course-in-hyderabad/

    ReplyDelete
  23. Nice Blog. Learn Digital Marketing Course & digital marketing video course
    at Digital Brolly

    ReplyDelete
  24. Earn Money Online
    Enroll in our Affiliate Marketing course in Hyderabad to learn how to earn money online by becoming an affiliate.

    Live Money Making Proof’s
    We will show you the live accounts that are making money for us and help you replicate the same.
    Affiliate Marketing Course in Hyderabad

    ReplyDelete
  25. Amazing Article ! I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.


    wm casino

    คลิปโป๊

    คลิปxxx

    คลิปโป๊ญี่ปุ่น

    คลิปโป้ไทย

    เรียนภาษาอังกฤษ

    poker online

    ReplyDelete
  26. nice blog
    Our Digital Marketing course in Hyderabad focuses on Making you employeable.

    We make sure you have the right skill to get a job in Digital Marketing.
    digital marketing course in hyderabad

    ReplyDelete
  27. Watch movies online sa-movie.com, watch new movies, series Netflix HD 4K, ดูหนังออนไลน์ watch free movies on your mobile phone, Tablet, watch movies on the web.

    SEE4K Watch movies, watch movies, free series, load without interruption, sharp images in HD FullHD 4k, all matters, ดูหนังใหม่ all tastes, see anywhere, anytime, on mobile phones, tablets, computers.

    GangManga read manga, read manga, read manga online for free, fast loading, clear images in HD quality, all titles, อ่านการ์ตูน anywhere, anytime, on mobile, tablet, computer.

    Watch live football live24th, watch football online, ผลบอลสด a link to watch live football, watch football for free.

    ReplyDelete
  28. SamudraBet Situs Judi Online Terbaik, Terpercaya dan Terbesar di Indonesia. Menyediakan permainan :

    Judi bola terbaik

    Judi slot online terpercaya

    Poker online terbaru

    Live casino online

    Judi sabung ayam

    ReplyDelete
  29. I'm glad I found this web site, I couldn't find any knowledge on this matter prior to.Also operate a site and if you are ever interested in doing some visitor writing for me if possible feel free to let me know, im always look for people to check out my web site. highqualitypvas

    ReplyDelete
  30. You’re so interesting! I don’t believe I’ve truly read something like this before. So great to find someone with genuine thoughts on this issue. Really.. many thanks for starting this up. This website is something that’s needed on the internet, someone with some originality!

    CBSE Schools In Thane
    CBSE Schools In Raigad
    CBSE Schools In Ratnagiri
    CBSE Schools In Sangli
    CBSE Schools In Satara
    CBSE Schools In Sindhudurg
    CBSE Schools In Wardha
    CBSE Schools In Washim
    CBSE Schools In Ahmednagar
    CBSE Schools In Akola

    ReplyDelete
  31. เราคือผู้นำด้านเกมพนันออนไลน์ Major168 เราคือผู้ให้บริการ คาสิโนออนไลน์ ที่ได้รับรองว่าดีที่สุดในประเทศไทย มีค่ายเกมส์ให้เล่นมากมาย Sagaming, Sexy bacarat, Dreamgame, Ebet, Wm casino, Vivo gaming ไม่ผ่านเอเย่นต์ ระบบปลอดภัยมีทีมงานดูแลตลอด 24ชม.

    SAGAME88 แหล่งรวมเกมส์พนันออนไลน์ คาสิโนสด บาคาร่า กำถั่ว คาสิโนออนไลน์ ไฮโล รูเล็ต รับเครดิตฟรีเล่นได้ทุกเกมส์ โบนัสสมาชิกใหม่เพียบ พร้อมระบบฝากถอนออโต้ 10วิ เรามีทุกค่ายเกมส์ให้คุณเลือกเดิมพัน SA Game Sexy bacarat Dreamgame WM Casino VIVO Gaming Ebet เล่นได้ทุกเกมส์

    Our website ufabet provides betting services in the system of Auto Deposit-Withdrawal. Our members are not only in Thailand. Online football betting UEFA Bet market And this makes a guarantee that Ufabet168 is another reliable football betting website

    ufa through the website UFABET1688 a web gambling online , one that integrated all the bets from online casinos , online casino , online , and also have a game a lot more to be chosen to play such games.

    ReplyDelete
  32. This is the best blog annualeventpost I ever see. Thanks for sharing information with us.

    ReplyDelete
  33. Buying Google Reviews can be a powerful way to promote your website or blog. You might think that having the voice of your company heard on Google is a good way to build up your brand, but unfortunately, that goodwill will not come cheap. On average, the price of a Google search range from $35 - $70, while the price varies considerably for different companies. It is therefore important to understand just what you can expect when you Buy Google Reviews USA, how to use them in your campaign, and where to buy them from, to make sure you get the best value for money.

    ReplyDelete
  34. Nice Blog !
    Our team at QuickBooks Customer Service are capable of handling all the troubles of QuickBooks in less time.

    ReplyDelete
  35. It helps us raise a lot of money and find exactly what you’re looking for the first time. In the artitle many of the queries i have never to know about them,but i will study it following this article. Thanks for your sharing. coffee beans australia

    ReplyDelete
  36. Sankey Diagram is the best visualization to improve your SEO. Sankey diagram is a very useful visualization to show the flow of data.ChartExpo provides you a better and easiest way to create the Sankey Diagram in no time without coding only on few clicks. Read more here : https://ppcexpo.com/blog/visualizing-customer-journey-with-sankey-diagram .

    ReplyDelete
  37. Are you searching for nursing assignment help? We offer you the best academic writing services including the help you need to create a nursing assignment. Visit our website for more help.

    ReplyDelete
  38. This obviously makes every readers to thank the blogger and hope the similar creative content in future too.
    Buy PVA Accounts

    ReplyDelete
  39. Excellent. High end information from your end. I love to read such type of article and blogs. Keep on contributing Online hopping in Pakistan

    ReplyDelete
  40. impossible to find well-informed people in this particular topic,but you sound like you know what you’re talking about! Skydive Fyrosity Las Vegas
    Columbia Interchange Omni Heat
    skydiving jobs

    ReplyDelete

Post a Comment