Half a croissant, on a plate, with a sign in front of it saying '50c'
h a l f b a k e r y
Idea vs. Ego

idea: add, search, annotate, link, view, overview, recent, by name, random

meta: news, help, about, links, report a problem

account: browse anonymously, or get an account and write.



Convert speech to animated moving lips

Cell Phone like device converts speech to visual moving lips
  [vote for,

Those hard of hearing often rely on being able to read a speaker's lips. Often the speaker is not aware of the listener's handicap and moves too far away, turns away from the listener, cover's their mouth while wiping their face, speaks very softly with little lip movement, etc. I would like to see a cell phone-like device (Iphone app possibly) that would convert speech to lip movements on an animated character's face on a cell phone-like screen. The character's lips would parrot the lip movements of the speaker in detail. The cell phone-like device could also act as a hearing aid and feed sound to the listener's earphone. The device / app could also translate frequencies the listener can't hear the frequencies the listener can hear. See the annotation to jutta's frequency translator idea on the attached link.
Sunstone, Jul 08 2010

Frequency translator frequency_20translator
Translate audio into vibrations [Sunstone, Jul 08 2010]

Lipsync MX http://www.lipsync-mx.com/
Pretty good. [wagster, Jul 09 2010]

Similar idea Animatronic_20Subti..._20for_20the_20Deaf
[hippo, Jul 09 2010]

Comedy lip movement device http://www.google.c...fc&biw=1012&bih=577
Not exactly what I'm thinking of but the idea is in the vicinity [Sunstone, Jun 28 2011]

Getting there http://www.mirror.c...uld-capture-7627465
Lip reading technology could help people solve crimes by deciphering what people are saying on security cameras [Sunstone, Mar 25 2016]

Talking penis https://de.wikipedi.../Besonders_wertvoll
[Toto Anders, Mar 26 2016]


       I wonder if professionally rendered CG faces (ie. those in the movie "Avatar") can be lip-read...
Cedar Park, Jul 09 2010

       Interestingly, I've just started using lipsync software (Lipsync MX - link) after my previous animation project went massively overtime due to the huge amount of lip-sync involved, which is one of the most time-consuming parts of character animation.   

       Lipsync software analyses speech for phonemes, the various sounds our mouth can make and converts them to visemes, the shapes our mouths make when we make the sounds. There's pretty much one shape for each vowel, one for all the consonants and a few extras, such as "L" and a closed mouth for non-speaking moments.   

       It's far from perfect and the finished result always needs editing before use. The analysis time on a decent computer is fairly fast and could easily run in real time, but I'm not sure an iPhone will be powerful enough. Another problem is the speech quality - I'm recording with a decent mic in a quiet room - call quality will hamper the process. Also, there will always be a slight delay as the technology will only ever be able to display the last phoneme spoken, but not the one being currently spoken.   

       So - it's possible, but we might have to wait until phones get better processors and the analysis algorithms get better (but with Apple or Google money...). The processing delay can be largely fixed by delaying the audio to match the display.   

wagster, Jul 09 2010

       very cool (+).
MisterQED, Jul 09 2010

       Headsets, or on speaker.
wagster, Jul 09 2010

       Will there be subtitles ?
8th of 7, Jul 09 2010

       And subtleties?
normzone, Jul 09 2010

       Subtle t's would be too difficult to animate.   

shudderprose, Jul 10 2010

       This has been done in Flash (RoboText or something similar) with a fairly low processor load. I doubt you could lip-read it though, but I'm sure it's possible with current mobile CPUs, it should certainly be less intensive than 30fps video encoding which is currently done in real-time on phones.
marklar, Jul 17 2010

       This I like! And would use... That's it then! Bun [+]
Grogster, Jul 17 2010

       There are several heroic efforts of the pre-animation age to create similar effects with talking penises <link>. To emulate the effect there would have to be a choice of "faces" in this app's user interface.
Toto Anders, Mar 26 2016


back: main index

business  computer  culture  fashion  food  halfbakery  home  other  product  public  science  sport  vehicle