Lumiera is the emerging new Video Editor for Linux. ()

mexicorarara | 1 Oct 20:24

Automatic Audio Syncing / Fingerprinting

I just re-read this
> http://pipapo.org/pipawiki/Lumiera/DesignProcess/DelectusShotEvaluator
>   
and thought that this is interesting point:

---
The application should be smart enough to fill in audio if the existing 
source is missing. For example, in a recent project I was working on a 
camera op accidently set the shotgun mike to test mode, ruining about 
10% of the audio for the gig. I was running sound, and luckily I had a 
backup copy of the main audio being recorded. This application should, 
when told that these two are of the same event at the same time, 
seamlessly overlay the backup audio over the section of the old audio 
that has been marked bad and not even play the bad audio. This is just 
background noise, and streamlining the immense task of sorting through 
footage needs to be simplified as much as possible.
---

Because still, at least in lower budget productions for flexibility 
reasons (no cables to the camera etc), external audio (Flash-Recorder / 
DAT) without TC sync is used. Even the more professional Prosumer 
cameras / Audio Equipment lack possibilities to sync for Freerun or 
other sync strategies. On the other hand, at least in video, often 
external Sound recorders and camera mic record the same sound from 
different perspectives. So, automatic syncing where possible by creating 
audio fingerprints could be nice. I dont know if this is what the part 
in the document referred to.

headers

Nicholas S-A | 1 Oct 22:17

Re: Automatic Audio Syncing / Fingerprinting

* mexicorarara <mexicorarara@...> [2008-10-01 13:24:39 -0500]:

>Because still, at least in lower budget productions for flexibility 
>reasons (no cables to the camera etc), external audio (Flash-Recorder / 
>DAT) without TC sync is used. Even the more professional Prosumer 
>cameras / Audio Equipment lack possibilities to sync for Freerun or 
>other sync strategies. On the other hand, at least in video, often 
>external Sound recorders and camera mic record the same sound from 
>different perspectives. So, automatic syncing where possible by creating 
>audio fingerprints could be nice. I dont know if this is what the part 
>in the document referred to.

I think that the fingerprinting is the next step. I was more thinking
about it as finding "bad" sound and replacing it -- the user is
responsible for defining key points in the two media streams for
syncing (and then Lumiera would do time interpolation of the samples
or use a similar technique between the two audio clips and hopefully
give a reasonable result [and the various options would be controlled
by the user of course]).  It is this job which would be removed by
fingerprinting, and would be very beneficial (especially in cases
where the primary or secondary recordings are started and stopped),
but I imagine that this would take much more work than we are able
to give at the present time. For future iterations of Lumiera (or
as a plugin), this is certainly a very real option.

nick

headers

prg | 1 Oct 20:51

Re: Automatic Audio Syncing / Fingerprinting

mexicorarara schrieb:
> I just re-read this
>> http://pipapo.org/pipawiki/Lumiera/DesignProcess/DelectusShotEvaluator
>> 
>> 
> and thought that this is interesting point:
>> --- The application should be smart enough to fill in audio if the
>> existing source is missing.
...
> 
> Even the more professional Prosumer cameras / Audio Equipment lack
> possibilities to sync for Freerun or other sync strategies. On the
> other hand, at least in video, often external Sound recorders and
> camera mic record the same sound from different perspectives. So,
> automatic syncing where possible by creating audio fingerprints could
> be nice. I dont know if this is what the part in the document
> referred to.

While I can't answer for the Proposal (nasa?), I'll add a note from
the view of the Proc-Layer (because I am working on this part of the
App). I wanted to provide two different mechanisms for dealing with
additional audio:

- a special kind of Placement, where the ref-location is not an Session
  time point, but a specific location within another media within
  another clip. Typically example being the frame where you see the
  flap. Then, the additional audio would be added as an independent
  clip, but with this sort of placement. Note this will use/calculate
  the offsets, i.e. even if you trim the ref clip, or the sound clip,
  the porition of the underlying media is used to keep the sync.

(Continue reading)