Possibilities Discussion - Video As Mask

[Lin Evans]

This post is simply an effort to stimulate discussion about possibilities for the "video as mask" feature....

One of the features several of us have asked for in a future version of PTE (hopefully the next one) is to be able to use video as a mask. This will open up a number of great possibilities for PTE's extended use not only in AV productions, but also in the business world.

A feature we sometimes see on the web is a website with a "talking head" or with a tiny "actor" walking back and forth and talking about the website or some of the features, etc. This is done using green screen (or blue screen) alpha channel extraction. With video as a mask, we could easily do this with PTE and create really interesting possibilities. Today, there are multiple video editors capable of doing alpha channel creation along with a normal video so that the user has both. How this ends up is a black and white video and a color video to match. Programs which create special effects, such as Particle Illusion can be used to create an alpha channel along with the regular video. Once we have the Alpha Channel black and white video which can be used as a mask - we would put the black and white in as the mask and run the video underneath. Then only the animation would be visible through the mask. We could then "animate" the mask as desired. For example, let's say we had an alpha channel extraction of a horse running. We could move this running horse across a pasture still image or a video. We could have a person walking and talking and this could easily be overlaid on a business presentation, etc. There are myriad possibilities once this feature is manifest in PTE.

How would this be possibly used in a dynamic AV presentation of the sort used in AV competitions? How would it be received by judges? These are questions which we might discuss. I have no knowledge or experience with AV competitions so am depending on those who have a background and experience to give their input and open minded commentary....

Best regards,

Lin

[Lin Evans]

Hi Tom,

Chroma Key would be nice, but not necessary. What I have in mind is just the ability to use video as a mask.

A number of software applications can create a video with Chroma Key technology which have both the black and white alpha channel as well as the color channel produced. In PTE with a using video as mask option, we then insert the alpha channel black and white video as the mask and inside the mask run the regular matching video. This then gives us the ability to use the video just as we now use a PNG transparency.

Let's say we have a video of the earth in rotation made on a black background. The alpha channel video is pure black and white. The Earth is simply a white circle within a black background. Then when the actual video is inserted within this video mask, we have the Earth rotating just as a still PNG cutout except with full video motion.

We can then overlay this "video" as a mask on any background and the only parts we see will be the earth in full color rotating on our choice of a still "or" video background. We can size and manipulate (pan, zoom, rotate, 3d transform, etc.) this rotating earth. The black surrounding the rotating Earth is transparent in the mask revealing everything "except" the Earth on the layer beneath (that being our choice of video or still backdrop). The white alpha channel revealing only the rotating Earth which can be manipulated as desired.

Now take this same concept and apply it to an alpha channel made by Chroma Key extraction of any subject whether static or dynamic and we have the ability to, for example, overlay a video of you made with a home video camera against a green or blue backdrop for extraction. You might be walking about, talking, sitting at a table, whatever. This alpha channel black and white (video as a mask) then coupled with the associated regular color video from which it was made, lets us place you and your table, etc., anywhere in the world. Let's say we have a video from Paris on La Seine by the Eiffel Tower. You could be sitting at a table as people walk by and it would appear that you were actually there.

This video as a mask has some seriously powerful implications I think...

Best regards,

Lin

Hi Lin,

Just to be sure I understand your video mask idea are we talking about chroma keying or something with additional features?

http://en.wikipedia....wiki/Chroma_key

Thanks,

Tom

[Boogie]

Lin

Interesting development. I looked briefly at the free VideoPad program that you mentioned some time ago. It has lots of facilities and has a Chroma key facility, but I haven't discovered if you can make a Black and white mask with it? Do you know?

Boogie

[Lin Evans]

Hi Boogie,

Here's what the "help" says:

Chroma Key

Select the Chroma Key color by clicking on it in the preview image. Then, using the preview image as a reference, adjust the Background and Foreground sliders until the desired transparency is achieved. Note you can click different positions on the timeline to see how the video will look. Click OK to apply changes.

If you would like to make changes later, first select the overlay in the list on the Overlay tab, then click the Change button next to the Use Chroma Key checkbox.

It's not very helpful, but I would assume that you can save the alpha channel as an AVI provided that you have the necessary Codec installed. There are several (at least 3) codecs which support the Alpha Channel. So this program at least lets you remove the Chroma Key color and leave transparency, but whether it can save this as a black and white alpha channel video or just overlay it on another video is the question. Once I get my system back and running, I'll try to find out what it can do. I can make a quick green screen video and test it then...

What we would need - to make this work with video masking is what is commonly referred to as the Luma Matte. The luma matte is essentially a black and white video. The white being opaque and the black being transparency. In PTE, the white would allow the chroma key portions (black in this case) to be transparent, and that part of the video which was not (green or blue, etc) to be visible.

Best regards,

Lin

Lin

Interesting development. I looked briefly at the free VideoPad program that you mentioned some time ago. It has lots of facilities and has a Chroma key facility, but I haven't discovered if you can make a Black and white mask with it? Do you know?

Boogie

[RobertAlbright]

'How would this be possibly used in a dynamic AV presentation of the sort used in AV competitions? How would it be received by judges? These are questions which we might discuss. I have no knowledge or experience with AV competitions so am depending on those who have a background and experience to give their input and open minded commentary'.

Can't really answer this , Lin, - the AV competition world is only just starting to adjust to the inclusion of video inserted between still images in transition (traditional AV). And no-one knows how judges will respond until we start to see the results of video insertion on a regular basis in national and international competitions. I suspect there would be negativity if people put forward video only work - there is already a thriving organisation in the UK for this kind of production (the IAC) and similar organisations internationally for movies.

[Lin Evans]

Hi Robert,

This is great information Robert.

What I believe may be the usefulness, is to overlay small chroma keyed videos over stills. Once we have the ability to use video as a mask, there are numerous types of things which, when used sparingly and with good taste can, I think, enhance the still image without becoming intrusive.

I think these type mini-animations might be useful especially in introductions and in closing scripts. An example of this was one I used some months ago in a show where I animated a "paint-brush" painting the subject in the first slide. I wouldn't use the paintbrush for a serious show, but the water-drop video I used in a couple places in the introduction and second slide might be the type of use which could be accepted by judges in time. Of course I didn't have "video as a mask" which would have made it very easy to do, but I did run the video "in a mask" and with opacity control was able to make it work.

Here's a link to that show:

http://www.learntomakeslideshows.net/almost/almost.zip (about 17 meg Windows pc exe file)

Best regards,

Lin

'How would this be possibly used in a dynamic AV presentation of the sort used in AV competitions? How would it be received by judges? These are questions which we might discuss. I have no knowledge or experience with AV competitions so am depending on those who have a background and experience to give their input and open minded commentary'.

Can't really answer this , Lin, - the AV competition world is only just starting to adjust to the inclusion of video inserted between still images in transition (traditional AV). And no-one knows how judges will respond until we start to see the results of video insertion on a regular basis in national and international competitions. I suspect there would be negativity if people put forward video only work - there is already a thriving organisation in the UK for this kind of production (the IAC) and similar organisations internationally for movies.

[RobertAlbright]

A lovely show, Lin. I tried very hard to watch the paintbrush movements but there were two other distractions which diverted my attention. Liked the ripple movement over the girl's toe in the water. Some of the scenes looked like Wales? Have you ever entered The RPS International AV Championship? - next September at Cirencester.

[Lin Evans]

Hi Robert,

LOL - those "mini-peaks" can be distracting for sure!

The ripples were an example of running and sizing a video inside a mask. Actually, it was the same video used in the opening slide. I suspect they (the images you mention) were Wales. Funny story - last time Sherry and I were in Wales I thought maybe I might look up some distant relatives. My great grandfather and his six brothers came to the U.S. from Wales and I thought perhaps I could learn a little about my genealogy. When I looked in the phone book in Cardiff, about half the book shared my last name - LOL. I decided that my relatives must be among, if not the most prolific rascals in Wales. I gave up that idea as a lost cause!

I've never entered any AV competitions, actually I've never seen one, so haven't a clue about what goes on. Everything I know about competitions I've learned from the forum here. Unfortunately, here in the U.S., there seem to be very few AV competitions, at least that I'm aware of. My experience with AV in any formal way is greatly limited. In past years, I made shows for business clients, primarily in the art world, but that's a fairly specialized venue and oriented toward selling art to customers. Years ago we had a camera club in Boulder, Colorado but interest waned, and it seems that there are not many who have the time or inclination any more for such. I guess a great deal of that is because of the internet. I envy your pervasive interest in the U.K. and in Europe in AV clubs, competition, etc. I think competition is a healthy thing for promoting overall improvement in the skills of presentation. Of course there are bound to be issues with judges and "opinions" about what makes a good AV presentation. I've always eschewed "politics" and probably wouldn't fit in very well.

Best regards,

Lin

A lovely show, Lin. I tried very hard to watch the paintbrush movements but there were two other distractions which diverted my attention. Liked the ripple movement over the girl's toe in the water. Some of the scenes looked like Wales? Have you ever entered The RPS International AV Championship? - next September at Cirencester.

[RobertAlbright]

Maureen is the one to ask about Welsh genealogy - but she has admitted mine is more interesting, including a very naughty grandfather in South Africa.

All the AV festivals I have attended, including as far away as Australia, are welcoming and fun. AV people are almost without exception helpful and generous. If you came to the UK International I'm sure you'd enjoy it (ask Igor) and see a wide range of approaches including, to come back on subject, video inserts.

Best wishes

Robert

[Ken Cox]

Robert wishes he was a male mermaid tickling her toes

- and making gentle ripples on the still water )

ken

[RobertAlbright]

I'm up for a tickle any time, Ken

[Boogie]

Hi Lin

I hope you have got your main system up and running without too much loss.

I have looked again at VideoPad and cannot see any way of creating a "Luma Matte" video. Have you had any joy with this program?

Cheers

Boogie

Hi Boogie,

It's not very helpful, but I would assume that you can save the alpha channel as an AVI provided that you have the necessary Codec installed. There are several (at least 3) codecs which support the Alpha Channel. So this program at least lets you remove the Chroma Key color and leave transparency, but whether it can save this as a black and white alpha channel video or just overlay it on another video is the question. Once I get my system back and running, I'll try to find out what it can do. I can make a quick green screen video and test it then...

What we would need - to make this work with video masking is what is commonly referred to as the Luma Matte. The luma matte is essentially a black and white video. The white being opaque and the black being transparency. In PTE, the white would allow the chroma key portions (black in this case) to be transparent, and that part of the video which was not (green or blue, etc) to be visible.

Best regards,

Lin

[Lin Evans]

Hi Boogie,

No luck with that. I think it takes a more sophisticated video editor to output both an alpha channel video and the luma matte.

Best regards,

Lin

Hi Lin

I hope you have got your main system up and running without too much loss.

I have looked again at VideoPad and cannot see any way of creating a "Luma Matte" video. Have you had any joy with this program?

Cheers

Boogie

[Boogie]

Thanks Lin

If you come across a free video editor that does what we are looking for, please let us know.

Cheers

Boogie

Hi Boogie,

No luck with that. I think it takes a more sophisticated video editor to output both an alpha channel video and the luma matte.

Best regards,

Lin

[Lin Evans]

Hi Boogie,

If I find one, I'll post a link.... The only software which I'm certain can do this is Adobe After Effects, but it's "way" over my budget. I have queries out at a few websites to see if I can get any suggestions for cheaper ways to accomplish this. After Effects is a wonderful package - you can even create videos with alpha channel without green screen/blue screen extraction using their new Rotoscope feature. It's extremely powerful, flexible, etc., but with all this power comes great "expense." Legit copies are hard to find for under $600 U.S.D. and generally run closer to $900 U.S.D.

Best regards,

Lin

[Lin Evans]

It occurred to me that actually, it wouldn't be all that difficult for PTE to make the Luma Matte without using an expensive package of any kind. The latest version of Adobe After Effects only works on 64 bit systems running Vista or later operating systems (Thanks Ken - for looking that up and reminding me) so in my case that's a non starter because I can't even afford After Effects let alone a new computer to run it on - LOL.

Bottom line is this (I'm simplifying a bit but the essence is correct): There are plenty of free software packages which can take a video and extract each frame as a jpg. Then there are free programs such as Windows Movie Maker which comes with the Windows operating system which can take the individual jpgs and make a video from them. So what can be easily done is to take a green screen video where one photographs the subject in front of and a few feet away from a solid green backdrop. Once the video is complete. First copy the original, then use freeware to extract each frame from the copy. Set up a batch action operation in Photoshop or similar program to use the select color range to select the green then inverse the selection and find and replace it in batch mode with 255,255,255 (RGB) white. Next, deselect and select the white, inverse and repeat the process replacing everything "except" white with 0,0,0 (RGB) black. Next use Windows Movie Maker or Quicktime with the MacIntosh to create a video matching the original in frame rate, aspect ratio and size.

Now you essentially have your Luma Matte for the mask itself and your green screen original to furnish the video portion inside the mask which will reveal only that portion of the video which is pure white, which is what you wanted to begin with. You then overlay this masked video on the video or still of your choice and manipulate the mask container for PZR, and 3d transform as desired. Voila! Green Screen Alpha Channel capabilities in PTE. All we need is the video as a mask!

It "shouldn't be very difficult to program the ability to extract frames, from a green screen video and convert it to a Luma Matte right in PTE once we have video as a mask incorporated. There are details I've omitted such as the necessity for a really "good" green screen capture with minimal "splash" over. This essentially means that if the subject to be extracted is too close to the green backdrop, green reflected light will "shadow" the subject and make it much more difficult to get a clean luma matte. The subject edges of both the luma matte and the green screen frame must be "softened" a bit with a slight edge blur so there is no abrupt change where green could show through. One way to help deal with that would be to also replace the "green" in the video to be included in the video mask with medium or dark gray. This would be much less noticeable than a slight "green" halo.

I've tested this theory in actual application using still images taken from a rather "bad" green screen capture and it works quite well so assuming that the precepts could be maintained, it should be a viable possibility for PTE.

Best regards,

Lin

[Ken Cox]

yesterday i received a link of Elvis singing blue xmas -- it said it was his daughter singing with him

- i sent it out to my dist list and was told promptly it was not his daughter

so i started googling )

ELVIS - BLUE XMAS

That isn’t his daughter…it is Martina McBride!!! But still really neat.

Here is one I found making the clip

and the finished product

This video is worth watching, it is a real technical feat, Elvis died in 77

a perfect example of green screen technology that Lin is talking about

ken

[davegee]

Hi Ken,

Thanks for the links.

Forgetting the technology - it was good to see that again!

DG

[Ken Cox]

so many times we say"how did they do that"?

well it is not perfect but close enough!

great clip of Elvis - one would think he was right there - and great to see the little twitches he made with his face -- he had a way!!

ken