Webcam and Linux – GStreamer tutorial

Logitech Quickcam Pro 4000 webcam (without

Image via Wikipedia

Nowadays almost every laptop comes equipped with the camera that is sitting a little bit above the LCD screen, staring at you. Desktops are a different story since monitors with built in cameras are not that common. Unless you’re one of those fruitcakes and you own the Cinema Display. However getting a camera isn’t all that difficult and you will be overwhelmed by the offer. Cameras connected to computers are mostly called webcams1 and they come in various shapes, sizes and prices. Prices range from $10 up to $100 and there is a real difference between $10 camera and $100 one.

Now you decided to become the Jenni of the next decade and you will be using free software to accomplish your goal, right? Before you run to the store and buy the most expensive piece of equipment make sure you know what you are buying. First stop would probably be the Linux UVC driver page where you can check which cameras are supported and which not. Other, non UVC cameras are also supported but the support is limited. You will find more about the other drivers in the LinuxTV wiki.

Your webcam is now sitting on top of your monitor, pointed at your face or under your desk pointed at, well, something, and you want to record whatever the camera is pointed at. I admit, I had no idea on how to do this in Linux. Two weeks ago when I bought the cam I was losing my virginity sort of speak. I got me a Philips SPC 1330NC and I was in luck since it is supported by Linux UVC drivers2.

There are number of programs that can record input from Video4Linux2 source. USB webcams are all v4l2 sources. I had trouble with all the programs I tried and sadly none of them was working like it should or the quality of the recording did not meet my criteria. I was bouncing between ffmpeg and cvlc3 and in both cases I had issues with audio/video sync and quality of the output picture was sometimes questionable.

GStreamer to the rescue

Then she said that I should use GStreamer. I did a little research since I was convinced that GStreamer is only a framework for building other applications. However, it comes equipped with the gst-launch utility which is the swiss-army knife of video and audio manipulation in Linux.

Now let’s get busy building a pipeline!

$ gst-launch v4l2src device=/dev/video0 ! \
'video/x-raw-yuv,width=640,height=480,framerate=30/1' ! \
xvimagesink

GStreamer has a simple pipeline workflow. First you define the source, then you tell what to do with the source and in the end you tell where to put it by specifying a sink. Sink can be almost anything: a file, your screen or a network which turns your computer into a streaming server. In the above case you tell gst-launch to take video4linux2 as a stream source. Then you link it to a capability, which is defined it by a mime type and a few optional properties, telling v4l2 that you want video, Giving it a proper resolution and a frame rate. In the end you link everything to xvimagesink. This sink will display your stream on the screen.

Now that you have a live feed on your screen you can try putting it in a file.

$ gst-launch v4l2src ! 'video/x-raw-yuv,width=640,height=480,framerate=30/1' ! \
queue ! videorate ! 'video/x-raw-yuv,framerate=30/1' ! theoraenc ! \
queue ! oggmux ! filesink location=me_funny_dancing.ogg

What is different this time? First there is queue which provides some buffer for the next element in the pipeline, videorate. Videorate will take each frame of the input and feed it to the next element at the requested framerate. This is achieved by duplicating or dropping frames. Stream is then linked to theoraenc which encodes raw video into theora stream. Then the stream is linked to oggmux which takes theora stream and muxes it into an ogg container. In the end a properly contained theora encoded video is linked to a filesink which writes all the data to a file. Try playing it!

$ mplayer me_funny_dancing.ogg

See yourself doing a funny dance. But oh, noes! The quality is kind of bad and you didn’t see yourself while you were recording. Let’s take it one step of at a time. First the recording feedback.

$ gst-launch v4l2src ! 'video/x-raw-yuv,width=640,height=480,framerate=30/1' ! \
tee name=t_vid ! queue ! xvimagesink sync=false t_vid. ! queue ! videorate ! \
'video/x-raw-yuv,framerate=30/1' ! theoraenc ! queue ! oggmux ! \
filesink location=me_funny_dancing.ogg

Yes, this pipeline is getting more and more complicated. With tee you can split data stream into multiple pads and handle each one of them separately. First you create an element tee and name it t_vid then you link first part with xvimagesink so it displays live stream. Then by appending t_vid. you tell gstreamer that you want to split the data stream and you link the rest of it to theora encoder and muxer and filesink like in previous example. You will end up with your pretty video on the screen which is also being recorded into a file.

As we all know that there is no home made porn without sound we somehow need to add sound to the video. But how? Simple, by complicating our already complicated pipeline.

$ gst-launch v4l2src ! 'video/x-raw-yuv,width=640,height=480,framerate=30/1' ! \
tee name=t_vid ! queue ! xvimagesink sync=false t_vid. ! queue ! videorate ! \
video/x-raw-yuv,framerate=30/1 ! theoraenc ! queue ! mux. \
alsasrc device=hw:1,0 ! audio/x-raw-int,rate=48000,channels=2,depth=16 ! \
queue ! audioconvert ! queue ! vorbisenc ! queue ! mux. \
oggmux name=mux ! filesink location=me_funny_dancing.ogg

Are your eyes bleeding? You’re not finished yet, so get ready for more. What in the name of a pipeline is this? You now already know all about splitting the pipeline and how to handle those two splits so let’s go straight to the changes. In the third line you can see that where video should be linked with oggmux it is actually linked with mux. which is a name for your future muxer that you will create later. Next you need to specify one additional source for sound. In my case it is Alsa source and the device I used is hw:1,0 which corresponds with the built in microphone on the webcam. Your device will probably be something else. Just like with the video we need to tell alsasrc what kind of data we want. Mime type is audio/x-raw-int sampling rate is 48kHz with two channels and 16 bit depth. This is sometimes enforced by the audio source and you might have to play with the settings a little bit. You will need to link it to audioconvert element which takes your audio and converts it to something that encoder expects. Vorbisenc is an audio encoder as you probably guessed and it is linked to mux. At last you will create that muxer you were referring to before and link it to the filesink.

There, you’re all set! No wait, don’t run off just yet. What about the quality? Satisfied? No? Then stick around! So, on the fly encoding to theora works but the quality of your recording is not that good. The problem with theora is that even setting the quality to maximum doesn’t help much. You will need something else, something more raw! Try this:

$ gst-launch v4l2src ! 'video/x-raw-yuv,width=640,height=480,framerate=30/1' ! \
tee name=t_vid ! queue ! videoflip method=horizontal-flip ! \
xvimagesink sync=false t_vid. ! queue ! \
videorate ! 'video/x-raw-yuv,framerate=30/1' ! queue ! mux. \
alsasrc device=hw:1,0 ! audio/x-raw-int,rate=48000,channels=2,depth=16 ! queue ! \
audioconvert ! queue ! mux. avimux name=mux ! \
filesink location=me_dancing_funny.avi

Don’t be surprised if you see your mirrored image on the screen. You should, some people prefer the mirrored preview. It certainly makes nose picking easier, right? If you don’t like it that then simply remove the videoflip element. With this pipeline your video will stay uncompressed and will be written in an avi container with the help of avimux. Please note that ten seconds of your video will require around 150 Megabytes of disk space and one minute is close to 1 Gigabyte if you record in 640×480@30fps! Yes it will be huge.

The problem with this raw video is that is not editable in Kdenlive. If you try to open it Kdenlive will simply crash and die in a puff of smoke. My main problem was how to record a video with a decent quality. I tried dozens of combinations, but nothing really worked and raw video was not editable.

And now the/my solution. Quality of H.264 would be good for me, but H.264 encoding takes a lot of time even on a Core2Quad CPU and it can’t be done on the fly. You need to record raw video and then convert to H.264. Here’s how:

$ gst-launch filesrc location=me_funny_dancing.avi ! \
decodebin name=decode decode. ! queue ! x264enc ! mp4mux name=mux ! \
filesink location=me_funny_dancing.mp4 decode. ! \
queue ! audioconvert ! faac ! mux.

This time you need to specify a different source. It needs to be the file which contains your recording. Then you link it to the decodebin which will create a pipeline that you can link to x264enc and then mp4mux. Decodebin works similar to tee, you get more than one pipeline out of it so that you can separetely work with audio and video. You will need to mux audio and video together with mp4mux and write it to file. Careful eye will notice that faac element was added before the ending mux. This is because mp4 container can’t contain raw audio so we convert it to mpeg2/4.

Conversion will take some time, but it is well worth it. Video that you get will be of a very high quality and it can be edited with Kdenlive and later rendered into your final product.

This concludes the first part of Gstreamer tutorial. Next time I will talk more about streaming live feeds to all the viewers out the on the internet.

Comments, suggestions, corrections and flames are well appreciated.

Update: Instead of hw:1,0 (or whatever device you are using) you might try with plughw:1,0. In layman’s terms, plughw is a software abstraction above the hardware device that will do a lot of converting instead of you. It will match sampling rates and channel numbers and so on and so on.

 

Reblog this post [with Zemanta]


Footnotes:
  1. Term coined in mid 90’s, before YouTube existed. []
  2. No, I didn’t previously check if it is supported, don’t be like me! []
  3. VLC without the GUI []

  • [...] understand and play with some cool ffmpeg “pipes” just modifying, for example, this interesting Capture2File GSVideo Library sketch (if you install [...]

    Reply
  • This is a great job!!! Just one question, I’m a newbie and I want to know if there is some kind of docs where I can learn about which gstreamer element I must use with other? I say, how to build a pipe by my own. I read the GStreamer application manual but I think it is not enough.

    Reply
  • On August 4, 2010 at 08:48 Stephen said:

    May I ask where I can find the hw:1,0 In your sample code of Webcam?

    Reply
  • On August 10, 2010 at 10:02 Stephen said:

    find the solution /proc/asound/devices

    Reply
  • Wise men were on vacation, so a little late answers.

    @Mems:

    Use gst-inspect to check which inputs and outputs are acceptable by a certain element.

    $ gst-inspect filesink
    or
    $ gst-inspect audioconvert

    And you can use gst-inspect without the parameters to get a list of all the elements.

    @Stephen:

    I usually run arecord -l to get list of devices.

    Reply
  • On September 7, 2010 at 08:30 youme Lee said:

    Thanks for you specific comment…
    It’s awesome…..

    Reply
  • This is great – the only “one liners” I found via google that actually worked!

    Only two comments from me.

    1. Where your scripts identify the soundcard as hw:1,0, I found I had to call this plughw:1,0 for my eyetoy webcam on Xubuntu 10.10.

    2. You switched from me_dancing_funny to me_funny_dancing halfway through :)

    Cheers, Jose

    Reply
  • hey this is very nice tutorial… do you know how to stream this or any video.. i tried but i couldn’t stream .. if you have any thing related to that please help…
    thanks

    Reply
  • Thanks – looks like these one liners have fixed my audio-video sync problems using guvcviewer.

    Thanks too to Jose where he commented that he had to change device=hw:1,0 to device=plughw:1,0

    That got it working for me. Using a Logitech cam with device=hw:1,0 would fail with this cryptic error message:

    ERROR: Pipeline doesn’t want to pause.
    ERROR: from element /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0: Could not negotiate format
    Additional debug info:
    gstbasesrc.c(2767): gst_base_src_start (): /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0:
    Check your filtered caps, if any

    device=plughw:1,0 saved the day.

    Thanks!!

    Reply
    • Yes, plughw is basically just a virtual device that talks to the physical device and supports more input/output configurations and formats.

      I will update the post accordingly.

      Reply
  • On March 27, 2011 at 05:10 sahil said:

    is there any way by which the web cam feed can be transmitted live over a network like as in if i want to make a video chat app.

    Reply
  • Brilliant tutorial and of immense use to me in a forthcoming project. I am indebted to you good (sir?)

    Reply
  • Useful; been trying to figure out how to do this for a while. Thx.

    Reply
  • Hi BigWhale, your’s very good article teach me many things… I’ve a request: how to register only 10sec of webcam video, or 10sec of udp streaming?
    Thanks a lot. Bye
    Riccardo

    Reply
    • You can add a num-buffers parameter to the source and you can specify number of frames that will be captured.

      I am not sure if there’s a time parameter.

      So, if you are capturing at 24 frames per second, then you need to specify num-buffers=240 to get 10 seconds of capture.

      Reply
  • On September 25, 2011 at 13:35 Nigel Cunningham said:

    “This concludes the first part of Gstreamer tutorial. Next time I will talk more about streaming live feeds to all the viewers out the on the internet.”

    I’ve been looking for the ‘next time’ and can’t find it. Am I blind, or did you not get around to it? If the latter, “Please…” :)

    Reply
    • Unfortunately you are right. The next part wasn’t written, because I never had to do the actual streaming then. I’ll probably write it somewhere in the feature. :)

      Reply
  • Would love to see the next part as well :-)

    Reply
  • Now thay you know how GStreamer works you can give a try to the “Cheese!” program which uses GStreamer itself under the hood :)

    Reply
  • [...] http://www.twm-kd.com/linux/webcam-and-linux-gstreamer-tutorial/ Share this:TwitterFacebookLike this:LikeBe the first to like this post. Posted by bmagyar Filed in Uncategorized Tagged: cheese, gscam, gstreamer, ros Leave a Comment » [...]

    Reply
  • On June 5, 2012 at 03:04 Anshul said:

    hi wisemen,
    may i know is there any possible way to see what type of packet does device give or any way to use webcam without gstreamer directly taking data from /dev/video0

    Reply
    • On June 5, 2012 at 03:30 Anshul said:

      hi
      i foud that my webcam is working only with framerate set to 10 or 15 neither 11 nor 16and ot even on 30 or 5
      can anyone explain why this behaviour

      Reply
  • On June 5, 2012 at 03:26 Anshul said:

    hi
    while setting framerate i found that when pipeline want to get paused and giving me error that pipeline don’t want to be paused

    Reply
  • Thank you, this information was very helpful as I reachad a road block trying to do the same thing in VLC. Even though we appear to have the same webcam, I had to make one small adjustment to each commandline: reduce the frame rate to 15 or less (in both places) as well as keeping the device=/dev/video0 parameter.

    Reply
  • [...] Het principe is dat er een pipeline wordt opgebouwd waardoor de video verstuurd kan worden. GStreamer kan ook encoderen en bestanden naar file schrijven. Vele mensen noemen het “The Swiss army knife of video and audio manipulation in Linux” zo ook hier. [...]

    Reply
  • On November 22, 2012 at 08:17 davidmax said:

    hi,BigWhale

    Reading your article is very useful for me to learning gstreamer.But when I try the commandline below:
    gst-launch v4l2src ! ‘video/x- raw-yuv,width=640,height=480,framerate=30/1′ ! tee name=t_vid ! queue ! xvimagesink sync=false t_vid. ! queue ! videorate ! ‘video/x-raw-yuv,framerate=30/1′ ! theoraenc ! queue ! oggmux ! filesink location=me_funny_dancing.ogg

    it’s always failed with some error message,like this:

    Setting pipeline to PAUSED …

    (gst-launch-0.10:4769): GStreamer-WARNING **: Pad list returned error on element t_vid
    Pipeline is live and does not need PREROLL …
    Setting pipeline to PLAYING …
    New clock: GstSystemClock

    (gst-launch-0.10:4769): GStreamer-WARNING **: Pad list returned error on element t_vid
    ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data flow error.
    Additional debug info:
    gstbasesrc.c(2507): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
    streaming task paused, reason not-negotiated (-4)

    Can you help me solve this problem?
    Thanks.
    davidmax

    Reply