Multimedia Production & Concepts

This page is to give you a general overview of the course and to cover what you are to do and to learn. With luck it will boil down all the other pages into those few topics we are covering.

For millenia humans have delivered media. It was only recently that we have delivered digital media. Before then physical media was categorized by the materials it was made from, paint, stone, canvas, papyrus, etcetera. Delivery could have been a barge down the Nile with a statue or a film delivery truck with reels of film or a radio signal. The way the media was handled depended on the material and physical size and weight.

Transport and Analog Transmission

Physical transport - Analog and Digital

Physical storage media, film, disk, cylinder, wire to be player by a physical machine such as a projector or recorder (tape, wire, cylinder) or photographic paper. Each type of analog media requires different physical materials and different handling. For example we use a projector to look at movies. We would not (normally) look at a movie by printing each movie frame on paper and view each frame separately like so many pictures on paper. We keep film in film cans and audio recordings on plastic or laquer disks or on audio tape on reels or cassettes and other methods.

Each analog method requires a completely different storage technique with different handling. With digital storage audio and visual data as well as text information are all stored in various arrangements of 1s and 0s in the same files. What differentiates the various media is the the way those patterns of 1s and 0s are stored. That is called encoding.

Once images, moving pictures and audio could be stored in digital form they could all be stored on the same devices, data tape, data disk both magnetic and optical and flash memory on cards and USB sticks. The same digital media storage devices could store varied media. Instead of a separate container for audio versus photo versus moving pictures, it is all now kept in digital files. Only the encoding changes for each media type.

Broadcast transmission

Sound or video and sound: radio and television - In a sense these are truly broadband because they send out the full signal with full sound and full picture all the time. They are also one-way methods, sending a signal with the assumption that someone "out there" is receiving them. Anyone within range can receive with no effect on the transmitter regardless of whether one person, one-million persons or no one at all is listening. The only way to know how many listeners or viewers are receiving the media is by external methods such as surveys.

Digital transmissions require a request from the receiver followed by a continuous "dialog" between the transmitter (the server) and the receiver (the client) until the transmission is completed or terminated.

Four Digital Transmission Methods

1 Downloading
2 Progressive Downloading
3 Segmented Downloads (HLS)
4 Streaming

You need to understand that there are only three basic methods of viewing or listening to media files delivered using the internet. This is the case whether you are using a web browser, a hardware television player hooked to the internet or a cell phone or computer or computer-based appliance hooked directly to a phone network or WiFi or Ethernet.

In the early days of streaming (from the mid 1990s) these differences were very visible in terms of the amount of time between clicking the media link and the start of media playing. Today, with much faster lines (more bandwidth) the time-to-play differences are not usually all that noticeable.

  1. Download offers the best quality, taking as long as needed to deliver the entire file in the best quality. In a sense, radio and television are inherently high bandwidth downloaded media in that a lot of material is sent and that there are no delivery controls for play, pause, etcetera.
  2. Progressive Download also delivers the entire file in whatever time is needed for full quality. However the player starts playing the file from disk (while still being saved) at a point during the download when the software estimates that the file download will complete before the file finishes playing.
  3. Segmented Download also known as HTTP Live Streaming - converts the source file into small (10-second) segments, then encapsulates those segments within an MPEG-2 container (transport stream) and delivers these segments via HTTP or HTTPS protocols.
  4. Streaming plays the file as it is delivered after first building a small "buffer" on disk or in memory in order to keep playing smoothly through any "hitches" in delivery. Standard player controls are available: play, stop, fast forward, etcetera. Because streaming must play the file in real time, streaming delivery will sacrifice quality in order to maintain continuous play (without breaks or waits). That will depend on traffic, routing and download speed (bandwidth).

Downloads - (handled by the browser - http protocol)

Download and save the file on the local disk drive.
RealMedia, QuickTime, Windows Media, MP3, Flash Movie, etc.

Once downloaded, the file is played. Because the file doesn't need to play in real time as it downloads, the transfer can maximize for quality.

Play, pause and other controls operate only on the local machine on the local file.

Because this is saved on your machine you can play it any time later unless there is some sort of digital rights management contained with the file.

Progressive Downloads - (handled by a player - typically http protocol)

Download and Begin Playing when the player plugin for your browser estimates that the entire file will be saved to the local disk before the playing is done. Saves locally as a temporary file.

The media player calculates how much time is left to finish downloading the file. Once the time left is less than the time it will take to play the file, it begins playing from the start of the file.

Play, pause and other controls operate mostly on the local machine

Repositioning the pointer ahead of the downloaded amount stalls the machine until downloading catches up.

Because the file is saved to the local disk, this protocol with a player leaves the file on disk, available for piracy. So someone with security concerns should be streaming the media, not doing downloading or progressive downloading.

DPD - (Digital Phono-record Deliveries) - such as iTunes

This is a different category which can combine any of the three primary methods of transport and play. The main difference here is that the download is done through a service, such as iTunes and any resulting files and their usage are controlled by that service. The service may mean storing files on the device it is downloaded to or it may mean continued access to and streaming replay via of certain files which are kept in "the cloud." The cloud is on a server located somewhere else, either on the internet or some other server accessed by the phone or by the service via phone or internet. In other words, you are renting music plays.

Full Permanent delivery

DPDs may be transferred to other devices or burned onto disks such as CDs, depending on any rules set by the record distributor for each DPD (digital phonorecord delivery, what is just a file to you). DPD are covered in Section 115 of the Copyright Act.

Limited, Tethered delivery

This is a digital file delivered electronically but with device, time or usage limits or a combination of all of those limits.

Limited-time download restricts how long the file is allowed to be used before media software will stop playing it, such as 30 days.

Limited-use download my restrict how many plays before the media player will stop using it. This requires that the media player keeps a count of the number of times that file has been used.

HTTP Live Streaming

If you are streaming to phones (such as iPhone) you should use HTTP Live Streaming as a matter of practice. Apple's specs for mobile apps on the app store require that a video clip longer than 10 minutes or a video clip which sends more than 5 megabytes of data in 5 minutes must use HTTP Live Streaming - or they reserve the right to remove your app. The stream you provide should be the old (low definition) POTS bandwidth of 64kb per second (POTS = Plain Old Telephone System).

Streaming - (media player software on your machine deals with the server - mms)

Play as you Download without saving anything to the local drive except as a temporary file in the folder for holding internet work files. Priority goes entirely to playing in "real time" rather than to quality of image. This requires compression techniques. Depending on internet traffic and bandwidth requirements (see below) some of the file may never get played. You may or may not notice. (RealPlayer, QuickTime, Windows Media Player, Flash player, DivX player, a player app on an iPad, iPhone, Android OS, hardware with internet connector such as and HD Television or and HD file player, etc.)

Tech note: Although a file is not saved to the hard drive with a message acknowledging name and location, there is a file saved to the temporary-files location which can be found if you look for it. The name may not be obvious but you can check by expected file size, time, date and file type.

Connection speed (bandwidth) makes a huge difference in what you can receive as a customer and, as a provider. It also makes a difference in how important streaming is compared to one of the download methods. When most connections were slower streaming was an important way to listen to or view files without waiting hours for a few minutes of play time. With faster connection speeds this distinction has gradually changed. For smaller files and faster connections, you might not even notice the difference between streaming and downloading.

When streaming, the media player begins playing the file as soon as possible, which is as soon as the player software estimates that it can build up a buffer to work from in the case of line delays or other glitches.

Streaming has two other characteristics:
a - lower quality - because the file needs to keep playing in real time meaning that transmission problems will cause drop outs. Data delivery can be slowed by either too much traffic or slower transmission lines on the route between server and client. If needed the media server will sometimes drop to a lower quality delivery to avoid drop-outs and choking. If the media server is unable to deliver lower-bandwidth streams the client (you) will experience interruptions where the signal stops and starts. This is why you need to provide variously sized files (or file-delivery methods) for the server to send to visitors (client machines).
b - play, pause and other controls operate across the internet, all the way back to the media server

Two Streaming modes

On Demand Streaming (links on a web page) is for existing files on a server made available for users who wish to play a specific song at a specific time but without saving the file to their own machine.

It has become standard for both radio and television shows to archive their shows as files online for on-demand streaming at later times. This way fans can review or catch up on shows they were not able to view on scheduled broadcasts. It also serves as a connection between shows and fans.

Live Streaming (real time) is analogous to a radio transmission and involves streaming a program or work as it is being created (on air, so to speak).

In both cases a link is created. For "On Demand" the link goes to the existing file.

For "Live Streaming" the link goes to a virtual location to which the feed from the live program is being sent. Normally the feed is sent to two locations, 1) a file created on disk as an archive and which can be used in the future for "on demand" use and 2) the virtual location on the server which then distributes the feed to each internet visitor requesting the live stream. The virtual location goes by a couple of names depending on the production software. In the case of Windows Media the virtual location is called a "Publishing Point," for QuickTime and "SDP" file (session description protocol), for RealServer a "MountPoint."

See also: LiveStreams.asp


Review Quiz

offers the best quality, taking as long as needed to deliver the entire file in the best quality.

Download also delivers the entire file in whatever time is needed for full quality. However the player starts playing the file from disk (while still being saved) at a point during the download when the software estimates that the file download will complete before the file finishes playing.

plays the file as it is delivered after first building a small buffer on disk or in memory in order to keep playing smoothly through any hitches in delivery.

On Streaming delivers existing files on a web server.

Streaming (real time) involves streaming a program or work as it is being created. It may or may not be getting saved to an archive at the same time.



Computer Magazine September 1979 cover on network protocols

Protocols - Instructions on how to Transmit Data

HTTP, FTP, RTP, RTSP, UDP, TCP, IP, MMS, HLS, HDS

A protocol is simply a set of rules for handling information between computers. It is similar to human-to-human communication in that all sides of the conversation need to talk to each other in some that everyone mutually understands, such as a common language. With computers, the protocol has to be understood by the programs on both ends of the line.

When you are streaming a file, each program at each end of the data stream (server and player) needs to know what the signals are for stop, start, pause, kill, re-send, etcetera. That information is written into the protocol. The first letters in a URL specify which protocol to use such as the http in "http://www.abc123.com/......"

TCP and TCP/IP - Some protocols handle all the needed transfer information and methods by themselves and sometimes they are divided into various layers to handle different parts of a larger task. The most commonly used example may be TCP/IP which started as TCP and later added IP (this is pronounced by listing the letters in the abbreviation: "tee cee pee, eye pee" [ NOT TeeCeePip ]).

TCP handles the transport (moving the data) and
IP handles the data itself (making sure it is complete and error free). Dividing the tasks simplifies the programming leading to fewer errors.

The TCP part of TCP/IP guarantees delivery of every bit sent. However, the method to guarantee transmission requires extra time which means starting with a larger buffer before playing. When TCP sees a data loss it does a retry and may have a timeout.

If data is lost on the way to the receiver, it will be requested again and eventually played (rather than being a dropped as with UDP) but the time to get the lost data may cause the media stream to stall. To avoid the stall, the original data buffer is larger.

If you are merely receiving data the larger buffer is not much of a problem. But if you have interactive needs such as online conferencing, the delays and buffering can be ackward to work with.

Here are some protocols we will be most concerned with:

HTTP - HyperText Transfer Protocol - the networking protocol used by the World Wide Web to distribute information (web pages) and to collaborate with others on the web through mutual editing of documents. Often forgotten is that Tim Berners-Lee, in designing the web was designing a tool that could be used to collaborate on documents using the internet.

RTP - Real-time Transfer Protocol - (and RTCP - Real-time Transport Control Protocol) - Defines the packet used to send audio and video across the internet (using Internet Protocol). Used not only for streaming entertainment audio and video but also for telephony (such as Skype) and video teleconferencing controls. This handles the transport, while RTSP handles the control over the stream. This runs on UDP (below) and as such can suffer dropouts if data is lost.

RTSP - Real Time Streaming Protocol - rules used to control media-streaming servers to provide typical play controls (play, pause, ...) The actual data movement is controlled by RTP. This will run on a number of transport protocols.

UDP - User Datagram Protocol - (a datagram is a "message") a very direct way of sending data without a preliminary conversation between machines. Because of this a lot of firewalls block this by default, especially corporate firewalls. Exceptions are specifically opened, usually by system admins, to specific addresses. UDP has no way of making sure of delivery. If data is lost gaps will occur in the materia.

MMS - Microsoft Media Server - Microsoft files (wmv, wma) - used with absolute addressing in Windows media metafiles. Otherwise, for relative addressing, the MMS is generally assumed by the Microsoft servers. This is a Microsoft proprietary protocol, not a standard.

* HLS - HTTP Live Streaming (NOT an adopted internet standard but an important new protocol). HLS is a commercial standard introduced by Apple for their QuickTime X and iPhone software systems) The market strength of the iPad and iPhone are shown by the quick adaption of this non-standard commercial protocol by Microsoft to their IIS web server and supported by Adobe and Google as well as other servers.

Unlike RTP, HLS uses regular HTTP but breaks the audio and/or the video stream into much smaller packets as individual, mini downloads. These are then put together at the other end. HLS starts the session by sending a complete playlist (an m3u file) to the client showing what options are available, such as the variety of data rates.

* HDS - HTTP Dynamic Streaming - Adobe created this protocol to get around restrictions on Flash, especially on Apple Devices. Apple's Steve Jobs has refused to allow Flash on his new devices, starting with the iPhone. As a result Adobe and Apple got into a very public dustup. HDS streams F4F (extension) files as a portion of the MPEG-4 container which uses h.264/AAC codecs. Basicially this is a tunnel through the Apple software barrier so that they can use the new protocol to deliver their own marketing into the Apple operating system.

* keep paying attention to these protocols

Types of Online Broadcast

Unicast

Protocols which are used for unicasting send a separate copy of the media stream to each receiving customer, just like sending individual page requests for regular web pages. Although this is the method used for most internet media streams, necessary because of architecture, unicasting is limited in how many people can get the stream at one time. As the number of users go up the performance decreases and reaches an upper limit. If you have one user you send out one stream. One hundred users requires one hundred streams and so forth.

Multicast

Protocols which can multicast work a little more like over-the-air broadcasts in that one stream is sent to multiple machines. They all take in the stream. However, this has to be done by addressing either a closed system or a single sub-IP address shared by a number of machines.. For what should be obvious reasons, you cannot simply send the same single broadcast to the internet in general. This kind of streaming prevents playback controls from being sent by each receiving machine back to the server. However, an intermediate solution as available by using buffered storage hardware at the receiving machine so that the material is received and stored as it comes in locally to each machine. Any playback (rewind, pause, go, etc.) is actually handled locally.

Generally groups of recipients on a LAN use Internet Group Management Protocol. If this is to a company or government group of people on their machines the administrators of the networks need to open a tunnel to let the signals through their routers and their firewalls and to specify which IP groups are getting the muliticast.

Peer-to-Peer (P2P)

Peer-to-peer protocols bypass central servers and allow streams to be sent directly from regular machines on the internet or network to other machines. This way the central servers don't have to shoulder the entire burden of sending to everyone. That brings up new problems including performance (the machines are not designed to be fast enough to be servers) and legal and business considerations.

A La Carte Service

Here a supplier allows you to download music by paying a purchase price for each piece you download. Amazon, iTunes and others are examples.

Online Music Subscription Services

Like the A La Carte services, you pay for the music you download, only here you pay a set fee over a set time period.