How to create spec compliant MP4 files How to mux Subtitles into MP4 files How to place Chapters in MP4 files How to author MP4 files (ie create DVD-like Menus) How to "anamorphic resize" with MP4 How to (live-)stream/play streamed MP4 files How To create Variable Framerate MPEG-4 video streams in MP4
How to create spec compliant MP4 files
detailed answer: How to mux Subtitles into MP4 files there are three possibilites to store subs in mp4: How to place Chapters in MP4 files chapters storage is also possible by using advanced content (eg by creating a menu), but Nero went another way and uses a MP4 feature called "user space" (udta atom) which allows you to attach anything you want in MP4 (iTunes uses the udta atom for storing tags and cd/dvd-covers ("coverart") in MP4 for example). How to author MP4 files (ie create DVD-like Menus) How to "anamorphic resize" with MP4 As the MPEG-4 Standard aims to be a "multi-talent", satisfying everyone's needs, it offers 3 possibilites: How to play MP4 files as one of the big advantages of MP4 is interoperability and being an open standard (license free!) there already exist a lot of different tools supporting it on as good as all systems including Mac, Linux, PocketPC and of course Windows: How to (live-)stream/play streamed MP4 files Apple and MPEG4IP offer nice and free tools when it comes to streaming MP4: How To create Variable Framerate MPEG-4 video streams in MP4
1) To create MP4 files you should preferably use audio/video formats covered by the MPEG-4 Standard:
- MP4 video formats: MPEG-4 SP/ASP: XviD, DivX5, 3ivx, ffmpeg/ffdshow, NeroDigital ASP... and MPEG-4 AVC/H.264: x264, NeroDigital AVC, Apple... and MPEG-2, MPEG-1 (not DivX3/MS Mpeg-4, WM9, RV9 or VP6) - doom9's quality comparisons: 1 2
- MP4 audio formats: Advanced Audio Coding (AAC FAQ): Quicktime/iTunes, Nero, Psytel, FAAC (binary)... and MP3, MP2, MP1, CELP (speech), TwinVQ (very low bitrates), SAOL (midi), ALS (lossless)
- MP4 subtitle/text format: MPEG-4 Timed Text
- MP4 picture formats: JPEG, PNG
2) with the following tools you can mux MPEG-4 compatible content into the MP4 container:
- GPAC's MP4Box, commandline, supports AVC, packed b-frames, advanced content/BIFS, TTXT and Chapters
- MPEG4IP's mp4creator, commandline, able to mux AVC, dont use it with AVI and packed b-frames!
- 3ivx mp4 muxer dshow filter, usable in Graphedit, supports packed AVI b-frames but no AVC (3ivx Guide)
- mp4UI, easy solution, based on MPEG4IP, dont use it with AVC, AVI and (packed) b-frames!
- VideoLAN Client, able to mux AVC, dont use it with (packed) b-frames!
- ffmpeg, buggy, commandline, especially dont use it with AVC, AVI and (packed) b-frames!
- IBM's Toolkit for MPEG-4 which includes XMTBatch and/or AVGen (also a simple version) based on Java
- dvd2mp4gui, GUI using mp4tool, supports muxing easily multiple audio and subtitle streams (as advanced content) into MP4
- mp4tool/mp4edit from ENST, similar to mp4box, mp4edit is the "light" GUI version of the commandline mp4tool
3) the following "all-in-one" MP4 encoding tools are existing:
- Nero Recode2, encodes into MP4 using ASP or AVC and AAC (Doom9 Guide), subs are DVD VobSubs, includes a dshow muxer
- Doom9's MeGUI, supports ASP (Xvid, libavcodec), AVC (x264) and AAC (Besweet - Nero) encoding
- HDX4 supports ASP and AAC
- Dicas' mpegable X4 live, easy solution, encodes directly into MP4 using Dicas' MPEG-4 codecs
- Envivio's 4Coder encodes to MPEG-4 AAC, ASP/SP and AVC
4) the following tools can be used for editing (eg splitting, appending) MP4 files:
- MP4Box splits MP4 (AAC, ASP, AVC, TTXT, MPEG-1/2, private streams) time- and sizewise, extracts parts and joins MP4 too
- Ulead's VideoStudio 8 with the MPEG-4 plugin
- Quicktime/Pro can be used to create (encode, mux) and edit (split, join) MP4 files, supports MPEG-4 Simple Profile, AAC, AVC
- In contrary to MPEG-1/2 the MPEG-4 Standard defines its own Text Stream/Subtitle Format under ISO 14496-17 (MPEG-4 Timed Text). Its text-based (not picture-based as DVD subs), uses UTF-8/16 (including cyrilic, arabic, chinese characters...), allows all sorts of animations (scrolling, colors, karaoke aso...), allows attaching of fonts and is streamable
MPEG-4 TTXT creation is already supported in MP4Box and Quicktime. Playback works with Haali's MP4 dshow Parser, Osmo4, Realplayer and Quicktime (only when placed in .3gp). Support in VideoLan is worked on
- Nero's Recode2 makes use of the possibility of storing private streams in MP4 (question 4) and simply takes the DVD subpicture streams (vobsubs) from the DVD and stores them in MP4 without modifications
playback of these streams works with the Nero filters, Haali's Parser, VideoLan (buggy) and some Hardware MP4 DVD Players
- another way covered by the MPEG-4 Standard for storing subs in MP4 is to translate the subtitle format into the XMT/BT text format and then to encode it as "advanced content" (BIFS binary format) in MP4. this should work with every subtitle format available (use mp4box or dvd2mp4gui for example to convert SRT subs)
for playback of advanced content you need a filter/player that can handle this kind of content (How to play MP4 files)
atm you can place chapters in MP4 automatically during encoding with Nero Recode2 or, on existing files, with MP4Box and the MP4 directshow muxer filter coming with Recode2 (usable in graphedit for example)
As players, not knowing the info stored in udta, will simply ignore it, you need a player/filter that can handle this special chapters info (Nero's and 3ivx' dshow filter handle this atm)
There are already some GUI oriented MP4 authoring tools existing, which should make this process much easier, by IBM, AVIPIX, Envivio, Mindego, iVAST, Digimax or Etri, but all not available to the broad public
So if you want to try MP4 authoring yourself there is no way around writing an own BT/XMT script atm
thanks to the GPAC guys there is already a great Tutorial available in english and french, which can help you a lot with this
to make this process much easier i created a sample script, called MP4Menu, which already supports all features a DVD Menu offers and you will only have to adopt to your needs.
if you want to know what for example can also be possible in MP4, check out this small in-video menu sample, but MPEG-4 Systems also offers things like animations in 2 and 3 Dimensions (think of movies like Toy Story or Finding Nemo), check out this small systems 2D animation sample (doesnt include a video stream!)
Make sure you use a Systems capable Player, like the GPAC Osmo4 player, to watch the samples.
1) on the video bitstream level: maybe the most practicable and widely used method atm
use a codec which allows the setting of a PAR (eg 3ivx, ffmpeg/ffvfw, XviD), than mux the resulting AVI into MP4 by using any MP4 muxer (for example the 3ivx mp4 muxer, MP4Box or mp4UI). On already existing bitstreams you can change the PAR on MPEG-4 ASP with Moitah's MPEG4 Modifier, on AVC with hhanh's ARChange.
Anamorphic playback also already works automatically with VideoLAN, MPlayer or on DShow with the 3ivx, Nero or Haali parsers together with the XviD (set AR to auto), 3ivx (enable "force overlay"), Nero or ffdshow (enable "overlay mixer") decoder filter
2) on the container level: MP4 offers the possibility to use a "Composition Matrix", which allows the changing of the AR, rotating of the picture, multilayer handling, mixing of two audio streams during playback (eg seperated speech and music in movies)...Quicktime/Pro already offers many composition possibilities via Movie -> Get Movie Properties -> Video/Sound Track -> Size/Layer/Volume/... and the correct playback of such composed MP4 files
3) on the BIFS level: which allows the broadest range of authoring possibilities (question 9) and therefore also changing of the AR (eg via Transform2D.scale)
for creating and playing with this BIFS control stream you can use the GPAC project tools: MP4Box and Osmo4
1) if you want to play any multimedia file in a directshow based player (for example media player classic, bsplayer, zoomplayer or the crappy windows media player) first of all two things are needed for playback:
- a splitter/parser filter, which splits up the container file into the included streams (audio, video, subtitles) during playback
- a decoder filter, which decodes the encoded streams (for example ffdshow, 3ivx, CoreAAC)
btw. the only reason why you don't have to install a splitter for AVI is that it is already installed on windows by default
there are some packages available, which include such filter:
- Haali's Media Splitter supports AAC, ASP, AVC, MPEG-1/2 Audio (eg MP2/MP3) and Video, TTXT, VobSubs and Chapters in MP4, audio/video/subtitle streams switching, is unlimited, partly opensource and freely available
- 3ivx is useable, like Haali's Splitter, with alternative decoders like ffdshow or CoreAAC, make sure that the "allow unsupported decoders" option is ticked in the 3ivx splitter. 3ivx handles MP2, MP3, AAC, ASP, chapters but NOT AVC in MP4!
- Moonlight offers a MP4 splitter able to handle MP3, AAC, ASP and AVC, together with ASP/AVC decoders
- Nero Vision Express supports AAC, ASP, AVC, VobSubs and Chapters (subs/chapters/multiple audio only in Nero Showtime!)
- HDX4 supports AAC and ASP in MP4
- EnvivioTV handles advanced content/BIFS (sample), AAC, ASP and AVC in MP4
- Dicas' mpegable, supports RTP streaming, AAC, ASP and AMR (.3gp audio) decoding
- Ligos' LSX-MPEG, Simple Profile Video only, Demo limited to 30 days
2) if you can't/won't use windows-centric directshow you can also use one of the following players for playing MP4:
- Apple's Quicktime, widely used, supports MPEG-4 SP, AVC and AAC
- Realnetworks' RealPlayer 10, supports MPEG-4 SP + B-frames (only in the "plus" version) and AAC
- GPAC's Osmo4, opensource (GPL), able to decode BIFS (subtitles, user interactivity, dvd-like menus...) and TTXT
- MPlayer, opensource (GPL), handles ASP and AVC, available for various OSes, like Win, Linux, Mac OS...
- Videolan, opensource (GPL), handles ASP and AVC, various OSes supported
- MPEG4IP's WMP4Player, opensource (LGPL/MPL), handles ASP and AVC
- xine, opensource (GPL), various OSes supported
- Philips' Platform4 Player, also offering a player for PocketPC
- ENST's Osmo4 (based on the reference source), able to decode advanced content/BIFS
- IBM's M4Play included in their Toolkit for MPEG-4, Java-based, able to decode advanced content/BIFS
3) hardware/standalone players: As the most popular MPEG-4 ASP implementations (DivX5/XviD) were often placed in AVI, thats the most widely supported format on hardware players atm. Still the guys from Nero, together with increasing consumer demand, try to push support for the MP4 container on hardware players, with increasing success:
The first MP4 (only ASP video support, not AVC) enabled hardware players are:
- KiSS's DP-558 supports AAC, ASP (no QPel/GMC), VobSubs and Chapters in MP4 (Sigma Designs EM8560)
- Siemssen's SCO 5000ND supports AAC (no MP3/MP2), ASP (QPel/1WP GMC), VobSubs, Chapters in MP4 (Zoran Vaddis 776)
- RJTech's RJ 1500DVX II supports AAC (5.1), MP3, ASP (QPel/1WP GMC/CQM), VobSubs, Chapters in MP4 (ESS Vibratto-II)
- Xoro's HSD 415/310 players supports AAC, ASP (no QPel/1WP GMC?), VobSubs, Chapters in MP4 (ESS Vibratto-II)
- Tevion DR 2004, supports AAC, ASP (QPel/GMC?) and VobSubs in MP4
- Tevion DVD-4000, with latest firmware supports ASP (no QPel/1WP GMC?) and MP3 audio (no AAC) in MP4 (ESS Vibratto-II)
- Philips's DVP630/632/642 players support ASP (no QPel/1WP GMC) and MP3 audio (no AAC) in MP4 (ESS Vibratto-II)
- M$'s Xbox with the Xbox Media Center is supposed to support MP2, MP3, AAC and AVC in MP4
- Sony's Playstation Portable (PSP) is supports AAC, SP (from Memory Stick) and AVC (from UMD) in MP4
- Apple's iPod supports AAC in MP4 (no video)
- Sigma Designs' Xcard handles hardware MPEG-4 decoding on PC including the MP4 container
As streaming server you can use Apple's Darwin Streaming Server, Guides on the process are available from everwicked and linuxjournal
for live streaming you can use mp4live from MPEG4IP (only for Linux), Guides from everwicked and MPEG4IP; on Windows you can use MPEGRecorder (which seems to be a port of mp4live)
Also have a look at the tools from ViTooKi (opensource Streaming Server, Player and other tools)
Another free and opensource MP4 streaming server is available from Cata
for MAC you can have a look at Live Channel
To play broadcasted MP4 files you can use Apple's Quicktime, Real's RealPlayer 10, MPEG4IP's WMP4Player, Dicas' mpegable and EnvivioTV (both work in dshow players) and GPAC's Osmo4 (the last two also support streamed advanced content/user interactivitiy)
the fact why it is often not used is probably caused by the crappy avi container, which (apart from many other issues) is not able to store real variable framerate streams
now with the upcoming of far more advanced containers (like matroska and mp4) able to handle vfr, i think we should explore the possibilites of vfr far more (and i hope i can encourage some people to help me with my currently ongoing testings):
as avi and vfw (used by xvid and divx5 in vdm) doesnt seem to be able to handle vfr at all there are two possibilities to create vfr mpeg-4 video streams
1) use a directshow encoder, like the one from 3ivx (suxendrol is already working on a xvid dshow enc filter) and dont even think about touching crappy outdated avi and vfw
2) use vfw encoders, with avi workarounds, and do some smart avi->mp4 conversions:
the xvid encoder already offers the so called "frame drop ratio":
with a setting of 0 only frames get dropped which are 100% the same as the frame before, with 100 all frames are dropped
this option's name and description might be a little misleading as the frames arent really totally dropped (leading to vfr or less frames in the output), also a value of 50 doesnt mean that every second frame is affected -> frames get simply not coded, which means so called n-vops are placed in the stream
this value specifies a threshold under which, if the difference of the following frame to the previous frame is below or equal to this threshold, a frame gets not coded (see it could even be that you set a value of 50, but still no frame gets not coded)
on the decoder side, everytime when the decoder comes to such a n-vop, it displays the frame before the n-vop again, therefore its some sort of "fake vfr stream"
now with the 3ivx or MP4Box MP4 muxers these n-vops can be really dropped and real vfr mpeg-4 streams get created inside the mp4 container
small howto
1) open your to-encode source in virtualdubmod
2) go to the xvid codec settings and choose the settings as you wish and set the frame drop ratio you want to use (on my testssample it seemed that i didnt get a dropped frame with a value of 15, with 90 there were (of course) far too much dropped frames) also it seems that the frame drop ratio seems to work better when packed bitstream (b-frames) is disabled
now you can decide whether you want to use the commandline MP4Box or the 3ivx directshow muxer to create a VFR MP4 file:
MP4Box:
3) download mp4box from rarewares
4) run the following commandline:
3) install 3ivx
4) open the resulting avi in graphedit
5) remove all filters, except the source and the avi splitter, load the 3ivx mp4 muxer and the file writer (*.mp4) and connect the 3ivx muxer between avi splitter and file writer
6) enable the "compress n-vops" option in the 3ivx muxer and push play
you should now be able to enjoy and play the created variable frame rate mp4 file with every mpeg-4 decoder (xvid, divx5, ffdshow, 3ivx etc...) in any dshow based player without loosing sync with the audio aso...
edit:
note that during my testings i found that the frame drop ratio was broken in the current release candidates (koepis rc3). syskin now fixed some bugs but it seems that there are still "unfixable" problems when using it with b-frames
therefore the frame drop option in xvid will only work without b-frames from now on
edit2: added gpac's mp4box, as it offers exactly the same features as 3ivx now too
1) xvid (tested rc4, till rc3 no n-vops were created with the default settings (framedrop ratio on 0)!!)
- when b-frames are disabled xvid will create n-vops even if the framedrop ratio is set to 0 (which means it only drops frames that are 100% the same) a higher value will lead to that even frames which are not 100% the same can be dropped
- with b-frames enabled it doesnt output n-vops
2) 3ivx (4.5.1) outputs n-vops
3) nero digital (first version), with and without b-frames, outputs n-vops
4) divx5 (5.1.1) doesnt output n-vops
5) ffvfw (20031117) doesnt output n-vops with the default settings (i didnt find an option which could enable n-vops creation)