JustDVB-IT 2.0 15/6/2006

Introduction:
JustDVB-IT is a MPEG2 transport stream data generator and packet manipulator.
It can be used to:
- generate many of the data carried by the transport stream, almost everything but audio, video and teletext, that's because justdvb-it IS NOT AN MPEG AUDIO/VIDEO ENCODER.
- manipulate transport streams stored on the hard disk.
- cypher and decypher transport streams
- multiplex constant bit rate transport streams
- output to DVB-ASI with pci hardware boards.
- output to DVB-S, DVB-T and DVB-C with pci hardware boards.
It doesn't encode audio and video streams, they are expected as input files.

Change log 2.0:
SciTE help included
Bluetop BSTC-12 DVB-ASI output: tested
Dektec DTA110T DVB-T modulator support: tested
MHEG5 DSMCC carousel generation: tested
Fixed support for python 2.4, currently both python 2.3 and 2.4 are working properly
Transport stream successfully tested against StreamAnalyzer of Panasonic from MHP KBD
Sample configuration timings successfully tested against TR-101i 290 with Dektec StreamXpert 
Removed debug prints
tools/tscrypt: added
extras/cwtest: added
extras/javasrc: added java source code of the example applications
extras/mainconcept: added a script to manage mainconcept output
bash-config/: added application storage option for MHP 1.1.2
bash-config/: added Yambo 2.0 example applications
bash-config/: added Test ste example application
SciTE help included
python-config/: new python only config, bash will phase out...
python-config/: it is possibile to specify module id for each file or only for some, check the .descriptors example
python-config/: added datacarousel example
tools/i3942ts: added
tools/tspcrmeasure: bug fix on print
tools/tsloop: cc fix
tools/zpipe: added
tools/DtPlay: removed, now it is necessary to get it from Dektec web site
tools/lib/DTAPI: removed, now it is necessary to get it from Dektec web site
libs/: added application storage descriptor (MHP 1.1.2)
libs/: added parental rating descriptor
libs/: added logical channel number descriptor
libs/: added support for compressed DSMCC modules
added support for very large files (>2GB)
manual.txt: Video and audio multiplexing revisited, renamed readme.txt (people miss it...)

Change log 1.2:
dvbobjects/STE.py: addded Stream Event Descriptor Section
dvbobjects/STEO.py: addded Stream Event Object
dvbobjects/DSMCC: addded Stream Event Object with Do-It-Now support
config/: stream event example added
tools/tsnullshaper: added files loop to reduce latency

Change log 1.1.1:
Just a very cosmetical fix

Change log 1.1:
Fix a JustDvb-It bug about bad string termination.

Change log 1.0:
DtRecord: removed, if u need it u can use the official from Dektec
EIT: bug fix
tstdt: added new, generates gmt time from server clock and restamp TDT packets
docs: manual.txt added
tools/tsloop: support multiple input, removed buffering, good for transport scheduling
config: shows an example of cbr muxing and multicarousel from MainConcept's input

Change log 0.5:
tools/tsnullshaper: bug fix        
tools/oc2sec: buf fix on debug prints 
tools/sec2ts: bug fix on .ts length
psi tables python look: same semantic different syntax
dvbobjects/dvbobjectsPMT.py: changed program_loop in stream_loop
config: added 
tools/data2sec: renamed oc2sec
tools/esaudio2pes: added
tools/esaudioinfo: added
tools/esvideo2pes: added
tools/esvideoinfo: added
tools/pes2es: added
tools/pes2ts: added
tools/pesinfo: added                                                                 
tools/ts2pes: added
tools/tscbrmuxer: added
tools/vbv: added

Change log 0.4:
moved to kernel 2.6
everything can be run without root privileges
new directories structure
tools/tsfilter: added new
tools/tspidmapper: modified input syntax 
tools/data2sec: rearchitected
config/: added
extern/: removed
set-ups/: removed
drivers/dvbcharts: removed

Chane log 0.3.2:
minor buf fixes

Chane log 0.3.1:
minor buf fixes
tools/sec2tspackets: renamed sec2ts 

Change log 0.3:
tools/ip2sec: added new
tools/tspidmapper: added new
tools/tspcrmeasure: added new
set-ups/iptrafficdemo: added new
tools/tsloop: enhanced with variable buffer support
tools/tsmodder: enhanced with variable buffer support
tools/tsfiller: enhanced with variable buffer support
tools/tsnullshaper: enhanced with variable buffer support, read packets to susbtitute only if they are read
extern/dvbasi/DtPlay-custom: DtPlay enhanced with variable buffer support
extern/dvbasi/DtRecord-custom: DtRecord enhanced with variable buffer support
extern/dvbasi/DtPlay: removed
extern/dvbasi/DtRecord: removed

Change log 0.2:
tools\tsnullshaper: added time insertion
tools\dvbdata: more bug fixes
tools\data2sec: added new MPEG and DVB descriptors, modules versioning from input parameters
set-ups\cindemo: added multiple application ait example and fixed time of tables insertion

Glossary:
AIT: Application Information Table (ETSI TS 102 812)
CBR: Constant Bit Rate
DVB: Digital Video Broadcasting (http://www.dvb.org/)
DVB-ASI: DVB-Asynchronous Serial Interface (EN 50083-9)
DSM-CC:  Digital Storage Media Command and Control (ISO/IEC 13818-6, EN 301 192)
IP: Internet Protocol (rfc 791)
MHP: Multimedia Home Platform (ETSI TS 102 812) 
MPE: Multi Protocol Encapsulation (ISO/IEC 13818-6, EN 301 192)
MHEG5:   (ISO/IEC 13522-5)
MPEG: Motion Picture Expert Group (http://multimedia.telecomitalialab.com/)
NIT: Network Information Tablet  (ISO/IEC 13818-1)
PAT: Program Association Table  (ISO/IEC 13818-1)
PID: Program IDentifier  (ISO/IEC 13818-1)
PMT: Program Map Table  (ISO/IEC 13818-1)
PTS: Presentation Time Stamp (ISO/IEC 13818-2)
PSI: Program Signalling Information (ISO/IEC 13818-1)
SPTS: Single Program Transport Stream (ISO/IEC 13818-1)
TS: Transport Stream (ISO/IEC 13818-1)
VBR: Variable Bit Rate
PES: Program Elementary Stream (ISO/IEC 13818-1)
ES: Elementary Stream (ISO/IEC 13818-1)

Features:
PSI generation (PAT, PMT, NIT, SDT, TDT, AIT, ...)
Object carousel generation (MHP, MHEG5, OTA, ...)
Data carousel generation
IP encapsulation (MPE)
Scrambling (ECD)
Null packet replacement
CBR transport stream muxing
PID filtering
PID remappaing
EPG/EIT generation
Null packet uniform insertion

Analisys:
ES to PES encapsulator
PES to ES decapsulator
TS to PES decapsulator
PES to TS encapsulator
Video buffer verifier
PCR timing verifier
PES headers analyzer
Video and audio ES header analyzer

DVB hardware tested from us:
Bluetop BSTC-12
Dektec Dta-100
Dektec Dta-107
Dektec Dta-110t
Dektec Dta-140

How does it work:
JustDVB-IT is programmed with 2 languages: python and C. Python is used for data generation, PSI, carousels and so on, while C is used for transport stream manipulation purpose. C tools use fifos for process communication between them.
Python tools generate transport stream data, C tools mux the generated data from input transport streams carrying audio, video and teletext generated by third parties encoders.
JustDVB-IT doesn't encode audio, video and teletext.

How JustDVB-IT is used:
Typical generated data are MHP/MHEG5/OTA carousels, PSI tables/EPG and TCP/IP encapsulation.
In a broadcasting environment justdvb-it server is connected to a multiplexer input to generate some or all the data for the multiplexed transport stream.
In a modulator laboratory justdvb-it is connected directly to a modulator carrying out also cbr multiplexing, the modulator can be also a pci modulator.

The directory structure:
/tools: C tools to generate and manipulate the TS bitstream
/libs: common lib for the tools and the scripts
/bash-config: example scripts using the tools and the libs with bash scripting
/python-config: example scripts using the tools and the libs with python scripting

How to compile and install JustDVB-IT: (tested in a Debian stable Linux environment)
This packets can be required: python-dev, libpcap-dev e zlib1g-dev
cd JustDVB-IT-2.0/libs/
make install
cd ../tools
make install 

How to use SciTe:
We really suggest you to use Scite (http://www.scintilla.org/SciTE.html) to edit python psi tables, folding and unfolding of structure is very usueful.
Open psimain.py, keep pressed ctrl and left click to rescursevly fold structures, later on just left click to unfold only one level and get a hiearchical view.

MPEG2 Transport Stream basics:
TS is made by packets, a packet is the minimum unit that cannot be splitted. 
A packet has a fixed length of 188 bytes and start with 0x47.
Every packet has an unique PID. A sequence of packets of the same PID is a PID stream.
On DVB networkes every TS has a stream with PID 0 that brings the PAT, a PID stream 16 that brings NIT, a PID stream 17 that brings SDT and optinally a PID stream 18 that brings EIT (aka EPG).
PAT and SDT define Programs, every Program has PMT, every PMT has a PID stream.
Every PMT has many PID stream for audio, video and data.
PAT, PMT, SDT, ... are all PSI tables and are defined into python scripts that generates the PSI TS packets from their textual description reported into the referenced standards.
Any DVB TS is going to use PSI, so PSI are important in any use of JustDVB-IT, the following lines reminds their bit rates from the standards: 
PAT 37600bps from (ISO/IEC 13818-1) it means every 0,040 seconds
PMT 37600bps from (ISO/IEC 13818-1) it means every 0,040 seconds
AIT every 10 seconds from (ETSI TS 102 812), it means 150,4bps * (number of packets)
SDT every 2 seconds from  (EN 300 468), it means 752bps * (number of packets)
NIT every 10 seconds from (EN 300 468), it means 150,4bps * (number of packets)
The .ts file generated are use in all the following case studies.

How to add an object carousel to a TS using bash config
# generate carousel from 3 directories: ocdir1, ocdir2, ocdir3
config/oc-update.sh ocdir1 0xB 1 2002 1 # <<<<<<<<<< paramenters are referenced by PMT and AIT
config/oc-update.sh ocdir2 0xC 1 2003 2 # <<<<<<<<<< paramenters are referenced by PMT and AIT
config/oc-update.sh ocdir3 0xD 1 2004 3 # <<<<<<<<<< paramenters are referenced by PMT and AIT
# "config/oc-update.sh" is a shell script to generate a .ts file with an object carousel generated from the following parameters: 
# 1 - carousel directory name
# 2 - association aka common tag (referenced by PMT and AIT into "config/psi-config.py")
# 3 - modules version
# 4 - pid (referenced by PMT into "config/psi-config.py")
# 5 - carousel id
# "config/oc-update.sh" uses "tools/oc2sec/file2mod.py" and "tools/oc2sec/mod2sec.py" that both use "lib/dvbobjects" and "lib/sectioncrc" libraries to generate object carousels and data carousels.
# launch many tsloop that will repeat the same TS fixing cc errors write the TS in their own FIFO
/usr/local/bin/tsloop config/ocdir1.ts > config/fifoocdir1.ts &
/usr/local/bin/tsloop config/ocdir2.ts > config/fifoocdir2.ts &
/usr/local/bin/tsloop config/ocdir3.ts > config/fifoocdir3.ts &
# tscbrmuxer: muxes carousels at the bitrate specified, the result is a TS with the bitrate equals to the addition of all the bitrates
/usr/local/bin/tscbrmuxer b:139872 config/fifoocdir1.ts b:645096 config/fifoocdir2.ts b:215032 config/fifoocdir3.ts > config/fifooc.ts &
# tsloop repeat an input TS file, as a basic example a TS with a video stream and some null packets is repeated, for informations about how to generate offline an input.ts see the next case study 
/usr/local/bin/tsloop config/input.ts > config/fifoinput.ts &
# tsnullshaper: changes dynamically input file null packets with the PSI and the muxed object carousel
/usr/local/bin/tsnullshaper config/fifoinput.ts +config/fifooc.ts t:40 config/pat.ts t:40 config/pmt.ts t:100
0 config/ait.ts t:500 config/sdt.ts t:500 config/nit.ts > config/fifoshaped.ts &
# the playout is driven by the input bit rate
/usr/local/bin/DtPlay config/fifoshaped.ts -r 6000000 # <<<<<<<<<<<< output bit rate fixed from input.ts

How to build a TS with video and audio input files
Audio and video TS can be muxed with tscbrmuxer if they are CBR TS and their bit rate is known, so all this section is about how to known the bitrate and encapsulate them.
Audio: Audio can be generate from any encoder, free and GPL are working enough good, it is just necessary to pay attention on some parameters to encapsulate ES into PES and later into TS.  The number of samples contained in a audio frame is constant and always 384 samples for layer I and 1152 for layers II and III.
Layer I frame es size is so ((12 * BitRate / SampleRate + Padding) * 4) byte while layer II and III frame es size is (144 * BitRate / SampleRate + Padding) byte.
Audio frame pes size is frame es size + 12 byte and ts size is (((frame pes size / 184) + (frame pes size % 184 != 0) + (frame pes size % 184 == 183)) * 188) bytes.
Audio frame PTS step for layer I is (384 / SampleRate) second while audio frame PTS step for layer II and III is (1152 / SampleRate) second.
Audio frame PTS step * 90000 is PTS step in ticks.
Audio TS packets can be muxed with tscbrmuxer knowing their ts bitrate because they are CBR by nature.
An ES video to be played correctly by a MPEG2 hardware decoder needs to pass the Video Buffer Verifier (VBV) test program, vbv tries to guess if the video will over or under go the hardware decoder buffer, if the video stream doesn't pass the vbv test it won't play correctly for sure on the cheapest MPEG2 hw decoders.
Video ES is the only kind of ES that can be Variable Bit Rate (VBR) or Constant Bit Rate (CBR), if it is VBR it can lead after its encapsulation to generate a VBR Transport Stream (TS).
VBR Video ES means that a Group Of Picture(GOP) may have a different bit rate from the others GOP, CBR Video ES means all GOPs have the same bit rate.
A VBR TS can be made a CBR TS with null packet insertion, a feature that JustDVB-IT supports with tsvbr2cbr tool.
In a digital video broadcasting architecture VBR makes sense if multiplexers and encoders talk to each other and realize a statistical multiplexing, on the other hand using a VBR encoding without a statistical multiplexing would be just a waste of band.
As far as we know there is not a standard about statistical multiplexing and every hardware vendor has its own lock-in protocol between encoder and multiplexer.
The MainConcept MPEG2 software encoder offers as outuput a CBR TS with VBV compliant Video ES for a good price, more over is available as stand alone or as a plug-ing for Adobe Premiere and on many platforms so it suggested for encoding video stream for JustDVB-IT.
Check extras directory to know more on how to use Mainconcept's encoder to generate a input transport stream for Justdvb-it.
It is worthless to say that any professional broadcasting industrial MPEG2 encoder hardware will be also suitable.
Some free or GPL mpeg2 encoders, like VLC using libavcodec from ffmpeg, can output a VBR TS with a VBR ES that can be filled to be a VBR ES with a CBR TS, however these encoders' target are usually DVD players and bit rate is too variable to be usefull for broadcasting without a big waste of bandwith, however you can give them a try because they often improve by time so maybe at the time of this writing they are just fine.
