[mp4-tech] inverse transforms data range

[Gary Sullivan]

Robin et al,
Encoders can actually rather easily ensure that those constraints are
followed (and using only 16-bit arithmetic in the encoder as well),
although there are some subtleties in the details that are important for
implementers to understand.
Recommended reading includes:
1. H. Malvar, A. Hallapuro, M. Karczewicz, and L. Kerofsky,
Low-complexity transform and quantization in H.264/AVC, IEEE
Transactions on Circuits and Systems for Video Technology, vol. 13, pp.
598-603, July 2003. (found at http://research.microsoft.com/~malvar/).
2. The VCODEX/Richardson tutorial information about the transform at
http://www.vcodex.com/h264.html.
3. The JVT's JM reference encoder software.
4. The JVT's JM text algorithm description.
5. Every single detail in the video codec standard itself (the latest
version that you can obtain).
Best Regards,
Gary Sullivan
+> -----Original Message-----
+> From: Robin Zoo [mailto:robin94539 yahoo.com] 
+> Sent: Friday, September 16, 2023 1:23 AM
+> To: Gary Sullivan; mp4-tech lists.mpegif.org
+> Subject: RE: [mp4-tech] inverse transforms data range
+> 
+> Dear Gary,
+> 
+> Thank you for your reply.
+> 
+> What I meant was, for BitDepthy=8,
+> the bounds of input data(dij), intermediate data(eij ~
+> mij) and final result(rij) of the 8x8 inverse
+> transform is -2^15 ~ 2^15-1, inclusive, which means no
+> data overflow by using 16-bit 2's complementary
+> arithmetic.
+> 
+> Since I can't image how the encoders guarantee such an
+> overflow won't happen by using 16-bit input(don't tell
+> me that they limit input data within 10 bits :), I
+> still don't know 
+> 1) whether the hardware can be reduced further;
+> 2) and, whether a simulation mis-match is caused by
+> hardware implementation error or incorrect bit stream.
+> 
+> thanks in advance!
+> 
+> Robin
+> 
+> --- Gary Sullivan <garysull windows.microsoft.com>
+> wrote:
+> 
+> > 
+> > Robin et al,
+> > 
+> > I'm not sure I understand your question.
+> > 
+> > There is no such statement in 8.5.10 or 8.5.11
+> > exactly saying that the
+> > "8x8 or 4x4 inverse transform is performed on 16-bit
+> > integer data",
+> > although I think I agree with the spirit of that
+> > statement.
+> > 
+> > Regarding how to guarantee that the constraints
+> > expressed in 8.5.10 or
+> > 8.5.11 are fulfilled, I call your attention to 3.132
+> > which says that
+> > when a "mandatory constraint [is expressed] on the
+> > values of syntax
+> > elements or on the results obtained by operation of
+> > the specified
+> > decoding process, it is the responsibility of the
+> > encoder to ensure that
+> > the constraint is fulfilled."
+> > 
+> > Thus if the bounds expressed in 8.5.10 or 8.5.11 are
+> > exceeded when
+> > decoding the data, the bitstream is not considered a
+> > proper, conforming,
+> > bitstream.  A decoder can do whatever it wants in
+> > response to an
+> > incorrect bitstream.
+> > 
+> > Best Regards,
+> > 
+> > Gary Sullivan
+> > 
+> > +> -----Original Message-----
+> > +> From: mp4-tech-bounces lists.mpegif.org 
+> > +> [mailto:mp4-tech-bounces lists.mpegif.org] On
+> > Behalf Of Robin Zoo
+> > +> Sent: Thursday, September 15, 2023 10:53 AM
+> > +> To: mp4-tech lists.mpegif.org
+> > +> Subject: [mp4-tech] inverse transforms data range
+> > +> 
+> > +> As specified in subclause 8.5.10/11 of H.264
+> > spec, the
+> > +> 8x8 or 4x4 inverse transform is performed on
+> > 16-bit
+> > +> integer data.
+> > +> 
+> > +> Could someone briefly explain to me how the
+> > encoded
+> > +> bit stream and dequantization guarantee that
+> > there is
+> > +> no data overflow since the overflow can't be
+> > avoided
+> > +> if the only constrain is that the input
+> > +> data(dequantized transform coefficient) is
+> > bounded
+> > +> within -2^16 ~ 2^16-1(for 8-bit video)?
+> > +> 
+> > +> By the way, how to simply generate test stream to
+> > +> maximize test coverage of the inverse transform
+> > though
+> > +> the transform operation is quite simple?
+> > +> 
+> > +> 
+> > +> 
+> > +> 
+> > +> 		
+> > +> __________________________________ 
+> > +> Yahoo! Mail - PC Magazine Editors' Choice 2005 
+> > +> http://mail.yahoo.com
+> > +> _______________________________________________
+> > +> NOTE: Please use clear subject lines for your
+> > posts. Include 
+> > +> [audio, [video], [systems], [general] or another 
+> > +> apppropriate identifier to indicate the type of
+> > question you have.
+> > +> 
+> > +> Note: Conduct on the mailing list is subject to
+> > the 
+> > +> Antitrust guidelines found at 
+> > +>
+> >
+> http://www.mpegif.org/public/documents/vault/mp-out-30042-Ant
+> > +> itrust.php
+> > +> 
+> > 
+> 
+> 
+> __________________________________________________
+> Do You Yahoo!?
+> Tired of spam?  Yahoo! Mail has the best spam protection around 
+> http://mail.yahoo.com 
+>