nsvdec: Be more careful with av_malloc().

Check results for av_malloc() and fix an overflow in one call.

Related to CVE-2011-3940.

Based in part on work from Michael Niedermayer.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit 8fd8a48263ff1437f9d02d7e78dc63efb9b5ed3a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit be524c186b50337db64d34a5726dfe3e8ea94f09)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 87007519c81c37d8a3de424de3db14078ae84333)

Conflicts:

	libavformat/nsvdec.c
1 file changed