vf_scale: replace ancient fallback image format selection

If video output and VO don't support the same format, a conversion
filter needs to be insert. Since a VO can support multiple formats, and
the filter chain also can deal with multiple formats, you basically have
to pick from a huge matrix of possible conversions.

The old MPlayer code had a quite naive algorithm: it first checked
whether any conversion from the list of preferred conversions matched,
and if not, it was falling back on checking a hardcoded list of output
formats (more or less sorted by quality). This had some unintended side-
effects, like not using obvious "replacement" formats, selecting the
wrong colorspace, selecting a bit depth that is too high or too low, and
more.

Use avcodec_find_best_pix_fmt_of_list() provided by FFmpeg instead. This
function was made for this purpose, and should select the "best" format.

Libav provides a similar function, but with a different name - there is
a function with the same name in FFmpeg, but it has different semantics
(I'm not sure if Libav or FFmpeg fucked up here).

This also removes handling of VFCAP_CSP_SUPPORTED vs.
VFCAP_CSP_SUPPORTED_BY_HW, which has no meaning anymore, except possibly
for filter chains with multiple scale filters.

Fixes #1494.

video/img_format.c

@@ -19,6 +19,7 @@
 #include <assert.h>
 #include <string.h>
 
+#include <libavcodec/avcodec.h>
 #include <libavutil/pixfmt.h>
 #include <libavutil/pixdesc.h>
 
@@ -263,6 +264,23 @@ int mp_imgfmt_find_yuv_planar(int xs, int ys, int planes, int component_bits)
     return 0;
 }
 
+#if LIBAVUTIL_VERSION_MICRO < 100
+#define avcodec_find_best_pix_fmt_of_list avcodec_find_best_pix_fmt2
+#endif
+
+// Compare the dst image formats, and return the one which can carry more data
+// (e.g. higher depth, more color components, lower chroma subsampling, etc.),
+// with respect to what is required to keep most of the src format.
+// Returns the imgfmt, or 0 on error.
+int mp_imgfmt_select_best(int dst1, int dst2, int src)
+{
+    enum AVPixelFormat dst1pxf = imgfmt2pixfmt(dst1);
+    enum AVPixelFormat dst2pxf = imgfmt2pixfmt(dst2);
+    enum AVPixelFormat srcpxf = imgfmt2pixfmt(src);
+    enum AVPixelFormat dstlist[] = {dst1pxf, dst2pxf, AV_PIX_FMT_NONE};
+    return pixfmt2imgfmt(avcodec_find_best_pix_fmt_of_list(dstlist, srcpxf, 0, 0));
+}
+
 #if 0
 
 #include <libavutil/frame.h>