Christopher's Broom Cupboard header image

Canon XF305 MXF problems in Premiere? Transcode your clips to ProRes with FFMPEG!

Recently encountered a weird problem with 1080i MXFs straight out of a Canon XF305. The files would play in VLC, but Premiere Pro CC 2017 on a brand new MacBook Pro or iMac failed soon after starting to decode the video, with a horrible red frame and MXF frame decode errors in the Log. It's pretty nasty.

Interestingly, an older Mac Pro (a silver tower running Yosemite 10.10.5 with an ATi 5770 1GB video card) running Premiere Pro CC 2017 did not have any issues decoding the MXFs, so I wonder if there's some additional / different version Pro Video Codecs installed on it...

I fired up MediaInfo to check the file. (If you don't have it, download the 64-bit GUI without installer from the MediaInfo site).

All looked to be OK:

Input #0, mxf, from 'E:\scratch\test\CLIPS001\XF0007\XF000701.MXF':
Metadata:
uid : 05f46a47-8205-4800-80aa-960001e012e3
generation_uid : 05f46a47-8205-4800-812a-960001e012e3
company_name : CANON
product_name : XF305
product_version : 1.00
product_uid : 060e2b34-0401-010d-0e15-005658460000
modification_date: 2018-05-09T02:10:05.000000Z
material_package_umid: 0x060A2B340101010501010D43130000000A05F46B478205800000850001E012E3
timecode : 09:51:05:02

So why is the file failing to work in Premiere?

This Canon MXF decode problem has been around for years: Google "Premiere MXF red frame". Lots of Canons seem to produce problematic MXFs for Premiere. (1, 2, 3, 4...)

The MXFs produced by this Canon XF305 contain MPEG-2 inter-frame videos, lossily compressed to produce I, B(idirectional predicted) and P(redicted) frames grouped as "GOPs" (Groups of Pictures). The highest quality MPEG-2 files you can produce are intra-frame, where the file is a stream of intra frames, each one being one complete picture, but these yield larger file sizes.

Whatever; we can deal with this - let's transcode to a proxy format, for example Apple ProRes 422 HQ. Easy, Premiere likes that, and there's negligible generational quality loss.

However, what happens if you attempt to do this natively in Premiere or After Effects is that you get the same result as attempting to import the MXFs to use them directly: failed encodes, frame decode errors, truncated files, pain and suffering. Yuck.

So, off to trusty FFMPEG!

The following guide assumes you're using Windows, although it's trivial to modify the instructions for Mac or Linux.

Download the latest stable FFMPEG from the FFMPEG web site and place it in a suitable location. No installation required as long as you grab the 'static' version (with all dependencies included in the main executable).

If your MXFs are gathered in a folder or folder structure, no problem. Make a new batch file (.bat) containing the following (obviously Windows only):

for /R %%f in (*.mxf) do (
D:\path\to\ffmpeg.exe -fflags +genpts -i "%%f" -pix_fmt yuv422p10le -vendor ap10 -top -1 - 
flags +ilme+ildct -qscale:v 8 -map 0 -c:v prores_ks -profile:v 3 -c:a copy "%%~df%%~pf%%~nf- 
prores422hq.mov" %*
)

So what does this do?

The for loop with the /R switch recursively looks through all files in the directory the script is run from. The caveat is that it assumes the start point is the directory the batch script is run from; this could be improved into an argument (so you could specify a different starting folder).

Once inside the loop, the script parses the full path to the video file, and encodes it to ProRes 422 HQ with the following settings:

  • Preserves format flags: PTS timestamps
  • Pixel format: YUV422P10LE
  • "Vendor": Apple (to fool some Macs into thinking it's encoded by a QuickTime/Apple encoder)
  • Top field first interlaced
    • Along with the flags options, this helps correctly set the interlaced flag for the container - c.f. the QuickTime 'fiel' atom - which indicates the streams are interlaced, not progressive
  • Sets flags "ildct" and "ilme" to indicate source video is interlaced, and forces FFMPEG to handle the video as interlaced while encoding to help preserve picture quality
  • Defines a QScale value of 8, which produces roughly 120 MBps files with the ProRes encoder (Qscale of 9 produces about 115 MBps)
  • 'Maps' all of the MXF's streams out to the destination file in their original order (FFMPEG can reorder or selectively copy a subset of streams if told to)
  • For codec:video, use FFMPEG's "prores_ks" encoder, written by Kostya Shishkov - deemed to be the highest quality one in testing, and the only choice if you want to encode Alpha channels (for various reasonsthere are three prores encoders in FFMPEG, but just use Kostya's.)
  • Use ProRes profile 3 - ProRes 422 HQ (there are 6 ProRes profiles, with indexes 0 to 5)
  • For all codec:audio streams, leave them untouched - do not reencode
  • "%%~df%%~pf%%~nf-prores.mov" = save the output file as a MOV, with the same name as the source but with "-prores" appended to it. This is some funky Windows batch file variable expansion...
  • %* = pass any further command-line arguments to the command
    • This is useful if you want to specify "-y" (or "-n!") to say 'yes' (or 'no' to overwriting any destination files if they already exist.

I get lovely ProRes 422 HQ files as a result, which are correctly flagged as interlaced (and play correctly deinterlaced with VLC or MPC-HC) and, most importantly, are usable in Premiere. Happy days. Thanks, once again, to FFMPEG.

If there's demand for it, I'll modify the script to work on Linux/OSX and publish here. Likewise, if you have any questions, leave them in the commands or tweet me @christopherw.

Tiplet: DO NOT USE the # symbol in Oracle Enterprise Communication Broker passwords!

I've been working with Oracle Session Border Controllers and Enterprise Communications Brokers a lot recently, and I encountered what I think is a new bug (it's with Oracle for investigation).

After deploying some new ECB instances, I changed the login and enable passwords, per the Oracle ACLI guide ("secret login" and "secret enable" at the SSH prompt). I used randomly generated strong passwords which included limited special characters -- "#", "%", "?" and "!".

The ECB appliances use two passwords per username - one to log on to the unit, and one to 'enable' (admin mode for configuring them, similar to Cisco's enable mode).

On one appliance, the new password I generated for it was accepted without complaint when typed and retyped at the confirmation prompt. However, attempting to then authenticate a new session using this password resulted in immediate refusal by the appliance. It effectively denied access, with no recovery method. I had to flatten and redeploy the appliance!

After I was locked out, I tried all sorts of alternatives, seeing if the password had been partially saved or corrupted - seeing whether the password was truncated at the # symbol, whether the appliance replaced it with a space character, whether it omitted the # symbol when it committed it to memory, or whether it simply swapped it for another character. I must have tried a dozen or so possible variants before giving up.

Despite the documentation saying "do not use special characters", one normally expects a password change routine to immediately reject any invalid special characters on submission using string validation. This isn't a new concept. However, the ECB appliances seem to happily accept invalid characters, even though one of them causes a service-affecting bug.

I've previously set passwords including !, ? and % characters -- those are in use on other ECB appliances using the same software revision (and SBCs) and they work fine. I have only encountered this problem so far using # symbols.

I see conflicting information about acceptable non-alphanumeric symbols in passwords in the Oracle documentation. However, the fact that simply entering a # symbol as part of a new password when updating credentials can lock you out of admin access on a production system seems pretty bad.

 

I can reproduce the fault with a fresh OVA deployment of ECB PCZ2.2.0 GA (Build 53), so this seems like a serious bug which can end up with an administrator permanently and irreparably locking themselves out of their own system. I'm only glad I was setting the passwords prior to deploying a configuration!

If you're working with Oracle ECBs or SBCs, be very careful about which special characters you use in your strong passwords.

Photography: ONE OK ROCK, Make Out Monday

I've been getting a bit more serious with my band photography recently. Here's two galleries of ONE OK ROCK (from Download Paris 2016) and Make Out Monday from MCM Comic-Con London (May 2016).

BAND-MAID MCM Comic-Con photos coming very soon!

ONE OK ROCK:
20160611 Download Paris - ONE OK ROCK

Make Out Monday (Listen on iTunes | Spotify | Google Play Music)
(Kicking Cars is streamable on GPMAA)
20160529 Make Out Monday at MCM Comic-Con London

This referendum should never have happened

My country voted to leave the EU on Thursday. Tonight, one small comment on Facebook tonight prompted me to pen a long response:

i can not believe the amount of hate from the remain supporters, most of the people bitching are students blaming OAP's but i see it as the older people have been around before the EU and lived through all 40 years of the EU and have seen how the country has changed and have lived on the breadline and led a working class life. surly they have more insight than students.

A lot of the remain complaints have come mainly from students who have no clue what the working class life is like they just live the life of getting student loans that 80 percent probably will never pay back and just spunk their loans on piss up's and traveling then get a degree in something useless and end up working in starbucks or some other job that has nothing to do with their degree, I think i only know 3 or 4 people who have a degree in something and actually work in the field they studied.
Also give leaving a chance ffs it has been less than 24 hours give it at least a bit of time it could work out for the better

It's not hate. It's despair, anger and a huge amount of worry about what's coming.

And you can't just pin it on the students. I'm pro EU. I'm in my thirties. Working class background. What I hate is how many people voted without fully understanding the implications of their vote, or that they allowed themselves to be so easily convinced by lies and spin.

I've seen the benefits of EU membership since I was a child - my Mum was a French teacher, and is still deeply involved in international French examinations. We were always in France for holidays through my childhood, staying with French families (mum set up a French exchange programme and made a LOT of contacts) so I got to experience France as the French saw it. I feel like I have a fairly good understanding of what EU membership brings as I have seen both the benefits and difficulties other countries face - and how being in the EU has made a difference for them.

I also know a lot of French people (they all think we are completely crazy for leaving). I love the injection of foreign culture and the alternative perspectives it brings. When we joined Europe, no longer were we one small island. we were part of a large collective group of countries all working together for collective benefit.

I take the wider view. We've been part of the EU common market, the EU bargaining force and the EU political 'bloc' for four decades. The world has moved on significantly since then and the EU today is one of the world's "superpowers" alongside China, USA and Russia. Forty countries bargaining together have a lot of power which we simply don't have any more as a lone country.

"Make the UK Great again" - but it wasn't Great before

In the past forty years, for various reasons, the UK's seen many of its heavy industries - coal, steel, oil - closed down or significantly reduced in size. Oil market has bottomed out due to the Saudis but the North Sea industry was contracting before that anyway. China and India obliterated our steel industry in one go simply because they felt like it. Coal is long dead. The UK now specialises in things like IT, high tech, financial and pharmaceuticals -- all of which significantly rely on Europe to source employees with the relevant skills from other European countries, to support multinational corporations and reciprocal trade.

So what happened in the meantime? Loads of things. Successive governments, Tory and Labour, gradually promoted everyone to go to University (I went to university) to the loss of apprenticeships, crafts and practical trades. More than half the country's young people ended up going straight from school into higher education and then into a "highly skilled" job. Only very recently has there been an increase in people taking an alternate route straight into industry through professional apprenticeships. The UK is playing catch-up, and it's only got itself to blame.

So why should we have stayed part of the EU? Cultural enrichment, business enrichment, trading benefits, generous financial subsidies for our key industries (e.g. farming and fishing), shared national infrastructure arrangements, some of the various European development funds which help regenerate cities and industries (Manchester and Liverpool for example benefited massively from this). More than that, we combine to have real political clout when dealing with other large and bossy nations.

We also keep a really preferential position in what's still a hugely powerful collection of countries. Our original deal with the EU has most the benefits of being an EU member with far fewer downsides (like being part of the Euro, but having the economic stability of being allied with them).

Please, don't go

If / when1) because who knows, we may never get to doing anything about Article 50... we quit the EU, we'll lose the ability to have ANY say on how the EU works. As we still rely massively on the EU for trade and day to day life, we'll now have to agree to all their agreements - with no preferential terms - and we will still HAVE to comply with all the EU regulations in order to sell our goods and services into Europe. We'll also still have to comply with all of the European rules on free movement of people if we want to travel and work in Europe.

Voting to leave the EU has set our country back 20 or 30 years in terms of its influence in global economy and politics. Economic migrants - foreign benefits scroungers - have been used by politicians as a reason to leave the EU, but they don't really exist at all. Other European citizens coming over here and taking jobs and benefits - perhaps true to a small extent, but there's lots of jobs in sectors which British people just aren't taking. I was reading someone who works in the medical sector yesterday saying that they struggle to get anyone to apply for the jobs, and those who do are from other European countries. Great! Frankly I don't care who does the jobs, if British people can't be bothered to apply for them that's their fault. Those problems are nowhere near as bad as some politicians described.

Bills, bills, bills

The cost of being in the EU, for what it gives us, is actually quite reasonable. A lot of the money we "send" to the EU is actually just ringfenced for us to spend on things in the UK. We also get a lot of support from the EU which is worth far more than the money. Europe isn't also THAT beaurocratic, nowhere near as bad as some people made out.

The nail in the coffin for me was all the Leave politicians (Farage and the £350 million mistake - full ITV interview here; Daniel Hannan's Free Movement bait-and-switch) who, straight after the result came in, reneged on their statements and promises. We won't have £350 million a week to spend on the NHS as we're not spending £350 million a week to begin with. We won't really be able to stop people entering our country if we want to keep travelling to Europe, as we'll have to allow European citizens for us to get in to Europe. We won't be able to make any difference to the number of refugees coming here -- actually it's a tiny amount and always has been. Qualified, skilled European citizens won't be stopped from coming to the UK for work and we would never have realistically had the ability to stop this.

But it's the Will of the People...
Er, could you explain the question again?

Lots of people didn't properly understand the purpose of the referendum question, and some people didn't understand the implications of leaving. Some people did it based on their dislike of overseas residents; some people did it because they're flat out racist. Some people did it because they don't understand how the the UK and European political systems are interwoven. A LOT of people thought that by voting Leave they would be putting the middle finger up at the current government, and they were persuaded by tabloid media run by individuals who have a vested interest in leaving the EU and don't really care about the problems it will cause. Some people voted Leave because they don't like the Government, but they thought that their vote wouldn't really make a difference. (It did.) I live in an area with a large amount of Leave voters and I'm yet to hear a convincing, sensibly explained reason for why they voted Leave.

We won't suddenly be able to stop complying with all the various EU rules and regulations (which are actually hugely beneficial and have given us real improvements to our life, our safety and our legal and human rights), so we'll end up taking new EU laws and just copy and pasting them as UK law to remain compatible. We'll keep on using all of the existing EU rules and regulations as they form the basis of a lot of UK laws brought in over the last 40 years. "Taking back control", yeah... no. The cost of rewriting four decades of laws for no reason would be so much we would go bankrupt.

Hell, until Mark Carney made his statement on Friday morning, for a couple of hours the UK was on its way to going bankrupt. I'm not kidding. I watched it happen on the markets.

So much to lose, little to gain

We'll lose much of the cultural enrichment and spontaneous opportunities which being part of the EU offers. Leaving adds cost and complication to international trading. Leaving adds cost and complication to international politics and lobbying. Leaving adds cost and complication to international peacekeeping. Leaving adds cost and complication to buying things from other countries. Leaving destabilises our currency as we're no longer part of the 'safe' group of 40 countries, which devalues our usefulness as a "safe haven" for investors.

I can't really see any benefits from us leaving the EU, just more cost and an uncertain future at every step.

The Lord Ashcroft poll on referendum voting preferences, analysed by demographic, is a very interesting read. I think it speaks clearly about the demographic division between those who voted to stay and who voted to leave. You can draw your own conclusions as to why these stats so clearly show the divide that they do, but I think they makes a clear statement about the intentions of Remain voters and the intentions of Leave voters.

Read: http://lordashcroftpolls.com/2016/06/how-the-united-kingdom-voted-and-why/

What's particularly concerning is how many older people (60+) voted Leave. Speaking selfishly, this won't affect old people anywhere near as much as it will affect the rest of us, but as so many of them voted Leave they swung the result over just enough. I feel that many of them voted because they were looking back to a past reality which simply does not exist any more - the younger generations were looking to the future. Sadly, it was not a representative result.

A guy called Hugh Hancock also spent a LONG time analysing everything the Remain and Leave campaigns said through their campaigning. His results are very interesting.

He writes, "At every single turn, I found that the Leave campaign’s arguments were founded on lies. Sorry, it’s as simple as that. I wish it wasn’t."

 

Bye bye, Europe. I hope I can come back soon.

europeflag

References   [ + ]

Katsura Sunshine at The Forge, London

The most excellent Rakugo storyteller Katsura Sunshine recently assumed his Artist in Residence position at The Forge in London - I was lucky enough to enjoy and document his performance. If you get a chance to see one of his shows whilst you're in London, do not pass up the opportunity, they're great fun and perfect for the whole family.

Katsura Sunshine at The Forge, 23 April 2016

The Forge also celebrated its seventh year of operation in April this year! Belated congratulations to them 🙂 If you go to the Forge for a visit, try the Camden Brewery beers. Dangerously delicious.