Christopher's Broom Cupboard header image

MaxMind GeoLite v1 databases discontinued - install GeoLite2

I noticed recently that a few web sites are miscategorising my ISP's static IP as being in the wrong country. I knew it was a recent reallocation of a new block and suspected the web sites were using a stale version of a GeoIP database - probably MaxMind's GeoLite v1 offering.

However, as I also run a few servers which themselves use the GeoIP tool (with

geoipupdate
) I wondered about how to update to the latest version. Sure enough, with the old v2 of geoipupdate I was using, the databases were definitely not available (and HTTP error codes were being returned when I tried to run an update).

When I started Googling in more detail, I discovered MaxMind has fully discontinued and removed the GeoLite databases. Yowzer.

But no matter! GeoLite2 is now available from the MaxMind Developers site - I

--force
installed the RPM, as yum was requesting to also uninstall ProFTPd, which I wasn't impressed by). Backed up the old binary and config just in case, but it was unnecessary. Verified the install afterwards with
geoipupdate -V
, then ran it with
geoipupdate -v
-- all good.

Getting Postfix working for outbound emails via SMTP and local relay for cron reports

I recently had to diagnose a couple of servers running Postfix which emailed results of rsync cronjobs when they returned a non-OK value. While Postfix was emailing the recipients on rsync failure, I noticed that the cronjob STDOUT was not being locally relayed correctly to the root mail.

I documented my fix on ServerFault, hopefully it's useful for someone else.

Having fixed local root cron reports, I then noticed on the CentOS7 box that the AIDE service was running, had been scanning for integrity checks against the original postfix install (subsequently upgraded to Postfix 3) and as such was throwing errors every night for no real reason, quite annoying.

To fix this I tried running

aide --update

but it didn't work (probably my fault from doing an

--init

first). I had to rm the

/var/lib/aide/aide.db.gz

and

/var/lib/aide/aide.db.new.gz

files, then

run aide --init

and rename the newly-made

aide.db.new.gz

to

aide.db.gz

. After that, it was happy.

Canon XF305 MXF problems in Premiere? Transcode your clips to ProRes with FFMPEG!

Recently encountered a weird problem with 1080i MXFs straight out of a Canon XF305. The files would play in VLC, but Premiere Pro CC 2017 on a brand new MacBook Pro or iMac failed soon after starting to decode the video, with a horrible red frame and MXF frame decode errors in the Log. It's pretty nasty.

Interestingly, an older Mac Pro (a silver tower running Yosemite 10.10.5 with an ATi 5770 1GB video card) running Premiere Pro CC 2017 did not have any issues decoding the MXFs, so I wonder if there's some additional / different version Pro Video Codecs installed on it...

I fired up MediaInfo to check the file. (If you don't have it, download the 64-bit GUI without installer from the MediaInfo site).

All looked to be OK:

Input #0, mxf, from 'E:\scratch\test\CLIPS001\XF0007\XF000701.MXF':
Metadata:
uid : 05f46a47-8205-4800-80aa-960001e012e3
generation_uid : 05f46a47-8205-4800-812a-960001e012e3
company_name : CANON
product_name : XF305
product_version : 1.00
product_uid : 060e2b34-0401-010d-0e15-005658460000
modification_date: 2018-05-09T02:10:05.000000Z
material_package_umid: 0x060A2B340101010501010D43130000000A05F46B478205800000850001E012E3
timecode : 09:51:05:02

So why is the file failing to work in Premiere?

This Canon MXF decode problem has been around for years: Google "Premiere MXF red frame". Lots of Canons seem to produce problematic MXFs for Premiere. (1, 2, 3, 4...)

The MXFs produced by this Canon XF305 contain MPEG-2 inter-frame videos, lossily compressed to produce I, B(idirectional predicted) and P(redicted) frames grouped as "GOPs" (Groups of Pictures). The highest quality MPEG-2 files you can produce are intra-frame, where the file is a stream of intra frames, each one being one complete picture, but these yield larger file sizes.

Whatever; we can deal with this - let's transcode to a proxy format, for example Apple ProRes 422 HQ. Easy, Premiere likes that, and there's negligible generational quality loss.

However, what happens if you attempt to do this natively in Premiere or After Effects is that you get the same result as attempting to import the MXFs to use them directly: failed encodes, frame decode errors, truncated files, pain and suffering. Yuck.

So, off to trusty FFMPEG!

The following guide assumes you're using Windows, although it's trivial to modify the instructions for Mac or Linux.

Download the latest stable FFMPEG from the FFMPEG web site and place it in a suitable location. No installation required as long as you grab the 'static' version (with all dependencies included in the main executable).

If your MXFs are gathered in a folder or folder structure, no problem. Make a new batch file (.bat) containing the following (obviously Windows only):

for /R %%f in (*.mxf) do (
D:\path\to\ffmpeg.exe -fflags +genpts -i "%%f" -pix_fmt yuv422p10le -vendor ap10 -top -1 - 
flags +ilme+ildct -qscale:v 8 -map 0 -c:v prores_ks -profile:v 3 -c:a copy "%%~df%%~pf%%~nf- 
prores422hq.mov" %*
)

So what does this do?

The for loop with the /R switch recursively looks through all files in the directory the script is run from. The caveat is that it assumes the start point is the directory the batch script is run from; this could be improved into an argument (so you could specify a different starting folder).

Once inside the loop, the script parses the full path to the video file, and encodes it to ProRes 422 HQ with the following settings:

  • Preserves format flags: PTS timestamps
  • Pixel format: YUV422P10LE
  • "Vendor": Apple (to fool some Macs into thinking it's encoded by a QuickTime/Apple encoder)
  • Top field first interlaced
    • Along with the flags options, this helps correctly set the interlaced flag for the container - c.f. the QuickTime 'fiel' atom - which indicates the streams are interlaced, not progressive
  • Sets flags "ildct" and "ilme" to indicate source video is interlaced, and forces FFMPEG to handle the video as interlaced while encoding to help preserve picture quality
  • Defines a QScale value of 8, which produces roughly 120 MBps files with the ProRes encoder (Qscale of 9 produces about 115 MBps)
  • 'Maps' all of the MXF's streams out to the destination file in their original order (FFMPEG can reorder or selectively copy a subset of streams if told to)
  • For codec:video, use FFMPEG's "prores_ks" encoder, written by Kostya Shishkov - deemed to be the highest quality one in testing, and the only choice if you want to encode Alpha channels (for various reasonsthere are three prores encoders in FFMPEG, but just use Kostya's.)
  • Use ProRes profile 3 - ProRes 422 HQ (there are 6 ProRes profiles, with indexes 0 to 5)
  • For all codec:audio streams, leave them untouched - do not reencode
  • "%%~df%%~pf%%~nf-prores.mov" = save the output file as a MOV, with the same name as the source but with "-prores" appended to it. This is some funky Windows batch file variable expansion...
  • %* = pass any further command-line arguments to the command
    • This is useful if you want to specify "-y" (or "-n!") to say 'yes' (or 'no' to overwriting any destination files if they already exist.

I get lovely ProRes 422 HQ files as a result, which are correctly flagged as interlaced (and play correctly deinterlaced with VLC or MPC-HC) and, most importantly, are usable in Premiere. Happy days. Thanks, once again, to FFMPEG.

If there's demand for it, I'll modify the script to work on Linux/OSX and publish here. Likewise, if you have any questions, leave them in the commands or tweet me @christopherw.

Tiplet: DO NOT USE the # symbol in Oracle Enterprise Communication Broker passwords!

I've been working with Oracle Session Border Controllers and Enterprise Communications Brokers a lot recently, and I encountered what I think is a new bug (it's with Oracle for investigation).

After deploying some new ECB instances, I changed the login and enable passwords, per the Oracle ACLI guide ("secret login" and "secret enable" at the SSH prompt). I used randomly generated strong passwords which included limited special characters -- "#", "%", "?" and "!".

The ECB appliances use two passwords per username - one to log on to the unit, and one to 'enable' (admin mode for configuring them, similar to Cisco's enable mode).

On one appliance, the new password I generated for it was accepted without complaint when typed and retyped at the confirmation prompt. However, attempting to then authenticate a new session using this password resulted in immediate refusal by the appliance. It effectively denied access, with no recovery method. I had to flatten and redeploy the appliance!

After I was locked out, I tried all sorts of alternatives, seeing if the password had been partially saved or corrupted - seeing whether the password was truncated at the # symbol, whether the appliance replaced it with a space character, whether it omitted the # symbol when it committed it to memory, or whether it simply swapped it for another character. I must have tried a dozen or so possible variants before giving up.

Despite the documentation saying "do not use special characters", one normally expects a password change routine to immediately reject any invalid special characters on submission using string validation. This isn't a new concept. However, the ECB appliances seem to happily accept invalid characters, even though one of them causes a service-affecting bug.

I've previously set passwords including !, ? and % characters -- those are in use on other ECB appliances using the same software revision (and SBCs) and they work fine. I have only encountered this problem so far using # symbols.

I see conflicting information about acceptable non-alphanumeric symbols in passwords in the Oracle documentation. However, the fact that simply entering a # symbol as part of a new password when updating credentials can lock you out of admin access on a production system seems pretty bad.

 

I can reproduce the fault with a fresh OVA deployment of ECB PCZ2.2.0 GA (Build 53), so this seems like a serious bug which can end up with an administrator permanently and irreparably locking themselves out of their own system. I'm only glad I was setting the passwords prior to deploying a configuration!

If you're working with Oracle ECBs or SBCs, be very careful about which special characters you use in your strong passwords.

Photography: ONE OK ROCK, Make Out Monday

I've been getting a bit more serious with my band photography recently. Here's two galleries of ONE OK ROCK (from Download Paris 2016) and Make Out Monday from MCM Comic-Con London (May 2016).

BAND-MAID MCM Comic-Con photos coming very soon!

ONE OK ROCK:
20160611 Download Paris - ONE OK ROCK

Make Out Monday (Listen on iTunes | Spotify | Google Play Music)
(Kicking Cars is streamable on GPMAA)
20160529 Make Out Monday at MCM Comic-Con London

I footnotes