Extracting audio from video and generating a cuesheet from a tracklist

I quite often listen to mixes on Youtube. Only problem is, you need an internet connection to listen to the music and on some videos the mix seems to be constanty interupted by ads.

The solution to this is download the video - there are plenty of free Youtube video download tools available. But then you have a video file that's much larger than it needs to be (because you only want the audio track), and can't skip from one track in the mix to another.

(If you feel guilty about downloading the video you can always donate a few dollars to Google and a few cents to the video uploader and ASCAP or the equivalent organisation in your country to make up for the lost ad revenue).

So to this end I decided to write a script that could be used to extract the audio from the MP4 video file and generate a cuesheet based on the tracklisting (copied from the video description). And I thought I might as well share it here.

For the initial setup you'll need MP4Box, which comes as part of GPAC.

Ensure GPAC is in your Path: System > Advanced System Settings > Advanced tab > Environment Variables... In User variables find Variable Path. Select it, click Edit, and add to the end: ;C:\Program Files\GPAC (Assuming that's where you installed GPAC to).

The scripts I created using powershell, so you need to allow executing powershell scripts. Run Powershell as admin. Then enter: Set-ExecutionPolicy Remote Signed.

Download the scripts: create-cue.ps1 and video-convert.ps1.

Now you need to allow running those scripts without signatures. From powershell:

Unblock-File create-cue.ps1
Unblock-File video-convert.ps1

That's the initial setup done, which you shouldn't need to do again. Now onto converting the video file.

To convert the video you need a folder containing the video file in mp4 format and a tracklist file. The tracklist file should be a plain text file containing lines in the format:

00:00 Artist Name - Song Title

Many (though sadly not all) Youtubers who upload mixes will have a tracklist in this format as part of the video description, so you can copy-paste from there.

The tracklist file and the video file should both have the same filename (with .txt and .mp4 extensions respectively), which should be in the format:

Album Artist - Album Title

You can run script in several ways:

• From powershell: path\to\video-convert.ps1 \path\to\video.mp4
• From powershell:
  cd \path\to\video
  path\to\video-convert.ps1
• Copy both .ps1 files to the video dir, then double-click on video-convert.ps1 to run it. (This is easiet IMO).

The script will extract audio from the mp4 video, wrap it in an mp4 (m4a) wrapper, and generate a cuesheet from the tracklist file. Because it just extracts the audio, not converting it, there is no loss in audio quality.

If you want the track split into separate files, there are various cue splitters available that will split an audio file based on a cuesheet.

Choosing the fields shown in the details pane in Windows Explorer

I just noticed that weirdly in the details pane of Windows Explorer it would not show the total file size when I selected a bunch of Fuji RAF files. Yet it did show this information when I selected a bunch of JPEGs. The other fields that were shown were also quite different between the two file formats.

Thankfully it is possible (though a bit messy) to change what information is shown in the details pane. The information that should be displayed is set on a per file extension basis and stored in the registry under HKEY_LOCAL_MACHINE\SOFTWARE\Classes\SystemFileAssociations\.

I found the answer here: Customising Explorer Details Pane? (If you read through that thread you'll notice that the OP has to answer their own question after being given irrelevant answers by people who couldn't be bothered to read what was being asked. Unfortunately a common problem on internet fora.

In case that post ever dissappears, the solution is as follows:

1. Run Regedit.
2. HKEY_LOCAL_MACHINE\SOFTWARE\Classes\SystemFileAssociations\.mp3 (Or whatever file type you want to edit.)
3. Edit the PreviewDetails string.
4. If needed, log off and back on.

eBay watchlist cleaning

It's time to clear out my eBay Watched Items list again, and make notes on any items I found that sold for high or low amounts.

NIKON AF-S NIKKOR ED 600mm f/4G IF VR Lens

sold for GBP 2730, which seems very cheap for this lens. Despite the auction finishing on 22nd March, no feedback for the seller has been left, so you can't be sure if the buyer did actually get the item in good condition for that amount.

Carl Zeiss Softar III 67mm screw fitting

sold for GBP 5.79 - not super cheap, but screw mount versions of these filters are quite rare - they tend to be Hasselblad bayonet mount.

W.Acall Kyoei 1:3.5 F=35mm UV-Objektiv f. Exakta

Didn't sell several times, often finishing around the EUR 20 mark or less. I don't know what the reserve was, but they had a EUR 150 buy it now.

Canon 400mm f2.8 L Lens EF mount (AF non IS) Pro Prime +BN= 1.4x Extender MK II

was listed with a BIN of GBP 1430 inc. P&P. However, the listing was ended by the seller because the item was lost or broken. I suspect they sold it to someone off eBay for a slightly lower price to avoid the eBay fees. Although this is the non-IS version, it seems a very good price to me, particularly with the TC.

Canon EF 600mm F/4.0 L Lens

ended for GBP 1,191 + 60 P&P, however it didn't meet the reserve price. It would've been very cheap otherwise. The body was a bit bashed up, but paint chips don't harm the lens' performance.

Ananda Shankar And His Music LP Rare Orig EMI HMV India Pressing 1976

Ended for GBP 19 inc. P&P, which is very cheap. This was one of those ones that I actually wanted to bid on, but had forgotten to force refresh my ebay watch list, so by the time I went to bid on it (near ending time), the auction had actually already ended.

Canon Lens EF 135 mm F2.8 Softfocus

sold for GBP 62.50 inc. P&P. Possibly a bargain, however the description does state the AF is UNTESTED and Optics do have some signs of dust and possibly a small amount of fungus although it is very difficult to see. The buyer left positive feedback though. (This item normally goes for around GBP 130).

Tiffen Standard Hot Mirror Filter 77mm

Sold for GBP 40 inc. P&P. This is a good price. I'm glad I didn't purchase it myself though, as on further research it seems to have the same problem on wide-angle lenses as my B+W 486 UV-IR cut filter has.

rayxar e50/075 made in holland

sold for GBP 20 + 12 P&P. This included not only the lens, but a large (and heavy looking) piece of equipment the lens was mounted in. Difficult to say what condition the lens was in, but looked okay. I would quite like one of these lenses, however from images I've seen taken with them, I don't think they are actually that great.

Sigma 800mm f/5.6 EX DG HSM APO Ultra Telephoto Lens for Nikon SIGMONSTER

sold for GBP 876 inc. P&P. I don't think these are listed often, I suspect this is probably the going price, neither cheap nor expensive.

Télé-objectif Canon FD 600mm f4.5

sold for EUR 371 + 100 P&P (to UK). Again, these don't seem to sell that often so I'm not sure if it's a good deal. Manual focus only, but I think this lens is meant to be pretty good.

Canon EF 400mm F2.8 Mk 1 lens

sold for GBP 860 + 40 P&P. Seems a good deal to me.

Wimberley Plamp

sold for GBP 40.56 + 10 P&P. The new model is 'only' GBP 45 new, so seems really expensive to me.

Very nice Petri Film Camera complete with Petri 1:3.5 f=35mm C.C Lens & Strap.

sold for GBP 27.15 + 2.85 P&P. This is quite a bit more than the other seller's Petri 1:3.5 lens didn't sell for. (Though of course this one includes a camera too).

Looking for photos

Today I was just looking for photos to illustrate an article. Yes, it really did take me all day. Quite annoying.

Elephant dream

This morning I had a dream that there was an Elephant on Welland Park road, and I think I was trying to wash its ear. I was around the same height as it though, maybe I was standing on top of a car or something. I'm not even sure it was actually an Elephant at this point either, it may have been a giant wood pigeon.

Anyway, the Elephant walked down the street and then turned left at the mini roundabout to go up Farndon road. As it started going up the road, I saw that the skin just under the backs of the knees on its rear legs was ripped. It was as if the elephant's skin was just an outer covering over its pink body, and the skin covering had been torn in this location.

There was a bloke walking the other way down the road, and he said the elephant had its trunk cut off. While the elephant actually had a trunk in my dream, it was also true at the same time that its trunk had been cut off. It must have been a quantum trunk.

I explained that it was from a circus (the point being that as it was from a circus, it had been badly abused). When I looked at it again all the skin had been ripped around its bum and you could see its pink buttocks through the hanging flaps of torn skin.

Then the bloke started riding the elephant. Suddenly we were on top of a cliff on the coast, and the elephant was running really fast towards the edge of the cliff. There was a single storey hotel (or similar) building that blocked my view as the elephant was heading over the edge of the cliff. So I ran into the hotel and went into the room at the far end of the building so I could look out of the window and see what happened.

I could see that the elephant had landed in the sea. I think it was still alive and swimming out into the ocean, but I can't quite remember. Near the shore there was a person coming out of the sea while holding another person in their arms. The person they were holding had either been killed or badly injured when the elephant jumped off the cliff into the sea.

Then the bloke who had been riding the elephant ran into the room with me to look out the window and see what was happening below. He had managed to jump off the elephant just as it jumped off the cliff. Then he had ran round the hotel and down the corridor into the room where I was really fast. But in the dream it wasn't weird how quickly he'd got there, or that you needed to look out the window in this hotel room to see what was happening over the cliff edge.

Then we ran away, back down the corridor and out the front of the hotel or whatever building it was. (It didn't have a reception area, but there were rooms along the corridor like in a hotel and the room we'd been in was like a hotel room). In my dream (and also now), I thought it was weird that you could just go into this building and go into an unlocked room.

Then I was feeling like I was partly responsible for the injury or death of the person who I had seen being carried out of the sea. And I wanted to get away in case anyone found out I had anything to do with the elephant, which could lead to me being arrested in connection with the incident.

That was the end of the dream.

BTW I don't actually think that all circus animals are badly abused, that was just dream logic.

Why am I not surprised a bank's website is rubbish?

Recently I decided to finally do something about my rubbish Alliance & Pester / Santander bank account that pays no interest, and go through the hassle of opening a new bank account that pays a better rate of interest. TSB offer quite a good rate, so I applied to open an account with them online. Their sign-up / security process is pretty bad though:

As you can see, it didn't like my username containing non-alphanumeric chars. Underneath the username section it says Your user IS must be between 9 and 30 characters long and can include letters, numbers and special characters (see the tip section above for ideas on which special characters to include). (emphasis mine).

Yet when you look at the tip section, it makes no mention of special characters. And with alphanumeric only password but 'special chars' in my username I was not able to continue. But without 'special chars' in the username I could continue onto the next page. So clearly special characters are not allowed in the username.

As well as giving incorrect information on the signup page, the fact that they don't allow special characters and also make the username / password case insensitive is bad for security. The smaller the number of possible character combinations, the easier it is for someone to make a brute force attack and guess a correct user ID / password combination.

It wouldn't surprise me if the user id and password are also ASCII only. While this also decreases security, it's more of a problem from an accessibility standpoint. If I was Chinese, my preferred User ID may well be something in Chinese. But you'd be forced to use something in ASCII. It's not a big problem - you could still use the same keystrokes combination, but you'd see a jumble of random letters rather than the Chinese characters, making identifying typos a bit more difficult.

Installing mysql binaries in non-standard location

Well, the I've spent the last few days just trying to get the latest version of mysql installed. Yesterday I spent ages installing the latest version of gcc on the webserver, as the one installed was too old to build mysql with. (Don't I love shared hosting). GCC takes hours to build and it was also complicated by the unusual way in which it must be configured, which I wasn't aware of until after a few failed attempts at compiling it in the normal way.

Installation of mysql was further complicated by it still trying to use the old version of gcc. Following the instructions to make clean and rm CMakeCache.txt didn't help. After trying a few different things I deleted the directory and extracted the tarball again to start clean. This did work.

But after all that work, my webhost killed all my processes while mysql was still building. I guess I can't blame them - I have a 512MB plan, and the mysql build process alone was taking up nearly a gig. I tried to see if there was any way to limit the amount of memory used during make, but couldn't find anything.

So today I checked if there was any way of changing where a pre-compiled binary package could be installed to. It turns out that for rpm (the server is CentOS and mysql offers an rpm package) you can use rpm --prefix= or rpm --relocate, but mysql packages are not relocatable.

However, I later found this information about using the generic binaries: Installing MySQL on Unix/Linux Using Generic Binaries. This indicates you can just extract the tarball to your preferred installation directory, and that's it. So I did that and it worked! Could've saved myself several days work and fustration!

mysqld_safe mysqld from pid file ended error with no error log entries

Today I was just doing more work on configuring nginx and trying to install the latest version of mysql. For things like mysql I install them first on my local PC, writing the commands in a text document using gedit. That way I can easily alter the commands when something doesn't work and try it again by just copy-pasting from gedit to the terminal.

When I tried configuring mysql today though, it was as if the escape new line character wasn't working - when I pasted in my multi-line with escaped line endings command, the shell just executed each line of the command separately (which of course didn't work). The problem was that the file had been saved with windows line endings (\r\n) instead of linux (\n). But how on earth that could happen I don't know. When I was using this file the other day I could copy and paste the multi-line command from it into the terminal no problem.

After compiling mysql again (twice) I got a bit further. But when I ran mysqld_safe, it just said:

160126 21:13:44 mysqld_safe Logging to './data/rusty-ubuntu.err'.
160126 21:13:44 mysqld_safe Starting mysqld daemon with databases from ./data
160126 21:13:45 mysqld_safe mysqld from pid file ./data/rusty-ubuntu.pid ended

Checking the log, all it contained was:

160126 19:27:48 mysqld_safe Starting mysqld daemon with databases from ./data
160126 19:27:48 mysqld_safe mysqld from pid file ./data/rusty-ubuntu.pid ended

I did quite a bit of googling, but all the results seemed to be either where the error log actually contained an error, or a permissions error. Then eventually I found this blog entry: MySQL error mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended, which seemed to be pretty similar to my problem with the lack of error log information. They suggest running mysql_safe, which doesn't exist in my bin dir and I think might just be a mistake in their write-up. But it gave me the thought of just running mysqld rather than mysqld_safe.

mysqld ran without any issues. So I then tried to access the mysql client to change the temporary password that had been assigned to root when running mysqld with the --initialize parameter to setup the data store. But this gave me an error that it couldn't connect to the mysql server:

ERROR 2002 (HY000): Can't connect to local MySQL server through socket './mysql.sock' (2)

The problem was that I was using relative paths in my.cnf and it seems the paths get resolved differently. mysqld had resolved ./mysql.sock to the data dir (no idea why). Whereas mysql resolved it to the bin dir. After replacing all the relative paths in my.cnf with absolute paths, I restarted mysqld and then tried mysql again.

mysql could now connect to the server, but said that the temporary password that had been assigned to root wasn't valid. So I then had to delete the data dir and initiailise it again. After this I could get in okay and change the root password. I then shut down mysqld and tried running the mysqld_safe script, and lo and behold, it actually worked.

So it seems the problems in mysqld_safe starting and ending straight away (or likely just not starting at all in reality) were down to the use of relative paths in the configuration options. Annoying that it didn't give any errors to indicate this though.

I still have to actually setup the databases and switch it over as the active mysql server, not to mention get it up and running on the web server, so there are still plenty more things that can go wrong yet.

Wasting time because my memory is rubbish

I spent quite a while today trying to figure out why my wordpress blog wouldn't work when a query string was added to the url. At first I just got an access denied error in the browser. Checking nginx's error log for the site in question I saw:

FastCGI sent in stderr: "Access to the script '/path/to/document_root/url' has been denied (see security.limit_extensions)" while reading response header from upstream

Searching for security.limit_extensions, I found it was a php-fpm configuration option (so php-fpm.conf, not php.ini). Adding security.limit_extensions= (so no value) to the php-fpm conf (and restarting php) I got a bit further. I now got a 404, and the error message in the nginx logs was:

FastCGI sent in stderr: "Unable to open primary script: /path/to/document_root/url (No such file or directory)" while reading response header from upstream

I then spent ages trying to figure out why it was trying to open this uri, instead of trying to open index.php as my try_files dictated.

After a while I traced the error down to the use of if ($query_string... within the location block for my blog. I remembered that I had actually had such an issue some time ago, and I'm pretty sure I spent ages trying to figure out the problem then too.

The problem is down to having the if statement within the location block. Move it outside the location block and all is fine.

I also had trouble with mysql today, (installed the latest version, but got stuck in getting it setup). I haven't resolved that yet. I've asked for help on the mysql forums, so I'll see if any is forthcoming before I try and fix it myself. (It is likely to involve trial and error in setting options when compiling mysql and mysql is extremely slooow to configure, make and install).

The other thing I spent some time on was trying to get Firefox to send an If-Modified-Since header. After some testing it only sends it if the cached response had a Last-Modified header. I suspect that's per the spec, though I had naively thought that it might just use the Date header in the absence of a Last-Modified header.

I thought it would be nice if nginx could just set the Last-Modified header for cacheable responses (that lack a Last-Modified header) to the current time, but I couldn't find how to do this. And it's not a big deal really, I don't think many people request the same page more than once unless they're expecting it to be updated. If it was a big deal, then sending the Last-Modified header from PHP would work fine.

Making a folder name end in a dot

I had a problem today when I tried copying some files from my Linux VM to my Windows host OS today. The problem was that one of the folders ended in a dot / period, and my backup software (Beyond Compare) couldn't create this directory. Looking into it, I found that apparently Windows / Windows Explorer doesn't support directory names ending in dots, but the underlying file system (NTFS) and Windows API do.

I managed to find the solution here: Making an folder end with a dot. Basically you need to specify the full path, and precede it with \\?\, e.g. mkdir "\\?\C:\path\to\dir.". After doing this, Beyond Compare could copy the files across to the folder with no problems.

However, weirdly the folder shows up in Windows Explorer, and is not hidden / inaccessible, unlike what all posts on the web seem to indicate. Also, a folder without the dot on the end of the name has been created. Both folders are identical, and I guess one is actually a link to the other.

Looking at the 8.3 directory names, the folder without the dot on the end, has a .C tagged onto the end of its 8.3 name (so it is 10 chars long rather than 8). It is definitely listed as a dir rather than as a file though.

No lib or include dir in Ghostscript?!!

Today I had a lot of problems trying to compile ImageMagick with Ghostscript support. When I configured ImageMagick using the --with-gslib option it would show in the configuration summary that it would be compiled without ghostscript support. (There are three columns, the first lists the delegate lib name (ghostscript), the second whether it was requested to include that lib (yes) and the third whether it would be included (no)).

So I downloaded and built ghostscript-9.18 from source, but when built the install directory contained no lib or include dir! Sadly I couldn't find anything similar by searching the web. I found some information that said you need to use make so rather than just make to build shared libs. But that didn't make any difference. Then in the ghostscript build / source dir I found there was a folder called sobin or binso I think, which contained two bin files and the shared libs.

So I created a lib dir inside the dir where ghostscript had been installed to, then copied the ghostscript .so files across. Running ImageMagick's configure again I now got further - no (failed tests) was the status for gslib. Checking config.log in the ImageMagick source / build dir I found it was looking for some missing ghostscript header files. I found the files it was looking for in the ghostscript source / build dir, inside the psi sub directory. Rather than try and pull just the needed files, I copied the entire contents of the psi directory to an include dir I created inside the ghostscript install dir.

That made no progress, but looking at ImageMagick's config.log again it was easy to see why - it was looking in a ghostscript dir inside the include dir, and I had just pasted the files straight into an include dir. So creating a ghostscript dir inside the include dir and moving the files from psi to there fixed that problem. Almost.

Running ImageMagick's configure again, there were still some missing ghostscript header files. I found one in the ghostscript source / build dir under the base sub directory, so again I just copied the entire contents of that directory into the ghostscript install dir include/ghostscript dir. And ImageMagick would now configure and give me a yes for Ghostscript!

I should mention that I have ghostscript installed to a non-standard dir using the prefix configure option. So I had to also specify LDFLAGS and CPPFLAGS when configuring ImageMagick, so that it could pick up the ghostscript include and lib dirs after I had manually created and filled them.

There was just one last issue - under delegate programs ImageMagick's configure output was showing an older version of ghostscript. My guess is that my OS (Ubuntu) has ghostscript installed, but not the development libraries. So to get ImageMagick to see my stupid install of ghostscript, I just had to specify the path to its bin folder in the PATH var when configuring ImageMagick.

Of course, it may well be that my ghostscript install is completely broken and ImageMagick won't actually be able to do anything ghostscript related when I try it. But at least I got it to compile with ghostscript OK!

Oh yeah, another weird thing when configuring ImageMagick was that when I specified the PKG_CONFIG_PATH for other libs (jpeg, tiff, lzma) I have in non-standard locations, it found some of them, but other ones it didn't. I had to use LDFLAGS and CPPFLAGS for the ones it didn't find.

P.S. PKG_CONFIG_PATH and PATH use a : colon as the separator, LDFLAGS and CPPFLAGS use a space.

Needless to say, I don't plan on trying to compile ImageMagick with ghostscript support on my actual web server.