Super HDD activity caused by Albion Prelude - fragmented video file

[X-Me]

Yesterday I started playing Albion Prelude.
I noticed something I'm not comfortable with and I'm here with the hope that the issue can be resolved or at least mitigated.

Whenever I Comm to somebody and his animated portrait appears in the top right corner of the screen... or also when I'm docked at a station and I look at the wares in the Trade window and the spinning blue image of the selected ware appears in the top right corner of the screen...
... my hard disk starts working like crazy.

The noise is unmistakable because the hard disk is currently outside of the PC case and I can listen to it very well.
As soon as that window is shown, and a video starts in it, my HDD begins working abnormally. As soon as the window is gone, the HDD stops working and is silent again. Every time.

While this might not be a concern to most, believe it or not, that super-activity is the kind that shortens the life of a unit considerably. So I would like to stop it.
Anybody knows what causes it? It's the animated video being shown, alright, but... 'why'?

I searched around for ways to prevent that window from showing, or at least animating, but haven't found anything.
Is it even possible to have a still image rather than an animation?

[Allan F]

Get a SSD, and install the game there.

[RAVEN.myst]

Hmm, this is strange indeed. I, too, am highly sensitive to hard-disk thrashing and its MTBF-shortening effect, and I also am still using magnetic platter drives, so I tend to notice it - very quickly, in fact. I won't annoy you with inane questions about anti-virus software running in the background, or such - what you describe seems to indicate a very clear correlation. I haven't ever noticed this behaviour in my installation, though I will fire up AP a bit later and make sure. At present, there is only one thought that occurs to me: as I am rabidly anti-Steam, I have thoroughly "de-Steamed" my installation (previously, this wasn't even necessary, as up until X3TC Steam was strictly optional) - if your game is in its default "Steamy" configuration, maybe what you are experiencing has something to do with that. This is pure conjecture on my part, though, please understand that. I will keep an eye open for this issue and possible causes and solutions, and will report here if I come across anything. (Sorry I can't be of more help.)

EDIT: It occurs to me that, just possibly, what you ask about replacing animations with stills is something that could be quite easy to mod. Maybe you could send out some feelers in that "world"? (ie. in the scripting & modding forum)

[X-Me]

Get a SSD, and install the game there.

Hi,
I'd like to keep using this HDD. It's the point of the post: I don't want to [be forced to] change disk.
As a last resort I got RAM to spare and may create a RAM disk to run the game from there. But it's overkill. To turn the light Off you hit the switch in the room, you don't severe the power cable into the wall

AP runs as smooth as silk on my rig, and I wouldn't complain about a thing if it wasn't for the abnormal HDD activity observed when that portrait window is shown (and I remark: only when _that_ window is shown).
Can we find a simple solution that won't involve a major change in my machine?

[Alan Phipps]

The game movie file MOV/00001.dat probably contains those ware clips and NPC character videos and it/they could be edited out I suppose. The file is just an mpeg file of all the various video clips added together. (Back it up or rename it first in case that breaks something important.)

I think MOV/00002.dat contains the floating advertising signs feed.

[X2-Illuminatus]

Adding to what Alan said, it's the X3 Terran Conflict\mov directory (not the one in the addon directory). Before trying to edit these files, I would suggest to simply rename them and test, whether this fixes the problem already. The communication screens (ware windows) should stay blank in this case.

As Alan mentioned, these files are just video files, so I wouldn't know why playing them would cause the problem you're describing with regards to your HDD activity. I also didn't experience such a problem myself until now. And while I don't have an external HDD, the one in my laptop is also a noise-weakness. So any activity (greater than normal) rarely stays unnoticed.

[X-Me]

The game movie file MOV/00001.dat probably contains those ware clips and NPC character videos and it/they could be edited out I suppose.

You nailed it!
If I rename the mov\00001.dat file, the portrait window shows up black/empty.
No movie playing, no disk trashing whatsoever. And the game doesn't seem to mind.

I can alt-tab to windows (with game running and portrait window shown) and undo the file-renaming, and when I'm back in game the portrait movie is now playing and the trashing resumes instantly.
Therefore it _is_ that movie.

And just now I noticed in the tray notification area a blue icon with the 'FFa' writing on it while I play AP.
Important question: Is this something that came with X3:TC/AP ?


@ X2-Illuminates:
Hi,
the fact that you guys don't have this problem is a good sign to me.
If it's not part of the game, there's hope I can sort it out. Fingers crossed

[Jimmy C]

A few questions, X-Me. Do you watch videos (local videos, not youtube) on your PC? If yes, do you use video codecs? Which ones? Are you using 32-bit or 64-bit windows?

The videos in X3TC and AP have trouble with certain codecs if they're installed on your system. I use 64-bit windows, so I was able to swtich to 64-bit codecs and media players and bypass the issue.

[Cycrow]

No its not doing that somes with X3:TC/AP.

its part of FFDShow, which is a codec, most likely instaleld as part of a codec pack.

It opens with the game because because it will have replaced the original codecs that the game tries to use.

This will be used to decoded the audio/video files which is what might be causing your problem.

if you are not using it for anything, you could probably just uninstall it. But its possible to edit the settings, so it doesn't enable for X3

[RAVEN.myst]

Talk about video files made me think two thoughts:

1. Is your drive compressed? If so, you could find some files (especially large ones) causing excessive drive activity (even though the actual conventional compressibility of video files is poor.)

2. Video codecs, as Jimmy C has also pointed out. I recall recently (by accident, while searching for something else) hitting lots of old forum posts about certain specific codecs (specifically FFDShow, as Cycrow points out) which cause trouble with the X3 games - this seems to track with your systray observation.

[UniTrader]

just asking because it was not mentoined specifically: you have enough RAM? because the HDD activity could also be caused by paging/swapping (your idea with the ramdisk suggests that you have far more than enough, but i want to know for sure )

[Sinxar]

Or it could be heavily fragmented. The most common cause of disk thrashing.

[X-Me]

Hi everyone,
I got good news and bad news. Let's go with order.

Just now I ran Albion Prelude in windowed mode, to keep an eye on the tray notification area.

Splash screen: no icon.
Egosoft logo movie: no icon.
Quote of the day: no icon.
Loading screen: no icon.
Main menu: the FFa icon appears in the tray.

Makes sense. At the main menu there's that floating ad-sign playing the 00002.dat movie, as mentioned by Alan Phipps.


@ UniTrader:
I use Windows 7 Ultimate x64, with 16 GB of RAM. The system idles at 1.67 GB usage, rarely at 1.84 GB. The rest is free for all.
Albion Prelude at the moment barely takes more than 1.1 GB of RAM. And with the Large Address Aware flag set in the exe it could well consume up to 3 more GB before resorting to paging (although some specific mechanisms in Windows use paging regardless of RAM present, and a pagefile -however small- is always good to have).


@ RAVEN.myst and Jimmy C:
The FFDShow was installed in my system by the military flight simulator 'Falcon BMS 4.32' a long time ago. That's why I asked if AP would install it as well, I thought that maybe I was having an incompatible version (newer or older) than what AP wanted to use. But turns out that AP got nothing to do with it...

I then found this thread from user Carlo the Curious.
I followed the instructions to have FFDShow ignore the game executables, and it works: FFDShow doesn't appear anymore in the tray... but nothing else changed, and the portrait window continued to cause abnormal HDD activity. Rebooting didn't help either.



And then Sinxar suggested the impossible: fragmentation.
So I selected the 00001.dat file and made a copy of it in the same folder (just: Ctrl-C, Ctrl-V).
- I renamed the original file in '00001.dat.bak'
- I renamed the copy in '00001.dat'
Then I launched the game.

With the copied file, the portrait window trashed my HDD a lot less. Barely, in fact.
I changed filenames again to reinstate the original situation, and the portrait window trashed the HDD badly.
I swapped files yet again, and the trashing was barely noticeable.

So... I guess that Sinxar has found the culprit :-/
Might be that the original 0001.dat file is split on a ton of fragments, hence the crazy I/O, while the copy I made is much less so, and the noisy I/O barely happens.
Strangely enough, Windows' Defrag utility won't give me a report. It keeps saying that I have 0% fragmentation on all partitions, which cannot be: I never defragmented this hard drive before.

What should I do now, defrag 250 GB? It's gonna take forever.
And it'll be an I/O hell in itself for this poor disk. Oh man...

[RAVEN.myst]

Wow, it's been a while since I've encountered problems due to drive fragmentation - because since Win7 fragmentation management has been built into the OS, it's been more or less eliminated as an issue - but not always, apparently. And yes, Windows Defragmenter has never been super-effective on demand (though it seems to limit fragmentation decently in the background - usually!)

So, how to solve your problem: I believe you already have. Simply switch to using your copied version of the culprit file permanently. (Incidentally, though rather obvious in retrospect, I wouldn't have thought of that way to test for fragmentation - most ingenious!) If you are feeling cautious, you can always keep the original one until you are confident, though I expect that shouldn't prove necessary, ultimately.

Alternatively, if you want to go the on-demand defrag route, you might consider looking around for a decent third-party one (I used to be quite an expert on these, until Win7 came out and it became unnecessary, so I don't know how it looks out there these days), and if you want to avoid a full defrag, maybe just run a "defragment files only" or equivalent (some old defragmenters even allowed specifying specific files to process.)

[X-Me]

Hi,
I'm willing to try a 3rd party defragmenter.
But my experience with them is nil -- I'd appreciate if you named me some good softwares you know, please.

If possible I'd avoid a full drive defragmentation. Better do just the problematic files, like the one found today.
Minutes ago I asked Windows' Defrag to fix my drive, but looks like it did absolutely nothing.
Over the years I learned to never fully trust Microsoft's tools anyway...

[Alan Phipps]

Just a word of caution. Many of the utility suites (and even things like Norton) have defrag optimisation tools included in their arsenal - but many of these just quietly hijack and call Windows defrag using a fancy branded GUI screen.

[Nanook]

There's another, and probably safer**, way to defrag your hard drive, if you have a spare drive around somewhere and an external case to put it in. Simply connect the new drive to the system and copy the files from the fragmented drive to the new one. Copying won't duplicate the fragmentation so all the files should have contiguous space on the new drive.

Then, if you still want to use the original drive, erase it and copy the files back to it. You now have a totally defragmented drive, and no third party utilities needed.

**I say 'safer' because when a drive is highly fragmented, there's always the possibility of disk read/write errors during the defragmentation process. Whichever way you go, it's best to run a check disk utility first, such as the one that comes with Win 7 Win 7 disk check, or just use chkdsk from the command line.

And as this is a tech support issue rather than gameplay, I'll move it to the Tech Support forum