Manifest issue when used in combination with asset_hash

[kseyedmehdi]

Hi,

I've tried the manifest flag but unfortunately it either doesn't work or it does work but not as I expect it.
When i run middleman build, the build steps for my site is in the following order:

1. copy over / generate updated html & images etc.
2. minify_javascript
3. minify_css
4. minify_html
5. imageoptim
6. asset_hash
7. gzip

So each time i deploy, step 1 updates all the images, causing step 5 to optimize the images from scratch which for me takes up to 20+ minutes.

What i kind of hoped that the manifest flag would do is that it would prevent step 1 to copy over images unless they are really updated. Now step 1 copies over the images to the build dir from source because the build version is not similar which is correct because it's optimized.

Any ideas on this?

[plasticine]

Hmm—interesting. The initial copy should be skipped as images already in the manifest should be seen as identical. Can you provide any more info? Platform, manifest contents, etc? Also are you able to run the test suite locally and have it pass?

Edit: btw, the test suite it run using ./script/spec from the command line.

[kseyedmehdi]

Yes, the test suite runs and passes locally.
I'm using a mac running OS X Yosemite 10.10.2

https://www.dropbox.com/s/lg2tow7u5hdhuo6/imageoptim.manifest.yml?dl=0

Screenshot tests passing:
https://www.dropbox.com/s/ac0awwd07gz6mqh/Screenshot%202015-01-07%2021.19.19.png?dl=0

[plasticine]

@kseyedmehdi hmm—thanks. Any chance I can grab the version of middleman also—just want to try and make a failing test case so I can reproduce. That manifest looks fine I think...

[kseyedmehdi]

@plasticine i'm using Middleman 3.2.2

[plasticine]

@kseyedmehdi As of 14d6b07 I’ve added the Appraisals gem to run the test suite in a bunch of Middleman versions (including 3.2.2). I was hoping that this might catch the issue, but didn’t: https://travis-ci.org/plasticine/middleman-imageoptim/builds/46257162

Are you able to try and make me a simple failing test-case so that I have something more to go off?

[kseyedmehdi]

@plasticine I tried some different things and i think i've figured out what is causing the issue.
asset_hash is probably causing the issue. If i disable asset_hash gem, the manifest flag works.

This is what is probably going on during the build process:

1st build run:

1. copy over / generate updated html & images etc.
2. imageoptim
3. asset_hash, adds hash to all asset filenames (jpg, css, js etc.) therefor touches the images

2nd build run:

1. copy over / generate updated html & images etc. (we expect this step to be skipped, but it doesn't because the images are touched by asset_hash in the previous run thus timestamps don't match
2. imageoptim (re-runs because the previous step is NOT skipped)
3. asset_hash, adds hash to all asset filenames (jpg, css, js etc.) therefor touches the images

In general I think it's maybe better to use a SHA2 of the file contents instead of a timestamp? Is this possible? ( i tried some things in a fork, but don't really know how the manifest prevents the initial copy.

[plasticine]

@kseyedmehdi Ahh—cool I guess that makes sense, though I’m surprised that asset_hash is changing the file mtimes. It doesn’t seem like it should need to.

Anyway, I like the idea of just hashing the files — that’s probably more robust anyway.

[kseyedmehdi]

@plasticine i continued digging into this and figured out that asset_hash is triggering the initial copy.

asset_hash does not have a manifest like imageoptim thus compares the images from the build with the images from the source folder, and these are of course different because the ones in the build are optimized, and this triggers the initial copy. So even the use of an asset for manifest instead of mtime will not help my situation... basically asset_hash should create a manifest based on the source images and check if those are updated and then decide to trigger the initial copy or not. Thanks anyways for looking into this!

[plasticine]

Ahh — of course it does. Hmm, yeah I’m not really sure if there is a way around that. Perhaps I’ll try and chase up the Middleman core devs.

[kseyedmehdi]

It would be awesome if you could ask the Middleman core devs to look into this.

In meanwhile i'm going to copy the optimized images back to the source folder, unfortunately this seems to be the only feasible solution/workaround in short-term.

Thanks in advance!

[codyrobbins]

I’m having the same issue because of asset_hash. Would a potential workaround be for middleman-imageoptim to maintain its own cache directory where it stores the manifest and keeps the results of its optimizations? Then it could check there and dip in to get an already optimized version of an image before it bothers to optimize something from scratch again. Even once this issue with asset_hash is resolved I’m still going to have this problem unless there’s some sort of caching because my build directory is cleared between builds and therefore it’ll still have to re-optimize every image on every build.

This may be a larger architectural flaw in Middleman itself that makes more sense to address there. I feel like it shouldn’t rely on the build directory persisting between builds but should have some sort of separate caching layer where it stores intermediate build products that it can re-use between builds. The gzip plugin has a similar problem to this situation where it’s maniacally deleting the previously gzipped versions of files on every build and then re-gzipping them at the end of the build even if the original files haven’t changed.