Yes, I know that I’ve said before that Windows Live Photo Gallery 2011 is a disaster, but more problems caused by using it just keep crawling out of the woodwork.
The first problem I stumbled across was that if you are a photographer who uses IPTC metadata to record information about where your photos were taken, then WLPG 2011 will write false GPS data into your photos without telling you that it is doing so.
I, and others, have reported this issue to Microsoft, and I understand that they are looking into ways of correcting it. Meanwhile, I’ve become aware of another issue with WLPG 2011. It also screws up the Exif section of the metadata in images.
While trying to scrub my images clean of the false GPS data inserted by WLPG 2011, I noticed ExifTool was reporting that many of my images had problems with their Exif metadata. Often it was a simple warning that the Makernotes in the Exif section have been damaged. This is a warning from ExifTool that another utility has written to the Exif section and damaged the structure in some way.
In many cases, however, ExifTool is reporting that more serious damage has occurred and some of the data written into the original Exif section by the camera that took the image has been corrupted.
Here’s a screenshot that shows an example of an image exhibiting both types of issue (warnings and corruption). The screenshot is of Geosetter, and the highlighted image shows errors being reported by ExifTool (Geosetter uses ExifTool under the covers to do all the heavy lifting). Click on the image to open it full sized in a new window.
You’ll notice that the thumbnail, and some of the others, have a dark blue marker pin in the top left corner. That indicates that the image contains GPS information. But the interesting thing is that, for that particular image, I did not supply GPS data; WLPG 2011 has inserted it by itself.
Now here’s a screenshot of one of the other thumbnails that Geosetter is indicating have GPS information. For these thumbnails, I explicitly inserted GPS information myself, in other words, WLPG 2011 has not had cause to write anything out to the files. Notice that for this thumbnail, ExifTool is not reporting any errors or warnings. That’s the case for all the images where I have explicitly added GPS information.
By the way, even though these two shots were taken at the same place, the GPS inserted (without my knowledge) by WLPG 2011 is wrong, and is located 500 metres distant.
I’m now going back through my images, and as far as I can see, all those which are being reported by ExifTool as having problems with their Exif metadata are ones that have had GPS information inserted by WLPG 2011.
The really irritating thing about this discovery is that WLPG has a track record of not dealing correctly with Exif metadata. Previous versions of WLPG have been reported as corrupting the Makernotes (data written by the camera manufacturer) in Exif.
It would appear that nothing has been done with WLPG 2011 to address this issue. So not only does it insert gratuitous, and false, GPS data into your images, it will also screw up their Exif metadata.
I repeat, this is a disaster.
24 Hours Later
So, I’ve been looking into this a bit more, but the more I look, the more I think: OMFG. The damage that has been done appears to be quite extensive, and will take some time to repair.
Today, for example, I decided to examine just one folder of photos and compare the contents with the contents of the same folder as it was in a backup taken on the 1st June 2010 – a date chosen because it was before any of the betas of WLPG 2011 had been released to the public. On that date, I would have had the previous version of WLPG installed and running. The first beta of WLPG 2011 wasn’t available to the public until the 24th June.
I looked for a folder that had photos containing entries in the IPTC Extension “Location Created” metadata fields. These fields are used by WLPG 2011 to store textual information for a location (e.g. the street address, city, state and country) in the image files. I don’t use these fields; I use the older IPTC Core “Location” fields for this purpose. So if I find an image file that contains IPTC Extension “Location Created” metadata, then I know it has been touched by WLPG 2011.
I chose a folder containing 24 photos that had been taken back in 2007, and which had IPTC Extension metadata present. I then got the same folder from the 1st June backup to compare the two side by side.
Here’s a screenshot of the folder, as it was on the 1st June, being displayed in Geosetter, with the metadata of the selected photo being shown (click on the screenshot to see it full-size in a new window):
Now here’s a screenshot of the same folder as it currently exists in my computer. The same image file has been selected to show its image metadata:
I’ve expanded some of the more interesting metadata sections. As you can see, the metadata has changed substantially. Let me list the ways:
- ExifTool is now listing a warning about a possibly incorrect Maker notes offset, together with three warnings about invalid camera data in the Exif section.
- While the original (backup) file had 98 elements of camera maker data in Exif, the current file has now only 11 left.
- The current file now has GPS metadata present in the Exif. This has been inserted by WLPG 2011, not by me. You will note from the other thumbnails in the second screenshot, that all the other files are also showing that they now have GPS data in them. None of the original files had GPS data. By the way, the GPS data is also incorrect by 300 metres.
- The original file had its Exif byte order in little-endian fashion; in the current file it is big-endian. According to the guidelines of the Metadata Working Group (of which Microsoft is a founding member), the “endianness” should be preserved, not reversed.
- The original file had a filesize of 3.1 MB; the current file has shrunk to a mere 1,553 KB.
- The current file now contains a JFIF block, which is not present in the original file. It also has changed YCbCr values, possibly as a result of this.
- The current file now contains an IPTC Extension metadata section, which lists textual information for the “Location Created”. This section is not present in the original file.
- The original file is showing that it was last modified on the 27th November 2009. The current file is showing that it was last modified on the 30th September 2010, which also happens to be the date of the final release of WLPG 2011. This is not a coincidence.
There may be other, more subtle, differences between the original and current versions of the files, but I’m already disheartened enough by the above list, particularly by the Exif corruption and by the fact that my JPEGs have been compressed in size without my knowledge or permission.
I suppose I can cut my losses by doing a full restore of the photo folders from the backup taken on the 1st June, but this will still not take account of new files that have been created since that time, or of older files that I have been working on.
What a mess. Thanks, WLPG 2011.
48 Hours Later
Oh gawd, it just keeps getting worse… I had hoped that WLPG 2011 was only corrupting Exif metadata when it actually changed the metadata, for example when it added (false) GPS coordinates to the Exif section.
After further examination of files today, I have discovered that WLPG 2011 will merrily corrupt Exif metadata even when it doesn’t need to change any of the Exif content.
You see, one of the new features of WLPG 2011 is automatic face recognition. When it discovers what it thinks is a face in a photograph, it will ask the user to confirm the person’s name. Once it gets confirmation, it will then write XMP metadata into the image file. This XMP metadata is structured according to Microsoft’s People Tag. However, when WLPG 2011 writes this XMP metadata out to the file it will also (a) corrupt the Exif metadata section and (b ) compress the JPEG image.
I’m afraid that I’ve been confirming face tags suggested by WLPG 2011. And now, every single one of those images that contain face tags has also had its Exif corrupted.
What a f*cking mess. Thanks, WLPG 2011.
Update 23 November 2010
I thought that it would be worthwhile to report that Microsoft are listening to those of us who are reporting tales of woe caused by using WLPG 2011. Please see here for a status report on where I think we are…
Update 2 December 2010
There’s an update to WLPG 2011 that addresses the geotagging issue. See here for more information.
Geoff Coupe's Blog 
Geoff, is this only happening because of WLPG? or is it from Windows in general?
JL – nope, this is being caused by WLPG alone. Remember, it’s only those images that have had their Exif touched by WLPG when it writes (false) GPS coordinates into them. Other images that have had their Exif touched by other tools are OK as far as I’ve found.
I have photos backed up to Carbonite that go back all the way to December 2009. Some of them at least. They’re slack about deleting files over 30 days old as they say is their policy. It would take forever to download them all. Right now I have a 2GB folder restoring and the estimated time is 4 hours. My total is about 30GB I think so that’s 60 hours. Some of it has been overwritten in the meantime so I’d be picking through looking for anything ‘less-damaged’.
There’s been so much change to my files even in the month before these other trials began it may be a thankless task anyway.
At the time I first discovered the geotag problem I had just spent 3 days getting rid of the AFCP-IPTC junk from thousands of pictures and backing up the changes to two external hard-drives. So there’s nowhere to retrieve recent ‘clean’ data from. AFCP was a total shock to me. I thought my collection was in pretty good shape.
The EXIF corruption is just the latest in a long series of problems. Is it reversible? Anything ExifTool can do to bring the right stuff back?
JL – I’m not sure whether ExifTool can remove the corruption, I think it can only rewrite the Exif to remove suspicious offsets.
However, I’m no expert with ExifTool – suggest you ask this in the ExifTool forums…
Corruption of maker notes is a long-standing problem with both Windows software and Picasa. I don’t really think the software engineers care about this problem or it would have been fixed long ago. These two are the worst out there. Other software such as PhotoMechanic does a much better job of handling the maker notes.
– Phil
JL – I’ve added some more info to this blog entry – see the “24 Hours Later” section. Hope you have a stiff drink handy…
There goes the wine cellar.
Thanks, Phil. Oh, and a heap of thanks for ExifTool – a really brilliant utility…
Here’s my theory, Phil. They (Microsoft and Picasa) think the ‘mainstream’ that they cater to won’t notice and, therefore, won’t care. And anyone else should be smart enough to stay away from their products.
Sorry for the errant italics tag. I think you’re set for 3 levels deep on the replies so I’ll continue the saga here and we’ve written an encyclopedia already.
-I have deleted multiple fields of AFCP-IPTC from thousands of photos except for a hundred or so that are so corrupted nothing will move them.
-I reset all GPS co-ordinates to zero on 10,000 photos and rewrote the valid to about 5,000. Another hundred or so too corrupted to be re-saved.
-I have removed reams of XP data one field at a time from untold thousands of pictures, except for a few dozen that won’t budge.
-I have reset zero co-ordinates on 10,000 photos to null. Except for the hundred that won’t accept change and send up long error messages.
-I have checked 10,000 photos for missing IPTC on 7 different fields. About 2,000 to go.
I’m sure I’m missing something because that sounds like such a short list.
Most of this was done before you dropped the EXIF bombshell. While your mind may have strayed only as far as the back garden, mine’s been all the way across town twice and impounded once.
Microsoft/Picasa/whoever is building cities on toxic chemical dump sites. There should be a certification process for photographic software. Software has to be built to standards; it has to be upgraded to standards. Either you get certified or you don’t. But no-one gets to screw around with the public anymore. I’ve spent every waking hour for two weeks already cleaning up this mess and if there’s a light at the end of the tunnel I’m sure it’s a train coming.
JL, I’ve noticed that many of my older photos have XP tags in them. However, I’ve got plenty of files with them where the Exif metadata is still OK, so I’ll probably not bother scrubbing them out. With the advent of Vista and Windows 7, the XP tags are no longer used anyway as far as I can see.
What I have found is plenty of proof that earlier examples of WLPG were also screwing up Exif, and unfortunately I no longer have backups that predate the damage…
If there’s an upside to be found anywhere, (and it does seem to be slim pickins’) a lot of my older photos were scanned. 125-year old portraits saved from the brink of total deterioration and many bags of photos sent to me by various people. These, of course, do not have makernotes. I’m not sure about anything else because I assume it would take a comparison, picture by picture, with “pre-damaged” versions.
However, clearly, some of them give a warning of “Bad IPTC data tag” and all kinds of other error messages when I tried to re-save after deleting errant tags and GPS. Thankfully, some of them had no address so were not hit by the geotag disaster.
I must say, if Microsoft thinks they can just apply a quick fix to ‘the geotag thing’ they may find themselves having to dig up the whole sewer. That could only be good.
I have so many photos (12,000) spread across so many folders (125 folders at last count) so I don’t yet have a detailed assessment of the damage. My own pictures taken on a Canon seem unaffected by EXIF problems, and yet photos sent from a cousin’s Nikon are riddled with EXIF warnings.
P.S. The camera model is curious. Most of my digital pictures have GPS co-ordinates. Strangely, the ones that don’t (but do have state/country addresses) seem to have been excluded from EXIF damage.
My cousin’s pictures don’t have GPS but they do have addresses so that would have included them in the WLPG train-wreck.
Nope, not curious. My mistake again. The ones that presently don’t have GPS had it when WLPG happened. I missed them when I was copying out the co-ords before mass-deletion. They’re on the U.S./Canada border and I didn’t look them up again yet.
I’m going through folders one at a time (this is going to be long) using GeoSetter. Hex dump on one monitor and thumbnails on the other. Click through one at a time. When there’s an error message I use Win+A from the EverNote v.2 Univeral Clipper to send the note over to EverNote. First folder results: 49/478 images corrupted. No pattern to it that I can see.
I suspect the cemeteries and the gravestones got hit the hardest because they’re digital, they have addresses and no GPS. There’s hundreds and hundreds of them. Going back undercover …
Second folder, gravestones: 55/258. The percentage just doubled and a lot of them were scanned from photographs, so that makes it even worse. Fortunately, this batch is salvageable, assuming my cousin hasn’t upgraded her computer lately.
There’s no way to get rid of the zero co-ordinates on 46 of them. Here’s an example of the output:
Warning: Truncated PreviewIFD directory. IFD dropped. – C:/Users/JL/Documents/ANCESTORS/BEEKEN HISTORY/Photographs/Beeken Graves/0002 grave, beeken, hector.jpg
Error: [minor] Bad PreviewIFD directory – C:/Users/JL/Documents/ANCESTORS/BEEKEN HISTORY/Photographs/Beeken Graves/0002 grave, beeken, hector.jpg
Warning: Truncated PreviewIFD directory. IFD dropped. – C:/Users/JL/Documents/ANCESTORS/BEEKEN HISTORY/Photographs/Beeken Graves/0002 grave, beeken, hector.jpg
Error: [minor] Bad PreviewIFD directory – C:/Users/JL/Documents/ANCESTORS/BEEKEN HISTORY/Photographs/Beeken Graves/0002 grave, beeken, hector.jpg
First batch of ANCESTORS, one side of family. 269 corrupted, some of it IPTC problems, a lot of it EXIF. Two folders of digital camera pics (about 150) entirely EXIF-damaged. Miraculously, my Source Library was untouched, even the gravestone copies included.
Going back into other side of family; the one with most of the photos. I dread this. I need some wino first. O dear, I drank the whole case yesterday.
I’m at 1956 now and slowing down. The safe ones: the scanned batch with addresses & GPS or no addresses & no GPS.
On some of the later digital ones it’s not as bad as I thought. I hadn’t gotten around to putting addresses on many of them yet. There’s something to be said for procrastination.
I spoke too soon. The whole 1957 folder is corrupted.
These are the folders affected:
1950
1957
1961-1978
1984-1993
1997-2003
I stopped marking individuals files because there’s too many of them. This seems to affect mostly pictures with addresses & GPS. If there’s neither it leaves them alone but not always. Many photos from the 1950’s have both and they’re fine.
The most common errors are:
Warning: [minor] Empty PhotoshopSettings data –
Warning: IPTC pointer references previous IPTC directory –
My digital camera starts in 2004, almost all pictures have addresses & GPS and the problems stop. Before then all photos were scanned.
A titch of good news. I’m able to use GeoSetter to get rid of the warnings on the hex dump although it’s still giving me a warning in the Save Changes dialog:
Warning: Duplicate PhotoshopSettings tag in IFD0
Methodology – search pictures with, say, common GPS, then make a small change, such as adding Altitude, then Save Changes and the warnings go away.
This is now the general survey of the damage.
The DESCENDANTS cleaned up pretty well. On many hundreds of photos GeoSetter said it couldn’t save the changes but ExifToolGUI shows the warnings gone now.
ANCESTORS did not fare so well. There are several folders with completely corrupted EXIF. The upside is that they can probably be replaced because they’re entire collections sent from cousins who (hopefully) still have them. One other can be replaced because I’m pretty sure I sent it on disc to a cousin for backup.
There are many other photos that came from online for my non-public projects that I might be able to hunt down again.
There are a dozens of others with a variety of error codes that GeoSetter doesn’t know what to do with.
Posted 5 hour ago on “Windows at Home”:
“Windows Live Photo Gallery is a free application and part of Windows Live Essentials. If you haven’t yet downloaded Windows Live Essentials, now’s the time! You can upload photos, albums, or slide shows for anyone to view or for just those people you want to share them with. You can post photos for others to comment on, tag photos, edit and organize your photos, and even create group albums or slide shows that you can publish to the Internet. Our article Create an online photo album is a great place to get tips and how-to instructions on setting up an online album.”
http://www.microsoft.com/athome/email/keepintouch.aspx
The Windows Live Essentials link goes to WLPG 2011. Do these people have a communication problem?
I found your latest ’48 Hours Later’ segment. That’s just obscene. I go from shock to rage to heartbreak in a continual loop.
At one point yesterday I was so distracted I let a pot of water boil dry on the stove and it melted two egg-poaching cups. I’d had those poaching cups for 20 years.
Certainly the EXIF is the worst of the damage so far. The GPS was time-consuming but, at least, there was an eventual end to it.
Except for the fifteen 14-hour days I’ve spent so far, it looks like I might come out of this surprisingly well. I ran everything through GeoSetter and now that I can see the list of the unfixables, I think most of them are retrievable, either from privately-held collections (where the EXIF damage is the worst) or from discs sent to my sisters within the past year or so. I didn’t remember I had those resources until I saw the list.
If it was any of my personal photos over the past 35 years I’d be totally out of luck since there’s been no copies sent away, except for what might be found at Carbonite. If they’re not with me they’re nowhere.
I’m not sure this signals light at the end of the tunnel for me (that really is light and not another freight-train hauling manure) since I’m working with a profound lack of technical knowledge in this area. What I’m hoping is that if GeoSetter doesn’t show errors that means there aren’t any of serious significance. Please tell me that’s how it works. Even if you have to lie.
JL – sorry about your cups; collateral damage, as they say…
As far as I know, Geosetter is just reporting what ExifTool is saying, so therefore if there are no errors or warnings being reported, then nothing’s been found by ExifTool itself.
I’ve been asked by Carmen to provide some samples, so I’m just assembling a little collection of before and after images to illustrate insertion of false GPS, Exif corruptions caused by this and other operations of WLPG such as face recognition, cropping, colour adjustment, etc. The list goes on. Basically, if WLPG touches your file, it will screw up the Maker notes section in Exif.
O, yeah … they need proof? How odd. This has been going on for how many years? Phil?
It’s about noon, haven’t had breakfast yet (no egg-poaching cups) and this is the state of my world:
-I was able to turn around dozens of stubborn ones by re-saving them in Adobe.
-I lost two Nikon slides from 1977 but got them back from Carbonite, last backed up in early September. They were already corrupted and still are.
-I found a couple more photos online. One was already corrupted.
-I sent a long list to a cousin of jpg’s with EXIF damage. I anxiously await their return.
-There’s 5 others with EXIF damage to retrieve from my mother.
Except for the remaining 2,000 to recheck for missing IPTC I think I’m done. Basically what saved my butt is that the majority of my photos were scanned.
I’m back in the queue of the walking wounded. More accurately, crawling on my knees.
My scanned tiff’s have also been affected by the EXIF damage. Not only does WLPG compress jpg’s it also compresses tiff’s.
I did not submit them to any of the functions within WLPG; tagging, face-tagging, color correction or anything else you mentioned have affected yours. They were simply exposed to WLPG by having their folder imported. WLPG was on my computer for one week before I uninstalled it. And only about a minute after I discovered the GPS damage.
I always save my scanned files as uncompressed tiff’s. I’ve never done it any other way. These are photos that go back as far as the 1860’s. There will not be another opportunity to scan them. They were manually restored across thousands of hours of back-breaking work and the originals are gone. Some of them were so old they disintegrated on site. This was literally the only way to save them. I scanned them uncompressed and large enough to have the best quality for any future reproduction. Thousands of these ancestral photographs are now showing LZW compression.
Some of them were available to restore from Carbonite Online Backup and here’s a sampling of file-size results:
7.5 MB now 5.1
6.2 MB now 3.2
8.5 MB now 5.3
11.8 MB now 6.8
34.0 MB now 25.0
7.4 MB now 2.4
I have yet to print out complete profiles to see what else has gone awry. I couldn’t stand to look right now.
Unfortunately, WLPG 2011 out of the box has file compression turned on. It can be turned off in the Options panel, but I suspect most people will miss this. I think the compression of your files would have occurred when WLPG was busily writing out the geotag information into them. My opinion is that to compress the image in the file while only metadata is being written out is sloppy design, but there you go…
Well, the whole thing’s sloppy, isn’t it? Although I think ‘sloppy’ sounds like praise compared to what it actually is.
And yes, it’s obvious the compression happened along with the geotagging. WLPG only had access to my photos for 3 days (of my computer-on hours) and in that time managed to add the trash GPS, rewrite the EXIF and compress 10,000 photos from one end to the other. I can see this from the modification dates; Oct 26th to Oct 28th. A few of the photos, bless their little hearts, got missed. It was so new to me, it wouldn’t have crossed my mind that early on that compression was even an option under Options.
Well, now we know where we are. What do you think you’d like to do next?
“What do you think you’d like to do next?”
Plan for a better backup strategy, I suppose. I think I’ve got off rather lightly in comparison with you. I can at least repair most of the damage to my files, but your losses are far worse than anything I have…
OK, I looked. Only because it’s 9 o’clock in the morning, I haven’t been to bed yet and I’m feeling a little goofy-brained. Except for the Maker notes, it’s everything you listed above. No grace whatsoever given to scanned tiff’s.
I may be able to retrieve and restore some of it from old backup discs sent to other people. Those are 1-3 years old. As you’ve already mentioned there are huge shortcomings in that method. Besides the AFCP, a nightmare in its own right, I would be moving updated IPTC across about 10,000 photos. And that’s only one step in a multi-step process.
99% of my photos between 1975 and 2003 have no backup that predates this damage. Right now I’m retrieving what I can (the 1%) from Carbonite.
“I can at least repair most of the damage to my files …”
Repair or replace? Do you have a magic formula for repairing compression?
Being a novice to some of the detailed innards, I’m not clear on what the EXIF damage means.
-I see that IPTCext adds extra data that’s unnecessary to anyone except Microsoft but is it hurting anything?
-I’m suspicious that YCbCr values are connected to color contortion from what I saw on one photo. I didn’t look further because there’s only so much a person can take.
-In a broad sense I know what Big-endian/little-endian is but does this affect photo quality?
-etc with your list.
The vastness of time required to replace what I can, magnified by that time being spent hunched over a monitor is more than I can fathom right now.
Most everything has an address of some kind. So, at most recent count this morning, the ones affected by EXIF damage and compression are 8,766. Yahoo! Not quite 10,000.
Over the past two weeks I got really tired of seeing, “IPTC pointer references previous AFCP IPTC directory.” Right now it’s the BEST thing I’ve ever seen. It means it’s my old uncompressed photos back from NeverLand. It looks like I’ll still be coming up a few thousand short but right now I’ll take anything I can get.
I’m designing a process map for putting these through multiple necessary steps to eventually re-integrate them with the permanently damaged ones. If there’s no copies, there’s no copies.
“Repair or Replace?” – Yep, you’re right; I repair the damage by replacing the file with a backup copy. This works for the damage caused by WLPG 2011 and its betas (the GPS and Face recognition stuff), but, alas, I no longer have backups for any damage caused by earlier incarnations of WLPG.
“The Exif damage”: well now, that’s an interesting point. The damage that I have seen is solely to the Makernotes section in Exif. I asked the question over on David Riecks’ “Controlled Vocabulary” group whether makernotes corruption is a heinous sin, or could we just say the equivalent of a couple of hail marys and carry on regardless? I honestly don’t know the answer to this, so I welcomed feedback from the group.
David has replied to say that, while preservation of makernotes is essential in RAW files, as for JPEG images, he ddidn’t think that makernotes were really of much use…. if they are even stored in them…
So what I take away from his comments is that, so far as JPEGs are concerned, makernote corruption, while regrettable, is not a heinous sin.
However, unwanted compression of the files certainly remains a heinous sin!
“IPTCExt”? I was just pointing out that WLPG places this metadata in the files to hold the textual data for its geotags. Actually, I’m rather relieved that Microsoft have used these fields, because they are part of the latest IPTC standards, and not proprietary to Microsoft. It’s actually a good decision by Microsoft, in my opinion. It doesn’t hurt to use these fields, and is in fact the correct use of them, and forward-looking.
“YCbCr”? – I don’t understand the significance of the change here myself, I’m afraid…
“Endianness”? – This only applies to the Exif portion of the file, not to the image data itself, so far as I’m aware, so it has no impact on photo quality. I was just pointing out that Microsoft, despite being a founding member of the Metadata Working Group, weren’t in fact following their own guidelines, which I found ironic. Even more so, since I now discover that earlier versions of WLPG did not reverse the endianness of the Exif section; so earlier versions were following the guidelines…
This has been such an education!
I think I would put YCbCr under ‘heinous sins’. As far as I can see, this one actually corrupts the color. I’d take that with a huge grain of salt though, because I know nothing, I’m just saying. I was looking at a photo I know quite well. It’s actually three photos taken of myself and a friend and made into a collage. The colors were all quite strange. It’s a black and white so I wasn’t expecting to see smudges of other colors on it. And one quite large smudge. It looks like I’ll be getting back the original so I can compare. Maybe I don’t remember; it’s been so long.
And what a roller-coaster ride! Yesterday morning I was thinking I might come out of this pretty well. By last night I was at the lowest point ever. A day later I’m back on top.
The really good news is that my mind came out of panic mode and I found the un-damaged files on Carbonite. I even think I may have found ALL of them but I won’t know for sure until they’re back home again. The ANCESTORS folder I have some doubts about because the size isn’t quite what it should be. But, what confused me at first and might still be confusing me is that the files are split between two different backup drives. One from my computer that crashed in August and one from this new computer I got on October 15th. So I had to go through all the folders on both drives.
Occasionally it would ask me if I wanted to replace this file that’s 3 MB with that file that’s 8 MB. O god Yes please.
At the moment it’s telling me that the Restore will take another 28 hours. May I sell you a subscription to Carbonite?
The most recent backup date on many of them is October 21st – 2 days before I met WLPG which means they’ll have the latest version of my efforts. A lot of other ones are mid-August before the old computer died. There are also backups as far away as May 2009. And to think I wrote to Carbonite a few months ago agitating about why they didn’t clean up their server more often! After your backup exceeds 200GB they slow the speed on it to snail’s pace. I shall never complain again.
I will also be making backup discs twice a year and sending them to various places around the continent. In the meantime I’ve harassed everyone about sending me copies of what they have, no matter how old.
I had 4 hours sleep sometime during the day today after being up all night thinking this must be another mid-to-late life crisis and wondering about the meaning of life and all that. Maybe it’s time to get out of this computer stuff and go back to gardening.
I wrote to Carmen and he wrote about 5 words back. So I’ll just watch over here for how things go.
There’s some hope here again. I have set for restore anything I could find on my old computer (before August 22nd) because I know those photos are OK. (Except for the AFCP, of course.) It’s taken 12 hours so far and is set for another 10.
Since I’ve never had to restore photos from Carbonite before, I’m learning lots. If I go through the folders (now up to 74GB although not all photos thankfully) that are deemed ‘Pending Backup’ (and this has to be done one photo at a time) and click on “Restore Previous Versions” a little box pops up with older versions, prior to WLPG. I click on the date I want and it’s added to the list for restoration. I’m only starting this part of the process now. It’s going to be interesting trying to sort out this mess once it’s on back on my computer. It’s going to be interesting restoring 12,000 photos one at a time.
When I could think logically it seemed those photos had to be somewhere. If they’re not on the old computer backup and the damaged ones are ‘pending backup’, where are the un-damaged ones? Must have fallen in a crack somewhere in between. I think I found it.
Now, if Carbonite will just leave their server clean-up alone for awhile longer …
I’m afraid to sleep. Time’s a-wastin’.
This morning Carbonite is telling me the restore is going to take another 41 hours.
The photos from the newer backup-drive are coming through under different names so all those will have to be changed. The older backups are coming through with the same names. I’ll probably end up clicking through one folder/one photo at a time comparing what I’ve got. I did a lot of change during the past two weeks’ of WLPG repair, prior to this stage and certainly since August where a lot of the files are coming from, or even earlier. Also because I don’t know if all the files are coming back or not. Some of the damaged ones may have been backed up overwriting the un-damaged ones. Backup files are wherever they were across 500 days of different ‘screenshots’ of my system so it’s a big mess.
Virtually every file was affected by the AFCP so that repair has to be re-done.
This is going to take months.
Back in the cesspool.
Any photos in the DESCENDANTS folder (4,420 photos without GPS) that live in the My Pictures folder, exposed to WLPG from the day I got Windows 7 on October 15th have irretrievable EXIF damage and file compression. I don’t have WLPG on my computer anymore, but I believe the My Pictures folder is included in the WLPG index by default. I didn’t even have to click a button to import it myself.
About 1,500 of those can be replaced by backup discs from my family. The rest are toast. About 3,000.
The only saving grace is that perhaps Carbonite was slow about overwriting the good ones with the damaged ones so it didn’t affect every single picture. Time will tell.
What I meant to say was, “Any restored photos in the DESCENDANTS folder…”
Gawd, JL, that’s a mess. Yep, you’re right that WLPG will include the My Pictures, and any folders that live beneath it, in the WLPG gallery by default.
I thought that you could restore previous versions from Carbonite? Can’t you just go back to the day before you first started up WLPG?
If you’re saying that October 15th was the day you first started up your new Windows 7 computer, then presumably, you must have first loaded it up with the data from your old computer? Is that data still not available? WLPG won’t actually do anything until you start it up (it doesn’t run as a background service all the time the computer is on), so there must have been a point where good data was sitting on your new computer up until you decided to start up WLPG to see what it was…
My new computer is a replacement so the old one had to be sent back, including the hard-drive that was nuked before I sent it.
According to when I posted on my blog about this, I had WLPG open from October 26-28. I was using the tag list to see old keywords that needed to be removed. I did not use WLPG for anything except viewing the tag list. On October 28 I realized the GPS damage and uninstalled it.
Prior to that, starting October 19th, I was having a conversation with Phil at his forum about AFCP damage. I do see talk there about WLPG and the ‘bad tags’ I was seeing. So that would explain it. I had opened WLPG earlier. Carbonite had plenty of time to back up the files changed by WLPG between the 19th (or earlier) and the 28th. And, as we know, these were unsolicited changes.
When I was choosing photos from Carbonite for restoration I saw many photos with October 20th or October 21st as the only ‘last backup date’. ALL of these photos have compression and EXIF damage.
For a few photos I noticed there are earlier backups associated with the old computer paths as well as the new one. For most, it’s one or the other. I won’t know exactly what’s what until I get all the photos restored and go through and that will take months.
It’s possible that the photos with backup dates of Oct 20-21 on the Windows 7 file-path also have earlier backups on the XP-drive path. I mean, they “should”. But it’s too much to determine exactly at this point. I’m downloading everything that’s there.
The next step will be to rename the files with extensions to distinguish them from the damaged ones; -olddrive, -newdrive. And then throw them together in each folder. There are about 125 folders and the complete multi-step process will have to done on each of them. I’m glad I have some time to decide on the process so I don’t rush in on the wrong track.
Probably, first, I’ll go through everything in Photo Mechanic copying and pasting the IPTC from the damaged photos (most recent version) to the backup files. This is insane to even think about. But, the thought of re-doing two year’s work on the IPTC is even more insane. Well, it’s a close call.
Them sort each folder by Name/Size columns in Windows Explorer so I can see which ones to delete. Then delete 12,000-24,000 photos, depending how many backups there are.
Rename 12,000 photos.
Then use ExifTool to remove the AFCP again. This is multi-fielded and pretty much one photo at a time through every single one.
Then run each folder through GeoSetter to clean out the bad tags.
Then run 45 addresses on all photos through Photo Mechanic to update the GPS. Again.
Meanwhile, make a list of anything without an undamaged backup so I can look for those on the old backup discs being sent to me. And start the whole process over again.
That’s briefly what comes next.
I’m still playing a guessing game here but maybe if I only merge the backups from the XP file-path that pre-date WLPG that’s all that’s required. It seems everything on the W7 backup would be corrupted by WLPG. So that cuts the process down a bit. It’s only 45% of the DESCENDANTS folder and 80% of the ANCESTORS.
I’ve now seen photos from October 16th on the W7 file-path that are not damaged so I can’t assume everything I need back is on the XP-path.
I made bulk GPS additions (but I don’t remember to which photos) before I realized that WLPG had already damaged the GPS, EXIF and compression on those, so I can’t sort by GPS/non-GPS either.
I don’t see any cheap short-cuts here. I’m still going to have to go through every photo. And more than one time.
Carbonite now says it’s finished with the restore, about 12 hours short of what I figured.
The XP-path says it has 19.2 GB. I got back only 17.2.
The W7-path says it has 13.1 GB. I got back only 12.6.
I have to go out to an appointment so will start dealing with Carbonite when I get home.
Nothing to be done with Carbonite. I wouldn’t know where to start. When I went back in to double-check the numbers they’d already changed to where the W7-drive was totally in sync on a file count with what I had restored. I don’t think that makes sense but never mind.
Moving on …
Tiptoeing into the massive restoration ahead – (I think for at least the next 8 months it could be called the MRA) – with 2010. My previous study told me it had no injury. Wrong again. But, it shouldn’t have because they all have GPS.
The originals obviously got hit by WLPG because there’s IPTCext. It’s likely they were compressed but there’s no good backup to show me. The backups have no IPTC or XMP because they were last backed up before I added it? Maybe 2010 is not a typical example.
Starting with photo 0146, the IPTC and XMP is suddenly present on the backups but no GPS. The originals have GPS and the byte order changes to Big-endian. File-sizes the same within 10K.
OMG, this is hard work.
Eventually my brain will compartmentalize to where I’m able to view 4 or 5 or 6 screens simultaneously without stress. When that happens I’ll send in my application to be an astronaut.
Many thanks Geoff for your hard work !
Following the message I’ve posted on WL Help center :
As mentionned by other users (http://windowslivehelp.com/thread.aspx?threadid=95b5150f-a69b-4854-a67d-99afdd782817), I was very surprised to see some geotags appearing in my pictures without any action from me !!
After 2 days investigation, I have concluded that WLPG 2011 (automatically arrived with the windows update KB2434419) did that without any information to the user !!
Could any Microsoft’s staff confirmed that this action is automatic or not ?
How is it possible to disable this function ? If it is impossible, I’ll uninstall Windows Live products from my PC !!
I would like to say that this way to implement new functiuns in a software is not professionnal at all (I’m an IT manager…) !
Many thanks for your answer !
René (a disappointed french user)
Welcome René. The bad news is that you can’t stop WLPG 2011 from writing false GPS information into your image files; it will do this automatically. The good news is that Microsoft are supposedly working on a solution to this. When it will appear, and what it will be, I don’t know.
In the meantime, if you don’t want this to happen, you should:
(a) Uninstall WLPG 2011 from your computer. This means that you will have to uninstall all of the Windows Live Essentials 2011 software from your PC.
(b) Install Windows Live Essentials for XP on your computer. This will give you the previous versions of all the Windows Live Essentials programs. See:
http://explore.live.com/windows-live-essentials-xp
(c) Run Windows Update, and then use it to “Hide” the “Important Update” that will try and update this version of Windows Live Essentials to the 2011 version again. That way, you will continue to run the older version of WLPG that did not have these problems.
Hi,
Many thanks for your answer !
You said : “The good news is that Microsoft are supposely working on a solution to this. When it will appear, and what it will be, I don’t know.”
We’ll wait but does Microsoft inform of the correction…I’m not sure :-((
Geoff – Your theory about ‘just’ going back to backups prior to WLPG was a good one. Unfortunately, that’s not how it works.
In an ANCESTORS sub-folder of about 900 photos I found only 7 unaltered on the XP backup-path. After the files were altered by WLPG, they were brought forward to the Windows 7 backup-path to be Pending Backup and removed from the XP backup-path since the previous most recent backup dates were more than 30 days old. They do not give the option of “Restore previous versions.”
These photos might have had backups remaining on the XP-path but I was without my computer for two months so the time factor was compromised.
And so it goes throughout. Only photos that were unchanged offer previous versions and, of course, I don’t need those. At last broad calculation I had 9,000 photos damaged by WLPG, and I got back only 2,300 from Carbonite. And I suspect those are the ones that were unchanged and consequently not added to the backup queue.
Welcome to our new visitor, René. Deep regrets for the circumstances under which we meet.
Using WinMerge to view side-by-side text outputs of metadata of an undamaged photo and an XP-path backup, I’m able to see that even photos that were not ‘damaged’ by WLPG were still altered in some ways:
-Modification Date, Original Date/Time, Date/Time Digitized, Date Created
-Thumbnail Offset
-Current IPTC Digest
I think that’s pretty funny. WLPG even claims the right to the Date/Time Original.
Right now I’m starting on the process of renaming the XP-path backups one folder at a time and adding the photos back in with the damaged or otherwise altered ones. So then I can move on to copying and pasting the newer version of the IPTC into the backup photos.
Geoff – Do you know what it means when there are no maker notes? I’ve found an entire folder modified by WLPG as well as August 9th backups with no maker notes.
For once, this looks to be not WLPG’s fault but I’d like to know whose it is.
… and I quote:
Photoshop is the biggest culprit when it comes to discarding maker notes.
– Phil
And also, I wouldn’t expect any scanned photos to have maker notes either…
The ones I was referring to were all from my digital camera. 2004-2007. They say “Adobe” on them, so it must have been from the old days of trying to use Adobe Elements Organizer. Yuck.
I got 2,434 photos restored to their rightful home today. Although I started on the easiest ones so I expect to bog down soon. I haven’t hit the AFCP yet. And that is so much fun.
I just found out I can restore the older backups, at least some of them, from Carbonite but it involves opening each folder and clicking through each box for each photo and choosing the backup that precedes the WLPG damage. Click, click, click, click, click. The 48 hours spent downloading the main folders were mostly a waste of time because Carbonite chooses the most recent backup by default, assuming that’s the smart thing to do.
Too many programmers lately thinking they know what the smart-thing-to-do is. They should consult us peons down in the trenches before they even think of getting up in the morning.
Coopster, if this crisis is ever over, I’ll miss ya.
Ha, well I’ll be here on the interwebs until I drop off the perch – so you’re welcome to drop by and wave if you want…
Ah, yes, Photoshop Elements Organizer – we go way back https://gcoupe.wordpress.com/2005/03/05/managing-photo-libraries-part-2/
I still hate it with a passion.
The Organizer crashed all the time when I tried to use it. Totally disorganized with the IPTC. Backups I never could fathom. I hated it and then I hated it more and then I gave up trying.
I also don’t like Picasa because most of my photos are tiff’s and they pretend they do IPTC on tiff’s but they don’t really. And a whole bunch of other reasons I’ve forgotten because it’s 2 o’clock in the morning. http://www.jgen.ws/jlog/tag/picasa/
‘Til the next time…
Wow folks. I have to admit, I’m not very knowledgeable in this area, but I can see this is a real mess. I’ve been using Picasa to manage my photos but have used W7/WLPG to import from my memory card. I can only assume my files are just as corrupt as everyone elses.
I guess my real question is – once fixed, what products do we use to import, edit and manage photos?
Does anyone have a SAFE workflow that they can recommend?
Norm, my main tool for my digital workflow is IDimager. Not free, but a great tool with good support and an enthusiastic user community in the forums.
Norm, I tried IDimager and found it way over my head or just confusing. I use Photo Mechanic, also not free but really good. They both have free trials.
Also, I’ve recently come to GeoSetter which IS free and excellent although I’m not sure if it’s what you’d call ‘workflow’ software. I’m too new to it.
XnView is also a possibility and it’s free.
Thanks for this information on WLPG 2011.
Are there any known similar issues for Picasa (v3.8)?
Martin, see here for what I found when I tried Picasa v3.8:
Thanks for your reply! I think Picasa fits my needs as I want to use it just to view and search images, all cataloging is done with IDimager.
I’d look for out for Picasa. I had problems with it repeatedly taking out the IPTC State/Province field and replacing it with a duplicate of the sub-Location field. On thousands of photos and I was only using it for ‘viewing’. Try XnView. Also fine for viewing and searching and much safer.
Geoff – How’s your restoration going? I’m aiming for 100 photos a day, although I think that will become too ambitious soon.
JL – I think I’ve got as many as I can back now. There are some – around a window during September 2010 – that I don’t have backups for (the backup images have also got the damage/been compressed), and some of the earlier ones have already been damaged by an earlier version of WLPG that was writing Microsoft-specific XMP. However, the damage is makernote corruption on JPEGs, so I’m just going to have to write that off as a lesson learned.
Your life is moving on it sounds like. How painful, though. I would say a lesson learned in how easy it is to get my stuff carelessly tossed about by something/anything online; about akin to a hurricane. And not something one would expect from an operating system.
Going backwards from 2009 I’m as far as 1990, and I think I’m still in the easy years. I’ve started doing some batch work in GeoSetter, like co-ordinates, to try to get a jump on things. That’s a great piece of software!
Theoretically, I’m committed to two hours a day, (times about 240 days I figure) so I can have a present-day life as well, although I find myself spending every spare minute while greatly stretching the meaning of ‘spare’, so it’s more like 10 hours or 12.
I go in circles in my process between Windows Explorer, ExifToolGUI, Photo Mechanic and GeoSetter. Everything I’ve seen so far tells me that every single photo has damage of some kind. Even the ones that were not affected by the GPS auto-write had other metadata altered. And, of course, the compression is just incredibly ‘heinous’ (to quote you) … 3MB down to 650K or 7MB reduced to 3.2MB. It’s just sickening. They’re my photos; I want them back the way they were. You know, like Florida orange juice – “unfooled around with”.
If I can get back to 1970 without finding any missing ones, the MIA damage may be less severe. The backup discs being sent back to me are quite old so my hope is tempered.
Pingback: What Lies Beneath | Geoff Coupe's Blog
Pingback: Windows Live Photo Gallery 2011 – A Status Report | Geoff Coupe's Blog
MakerNotes have always been overwritten by Picasa and WLPG. Picasa seems a little better about wiping them out – but there are certain file/save actions that will completely wipe out Makernotes, at least from my Canon. I’ve complained about the issue in both forums, but the answers from Picasa and WLPG diehards always lay the blame at the camera manufacturers’ doors – “MakerNotes? You can’t expect us to preserve MakerNotes….every camera outfit does them differently…that would be insane!” Frack.
Hi Jason – I’ve at least had an acknowledgement from the Lead Project Manager of the WLPG team that the Makernotes issue is understood, so who knows, perhaps it will at last be addressed in a future release. Here’s hoping.
Geoff,
Thank you for your reply, and for all you do on behalf of our metadata. I suppose my real concern (after reading through this thread) is what (if anything) will be the end result of all this massaging and re-massaging of our digital assets? WLPG does such-and-such to our jpegs, then after a year or so, we use Picasa, and it overwrites our metadata again. Then, next year, maybe we try Photoshop, and it roots around in our XMP and EXIF. Theoretically, there should be no disturbance to our real treasure – the image contained in the jpeg….but do we know that? for certain?
I’ve already experienced difficulties with XP being unable to open photos that were acted on by both Picasa and WLPG. Can we be sure that long term, we are not all going to experience digital rot of some sort or another? Personally, I’ve halted my efforts to tag my photos until I’m assured that nothing destructive can come of adding “Grandma – Xmas 2007” to a caption line.
I have had similar thoughts having put my photos through the wringer with many different IPTC-annotating programs. I’m still recovering my photos from the WLPG disaster. Makernotes destroyed from brief passes through Adobe Elements Organizer, well, it’s too late for those.
At this point, I’m down to ExifToolGUI and GeoSetter (that’s run by ExifTool) and Photo Mechanic which I believe is based on high standards and safe. For a browser I use XnView but I don’t use it for writing metadata. Everything else is off the board.
I know people ‘love’ Windows Live Photo Gallery and Picasa because they’re free, and Adobe because they ‘seem’ to know what they’re doing. Well, ExifTool and GeoSetter are both free too and I’m way more confident from my recent and painful education that Phil Harvey knows what he’s doing
Pingback: 2010 in review | Geoff Coupe's Blog
Please help me, I am so confused and not all that computer literate..suddenly, literally this morning after a year on Windows 7, everytime I try to rotate a picture the computer just deletes it. It is not in the “trash”…it is nowhere….it just disappears. And there is no option to “undo”. And, yes, I have been using Windows Live Photo Gallery. Thanks for any help or explanation.
Stephanie, I think what’s happening is that the edited (or rotated) picture is having the “hidden” attribute accidently turned on. Are you using McAfee as your security program? I think that is what is probably going on.
Please take a look at this post in the Photo Gallery Help forum:
http://windowslivehelp.com/thread.aspx?threadid=be95d669-75b7-42ec-b93a-5513c1eb9158
As a matter of fact that is exactly what I am using. I had just renewed the subscription recently and this problem presented on both mine and my grandaughter’s laptop. Thank you for this help…it worked like a charm. The question now is, will McAfee fix this or is everyone who uses it going to be pulling their hair out?
According to the McAfee document referenced in the Photo Gallery Help forum, McAfee are working on a fix, so the problem should be temporary… There is a workaround given by McAfee in that document, but it involves diving into the innards of Windows – probably not a good idea unless you know what you are doing.
Thanks Geoff for your solution to Stephanie’s problem. I experienced the same problem as Stephanie, i.e. edited photo in WLPG 2011 results in the photo being deleted.
With your suggestions, I was able to locate my edited photo as a hidden file and recover the photos. But still cannot edit photos without getting the “hidden” attribute. Waiting for McAfee’s fix.
Thanks again, Ivan
The majority of photos in Windows live photo gallery has been some erased again–2nd time in a month–the various file folders in which the photos were stored are all still accessible but when opened there are no pitcures in the folders anymore.Fortunately I have backups.
Can you tell me why this happens and how to avoid it in future.
Thanks
Cam
Cam, I’m not sure what is going on in your particular case – I’ve not seen this problem. Sorry.
Can someone tell me what the latest is with Picasa & WLPG? I was just wondering if the problems have been addressed and corrected yet.
Norm, I can only speak for WLPG, I haven’t looked at Picasa since version 3.8 in September 2010, and while Picasa has had a couple of minor updates, it’s still at 3.8, and the geotag issue I reported has not been addressed according to Picasa’s release notes.
Re WLPG, the geotag issue was addressed a year ago, but the Makernotes corruption is still there. Until that is cleared up, I will only ever use WLPG as a browser, I will not use it to edit metadata.
Norm, I see that Picasa version 3.9 has just been released. I’m happy to say that this has corrected the issue of not displaying Geotags on the map correctly. It also doesn’t seem to corrupt Makernotes when it writes metadata back into image files. So I think Picasa has now pulled ahead of WLPG. I’ll be blogging about this soon, once I’ve had a chance to poke about a bit more.
Pingback: A False Sense of Security | Geoff Coupe's Blog
Geoff, I use Win 7 and just since Monday 5/21/12 I now cannot open jpg pics sent in my email. I get Windows Live Photo Gallery encountered an error loading WLXPhotoviewer.dll and can’t start. Error code 0x8007007e.
I have read and reread different ways to handle this. Please help before I lose patience. Thanks Jud
Jud, I’m sorry, but I don’t have a quick and easy answer for you. Either you have installed some other software that has broken WLPG, or Windows Update has installed something that has had the same result.
All I can suggest is that you post your issue on the Microsoft Answers forum:
http://answers.microsoft.com/en-us/windowslive/forum/gallery
I hope that someone there can help.
Geoff, FYI – I went to
For Windows 7 or for Windows Vista1.
Start
Control Panel.
2.Under Programs, click Uninstall a program.
3.In the Uninstall or change program list, click Windows Live Essentials. and then click Uninstall/Change.
INSTEAD of removing and reinstalling, I choose Repair Windows Live
AND IT WORKED. Just thought you’d like to know to pass on. Jud
Great, glad that it worked.
Boy, I was starting to think I was loosing my mind. For some reason just recently it seemed as if my photos were’nt loading correctly. I thought it was me taking underexposed shots. My Lcd screen on camera they would look ok but after uploading them with Windows Live they looked Horrible Absolutley Horrible. I had also noticed older pics in folders seemed to have changed . I was like what is happenning did I over-correct at one time. Did I take on too much editing on one of my late nite editing Binges. I guess now I Too have a bunch of DEstroyed Files> Unlike Geoff I Most likely will never be able to fix. Im pretty much at the mercy of my computer I do pretty much what it tells me . As for as fixing problems I’m a computer Dummy. So whats a Dummy like me suppose to Do??? I have !000’s of Images and dont know where to begin to change these files. I do use another Photo editing software, Photoscape and my Photos seem still to be fine in there. Its just a harder program to share with To Say like Flickr. So is there any recommendations for any other Photo Orginiser that is non destructive
Tom, it sounds to me that there’s something else going on with your photos – possibly caused by something else in your Windows and PC configuration. I’ve never had problems with WLPG’s handling of images, just with the metadata. You might want to ask on the Microsoft Answers forum here: http://answers.microsoft.com/en-us/windowslive/forum/gallery
Pingback: Photo Editor Apps | Geoff Coupe's Blog