UPDATE: It seems that now the notes from personal documents also show up in the Your Highlights page. So there is no more need for all the tricks below. I’ll keep this post here just for historical reasons anyway.
UPDATE 2 (01/04/2014): I’ve been making more tests and it looks like only my Paperwhite started uploading the notes from Personal Documents, after the latest firmware update. However, that didn’t happen for any personal document, just to one book specifically. I’m still investigating what made that book special in Amazon’s view… But for now, it seems like the post below is, after all, still relevant.
Now that I am using my Kindle to read academically, I’ve realized that I will need to make a very good use of the Notes and Highlights feature. I am still working out a system on how to make my notes more useful in my reading (which will become a post on its own), but before I put such a huge effort in note taking, I want to make sure that the notes will be easily retrieved and safely stored outside of the Kindle.
For books that have been bought from Amazon, getting the notes out of the Kindle is not a problem at all. Amazon has a page that shows all your notes from all your Kindle books, taken in any of the devices that you have connected to your account (you have to be logged in to Amazon to be able to see it). You can just copy and paste from that page and that’s it. Or even better, you can clip it to Evernote.
The problem is when you are annotating documents that have not been bought from the Amazon store – what Amazon calls “Personal Documents”. For those documents, notes and highlights can be synchronized across devices, which is very handy, but unfortunately they are not uploaded to the magic page with all your highlights.
“My clippings.txt”, almost a solution, but not quite
The “My clippings.txt” file is great and there are amazing tools that extract the notes from this text file and allow you to export them to many formats (this one is my favorite).
For those who don’t know this, each time you take notes in your Kindle device (not in the desktop or mobile apps), the device writes the note to that text file. It is a one-way operation, though; more like a back-up. If you delete a note from the book, it will not be deleted from the “My clippings.txt” file. If you edit the file and put it back in the Kindle, it will not carry your edits back to the book annotations.
But there are two problems, at least for me, with relying on this text file for my note-liberation needs. One is that this file is device-dependent, so the notes that I take while reading a Personal Document in another device will not show up on my Paperwhite “My clippings.txt”. The second problem is that the Kindle reading apps (for desktop and mobile) do not produce such a file (otherwise it would just be a matter of merging different files and exporting them).
In practice, this means that any notes that I take while reading my Personal Document from my Android tablet will appear on my Paperwhite while I’m reading the book, but they will not be exported in text format to outside of the devices.
.mbp and .empb, the annotations file
As I said above, the “My clippings.txt” file is merely a backup of your notes. The notes themselves are stored in a “side” file, named exactly like the e-book file itself but with a different extension: mbp, which is a file that holds Mobipocket annotations (Mobipocket was bought by Amazon and now it has basically disappeared).
These files are not text files, and although I haven’t found any software that can edit them directly (except from the Kindle for PC itself, which opens them without troubles together with the original book, but doesn’t allow for bulk exporting of all the notes), there is a software that some nice soul posted online that is able to extract the mbp notes to text.
(One tip for those using the MBP note extracter above: edit the file mbp.parameters.txt and remove the “#” from the beginning of the line that shows BKMK_PAGE_FACTOR=150 – as this will write the location of the notes to the file.)
But there is one problem here. Up until late 2012, these files were just as Mobipocket had created them, and using tools such as the one posted above or software like Buzzworthy, you could simply extract the notes and live happily ever after. But since then, Amazon started encrypting the notes (file extension .mbp1 or .embp or .mbs) and those software no longer work. Not fun.
The solution to this issue is posted on Buzzworthy’s website: you have to downgrade the Kindle app in your mobile/tablet in order to have your notes in the non-encrypted format to be able to read the .mbp file.
So, to sum up all that is written above, this is how I am managing my notes in my Personal Documents in all my Kindle apps/devices (one Kindle Paperwhite, one Android Smartphone, one Android tablet, and two Kindle for PC installations):
1) I make sure my Personal Document is synchronized across my devices by using “Send to Kindle”. Although it is possible to add personal documents directly to my device(s), I use Send to Kindle, so that the document and the notes are synchronized across all the devices (Cloud and Desktop readers, unfortunately, do not sync personal documents). If you are sending the document via e-mail, make sure to add the word “CONVERT” to the subject of the message.
2) To extract the notes I take directly in the Paperwhite, I export the “My Clippings.txt” file to Clippings Converter and from there to Evernote. And because this is the simplest method, I am trying my best to do all my annotations there. But if I need to make notes on the phone or the tablet, it’s not a big problem. Read on.
3) I keep the Kindle App in my Smartphone with an old version of the software (namely 126.96.36.199). This way, I can get the unencrypted .mbp file out of the phone if I need, and as I sent the document via Send my Kindle, this file will have all my notes, even from the other devices with newer Kindle apps. I just have to make sure that the notes have been synced before transferring the file from the phone to the computer (via USB, Dropbox, Bluetooth, whatever works). The .mbp files will be inside the “kindle” folder, together with the books themselves.
4) If I need to extract those notes from the .mbp file, I use the handy software mobipocket notes file extractor to get them safely out of the Kindle ecosystem. Setting the “BKMK_PAGE_FACTOR=150” preference makes the software print out the locations of the notes as well.
5) If I need to see the notes in my Desktop, I copy the personal document and the unencrypted .mbp file manually from the phone to the Kindle for PC, by simply dropping those files in the “My Kindle Content” folder in the computer. Then I can open the book and all my markings in my Desktop are there (and this is very useful while cleaning up my notes on Evernote Desktop, using a proper keyboard, and with the book and notes open right in front of me).
I tried adding notes in the desktop and sending the .mbp file back to the phone to see if the new notes would sync and show up in the other devices, but unfortunately this didn’t seem to work. I could see the note in the phone, but it was not replicated to the other devices. It only became visible in the Paperwhite once I edited it on the phone directly. So I consider the installation of Kindle for PC as read-only – I never edit the notes there so that they won’t get lost.
So that’s it. It’s a lot of pain to go through just to get your Personal Documents’ notes out of the Kindle. It bothers me a lot to think that this is Amazon’s way of pushing us towards buying our e-books from them so that we can enjoy the much better synchronization possibilities that Amazon books have in that matter. I have always preferred to buy my books from Kobo and convert them to .mobi using Calibre to send to the Kindle, but I don’t know if I want to go through all this trouble all the time… I can’t wait for Open Annotations in epubs to become a reality and to have some impact in Amazon’s way of dealing with this. After all, those are my notes and I want to make sure they are easily readable!
The next post will be more about the quality of the notes and how I am planning to use them in aiding me in reading critically using an e-book reader.