UPDATE: It seems that now the notes from personal documents also show up in the Your Highlights page. So there is no more need for all the tricks below. I’ll keep this post here just for historical reasons anyway.
UPDATE 2 (01/04/2014): I’ve been making more tests and it looks like only my Paperwhite started uploading the notes from Personal Documents, after the latest firmware update. However, that didn’t happen for any personal document, just to one book specifically. I’m still investigating what made that book special in Amazon’s view… But for now, it seems like the post below is, after all, still relevant.
Now that I am using my Kindle to read academically, I’ve realized that I will need to make a very good use of the Notes and Highlights feature. I am still working out a system on how to make my notes more useful in my reading (which will become a post on its own), but before I put such a huge effort in note taking, I want to make sure that the notes will be easily retrieved and safely stored outside of the Kindle.
For books that have been bought from Amazon, getting the notes out of the Kindle is not a problem at all. Amazon has a page that shows all your notes from all your Kindle books, taken in any of the devices that you have connected to your account (you have to be logged in to Amazon to be able to see it). You can just copy and paste from that page and that’s it. Or even better, you can clip it to Evernote.
The problem is when you are annotating documents that have not been bought from the Amazon store – what Amazon calls “Personal Documents”. For those documents, notes and highlights can be synchronized across devices, which is very handy, but unfortunately they are not uploaded to the magic page with all your highlights.
“My clippings.txt”, almost a solution, but not quite
The “My clippings.txt” file is great and there are amazing tools that extract the notes from this text file and allow you to export them to many formats (this one is my favorite).
For those who don’t know this, each time you take notes in your Kindle device (not in the desktop or mobile apps), the device writes the note to that text file. It is a one-way operation, though; more like a back-up. If you delete a note from the book, it will not be deleted from the “My clippings.txt” file. If you edit the file and put it back in the Kindle, it will not carry your edits back to the book annotations.
But there are two problems, at least for me, with relying on this text file for my note-liberation needs. One is that this file is device-dependent, so the notes that I take while reading a Personal Document in another device will not show up on my Paperwhite “My clippings.txt”. The second problem is that the Kindle reading apps (for desktop and mobile) do not produce such a file (otherwise it would just be a matter of merging different files and exporting them).
In practice, this means that any notes that I take while reading my Personal Document from my Android tablet will appear on my Paperwhite while I’m reading the book, but they will not be exported in text format to outside of the devices.
.mbp and .empb, the annotations file
As I said above, the “My clippings.txt” file is merely a backup of your notes. The notes themselves are stored in a “side” file, named exactly like the e-book file itself but with a different extension: mbp, which is a file that holds Mobipocket annotations (Mobipocket was bought by Amazon and now it has basically disappeared).
These files are not text files, and although I haven’t found any software that can edit them directly (except from the Kindle for PC itself, which opens them without troubles together with the original book, but doesn’t allow for bulk exporting of all the notes), there is a software that some nice soul posted online that is able to extract the mbp notes to text.
(One tip for those using the MBP note extracter above: edit the file mbp.parameters.txt and remove the “#” from the beginning of the line that shows BKMK_PAGE_FACTOR=150 – as this will write the location of the notes to the file.)
But there is one problem here. Up until late 2012, these files were just as Mobipocket had created them, and using tools such as the one posted above or software like Buzzworthy, you could simply extract the notes and live happily ever after. But since then, Amazon started encrypting the notes (file extension .mbp1 or .embp or .mbs) and those software no longer work. Not fun.
The solution to this issue is posted on Buzzworthy’s website: you have to downgrade the Kindle app in your mobile/tablet in order to have your notes in the non-encrypted format to be able to read the .mbp file.
Wrap up
So, to sum up all that is written above, this is how I am managing my notes in my Personal Documents in all my Kindle apps/devices (one Kindle Paperwhite, one Android Smartphone, one Android tablet, and two Kindle for PC installations):
1) I make sure my Personal Document is synchronized across my devices by using “Send to Kindle”. Although it is possible to add personal documents directly to my device(s), I use Send to Kindle, so that the document and the notes are synchronized across all the devices (Cloud and Desktop readers, unfortunately, do not sync personal documents). If you are sending the document via e-mail, make sure to add the word “CONVERT” to the subject of the message.
2) To extract the notes I take directly in the Paperwhite, I export the “My Clippings.txt” file to Clippings Converter and from there to Evernote. And because this is the simplest method, I am trying my best to do all my annotations there. But if I need to make notes on the phone or the tablet, it’s not a big problem. Read on.
3) I keep the Kindle App in my Smartphone with an old version of the software (namely 3.6.0.87). This way, I can get the unencrypted .mbp file out of the phone if I need, and as I sent the document via Send my Kindle, this file will have all my notes, even from the other devices with newer Kindle apps. I just have to make sure that the notes have been synced before transferring the file from the phone to the computer (via USB, Dropbox, Bluetooth, whatever works). The .mbp files will be inside the “kindle” folder, together with the books themselves.
4) If I need to extract those notes from the .mbp file, I use the handy software mobipocket notes file extractor to get them safely out of the Kindle ecosystem. Setting the “BKMK_PAGE_FACTOR=150” preference makes the software print out the locations of the notes as well.
5) If I need to see the notes in my Desktop, I copy the personal document and the unencrypted .mbp file manually from the phone to the Kindle for PC, by simply dropping those files in the “My Kindle Content” folder in the computer. Then I can open the book and all my markings in my Desktop are there (and this is very useful while cleaning up my notes on Evernote Desktop, using a proper keyboard, and with the book and notes open right in front of me).
I tried adding notes in the desktop and sending the .mbp file back to the phone to see if the new notes would sync and show up in the other devices, but unfortunately this didn’t seem to work. I could see the note in the phone, but it was not replicated to the other devices. It only became visible in the Paperwhite once I edited it on the phone directly. So I consider the installation of Kindle for PC as read-only – I never edit the notes there so that they won’t get lost.
—
So that’s it. It’s a lot of pain to go through just to get your Personal Documents’ notes out of the Kindle. It bothers me a lot to think that this is Amazon’s way of pushing us towards buying our e-books from them so that we can enjoy the much better synchronization possibilities that Amazon books have in that matter. I have always preferred to buy my books from Kobo and convert them to .mobi using Calibre to send to the Kindle, but I don’t know if I want to go through all this trouble all the time… I can’t wait for Open Annotations in epubs to become a reality and to have some impact in Amazon’s way of dealing with this. After all, those are my notes and I want to make sure they are easily readable!
The next post will be more about the quality of the notes and how I am planning to use them in aiding me in reading critically using an e-book reader.
Thanks for the post. I can’t believe Amazon is doing this to us. I stored tons of notes in my Kinde for PC and then I had a problem with my app and had to reinstall it as recommended by an Amazon representative. I did not realized until months later that I had lost all my notes that I had collected for years. This is sick!
There is a really simple yet robust tool for extracting highlights, notes and images from your pdf-files available at: http://www.sumnotes.net . Not only it supports various advanced features like exporting to DOC and TXT formats, but it also allows you to send extracted highlights into your Evernote account or email. All desktop browsers and operating systems are supported. We are in cloud, so no installation is needed. Try it out.
Thanks! I’ll definitely be making use of clippingsconverter.com thanks to your advise. It’s mind-boggling that there isn’t an easier way to keep track of these notes. Amazon is seriously defying their own image as a resource for academia (let alone others who wish to conveniently read through and mark up documents) by making it so difficult to make proper use of nonAmazon docs on the Kindle. Personally, I try to use Kindle to read and annotate manuscripts but when it comes time to make those edits in the original document, going over my in-Kindle notes is a real pain in the butt.
advice*
Thank you for this post. Yours is the only place I’ve seen the specific link to https://kindle.amazon.com/your_highlights ; if you just go to Amazon.com and click manage your Kindle, it can take forever to navigate down to this highlights page.
However, I don’t agree that “It seems that now the notes from personal documents also show up in the Your Highlights page. So there is no more need for all the tricks below. I’ll keep this post here just for historical reasons anyway.” I’ve got several E-books from Project Gutenberg free library, and NONE of them back up my notes to Amazon. Such a pain! And I makes me afraid that if I were to lose my specific Kindle Reader, I’d lose all those thoughts and observations in my books!
Holden, any chance Sumnotes might support .mobi files in the near future?
Thank you for the great post.
I was looking for a way to export my notes from Kindle Fire that does not backup notes into “My Clippings.txt”.
So the only way to extract notes is to decode .mbp with mobipocket notes file extractor.
Yes, I have also found that highlights from personal documents don’t show up on the highlights page yet.
I’ve found a way to consistently get personal document notes off of my kindle touch by using “share to twitter” from my kindle. This forces kindle to upload the annotations/highlights to your “recent activities” page on kindle.amazon.com. I like doing it this way over messing with the files on my kindle since it’s all hands-off and nicely formatted. I’ve documented the full process here: http://hub-mag.co.uk/how-to-automatically-export-kindle-notes-and-highlights-to-trello/
I’m with the other more recent commentators. I do NOT see a way to find my highlights and notes from personal documents on the Amazon “Your Highlights” page.
Thanks for the “old way” as it still serves as the new way for me…
this is really helpful. thanks a lot!!
Hello, thank you for this post. This is really helpful. Thank you! I have the same question with the other recent commentators- I don’t see my highlights on personal documents under Your Highlights either ( as of Feb 2015)? Could it be that mine is a Kindle 2014 (the cheapest one) but yours is a KPW?
@Liv:
I don’t think it has to do with mine being Kindle Paperwhite and yours being an older version. I think that for some reason Amazon identified my Personal Document as being an Amazon book and then incorporated it in the Amazon “ecosystem”, treating it as any other Amazon book. This happened just with one document. All the other Personal Documents I have don’t show up in Your highlights either :(
I should open that book and see what does it have that is special and makes Amazon see it that way… I’ll post back with any discoveries I make!
I read your post over a year ago and devised a method of my own to manually sync the highlights in a personal doc on a kindle with the kindle app on the desktop. I planned to document the whole process, but was never able to do that. Here is a rough outline of what I did.
Situation: Highlights were done on a kindle device.
Objective: View those highlights on the kindle software on the computer.
Workflow:
> Get the encrypted mpb file from the kindle’s storage over usb. Open it with Notepad++.
Among the “garbage” there are a set of numbers inside brackets in a repeated pattern. Those numbers are locations of the start and ends of the highlighted portions.
Via a (set of) regex find and replaces, extract those numbered pattern.
> Highlight something on an official kindle book (like the free Alice in Wonderland) on your kindle device.
Let it sync to the desktop app. In the storage folder, an uncrypted mpb file will get created. Inside that file you can see the positions of the highlight you just made on the kindle.
This new mpb file follows a particular syntax.
> Now arrange the set of numbers you have extracted from the encrypted mpb file and arrange them as per the syntax of the unencrypted mpb file. Keep the headers and footers of the unencrypted mpb files as they were.
> rename the the newly created mpb file to match with the personal doc’s file name on your computer. Open the personal doc in the desktop app. All the highlights will be visible now.
Similarly, the mpb can be transferred to a kindle app on android (that can still read unencrypted mpbs). I havent tried this, but it should work.
Comments: There is no easy way to generate a highlighted pdf out of a personal doc. This is the closest I got in viewing the highlighted doc on the desktop with the highlights. I still plan to make the procedure semi automatic via scripts, or atleast explain it somewhere in detail, but never got the time to do it. Till that day comes, I will let this stick in your post, hoping it will help/guide someone.
Thanks.