We've got 613 practice logs on this site, and I've often wondered what nuggets of gold are buried deep in that archive. With inspiration from Pixelcloud and Chris M., I decided to scrape all the logs and have Claude Sonnet 3.5 dynamically label them. You can check out the database here.

For each log, Claude notes any themes (listed in order of prominence), techniques used, teachers mentioned, map territory discussed, claims to attainments, and apparent attainments. He assesses how helpful of a read the log is and the level of disagreement between posters in the log. He also generates a pitch for each log, including the type of practitioner who would likely benefit from reading it, and picks out a couple memorable quotes.

I invite you to look through the database and post any goodies you find here. You can use the filter and sort features on Google Sheets to narrow down your results (e.g. filtering to logs that discuss third path, or logs that discuss addiction, or logs that were super controversial, or sorting logs by perceived helpfulness).

To filter or sort, go to Data -> Create Filter View, and then click the three-lines icon at the top of the column you're interested in. You can then sort it alphabetically. If you want to filter by a specific search term, click Filter By Condition -> Text Contains. By clicking "Save view," you can save your filter/sort combo and return to it later. 

The model is intelligent, and I gave it a crash-course in our conceptual frameworks, but still, hold its outputs lightly. I haven't edited the outputs, and they don't reflect my personal views. I told it to be extra harsh in assessing usefulness. That's not the most polite thing to do, so I apologize, but I think it's worth it.

I can't program; Claude wrote all the code necessary to do this, and I merely pasted it into VSCode and fed him back the error codes until it worked. Generating this database cost me $40 in API credits. Here's the code I used, in case you want to remix or continue this project, as well as readable .txt versions of all the logs.

Amazing! 
Thanks John L, and Claude ;)

I wonder, will this database be updated periodically?

​​​​​​​Happy holidays

Ha ha I just looked through my papa che log 2 and I enjoyed reading it especially because some old friends were replying to it Thank you! 

Yup, I plan on refreshing it periodically, maybe every 3 to 6 months. I want to eventually include the Kenneth Folk Dharma logs, but I'm out of time for now. 

Nice work !!

​​​​​​​I could hypothetically make a UI for this so people can sort by topic, which I think would be a cool thing to have, I had some additional ideas along those lines recently. Cool filters and fancy search criteria.

It'd be easy enough to set it up to automate a Database sync with new logs if it was on a server.

I'm a busy boy but I could defintiely do some work on this if it was something people wanted... at some point.

Oh my friend is developing an AI tool I actually think would be perfect for the loggers on here. Basically, imagine a spreadsheet where the rows are voice memos you send it, and the columns are customizable text prompts for chatgpt, to intelligently process the outcome (so you could imagine having a prompt that labels less than obvious emotions, or tone, or any other thing you can think of. It's currently in early development, so he is looking for people to test it and give feedback.

The voice tool called Ramble. It lets you ramble about whatever’s on your mind, and the tool helps organize your thoughts into something more structured.

Here’s the link to their prototype: https://www.intfc.co/notes
www.intfc.co

Amazing John! This is incredible work, bringing clarity and accessibility to DhO's rich archive, an invaluable resource for the community!

This is great!

I've had fun this morning using search (Command-F) to zoom in on various interesting key words and topics.

THANK YOU!

John, check your messages! I just sent you an uncompressed version of my mahasi notes, so you can recompress it

Chris, can we make a category of auxiliary resources created by DhO people? Im thinking stuff like daniel/shargrols post compliation, john's stuff, etc. They are extremely helpful. But right now they are just kinda buried in random threads (no ill will towards the random threads, they are very nice!)

Yes, you certainly can, Geoffrey. Where do you suggest it belongs?

Oh I just made a post on the books/website. It might work better as a category (which im not sure i can make?), but it's probably fine as a post for right now?

It's fine as is—let's see how it works before I move it to another category.

Recompressed! It's in the sheet now; thanks for sending over that log. 

I personally am taking this AI model's evaluation of 'Likely Highest Attainment' very lightly.  My gut tells me that this AI is probably very easy to fool into thinking a poster has attained a certain path when they may have not, and that if one were to project that they are on a higher path, the AI would probably agree with them a lot of the time even if there is no real description of practice and no peer review or confirmation.

For example, in one of my earlier logs from several years ago, which was really just a single post, where I made a claim to 1st path which is 'verified' by this AI model, despite a lack of any practice description, no peer verification or review.  

It's also interesting to note, looking through the excel sheet, there are some log series where the poster makes a claim to a higher path which is 'verified' by the AI in earlier logs, only to drop down to a lower path in later logs.  

All that to say, IMHO, nothing beats real peer review from experienced meditators who have a deep familiarity with the 4 path model, and it is always better to underestimate one's own progress than to overestimate.  

John L --

In the earliest era of DhO the platform we were hosted on was called "WetPaint." There were thousands of comments and scores of posters in that earlier version of DhO. Those posts and posters were ported here to the LifeRay version of DhO when we moved. I think that deeper history would be a nice addition to what you've created. As far as I can tell, all of that content is still here on the LifeRay system. Here's one example of this kind of content:

https://www.dharmaoverground.org/discussion/-/message_boards/category/78028

According to the LifeRay data, there are 22,924 WetPaint posts available across DhO today, which accounts for only a fraction of the total posts that were ported over, representing only those users who had no DhO/LifeRay account that corresponded to a DhO/WetPaint account.

This is just a thought to help add more content to the DhO database.

Hi Chris, 

I appreciate the tip. It looks like most of the WetPaint threads got sorted into a proper subject-matter category, while the rest ended up in the WetPaint Migration 4 category. With that understanding, I think I can just scan the WetPaint Migration 4 category for practice logs, process them, and then the sheet will have all the relevant WetPaint content. 

P.S.: I'm interested, for my own historical understanding: did you really have the first ever internet practice log in the pragmatic dharma scene? It sure seems that way to me. Pretty cool. 

Wait here's an interesting idea. Could we use an AI transcriber on all of dharma seed, and then make a similar table based on all the western dharma talks?

Like what if we just make *the* western meditation AI?  

​​​​​​​I bet I could find us some help! (I met an OpenAI llm guy on a retreat, and one yogi I know has a big google drive link of books)

Also, once we make it, I bet we can get it hosted for free on Google, AWS, etc 

https://www.dharmaoverground.org/discussion/-/message_boards/message/33380245

Yes, as far as I know. However, it wasn't my idea—it was Kenneth Folk's. I wasn't keen on doing a practice log since it would be very personal and "out there" for everyone to see, but he managed to talk me into it.

John, do you mean the "KFD" online logs now on AwakeNetwork? If so, they are private and were originally posted under that assumption behind a login that protected the content from anyone but the members of that message board. I can help you navigate which ones you can use here and which need to remain private, as intended. It would be very bad karma indeed to violate that agreement, so it would be best to obtain permission from the original posters.

Sorry to be a pain, but ethics and morality, right?

Oh that's a surprise! I definitely want to respect people's privacy. So far, I've scraped the threads from the Kenneth Folk Dharma Archive (Original and V2) and, using GPT 4o-mini and some elbow grease, figured out which of those are practice logs. That left me with 278. I've looked at a good amount of them, and they've seemed to be public-facing, with people participating. It's strange that Awake Network would host them if they're private.

If some of these were meant to be private, maybe I can send you a folder of what I've got, and you can take a look? Perhaps a shortcut is to green-light threads with multiple participants. 

Yes, I'll take a look. Did you use an Awakenetwork login to obtain the KFD content?

I'll DM you my email address using the DhO messaging system. It's down right now, but I'll get it working again shortly.

EDIT: DM sent.

We need to do the right things here from a legal standpoint. Just because the content is publicly available doesn't mean it's legal to republish it. Please note that OpenAI is being sued by the New York Times, other publications, and various artists for scraping publicly available content to use for training data purposes which then gets republished almost verbatim.

Ah, I was a little sleep deprived when I wrote my last post; I now understand the situation. Since these posts were only meant to be available to those with a registered account, maybe we can similarly restrict the KF log spreadsheet. As it stands, these logs are accessible without an AN login, so maybe we should relay that to the AN site admin. I'll send you an email now.

Email received. I'll look into what can be done with the content.

After conferring with Chris, I've added 277 practice logs from Kenneth Folk Dharma to the spreadsheet. Enjoy! 

While digging through the archive, I found a nice nugget of practice wisdom from telecaster — "looking at vs. looking for". 

Thanks, John.

*** Note: The KFD logs John has added are publicly posted on the KFD segment over at AwakeNetwork. You can navigate to those logs without having to become a member of AN and without logging in. There are many, many more *private* logs on AN that have not been added to John's spreadsheet.