Home
Softono
t

transitive-bullshit

Professional software vendor delivering innovative solutions on the Softono platform. Specialized in both open-source and proprietary software development.

Total Products
4

Software by transitive-bullshit

nextjs-notion-starter-kit
Open Source

nextjs-notion-starter-kit

<p align="center"> <a href="https://transitivebullsh.it/nextjs-notion-starter-kit"> <img alt="Example article page" src="https://user-images.githubusercontent.com/552829/160132094-12875e09-41ec-450a-80fc-ae8cd488129d.jpg" width="689"> </a> </p> # Next.js Notion Starter Kit <!-- omit from toc --> > The perfect starter kit for building websites with Next.js and Notion. [![Build Status](https://github.com/transitive-bullshit/nextjs-notion-starter-kit/actions/workflows/build.yml/badge.svg)](https://github.com/transitive-bullshit/nextjs-notion-starter-kit/actions/workflows/build.yml) [![Prettier Code Formatting](https://img.shields.io/badge/code_style-prettier-brightgreen.svg)](https://prettier.io) - [Intro](#intro) - [Features](#features) - [Demos](#demos) - [Setup](#setup) - [Vercel Configuration](#vercel-configuration) - [URL Paths](#url-paths) - [Preview Images](#preview-images) - [Redis](#redis) - [Styles](#styles) - [Dark Mode](#dark-mode) - [Automatic Social Images](#automatic-social-images) - [Automatic Table of Contents](#automatic-table-of-contents) - [Responsive](#responsive) - [Analytics](#analytics) - [Fathom Analytics](#fathom-analytics) - [PostHog Analytics](#posthog-analytics) - [Environment Variables](#environment-variables) - [Contributing](#contributing) - [License](#license) ## Intro This repo is what I use to power my personal blog and portfolio site [transitivebullsh.it](https://transitivebullsh.it). It uses Notion as a CMS, [react-notion-x](https://github.com/NotionX/react-notion-x), [Next.js](https://nextjs.org/), and [Vercel](https://vercel.com). ## Features - Setup only takes a few minutes ([single config file](./site.config.ts)) 💪 - Robust support for Notion content via [react-notion-x](https://github.com/NotionX/react-notion-x) - Built using Next.js, TS, and React - Excellent page speeds - Smooth image previews - Automatic social images - Automatic pretty URLs - Automatic table of contents - Full support for dark mode - Quick search via CMD+K / CMD+P - Responsive for different devices - Optimized for Next.js and Vercel ## Demos - [Default demo](https://nextjs-notion-starter-kit.transitivebullsh.it) - Deployed from the `main` branch - [My site](https://transitivebullsh.it) - Deployed from the `transitive-bullshit` branch ## Setup **All config is defined in [site.config.ts](./site.config.ts).** This project requires a recent version of Node.js (we recommend >= 16). 1. Fork / clone this repo 2. Change a few values in [site.config.ts](./site.config.ts) 3. `npm install` 4. `npm run dev` to test locally 5. `npm run deploy` to deploy to vercel 💪 6. Double check your [Vercel project settings](#vercel-configuration) I tried to make configuration as easy as possible — All you really need to do to get started is edit `rootNotionPageId`. We recommend duplicating the [default page](https://notion.so/7875426197cf461698809def95960ebf) as a starting point, but you can use any public notion page you want. Make sure your root Notion page is **public** and then copy the link to your clipboard. Extract the last part of the URL that looks like `7875426197cf461698809def95960ebf`, which is your page's Notion ID. In order to find your Notion workspace ID (optional), just load any of your site's pages into your browser and open up the developer console. There will be a global variable that you can access called `block` which is the Notion data for the current page. If you enter `block.space_id`, it will print out your page's workspace ID. I recommend setting up a collection on your home page that contains all of your articles / projects / content. There are no structural constraints on your Notion workspace, however, so feel free to add content as you normally would in Notion. ### Vercel Configuration **Social media preview images won't work by default on Vercel**. You'll need to ensure that your site doesn't require auth. From your Vercel project settings, you'll want to **disable Vercel Authentication** from `Project -> Settings -> Deployment Protection`. ![How to disable Vercel Deployment Protection setting](https://github.com/user-attachments/assets/a1eb5a1f-da7a-497e-b4f6-f7e851a6cd8a 'How to disable Vercel Deployment Protection setting which causes social media preview image endpoint to return 401 Unauthorized') 💡 If you forget to do this your site will return `401 Unauthorized` responses when crawlers are trying to retrieve the images. ## URL Paths The app defaults to slightly different URL paths in dev vs prod (though pasting any dev pathname into prod will work and vice-versa). In development, it will use `/nextjs-notion-blog-d1b5dcf8b9ff425b8aef5ce6f0730202` which is a slugified version of the page's title suffixed with its Notion ID. I've found that it's really useful to always have the Notion Page ID front and center during local development. In production, it will use `/nextjs-notion-blog` which is a bit nicer as it gets rid of the extra ID clutter. The mapping of Notion ID to slugified page titles is done automatically as part of the build process. Just keep in mind that if you plan on changing page titles over time, you probably want to make sure old links will still work, and we don't currently provide a solution for detecting old links aside from Next.js's built-in [support for redirects](https://nextjs.org/docs/api-reference/next.config.js/redirects). See [mapPageUrl](./lib/map-page-url.ts) and [getCanonicalPageId](https://github.com/NotionX/react-notion-x/blob/master/packages/notion-utils/src/get-canonical-page-id.ts) for more details. You can override the default slug generation on a per-page basis by adding a `Slug` text property to your database. Any page which has a `Slug` property will use that as its slug. NOTE: if you have multiple pages in your workspace with the same slugified name, the app will throw an error letting you know that there are duplicate URL pathnames. ## Preview Images <p align="center"> <img alt="Example preview image" src="https://user-images.githubusercontent.com/552829/160142320-35343317-aa9e-4710-bcf7-67e5cdec586d.gif" width="458"> </p> We use [next/image](https://nextjs.org/docs/api-reference/next/image) to serve images efficiently, with preview images optionally generated via [lqip-modern](https://github.com/transitive-bullshit/lqip-modern). This gives us extremely optimized image support for sexy smooth images. Preview images are **enabled by default**, but they can be slow to generate, so if you want to disable them, set `isPreviewImageSupportEnabled` to `false` in `site.config.ts`. ### Redis If you want to cache generated preview images to speed up subsequent builds, you'll need to first set up an external [Redis](https://redis.io) data store. To enable redis caching, set `isRedisEnabled` to `true` in `site.config.ts` and then set `REDIS_HOST` and `REDIS_PASSWORD` environment variables to point to your redis instance. You can do this locally by adding a `.env` file: ```bash REDIS_HOST='TODO' REDIS_PASSWORD='TODO' ``` If you're not sure which Redis provider to use, we recommend [Redis Labs](https://redis.com), which provides a free plan. Note that preview images and redis caching are both optional features. If you’d rather not deal with them, just disable them in your site config. ## Styles All CSS styles that customize Notion content are located in [styles/notion.css](./styles/notion.css). They mainly target global CSS classes exported by react-notion-x [styles.css](https://github.com/NotionX/react-notion-x/blob/master/packages/react-notion-x/src/styles.css). Every notion block gets its own unique classname, so you can target individual blocks like this: ```css .notion-block-260baa77f1e1428b97fb14ac99c7c385 { display: none; } ``` ## Dark Mode <p align="center"> <img alt="Light Mode" src="https://transitive-bs.notion.site/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F83ea9f0f-4761-4c0b-b53e-1913627975fc%2Ftransitivebullsh.it_-opt.jpg?table=block&id=ed7e8f60-c6d1-449e-840b-5c7762505c44&spaceId=fde5ac74-eea3-4527-8f00-4482710e1af3&width=2000&userId=&cache=v2" width="45%"> &nbsp; &nbsp; &nbsp; &nbsp; <img alt="Dark Mode" src="https://transitive-bs.notion.site/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fc0839d6c-7141-48df-8afd-69b27fed84aa%2Ftransitivebullsh.it__(1)-opt.jpg?table=block&id=23b11fe5-d6df-422d-9674-39cf7f547523&spaceId=fde5ac74-eea3-4527-8f00-4482710e1af3&width=2000&userId=&cache=v2" width="45%"> </p> Dark mode is fully supported and can be toggled via the sun / moon icon in the footer. ## Automatic Social Images <p align="center"> <img alt="Example social image" src="https://user-images.githubusercontent.com/552829/162001133-34d4cf24-123a-4569-a540-f683b22830d1.jpeg" width="600"> </p> All Open Graph and social meta tags are generated from your Notion content, which makes social sharing look professional by default. Social images are generated automatically using [Vercel OG Image Generation](https://vercel.com/docs/concepts/functions/edge-functions/og-image-generation). You can tweak the default React template for social images by editing [api/social-images.tsx](./pages/api/social-image.tsx). You can view an example social image live in production [here](https://transitivebullsh.it/api/social-image?id=dfc7f709-ae3e-42c6-9292-f6543d5586f0). ## Automatic Table of Contents <p align="center"> <img alt="Smooth ToC Scrollspy" src="https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fcb2df62d-9028-440b-964b-117711450921%2Ftoc2.gif?table=block&id=d7e9951b-289c-4ff2-8b82-b0a61fe260b1&cache=v2" width="240"> </p> By default, every article page will have a table of contents displayed as an `aside` on desktop. It uses **scrollspy** logic to automatically update the current section as the user scrolls through your document, and makes it really easy to jump between different sections. If a page has less than `minTableOfContentsItems` (default 3), the table of contents will be hidden. It is also hidden on the index page and if the browser window is too small. This table of contents uses the same logic that Notion uses for its built-in Table of Contents block (see [getPageTableOfContents](https://github.com/NotionX/react-notion-x/blob/master/packages/notion-utils/src/get-page-table-of-contents.ts) for the underlying logic). ## Responsive <p align="center"> <img alt="Mobile article page" src="https://user-images.githubusercontent.com/552829/160132983-c2dd5830-80b3-4a0e-a8f1-abab5dbeed11.jpg" width="300"> </p> All pages are designed to be responsive across common device sizes. ## Analytics Analytics are an optional feature that are easy to enable if you want. ### Fathom Analytics [Fathom](https://usefathom.com/ref/42TFOZ) provides a lightweight alternative to Google Analytics. To enable, just add a `NEXT_PUBLIC_FATHOM_ID` environment variable, which will only be used in production. ### PostHog Analytics [PostHog](https://posthog.com/) provides a lightweight, **open source** alternative to Google Analytics. To enable, just add a `NEXT_PUBLIC_POSTHOG_ID` environment variable, which will only be used in production. ## Environment Variables If you're using Redis, analytics, or any other feature which requires environment variables, then you'll need to [add them to your Vercel project](https://vercel.com/docs/concepts/projects/environment-variables). If you want to test your redis builds with GitHub Actions, then you'll need to edit the [default build action](./.github/workflows/build.yml) to add `REDIS_HOST` and `REDIS_PASSWORD`. Here is an [example from my personal branch](https://github.com/transitive-bullshit/nextjs-notion-starter-kit/blob/transitive-bullshit/.github/workflows/build.yml#L17-L21). You'll also need to add these environment variables to your GitHub repo as [repository secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets). ## Contributing See the [contribution guide](contributing.md) and join our amazing list of [contributors](https://github.com/transitive-bullshit/nextjs-notion-starter-kit/graphs/contributors)! ## License MIT © [Travis Fischer](https://transitivebullsh.it) Support my open source work by <a href="https://x.com/transitive_bs">following me on twitter <img src="https://storage.googleapis.com/saasify-assets/twitter-logo.svg" alt="twitter" height="24px" align="center"></a>

CMS & Blogging Static Site Generators
7K Github Stars
chatgpt-twitter-bot
Open Source

chatgpt-twitter-bot

> [!IMPORTANT] > I highly recommend you use V2 of this project, which is a significant rewrite: https://github.com/dexaai/xbot --- <p align="center"> <a href="https://twitter.com/ChatGPTBot"> <img alt="Example Twitter thread using @ChatGPTBot" src="/media/demo.jpg"> </a> </p> # ChatGPT Twitter Bot <!-- omit in toc --> > Twitter bot powered by OpenAI's ChatGPT API. [![Build Status](https://github.com/transitive-bullshit/chatgpt-twitter-bot/actions/workflows/test.yml/badge.svg)](https://github.com/transitive-bullshit/chatgpt-twitter-bot/actions/workflows/test.yml) [![MIT License](https://img.shields.io/badge/license-MIT-blue)](https://github.com/transitive-bullshit/chatgpt-twitter-bot/blob/main/license) [![Prettier Code Formatting](https://img.shields.io/badge/code_style-prettier-brightgreen.svg)](https://prettier.io) - [Intro](#intro) - [Usage](#usage) - [Note](#note) - [Related](#related) - [License](#license) ## Intro [@ChatGPTBot](https://twitter.com/ChatGPTBot) is a Twitter bot that you can @mention with a prompt, and it will respond with a twitter thread containing the [ChatGPT](https://github.com/transitive-bullshit/chatgpt-api) response. It uses the [chatgpt](https://github.com/transitive-bullshit/chatgpt-api) Node.js package under the hood. ## Usage Just create a tweet @mentioning `@ChatGPTBot` containing your prompt: ``` @ChatGPTBot What is 1 + 1? ``` Then wait until the bot responds, which can be anywhere from a few seconds to a few minutes (hopefully not longer), depending on how much usage the bot receives. The ChatGPT response will be split up into multiple tweet-sized replies. ## Note **Don't be surprised if it takes awhile for the bot to respond to you**. This is due to: 1. Twitter heavily rate-limits bots. 2. This bot is just running on my local dev machine, so please be kind 🙏 > **Warning** > Any improper use of ChatGPTBot will result in an immediate block on Twitter. Improper use covers everything [OpenAI](https://openai.com/blog/chatgpt/)'s API would consider disallowed / harmful / banned. ## Related - V2 of this project is here: https://github.com/dexaai/xbot - Powered by [Agentic](https://github.com/transitive-bullshit/agentic) ## License MIT © [Travis Fischer](https://transitivebullsh.it) > **Note** > This project is not affiliated with OpenAI in any way. It is just a fun, open source side project that uses their API. If you found this project interesting, please consider [sponsoring me](https://github.com/sponsors/transitive-bullshit) or <a href="https://twitter.com/transitive_bs">following me on twitter <img src="https://storage.googleapis.com/saasify-assets/twitter-logo.svg" alt="twitter" height="24px" align="center"></a>

Live Chat & Chatbots Social Media Management
760 Github Stars
yt-semantic-search
Open Source

yt-semantic-search

<p align="center"> <a href="https://all-in-on-ai.vercel.app"> <img alt="Search the All-In Podcast using AI" src="/public/social.jpg" width="600"> </a> </p> # YouTube Semantic Search <!-- omit in toc --> > OpenAI-powered semantic search for any YouTube playlist — featuring the [All-In Podcast](https://all-in-on-ai.vercel.app) 🔥 [![Build Status](https://github.com/transitive-bullshit/yt-channel-search/actions/workflows/test.yml/badge.svg)](https://github.com/transitive-bullshit/yt-channel-search/actions/workflows/test.yml) [![MIT License](https://img.shields.io/badge/license-MIT-blue)](https://github.com/transitive-bullshit/yt-channel-search/blob/main/license) [![Prettier Code Formatting](https://img.shields.io/badge/code_style-prettier-brightgreen.svg)](https://prettier.io) - [Intro](#intro) - [How to get started](#How-to-get-started) - [Example Queries](#example-queries) - [Screenshots](#screenshots) - [How It Works](#how-it-works) - [TODO](#todo) - [Feedback](#feedback) - [Credit](#credit) - [License](#license) ## Intro I love the [All-In Podcast](https://www.youtube.com/channel/UCESLZhusAkFfsNsApnjF_Cg). But search and discovery with podcasts can be really challenging. I built this project to solve this problem... and I also wanted to play around with cool AI stuff. 😂 This project uses the latest models from [OpenAI](https://openai.com/) to build a semantic search index across every episode of the Pod. It allows you to find your favorite moments with Google-level accuracy and rewatch the exact clips you're interested in. You can use it to power advanced search across _any YouTube channel or playlist_. The demo uses the [All-In Podcast](https://www.youtube.com/channel/UCESLZhusAkFfsNsApnjF_Cg) because it's my favorite 💕, but it's designed to work with any playlist. ## How to get started - Clone the repository to your local machine. - Navigate to the root directory of the repository in your terminal. - Run the command `npm install` to install all the necessary dependencies. - Run the command `npx tsx src/bin/resolve-yt-playlist.ts` to download the English transcripts for each episode of the target playlist (in this case, the All-In Podcast Episodes Playlist). - Run the command `npx tsx src/bin/process-yt-playlist.ts` to pre-process the transcripts and fetch embeddings from OpenAI, then insert them into a Pinecone search index. - You can now run the command `npx tsx src/bin/query.ts` to query the Pinecone search index. (Optional) Run the command `npx tsx src/bin/generate-thumbnails.ts` to generate timestamped thumbnails of each video in the playlist. This step takes ~2 hours and requires a stable internet connection. - The frontend of the project is a Next.js webapp deployed to Vercel that uses the Pinecone index as a primary data store. You can run the command npm run dev to start the development server and view the webapp locally. Note that a few episodes may not have automated English transcriptions available, and that the project uses a hacky HTML scraping solution for this, so a better solution would be to use Whisper to transcribe the episode's audio. Also, the project support sorting by recency vs relevancy. ## Example Queries - [sweater karen](https://all-in-on-ai.vercel.app/?query=sweater+karen) - [best advice for founders](https://all-in-on-ai.vercel.app/?query=best+advice+for+founders) - [poker story from last night](https://all-in-on-ai.vercel.app/?query=poker+story+from+last+night) - [crypto scam ponzi scheme](https://all-in-on-ai.vercel.app/?query=crypto+scam+ponzi+scheme) - [luxury sweater chamath](https://all-in-on-ai.vercel.app/?query=luxury+sweater+chamath) - [phil helmuth](https://all-in-on-ai.vercel.app/?query=phil+helmuth) - [intellectual honesty](https://all-in-on-ai.vercel.app/?query=intellectual+honesty) - [sbf ftx](https://all-in-on-ai.vercel.app/?query=sbf+ftx) - [science corner](https://all-in-on-ai.vercel.app/?query=science+corner) ## Screenshots <p align="center"> <a href="https://all-in-on-ai.vercel.app"> <img alt="Desktop light mode" src="/public/images/screenshot-desktop-light.jpg" width="45%"> </a> &nbsp; &nbsp; &nbsp; &nbsp; <a href="https://all-in-on-ai.vercel.app"> <img alt="Desktop dark mode" src="/public/images/screenshot-desktop-dark.jpg" width="45%"> </a> </p> ## How It Works Under the hood, it uses: - [OpenAI](https://openai.com) - We're using the brand new [text-embedding-ada-002](https://openai.com/blog/new-and-improved-embedding-model/) embedding model, which captures deeper information about text in a latent space with 1536 dimensions - This allows us to go beyond keyword search and search by higher-level topics. - [Pinecone](https://www.pinecone.io) - Hosted vector search which enables us to efficiently perform [k-NN searches](https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm) across these embeddings - [Vercel](https://vercel.com) - Hosting and API functions - [Next.js](https://nextjs.org) - React web framework We use Node.js and the [YouTube API v3](https://developers.google.com/youtube/v3/getting-started) to fetch the videos of our target playlist. In this case, we're focused on the [All-In Podcast Episodes Playlist](https://www.youtube.com/playlist?list=PLn5MTSAqaf8peDZQ57QkJBzewJU1aUokl), which contains 108 videos at the time of writing. ```bash npx tsx src/bin/resolve-yt-playlist.ts ``` We download the English transcripts for each episode using a hacky HTML scraping solution, since the YouTube API doesn't allow non-OAuth access to captions. Note that a few episodes don't have automated English transcriptions available, so we're just skipping them at the moment. A better solution would be to use [Whisper](https://openai.com/blog/whisper/) to transcribe each episode's audio. Once we have all of the transcripts and metadata downloaded locally, we pre-process each video's transcripts, breaking them up into reasonably sized chunks of ~100 tokens and fetch it's [text-embedding-ada-002](https://openai.com/blog/new-and-improved-embedding-model/) embedding from OpenAI. This results in ~200 embeddings per episode. All of these embeddings are then upserted into a [Pinecone search index](https://www.pinecone.io) with a dimensionality of 1536. There are ~17,575 embeddings in total across ~108 episodes of the All-In Podcast. ```bash npx tsx src/bin/process-yt-playlist.ts ``` Once our Pinecone search index is set up, we can start querying it either via the webapp or via the example CLI: ```bash npx tsx src/bin/query.ts ``` We also support generating timestamp-based thumbnails of every YouTube video in the playlist. Thumbnails are generated using [headless Puppeteer](https://pptr.dev) and are uploaded to [Google Cloud Storage](https://cloud.google.com/storage). We also post-process each thumbnail with [lqip-modern](https://github.com/transitive-bullshit/lqip-modern) to generate nice preview placeholder images. If you want to generate thumbnails (optional), run: ```bash npx tsx src/bin/generate-thumbnails.ts ``` Note that thumbnail generation takes ~2 hours and requires a pretty stable internet connection. The frontend is a [Next.js](https://nextjs.org) webapp deployed to [Vercel](https://vercel.com) that uses our Pinecone index as a primary data store. ## TODO - Use [Whisper](https://github.com/m-bain/whisperX) for better transcriptions - Support sorting by recency vs relevancy ## Feedback Have an idea on how this webapp could be improved? Find a particularly fun search query? Feel free to send me feedback, either on [GitHub](https://github.com/transitive-bullshit/yt-semantic-search/issues/new) or [Twitter](https://twitter.com/transitive_bs). 💯 ## Credit - Inspired by [Riley Tomasek's project](https://twitter.com/rileytomasek/status/1603854647575384067) for searching the [Huberman YouTube Channel](https://www.youtube.com/@hubermanlab) - Note that this project is not affiliated with the All-In Podcast. It just pulls data from their [YouTube channel](https://www.youtube.com/channel/UCESLZhusAkFfsNsApnjF_Cg) and processes it using AI. ## License MIT © [Travis Fischer](https://transitivebullsh.it) If you found this project interesting, please consider [sponsoring me](https://github.com/sponsors/transitive-bullshit) or <a href="https://twitter.com/transitive_bs">following me on twitter <img src="https://storage.googleapis.com/saasify-assets/twitter-logo.svg" alt="twitter" height="24px" align="center"></a> The API and server costs add up over time, so if you can spare it, [sponsoring on Github](https://github.com/sponsors/transitive-bullshit) is greatly appreciated. 💕

Podcast Tools Search Engines
540 Github Stars
kindle-ai-export
Open Source

kindle-ai-export

# Kindle AI Export <!-- omit from toc --> > Export any Kindle book you own as text, PDF, EPUB, or as a custom, AI-narrated audiobook. 🔥 <p> <a href="https://github.com/transitive-bullshit/kindle-ai-export/actions/workflows/main.yml"><img alt="Build Status" src="https://github.com/transitive-bullshit/kindle-ai-export/actions/workflows/main.yml/badge.svg" /></a> <a href="https://github.com/transitive-bullshit/kindle-ai-export/blob/main/license"><img alt="MIT License" src="https://img.shields.io/badge/license-MIT-blue" /></a> <a href="https://prettier.io"><img alt="Prettier Code Formatting" src="https://img.shields.io/badge/code_style-prettier-brightgreen.svg" /></a> </p> - [Intro](#intro) - [How does it work?](#how-does-it-work) - [Audiobook Examples 🔥](#audiobook-examples-) - [Why is this necessary?](#why-is-this-necessary) - [Usage](#usage) - [Setup Env Vars](#setup-env-vars) - [Extract Kindle Book](#extract-kindle-book) - [Transcribe Book Content](#transcribe-book-content) - [(Optional) Export Book as PDF](#optional-export-book-as-pdf) - [(Optional) Export Book as EPUB](#optional-export-book-as-epub) - [(Optional) Export Book as Markdown](#optional-export-book-as-markdown) - [(Optional) Export Book as AI-Narrated Audiobook 🔥](#optional-export-book-as-ai-narrated-audiobook-) - [Disclaimer](#disclaimer) - [Author's Notes](#authors-notes) - [Alternative Approaches](#alternative-approaches) - [How is the accuracy?](#how-is-the-accuracy) - [License](#license) ## Intro This project makes it easy to export the contents of any ebook in your Kindle library as text, PDF, EPUB, or as a custom, AI-narrated audiobook. It only requires a valid Amazon Kindle account and an OpenAI API key. _You must own the ebook on Kindle for this project to work._ ### How does it work? It works by logging into your [Kindle web reader](https://read.amazon.com) account using [Playwright](https://playwright.dev), exporting each page of a book as a PNG image, and then using a vLLM (defaulting to `gpt-4.1-mini`) to transcribe the text from each page to text. Once we have the raw book contents and metadata, then it's easy to convert it to PDF, EPUB, etc. 🔥 This [example](./examples/B0819W19WD) uses the first page of the scifi book [Revelation Space](https://www.amazon.com/gp/product/B0819W19WD?ref_=dbs_m_mng_rwt_calw_tkin_0&storeType=ebooks) by [Alastair Reynolds](https://www.goodreads.com/author/show/51204.Alastair_Reynolds): <table> <tbody> <tr> <td> The automated script starts from the Kindle web reader's library page and selects the book we want to export. </td> <td> <img src="./examples/B0819W19WD/kindle-reader-library-example.jpg" alt="Kindle web reader library"> </td> </tr> <tr> <td> We use Playwright to navigate to each page of the selected book. </td> <td> <img src="./examples/B0819W19WD/kindle-reader-page-example.png" alt="Kindle web reader page"> </td> </tr> <tr> <td> For each page, we use Playwright to export a scaled down PNG screenshot of the page's rendered content. </td> <td> <img src="./examples/B0819W19WD/pages/0000-0001.png" alt="First page of Revelation Space by Alastair Reynolds"> </td> </tr> <tr> <td> We then convert each page's screenshot into text using one of OpenAI's vLLMs (<strong>gpt-4.1-mini</strong>. </td> <td> <p>Mantell Sector, North Nekhebet, Resurgam, Delta Pavonis system, 2551</p> <p>There was a razorstorm coming in.</p> <p>Sylveste stood on the edge of the excavation and wondered if any of his labours would survive the night. The archaeological dig was an array of deep square shafts separated by baulks of sheer-sided soil: the classical Wheeler box-grid. The shafts went down tens of metres, walled by transparent cofferdams spun from hyperdiamond. A million years of stratified geological history pressed against the sheets. But it would take only one good dustfall—one good razorstorm—to fill the shafts almost to the surface.</p> <p>“Confirmation, sir,” said one of his team, emerging from the crouched form of the first crawler. The man’s voice was muffled behind his breather mask. “Cuvier’s just issued a severe weather advisory for the whole North</p> </td> </tr> <tr> <td> After doing this for each page, we now have access to the book's full contents and metadata, so we can export it in any format we want. 🎉 </td> <td> <p>Here are some output previews containing only the first page of this book:</p> <ul> <li> <a href="./examples/B0819W19WD/book-preview.pdf">PDF output preview</a> </li> <li> <a href="./examples/B0819W19WD/book-preview.epub">EPUB output preview</a> </li> <li> <a href="./examples/B0819W19WD/book-preview.md">Markdown output preview</a> </li> <li> <a href="#audiobook-examples-">Audiobook examples</a> </li> </ul> </td> </tr> </tbody> </table> ### Audiobook Examples 🔥 We can even use TTS to generate custom audiobooks. Here are some auto-generated examples using a few different TTS providers & voices, containing only the first page of this book as a preview: <table> <tbody> <tr> <td align="center"> OpenAI tts-1-hd "alloy" voice <br />(female; solid quality but more expensive) </td> <td> <video src="https://github.com/user-attachments/assets/f634f2cc-cc65-4381-ba04-5fc59df69668"></video> </td> </tr> <tr> <td align="center"> OpenAI tts-1-hd "onyx" voice <br />(male; solid quality but more expensive) </td> <td> <video src="https://github.com/user-attachments/assets/5cc86ae3-9f82-414c-a69f-a2ab40db4ce1"></video> </td> </tr> <tr> <td align="center"> Unreal Speech "Scarlett" voice <br />(female; medium quality but cheaper) </td> <td> <video src="https://github.com/user-attachments/assets/232e5258-9f89-4493-a06b-5523ddf93226"></video> </td> </tr> </tbody> </table> ### Why is this necessary? **Kindle uses a [custom AZW3 format](https://en.wikipedia.org/wiki/Kindle_File_Format) which includes heavy DRM**, making it very difficult to access the contents of ebooks that you own. It is possible to [strip the DRM using existing tools](#alternative-approaches), but it's a serious pain in the ass, is very difficult to automate, and the "best" solution is expensive and not open source. This project changes that. _Why?_ Because I love reading books on Kindle (especially scifi books!!), but none of the content is _hackable_. The official Kindle apps are also lagging behind in their AI features, so my goal with this project was to make it easy to build AI-powered experiments on top of my own Kindle library. In order to do that, I first needed a reliable way to export the contents of my Kindle books in a reasonable format. I also created an [OSS TypeScript client for the unofficial Kindle API](https://github.com/transitive-bullshit/kindle-api), but I ended up only using some of the types and utils since Playwright + vLLMs allowed me to completely bypass their API and DRM. This approach should also be a lot less error-prone than using their unofficial API. ## Usage Make sure you have `node >= 18` and [pnpm](https://pnpm.io) installed. 1. Clone this repo 2. Run `pnpm install` 3. Set up environment variables ([details](#setup-env-vars)) 4. Run `src/extract-kindle-book.ts` ([details](#extract-kindle-book)) 5. Run `src/transcribe-book-content.ts` ([details](#transcribe-book-content)) 6. (Optional) Run `src/export-book-pdf.ts` ([details](#optional-export-book-as-pdf)) 7. (Optional) Export book as EPUB ([details](#optional-export-book-as-epub)) 8. (Optional) Run `src/export-book-markdown.ts` ([details](#optional-export-book-as-markdown)) 9. (Optional) Run `src/export-book-audio.ts` ([details](#optional-export-book-as-ai-narrated-audiobook-)) ### Setup Env Vars Set up these required environment variables in a local `.env`: ```sh AMAZON_EMAIL= AMAZON_PASSWORD= ASIN= OPENAI_API_KEY= ``` You can find your book's [ASIN](https://en.wikipedia.org/wiki/Amazon_Standard_Identification_Number) (Amazon ID) by visiting [read.amazon.com](https://read.amazon.com) and clicking on the book you want to export. The resulting URL will look like `https://read.amazon.com/?asin=B0819W19WD&ref_=kwl_kr_iv_rec_2`, with `B0819W19WD` being the ASIN in this case. ### Extract Kindle Book ```sh npx tsx src/extract-kindle-book.ts ``` - _(This takes a few minutes to run)_ - This logs into your [Amazon Kindle web reader](https://read.amazon.com) using headless Chrome ([Playwright](https://playwright.dev)). It can be pretty fun to watch it run, so feel free to tweak the script to use `headless: false` to watch it do its thing. - If your account requires 2FA, the terminal will request a code from you before proceeding. - It uses a persistent browser session, so you should only have to auth once. - Once logged in, it navigates to the web reader page for a specific book (`https://read.amazon.com/?asin=${ASIN}`). - Then it changes the reader settings to use a single column and a sans-serif font. - Then it extracts the book's table of contents. - Then it goes through each page of the book's main contents and saves a PNG screenshot of the rendered content to `out/${asin}/pages/${index}-${page}.png`. - Example: [examples/B0819W19WD/pages](./examples/B0819W19WD/pages) - Lastly, it resets the reader to the original position so your reading progress isn't affected. - It also records some JSON metadata with the TOC, book title, author, product image, etc to `out/${asin}/metadata.json`. - Example: [examples/B0819W19WD/metadata.json](./examples/B0819W19WD/metadata.json) > [!NOTE] > I'm pretty sure Kindle's web reader uses WebGL at least in part to render the page contents, because the content pages failed to generate when running this on a VM ([Browserbase](https://www.browserbase.com)). So if you're getting blank or invalid page screenshots, that may be the reason. ### Transcribe Book Content ```sh npx tsx src/transcribe-book-content.ts ``` - _(This takes a few minutes to run)_ - This takes each of the page screenshots and runs them through a vLLM (defaulting to `gpt-4.1-mini`) to extract the raw text content from each page of the book. - It then stitches these text chunks together, taking into account chapter boundaries. - The result is stored as JSON to `out/${asin}/content.json`. - Example: [examples/B0819W19WD/content.json](./examples/B0819W19WD/content.json) ### (Optional) Export Book as PDF ```sh npx tsx src/export-book-pdf.ts ``` - _(This should run instantly)_ - It uses [PDFKit](https://github.com/foliojs/pdfkit) under the hood. - It includes a valid table of contents for easy navigation. - The result is stored to `out/${asin}/book.pdf`. - Example: [examples/B0819W19WD/book-preview.pdf](./examples/B0819W19WD/book-preview.pdf) ### (Optional) Export Book as EPUB If you want, you can use [Calibre](https://calibre-ebook.com) to convert your book's PDF to the EPUB ebook format. On a Mac, you can install `calibre` using Homebrew (`brew install --cask calibre`). ```sh # replace B0819W19WD with your book's ASIN ebook-convert out/B0819W19WD/book.pdf out/B0819W19WD/book.epub --enable-heuristics ``` _([ebook-convert docs](https://manual.calibre-ebook.com/generated/en/ebook-convert.html))_ ### (Optional) Export Book as Markdown ```sh npx tsx src/export-book-markdown.ts ``` - _(This should run instantly)_ - The result is stored to `out/${asin}/book.md`. - Example: [examples/B0819W19WD/book-preview.md](./examples/B0819W19WD/book-preview.md) ### (Optional) Export Book as AI-Narrated Audiobook 🔥 ```sh npx tsx src/export-book-audio.ts ``` - _This takes a few minutes to run._ - We support two TTS engines: [OpenAI TTS](https://platform.openai.com/docs/models/tts) and [Unreal Speech TTS](https://unrealspeech.com). - To use OpenAI, set `TTS_ENGINE=openai` (the default) - To use Unreal Speech, set `TTS_ENGINE=unrealspeech` and `UNREAL_SPEECH_API_KEY=(your-api-key)` - OpenAI is higher quality but more expensive; Unreal Speech is medium quality and cheaper - To set the OpenAI voice, use `OPENAI_TTS_VOICE=onyx` (defaults to `alloy`) - To set the Unreal Speech voice, use `UNREAL_SPEECH_VOICE='Scarlett'` (defaults to `Scarlett`) - OpenAI TTS for a full novel (~1M tokens) is approximately **$30** (1.5GB MP3 ~21 hours long) - Unreal Speech TTS for a full novel (~1M tokens) is approximately **$2** (1.7GB MP3 ~23 hours long) - It should be pretty easy to support other TTS providers in the future. - The TTS will be broken up into reasonly sized chunks and stored in `mp3` files under `out/${asin}/audio/<tts-engine-hash>/`. - The `<tts-engine-hash>` directory is based on the TTS engine settings and book contents - After generating audio for each chunk, we use `ffmpeg` to concat them together. - You need to have `ffmpeg` installed locally for this to work - On Mac, `brew install ffmpeg` ([or install with more options](https://stackoverflow.com/a/55108365/2353599)) - The resulting audiobook is stored to `out/${asin}/audio/<tts-engine-hash>/audiobook.mp3`. - Examples: [examples/B0819W19WD/audio-previews](./examples/B0819W19WD/audio-previews) ## Disclaimer **This project is intended purely for personal and educational use only**. It is not endorsed or supported by Amazon / Kindle. By using this project, you agree to not hold the author or contributors responsible for any consequences resulting from its usage. ## Author's Notes This project will only work on Kindle books which you have access to in your personal library. **Please do not share the resulting exports publicly** – _we need to make sure that our authors and artists get paid fairly for their work_! With that being said, I also feel strongly that we should individually be able to use content that we own in whatever format best suits our personal needs, especially if that involves building cool, open source experiments for LLM-powered book augmentation, realtime narration, and other unique AI-powered UX ideas. I expect that Amazon Kindle will eventually get around to supporting some modern LLM-based features at some point in the future, but [ain't nobody got time to wait around for that](https://youtu.be/waEC-8GFTP4?t=25). ### Alternative Approaches If you want to explore other ways of exporting your personal ebooks from Kindle, [this article](https://www.digitaltrends.com/mobile/how-to-convert-kindle-to-pdf/) gives a great breakdown of the options available, including [Calibre](https://calibre-ebook.com) (FOSS) and [Epubor Ultimate](https://www.epubor.com/ultimate.html) (paid). Trying to use the most popular [free online converter](https://cloudconvert.com/azw3-to-pdf) will throw a DRM error. Compared with these approaches, the approach used by this project is much easier to automate. It also retains metadata about Kindle's original sync positions which is very useful for cases where you'd like to interoperate with Kindle. E.g., be able to jump from reading a Kindle book to listening to an AI-generated narration on a walk and then jumping back to reading the Kindle book and having the sync positions "just work". The main downside is that it's possible for some transcription errors to occur during the `image ⇒ text` step - which uses a multimodal LLM and is not 100% deterministic. In my testing, I've been remarkably surprised with how accurate the results are, but there are occasional issues mostly with differentiating whitespace between paragraphs versus soft section breaks. Note that both Calibre and Epubor also use heuristics to deal with things like spacing and dashes used by wordwrap, so the fidelity of the conversions will not be 100% one-to-one with the original Kindle version in any case. The other downside is that the **LLM costs add up to a dollars per book using `gpt-4.1-mini`**. With LLM costs constantly decreasing and local vLLMs, this cost per book should be free or almost free soon. The screenshots are also really good quality with no extra content, so you could swap any other OCR solution for the vLLM-based `image ⇒ text` quite easily. ### How is the accuracy? The accuracy / fidelity has been very close to perfect in my testing, with the only discrepancies being occasional whitespace issues. I'm sure there will be edge cases and ebook features that are missing (like embedded images), but it shouldn't be too hard to add those if there's enough interest. ## License MIT © [Travis Fischer](https://x.com/transitive_bs) If you found this project interesting, [consider following me on Twitter](https://x.com/transitive_bs).

E-book Management
305 Github Stars