Home
Softono
f

frigadehq

Professional software vendor delivering innovative solutions on the Softono platform. Specialized in both open-source and proprietary software development.

Total Products
2

Software by frigadehq

remote-storage
Open Source

remote-storage

[![npm version](https://img.shields.io/npm/v/remote-storage)](https://www.npmjs.com/package/remote-storage) [![tests](https://github.com/FrigadeHQ/remote-storage/actions/workflows/tests.yml/badge.svg)](https://github.com/FrigadeHQ/remote-storage/actions/workflows/tests.yml) [![npm license](https://img.shields.io/npm/l/remote-storage)](https://www.npmjs.com/package/remote-storage) [![code style: prettier](https://img.shields.io/badge/code_style-prettier-ff69b4.svg)](https://github.com/prettier/prettier) <H3 align="center"><strong>remoteStorage</strong></H3> <div align="center">remoteStorage is a simple library that combines the localStorage API with a remote server to persist data across browsers and devices.</div> <br /> <div align="center"> <a href="https://remote.storage">Website</a> <span> Β· </span> <a href="https://codesandbox.io/p/sandbox/remote-storage-demo-35hgqz?file=%2Fsrc%2Findex.ts">Live Demo</a> <span> Β· </span> <a href="https://github.com/FrigadeHQ/remote-storage">Source</a> <span> Β· </span> <a href="https://github.com/FrigadeHQ/remote-storage">Docs</a></div> <br /> ## Why Storing data in localStorage is useful, but it's not a good solution when you store data that needs to be shared across multiple devices or browsers. For instance, let's say you want to show a welcome modal to all new users that sign up for your product. If you use localStorage to track if a user has already seen this modal, your users will continue to get the experience over and over again every time they switch devices or browsers. That's where remoteStorage comes in. Using the same API as localStorage, remoteStorage allows you to easily read and write data on the fly while maintaining state across browsers and devices in order to provide a better user experience. ## Features - ✨ Simple API (same as localStorage) - πŸ” Secure (built-in JWT support) - πŸ‘Œ Works with all Javascript frameworks - πŸ“¦ Lightweight (~1 kB minified) - πŸ”“ Open source server and client (MIT license) - πŸ†“ Free hosted community server ## Quick start Install the library using your favorite package manager: ```bash npm install remote-storage ``` Or simply include it in your HTML: ```html <script src="https://unpkg.com/remote-storage@latest/dist/remote-storage.min.js" sync></script> ``` Import the library and use it like you would localStorage: ```javascript import { RemoteStorage } from 'remote-storage' const remoteStorage = new RemoteStorage({ userId: "my-user-id" }) const hasSeenNewFeature = await remoteStorage.getItem('hasSeenNewFeature') if (!hasSeenNewFeature) { await remoteStorage.setItem('hasSeenNewFeature', true) // Highlight your new and exciting feature! } ``` That's it! ## Documentation ### User IDs remoteStorage uses user IDs to identify users. A user ID is a string that uniquely identifies a user. It can be anything you want, but we recommend using a non-iterable UUID to prevent users from guessing other user IDs and accessing their data. The User ID is set when you create a new instance of remoteStorage: ```javascript const remoteStorage = new RemoteStorage({ userId: '123e4567-e89b-12d3-a456-426614174000' }) ``` If you don't provide a user ID, remoteStorage will generate a random UUID which will change every time the user visits your site. This is useful for testing, but defeats the purpose of remoteStorage since the data will not persist across devices or browsers. ### Instance IDs remoteStorage uses instance IDs to identify the application instance that is making the request. An instance ID is a string that uniquely identifies an application instance. Typically you would use the same instance ID for all requests from the same application instance. The instance ID is set when you create a new instance of remoteStorage: ```javascript const remoteStorage = new RemoteStorage({ userId: '123e4567-e89b-12d3-a456-426614174000', instanceId: 'my-cool-app' }) ``` ### Server We offer a free hosted community server at `https://api.remote.storage` (the default behavior if no `serverAddress` is provided). This hosted server should not be used for production apps, but it's great for testing and prototyping. To use a different server, simply pass the `serverAddress` option when creating a new instance of remoteStorage: ```javascript const remoteStorage = new RemoteStorage({ serverAddress: 'https://api.remote.storage', userId: '123e4567-e89b-12d3-a456-426614174000', instanceId: 'my-cool-app' }) ``` The server can be spun up using Docker in a few minutes. See the [server documentation](/apps/remote-storage-server/README.md) for more information. ### FAQ #### What data should I store in remoteStorage? remoteStorage should only be used for non-sensitive data. We recommend using it for things like user preferences, settings, and other non-sensitive data. Due to the nature of the public API, it's not a good fit for storing sensitive data like passwords or PII. #### How is remoteStorage different from localStorage? localStorage is a browser API that allows you to store data in the browser. The data is stored locally on the user's device and is not shared across devices or browsers. remoteStorage is a library that combines the localStorage API with a remote server to persist data across browsers and devices. #### How do I authenticate requests to remoteStorage? remoteStorage can be used without any authentication, but we highly recommend using JSON Web Tokens (JWT) to authenticate requests to the server. This can be done by setting the `JWT_SECRET` environment variable in `.env` to your JWT secret for the server. See the [server documentation](/apps/remote-storage-server/README.md) for more information. ## Contributing Pull requests are always welcome. Note that if you are going to propose drastic changes, make sure to open an issue for discussion first. This will ensure that your PR will be accepted before you start working on it. For any existing issues that do not yet have an assigned contributor, feel free to comment on the issue if you would like to work on it. We will assign the issue to you if we think you are a good fit. **Making changes:** implement your bug fix or feature, write tests to cover it and make sure all tests are passing. Ensure your commit leverages [Semantic Commit Messages](https://gist.github.com/joshbuchea/6f47e86d2510bce28f8e7f42ae84c716) and that your commit message follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) format. Then open a pull request to the main branch.

Storage & Backup NoSQL Databases
1.4K Github Stars
trench
Open Source

trench

<br /> <div align="center"> <a href="https://trench.dev?utm_source=github" target="_blank"> <picture> <source media="(prefers-color-scheme: dark)" srcset="/img/trench-dark.png"> <img alt="Trench Logo" src="/img/trench-light.png" width="220"/> </picture> </a> </div> <br/> <h1 align="center">Open-Source Analytics Infrastructure</h1> <p align="center"> <br /> <a href="https://docs.trench.dev" rel="dofollow"><strong>Documentation</strong></a> Β· <a href="https://trench.dev" rel="dofollow"><strong>Website</strong></a> Β· <a href="https://join.slack.com/t/trench-community/shared_invite/zt-2sjet5kh2-v31As3yC_zRIadk_AGn~3A"><strong>Slack Community</strong></a> Β· <a href="https://demo.trench.dev"><strong>Demo</strong></a> <br/> <br/> </p> ## 🌊 What is Trench? Trench is an event tracking system built on top of Apache Kafka and ClickHouse. It can handle large event volumes and provides real-time analytics. Trench is no-cookie, GDPR, and PECR compliant. Users have full control to access, rectify, or delete their data. Our team built Trench to scale up the real-time event tracking pipeline at <a href="https://frigade.com?utm_source=github-trench-readme" target="_blank">Frigade</a>. <picture> <source media="(prefers-color-scheme: dark)" srcset="/img/trench-cover.png"> <img src="/img/trench-cover.png" width="100%" alt="Trench Code Snippet" /> </picture> ## ⭐ Features - 🀝 Compliant with the Segment API (Track, Group, Identify) - 🐳 Deploy quickly with a single production-ready Docker image - πŸ’» Process thousands of events per second on a single node - ⚑ Query data in real-time - πŸ”— Connect data to other destinations with webhooks - πŸ‘₯ Open-source and MIT Licensed ## πŸ–₯️ Demo **Live demo:** [https://demo.trench.dev](https://demo.trench.dev) **Video demo:** Watch the following demo to see how you can build a basic version of Google Analytics using Trench and Grafana. https://github.com/user-attachments/assets/e3f64590-6e7e-41b9-b425-7adb5a1e19b1 ## πŸš€ Quickstart Trench has two methods of deployment: 1. **Trench Self-Hosted**: An open-source version to deploy and manage Trench on your own infrastructure. 2. **Trench Cloud**: A fully-managed serverless solution with zero ops, autoscaling, and 99.99% SLAs. ### 1. Trench Self-Hosted πŸ’» Follow our self-hosting instructions below and in our [quickstart guide](https://docs.trench.dev/quickstart) to begin using Trench Self-Hosted. If you have questions or need assistance, you can join our [Slack group](https://join.slack.com/t/trench-community/shared_invite/zt-2sjet5kh2-v31As3yC_zRIadk_AGn~3A) for support. #### Quickstart 1. **Deploy Trench Dev Server**: The only prerequisite for Trench is a system that has Docker and Docker Compose installed [see installation guide](https://docs.docker.com/compose/install/). We recommend having at least 4GB of RAM and 4 CPU cores for optimal performance if you're running a production environment. After installing Docker, you can start the local development server by running the following commands: ```sh git clone https://github.com/frigadehq/trench.git cd trench/apps/trench cp .env.example .env docker-compose -f docker-compose.yml -f docker-compose.dev.yml up --build --force-recreate --renew-anon-volumes ``` The above command will start the Trench server that includes a local ClickHouse and Kafka instance on `http://localhost:4000`. You can open this URL in your browser and you should see the message `Trench server is running`. You shouldupdate the `.env` file to change any of the configuration options. 2. **Send a sample event**: You can find and update the default public and private API key in the `.env` file. Using your public API key, you can send a sample event to Trench as such: ```sh curl -i -X POST \ -H "Authorization:Bearer public-d613be4e-di03-4b02-9058-70aa4j04ff28" \ -H "Content-Type:application/json" \ -d \ '{ "events": [ { "userId": "550e8400-e29b-41d4-a716-446655440000", "type": "track", "event": "ConnectedAccount", "properties": { "totalAccounts": 4, "country": "Denmark" }, }] }' \ 'http://localhost:4000/events' ``` 3. **Querying events**: You can query events using the `/events` endpoint (see [API reference](https://docs.trench.dev/api-reference/events-get) for more details). You can also query events directly from your local Trench server. For example, to query events of type `ConnectedAccount`, you can use the following URL: ```sh curl -i -X GET \ -H "Authorization: Bearer private-d613be4e-di03-4b02-9058-70aa4j04ff28" \ 'http://localhost:4000/events?event=ConnectedAccount' ``` This will return a JSON response with the event that was just sent: ```json { "results": [ { "uuid": "25f7c712-dd86-4db0-89a8-d07d11b73e57", "type": "track", "event": "ConnectedAccount", "userId": "550e8400-e29b-41d4-a716-446655440000", "properties": { "totalAccounts": 4, "country": "Denmark" }, "timestamp": "2024-10-22T19:34:56.000Z", "parsedAt": "2024-10-22T19:34:59.530Z" } ], "limit": 1000, "offset": 0, "total": 1 } ``` 4. **Execute raw SQL queries**: Use the queries endpoint to analyze your data. Example: ```sh curl -i -X POST \ -H "Authorization:Bearer public-d613be4e-di03-4b02-9058-70aa4j04ff28" \ -H "Content-Type:application/json" \ -d \ '{ "queries": [ "SELECT COUNT(*) FROM events WHERE userId = '550e8400-e29b-41d4-a716-446655440000'" ] }' \ 'http://localhost:4000/queries' ``` Sample query result: ```json { "results": [ { "count": 5 } ], "limit": 0, "offset": 0, "total": 1 } ``` ### Kafka authentication Trench supports connecting to Kafka clusters that require SASL and/or SSL. Configure via the following environment variables (all optional): - `KAFKA_SSL_ENABLED`: Enable SSL/TLS when connecting to brokers. Values: `true`/`false` (default: `false`). - `KAFKA_SSL_REJECT_UNAUTHORIZED`: Whether to verify broker certificates. Values: `true`/`false` (default: `true`). Set to `false` when using self-signed certs in development. - `KAFKA_SSL_CA`: CA certificate contents (PEM). Use when brokers use a custom CA. - `KAFKA_SSL_CERT`: Client certificate contents (PEM) if mutual TLS is required. - `KAFKA_SSL_KEY`: Client private key (PEM) if mutual TLS is required. - `KAFKA_SASL_MECHANISM`: One of `plain`, `scram-sha-256`, or `scram-sha-512`. - `KAFKA_SASL_USERNAME`: SASL username (required when `KAFKA_SASL_MECHANISM` is set). - `KAFKA_SASL_PASSWORD`: SASL password (required when `KAFKA_SASL_MECHANISM` is set). Notes: - For SSL cert variables, provide the PEM content directly (including header/footer) or mount files and load into env before starting. - When using Bitnami Kafka images locally, the default `docker-compose.yml` uses PLAINTEXT; set the appropriate broker listeners and advertise SSL/SASL endpoints in your Kafka deployment if required. - Kafka authentication for ClickHouse should be configured in ClickHouse server configuration files, not in SQL migrations. ### ClickHouse Kafka Authentication For ClickHouse to connect to authenticated Kafka clusters, you need to configure authentication in ClickHouse server configuration files. ### 2. Trench Cloud ☁️ If you don't want to selfhost, you can get started with Trench in a few minutes via: * our [Cloud web interface](https://app.trench.dev/sign-up) * our [Cloud quickstart guide](https://docs.trench.dev/cloud-quickstart) <picture> <source media="(prefers-color-scheme: dark)" srcset="/img/trench-dashboard-dark.png"> <img src="/img/trench-dashboard.png" width="100%" alt="Trench Code Snippet" /> </picture> ## πŸ”— Links - [Website](https://trench.dev?utm_campaign=github-readme) - [Documentation](https://docs.trench.dev/) - [Slack community](https://join.slack.com/t/trench-community/shared_invite/zt-2sjet5kh2-v31As3yC_zRIadk_AGn~3A) ### Kafka Authentication Examples Trench supports Kafka authentication with SASL and SSL. Here are examples of how to configure both Node.js KafkaJS client and ClickHouse for authenticated Kafka connections. **Quick Start:** - **SASL-only**: `docker compose -f docker-compose.sasl.yml up -d --build` - **SSL+SASL**: Generate certs first with `./scripts/generate-kafka-certs.sh`, then run the compose file #### SASL-Only Authentication ```bash # Run with SASL authentication (no SSL) docker compose -f docker-compose.sasl.yml up -d --build ``` This setup uses: - **Node.js KafkaJS**: `KAFKA_SASL_MECHANISM=PLAIN`, `KAFKA_SASL_USERNAME=kafka_user`, `KAFKA_SASL_PASSWORD=kafka_password` - **ClickHouse**: Pre-configured with `clickhouse-kafka-auth-config-example/clickhouse-sasl.xml` for Kafka authentication - **Kafka**: SASL_PLAINTEXT listener on port 9095 #### SSL+SASL Authentication ```bash # Step 1: Generate SSL certificates ./scripts/generate-kafka-certs.sh # Step 2: Set environment variables for SSL certificates export KAFKA_SSL_CA=$(cat ./certs/ca.pem) export KAFKA_SSL_CERT=$(cat ./certs/client.pem) export KAFKA_SSL_KEY=$(cat ./certs/client-key.pem) # Step 3: Run with both SSL and SASL authentication docker compose -f docker-compose.ssl-sasl.yml up -d --build ``` This setup uses: - **Node.js KafkaJS**: SSL+SASL authentication via environment variables - **ClickHouse**: Pre-configured with `clickhouse-kafka-auth-config-example/clickhouse-ssl-sasl.xml` for Kafka authentication - **Kafka**: SASL_SSL listener on port 9095 with SSL certificates ## πŸ“š Authors Trench is a project built by [Frigade](https://frigade.com). ## πŸ“„ License MIT License

Analytics & BI Knowledge Bases & RAG Event Tracking & CDP
1.6K Github Stars