This is a proof of concept to generate Airbyte low-code YAML connectors from API documentation. We want this to serve as inspiration to what can be done with LLMs. Here's how it works:
- Specify a goal, e.g "Fetch all pages posts"
- Provide one or more links to documentation, e.g for Notion: API Intro, API Versioning and Search Endpoint
This will generate an OpenAPI specificatin and a Airbyte low-code connector. Try it!
📽️ Demo
Notion
Perplexity
✅ Key features
- Co-pilot for generating ETL code for HTTP APIs
- Support Airbyte low-code YAML connectors
- Scrapes API documentation
- Produces OpenAPI specifications
📦 Installation
-
Clone the repository
git clone https://github.com/skyffel/airbyte-connector-generator-poc cd airbyte-connector-generator-poc -
Setup virtual environment
# Using virtualenv virtualenv env source env/bin/activate # Or using venv python3 -m venv env source env/bin/activate # Or using poetry poetry shell -
Install required packages
poetry install playwright install
🚀 How to use
Generates an Airbyte low-code YAML connector using the API documentation provided via URLs.
Set
DEBUG=truein.envto enable logs
skyffel --goal "<MY ETL GOAL>" --urls "<URL DOC 1>" --urls "<URL DOC 2>"
Here we generate a connector for extracting all blog posts from the Department of Justice.
skyffel \
--goal "extract all blog entries from department of justice" \
--urls https://www.justice.gov/developer/api-documentation/api_v1
📥 Import to Airbyte
After generating the connector, you need to import it to Airbyte. Eventually they might expose an API to do this programatically 🤞 Until then, here's how:
-
Go to your Airbyte workspace
-
Click on Builder [BETA] in the menu
-
Click “New custom connector” in the upper right corner
-
Click “Import a YAML” and select the generated
airbyte_connector.yaml -
Fill in the test values and run the test
-
Press “Publish to workspace”