kunthawat/ALwrity
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

main_config changes - WIP

Browse Source
This commit is contained in:
AjaySi
2024-03-27 22:19:16 +05:30
parent e5a5372a29
commit 74b7bc3cbe
16 changed files with 63 additions and 543 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
README.md
View File

@@ -13,17 +13,17 @@ Leveraging AI technologies, it assists content creators and digital marketers in
To start using this tool, simply follow one of the options below:
---
### Option 1: Local Laptop Install 💻 (Recommended)
### Option 1: 𝗙𝗼𝗹𝗹𝗼𝘄 𝗺𝗲 Local Laptop Install 💻 (Recommended)
**Step 0**️⃣: **Pre-requisites:** Git, Python3
**Installing Python on Windows:**
**Installing Python on Windows:🐍🪟**
- Open PowerShell as admin: Press `Windows Key + X`, then select "Windows PowerShell (Admin)".
- Type `python`. If Python is not installed, Windows will prompt you to 'Get Python'.
- If Python is installed, you should see '>>>>>'.
**Installing Git on Windows:**
**Installing Git on Windows:🛺**
- Open PowerShell or Windows Terminal: Press `Windows Key + X`, then select "Windows Terminal".
- Paste or type and press enter:⏎.⏎.<br>

2
alwrity.py
View File

@@ -225,7 +225,7 @@ def blog_from_keyword():
break
else:
message_dialog(
title='Warning',
title='Error',
text='🚫 Blog keywords should be at least two words long. Please try again.'
).run()
if blog_keywords:

0
lib/gpt_providers/save_image.py → lib/blog_postprocessing/save_image.py
View File

11
lib/gpt_providers/README.md
View File

@@ -1,11 +0,0 @@
# OpenAI ChatGPT Integration for Enhanced Blog Generation
## Introduction
This toolkit, written in Python, integrates OpenAI's ChatGPT and other AI services for comprehensive blog generation. It allows for selecting and fine-tuning OpenAI models to suit various content creation needs, including text generation, image analysis, and speech-to-text conversion.
## Key Features
- **AI-Powered Text Generation**: Leverages OpenAI's ChatGPT for creating engaging and contextually relevant text based on user inputs.
- **Image Analysis and Detail Extraction**: Utilizes OpenAI's Vision API to analyze images and extract important details like Alt Text, Description, Title, and Caption.
- **Dynamic Image Generation**: Generates images from textual descriptions using DALL-E 2 and DALL-E 3 models, enhancing blog visual content.
- **Speech-to-Text Transcription**: Converts audio from YouTube videos to text, enabling easy content repurposing for blogs.
- **Image Variation Creation**: Produces variations of existing images, offering creative flexibility and maintaining topical relevance.

12
lib/gpt_providers/gpt_vision_img_details.py
View File

@@ -92,15 +92,3 @@ def analyze_and_extract_details_from_image(image_path):
except Exception as e:
logger.error(f"Unexpected error occurred during image analysis: {e}")
sys.exit(f"Exiting due to an unexpected error: {e}")
# Example usage
if __name__ == "__main__":
image_path = "path/to/your/image.jpg"
try:
details = analyze_and_extract_details_from_image(image_path)
if details:
print(f"Extracted image details: {details}")
else:
print("No details extracted from the image.")
except SystemExit as e:
print(f"Terminated: {e}")

0
lib/gpt_providers/gen_dali2_images.py → lib/gpt_providers/image_generation/gen_dali2_images.py
View File

0
lib/gpt_providers/gen_dali3_images.py → lib/gpt_providers/image_generation/gen_dali3_images.py
View File

0
lib/gpt_providers/gen_variation_img.py → lib/gpt_providers/image_generation/gen_variation_img.py
View File

4
lib/gpt_providers/openai_chat_completion.py
View File

@@ -23,7 +23,7 @@ def openai_chatgpt(prompt, model="gpt-3.5-turbo-0125", temperature=0.2, max_toke
prompt (str): The input text to generate completion for.
model (str, optional): Model to be used for the completion. Defaults to "gpt-4-1106-preview".
temperature (float, optional): Controls randomness. Lower values make responses more deterministic. Defaults to 0.2.
max_tokens (int, optional): Maximum number of tokens to generate. Defaults to 8192.
max_tokens (int, optional): Maximum number of tokens to generate. Defaults to 4096
top_p (float, optional): Controls diversity. Defaults to 0.9.
n (int, optional): Number of completions to generate. Defaults to 1.
@@ -34,7 +34,7 @@ def openai_chatgpt(prompt, model="gpt-3.5-turbo-0125", temperature=0.2, max_toke
SystemExit: If an API error, connection error, or rate limit error occurs.
"""
# Wait for 10 seconds to comply with rate limits
for _ in range(10):
for _ in range(5):
time.sleep(1)
try:

53
lib/gpt_providers/openai_chat_completion_streaming.py
View File

@@ -1,53 +0,0 @@
import sys
import logging
import openai
# Configure standard logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
def openai_chatgpt_streaming_text(user_prompt):
"""
Uses streaming functionality to get real-time output from OpenAI's GPT model.
Args:
user_prompt (str): The prompt to send to the model.
Returns:
str: The complete text generated by the model in response to the prompt.
Raises:
SystemExit: If an error occurs in connecting to the OpenAI API or during streaming.
"""
try:
client = openai.OpenAI()
response = client.chat.completions.create(
model="gpt-3.5-turbo-16k",
messages=[{"role": "user", "content": user_prompt}],
max_tokens=8192,
temperature=0.9,
n=1,
stream=True
)
collected_events = []
completion_text = ''
logger.info("Starting to receive streaming responses...")
for chunk in response:
collected_events.append(chunk) # Save the event response
event_text = chunk.choices[0].delta.content # Extract the text
completion_text += event_text # Append the text
sys.stdout.write(event_text)
sys.stdout.flush()
logger.info("Completed receiving streaming responses.")
return completion_text
except openai.OpenAIError as e:
logger.error(f"OpenAI API Error: {e}")
sys.exit("Exiting due to OpenAI API error.")
except Exception as e:
logger.error(f"Unexpected error during streaming: {e}")
sys.exit("Exiting due to an unexpected error.")

363
lib/gpt_providers/openai_gpt_provider.py
View File

@@ -1,363 +0,0 @@
########################################################
#
# openai chatgpt integration for blog generation.
# Choosing a model from openai and fine tuning its various paramters.
#
########################################################
import os
import sys
import requests
import re
import base64
from tqdm import tqdm, trange
import time # I wish
import openai
from openai import OpenAI
from pytube import YouTube
import tempfile
import datetime
from PIL import Image
from loguru import logger
logger.remove()
logger.add(sys.stdout,
colorize=True,
format="<level>{level}</level>|<green>{file}:{line}:{function}</green>| {message}"
)
def analyze_and_extract_details_from_image(image_path):
"""
Analyzes an image using OpenAI's Vision API and extracts Alt Text, Description, Title, and Caption.
This module provides functionality to analyze images using OpenAI's Vision API.
It encodes an image to a base64 string and sends a request to the OpenAI API
to interpret the contents of the image, returning a textual description.
Args:
image_path (str): Path to the image file.
api_key (str): Your OpenAI API key.
Returns:
dict: Extracted details including Alt Text, Description, Title, and Caption.
"""
logger.info(f"analyze_and_extract_details_from_image: Encoding image to base64")
def encode_image(path):
""" Encodes an image to a base64 string. """
with open(path, "rb") as image_file:
return base64.b64encode(image_file.read()).decode('utf-8')
base64_image = encode_image(image_path)
logger.info("Using GPT-4 Vision to get generated image details and tags.")
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}"
}
payload = {
"model": "gpt-4-vision-preview",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "The given image is used in blog content. Analyze the given image and suggest the following: Alternative text(Alt Text), description, title, caption."
},
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{base64_image}"
}
}
]
}
],
"max_tokens": 300
}
try:
response = requests.post("https://api.openai.com/v1/chat/completions", headers=headers, json=payload)
response.raise_for_status()
assistant_message = response.json()['choices'][0]['message']['content']
# Extracting details using regular expressions
alt_text_match = re.search(r'Alt Text: "(.*?)"', assistant_message)
description_match = re.search(r'Description: (.*?)\n\n', assistant_message)
title_match = re.search(r'Title: "(.*?)"', assistant_message)
caption_match = re.search(r'Caption: "(.*?)"', assistant_message)
image_details = {
'alt_text': alt_text_match.group(1) if alt_text_match else None,
'description': description_match.group(1) if description_match else None,
'title': title_match.group(1) if title_match else None,
'caption': caption_match.group(1) if caption_match else None
}
logger.info(f"analyze_and_extract_details_from_image: {image_details}")
return image_details
except requests.RequestException as e:
#sys.exit(f"Error: GPT-Vision: Failed to communicate with OpenAI API. Error: {e}")
logger.error(f"Error: GPT-Vision: Failed to communicate with OpenAI API. Error: {e}")
except Exception as e:
#sys.exit(f"Error occurred- GPT-Vision: {e}")
logger.error(f"Error occurred- GPT-Vision: {e}")
def openai_chatgpt(prompt, model="gpt-4-1106-preview", temperature=0.2, max_tokens=4096, top_p=0.9, n=1):
"""
Wrapper function for openai chat Completion
"""
# Error in generating topic content: Rate limit reached for default-global-with-image-limits
# in free account on requests per min. Limit: 3 / min. Please try again in 20s.
for i in trange(10):
time.sleep(1)
try:
client = OpenAI()
except Exception as err:
print("Error: OpenAI Client.")
exit(1)
try:
# using OpenAI's Completion module that helps execute any tasks involving text
response = client.chat.completions.create(
# model name used, there are many other models available under the umbrella of GPT-3
model=model,
# passing the user input
messages=[{"role": "user", "content": prompt}],
# generated output can have "max_tokens" number of tokens
max_tokens=max_tokens,
# number of outputs generated in one call
n=n,
top_p=top_p,
#frequency_penalty=0,
#presence_penalty=0
)
except openai.APIError as e:
#Handle API error here, e.g. retry or log
SystemError(f"OpenAI API returned an API Error: {e}")
except openai.APIConnectionError as e:
#Handle connection error here
SystemError(f"Failed to connect to OpenAI API: {e}")
except openai.RateLimitError as e:
#Handle rate limit error (we recommend using exponential backoff)
SystemError(f"OpenAI API request exceeded rate limit: {e}")
except Exception as err:
SystemError(f"OpenAI client Error: {err}")
return response.choices[0].message.content
def openai_chatgpt_streaming_text(user_prompt):
"""
Function to use stream=True for real time output from openai
"""
client = OpenAI()
response = client.chat.completions.create(
model="gpt-3.5-turbo-16k",
messages=[{"role": "user", "content": f"{user_prompt}"}],
max_tokens = 8192,
temperature = 0.9,
n=1,
stream=True
)
# Create variables to collect the stream of events, iterate through the stream of events
collected_events = []
completion_text = ''
print("\n\n.....COME ONE...\n\n")
for chunk in response:
collected_events.append(chunk) # save the event response
event_text = chunk.choices[0].delta.content # extract the text
completion_text += event_text # append the text
sys.stdout.write(chunk.choices[0].delta.content)
sys.stdout.flush()
print(f"COMLETION: {completion_text}")
return completion_text
def generate_dalle2_images(user_prompt, image_dir, num_images=1, img_size="512x512", response_format="url"):
"""
The generation API endpoint creates an image based on a text prompt.
Required inputs:
prompt (str): A text description of the desired image(s). The maximum length is 1000 characters.
Optional inputs:
--> num_images (int): The number of images to generate. Must be between 1 and 10. Defaults to 1.
--> size (str): The size of the generated images. Must be one of "256x256", "512x512", or "1024x1024".
Smaller images are faster. Defaults to "1024x1024".
-->response_format (str): The format in which the generated images are returned.
Must be one of "url" or "b64_json". Defaults to "url".
--> user (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse.
"""
logger.info(f"Generated Dall-e-2 blog images will be stored at: {image_dir=}")
try:
client = OpenAI()
img_generation_response = client.images.generate(
model="dall-e-2",
prompt=user_prompt,
n=num_images,
size=img_size
)
except openai.OpenAIError as e:
logger.error(f"Dalle-2 image generate error: {e.http_status}")
logger.error(f"{e.error}")
except Exception as aerr:
logger.info(f"Failed to generate Image with Dalle2, Error: {aerr}")
else:
img_path = save_generated_image(img_generation_response, image_dir)
return img_path
def generate_dalle3_images(img_prompt, image_dir, size="1024x1024", quality="hd", n=1):
""" Function to create images using Dalle3 """
client = OpenAI()
logger.info("Generating Dall-e-3 image for the blog.")
try:
img_generation_response = client.images.generate(
model="dall-e-3",
prompt=f"{img_prompt}",
size=size,
quality=quality,
n=1,
)
except openai.OpenAIError as e:
logger.error(f"Dalle-3 image generate error: {e.http_status}")
logger.error(f"{e.error}")
except Exception as e:
SystemError("Failed to Generate images with Dalle3.")
else:
#image_url = response.data[0].url
img_path = save_generated_image(img_generation_response, image_dir)
return img_path
def speech_to_text(video_url, output_path='.'):
""" Transcribes speech to text from a YouTube video URL. """
try:
# Create a YouTube object
print(f"Accessing YouTube URL: {video_url}")
yt = YouTube(video_url)
# Select the highest quality audio stream
print("Fetching audio stream. Select the highest quality audio stream")
audio_stream = yt.streams.filter(only_audio=True).first()
if audio_stream is None:
print("No audio stream found for this video.")
return
else:
# Download the audio stream
print(f"Downloading audio for: {yt.title}")
audio_file = audio_stream.download(output_path)
print(f"Downloaded: {yt.title} to {output_path}")
try:
# Check if the audio file size is less than 24MB
max_file_size = 24 * 1024 * 1024 # 24MB in bytes
file_size = os.path.getsize(audio_file)
if file_size > max_file_size:
print("Error: File size exceeds 24MB limit.")
exit(1)
# File uploads are currently limited to 25 MB and the following input
# file types are supported: mp3, mp4, mpeg, mpga, m4a, wav, and webm.
try:
client = OpenAI()
except Exception as err:
SystemExit("Unable to get openai client object: {err}")
print("Transcribing using Openai whisper.")
transcript = client.audio.transcriptions.create(
model="whisper-1",
file=open(audio_file, "rb"),
response_format="text"
)
return transcript
except Exception as err:
print(f"Failed in whisper transcription: {err}")
exit(1)
except Exception as e:
print(f"YT video download, An error occurred: {e}")
exit(1)
os.remove(audio_file)
# The idea is to download images from other blogs and recreate from it.
# This helps us generate images very close to the topic and also not worry about prompt message.
def gen_new_from_given_img(img_path, image_dir, num_img=1, img_size="1024x1024", response_format="url"):
"""
This function will take an image and produce a variant of it.
Required inputs:
image (str): The image to use as the basis for the variation(s). Must be a valid PNG file, less than 4MB, and square.
Optional inputs:
n (int): The number of images to generate. Must be between 1 and 10. Defaults to 1.
size (str): The size of the generated images. Must be one of "256x256", "512x512", or "1024x1024".
Smaller images are faster. Defaults to "1024x1024".
response_format (str): The format in which the generated images are returned. Must be one of "url" or "b64_json". Defaults to "url".
user (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse.
"""
logger.info(f"Generating a variation of the image at: {img_path}")
try:
client = OpenAI()
png = Image.open(img_path).convert('RGBA')
background = Image.new('RGBA', png.size, (255, 255, 255))
alpha_composite = Image.alpha_composite(background, png)
alpha_composite.save(img_path, 'PNG', quality=80)
variation_response = client.images.create_variation(
image=open(img_path, "rb"),
n=num_img,
size=img_size,
response_format=response_format,
)
except Exception as err:
logger.error(f"An error occured in Image.create_variation::: {err}")
SystemExit(1)
try:
img_path = save_generated_image(variation_response, image_dir)
except Exception as err:
logger.error(f"An error in Saving Image.create_variation::: {err}")
SystemExit(1)
else:
return img_path
def save_generated_image(img_generation_response, image_dir):
"""
Common util function to save the generated images for blog.
"""
# save the image
# We need to change the image name to unique, overwrite and for SEO considerations.
# Note: filetype should be *.png
generated_image_name = f"generated_image_{datetime.datetime.now():%Y-%m-%d-%H-%M-%S}.png"
generated_image_filepath = os.path.join(image_dir, generated_image_name)
# extract image URL from response
generated_image_url = img_generation_response.data[0].url
# We use the requests library to fetch the image from URL
logger.info(f"Fetch the image from url: {generated_image_url}")
response = requests.get(generated_image_url, stream=True)
# We use the Image Class from PIL library to open the image
Image.open(response.raw)
# Download the image.
try:
generated_image = requests.get(generated_image_url).content
except requests.exceptions.RequestException as e:
raise SystemExit(f"Failed to get generted image content: {e}")
else:
logger.info(f"Saving image at path: {generated_image_filepath}")
with open(generated_image_filepath, "wb") as image_file:
# Write the image to a file and store.
image_file.write(generated_image)
#logger.info(generated_image_filepath)
logger.info("Display the generated image.")
img = Image.open(generated_image_filepath)
img.show()
return generated_image_filepath

74
lib/gpt_providers/stt_audio_blog.py.bk
View File

@@ -1,74 +0,0 @@
from pytube import YouTube
import os
import sys
from loguru import logger
from openai import OpenAI
from tenacity import (
retry,
stop_after_attempt,
wait_random_exponential,
) # for exponential backoff
@retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(6))
def speech_to_text(video_url, output_path='.'):
"""
Transcribes speech to text from a YouTube video URL using OpenAI's Whisper model.
Args:
video_url (str): URL of the YouTube video to transcribe.
output_path (str, optional): Directory where the audio file will be saved. Defaults to '.'.
Returns:
str: The transcribed text from the video.
Raises:
SystemExit: If a critical error occurs that prevents successful execution.
"""
try:
logger.info(f"Accessing YouTube URL: {video_url}")
yt = YouTube(video_url)
logger.info("Fetching the highest quality audio stream")
audio_stream = yt.streams.filter(only_audio=True).first()
if audio_stream is None:
logger.warning("No audio stream found for this video.")
return None
logger.info(f"Downloading audio for: {yt.title}")
audio_file = audio_stream.download(output_path)
logger.info(f"Audio downloaded: {yt.title} to {output_path}")
# Checking file size
max_file_size = 24 * 1024 * 1024 # 24MB
logger.info(f"Downloaded Audio Size is: {max_file_size}")
file_size = os.path.getsize(audio_file)
if file_size > max_file_size:
logger.error("File size exceeds 24MB limit.")
sys.exit("File size limit exceeded.")
try:
logger.info("Initializing OpenAI client for transcription.")
client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))
logger.info("Transcribing using OpenAI's Whisper model.")
transcript = client.audio.transcriptions.create(
model="whisper-1",
file=open(audio_file, "rb"),
response_format="text"
)
return transcript, yt.title
except Exception as e:
logger.error(f"Failed in Whisper transcription: {e}")
sys.exit("Transcription failure.")
except Exception as e:
logger.error(f"An error occurred during YouTube video processing: {e}")
sys.exit("Video processing failure.")
finally:
if os.path.exists(audio_file):
os.remove(audio_file)
logger.info("Temporary audio file removed.")

0
lib/audio_to_blog/main_audio_to_blog.py → lib/speech_to_blog/main_audio_to_blog.py
View File

0
lib/audio_to_blog/main_youtube_research_blog.py → lib/speech_to_blog/main_youtube_research_blog.py
View File

0
lib/audio_to_blog/write_blogs_from_youtube_videos.py → lib/speech_to_blog/write_blogs_from_youtube_videos.py
View File

81
main_config
View File

@@ -1,38 +1,71 @@
###################################################
#
# Define Blog Content charateristics:
# This is the main config file which drives the code.
# This config will restrict code modifications and hence
# ease of usuability.
#
##################################################
###################################################
#
# Define Blog Content charateristics
# This config will restrict code modifications and hence ease of usuability.
#
###################################################
blog_tone="professional, how-to, begginer, research, programming,"
blog_character="???"
blog_tempo="???"
blog_audience="???"
blog_geographic="COUNTRY, hyper local"
# Length of blogs Or word count. Note: It wont be exact and depends on GPT providers and Max token count.
blog_length = 2000
search_intent="informational, commercial, company, news, finance, competitor, programming, scholar"
search_language="EN"
# professional, how-to, begginer, research, programming, casual, etc
blog_tone = "professional"
##################################################
#
# Blog postprocessing.
#
##################################################
# Target Audience, Gen-Z, Tech-savvy, Working professional, students, kids etc
blog_demographic = "All"
# informational, commercial, company, news, finance, competitor, programming, scholar etc
blog_type = "Informational"
# German, Chinese, Arabic, Nepali, Hindi, Hindustani etc
blog_language = "English"
# Specify the output format of the blog as: HTML, markdown, plaintext. Defaults to markdown.
blog_output_format="markdown"
blog_output_format = "markdown"
# Specify full path to folder where the final blog should be stored. ex: _posts
blog_output_folder=""
blog_output_folder = ""
# Specify full path to folder where blog images will be stored. ex: assets
blog_image_output_folder=""
blog_image_output_folder = ""
############################################################
#
# Blog Images details.
# Note: The images are created from the blog content. Blog title is used,
# the title is modified for image generation prompt.
#
############################################################
# Options are dalle2, dalle3, stable-diffusion.
image_gen_model = "stable-diffusion"
# Number of blog images to include.
num_images = 1
###########################################################
#
# Define LLM and its charateristics for fine control on output
# Note:
###########################################################
# Choose one of following: Openai, Google, Minstral
gpt_provider = "openai"
# Mention which model of the above provider to use.
model="gpt-3.5-turbo-0125"
# Temperature is a parameter that controls the “creativity” or randomness of the text generated by GPT.
# greater determinism and higher values indicating more randomness.
# while a lower temperature (e.g., 0.2) makes the output more deterministic and focused (thus, getting flagged as AI content).
temperature = 0.6
top_p=0.9
max_tokens=4096
n=1