top of page

Let's try making an AI 2

Jun 3
3 min read

Since I last started developing AI, I've encountered several limitations.

For example, if you manage training data in CSV format, the amount of data will become enormous.

Furthermore, manually updating the data each time, like in master data maintenance, is time-consuming.

So, I decided to try using the SQLite library that Python apparently has.

This is also a good topic for learning how to work with databases using Python.

We have implemented two tables so far.


Training data

Personality data


I also confirmed that the data was registered in SQLite Web.

By the way, I have an Oracle vendor certification,

This was actually the first time I had ever written CREATE and DROP statements myself.

Creating and deleting tables is something I rarely do in my work, so it was a bit of a novel experience.


As we implemented it, several problems arose.

For example, variations in lettering.

Hello

Hello


These differences in expression end up being registered as separate data entries.

Even if the meaning is the same, different expressions result in separate records.

If this continues, the database will become overloaded.

Therefore, I realized that a mechanism to absorb fluctuations was necessary.


Furthermore, the system was designed so that the learned content would remain permanently,

Incorrect data will remain unchanged.

As a countermeasure, we changed to a method that regenerates the training data each time.

However, this method was resource-intensive, so we also implemented caching.


In other words, the system maintains processing speed by storing the results of the learned data as a cache and reusing them.


When developing AI, I realized that it's harder to make it forget what it's wrong than to teach it what it's right.


import sqlite3

import pandas as pd

import numpy as np


from sklearn.feature_extraction.text import TfidfVectorizer

from sklearn.metrics.pairwise import cosine_similarity


# -------------------------------

# Cache variables

# -------------------------------

cached_vectorizer = None

needs_update = True



# -------------------------------

# Similarity search

# -------------------------------

def get_best_response(user_input, situation):

global cached_vectorizer, needs_update


conn = sqlite3.connect('taka_system.db')


df_candidates = pd.read_sql_query(

"SELECT question, answer, emortion FROM training_data WHERE situation = ?",

conit,

params=(situation,)

)


conn.close()


if df_candidates.empty:

return "UNKNOWN", "Normal"


# n-gram for handling fluctuations

if needs_update or cached_vectorizer is None:

cached_vectorizer = TfidfVectorizer(

analyzer='char',

ngram_range=(2, 3)

)

db_vecs = cached_vectorizer.fit_transform(

df_candidates["question"].tolist()

)

needs_update = False

else:

db_vecs = cached_vectorizer.transform(

df_candidates["question"].tolist()

)


user_vec = cached_vectorizer.transform([user_input])


similarities = cosine_similarity(user_vec, db_vecs)


best_score = np.max(similarities)

best_idx = np.argmax(similarities)


# Similarity threshold

if best_score < 0.2:

return "UNKNOWN", "Normal"


return (

df_candidates.iloc[best_idx]["answer"],

df_candidates.iloc[best_idx]["emortion"]

)



# -------------------------------

# Save learning

# -------------------------------

def save_new_interaction(question, answer, emotion, situation):

global needs_update


conn = sqlite3.connect('taka_system.db')

cursor = conn.cursor()


cursor.execute('''

INSERT INTO training_data (question, answer, emortion, situation)

VALUES (?, ?, ?, ?)

''', (question, answer, emotion, situation))


conn.commit()

conn.close()


needs_update = True


print(f"("Saved as training data. Emotion: {emotion}"))")



# -------------------------------

# Main process

# -------------------------------

def main():

current_situation = "Private"


print("System startup. You can switch to Business with [/set Business].")


while True:

user_input = input("\n[Input]: ")


if user_input.lower() == "exit":

break


if user_input.startswith("/set "):

current_situation = user_input.split(" ")[1]

print(f"The situation has been changed to {current_situation}.")

continue


response, emotion = get_best_response(

user_input,

current_situation

)


if response == "UNKNOWN":


However, although this has made it possible to organize the input,

My personality is not reflected at all in my output.


Even though we've gone to the trouble of storing personality data in the database,

I felt that simply returning past answers wasn't truly "my own AI."

Since we've gone to the trouble of providing personality data, we believe it's only meaningful if the response changes depending on the current state and tendencies.

As the next step, I plan to incorporate a mechanism that reflects the characteristics of the output itself.

 
 
 

Comments


bottom of page