Langchain - ConversationalRetrievalChain with memory and customized prompt

I'm trying to create a ConversationalRetrievalChain to answer based on a specific context provided by a pdf file. I can get good answers. The issue is that the memory is not working.

The code:

template2 = """
    Your name is Bot.
    You are a chatbot specialized in human resources. 
    Use the following context (delimited by <ctx></ctx>) to answer the questions.
    If there is any history of previous conversations, use it to answer (delimited by <hs></hs>)
    If you don't know the answer just answer that you don't know. 
    ------
    <ctx>
    {context}
    </ctx>
    ------
    <hs>
    {chat_history}
    </hs>
    ------
    Question:
    {question} 
    """
    
    
prompt2 = PromptTemplate(
                template=template2, 
                input_variables=["context", "chat_history", "question"])


def querying_V1(query, chat_history,prompt):
     memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True, output_key='answer')

    conv_chain = ConversationalRetrievalChain.from_llm(
          llm=llm,
          chain_type="stuff",
          retriever=vectorstore.as_retriever(search_kwargs={"k": 2}), 
          memory=memory,
          # condense_question_prompt=prompt,
          combine_docs_chain_kwargs={"prompt": prompt},
          output_key='answer',
          # return_source_documents=True,
          get_chat_history=lambda h : h,
          verbose = False
    )

    result = conv_chain({"question": query, "chat_history": chat_history})
    return result["answer"].strip()

while True:
    query = input("Prompt: ")
    if query == "q":
         sys.exit()
    result = querying_V1(query,chat_history,prompt2)
    print("\n" + result)
    chat_history.append((query, result))

This works to answer questions based on the context provided in the vectorstore. But if I ask for instance "what was the last question that I made?", the result is "I don't know".

I've read here Why doesn't langchain ConversationalRetrievalChain remember the chat history, even though I added it to the chat_history parameter? that if the ConversationalRetrievalChain object is being created in every iteration of the while loop, the new memory will overwrite the previous one.

If I define the memory and the conv_chain outside the function and call the conv_chain as input:

def querying_V2(query : str, conv_chain: object, chat_history):
    result = conv_chain({"question": query, "chat_history": chat_history})
    return result["answer"].strip()

The memory works but it seems to forget the context passed on the prompt...

I've already tried to change the result for result = conv_chain({"question": query) and some other suggestions based on similar issues. But until now, I was not able to have the ConversationalRetrievalChain answering based on the context provided in the prompt template and also based on the chat_history.