Large Language Models (LLM) coupled with multiple AI capabilities are able to generate images and text, and also approach/achieve human level performance on a number of tasks. The world is going through a revolution in art (DALL-E, MidJourney, Imagine, etc.), science (AlphaFold), medicine, and other key areas, and this approach is playing a role in this revolution.
to explore strategies that generate prompts for LLMs to extract relevant entities from job descriptions and also to classify web pages given only a few examples of human scores.
A client has a system that collects news artifacts from web pages, tweets, facebook posts, etc. The client is interested in scoring a given new artifact against a topic. The client has hired experts to score a few of these news items. The range of results between 0 and 10 signifies the degree of relevance of the news item to the topic “breaking news that may lead to public unrest”.
Some columns of this data are as follows
- Title: title of the item
- Description: the content of the item
- Body: the content of the item
- Analyst_Average_Score: target variable - the score to be estimated
The data are job descriptions ( together named entities) and relationships between entities in json format. To understand more about where the data comes from, read How to Train a Joint Entities and Relation Extraction Classifier using BERT Transformer with spaCy 3 | by Walid Amamou | Towards Data Science
- Dataset Dev: For development and training
- Dataset Test: For testing and final reportin
- EDA is done in notebook data_eda.ipynb
- Classification of news items using Cohere’s Few-shot classification in notebook classification.ipynb
- Entity Extraction from job descriptions using Cohere's generative models in notebook entity_extraction.ipynb
git clone https://github.com/emtinanseo/Prompt-Engineering.git
cd Prompt-Engineering
pip install -r requirements.txt