POI GPT : Extracting POI Information from Social Media Text Data

: Point of Interest (POI) is an important intermediary connecting geo data and text data in smart cities, widely used to extract and identify urban functional areas. While computer uses numerical coordinates, human uses places names or addresses to find location, leading to spatial-semantic ambiguities. However, traditional methods of extracting POIs are time-consuming and costly, and has the limitation of the lack of integration of functionalities such as information extraction(IE), information searching. Also, previous models have low accessibility and high barriers for users. With the advent of Large Language Models(LLMs) we propose a method that connects LLM models and POI information based on social media text data. By employing two steps, named entities recognition(NER) and POI information searching, we introduce POI GPT, the specialized model for providing precise location of POIs in social media text data. We compared its results with those obtained by human experts, NER model and zero-shot prompts. The findings show that our model effectively found the POI and precise location from social media text data. In result, POI GPT is a effective model that solves the existing POI extraction problems. We provide new extraction technique of POI GPT which is a new paradigm in traditional urban research methodologies and be actively utilized in urban studies in the future.


INTRODUCTION
Point of Interest (POI) is an important intermediary connecting geo data and text data in smart cities, widely used to extract and identify urban functional areas.Most human activities occur at POIs, and their lightweight and accessible nature allows for effective and accurate understanding of trends in the relationship between humans and cities (Gao et al., 2017;Zhang et al., 2021).Typically, POI is provided as georeferenced information in dataset and place name, address is included in POI information.Georeferencing by place names (known as toponyms) is the most common way of associating textual information with a geographic location (Kuai et al., 2020).While computers use numerical coordinates(latitude-longitude) to recognize places, people use place names or addresses to find it.This, however, makes confusion between finding exact location since coordinates represent only one location but place names do not.Some places that share same name, such as franchise, may be commonly identified with several locations (Zhu et al., 2016), leading to spatial-semantic ambiguities.In an effort to reduce this gap, numerous studies have been conducted to extract accurate place names from text data, for example NER.However, traditional methods of extracting POIs and searching each information are timeconsuming and costly, and extracting exact POIs from text is very difficult unless the text is geotagged.
Meanwhile, the accumulation of various big data from social media has led to the accumulation of text data containing the opinions and experiences of urban residents.This facilitates the analysis of urban residents' perceptions and urban space utilization.Big data from social media has been widely used for urban vitality, epidemic detection, satisfaction analysis (Park et al., 2021;Wakamiya et al., 2018;Kwon & Lee, 2023), etc. Especially since the outbreak of COVID-19, non-face-to-face online platforms have been activated and people are actively communicating online, making social media data-based analysis seem useful (Kim, 2023).Due to the sharp increase in online activities following COVID-19, there is a need to use social media text data to analyse residents' experiences and capture urban activities.
On the other hand, with the advent of Generative Pre-trained Transformer-based Large Language Models (LLMs) in recent years, the field of Natural Language Processing (NLP) has rapidly advanced.Notable models include GPT-4, LLaMA, and PaLM.These LLMs, with their large language models and training datasets, have shown powerful and impressive performance in understanding and generating meaning, as well as in some complex tasks.Especially, OpenAI recently released 'GPTs' which provides custom ChatGPT creation.It supports knowledge files, web browsing, API use, and instructions for creating custom ChatGPTs (OpenAI, 2023).This demonstrates the potential for tools that not only extract POI place names but also provide information about the POIs.Despite of amazing performance and potential of ChatGPT, there are still several critical limitations that impede its reliability and broad adoption (Fu et al., 2024).First, ChatGPT's performance compared with humans and other NLP techniques, is currently unknown.Second, ChatGPT's performance is highly sensitive on how prompts are crafted.
Accordingly, this study utilizes social media data with text geotagging and narrows the gap between spatial location and text data containing place names.We developed a model called POI GPT, using ChatGPT to extract and find additional information of POIs.The model utilizes OpenAI's ChatGPT, which has recently gained prominence for its exceptional performance and in particular, we chosen GPTs to provide a more convenient user interface.By leveraging ChatGPT's inherent understanding, learning, and generative capabilities, POI GPT can overcome the limitations of existing POI  (Chuang, 2016).Previous approaches to the IE categorized into three types by web pages' structure: structured, semi-structured and unstructured (Zhang et al., 2010).First, structured approach is about structured webpage which has format of predefined and strict.This kind of information can be easily be extracted using frame model.Second, semi-structured approach is an intermediate between structured format and unstructured text.This approach helps disambiguation on free texts.Lastly, unstructured approach is about unstructured webpage with free text of natural language.Generally used natural language techniques and rules are typically based on patterns involving syntactic relations between words, part-of-speech, and phrases or named entities (Downey et al., 2007).
There are various kinds of IE tasks such as postal address extraction, named entity recognition(NER).Postal address extraction is task about extracting complete addresses based on certain geographic terms (Sanderson & Kohler, 2004).NER is another major task in IE that seeks text's location and classify in pre-defined categories, such as location, personal name, organization (Sun et al., 2018).Early NER used rule-based approaches, such as lexical, contextual, morphological lexicon rules, but no longer mainstream approaches since built with less of no labelled corpus have been allowed.Such technological advancements have brought machine learning-based NER to the forefront.As the necessity for large volumes of language knowledge, in the form of lexicons and feature engineering, to achieve higher performance has increased, deep learning-based NER has concurrently risen to prominence (Sun et al., 2018).
LLMs have primarily been used in the field of urban planning, extensively for purposes such as evaluation and planning support system(PSS) across a broad spectrum.Fu et al.(2024) analyzed public feedback data in urban planning by employing ChatGPT and standard NLP techniques.As compared results shown, ChatGPT had few limitations but presented a transformative opportunity for planners especially on public feedback big data.Others proposed planning multi-agent collaboration framework by applying LLMs in original urban for participatory urban planning (Zhou et al., 2024) or by advanced tooling capabilities (Zhu et al., 2024).In terms of model proposing, Feng et al. ( 2024) developed POI recommendation system based on ChatGPT prompting strategies based on considering the geographical influence and sequential transitions.As mentioned earlier, with the increase in machine learning and deep learning approaches to NER models, applications of LLMs have also been active.Various studies have identified place names based on Twitter messages in disaster situations (Hu et al., 2023) and from drone delivery addresses (Yao et al., 2024).All these studies utilized NER to convert raw user addresses into precise locations using language models.In the same vein, our study aims to utilize LLMs to recognize POI place names from social media text data, converting users' raw natural language place names into precise addresses.
Although interest in LLM applications is rapidly growing, there are two major gaps in this emerging field within the information extraction.The first gap identified is the lack of integration of functionalities in the use of LLMs.While fields such as Information Extraction(IE) and appropriate information retrieval have been developing independently, attempts to combine these have been scarce.However, with the advent of LLMs and advances in custom GPT development, it is now possible to merge traditional IE capabilities with LLM's information retrieval and text generation skills.This integration allows for the creation of models that can perform POI extraction and POI information search, and explain the findings in a manner understandable to users.Second gap is the accessibility of NLP techniques for users.Previous models have been critiqued for being technology-driven without considering the user experience (Geertman and Stillwell, 2020).Recognizing that a model becomes meaningful when used by individuals, there is a pressing need for NLP models that consider user experience to improve accessibility.Consequently, this study employs GPTs using the ChatGPT Interface to develop a more user-friendly model, thereby overcoming the accessibility limitations that have been challenging for users to navigate.

METHOD
The objective of this study is to benchmark our POI GPT against experts, NER model and zero-prompt model with Chat GPT to accomplish extraction of POI information from blog text data.

Dataset
In this study, we selected a corpus of crowdsourced travel blog, travelblog.org.It is a worldwide popular travel blogging website and in each post, author describes their travel experiences(Ballatore and Adams, 2015; Davydova, 2012).The main locations are organized by hierarchical vocabulary in order of continent, country, administrative regions.Blog dates, number of words are also provided.
To collect the data, we used web crawling in Python with Selenium package and BeautifulSoup library.In total, we collected 4,290 travel blogs text data from 2019 to 2023.For the purpose of including only text data that contains POIs within, a preprocessing procedure was undertaken.This process involves retaining data with a word count exceeding 100 and below 300.Also to retain only reviews written in English, we employed the langdetect library in Python to remove reviews that were not in English.Through this procedure, total 63 travel blogs remained in dataset.Finally, we examined the data distribution by continent to check data's continent distribution (Figure 1).Example of acquired travel blog text data is provide in Table 1.

GPT model
Our model is done through 'GPTs', which provides custom ChatGPT creation.publicly shared services in ChatGPT for customers and allows easy integration with users' data.Despite the existence of the GPT-4 APIs, we utilized GPTs over the API due to their provision of a GPT interface.Since GPTs offers a more user-friendly interface, it enhances applicability.This approach is significant in that it overcomes the limitations of supply-driven(technology-driven) urban technologies (Geertman and Stillwell, 2020), such as PSS, by offering a demand-driven, user-friendly environment adopting GPTs.Specifically, GPTs offers functionalities such as support for knowledge files, web browsing capabilities, API usage, and instructions for creating custom ChatGPTs to specific purposes (OpenAI, 2023).
The model in this study is composed of two steps.First, POI GPT performs Named Entity Recognition(NER) to identify POIs within text data.Second, it finds information for each POI, such as POI categorization, longitude, and latitude.Ultimately, the identified POIs and their associated information are compiled and presented to the user.Figure 2 provides an overview of our model.

POI extraction :
Large language models can extract key information from various textual sources, including travel blog text data, public comments, and academic studies (Zhu et al., 2024).This process is about finding the POI names from the travel blog text data.

POI information :
GPTs also offers web browsing and utilizing API by actions.While our model uses web browsing to access in public big data, to enhance accuracy and reliability, we employed GoogleMap API from SerpAPI to find exact information of address, POI category and coordinates.and then to answer the same questions as those asked in the zero-shot prompts.

Evaluation
This comparison allows us to find ChatGPT's effectiveness and develop prompting guidelines, providing insights for to effectively use this tool in analysing social media text data.Since this model has two phases, NER and POI information search, we also evaluate in two phases.First, we evaluate about our model's NER performance with expert's, NER model's, and zero-shot prompting's results.Second, we will evaluate the accuracy of searched POI information with expert's and zeroshot prompting's results.Since NER model does not provides information of POI, it cannot be used for evaluating the entire process of this model and can only be utilized for assessing the first phase, NER part.We employed accuracy, precision, recall and F1-score for the evaluation metrics.

Expert :
To validate and assess our POI GPT's POI extraction and searching information performance, we asked two POI experts to extract POI from travel blog text data and search information about each POI.This process mirrors the operations conducted by our model, POI GPT.

Dates
Country Contents

2023-11-06 United-States
We have recently returned from an amazing trip to Oahu and Big IslandWe booked our flights with British Airways from Manchester via Heathrow and Seattle.… .A very Comfortable room with a kitchenette as we knew Hawaii was expensive!

2024-01-23 Spain
It was nice coming back to Tarifa.It would be a nice place to spend the wi ter.I stayed two nights to give me a chance to cycle to La Janda.I was hoping to see back with get kites again.No luck, but I did get decent views of red legged partridge.

2019-04-01 United-Arab-Emirates
The Burj Khalifa is currently the world's tallest building at over 800 metres.We too the brave step to firstly spend money and then to soar to level 148 for a sunset view over Dubai.… only to be met by an influx of people coming back from the dancing fountains.location, organization etc(Li et al., 2020).While NER is a traditional task of NLP, we adopted NER model to evaluate our model's NER performance.We used WikiNEuRal-Multilingual NER model from Tedeschi et al.(2021).WikiNEuRal is neural BERT-based model and shown substantial performance improvements over previous state-of-the-art approaches.To solely identify POIs, we extracted only the 'LOC(Location)' tags from among the various tags available.

Zero-shot prompting :
We initially used zero-shot prompts with ChatGPT, which means giving prompts without specific instructions or a data set.This is the most widely used method (Fu et al., 2024) and can also represent the generalization of ChatGPT.We asked ChatGPT to "extract the POI in this text and find each POI's address, latitude, longitude, POI classification."The outputs were typically in a consistent format.

Result Interface
Our model POI GPT implements an interface using GPTs.An example of our model implementation is shown in Figure 3. Specifically, when the raw text data of a travel blog is input as a query, it automatically undergoes preprocessing, and if necessary, calls an API to provide the Place Name, Address, POI classification, Latitude, and Longitude in a table format.Additionally, our model mentions the Lab's website at the end of the response.

Evaluation on extracting POI
Initially, we conducted an evaluation of POI extraction.Our results, as sown in Table 3, confirm that our model, POI GPT, exhibited the best performance.The evaluation benchmark was based on Expert's POI extraction, focusing solely on locations that the author of the text actually visited.As a result, NER models that recognized all place names showed the lowest accuracy.This was also due to the extraction of incorrect POIs, often inaccurately recognizing terms like 'North' and 'South'.These are considered limitations of the NER model and highlight the need for utilizing GPT.In the case of zero-shot prompting, our model, POI GPT, showed high accuracy with only a slight difference.

Evaluation on POI information accuracy
Next, we evaluated the second phase, which involves the collection of information for each POI.The results are presented in Table 4 and indicate that our model, POI GPT, consistently demonstrated the best performance.The evaluation benchmark was based on POI information directly collected by an expert using Google Maps, including address, latitude, longitude, and POI classification.The assessment was conducted only on POIs present in all three models, total 72 POIs.We verified the accuracy of information among the models through manual review.Although the differences were slight, our model, POI GPT, showed higher accuracy compared to zero-prompting.The overall lower accuracy can be attributed to the small sample size and challenges in clearly identifying addresses when POIs were closely located or involved broad districts.

DISCUSSION
After reviewing the literature on information extraction and the application of LLMs in urban studies, we identified two major gaps and developed a LLM-based model named POI GPT to overcome them.Our model extracts POIs from raw social media text data and collects location information for each POI, integrating functionalities of existing models while also providing a user-friendly interface.Given the ongoing increase in online activities and the availability of online text big data post-COVID-19, efforts to utilize this resource should continue, aligning with our research's focus on integration and convergence.
This study advances beyond previous research by combining POI extraction and information collection functionalities, demonstrating higher accuracy and usability compared to other models.However, there are limitations as the study used a small sample, and it does not fully represent the entire blog text.Additionally, the text data distribution was limited to Asia and Europe, not considering other continents.Future work should aim to acquire extensive blog data from all continents for further analysis.Moreover, while our study has improved interface and visual-accessibility limitations of previous research by using GPTs with a user-friendly interface, it faces challenges in processing large volumes of data simultaneously and only accessible to paid users of ChatGPT.Future research will endeavor to enhance user accessibility by simultaneously considering APIs.

Table 1 .
Example of travel blog text data acquired The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVIII-4/W10-2024 8th International Conference on Smart Data and Smart Cities (SDSC), 4-7 June 2024, Athens, Greece

Table 2 .
Example outcomes of POI GPT

Table 4 .
Evaluation results of POI information accuracy