AUTOMATING TWITTER DATA COLLECTION: A RAPIDMINER-BASED CRAWLING SOLUTION

Social media plays a crucial role in numerous aspects of human life. It is a medium for users to share information, communicate with each other, and share various types of content such as photos and videos. Social media is a digital platform that facilitates these activities (Auxier & Anderson, 2021; Putri, 2021; Rachmawaty, 2021). There are currently 2.8 billion active social media users, with Indonesia alone contributing more than 100 million active social media users (Fadly & Wantoro,2019; Yudhana, Riadi & Zuhriyanto, 2019). Depending on the type of social media they use, users fall into several categories. Social media is typically divided into five categories: social networks, such as Facebook and LinkedIn; microblogging, including Reddit and Twitter; photo sharing, such as Pinterest, Instagram, and Snapchat; video sharing, including TikTok, Twitch, Snackvideo, and Youtube; and instant messaging platforms, such as WhatsApp and Line. Currently, Twitter has 300 million social media users (Juditha, 2019; Al Asad et al., 2019; Mustafa et al., 2020). Interactions occur on Twitter for approximately 700 million tweets per day, with an average of 7,500 tweets per second (Furqon & Setiawan 2020). Twitter serves as a massive data warehouse, storing billions of data from around the world. If processed and analyzed effectively, the data collected from Twitter can provide valuable insights and information. However, before the data can be transformed into useful information, it must first be collected and extracted using techniques such as crawling. Users can leverage crawling techniques from the Internet to efficiently collect data from Twitter and prepare it for further analysis (Dakhi & Aji, 2022; Sulastomo et al., 2022; Pakpahan & Simanullang, 2022). In this study, the author has utilized a crawling technique to gather data from Twitter. The outcome of this study can be further utilized to analyze and extract valuable information. Specifically, the crawling technique utilized in this study involves the use of the RapidMiner application. This research has a main focus on how the implementation of data crawling techniques on Twitter social media, post-processing techniques or data processing that has been collected using crawling techniques is not discussed in this study. At the end of the study, testing will be carried out by crawling several keywords. Next, the results of crawling data will be displayed on several keywords that are currently the topic of chat or popular hashtags on Twitter on a certain day. The development of technology and social media, especially Twitter, has had a major impact on various aspects of people's lives. In the academic field, the data generated by Twitter can be a valuable source of information for research in different fields, such as sociology, psychology, political science, medical science, and others. Twitter is a massively popular social media platform with an enormous user base and an astonishing number of tweets produced every second. Therefore, researchers need to have efficient and effective way to collect Twitter data. One way to collect Twitter data is by doing data crawling. However, the process of collecting data manually can take a lot of time and effort. Therefore, research on automating Twitter data collection can provide an efficient and effective solution for collecting large amounts of Twitter data. Twitter is a commonly utilized social media platform where individuals can obtain information. Twitter has the advantage of displaying data in the form of sentences accompanied by pictures (Kurniawan & Maharani 2020; Pattiiha & Hendry, 2022). Twitter itself is a social media that has information in the form of sentences that can then be easily traced back by readers using the hashtag (#). Twitter, owned by the US-based company Twitter, Inc., is a social networking platform that relies on microblogging. Twitter users can interact with one another by posting tweets on their home page, and they can also create and join tweet communities. Additionally, Twitter allows users to exchange personal messages through chat (Murthy, 2018). Twitter users can post, like, and retweet tweets, Twitter can also be accessed without registering for an account, but users can only read public tweets without interacting with others. Users can interact with Twitter through a browser or mobile interface, or programmatically via its APIs (Chadijah, 2022; Seprina, 2019; Edwanto, 2019). Tweets are by default public and viewable, but users can limit who sees their tweets by making their account private, which restricts messages to their followers only. This can be done by enabling the account's privacy settings. Users have the option to mute other users they don't want to interact with, hide their tweets from the account, and remove the account from their follower list. Twitter is what is going on withinside the globally and what humans are speaking approximately proper now. Twitter can be accessed via applications (on mobile devices) and also through internet browsers. With the purposes of sharing information on Twitter each time viable, Programmatic access to Twitter is offered by Twitter to businesses, developers, and users through a Twitter Application Programming Interface (API). an API, which stands for application programming interface, facilitates communication between software programs by enabling them to exchange requests and data. APIs facilitate the interaction between two or more applications over the internet, functioning as a concept that enables software developers to leverage their programming skills in building new and innovative products (Karami & Collins, 2018; Lee & Kwon, 2018; Muri, Utomo & Sayyidati, 2019). API linking is done by allowing software applications to call an endpoint called: API endpoints that match a particular type of information and later provide an access point (endpoints are usually unique, like phone numbers). Twitter provides access to certain parts of Twitter services through APIs to allow users to create software that integrates with Twitter (Darmawan, 2020; Raharjo, 2021; Sanusi, 2020), such as solutions that help companies reply and feedback to customer comments on Twitter. The information available on Twitter stands out from data shared on other social media platforms as it portrays details that users have voluntarily made public. By using the Twitter API Platform, individuals can gain entry to a broad range of public Twitter data that is shared worldwide. Twitter APIs also support users in managing their privacy by allowing them to control information sources on Twitter, such as direct messages, and make it available to developers who are approved to access it. Web crawling begins with a list of URLs to access. The first URL on the list is known as the seed. When the crawler visits this URL, it communicates with the server and retrieves all the hyperlinks available on the web page. Afterward, it generates a collection of website links that need to be explored, known as the "crawl frontier," utilizing a set of criteria, based on a set of guidelines. The URLs that are visited are determined exclusively by the guidelines. If a user wants to archive a website, they can copy and save information during the crawling process. Archives are usually saved for viewing, reading, navigating, and browsing as if they were on the live web (Khder, 2021). Archives or repositories are specialized platforms created to manage and store collections of website pages. These repositories store individual HTML pages as separate files. Although they function similarly to modern databases in terms of storing and managing data, repositories do not always require the full range of features that a database system provides. Repositories store the latest versions of web pages retrieved by crawlers (Ang, et al., 2021; Pratama, 2018; Sooai & Laniwati, 2021). When dealing with a large amount of data, data crawlers must be selective in downloading web pages due to the limitation of resources. This means that they need to prioritize which pages to download first. Additionally, due to the high frequency of updates and deletions of web pages, it is important to ensure that the downloaded pages are current and accurate. Due to the vast number of URLs that can be generated by server-side software, web crawlers often struggle to avoid retrieving duplicate content. This is because there are countless combinations of HTTP GET parameters that may lead to identical content. For instance, an online photo archive may allow users to select from several options via the HTTP GET parameter in the URL, resulting in 48 different URLs that lead to the same set of images. As a result, crawlers must navigate through an enormous number of possible script changes to extract unique content, presenting a challenge (Rennhard et al., 2022; Yadav, 2013). RapidMiner is a software used for data science that was created by the company bearing the same name as the software, integrated and providing for environment data preparation, machine learning, deep learning, text mining, and predictive analytics. RapidMiner is a software platform that can be used for data processing activities until the data is finally processed into information (Hanif et al., 2018; Nofitri & Irawati, 2019; Sudarsono et al., 2021). RapidMiner is a versatile data science software that can serve multiple functions, including business and commerce, research, education, training, rapid prototyping, and application development. It covers all aspects of the machine learning process, such as data preparation, result visualization, model validation, and optimization (Hofmann & Klinkenberg, 2016). RapidMiner is developed on an open-core model. The RapidMiner Studio Free Edition is limited to one logical processor and 10,000 rows of data. RapidMiner uses a client/server model with servers provisioned on-premises or on public or private cloud infrastructure. RapidMiner provides advanced analytics solutions through a model-driven framework that speeds deployment and reduces errors by virtually eliminating the need for coding. RapidMiner is a versatile data science software created by a company that shares its name. It has a multitude of potential applications, including business, research, education, training, rapid prototyping, and application development. RapidMiner can handle all stages of the machine learning process, such as preparing data, validating models, visualizing results, and optimizing the final output. Alternatively, the tool can be called from another program or used as an API (Norris, 2013). RapidMiner offers command-line access to its functions, enabling users to call specific functions individually. Additionally, the software includes various learning schemas, models, and algorithms that can be leveraged, and it can also be extended with R and Python scripting for further functionality. Research on Twitter data collection automation with RapidMiner can make a significant contribution to the development of data collection methodologies in the academic field. This could open up opportunities for researchers to use similar methods to collect data from other social media platforms, as well as develop more efficient and effective tools for future data collection.

The research carried out to implement the crawling technique on Twitter social media uses the literature study method of library science and information science, utilizing related research as a reference in this study. This study also employs the RapidMiner application operating guidelines for Twitter as follows. Text mining aims to generate information from a set of documents. Text mining is capable of generating information through processing, grouping, and analyzing large amounts of unstructured data (Ruhyana, 2019; Samsir et al., 2021). Text mining is used to obtain useful information from a series of documents with data sources in the text that have an unstructured format. Picture. 1 Research Guidelines In the picture, the stages of the process carried out by researchers to implement the crawling technique are explained. The process starts with obtaining the Twitter API key and continues with the use of the Twitter library. Then, the API is integrated with RapidMiner, followed by connecting the program code with the Twitter Searching API. The last stage involves finding and saving the data that has been collected into a file or database.

Twitter API Making Picture. 2 Twitter API Making The first step in crawling data from Twitter is to obtain the Twitter API source, which will be used to retrieve or crawl data from the social media platform. The API will then be connected to the RapidMiner application using a token obtained from the API confirmation Twitter API Connecting Picture. 3 Twitter API Connecting Connecting the API with RapidMiner, at this stage the connection between the API and RapidMiner must be established, as it is crucial for the proper execution of the data crawling stage. The process of connecting the API with RapidMiner is the most important aspect to consider, as a series of data crawling processes depend on this step. Twitter Data Crawling Picture. 4 Twitter Data Crawling We can start data crawling by using the query feature. In this feature, we can enter search keywords for information or tweets on Twitter. Then, we can choose results based on the popularity of the tweet and/or the time the tweet was made. The limit is how much data we want to retrieve from Twitter. The maximum amount of data that Rapidminer can retrieve in one operation is 10,000. We can also search by language, location, and certain dates. Twitter Data Result Picture. 5 Twitter Data Result After running the query with the keyword 'ipusnas', we obtained 100 tweets containing the word 'ipusnas' from the Twitter search engine. The obtained data can be processed to become useful information, which can be analyzed for sentiment analysis, marketing, information literacy, or other purposes that can benefit from the data analysis

Twitter is one of the largest social media platforms in the world with over 300 million users, and 700 million tweets are sent per day, making it a valuable source of data for various research disciplines. Data crawling is an efficient and effective way to collect Twitter data, and research on automating Twitter data collection can provide a solution for collecting large amounts of Twitter data. Twitter users can post, like, and retweet tweets, and tweets are publicly viewable by default. Users can interact with Twitter via a browser, mobile application, or programmatically via its APIs. Overall, Twitter is a powerful social media platform that provides an extensive source of data for research and other purposes. In conclusion, the use of RapidMiner in crawling data on Twitter can provide researchers with a reliable and automated solution for collecting large amounts of structured data. With RapidMiner's efficient data collection and processing features, researchers can optimize their data analysis efforts and enhance their understanding of social media behavior. Additionally, this research could inspire further advancements in data collection methodologies, leading to new and innovative solutions for social media data collection and analysis.