BlogIdeas
BlogIdeas is a SEO tool which helps bloggers find ideas for a blog-post using the "Alphabet Soup" Google search technique. It relies heavily on scrapping Google's autosuggestion box automatically putting letters from alphabet to expand on the original keyword. This technique is used to find keywords which are searched by real users.
Project Purpose and Goals
In this project I needed to work on both backend and frontend. The backend was needed to scrap things like search engine results, Google trends, and create a blog post planner. To make it easier to work on both of these things, I decided to keep the project's code in one monorepo using NX.
To lazily load the scrapped data into Redux in real-time, I used Socket.io. Another challenge was persisting user-data after the webapp is reloaded. To accomplish that, I used the localStorage.
Technology Stack
In order to scrap keywords from Google, I used Playwright and Cheerio. When it comes to backend Node.js app, I prefer using Nest.js because it uses TypeScript by default and decorators similar to Angular's. For the web-frontend, unless I have to use something different, I always use React and Redux.
React hooks, Redux, and local storage is used to persist data whilst Websockets are used to send and receive data from the server.
Problems and Thought Process
Whenever I work on a project where I need to work both on frontend and backend code, I usually choose to keep the code in a mono-repo. The easiest tool right now for this is NX. If a project's code-base is medium to big, planning how it should be structured is as important as the code itself.
One of the biggest challenges of this project was keeping the server from being flooded with scrap-requests. To solve this problem, I used a library for task-scheduling and rate limiting called Bottleneck.
Lessons Learned
There were many skills I refreshed doing this project. But the biggest lesson I learned was the importance of task-scheduling and rate limiting heavy user requests. It makes a lot of sense to schedule non-mission critical heavy tasks for later.