Services and scripts
Where to host a web scraper so it runs without your computer
In short
To host a web scraper you need a place that simply keeps the process running: a scraper is a background script, it has no public link and nobody opens it. Upload the code to Netrun and the platform installs the dependencies, starts the scraper and brings it back up after a crash, with no server to rent and no autostart to configure. The schedule has to live inside your own code, as a pause in a loop or a scheduling library, because there is no built-in cron on the platform. Write the collected data into the permanent /data folder or an external database, otherwise the files are recreated the next time you publish.
The scraper is written and it works: once an hour it walks a list of pages, puts prices into a table and sends the difference to a chat. It lives exactly as long as your computer stays on. Close the laptop, let it sleep, lose Wi-Fi, and the collection stops. You notice a week later, from a gap in the data. The usual answer to where to host a web scraper is dull: rent a server, install Python on it, set up autostart so the process returns after a reboot, and keep an eye on system updates.
There is a shorter path: hand the code to a platform that keeps the process running for you. Netrun takes the folder of code, works out the language, installs dependencies, starts the scraper and brings it back after a crash. The important part is how a scraper differs from a website: it has no public link and nobody opens it. That leads to three things worth deciding up front, which are the schedule, the place for collected data, and the limit of the free plan. If your scraper drives a real browser through Selenium or Playwright, that is a separate case with its own guide.
Put the schedule inside your code#
There is no built-in cron or scheduler on the platform, and it helps to know that in advance: a script that runs once and exits counts as finished, and nothing will start it again an hour later. So the schedule has to live inside the scraper itself: an endless loop with a pause between runs, or a scheduling library such as schedule and APScheduler in Python, or node-cron in Node.js. The process then never exits and wakes itself up for the next pass.
Write results into permanent storage#
Files the scraper writes next to the code live only until the next publish: on a code update the project is built again, so a CSV table or a SQLite file is created from scratch. Anything that should accumulate belongs in the permanent /data folder of the project, which survives both code updates and restarts, and the Data tab in your dashboard lists the files so you can download them. If there is a lot of data, or another app reads it, use a database: run one as a second service in the same project with your own docker-compose file, or connect an external one. Projects that bring their own docker-compose file have no /data folder, because there the permanent storage is the named volumes from that file.
Move keys and logins out of the code#
Read API keys, the login and password for a third-party service, and database connection strings from environment variables rather than from strings in the code. During project setup Netrun asks for these values and passes them in encrypted, so they never sit in the archive or in the repository. Changing them later is easier too: you edit the value in your dashboard instead of hunting through files.
Slow down between requests#
A scraper with no pauses hits a third-party site with dozens of requests per second, which is the fastest way to get blocked and to annoy the owner. Add a delay between requests, do not fetch the same pages more often than they change, and check what the site allows in robots.txt and its terms of use. A calm scraper that walks its list once an hour lives far longer than a greedy one.
Upload the code and check the first pass in the logs#
Bring the project in as a ZIP archive or import a repository from GitHub, including a private one. Netrun detects the language, installs dependencies and starts the scraper, so there is no Python to install on a server and no autostart to write. A scraper has no public link and should not have one, so watch the logs in your dashboard instead: the output is there in real time and shows that the first pass ran and the data was written. If the scraper crashes, the platform brings it back and sends you a notification.
Decide what happens after three days on the free plan#
A scraper has no public link, so it cannot fall asleep when idle and wake up on request the way a website does: it is busy all the time. On the free plan a project like this gets three days of work and then shuts down, with a warning a day before. Code, settings and keys stay in place, and three days is enough to confirm that the scraper builds, runs on schedule and writes data where it should. To run a scraper 24/7 you need the Pro plan: the platform has no free around-the-clock option for background scripts, and it is fairer to say so up front.
A scraper that runs without your computer comes down to three decisions: the schedule inside the code, collected data in permanent storage or an external database, and keys in secrets. After that the platform keeps the process running and restarts it on failure, while you only drop in to read the logs and pick up the results. Try Netrun.
Common questions
Can I set up cron so the scraper runs once an hour?
There is no separate scheduler on the platform, so there is nowhere to configure cron. The schedule is written inside the scraper: an endless loop with a pause between runs, or a scheduling library such as schedule, APScheduler or node-cron. The script never exits and wakes itself up at the right moment.
Where should the scraped data go?
Into the permanent /data folder of the project: anything written there survives code updates and restarts, and the Data tab in your dashboard lists the files with downloads. Files written outside that folder are recreated the next time you publish. If there is a lot of data, or another app reads it, use a database instead: run it as a second service in the same project with your own docker-compose file, or connect an external one.
What do I do if the site blocks my scraper?
Start by slowing down: a pause between requests and a sensible crawl frequency remove most blocks. Check that you are not breaking the rules of the site and not fetching pages that are closed to you. The platform does not rotate addresses, so if the site blocks by address, plug a proxy service into your code and keep its key in secrets.
Do I need a browser and Selenium for scraping?
Usually not. If the page returns what you need in the HTML, a plain request plus an HTML parser is enough, and that kind of scraper is lighter, faster and more stable. A browser is only needed when the content is drawn by scripts on the page. The automatic build has no browser, so Selenium or Playwright requires your own Dockerfile that installs one.
How long does a scraper run on the free plan?
Three days, after which the project shuts down, with a warning sent a day earlier. Nothing is deleted: the code, settings and keys stay, and the project starts again once you move to Pro. The countdown belongs to the account rather than to each project, so replacing the scraper with a new one does not reset the timer.
Can I run a Python project — Django, Flask or FastAPI?
Yes. Put your dependencies in requirements.txt — Netrun detects Python and builds the project for you. Django, Flask and FastAPI are supported; the app should listen on the port from the environment variable, and keys and database access are set as secrets rather than in the code.
Do I need my own server or VPS?
No. Netrun deploys your project on its own servers. There is no VPS to rent, no systemd or nginx to configure and no certificates to renew — all of that runs for you.
What happens if my app crashes?
We watch over projects and restart them if they fail. If something goes wrong, you can see the status and logs in your account and you get a notification.
Where can I see the logs and status of my project?
The project page shows the status, the logs and the event history — together they show what is happening with the project right now. The log view also reaches further back: scroll up and earlier lines load on their own.