#Raster of guk 64 Bit#
#Raster of guk generator#

I have also download the 64 bit version of Java and increased the memory allocation for Screaming Frog to 12GB (default limit is 512mb) - here's how - (look at the section Increasing Memory on Windows 32 & 64-bit)

I agree with Moosa and Danny - in terms of I use Screaming Frog (full paid version) on a stripped down windows machine with an SSD and 16GB of performance RAM. But still, how will the program know not to crawl and download the same urls, on all the files? In general, I would like to ask for better explanation, on how this needs to be done. I actually got to understanding what you mean, get 8 separate files (can be 6 or, lets say 10) and run them all at the same time. I also dont see how this could do this large website in one day, or lets say even five days. Should I wait from this point, then stop the program, and divide the file into 8 separate files, and load it to the program separately? Then the program will recognize these separate files as one, and it will continue crawling for new urls? If possible, please give better information on how this would need to be done, as I dont fully understand. I am not sure, based on your advice, how I could speed it up this process. I am manually saving it to a file, every now and then, as there is no way to auto save, as far as I was checking (there could be though, I am not sure, there is no too many options there). I've been running Xenu on a 33,500,000 pages site for a little over 4 hours and 15 minutes, and I have something like this, so far:Ĭlose to 500,000 urls recognized, and only 115,000 processed, it looks like. It is good to hear, that there is a way to do, of what I am trying to do, especially on 50 or more sites, large. I've crawled close to that in a day for a scrape once. If you really paid attention to it and created smaller files but ran them more frequently, you could get 4-5 million, I think. Doing it this way, I think you could expect to crawl about 2-3 million URLs a day. Recrawl those files in Xenu with the 'file' option.īuild them back up to 800k or so recognized URLs again and repeat.Īfter a few (4-6) iterations of this, you'll have most URLs crawled on most sites no matter how large. (Export, put the links in as an html file and start over.) Let me break it down the best I can:Ĭrawl your main (seed) URL until you've recognized 800k.Ĭreate an html file with the URLs from the export - separated 50k to 100k at a time. There's no way to keep it from re-crawling some of the URLs, as far as I know.īut yes, get it to recognize 600-800k URLs and then split the file. You will crawl some of the same URLs - that's why you remove duplicates at the end. The method that you are suggesting is not perfect, but I dont have two months to wait too, obviously. I will now how the number of urls compares to the 33,500,000 figure, obviously, but whats indexed in Google is not necessarily the complete website too. I dont have too much time for experimenting with it too. I will try your method, and see what I will get. Definitely not something that can be done, and I would say that it should be possible, software-wise.

And besides that I was planning to do it on up to 100 sites. This is close to two months of running the program 24 hours a day, which does not make sense. This means, that it would take 1340 hours to do the entire site. I've been running the tool for close to 5 hours, and I have around 125,000 urls, so far. This is one of the parts of working on the sites (I've never needed it, but I am working on something like this now), and there is no good tool, which would do the work. I may have 80 or even less percent of the total site, and not know about it, I would assume. So there is probably no way to tell, whether I have all the urls of the site, or what percentage I have. So, in general, what is the best way to work on something like this, also time efficient. Seo Spider from Screaming Frog is not good for large websites. I know about also, but this one is paid, and it would be very expensive with this amount of pages and websites too (5 million ulrs is $1750 per month, I could get a better deal on multiple websites, but this obvioulsy does not make sense to me, it needs to be free, more or less). I know that Scrapebox can scrape title tags from list of url, but this is not needed, since this comes with both of the above mentioned tools. So basically, the second one looks like it wont be good for websites of this size.

#Raster of guk generator#

Sitemap Generator from seems to be working too, but it starts handing up, when a sitemap file, the tools is working on,becomes too large. Xenu's Link Sleuth seems to be the best option for this, at this point. I am working on scraping title tags from websites with 1-5 million pages.