search spiders show some interesting parts of Belgian sites

It was by looking for the statistics and demographics of the Belgian webspace that I saw that the webcrawlers indexed more than they ought to do and that some webmasters in Belgium have still learned nothing about robot.txt or didn't implement it effectively or didn't check it.


Because this for example shouldn't be seeable for searchengines and bots

dossiermanager.vdab.be and what autoscout24.be does is really inviting(http://data.autoscout24.be/)


site:assets.gva.be if you type that in Google you are in and can look at hundreds of pics and other things they call assets. I presume this wasn't the intention but for the time being this is so.


 site:cms.vtm.be  gives you the link to see all the different poll questions that are asked here


http://preview.standaard.be/  is where they place the testpages from destandaard, interesting for competitors to know and I hope security is not being forgotten and the financieel economische tijd yesterday can be read here;.


but sometimes this is even very practical like information about trains (so no homepage, just some practical links)


http://plannerint.b-rail.be/  This is an international belgian train time planner in different languages, much easier than on the homepage. Strangely enough it is called <HTML><HEAD><TITLE>HAFAS SITE</TITLE> or you can use this link. Prices you should find one way or another here and more info about it here. Information about incidents and work on the railways in Belgium can be found here directly.


oh yes, it will start again soon, googling the Belgian web space

13:54 Gepost door technology changes fast not a lot in Algemeen | Permalink | Commentaren (0) |  Facebook |

De commentaren zijn gesloten.