OpenStreetMap 開放街圖
How many days or weeks does it take to upload the whole planet via Osmosis ?
於 2015年九月15日 由 baditaflorin 以English發表。I am now in day 6 of waiting
There is a 16 GB RAM server, octacore environment, running ubuntu 14.10 ( don`t ask why )
I have a 317 GB file generated called copy********n
And there is a file called copy******w that is still being generated, growing by 10-20 GB per day , now is at 76 GB
And a file called copy*******wn at 30 GB
i am guessing that the n means nodes and w means ways, but how much should it grow, procentualy compared with the 317 GB n file ?
The command that i had used to give the command looks like this
sudo osmosis –rbf latest-planet.osm.pbf –wp host=localhost database=mydatabase user=*** password=****
討論
由 Warin61 於 2015年09月15日 06時57分 發表的評論
Your probably on the lowest priority. Why? Probably the reason for your load is unknown. Try contacting the administrator of the site, have a justification for your load on the site ready for them .. not here.
Do you really really need to whole planet? Already tested what you want to do with a small sample? These are the kind of questions I’d have answers ready.
由 baditaflorin 於 2015年09月15日 08時03分 發表的評論
I have access to the server, what should i change / could i change
I want the whole planet, so i can do some statistics for the whole planet, not just a country or a continent.
top - 11:00:23 up 62 days, 15:27, 7 users, load average: 2.03, 2.07, 2.10 Tasks: 305 total, 2 running, 280 sleeping, 23 stopped, 0 zombie %Cpu(s): 3.5 us, 9.6 sy, 0.0 ni, 73.7 id, 13.2 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 16341788 total, 16174748 used, 167040 free, 25816 buffers KiB Swap: 16684028 total, 16521344 used, 162684 free. 9697612 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21798 postgres 20 0 369660 190580 144392 R 0.0 1.2 16935:45 postgres 22127 postgres 20 0 295484 144512 144468 S 0.0 0.9 600:18.35 postgres
由 Jean-Marc Liotier 於 2015年09月15日 08時13分 發表的評論
sudo ? Why ?
由 baditaflorin 於 2015年09月15日 08時16分 發表的評論
@jean osmosis have a bug and it does not work to process a file and upload it to postgres without the sudo command.
由 Vincent de Phily 於 2015年09月15日 08時41分 發表的評論
I can’t imagine an osmosis bug that requires root to solve. Can you psql to the db ? Fix you pg_hba.conf if not. Can you write to temp files that osmisis creates ? Use chmod/chown if not.
16GB of ram is at the lower end of usable nowadays. And if you aren’t running this on an SSD, you should probably forget about importing the whole world, it’ll take months.
Always start with a smaller extract, not the whole world. Try Africa first, for example. You say you want to run stats, could you run stats on individual continent databases ?
The biggest speed improvement you can get is by osmfilter-ing out data you don’t need before the import. Do you really need building outlines for example ?
由 baditaflorin 於 2015年09月15日 08時52分 發表的評論
i want to do statistics about everything that is on the planet file, from every piece of the world.
I know that i could just import each continent and then do some join to stitch all together, but then whats the point of a planet file.
I already Have Africa loaded, some more countries loaded, but the aim of my statistics will be to show everything that is on the map, how it relates, what particularities exists in different parts of the world
由 baditaflorin 於 2015年09月15日 15時23分 發表的評論
I will kill the task and try 2 different version :
1 imposm3
the only information that found says that i still have to wait at least 7 more days http://ksmapper.blogspot.ro/2011/04/parse-planet-into-pgsnapshot-database.html
I think i got an answer, based on
If the copy*****n it`s 317 GB
What is the approximate size of copy****w 80 % ( 240~ GB ) copy*****wn 10-20 % (50-60 GB)
由 pnorman 於 2015年09月16日 01時18分 發表的評論
What area did you use for testing your command, and how long did it take?
Did you load in either the linestring or bbox columns?
What speed drives do you have?
由 species 於 2015年09月24日 06時56分 發表的評論
I see the load is only 2, which should be at least 8 on an octa-core. And your Mem and Swap are both 100% full… More interesting than “top” would be the output of “iotop” - I guess its your I/O that’s limiting you.
Do you have an SSD? Without that, forget importing the whole planet^^. Africa may has worked because the DB fitted (at least important parts) inside your 16GB RAM. You need either enough RAM (512G?) or an SSD for OSM planet imports :-)