datafile is here:
http://ia341329.us.archive.org/1/items/VitoriaOpenstreetmapData/Planimetria_de_Vitoria-polygons.osm.bz2

Ahh,
Ok. Well the input and output are just

x=358376.5
y=7753537

outputs are :
ry=-20.3106
rx=-40.3564

see the main routine...
http://github.com/h4ck3rm1k3/TwoNickels/blob/master/proj/proj.cpp

it is all working now, so dont worry about it.

Sorry for the confusion.

MText is the new part of the Dime (two nickels)
http://github.com/h4ck3rm1k3/TwoNickels/blob/master/src/entities/MText.cpp

I have made a simple proj tool in dime as well here, but I forget to add that to git.
http://github.com/h4ck3rm1k3/TwoNickels/tree/master/proj/

On of my problems that was in the old code was the radians, I had converted the northing and easting to radians before converting. Now I do that after I get the results.

In fact, It works with the standard proj lib now, after I made the needed changes. Here is the current code :
So I dont *need* the proj patches, all the better.

Still, the code changes to proj are good, there were alot of bad typecasts, removing the constant from chars etc.

Ideally we could have a set of templates, one for each projection and do the code inline. It would be much faster.

to answer your questions : I had posted the parameters and results here with the code :

The new code using the standard proj interface :

void convertPoint(double x, double y, double & rx, double & ry) 

{

  projPJ fromProj, toProj;

  if (!(fromProj = pj_init_plus("+proj=utm +south +ellps=intl +zone=24K +units=m -f \"%.7f\" ")) )

    exit(1);

  if (!(toProj = pj_latlong_from_proj( fromProj )))

    exit(1);

  int islatlon= pj_is_latlong( fromProj );

  int isgeocent=pj_is_geocent( fromProj );

  char * name = pj_get_def( fromProj , 0);

  std::cerr  << "FROm NAME:"<< name << "|" << islatlon << "|" << isgeocent << std::endl;


  islatlon= pj_is_latlong( toProj );

  isgeocent=pj_is_geocent( toProj );

  name = pj_get_def( toProj , 0);

  std::cerr  << "TO NAME:"<< name << "|" << islatlon << "|" << isgeocent << std::endl;

  std::cerr.precision(10);

  std::cerr.width(10);

  std::cerr << x  << "|" << y  << std::endl;

  //  const double D2R =DEG_TO_RAD;

  double ax = x;// * DEG_TO_RAD;

  double ay = y;// * DEG_TO_RAD ;

  double az = 0;

  

  /** end of "caution" section. */

  std::cerr.precision(10);

  std::cerr.width(10);

  std::cerr << ax  << "|" << ay  << "|" << std::endl;

  int ret = pj_transform(fromProj, toProj,1,1, &ax, &ay, &az);

 

  std::cerr.precision(10);

  std::cerr.width(10); 

  std::cerr << ax  << "|" << ay  << "|" << ret  << "|" << std::endl;

  

  ax *= RAD_TO_DEG;

  ay *= RAD_TO_DEG;

  std::cerr << ax  << "|" << ay  << "|" << ret  << "|" << std::endl;

  rx = ay;

  ry = ax;

    

  pj_free(fromProj); 

  pj_free(toProj); 

}

the outputs :
FROm NAME: +proj=utm +south +ellps=intl +zone=24K +units=m|0|0
TO NAME: +proj=latlong +ellps=intl|1|0
358376.5|7753537
358376.5|7753537|
-0.7043522911|-0.3544868135|0|
-40.35641357|-20.31059831|0|
-20.3106:-40.3564

the convert point goes from a UTM northing/easting to lat/lon.
mike

I have added the first results of the dxf-osm converted,
the points with names are created.
http://www.archive.org/details/VitoriaOpenstreetmapData

code checked in
http://github.com/h4ck3rm1k3/ProjC--
http://github.com/h4ck3rm1k3/TwoNickels

More to come.

mike

Data File:
http://filebin.ca/ewytu/test.osm

Memory leaks using xerces transcode. It was making copies of all the strings. Have replaced/removed all the unneeded copies of the strings for comparison and parsing of ints.

I have used valgrind to debug the memory leaks. The problems are removed. Also I turned off the reverse lookup by the string of the Point and removed it from the memory representation. Will look into using a rtree or a quadtree for that later.

mike

Now, to be fair, the osm2pgsql does process the osm files in a very similar way, but it is not intended on being a generic processor.

I am working on processing a much larger file now, the entire state file and also have started looking into the bzip2 processing inline and the sax reader.

We should be able to fetch just parts of a world.osm.bz2 file and process it while downloading (using the blocks as they complete)

But that is for the future, for now I will focus on the county processing.

Well, here are the results of wordcount on the uncompressed NJ file: 18 million nodes, counted in 1.5 minutes.

wget http://downloads.cloudmade.com/north_america/united_states/new_jersey/new_jersey.osm.bz2
bunzip2 new_jersey.osm.bz2

time wc new_jersey.osm
18612601 78918048 1249629139 new_jersey.osm
real 1m21.038s

One thing that I have observed, the processing takes up the entire processor, but only one of four. That is why we need these splitting routines in general so that we can process on mutiple processors easily. Osmosis is nice, but I dont feel confortable with using it, it is pretty complex. Ideally

Now, just running my county extractor on that takes a long time. I need to find out why....

mike

I have checked in the Makefile, using gcc -O4, have now the bounding box calculation and refactored the classes.
The time make test produces:
real 0m5.927s

here are my system details :
Ubuntu 9.10
ProblemType: Bug
Architecture: i386
Date: Fri Dec 25 10:52:00 2009
DistroRelease: Ubuntu 9.10
InstallationMedia: Ubuntu-Netbook-Remix 9.10 "Karmic Koala" - Release i386 (20091028.4)
NonfreeKernelModules: nvidia
Package: gcc 4:4.4.1-1ubuntu2
ProcEnviron:
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-17.54-generic
SourcePackage: gcc-defaults
Tags: ubuntu-unr
Uname: Linux 2.6.31-17-generic i686

processor : 3
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 265
stepping : 2
cpu MHz : 1808.403
cache size : 1024 KB
physical id : 1
siblings : 2
core id : 1
cpu cores : 2
apicid : 3
initial apicid : 3
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni lahf_lm cmp_legacy
bogomips : 3616.92
clflush size : 64
power management: ts fid vid ttp

OK, I have a perl script to generate xml constants here:
http://bazaar.launchpad.net/%7Ejamesmikedupont/%2Bjunk/EPANatReg/annotate/head%3A/makenames.pl

A scheme file here:
http://bazaar.launchpad.net/%7Ejamesmikedupont/%2Bjunk/EPANatReg/annotate/head%3A/schema.txt

The latest version has a makefile and also I have generated a list of fields:
http://bazaar.launchpad.net/%7Ejamesmikedupont/%2Bjunk/EPANatReg/annotate/head%3A/OSMAttributes.h

This is just a first version, will need to put more work into creating an optimum recogniser for the schema. It should be possible to generate a lex like structure to process the rest.

but for now, I am doing switches based on the field names.

Now, this version looks up each node reference in the id -> coords table and also outputs the entire names database of the nodes, ways and relations.

it runs in 10 seconds on my computer with a larger version of the osm file with some duplicates where i tried to resolve the missing nodes in the extract file.
real 0m10.667s

for comparison, wordcount needs 5x less.
time wc lint.osm
393773 1974640 30893704 lint.osm
real 0m1.896s

So, it is still fast even though it is doing much more processing.
I think this is a real winner folks.

I am going to make some template classes for the processing of fields and defining structures... here is a start that I have not even compiled :
http://bazaar.launchpad.net/%7Ejamesmikedupont/%2Bjunk/EPANatReg/annotate/head%3A/Field.h

Yes, I am rewriting that perl script in c++ now,
In the end you will be able to define filters on what attributes you want to collect and the get them in a callback.

I dont want to collect any huge memory structure in the parser, the client should be able to do that.

mike

The upload is finished :
http://www.archive.org/details/NJ_Counties

Here is the second part!
http://www.archive.org/details/ZCTA_NJ2

You can see the difference between the zcta and the "zip codes"

Here are the ztcas :
http://maps.huge.info/zcta.htm

There are differences that you can see between the two versions.
I understand the panic better now.

But we have to start somewhere!

The first part is finished uploading :
http://ia341335.us.archive.org/2/items/ZCTA_NJ/
07001- 07878

I found a mashup that shows just want I am planning on doing,
http://maps.huge.info/zip.htm

Here are more infos on zipcodes :
http://en.wikipedia.org/wiki/ZIP_Code_Tabulation_Area
http://en.wikipedia.org/wiki/ZIP_code

So if anyone wants to add any information about them, do it there.

mike

http://zip4.usps.com/zip4/welcome.jsp Here is a nice tool to double check the zip code if there are any questions.

I have been playing with qgis, and it looks like there is a feature to create a convex hull based on an attribute value.

So, you could take these attribute values (post codes) and create a convex hull and then compare this to the ZCTA. That would give you a good start because you could compare the areas that have the biggest difference first.

The other thing is that you can flag the nodes and ways that are outside of the ZCTA, that is what I was doing to check them. Maybe other states have more problems with the zipcodes, but NJ looks very stable.

mike

I was just following the wiki,
personally I would like zipcode, but
here :osm.wiki/Key:postal_code

Yes, of course. In Germany I found power lines, security cameras and trees.
But we still need a better staging system. Why should we just throw it all into a single database. We could have many databases for various layers. This is a design issue. In fact, why do we need a monster database at all? Cannot we deal with lots of small files and a smart editor that commits them in the right way so that we don't need anything more than a smart distributed version control system?

I have hacked osm2pgsql so that it imports the data from my feeds :
http://fmtyewtk.blogspot.com/2009/12/osm2pgsql-hack-for-importing-id-ways.html

The data is loaded in qgis.

I will be creating some postgres queries to split up the data and process it. that is at least my plan.

I dont care if the monolithic OSM database stores this data or not. In fact, I think it would be better to keep it separate until we find a better way to add in layers.

Ideally the chunks of data will be usable directly from some GIT repository and we split them into very small but useful peices.

mike