Why doesn't MapServer recognize shp file created from ogrtindex?

Question

I am trying to create a MapServer layer from tif pyramid but always facing the same error :

msDrawMap(): Image handling error. Failed to draw layer named 'ign_2020'.; msDrawRasterLayerLow(): Unable to access file. Corrupt, empty or missing file '/srv/shapes/ign_2020/PLANIGN08_TIF_LAMB93_D074/dalles.shp,0' for layer 'ign_2020'. /srv/shapes/ign_2020/PLANIGN08_TIF_LAMB93_D074/dalles.sh,0: No such file or directory

Here is how my /srv/shapes directory looks like :

.
`-- ign_2020
    |-- PLANIGN08_TIF_LAMB93_D074
    |   |-- PLANIGN08_0840_6550_L93.tab
    |   |-- PLANIGN08_0840_6550_L93.tif
    |   |-- PLANIGN08_0840_6750_L93.tab
    |   |-- PLANIGN08_0840_6750_L93.tif
    |   |-- dalles.dbf
    |   |-- dalles.prj
    |   |-- dalles.shp
    |   `-- dalles.shx
    |-- PLANIGN09_TIF_LAMB93_D074
    |   |-- PLANIGN09_0840_6550_L93.tab
    |   |-- PLANIGN09_0840_6550_L93.tif
    |   |-- PLANIGN09_0840_6750_L93.tab
    |   |-- PLANIGN09_0840_6750_L93.tif
    |   |-- dalles.dbf
    |   |-- dalles.prj
    |   |-- dalles.shp
    |   `-- dalles.shx
    |-- PLANIGN10_TIF_LAMB93_D074
    |   |-- PLANIGN10_0840_6550_L93.tab
    |   |-- PLANIGN10_0840_6550_L93.tif
    |   |-- PLANIGN10_0840_6750_L93.tab
    |   |-- PLANIGN10_0840_6750_L93.tif
    |   |-- dalles.dbf
    |   |-- dalles.prj
    |   |-- dalles.shp
    |   `-- dalles.shx
    |-- tileindex.dbf
    |-- tileindex.prj
    |-- tileindex.shp
    `-- tileindex.shx

And many other directories like that named PLANIGN##_TIF_LAMB93_D074 where ## is a layer of the tiff pyramid. Now, according to this page, the command i ran to generate every dalles.shp :

for d in /srv/shapes/ign_2020/PLAN*; do gdaltindex $d/dalles.shp $d/*.tif; done

And the one i used to create the main tileindex.shp into ign_2020 directory :

ogrtindex srv/shapes/ign_2020/tileindex.shp /srv/shapes/ign_2020/*/*.shp

Note that i only use absolute path to avoid MapServer to use a random relative path. Finally my MapServer version and the layer file associated :

MapServer version 7.4.0 OUTPUT=PNG OUTPUT=JPEG SUPPORTS=PROJ SUPPORTS=AGG SUPPORTS=FREETYPE SUPPORTS=CAIRO SUPPORTS=ICONV SUPPORTS=XMP SUPPORTS=FRIBIDI SUPPORTS=WMS_SERVER SUPPORTS=WMS_CLIENT SUPPORTS=WFS_SERVER SUPPORTS=WFS_CLIENT SUPPORTS=WCS_SERVER SUPPORTS=FASTCGI SUPPORTS=GEOS SUPPORTS=POINT_Z_M INPUT=JPEG INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE

LAYER
    NAME                    "ign_2020"
    TYPE                    RASTER
    TILEINDEX               "ign_2020/tileindex.shp"
    TILEITEM                "LOCATION"
END

So MapServer finds tileindex.shp but then it looks like ogrtindex added a ,0 after path of every dalles.shp files as you can see in the error message. Did i miss something ?

Answer 1

To clarify: ogrtindex is for indexing vectors, and gdaltindex is for indexing rasters in MapServer. Therefore here are some possible steps for you:

1. Let's assume you have the following directory structure:

   - mainfolder
     - data
       - ign_2020
         - PLANIGN08_TIF
         - PLANIGN09_TIF
         ...
     - tileindex.map
   

2. Generate your tileindex for the .tif files. Because you have various subfolders with data, we can use the --optfile switch for gdaltindex to pass a list of files to use. Here are the commands (for Windows and Unix):

   cd data
   #windows (create a file listing all .tif in subfolders)
   dir /s/b *.tif > tif_list.txt
   #unix
   find . -name "*.tif" > tif_list.txt
   

   Because we should always use relative paths, open that 'tif_list.txt' in a text editor, and do a search-and-replace for the path upto data, so instead of the full path such as:

   D:\mainfolder\data\ign_2020\PLANIGN08_TIF\PLANIGN08_0840_6550_L93.tif
   D:\mainfolder\data\ign_2020\PLANIGN08_TIF\PLANIGN08_0840_6750_L93.tif
   D:\mainfolder\data\ign_2020\xxxxxx1\xxxx1.tif
   D:\mainfolder\data\ign_2020\xxxxxx1\xxxx2.tif
   

   it lists this instead:

   ign_2020\PLANIGN08_TIF\PLANIGN08_0840_6550_L93.tif
   ign_2020\PLANIGN08_TIF\PLANIGN08_0840_6750_L93.tif
   ign_2020\xxxxxx1\xxxx1.tif
   ign_2020\xxxxxx1\xxxx2.tif
   

   Now you are ready to generate the index:

   gdaltindex gdaltindex.shp --optfile tif_list.txt
   

3. load that gdaltindex.shp file into QGIS and look at its attribute table, the 'location' field will contain those nice relative paths.

4. #protip: you can run that index file through shptree so that MapServer finds the feature/record fast:

   shptree gdaltindex.shp
   

5. Now you are ready to include that layer in your mapfile. Remember your mapfile lives in the 'mainfolder', above the 'data' folder. Here is an example mapfile:

   MAP
     NAME "tileindex-map"
     STATUS ON
     SIZE 600 400
     SYMBOLSET "../etc/symbols.txt"
     #output extents
     EXTENT -99.99 43.13 -86.70 49.60
     UNITS DD
     SHAPEPATH "./data"
     IMAGECOLOR 255 255 255
     FONTSET "../etc/fonts.txt"
     #output projection
     PROJECTION 
       "init=epsg:4269"
     END # projection
   
     /* your index layer */
     LAYER
       NAME "ortho"
       TYPE RASTER
       STATUS ON
       TILEITEM "LOCATION"
       TILEINDEX "gdaltindex.shp"
       #source projection
       PROJECTION
         "init=epsg:26915"
       END # projection
     END # layer
   
   END # Map File
   

   Notice the SHAPEPATH points to the data folder, so, when MapServer finds that index record, it uses a path of SHAPEPATH/LOCATION relative to the mapfile location, such as

   /data/ign_2020/PLANIGN08_TIF/PLANIGN08_0840_6550_L93.tif
   

6. Finally, test your mapfile with shp2img, to generate a map image, such as:

   #make sure you cd into /mainfolder (that's where you mapfile lives)
   shp2img -m tileindex.map -o ttt.png -map_debug 3
   

   You should see output listing the draw speeds, and a file 'ttt.png' should be created in your 'mainfolder'

I hope this helps. I also today added all of this (as 2 full examples, with images, for ogrtindex and gdaltindex) to the main tileindex document at https://mapserver.org/optimization/tileindex.html#long-ogrtindex-example

Thanks,

 Jeff McKenna
 GatewayGeo
 @mapserving