Problems, need help? Have a tip or advice? Post it here.
7 posts Page 1 of 1
Hi folks!
I'm trying to import a quite large CSV file - there are 34000 rows - via CSV importer module, but a lot of duplicate rows appear in database. What I'm doing wrong, or how can I get only one record / CSV row? Thanks a lot!

P.S. I'm using paginate='1' , limit='100' and use_cache='1' as csv_reader parameters.

Best regards,
Pandele
Hi,

The expected behavior using the CSV importer is to have only as many records (pages) as there are rows in the CSV file (assuming you are indeed creating one record per CSV row).

To validate this, instead of creating records, write the values as simple lines in a text file (using <cms:write> -https://www.couchcms.com/forum/viewtopic.php?f=8&t=11377) or even recreate a new CSV with the same values (viewtopic.php?f=8&t=11378).

The resulting output should have the same number of rows as the source CSV.
Please check and let me know.
Hi KK and thank you for your fast response!

I tried to write the results in a file, with the help of cms:write, and unfortunately there are also duplicates in there. :cry:
Well, that is unexpected :(
Could you PM me your CSV along with your script please?
Allow me see it for myself.
Thanks for sending me the files.

I tested by using the CSV importer to loop through your source CSV and use the output to create another CSV.
The result was a binary identical file.

So, nothing wrong with the CSV reader that would cause duplicate records.

As to why then you are getting duplicate records?
One reason (and I actually ran into it in my testing) is that the script gets called twice (for each set of paginated records) because the generated URL used for navigating to the next set of records does not match the one expected by Couch and so Couch redirects it to the correct URL.

This happened to me because my setup was using 'multilingual' addon and the URLs expected a 'lc' param which was absent in the script and Couch corrected this each time by using redirection. There could be some other reason in your case that you'll need to figure out. Use the dev console of your browser to see if a redirection is involved while the CSV importer is in use.

Hope this helps.
Thank you! You were absolutely right, I am using multilingual addon, but completely overlooked the lc=<language code> part from JS pagination piece of code! :roll: After making the necessary adjustments, it works flawlessly. So, for all interested, the CSV Importer pagination - as in example here https://www.couchcms.com/forum/viewtopic.php?f=5&t=8803 - should be written as:
Code: Select all
...
var myVar;
myVar = window.setTimeout( 'location.href="<cms:show k_paginate_link_next />&lc=en";', 1000 );
...

where en is one of the language codes you are using, or get it dynamically via <cms:show k_lang />.

Best regards!
I am glad I could help :)

Mentioning for documentation sake - I used <cms:show_with_lc> tag to get the link with correct language code
Code: Select all
..
var myVar;
myVar = window.setTimeout( 'location.href="<cms:show_with_lc k_paginate_link_next />";', 100 );
..

Also, again for documentation, following is the code I used to create a duplicate of the input CSV through the importer (might come in handy to someone wishing to check the fidelity of the import process) -
Code: Select all
<cms:set mystart="<cms:gpc 'import' method='get' />" />
<cms:if mystart >
    <cms:set my_delimiter=',' />
    <cms:set my_file='output.csv' />


    <cms:csv_reader
        file="<cms:show k_site_path />input.csv"
        paginate='1'
        limit='1000'
        delimiter=my_delimiter
        startcount='2'
    >
       
        <cms:if k_paginated_top >
            <cms:if k_current_page='1'>
                <!-- Header. 'truncate' starts a new file -->
                <cms:write my_file add_newline='1' truncate='1'><cms:csv_headers><cms:show value /><cms:if k_count ne k_total_records><cms:show my_delimiter /></cms:if></cms:csv_headers></cms:write>
            </cms:if>
           
            <cms:if k_paginate_link_next >
                <script language="JavaScript" type="text/javascript">
                    var myVar;
                    myVar = window.setTimeout( 'location.href="<cms:show_with_lc k_paginate_link_next />";', 100 );
                </script>
                <button onclick="clearTimeout(myVar);">Stop</button>
            <cms:else />
                <cms:set write_footer='1' />
                Done!   
            </cms:if>
           
            <h3><cms:show k_current_page /> / <cms:show k_total_pages /> pages (Total <cms:show k_total_records /> records. Showing <cms:show k_paginate_limit /> records per page)</h3>
        </cms:if>
       
        <h3><cms:show k_current_record /></h3>
                   
        <!-- CSV row -->
        <cms:write my_file add_newline='1'><cms:csv_columns><cms:format_csv value/><cms:if k_count ne k_total_records><cms:show my_delimiter /></cms:if></cms:csv_columns></cms:write>
       
        <cms:if k_paginated_bottom >
            <hr>
           
            <!-- Footer -->
            <cms:if write_footer>
                <!-- CSV does not require a footer so doing nothing here but for XML this could be used to output the document closing tags -->
            <cms:else />
                <cms:ignore><cms:paginator simple='1' /></cms:ignore>
                <cms:if k_paginate_link_prev >
                    <a href="<cms:show_with_lc k_paginate_link_prev />">prev</a>
                </cms:if>
                <cms:if k_paginate_link_next >
                    <a href="<cms:show_with_lc k_paginate_link_next />">next</a>
                </cms:if>
            </cms:if>   
        </cms:if>
       
    </cms:csv_reader>
<cms:else/>
    <button onclick='location.href="<cms:add_querystring k_page_link 'import=1' />"'>Start!</button>
</cms:if>
7 posts Page 1 of 1