Properly determine CSV delimiter (#17459)

* Fixes #16558 CSV delimiter determiner

* Fixes #16558 - properly determine CSV delmiiter

* Moves quoteString to a new function

* Adds big test with lots of commas for tab delimited csv

* Adds comments

* Shortens the text of the test

* Removes single quotes from regexp as only double quotes need to be searched

* Fixes spelling

* Fixes check of length as it probalby will only be 1e4, not greater

* Makes sample size a const, properly removes truncated line

* Makes sample size a const, properly removes truncated line

* Fixes comment

* Fixes comment

* tests for FormatError() function

* Adds logic to find the limiter before or after a quoted value

* Simplifies regex

* Error tests

* Error tests

* Update modules/csv/csv.go

Co-authored-by: delvh <[email protected]>

* Update modules/csv/csv.go

Co-authored-by: delvh <[email protected]>

* Adds comments

* Update modules/csv/csv.go

Co-authored-by: delvh <[email protected]>

Co-authored-by: wxiaoguang <[email protected]>
Co-authored-by: zeripath <[email protected]>
Co-authored-by: delvh <[email protected]>

This commit is contained in:

Richard Mahn

2021-10-30 23:50:40 +08:00

committed by

GitHub

co-authored by

wxiaoguang

zeripath

delvh

parent 63c0dc89ef

commit 40c8451b7d

5 changed files with 642 additions and 94 deletions

									
										modules/markup/csv/csv.go
									
		+1
		-1
	
												View File
												
				@@ -102,7 +102,7 @@ func (Renderer) Render(ctx *markup.RenderContext, input io.Reader, output io.Wri

						return err

					}

					rd, err := csv.CreateReaderAndGuessDelimiter(bytes.NewReader(rawBytes))

					rd, err := csv.CreateReaderAndDetermineDelimiter(ctx, bytes.NewReader(rawBytes))

					if err != nil {

						return err

					}