Is there a program...

[LeetViet]

Is there a program that searches a plain txt file for duplicate entries?

Say like if I type 123 on one line and at the bottom type 123 it'll find it and delete it.

 

[AFB]

Sounds like a job for scripting and a map.

 

[LeetViet]

Originally posted by: amdfanboy
Sounds like a job for scripting and a map.

I have some txt files that I'd like to import to my MySQL database but Navicat messes up when there are duplicate entries and sometimes not all the entries are added (although it continues on).

 

[cjr22]

On linux, I'd use sort and uniq to tell me which lines were duplicated, and then use sed or something to trim out those lines. Post again if this is helpful but you need more explanation.

 

[LeetViet]

Trying to do this in Windows.

I have a .txt file that contains some duplicate entries

i.e.

1 Test
2 Test2
3 Test3
1 Test

I'm looking for a program that will find and delete the duplicates without me having to manually search through every one of the unique entries.

 

[AFB]

Each one is on a different line? I would do it, but I'm kind of crammed for time. I would just use a map or something to keep track of what lines I have read . I would read each line in, if it was already there , I wouldn't write it to the output file.

 

[mattg1981]

if you are on linux you could create a simple little script using sed or awk that could easily do it

 

[LeetViet]

Yes, they're on separate lines.

Is there anything for Windows? I'd like to learn to do this myself so I can be self reliant in the future.

 

[CTho9305]

get this and once it's set up, "cat foo.txt |sort |uniq > foo2.txt"

 

[LeetViet]

Originally posted by: CTho9305
get this and once it's set up, "cat foo.txt |sort |uniq > foo2.txt"

http://www.cygwin.com/setup.exe
Right?

 

[CTho9305]

Yup. If you don't want to install the whole thing, I can send you sort and uniq and the DLL, but for various reasons, you should use their installer .

 

[LeetViet]

Originally posted by: CTho9305
Yup. If you don't want to install the whole thing, I can send you sort and uniq and the DLL, but for various reasons, you should use their installer .

I'd actually prefer that, that's what I did for patch.exe and diff.exe

 

[CTho9305]

http://ctho.ath.cx/tmp/cygwin

 

[LeetViet]

Got it. Thanks!

So just go to command prompt with the files in the same DIR and type what yous aid?

 

[CTho9305]

Replace "cat" with "type" since I didn't provide cat.exe .

 

[LeetViet]

Originally posted by: CTho9305
Replace "cat" with "type" since I didn't provide cat.exe .

:heart::heart::heart::heart::heart: