An impressive duo for processing flat DBs: csvfix & CSVpad

Word processors, spreadsheets, presentations, translation, etc.
Message
Author
musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

An impressive duo for processing flat DBs: csvfix & CSVpad

#1 Post by musher0 »

Update, Saturday, Dec. 20, 2014
Please see second message on page 2 for csvfix's new companion, CSVpad-1.2.
Although an ok program, dmscsv is not as powerful. Thanks.
~~~~~~
Update, Friday, Dec. 19, 2014
Please see on top of page 2 for csvfix-1.60. Thanks.
~~~~~~
Update, Sat., Jan. 12, 2013.

Hello.

A year almost to the day, here is csvfix 1.50. The pet archive for this version is
basically structured the same, except that csvfix and the docs have been updated to
the latest version, v. 1.50.

You can see what has changed here: http://code.google.com/p/csvfix/wiki/ChangeLog.
In particular, the new

Code: Select all

csvfix money
command may come in handy in
some situations.

The complete list of csvfix commands is quoted 5-6 messages below.

Please download this 1.50 version of csvfix from here : http://limelinx.com/chc90.
(EDIT: now a dead link. Please see at top of page 2 for a valid link to a newer version.)

Enjoy!

musher0
//////////////////////////
Hello, everyone.

I just thought I'd share with you that there's a new multi-platorm csv processor out
there: CSVfix 1.20. It was released under a MIT licence late last year.

I've compiled it from source and here it is. gposil already presented a DBF type
database in a personal base thread. As well, there are a few SQL incarnations on
this board, so this good csv processor is just the thing to round out Puppy's
databasing capacity.

I've started using it, toying with a copy of the icewm "menu" present on all
Puppies-- to make it into a real csv file, among other things. I have very little
experience of this program yet, but it seems to be doing what you want it to do.

About the attached pet: in order to save space, the executable has been upx'd (still
works great; upx'ing does not affect performance), and the docs have been
archived in 7z format. You'll have to unpack these html docs to read them. Once
you've unpacked them, you'll find a convenient link to the Introduction in ~/my-
documents/Docs-Linux.

Finally, also included is a simple script I made base on "less" to help you view *.txt
and *.csv files. The script in ~/my-applications/bin, and you might want to link it in
~/.config/rox.sourceforge.net/OpenWith/.text_plain and
~/.config/rox.sourceforge.net/OpenWith/.text_csv

That way, you can view or follow what's happening to your csv file as csvfix
processes it, almost in real time, so to speak.

Although csvfix is not a "window"-type application, the pet contains a loader (edit)
which reminds you what command to type to launch csvfixin the terminal. As you
can realize, it might difficult to make a GUI for csvfix,given that it has so many
functions and parameters. That said, used in conjunction with less (above) and
dmcsv (see a couple of posts down), you get a very powerful tool to process csv
(aka flat-file) databases.

Finally, reading the doc is a must if you want to make the most out of this program.

Hopefully all this will be of use to you.

Enjoy!
Last edited by musher0 on Sat 20 Dec 2014, 22:18, edited 12 times in total.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#2 Post by musher0 »

Hello again.

The following will speak volumes, to whoever got headaches trying to
successfully process a csv database, about the usefulness of this
program. It also describes what csvfix command to use to solve a
particular problem. From the docs.

TWYL.

musher0

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Solutions to Common Problems
Introduction ››

Here are some of the common problems that you may experience when
dealing with CSV data, and brief suggestions as to how CSVfix can help
you with them:

"The fields in the data are in the wrong order"
* Reorder them using the order command.

"The records in the data are in the wrong order"
* Sort them using the sort command. Remove duplicates with the unique
command.

"Some fields are missing from my data"
* If you simply need to add empty fields to keep your application happy,
use the pad command. If you need to enrich your data with values from
another file, use the join command. If you need to add a fixed string to
the data, use the put command.

"My CSV data isn't comma-separated!"
* Use the -sep and -rsep flags to tell CSVfix what separator to use.

"I need to read tab-separated data"
* Use the read_dsv command with the -s '\t' flag to convert tab-separated
data to CSV.

"I need to convert XML to CSV"
* The from_xml command probably does what you need.

"I need to perform calculations on some of the fields in my CSV data"
* The eval command provides arithmetic and other functions you can
apply to your CSV data.

"I need to extract data from the middle of a field"
* Use the split_fixed and split_char commands to break fields up into sub
fields. If you need to build a field from values in other fields, use the
merge command.

"I need to remove some lines from a CSV file before processing it further"
* The find and remove commands allow you to filter your CSV files using
regular expressions, value ranges and field lengths. You can also create
exclusion lists and use the join command with the -inv flag to exclude
selected rows.

"I need to merge the data from two CSV files"
* Use the order command to get the fields in the same order, and then
the unique command to merge them, discarding duplicates.

"I want to add record numbers to my CSV data"
* The sequence command provides flexible record numbering.

"I want to split my data into different files"
* The file_split command can split CSV data streams depending on field
values.

"My data is full of duplicate values"
* Remove duplicates using the unique command.

"The data I get given is always full of invalid values"
* Validate your data using the validate command. Remove bad values
with the remove command.

"I need to convert CSV data into XML"
* The to_xml command does exactly this.

"I need to convert CSV data to this weird format"
* Use write_fixed, write_dsv and the printf and template commands to
format your output.

"I want to extract data from a SQL database as CSV"
* In the Windows version of CSVfix, the odbc_get command allows you to
extract data from SQL databases.

"I need to import data into a database, but the data is in a weird format"
* Use CSVfix to convert the data to CSV using the read_fixed, read_dsv,
and/ot read_multi commands, and then convert the CSV to SQL INSERT
statements using sql_insert.

"I need special date formatting"
* Use the date_format and date_iso comnands to read and re-format
dates.

Copyright © 2011 Neil Butterworth

~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Last edited by musher0 on Wed 31 Aug 2016, 12:16, edited 1 time in total.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

gcmartin

#3 Post by gcmartin »

Thanks @Musher0

I seem to remember few years ago a similar Windows Program for creating and processing CSVs. This was very useful for prepping data for DB entry.

Does CSVFIX has a front-ent gui for processing CSVs or is it a CLI only tool.

Thanks in advance

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#4 Post by musher0 »

Update, Saturday, Dec. 20, 2014
Please see second message on page 2 for csvfix's new companion, CSVpad-1.2.
Although an ok program, dmscsv is not as powerful. Thanks.
~~~~~~~~~~~~~~~~~

Hi, gcmartin.

No csvfix is only CLI. But I "intuited" :) your question, as I came back to
upload dmcsv.

dmcsv is a simple csv viewer -- don't expect CSVed for Linux, here --, but
still it will help you view the results of your processing with csvfix -- if
you're unsure in your mind of what the process will do. It is less abstract
of course than viewing your modified csv file with less -- which is fine, too.

In any case, enjoy! And have a great day!

P.S. Here is a very short description of dmcsv, as given by the author:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
DMcsvEditor is simple csv/tab files editor
darhmedia.hu [where you'll find the archive of the file]
darhmedia@gmail.com
goo.gl/77BAE
*** this version developing on Trisquel GNU/Linux ***
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

And a screenshot.
Attachments
dmcsv-0.6.jpg
Screenshot of dmcsv-0.6 for Linux in action
(52.46 KiB) Downloaded 2542 times
Last edited by musher0 on Wed 31 Aug 2016, 12:17, edited 2 times in total.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#5 Post by musher0 »

Here is an easy way to integrate the two.

And at the same it will give the opportunity to explain some csvfix commands.

Let's start with the csvfix commands.

For example, let's say you wish to number a list of clients, putting the entry number as the first field.

You would thus use the command

Code: Select all

csvfix sequence
with its parameters:

-n 1 -> start at number one
-i 1 -> with an interval of 1
-p 000 -> pad for up to 999 entries (you want the sequencing column to be three digits wide)
-ibl -> remove blank entries if any
-sep , -> specifies that you wish to have the comma as field separator
-smq -> use smart quotes in output file
-o [name of output file] -> put here a filename you invent to avoid confusion -- without the brackets, of course.
And finally the filename of the file you wish to process (your input file).

Here's the full command line:

Code: Select all

csvfix sequence -n 1 -i 1 -p 000 -ibl -sep , -smq -o clients-mod2.csv CLIENTS.csv && dmcsv clients-mod2.csv &
Now you'll notice I've added an extra segment to this line starting with "&&". That's for loading dmcsv.

It means : once csvfix has finished processing, load the newly created file in dmcsv so the operator (you!) can see the result.

You'll notice that the filename I ask dmcsv to load is the same as the one vis-a-vis the "-o" parameter for csvfix.

Et voila. Integration of two csv programs through bash!

A variant would be to see the newly created file with less instead of dmcsv, like so:

Code: Select all

csvfix sequence -n 1 -i 1 -p 000 -ibl -sep , -smq -o clients-mod2.csv CLIENTS.csv && less -~ clients-mod2.csv &
Have a good one!
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#6 Post by musher0 »

If it's of any help, the latest version of less is here:

http://murga-linux.com/puppy/viewtopic. ... 25&t=75161

TWYL.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#7 Post by musher0 »

Hi.

Here is the complete list of commands for csvfix, v. 1.50.

Code: Select all

usage: csvfix command [flags] file ...
where 'command' is one of: 

    ascii_table  produce ascii table
    block        perform actions on blocks of CSV records
    check        check CSV record format conforms to CSV standard
    date_format  perform date formatting
    date_iso     convert dates to ISO format
    diff         compare two CSV files
    echo         echo input to output
    edit         edit fields
    escape       escape special characters
    eval         expression evaluation
    exclude      exclude fields from output
    exec         execute external command
    file_info    add file information to output
    file_merge   merge multiple sorted CSV files
    file_split   split into files
    find         find rows matching regular expression or range
    flatten      flatten to single row
    from_xml     convert from XML to CSV
    head         list first CSV records
    inter        interleave fields from two CSV sources
    join         perform relational join
    lower        convert to lowercase
    map          map betwen CSV field values
    merge        merge fields
    mixed        convert to mixed case
    money        format fields as money/currency values
    order        change field order
    pad          pad CSV records to fixed number of fields
    printf       printf-style formatting
    put          put literal or env variable into CSV output
    read_dsv     convert DSV (delimiter separated variables) data to CSV
    read_fixed   convert fixed format data to CSV
    read_multi   read multi-line data
    remove       remove rows matching regular expression or range
    rmnew        remove embedded newlines
    sequence     add sequence numbers
    shuffle      randomly shuffle rows or fields
    sort         sort CSV input on one or more fields
    split_char   split at character or character type transition
    split_fixed  split input using fixed sizes
    sql_delete   generate SQL DELETE statements
    sql_insert   generate SQL INSERT statements
    sql_update   generate SQL UPDATE statements
    stat         produce record/field stats for CSV files
    summary      summarise CSV data
    tail         list last CSV records
    template     output via template
    timestamp    add timestamp to CSV data
    to_xml       convert CSV to XML
    trim         trim leading/trailing spaces
    truncate     truncate CSV records
    unflatten    convert single row to multiple rows
    unique       filter duplicate CSV records
    upper        convert to uppercase
    validate     validate CSV data files
    write_dsv    convert CSV to DSV format
    write_fixed  convert CSV to fixed format
Best regards.

musher0
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#8 Post by amigo »

Where did you find dmcsv at?

Distracting a slight bit from the original subject, I recently ram across another light-weight database called 'recutils'. It's a CLI database which lends itself well to scripting or front-ending with some dialog program. Also, it uses flat plain-text files so the databse can easily be worked with using grep, awk, sed or any text editor you like.
recutils is in the ubuntu repos so you can look for it there.

My search for dmcsv came up empty.

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#9 Post by musher0 »

Hello, amigo.

Thanks for enquiring. These are the search results I got with ask:
http://www.ask.com/web?q=dmcsv%20csv&o=101451&l=dis
I had to insist with the search parameters : "dmcsv csv".

The Linux version is apparently here :
http://linux.softpedia.com/get/Text-Edi ... 6283.shtml

However, that may be an error, since the latest available Linux version listed on the author's site appears to be v. 0.6... Please see : http://code.google.com/p/dmcsveditor/do ... bel:Latest

(Linux version 0.6 of dmcsv is provided above in this thread.)

dmcsv obviously exists for Windows :
http://www.softpedia.com/get/Programmin ... itor.shtml

and even as a portable app:
http://www.portablefreeware.com/?id=2183

Best regards.

musher0
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#10 Post by musher0 »

To download recutils : http://archive.ubuntu.com/ubuntu/pool/u ... 1_i386.deb

I'll be testing it shortly. Thanks again for the tip, amigo.

musher0
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#11 Post by amigo »

Eeeww! dmcsv=Lazarus/Delphi

Still, since csvfix is all CLI, it would also be possible to make a frontend using gtkdialog or other such animal.

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#12 Post by musher0 »

amigo wrote:Eeeww! dmcsv=Lazarus/Delphi

Still, since csvfix is all CLI, it would also be possible to make a frontend using gtkdialog or other such animal.
Are you suggesting or volunteering ? :D
The Linux community would need something like this! :wink:

musher0
Last edited by musher0 on Mon 06 Feb 2017, 23:20, edited 1 time in total.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#13 Post by amigo »

Oh, I'm definitely suggesting! At least until I have a need or an itch to satisfy somebody else's need. Come on, there's lots of gkdialog programmers around here -even though I rather see a regular gtk GUI or something done with something beside gtkdialog... Far be it from me to implement something which others could do instead... I have my hands full already. But you can (nearly) always count on me for suggestions and wry criticism!

Pelo

CSVpad helping Gnumeric and OSMO

#14 Post by Pelo »

Sort is easy with CSVpad, just go on the choosen colum an d sort A to Z, or Z to A, as you like.
Import the csv external data in Gnumeric, Well done !
Attachments
CSVpad.jpg
(72.48 KiB) Downloaded 821 times
Last edited by Pelo on Sat 03 Jan 2015, 17:40, edited 3 times in total.

Pelo

dmcsv and separators

#15 Post by Pelo »

Space free, because solved above
Last edited by Pelo on Sat 03 Jan 2015, 17:42, edited 1 time in total.

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#16 Post by musher0 »

Good afternoon, csv table lovers! :)

I've uploaded version 1.60 of csvfix (released circa mid-2014) at
http://www.datafilehost.com/d/672b6210

I compiled it from source on Slacko-6.0b. Since this powerful csv file processor
is a console app, there should be no compatibility problems with other Puppies.

Please note that the main site for csvfix is now: http://neilb.bitbucket.org/csvfix/

The docs are included in the *.pet package, but they are also available online
at http://neilb.bitbucket.org/csvfix/manua ... svfix.html

Enjoy.

musher0
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#17 Post by musher0 »

Hello!

Just a note to say that I have made CSVpad-1.2, a much more powerful
replacement for dmcsv, available through:
http://murga-linux.com/puppy/viewtopic. ... ost#815999

BFN.

musher0
~~~~~~~~~~
Late thanks and apologies to pelo... :oops:
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

Pelo

CSVpad will help untill libgcong found.

#18 Post by Pelo »

CSVpad-1.2_Linux32.pet: 1734 K
dmcsv-0.6.pet: 1013 K
Attachments
tivadar.png
Développeurs
(14.57 KiB) Downloaded 231 times
csvpad.png
obprecise
(60.93 KiB) Downloaded 233 times
Last edited by Pelo on Fri 29 Sep 2017, 02:55, edited 4 times in total.

ac2011
Posts: 134
Joined: Wed 09 Feb 2011, 08:22

#19 Post by ac2011 »

musher0 wrote:Good afternoon, csv table lovers! :)
I compiled it from source on Slacko-6.0b. Since this powerful csv file processor
is a console app, there should be no compatibility problems with other Puppies.
The link is dead but I compiled CSVfix 1.60 in Tahr and it works fine there, but unfortunately not in Wary (GlibC errors) despite being a console app.

If I try to compile it in Wary I get issues with g++ errors. Don't suppose anyone's managed to compile this on an in-between version of Puppy so it'll work in Wary?

Pelo

Musher0, please check your links, s'il vous plait..

#20 Post by Pelo »

Musher0, please check your link, s'il vous plait..
Attachments
CSVpad.jpg
Depannage avec vieille version.
(48.43 KiB) Downloaded 415 times

Post Reply