Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Sun 21 Dec 2014, 18:27
All times are UTC - 4
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » Documents
An impressive duo for processing flat DBs: csvfix & CSVpad
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
Page 1 of 2 Posts_count   Goto page: 1, 2 Next
Author Message
musher0


Joined: 04 Jan 2009
Posts: 4465
Location: Gatineau (Qc), Canada

PostPosted: Mon 09 Jan 2012, 04:11    Post_subject:  An impressive duo for processing flat DBs: csvfix & CSVpad
Sub_title: A sophisticated CLI manager and a good viewer/editor for csv databases now available on PuppyLinux
 

Update, Saturday, Dec. 20, 2014
Please see second message on page 2 for csvfix's new companion, CSVpad-1.2.
Although an ok program, dmscsv is not as powerful. Thanks.
~~~~~~
Update, Friday, Dec. 19, 2014
Please see on top of page 2 for csvfix-1.60. Thanks.
~~~~~~
Update, Sat., Jan. 12, 2013.

Hello.

A year almost to the day, here is csvfix 1.50. The pet archive for this version is
basically structured the same, except that csvfix and the docs have been updated to
the latest version, v. 1.50.

You can see what has changed here: http://code.google.com/p/csvfix/wiki/ChangeLog.
In particular, the new
Code:
csvfix money
command may come in handy in
some situations.

The complete list of csvfix commands is quoted 5-6 messages below.

Please download this 1.50 version of csvfix from here : http://limelinx.com/chc90.
(EDIT: now a dead link. Please see at top of page 2 for a valid link to a newer version.)

Enjoy!

musher0
//////////////////////////
Hello, everyone.

I just thought I'd share with you that there's a new multi-platorm csv processor out
there: CSVfix 1.20. It was released under a MIT licence late last year.

I've compiled it from source and here it is. gposil already presented a DBF type
database in a personal base thread. As well, there are a few SQL incarnations on
this board, so this good csv processor is just the thing to round out Puppy's
databasing capacity.

I've started using it, toying with a copy of the icewm "menu" present on all
Puppies-- to make it into a real csv file, among other things. I have very little
experience of this program yet, but it seems to be doing what you want it to do.

About the attached pet: in order to save space, the executable has been upx'd (still
works great; upx'ing does not affect performance), and the docs have been
archived in 7z format. You'll have to unpack these html docs to read them. Once
you've unpacked them, you'll find a convenient link to the Introduction in ~/my-
documents/Docs-Linux.

Finally, also included is a simple script I made base on "less" to help you view *.txt
and *.csv files. The script in ~/my-applications/bin, and you might want to link it in
~/.config/rox.sourceforge.net/OpenWith/.text_plain and
~/.config/rox.sourceforge.net/OpenWith/.text_csv

That way, you can view or follow what's happening to your csv file as csvfix
processes it, almost in real time, so to speak.

Although csvfix is not a "window"-type application, the pet contains a loader (edit)
which reminds you what command to type to launch csvfixin the terminal. As you
can realize, it might difficult to make a GUI for csvfix,given that it has so many
functions and parameters. That said, used in conjunction with less (above) and
dmcsv (see a couple of posts down), you get a very powerful tool to process csv
(aka flat-file) databases.

Finally, reading the doc is a must if you want to make the most out of this program.

Hopefully all this will be of use to you.

Enjoy!
csvfix-1.20.pet
Description  (EDIT) This version is still ok, of course, but it's been superseded by version 1.70...
See top of page 2 on this thread.
pet

 Download 
Filename  csvfix-1.20.pet 
Filesize  363.32 KB 
Downloaded  454 Time(s) 

_________________
"Logical entities must not be multiplied needlessly." / "Il ne faut pas multiplier les êtres logiques inutilement." (Ockham)

Edited_times_total
Back to top
View user's profile Send_private_message Visit_website 
musher0


Joined: 04 Jan 2009
Posts: 4465
Location: Gatineau (Qc), Canada

PostPosted: Mon 09 Jan 2012, 04:20    Post_subject:
Sub_title: Excerpt from the csvfix documentation
 

Hello again.

The following will speak volumes, to whoever got headaches trying to successfully process a csv database, about the usefulness of this program. It also describes what csvfix command to use to solve a particular problem. From the docs.

TWYL.

musher0

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Solutions to Common Problems
Introduction ››

Here are some of the common problems that you may experience when dealing with CSV data, and brief suggestions as to how CSVfix can help you with them:

"The fields in the data are in the wrong order"
* Reorder them using the order command.

"The records in the data are in the wrong order"
* Sort them using the sort command. Remove duplicates with the unique command.

"Some fields are missing from my data"
* If you simply need to add empty fields to keep your application happy, use the pad command. If you need to enrich your data with values from another file, use the join command. If you need to add a fixed string to the data, use the put command.

"My CSV data isn't comma-separated!"
* Use the -sep and -rsep flags to tell CSVfix what separator to use.

"I need to read tab-separated data"
* Use the read_dsv command with the -s '\t' flag to convert tab-separated data to CSV.

"I need to convert XML to CSV"
* The from_xml command probably does what you need.

"I need to perform calculations on some of the fields in my CSV data"
* The eval command provides arithmetic and other functions you can apply to your CSV data.

"I need to extract data from the middle of a field"
* Use the split_fixed and split_char commands to break fields up into sub fields. If you need to build a field from values in other fields, use the merge command.

"I need to remove some lines from a CSV file before processing it further"
* The find and remove commands allow you to filter your CSV files using regular expressions, value ranges and field lengths. You can also create exclusion lists and use the join command with the -inv flag to exclude selected rows.

"I need to merge the data from two CSV files"
* Use the order command to get the fields in the same order, and then the unique command to merge them, discarding duplicates.

"I want to add record numbers to my CSV data"
* The sequence command provides flexible record numbering.

"I want to split my data into different files"
* The file_split command can split CSV data streams depending on field values.

"My data is full of duplicate values"
* Remove duplicates using the unique command.

"The data I get given is always full of invalid values"
* Validate your data using the validate command. Remove bad values with the remove command.

"I need to convert CSV data into XML"
* The to_xml command does exactly this.

"I need to convert CSV data to this weird format"
* Use write_fixed, write_dsv and the printf and template commands to format your output.

"I want to extract data from a SQL database as CSV"
* In the Windows version of CSVfix, the odbc_get command allows you to extract data from SQL databases.

"I need to import data into a database, but the data is in a weird format"
* Use CSVfix to convert the data to CSV using the read_fixed, read_dsv, and/ot read_multi commands, and then convert the CSV to SQL INSERT statements using sql_insert.

"I need special date formatting"
* Use the date_format and date_iso comnands to read and re-format dates.

Copyright © 2011 Neil Butterworth

~~~~~~~~~~~~~~~~~~~~~~~~~~~~

_________________
"Logical entities must not be multiplied needlessly." / "Il ne faut pas multiplier les êtres logiques inutilement." (Ockham)
Back to top
View user's profile Send_private_message Visit_website 
gcmartin


Joined: 14 Oct 2005
Posts: 4507
Location: Earth

PostPosted: Mon 09 Jan 2012, 12:57    Post_subject:  

Thanks @Musher0

I seem to remember few years ago a similar Windows Program for creating and processing CSVs. This was very useful for prepping data for DB entry.

Does CSVFIX has a front-ent gui for processing CSVs or is it a CLI only tool.

Thanks in advance

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile
Back to top
View user's profile Send_private_message 
musher0


Joined: 04 Jan 2009
Posts: 4465
Location: Gatineau (Qc), Canada

PostPosted: Fri 13 Jan 2012, 01:40    Post_subject:  

Update, Saturday, Dec. 20, 2014
Please see second message on page 2 for csvfix's new companion, CSVpad-1.2.
Although an ok program, dmscsv is not as powerful. Thanks.
~~~~~~~~~~~~~~~~~

Hi, gcmartin.

No csvfix is only CLI. But I "intuited" Smile your question, as I came back to upload dmcsv.

dmcsv is a simple csv viewer -- don't expect CSVed for Linux, here --, but still it will help you view the results of your processing with csvfix -- if you're unsure in your mind of what the process will do. It is less abstract of course than viewing your modified csv file with less -- which is fine, too.

In any case, enjoy! And have a great day!

P.S. Here is a very short description of dmcsv, as given by the author:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
DMcsvEditor is simple csv/tab files editor
darhmedia.hu [where you'll find the archive of the file]
darhmedia@gmail.com
goo.gl/77BAE
*** this version developing on Trisquel GNU/Linux ***
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

And a screenshot.
dmcsv-0.6.jpg
 Description   Screenshot of dmcsv-0.6 for Linux in action
 Filesize   52.46 KB
 Viewed   1632 Time(s)

dmcsv-0.6.jpg

dmcsv-0.6.pet
Description  A nice little viewer to go with csvfix (although not same author).
pet

 Download 
Filename  dmcsv-0.6.pet 
Filesize  1012.84 KB 
Downloaded  456 Time(s) 

_________________
"Logical entities must not be multiplied needlessly." / "Il ne faut pas multiplier les êtres logiques inutilement." (Ockham)

Edited_time_total
Back to top
View user's profile Send_private_message Visit_website 
musher0


Joined: 04 Jan 2009
Posts: 4465
Location: Gatineau (Qc), Canada

PostPosted: Fri 13 Jan 2012, 02:50    Post_subject:  

Here is an easy way to integrate the two.

And at the same it will give the opportunity to explain some csvfix commands.

Let's start with the csvfix commands.

For example, let's say you wish to number a list of clients, putting the entry number as the first field.

You would thus use the command
Code:
csvfix sequence
with its parameters:

-n 1 -> start at number one
-i 1 -> with an interval of 1
-p 000 -> pad for up to 999 entries (you want the sequencing column to be three digits wide)
-ibl -> remove blank entries if any
-sep , -> specifies that you wish to have the comma as field separator
-smq -> use smart quotes in output file
-o [name of output file] -> put here a filename you invent to avoid confusion -- without the brackets, of course.
And finally the filename of the file you wish to process (your input file).

Here's the full command line:

Code:
csvfix sequence -n 1 -i 1 -p 000 -ibl -sep , -smq -o clients-mod2.csv CLIENTS.csv && dmcsv clients-mod2.csv &


Now you'll notice I've added an extra segment to this line starting with "&&". That's for loading dmcsv.

It means : once csvfix has finished processing, load the newly created file in dmcsv so the operator (you!) can see the result.

You'll notice that the filename I ask dmcsv to load is the same as the one vis-a-vis the "-o" parameter for csvfix.

Et voila. Integration of two csv programs through bash!

A variant would be to see the newly created file with less instead of dmcsv, like so:

Code:
csvfix sequence -n 1 -i 1 -p 000 -ibl -sep , -smq -o clients-mod2.csv CLIENTS.csv && less -~ clients-mod2.csv &


Have a good one!

_________________
"Logical entities must not be multiplied needlessly." / "Il ne faut pas multiplier les êtres logiques inutilement." (Ockham)
Back to top
View user's profile Send_private_message Visit_website 
musher0


Joined: 04 Jan 2009
Posts: 4465
Location: Gatineau (Qc), Canada

PostPosted: Fri 13 Jan 2012, 04:32    Post_subject:  

If it's of any help, the latest version of less is here:

http://murga-linux.com/puppy/viewtopic.php?search_id=200079825&t=75161

TWYL.

_________________
"Logical entities must not be multiplied needlessly." / "Il ne faut pas multiplier les êtres logiques inutilement." (Ockham)
Back to top
View user's profile Send_private_message Visit_website 
musher0


Joined: 04 Jan 2009
Posts: 4465
Location: Gatineau (Qc), Canada

PostPosted: Sat 12 Jan 2013, 09:42    Post_subject:  

Hi.

Here is the complete list of commands for csvfix, v. 1.50.
Code:

usage: csvfix command [flags] file ...
where 'command' is one of:

    ascii_table  produce ascii table
    block        perform actions on blocks of CSV records
    check        check CSV record format conforms to CSV standard
    date_format  perform date formatting
    date_iso     convert dates to ISO format
    diff         compare two CSV files
    echo         echo input to output
    edit         edit fields
    escape       escape special characters
    eval         expression evaluation
    exclude      exclude fields from output
    exec         execute external command
    file_info    add file information to output
    file_merge   merge multiple sorted CSV files
    file_split   split into files
    find         find rows matching regular expression or range
    flatten      flatten to single row
    from_xml     convert from XML to CSV
    head         list first CSV records
    inter        interleave fields from two CSV sources
    join         perform relational join
    lower        convert to lowercase
    map          map betwen CSV field values
    merge        merge fields
    mixed        convert to mixed case
    money        format fields as money/currency values
    order        change field order
    pad          pad CSV records to fixed number of fields
    printf       printf-style formatting
    put          put literal or env variable into CSV output
    read_dsv     convert DSV (delimiter separated variables) data to CSV
    read_fixed   convert fixed format data to CSV
    read_multi   read multi-line data
    remove       remove rows matching regular expression or range
    rmnew        remove embedded newlines
    sequence     add sequence numbers
    shuffle      randomly shuffle rows or fields
    sort         sort CSV input on one or more fields
    split_char   split at character or character type transition
    split_fixed  split input using fixed sizes
    sql_delete   generate SQL DELETE statements
    sql_insert   generate SQL INSERT statements
    sql_update   generate SQL UPDATE statements
    stat         produce record/field stats for CSV files
    summary      summarise CSV data
    tail         list last CSV records
    template     output via template
    timestamp    add timestamp to CSV data
    to_xml       convert CSV to XML
    trim         trim leading/trailing spaces
    truncate     truncate CSV records
    unflatten    convert single row to multiple rows
    unique       filter duplicate CSV records
    upper        convert to uppercase
    validate     validate CSV data files
    write_dsv    convert CSV to DSV format
    write_fixed  convert CSV to fixed format


Best regards.

musher0

_________________
"Logical entities must not be multiplied needlessly." / "Il ne faut pas multiplier les êtres logiques inutilement." (Ockham)
Back to top
View user's profile Send_private_message Visit_website 
amigo

Joined: 02 Apr 2007
Posts: 2294

PostPosted: Sat 19 Jan 2013, 15:25    Post_subject:  

Where did you find dmcsv at?

Distracting a slight bit from the original subject, I recently ram across another light-weight database called 'recutils'. It's a CLI database which lends itself well to scripting or front-ending with some dialog program. Also, it uses flat plain-text files so the databse can easily be worked with using grep, awk, sed or any text editor you like.
recutils is in the ubuntu repos so you can look for it there.

My search for dmcsv came up empty.
Back to top
View user's profile Send_private_message 
musher0


Joined: 04 Jan 2009
Posts: 4465
Location: Gatineau (Qc), Canada

PostPosted: Sat 19 Jan 2013, 15:53    Post_subject:  

Hello, amigo.

Thanks for enquiring. These are the search results I got with ask:
http://www.ask.com/web?q=dmcsv%20csv&o=101451&l=dis
I had to insist with the search parameters : "dmcsv csv".

The Linux version is apparently here :
http://linux.softpedia.com/get/Text-Editing-Processing/Others/Text-Editor/DMcsvEditor-56283.shtml

However, that may be an error, since the latest available Linux version listed on the author's site appears to be v. 0.6... Please see : http://code.google.com/p/dmcsveditor/downloads/list?q=label:Latest

(Linux version 0.6 of dmcsv is provided above in this thread.)

dmcsv obviously exists for Windows :
http://www.softpedia.com/get/Programming/File-Editors/DMcsvEditor.shtml

and even as a portable app:
http://www.portablefreeware.com/?id=2183

Best regards.

musher0

_________________
"Logical entities must not be multiplied needlessly." / "Il ne faut pas multiplier les êtres logiques inutilement." (Ockham)
Back to top
View user's profile Send_private_message Visit_website 
musher0


Joined: 04 Jan 2009
Posts: 4465
Location: Gatineau (Qc), Canada

PostPosted: Sat 19 Jan 2013, 16:04    Post_subject:  

To download recutils : http://archive.ubuntu.com/ubuntu/pool/universe/r/recutils/recutils_1.5-1_i386.deb

I'll be testing it shortly. Thanks again for the tip, amigo.

musher0

_________________
"Logical entities must not be multiplied needlessly." / "Il ne faut pas multiplier les êtres logiques inutilement." (Ockham)
Back to top
View user's profile Send_private_message Visit_website 
amigo

Joined: 02 Apr 2007
Posts: 2294

PostPosted: Sat 19 Jan 2013, 16:11    Post_subject:  

Eeeww! dmcsv=Lazarus/Delphi

Still, since csvfix is all CLI, it would also be possible to make a frontend using gtkdialog or other such animal.
Back to top
View user's profile Send_private_message 
musher0


Joined: 04 Jan 2009
Posts: 4465
Location: Gatineau (Qc), Canada

PostPosted: Sat 19 Jan 2013, 16:40    Post_subject:  

amigo wrote:
Eeeww! dmcsv=Lazarus/Delphi

Still, since csvfix is all CLI, it would also be possible to make a frontend using gtkdialog or other such animal.


Are you suggesting or volunteering ? Very Happy
The Linux community would need something liuke this! Wink

musher0

_________________
"Logical entities must not be multiplied needlessly." / "Il ne faut pas multiplier les êtres logiques inutilement." (Ockham)
Back to top
View user's profile Send_private_message Visit_website 
amigo

Joined: 02 Apr 2007
Posts: 2294

PostPosted: Sat 19 Jan 2013, 17:21    Post_subject:  

Oh, I'm definitely suggesting! At least until I have a need or an itch to satisfy somebody else's need. Come on, there's lots of gkdialog programmers around here -even though I rather see a regular gtk GUI or something done with something beside gtkdialog... Far be it from me to implement something which others could do instead... I have my hands full already. But you can (nearly) always count on me for suggestions and wry criticism!
Back to top
View user's profile Send_private_message 
Pelo


Joined: 10 Sep 2011
Posts: 3452
Location: Mer méditerrannée (1 kms°)

PostPosted: Sun 20 Jan 2013, 01:39    Post_subject: csvfix is chinese for us, Mummy and me
Sub_title: Hard job
 

sh-4.1# csvfix

CSVfix 1.2
Copyright (C) 2011 Neil Butterworth

csvfix is a CSV stream editor
use 'csvfix help' to see a list of commands
use 'csvfix help command' to see help on a specific command
sh-4.1# csvfix check pelo.csv
sh-4.1# csvfix help check
check that records in file conform to CSV standards
usage: csvfix check [flags] [files ...]
where flags are:
-nl allow embedded newlines in quoted fields
-q run quietly - no output but return error indicator
-s sep use sep as the field separator (default is ',' )
-v print OK message for inputs that pass validation
Chinese would be easier, even for me

Edited_times_total
Back to top
View user's profile Send_private_message YIM 
Pelo


Joined: 10 Sep 2011
Posts: 3452
Location: Mer méditerrannée (1 kms°)

PostPosted: Sun 20 Jan 2013, 01:41    Post_subject: dmcsv and separators
Sub_title: Nice tool, Mummy is fond of it !
 

Separators semi column needed by homebank.
http://www.trustfm.net/ (site of some interest, sure Very Happy
CSVpad is a heavily modifided version of the DMcsvEditor. (CSVpad-v10-linux.zip: 856 K)
Smile The machine is on test bench at at pelo's workshops.
Séparated par de virgules, je voudrais des points virgules pour homebank (Translated french with google traduction)
csvpad.jpg
 Description   
 Filesize   74.91 KB
 Viewed   276 Time(s)

csvpad.jpg

Back to top
View user's profile Send_private_message YIM 
Display_posts:   Sort by:   
Page 1 of 2 Posts_count   Goto page: 1, 2 Next
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » Documents
Jump to:  

Rules_post_cannot
Rules_reply_cannot
Rules_edit_cannot
Rules_delete_cannot
Rules_vote_cannot
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1169s ][ Queries: 13 (0.0051s) ][ GZIP on ]