The time now is Sat 21 Apr 2018, 08:14
All times are UTC - 4 |
Page 1 of 3 [33 Posts] |
Goto page: 1, 2, 3 Next |
Author |
Message |
musher0

Joined: 04 Jan 2009 Posts: 12074 Location: Gatineau (Qc), Canada
|
Posted: Mon 09 Jan 2012, 04:11 Post subject:
An impressive duo for processing flat DBs: csvfix & CSVpad Subject description: A sophisticated CLI manager and a good viewer/editor for csv databases now available on PuppyLinux |
|
Update, Saturday, Dec. 20, 2014
Please see second message on page 2 for csvfix's new companion, CSVpad-1.2.
Although an ok program, dmscsv is not as powerful. Thanks.
~~~~~~
Update, Friday, Dec. 19, 2014
Please see on top of page 2 for csvfix-1.60. Thanks.
~~~~~~
Update, Sat., Jan. 12, 2013.
Hello.
A year almost to the day, here is csvfix 1.50. The pet archive for this version is
basically structured the same, except that csvfix and the docs have been updated to
the latest version, v. 1.50.
You can see what has changed here: http://code.google.com/p/csvfix/wiki/ChangeLog.
In particular, the new command may come in handy in
some situations.
The complete list of csvfix commands is quoted 5-6 messages below.
Please download this 1.50 version of csvfix from here : http://limelinx.com/chc90.
(EDIT: now a dead link. Please see at top of page 2 for a valid link to a newer version.)
Enjoy!
musher0
//////////////////////////
Hello, everyone.
I just thought I'd share with you that there's a new multi-platorm csv processor out
there: CSVfix 1.20. It was released under a MIT licence late last year.
I've compiled it from source and here it is. gposil already presented a DBF type
database in a personal base thread. As well, there are a few SQL incarnations on
this board, so this good csv processor is just the thing to round out Puppy's
databasing capacity.
I've started using it, toying with a copy of the icewm "menu" present on all
Puppies-- to make it into a real csv file, among other things. I have very little
experience of this program yet, but it seems to be doing what you want it to do.
About the attached pet: in order to save space, the executable has been upx'd (still
works great; upx'ing does not affect performance), and the docs have been
archived in 7z format. You'll have to unpack these html docs to read them. Once
you've unpacked them, you'll find a convenient link to the Introduction in ~/my-
documents/Docs-Linux.
Finally, also included is a simple script I made base on "less" to help you view *.txt
and *.csv files. The script in ~/my-applications/bin, and you might want to link it in
~/.config/rox.sourceforge.net/OpenWith/.text_plain and
~/.config/rox.sourceforge.net/OpenWith/.text_csv
That way, you can view or follow what's happening to your csv file as csvfix
processes it, almost in real time, so to speak.
Although csvfix is not a "window"-type application, the pet contains a loader (edit)
which reminds you what command to type to launch csvfixin the terminal. As you
can realize, it might difficult to make a GUI for csvfix,given that it has so many
functions and parameters. That said, used in conjunction with less (above) and
dmcsv (see a couple of posts down), you get a very powerful tool to process csv
(aka flat-file) databases.
Finally, reading the doc is a must if you want to make the most out of this program.
Hopefully all this will be of use to you.
Enjoy!
_________________ musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
Last edited by musher0 on Sat 20 Dec 2014, 18:18; edited 12 times in total
|
Back to top
|
|
 |
musher0

Joined: 04 Jan 2009 Posts: 12074 Location: Gatineau (Qc), Canada
|
Posted: Mon 09 Jan 2012, 04:20 Post subject:
Subject description: Excerpt from the csvfix documentation |
|
Hello again.
The following will speak volumes, to whoever got headaches trying to
successfully process a csv database, about the usefulness of this
program. It also describes what csvfix command to use to solve a
particular problem. From the docs.
TWYL.
musher0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Solutions to Common Problems
Introduction ››
Here are some of the common problems that you may experience when
dealing with CSV data, and brief suggestions as to how CSVfix can help
you with them:
"The fields in the data are in the wrong order"
* Reorder them using the order command.
"The records in the data are in the wrong order"
* Sort them using the sort command. Remove duplicates with the unique
command.
"Some fields are missing from my data"
* If you simply need to add empty fields to keep your application happy,
use the pad command. If you need to enrich your data with values from
another file, use the join command. If you need to add a fixed string to
the data, use the put command.
"My CSV data isn't comma-separated!"
* Use the -sep and -rsep flags to tell CSVfix what separator to use.
"I need to read tab-separated data"
* Use the read_dsv command with the -s '\t' flag to convert tab-separated
data to CSV.
"I need to convert XML to CSV"
* The from_xml command probably does what you need.
"I need to perform calculations on some of the fields in my CSV data"
* The eval command provides arithmetic and other functions you can
apply to your CSV data.
"I need to extract data from the middle of a field"
* Use the split_fixed and split_char commands to break fields up into sub
fields. If you need to build a field from values in other fields, use the
merge command.
"I need to remove some lines from a CSV file before processing it further"
* The find and remove commands allow you to filter your CSV files using
regular expressions, value ranges and field lengths. You can also create
exclusion lists and use the join command with the -inv flag to exclude
selected rows.
"I need to merge the data from two CSV files"
* Use the order command to get the fields in the same order, and then
the unique command to merge them, discarding duplicates.
"I want to add record numbers to my CSV data"
* The sequence command provides flexible record numbering.
"I want to split my data into different files"
* The file_split command can split CSV data streams depending on field
values.
"My data is full of duplicate values"
* Remove duplicates using the unique command.
"The data I get given is always full of invalid values"
* Validate your data using the validate command. Remove bad values
with the remove command.
"I need to convert CSV data into XML"
* The to_xml command does exactly this.
"I need to convert CSV data to this weird format"
* Use write_fixed, write_dsv and the printf and template commands to
format your output.
"I want to extract data from a SQL database as CSV"
* In the Windows version of CSVfix, the odbc_get command allows you to
extract data from SQL databases.
"I need to import data into a database, but the data is in a weird format"
* Use CSVfix to convert the data to CSV using the read_fixed, read_dsv,
and/ot read_multi commands, and then convert the CSV to SQL INSERT
statements using sql_insert.
"I need special date formatting"
* Use the date_format and date_iso comnands to read and re-format
dates.
Copyright © 2011 Neil Butterworth
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
_________________ musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
Last edited by musher0 on Wed 31 Aug 2016, 08:16; edited 1 time in total
|
Back to top
|
|
 |
gcmartin
Joined: 14 Oct 2005 Posts: 6730 Location: Earth
|
Posted: Mon 09 Jan 2012, 12:57 Post subject:
|
|
Thanks @Musher0
I seem to remember few years ago a similar Windows Program for creating and processing CSVs. This was very useful for prepping data for DB entry.
Does CSVFIX has a front-ent gui for processing CSVs or is it a CLI only tool.
Thanks in advance
_________________ Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engines or use DogPile
|
Back to top
|
|
 |
musher0

Joined: 04 Jan 2009 Posts: 12074 Location: Gatineau (Qc), Canada
|
Posted: Fri 13 Jan 2012, 01:40 Post subject:
|
|
Update, Saturday, Dec. 20, 2014
Please see second message on page 2 for csvfix's new companion, CSVpad-1.2.
Although an ok program, dmscsv is not as powerful. Thanks.
~~~~~~~~~~~~~~~~~
Hi, gcmartin.
No csvfix is only CLI. But I "intuited" your question, as I came back to
upload dmcsv.
dmcsv is a simple csv viewer -- don't expect CSVed for Linux, here --, but
still it will help you view the results of your processing with csvfix -- if
you're unsure in your mind of what the process will do. It is less abstract
of course than viewing your modified csv file with less -- which is fine, too.
In any case, enjoy! And have a great day!
P.S. Here is a very short description of dmcsv, as given by the author:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
DMcsvEditor is simple csv/tab files editor
darhmedia.hu [where you'll find the archive of the file]
darhmedia@gmail.com
goo.gl/77BAE
*** this version developing on Trisquel GNU/Linux ***
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
And a screenshot.
 |
Description |
Screenshot of dmcsv-0.6 for Linux in action |
Filesize |
52.46 KB |
Viewed |
2268 Time(s) |

|
_________________ musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
Last edited by musher0 on Wed 31 Aug 2016, 08:17; edited 2 times in total
|
Back to top
|
|
 |
musher0

Joined: 04 Jan 2009 Posts: 12074 Location: Gatineau (Qc), Canada
|
Posted: Fri 13 Jan 2012, 02:50 Post subject:
|
|
Here is an easy way to integrate the two.
And at the same it will give the opportunity to explain some csvfix commands.
Let's start with the csvfix commands.
For example, let's say you wish to number a list of clients, putting the entry number as the first field.
You would thus use the command with its parameters:
-n 1 -> start at number one
-i 1 -> with an interval of 1
-p 000 -> pad for up to 999 entries (you want the sequencing column to be three digits wide)
-ibl -> remove blank entries if any
-sep , -> specifies that you wish to have the comma as field separator
-smq -> use smart quotes in output file
-o [name of output file] -> put here a filename you invent to avoid confusion -- without the brackets, of course.
And finally the filename of the file you wish to process (your input file).
Here's the full command line:
Code: | csvfix sequence -n 1 -i 1 -p 000 -ibl -sep , -smq -o clients-mod2.csv CLIENTS.csv && dmcsv clients-mod2.csv & |
Now you'll notice I've added an extra segment to this line starting with "&&". That's for loading dmcsv.
It means : once csvfix has finished processing, load the newly created file in dmcsv so the operator (you!) can see the result.
You'll notice that the filename I ask dmcsv to load is the same as the one vis-a-vis the "-o" parameter for csvfix.
Et voila. Integration of two csv programs through bash!
A variant would be to see the newly created file with less instead of dmcsv, like so:
Code: | csvfix sequence -n 1 -i 1 -p 000 -ibl -sep , -smq -o clients-mod2.csv CLIENTS.csv && less -~ clients-mod2.csv & |
Have a good one!
_________________ musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
|
Back to top
|
|
 |
musher0

Joined: 04 Jan 2009 Posts: 12074 Location: Gatineau (Qc), Canada
|
Posted: Fri 13 Jan 2012, 04:32 Post subject:
|
|
If it's of any help, the latest version of less is here:
http://murga-linux.com/puppy/viewtopic.php?search_id=200079825&t=75161
TWYL.
_________________ musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
|
Back to top
|
|
 |
musher0

Joined: 04 Jan 2009 Posts: 12074 Location: Gatineau (Qc), Canada
|
Posted: Sat 12 Jan 2013, 09:42 Post subject:
|
|
Hi.
Here is the complete list of commands for csvfix, v. 1.50.
Code: |
usage: csvfix command [flags] file ...
where 'command' is one of:
ascii_table produce ascii table
block perform actions on blocks of CSV records
check check CSV record format conforms to CSV standard
date_format perform date formatting
date_iso convert dates to ISO format
diff compare two CSV files
echo echo input to output
edit edit fields
escape escape special characters
eval expression evaluation
exclude exclude fields from output
exec execute external command
file_info add file information to output
file_merge merge multiple sorted CSV files
file_split split into files
find find rows matching regular expression or range
flatten flatten to single row
from_xml convert from XML to CSV
head list first CSV records
inter interleave fields from two CSV sources
join perform relational join
lower convert to lowercase
map map betwen CSV field values
merge merge fields
mixed convert to mixed case
money format fields as money/currency values
order change field order
pad pad CSV records to fixed number of fields
printf printf-style formatting
put put literal or env variable into CSV output
read_dsv convert DSV (delimiter separated variables) data to CSV
read_fixed convert fixed format data to CSV
read_multi read multi-line data
remove remove rows matching regular expression or range
rmnew remove embedded newlines
sequence add sequence numbers
shuffle randomly shuffle rows or fields
sort sort CSV input on one or more fields
split_char split at character or character type transition
split_fixed split input using fixed sizes
sql_delete generate SQL DELETE statements
sql_insert generate SQL INSERT statements
sql_update generate SQL UPDATE statements
stat produce record/field stats for CSV files
summary summarise CSV data
tail list last CSV records
template output via template
timestamp add timestamp to CSV data
to_xml convert CSV to XML
trim trim leading/trailing spaces
truncate truncate CSV records
unflatten convert single row to multiple rows
unique filter duplicate CSV records
upper convert to uppercase
validate validate CSV data files
write_dsv convert CSV to DSV format
write_fixed convert CSV to fixed format
|
Best regards.
musher0
_________________ musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
|
Back to top
|
|
 |
amigo
Joined: 02 Apr 2007 Posts: 2641
|
Posted: Sat 19 Jan 2013, 15:25 Post subject:
|
|
Where did you find dmcsv at?
Distracting a slight bit from the original subject, I recently ram across another light-weight database called 'recutils'. It's a CLI database which lends itself well to scripting or front-ending with some dialog program. Also, it uses flat plain-text files so the databse can easily be worked with using grep, awk, sed or any text editor you like.
recutils is in the ubuntu repos so you can look for it there.
My search for dmcsv came up empty.
|
Back to top
|
|
 |
musher0

Joined: 04 Jan 2009 Posts: 12074 Location: Gatineau (Qc), Canada
|
Posted: Sat 19 Jan 2013, 15:53 Post subject:
|
|
Hello, amigo.
Thanks for enquiring. These are the search results I got with ask:
http://www.ask.com/web?q=dmcsv%20csv&o=101451&l=dis
I had to insist with the search parameters : "dmcsv csv".
The Linux version is apparently here :
http://linux.softpedia.com/get/Text-Editing-Processing/Others/Text-Editor/DMcsvEditor-56283.shtml
However, that may be an error, since the latest available Linux version listed on the author's site appears to be v. 0.6... Please see : http://code.google.com/p/dmcsveditor/downloads/list?q=label:Latest
(Linux version 0.6 of dmcsv is provided above in this thread.)
dmcsv obviously exists for Windows :
http://www.softpedia.com/get/Programming/File-Editors/DMcsvEditor.shtml
and even as a portable app:
http://www.portablefreeware.com/?id=2183
Best regards.
musher0
_________________ musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
|
Back to top
|
|
 |
musher0

Joined: 04 Jan 2009 Posts: 12074 Location: Gatineau (Qc), Canada
|
Posted: Sat 19 Jan 2013, 16:04 Post subject:
|
|
To download recutils : http://archive.ubuntu.com/ubuntu/pool/universe/r/recutils/recutils_1.5-1_i386.deb
I'll be testing it shortly. Thanks again for the tip, amigo.
musher0
_________________ musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
|
Back to top
|
|
 |
amigo
Joined: 02 Apr 2007 Posts: 2641
|
Posted: Sat 19 Jan 2013, 16:11 Post subject:
|
|
Eeeww! dmcsv=Lazarus/Delphi
Still, since csvfix is all CLI, it would also be possible to make a frontend using gtkdialog or other such animal.
|
Back to top
|
|
 |
musher0

Joined: 04 Jan 2009 Posts: 12074 Location: Gatineau (Qc), Canada
|
Posted: Sat 19 Jan 2013, 16:40 Post subject:
|
|
amigo wrote: | Eeeww! dmcsv=Lazarus/Delphi
Still, since csvfix is all CLI, it would also be possible to make a frontend using gtkdialog or other such animal. |
Are you suggesting or volunteering ?
The Linux community would need something like this!
musher0
_________________ musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
Last edited by musher0 on Mon 06 Feb 2017, 19:20; edited 1 time in total
|
Back to top
|
|
 |
amigo
Joined: 02 Apr 2007 Posts: 2641
|
Posted: Sat 19 Jan 2013, 17:21 Post subject:
|
|
Oh, I'm definitely suggesting! At least until I have a need or an itch to satisfy somebody else's need. Come on, there's lots of gkdialog programmers around here -even though I rather see a regular gtk GUI or something done with something beside gtkdialog... Far be it from me to implement something which others could do instead... I have my hands full already. But you can (nearly) always count on me for suggestions and wry criticism!
|
Back to top
|
|
 |
Pelo
Joined: 10 Sep 2011 Posts: 12591 Location: Mer méditerrannée (1 kms°)
|
Posted: Sun 20 Jan 2013, 01:39 Post subject:
CSVpad helping Gnumeric and OSMO Subject description: Sort is easy with CSVpad, |
|
Sort is easy with CSVpad, just go on the choosen colum an d sort A to Z, or Z to A, as you like.
Import the csv external data in Gnumeric, Well done !
Description |
|
Filesize |
72.48 KB |
Viewed |
553 Time(s) |

|
Last edited by Pelo on Sat 03 Jan 2015, 13:40; edited 3 times in total
|
Back to top
|
|
 |
Pelo
Joined: 10 Sep 2011 Posts: 12591 Location: Mer méditerrannée (1 kms°)
|
Posted: Sun 20 Jan 2013, 01:41 Post subject:
dmcsv and separators Subject description: Nice tool, Mummy is fond of it ! |
|
Space free, because solved above
Last edited by Pelo on Sat 03 Jan 2015, 13:42; edited 1 time in total
|
Back to top
|
|
 |
|
Page 1 of 3 [33 Posts] |
Goto page: 1, 2, 3 Next |
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum You cannot attach files in this forum You can download files in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|