fields.awk database

Under development: PCMCIA, wireless, etc.
Message
Author
Pelo

Databases ... for pleasure...

#81 Post by Pelo »

"My view is that Windows users may be even more reluctant than Linux ones when it comes to using the CLI."
First CLI needs to speak english, what can eject 95% of people having a computer.

cabbie0
Posts: 26
Joined: Thu 23 Jun 2016, 05:19
Location: British Columbia, Canada

Scripting LANGUAGE

#82 Post by cabbie0 »

Pelo wrote:
First CLI needs to speak english, what can eject 95% of people having a computer.
I hadn't thought of that, very true.

@musher0
BONNE FÊTE DU CANADA!

User avatar
Pete
Posts: 660
Joined: Sun 02 Mar 2014, 18:36

#83 Post by Pete »

@Pelo

I do agree with you that the CLI can be rather cryptic at times.
One way to make it more "English" would be to use Aliases.

In general I don't like aliases as it makes your script less portable but
if it helps in the beginning then go for it.
You can always still use the original command as well.

You could even make it more "French" if you like. :)

For example if you type this:

alias imprimer='echo'

Then you can use imprimer instead of echo, like so:

imprimer "Bonjour Monsieur Pelo"

Just don't use reserved words and to make the aliases persistent across reboots, place them all in ~/.bashrc or ~/.bash_aliases depending on your setup.

Pelo

If English speaking people want to use a Database;

#84 Post by Pelo »

it was just a remark. If English speaking people want to use a Database; it's a good idea.
As i Told somewhere, Puppy package manager is a database (data are stored in hidden .packages under root. You can use these data to test fields.awk.
separator | can easily be replaced by comma or semicolon

User avatar
Pete
Posts: 660
Joined: Sun 02 Mar 2014, 18:36

#85 Post by Pete »

@Pelo

OK, understood.

User avatar
Pete
Posts: 660
Joined: Sun 02 Mar 2014, 18:36

#86 Post by Pete »

@cabbie0

I got my Android toy and downloaded and installed ZShaolin.
Works great and I like the feature that it has 4 terminals, but how does one enter
CNTL+C?
Sometimes it gets stuck and I need to terminate a script.
Tried volume down + c but no go.
Any ideas?

T.I.A.

cabbie0
Posts: 26
Joined: Thu 23 Jun 2016, 05:19
Location: British Columbia, Canada

Zshaolin Special Keys Mapping

#87 Post by cabbie0 »

@Pete

Hi,
Glad to hear you've managed to get Zshaolin installed on an Android device.
I use the Volume_Up button for CTRL (and Volume_Down for ESC).
But that is because I specifically mapped them to those keys.
In the startup screen for Zshaolin, you should see a 6-part menu:
[Start ZShaolin] [Install System] [Shutdown] [Keyboard] [Options] [Website]
Select [Options] (I know, you'd think it would be Keyboard....)
At the bottom of [Options], see [SPECIAL KEYS MAPPING]
There is has special options for both the CTRL and ESC keys you can select.
In each case the opions (on my version) are: Jog ball, @key, Left Alt, Right Alt, Vol Up, Vol Down, Camera Key, or None.

Hope that helps.

cabbie0
Posts: 26
Joined: Thu 23 Jun 2016, 05:19
Location: British Columbia, Canada

Read Search Output file processing, continued.

#88 Post by cabbie0 »

@all

Follow up to previous post on filtering "Read Search Output" files for displaying 2 or 3 selected Fields only.

Although the "egrep" command sequence works, it is a bit too complex to be practical
unless you could make a "one-liner" script which takes your selected fields in the form of command-line arguments and inserts them into the "egrep" sequence, so you don't have to remember the syntax every time.

I tried this, but found that transferring command line arguments as variables into such a script is VERY problematic.
The syntax section " '[SELECTION1]|[SELECTION2]' "
is the sticking point as the single quotes on either side render the variables ($SELECT1 etc) as meaningless text and if you escape the single quote with "\" then the syntax itself is rendered inoperable. -- "Catch22"

So instead I decided to use AWK for both sections of the script; that is, the the multiple-argument section and the blank line insertion section.
Two separate "one-liner" filter scripts called OPselect2 and OPselect3 for filtering out 2 or 3 FIELDS respectively.
The active code for each as follows:

Note each code string is two separate awk scripts; the output from the first (highlighted red) being piped as input to the second. Consequently the entire code string must be on a single line and not wrapped into two separate lines.
The first parts (in red) select only lines containing the FIELD names you enter on the command line.
The second part (in black) is the same for each script since it simply inserts a blank line in front of the first selection, for easier readability.
The command sequence is simple to remember:
OPselect2 [Field_selection1] [Field_selection2] [filename] (where filename is your output file generated by fields.awk)

EXAMPLE: Let's say we have a short output file from a database of tree types. Filename=treesout.txt
==> Matching record 1 :
COMMON NAME = Maple
LATIN NAME = Acer
FOLIAGE TYPE = deciduous
*******************************************

==> Matching record 2 :
COMMON NAME = Oak
LATIN NAME = Quercus
FOLIAGE TYPE = deciduous
*******************************************

==> Matching record 3 :
COMMON NAME = Pine
LATIN NAME = Pinus
FOLIAGE TYPE = evergreen
*******************************************

==> Matching record 4 :
COMMON NAME = Willow
LATIN NAME = Salix
FOLIAGE TYPE = deciduous
*******************************************

==> Matching record 5 :
COMMON NAME = Yew
LATIN NAME = Taxis
FOLIAGE TYPE = evergreen
*******************************************

This is a special case where the FIELD names are actually two separate words. The command line arguments can only be one-word, but the script only needs a recognizable tag so here we can just enter the first word from each FIELD as arguments:
OPselect2 COMMON LATIN treesout.txt

and the output should be:

COMMON NAME = Maple
LATIN NAME = Acer

COMMON NAME = Oak
LATIN NAME = Quercus

COMMON NAME = Pine
LATIN NAME = Pinus

COMMON NAME = Willow
LATIN NAME = Salix

COMMON NAME = Yew
LATIN NAME = Taxis

------------------------------------
Attaching a copy of each filter script for those who would like to try them.
Note these are FAKE zip files. Just rename them OPselect2 and Opselect3.

cheers
Attachments
Opselect3.zip
Filter script for selecting THREE fields from a fields.awk Read/Search output file.
Syntax = OPselect3 [Field_selection1] [Field_selection2] [Field_selection3] [filename]
(758 Bytes) Downloaded 190 times
Opselect2.zip
Filter script for selecting TWO fields from a fields.awk Read/Search output file.
Syntax = OPselect2 [Field_selection1] [Field_selection2] [filename]
(664 Bytes) Downloaded 191 times
OPselect_code.jpg
One-liner awk scripts for filtering Read-Search output files.
(72.63 KiB) Downloaded 233 times

User avatar
Pete
Posts: 660
Joined: Sun 02 Mar 2014, 18:36

Re: Zshaolin Special Keys Mapping

#89 Post by Pete »

cabbie0 wrote:@Pete
....
There is has special options for both the CTRL and ESC keys you can select.
In each case the opions (on my version) are: Jog ball, @key, Left Alt, Right Alt, Vol Up, Vol Down, Camera Key, or None.

Hope that helps.
Perfect, thank you.
I thought that the vol up/dn + letter key combo was an automatic Android thing.
I'm such an Android noob. :lol:
Last edited by Pete on Sun 03 Jul 2016, 09:43, edited 1 time in total.

User avatar
Pete
Posts: 660
Joined: Sun 02 Mar 2014, 18:36

#90 Post by Pete »

@cabbie0

Nice addition those two scripts.
A thought, since the syntax after the pipe command is identical for both one-liners, perhaps including a bit of code to detect how many arguments the user supplied and the script would automatically branch to either 2 or 3 fields.
The number of arguments available in $#.

Another idea, I'm thinking of creating a sort of launch script from where one could start fields.awk or addcols or your latest addition.
So this launch script could be a main menu with options something like:

(S) Start fields.awk
(A) Add columns
(E) Extended search - 2 or 3 fields
:
:
:
:
(X) Exit

This way we can add new functions/one-liners easily.

cabbie0
Posts: 26
Joined: Thu 23 Jun 2016, 05:19
Location: British Columbia, Canada

Extensions

#91 Post by cabbie0 »

@Pete
Hi,

Yes I thought about using $# to automatically detect the number of fields, and it could work.

But there is one thing to consider. As is, OPselect2/3 will output to the screen.
If you want to save the output to a file, you'd be adding something like "> [savefilename]" at the end, which would be adding to the command-line argument total.
Could be done, but you'd have to decide beforehand whether the save_output function is default in order to get the argument numbering correct.

As for the other "launch script" idea, yes cool.
I gather the extended features (addcol etc) would automatically be acting upon output files in some common directory?
Mind, filtering can be done on either search-output files as with OPselect, OR it can also be done on the raw database files in DB_FILES storage directory.
That is usually a better option for "addcol" because it's adding the entries in whatever field contains some numerical data.
And you can combine it with a "grep" command to add the numerical data column for only those records that your grep filter defines.

OPselect type filters could be added to the "READ" function of fields.awk, but it would be at the cost of additional prompting every time you wanted to save a search output file...
Hmm.

ttyl

User avatar
Pete
Posts: 660
Joined: Sun 02 Mar 2014, 18:36

#92 Post by Pete »

Hi cabbie0

What I was thinking is using a temp file similar to what you use for searches.
As regards directories, fields.awk stores which directory is used for the DB files, so could use a similar arrangement for where the search files are created.
At the moment they default to ~/Desktop but could also have an option to change it via a fields.awk setup menu option.

It is a bit more work in the beginning but once it's written, it will allow for easy inclusion of more features to fields.awk by way of menu selections which would invoke different scripts or "modules".

Your thoughts?

cabbie0
Posts: 26
Joined: Thu 23 Jun 2016, 05:19
Location: British Columbia, Canada

Output filtering module idea

#93 Post by cabbie0 »

@Pete

Hi,
Yes, that is a possibility alright.
At the end of the read_from function where printing of the search output occurs:

case $PRINTORNOT in
[Yy]*) printf "`cat $OP`\n" | fold -s >> ~/Desktop/$SAVEAS.txt
echo "Your file was saved as $SAVEAS.txt in ~/Desktop.";;

Instead of immediately ending the "case" module with "esac" and returning to "main", you could clear the screen before the "echo" line and then issue a prompt:
echo "Filter output file? "; read REPLY

Only if the answer was Y would further prompts appear for selection of type of filter (addcol, OPselect, etc) and their appropriate sub-prompts and branch codes.
Otherwise the code would return to the main screen as usual.

You'd have to remember to save the "esac" module end (and the double semicolon ;; oops that would require care) for the very end of the submenus...
I won't be doing this immediately ;)

User avatar
Pete
Posts: 660
Joined: Sun 02 Mar 2014, 18:36

#94 Post by Pete »

Thanks for the tips cabbie0.
I will see what I can can up with during my spare time.

cabbie0
Posts: 26
Joined: Thu 23 Jun 2016, 05:19
Location: British Columbia, Canada

Enhanced Fields.awk

#95 Post by cabbie0 »

Hello all,

@Pete
You got me thinking about some good additional functions, and before I knew it I was coding again...

I have added some new functionality to fields.awk.
Feel free to download and tinker with the upgraded version, fields_ext.awk.
This is a real zip archive (.bz2) so you have to unzip it.

The two main additions are:
1) a column-adding feature (function "addcol") added to the main menu (select T for tally/total)
2) a search output filtering option, offered whenever you save a read-search to file.

Also, though the default search-output file folder is still ~/Desktop, you can now easily change that
by editing the line (found at the script beginning just after the preamble):
SEARCH_DIR=~/Desktop
to reflect your desired location. Just make sure it actually exists ;)

A handy short database file (weather_data.txt) for testing the "tally" function:
@@@DATE::mm_RAIN::uV_rads::lightning_strikes::
05-10-16::60::0.89::3::
05-15-16::12::0.54::1::
05-20-16::74::2.1::6::
05-25-16::98::0.34::3::
05-30-16::987::.87::7::
06-05-16::0::.23::0::
06-10-16::45::0::14::
06-15-16::10::.59::1::
06-20-16::10::1.0::2::
06-25-16::36::.10::15::

Select this file and choose "T" from main menu, then select field#2 for adding:
Note it offers an option to filter the tally, that is add the numbers in selected field only for records pertaining to a search string.
In this example I've selected for only the "June" entries by using the search string "06-"

DATE || mm_RAIN || uV_rads || lightning_strikes ||

Which field column would you like to total? (Enter an integer)
2
Total of column#2 = 1332.00
___________________________________________
Try a filtered tally?
y
Enter filter string:
(Default case-insensitive; only those records including the string will be tallied)
06-
The following is your _filtered_ tally result:

Total of column#2 = 101.00
___________________________________________
Hit ENTER to return to main menu ...

==============================================================

For the search output filter function using the same small database, if you save a search file including all the records, then select a filtered output including only DATE and mm_RAIN, the interface looks like this:

Your file was saved as weather.txt in /home/iforsyth/DB_FILES/SEARCHES.
Filter output?
y
DATE || mm_RAIN || uV_rads || lightning_strikes ||

Select # of fields to display (2-4)
2
Enter string for field selection#1:
DATE
Enter string for field selection#2:
mm_RAIN
Your filtered output was saved as weather_f2.txt in /home/iforsyth/DB_FILES/SEARCHES.
Hit ENTER to return to main menu ...


Note: the output filter can handle either 2, 3, or 4 field selections for filtering.
Hope you find these additions useful.
Cheers,
Ian
Attachments
fields_ext.awk.bz2
Enhanced fields.awk, includes "addcol" function plus output filtering.
(4.72 KiB) Downloaded 210 times

User avatar
Pete
Posts: 660
Joined: Sun 02 Mar 2014, 18:36

#96 Post by Pete »

Hi cabbie0

That is fantastic, thank you.
Up to now been adding fields in a separate script which was a bit cumbersome.
New version downloaded and already put to good use. :D

cabbie0
Posts: 26
Joined: Thu 23 Jun 2016, 05:19
Location: British Columbia, Canada

Your Welcome

#97 Post by cabbie0 »

@Pete

Outstanding :)
Glad you like it.

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#98 Post by greengeek »

I haven't used fileds.awk for a while (and don't recall testing the newest version) but now have a need to build a database of customers, addresses, contact numbers, invoice numbers and possibly some other data.

I have two main questions:

1) I know that immediately after building the database the first time I will forget the order of each field - so I wondered about entering data something like this:

NAME,DaveSmith::NUMBER,094123545::ADDRESS,5WilsonStDunedin::INVOICE,457143::VARIABLE1,Holden::VARIABLE2,Thursday::VARIABLE3,whatever

etc, etc

This way every line of the raw file would contain the "definitions" of what each field contains.
Note that there is a comma as a delimiter between NAME and DaveSmith etc - the idea being that if I look at the raw text file I can see exactly what each field is, without referring to the original title divisions. This would be especially helpful when the file is large. It would provide a "visual prompt". There may be times when all I have is the raw txt file and be viewing it on a tablet or just a sheet of paper (without fields.awk being available) so it might be helpful then.

Would entering data in this format upset fields.awk? Would it freak out about the commas? Would I be better using a different delimiter? Could fields.awk be made to ignore the extra info preceeding the comma, or even to insert it automatically into the raw file to enhance "human readability" of that file?

2) There may be times where it would be useful to add multiple entries into a specific field. For example, one customer may be linked to 5 invoice numbers. Could it ever be possible to "stretch" the invoice field in some way that it was "infinitely expandable" and accommodate multiple entries without displacing other fields? (maybe the last field in the line could be an expandable "multifield"???)

User avatar
6502coder
Posts: 677
Joined: Mon 23 Mar 2009, 18:07
Location: Western United States

#99 Post by 6502coder »

@ greengeek: I PM'd you with some questions about your proposals.

cabbie0
Posts: 26
Joined: Thu 23 Jun 2016, 05:19
Location: British Columbia, Canada

Commas and Multifields in fields.awk

#100 Post by cabbie0 »

@greengeek
Hi, just noticed your (almost) recent questions about fields.awk.

As for the idea of automatically inserting the field_name info into every line of the data file, you _could_ make that happen by altering one line in the END section of the write_to() module:
printf ("%s::", fieldvalues) >> datafile } # end for
change to:
printf ("%s,%s::", fieldnames, fieldvalues) >> datafile }

Then when you enter your field values interactively, they would be stored in the manner you suggest.
However I don't recommend it as the read-output would then look like this:
NAME = NAME,DaveSmith
NUMBER = NUMBER,094123545
ADDRESS = ADDRESS,5WilsonStDunedin
INVOICE = INVOICE,457143
VARIABLE1 = VARIABLE1,Holden
VARIABLE2 = VARIABLE2,Thursday
VARIABLE3 = VARIABLE3,whatever

An easier way to produce that human-readable file is to do a read function on your data file, and simply hit ENTER at the search prompt so it spits out the entire database in the form
==> Matching record 1 :
NAME = DaveSmith
NUMBER = 094123545
ADDRESS = 5WilsonStDunedin
INVOICE = 457143
VARIABLE1 = Holden
VARIABLE2 = Thursday
VARIABLE3 = whatever
*******************************************

==> Matching record 2 : etc etc

__________________________________________________________________

As for multiple entries in a single field, that can be done using "\n\t" as separators between the invoice numbers for example. The read output command:
printf "`cat $OP`\n" | fold -s >> $SEARCH_DIR/$SAVEAS.txt

uses the shell command "printf" which interprets the \n\t sequence as newline+tab, resulting in a very readable list of multiple entries in a single field.
In your "invoice" example, let's say you want to enter three invoice numbers
457143\n\t457980\n\t458231
in the INVOICE field when prompted.
Then your output for the INVOICE field would look like this:

INVOICE = 457143
457980
458231

I use this for multiple lines in mailing addresses etc.
If you find "\n\t" cumbersome to type, you can use copy/paste, or if you find they look confusing in the raw file you can add white-space on either side of them like so:
457143 \n\t 457980 \n\t 458231::
and it won't unduly affect the printout.

Hope some of this helps.

Post Reply