fields.awk database

[cabbie0]

@Pete
Hi,

Yes I thought about using $# to automatically detect the number of fields, and it could work.

But there is one thing to consider. As is, OPselect2/3 will output to the screen.
If you want to save the output to a file, you'd be adding something like "> [savefilename]" at the end, which would be adding to the command-line argument total.
Could be done, but you'd have to decide beforehand whether the save_output function is default in order to get the argument numbering correct.

As for the other "launch script" idea, yes cool.
I gather the extended features (addcol etc) would automatically be acting upon output files in some common directory?
Mind, filtering can be done on either search-output files as with OPselect, OR it can also be done on the raw database files in DB_FILES storage directory.
That is usually a better option for "addcol" because it's adding the entries in whatever field contains some numerical data.
And you can combine it with a "grep" command to add the numerical data column for only those records that your grep filter defines.

OPselect type filters could be added to the "READ" function of fields.awk, but it would be at the cost of additional prompting every time you wanted to save a search output file...
Hmm.

ttyl

[Pete]

Hi cabbie0

What I was thinking is using a temp file similar to what you use for searches.
As regards directories, fields.awk stores which directory is used for the DB files, so could use a similar arrangement for where the search files are created.
At the moment they default to ~/Desktop but could also have an option to change it via a fields.awk setup menu option.

It is a bit more work in the beginning but once it's written, it will allow for easy inclusion of more features to fields.awk by way of menu selections which would invoke different scripts or "modules".

Your thoughts?

[cabbie0]

@Pete

Hi,
Yes, that is a possibility alright.
At the end of the read_from function where printing of the search output occurs:

case $PRINTORNOT in
[Yy]*) printf "`cat $OP`\n" | fold -s >> ~/Desktop/$SAVEAS.txt
echo "Your file was saved as $SAVEAS.txt in ~/Desktop.";;

Instead of immediately ending the "case" module with "esac" and returning to "main", you could clear the screen before the "echo" line and then issue a prompt:
echo "Filter output file? "; read REPLY

Only if the answer was Y would further prompts appear for selection of type of filter (addcol, OPselect, etc) and their appropriate sub-prompts and branch codes.
Otherwise the code would return to the main screen as usual.

You'd have to remember to save the "esac" module end (and the double semicolon ;; oops that would require care) for the very end of the submenus...
I won't be doing this immediately

[Pete]

Thanks for the tips cabbie0.
I will see what I can can up with during my spare time.

[cabbie0]

Hello all,

@Pete
You got me thinking about some good additional functions, and before I knew it I was coding again...

I have added some new functionality to fields.awk.
Feel free to download and tinker with the upgraded version, fields_ext.awk.
This is a real zip archive (.bz2) so you have to unzip it.

The two main additions are:
1) a column-adding feature (function "addcol") added to the main menu (select T for tally/total)
2) a search output filtering option, offered whenever you save a read-search to file.

Also, though the default search-output file folder is still ~/Desktop, you can now easily change that
by editing the line (found at the script beginning just after the preamble):
SEARCH_DIR=~/Desktop
to reflect your desired location. Just make sure it actually exists

A handy short database file (weather_data.txt) for testing the "tally" function:
@@@DATE::mm_RAIN::uV_rads::lightning_strikes::
05-10-16::60::0.89::3::
05-15-16::12::0.54::1::
05-20-16::74::2.1::6::
05-25-16::98::0.34::3::
05-30-16::987::.87::7::
06-05-16::0::.23::0::
06-10-16::45::0::14::
06-15-16::10::.59::1::
06-20-16::10::1.0::2::
06-25-16::36::.10::15::

Select this file and choose "T" from main menu, then select field#2 for adding:
Note it offers an option to filter the tally, that is add the numbers in selected field only for records pertaining to a search string.
In this example I've selected for only the "June" entries by using the search string "06-"

DATE || mm_RAIN || uV_rads || lightning_strikes ||

Which field column would you like to total? (Enter an integer)
2
Total of column#2 = 1332.00
___________________________________________
Try a filtered tally?
y
Enter filter string:
(Default case-insensitive; only those records including the string will be tallied)
06-
The following is your _filtered_ tally result:

Total of column#2 = 101.00
___________________________________________
Hit ENTER to return to main menu ...

==============================================================

For the search output filter function using the same small database, if you save a search file including all the records, then select a filtered output including only DATE and mm_RAIN, the interface looks like this:

Your file was saved as weather.txt in /home/iforsyth/DB_FILES/SEARCHES.
Filter output?
y
DATE || mm_RAIN || uV_rads || lightning_strikes ||

Select # of fields to display (2-4)
2
Enter string for field selection#1:
DATE
Enter string for field selection#2:
mm_RAIN
Your filtered output was saved as weather_f2.txt in /home/iforsyth/DB_FILES/SEARCHES.
Hit ENTER to return to main menu ...


Note: the output filter can handle either 2, 3, or 4 field selections for filtering.
Hope you find these additions useful.
Cheers,
Ian

[Pete]

Hi cabbie0

That is fantastic, thank you.
Up to now been adding fields in a separate script which was a bit cumbersome.
New version downloaded and already put to good use.

[cabbie0]

@Pete

Outstanding
Glad you like it.

[greengeek]

I haven't used fileds.awk for a while (and don't recall testing the newest version) but now have a need to build a database of customers, addresses, contact numbers, invoice numbers and possibly some other data.

I have two main questions:

1) I know that immediately after building the database the first time I will forget the order of each field - so I wondered about entering data something like this:

NAME,DaveSmith::NUMBER,094123545::ADDRESS,5WilsonStDunedin::INVOICE,457143::VARIABLE1,Holden::VARIABLE2,Thursday::VARIABLE3,whatever

etc, etc

This way every line of the raw file would contain the "definitions" of what each field contains.
Note that there is a comma as a delimiter between NAME and DaveSmith etc - the idea being that if I look at the raw text file I can see exactly what each field is, without referring to the original title divisions. This would be especially helpful when the file is large. It would provide a "visual prompt". There may be times when all I have is the raw txt file and be viewing it on a tablet or just a sheet of paper (without fields.awk being available) so it might be helpful then.

Would entering data in this format upset fields.awk? Would it freak out about the commas? Would I be better using a different delimiter? Could fields.awk be made to ignore the extra info preceeding the comma, or even to insert it automatically into the raw file to enhance "human readability" of that file?

2) There may be times where it would be useful to add multiple entries into a specific field. For example, one customer may be linked to 5 invoice numbers. Could it ever be possible to "stretch" the invoice field in some way that it was "infinitely expandable" and accommodate multiple entries without displacing other fields? (maybe the last field in the line could be an expandable "multifield"???)

[6502coder]

@ greengeek: I PM'd you with some questions about your proposals.

[cabbie0]

@greengeek
Hi, just noticed your (almost) recent questions about fields.awk.

As for the idea of automatically inserting the field_name info into every line of the data file, you _could_ make that happen by altering one line in the END section of the write_to() module:
printf ("%s::", fieldvalues) >> datafile } # end for
change to:
printf ("%s,%s::", fieldnames, fieldvalues) >> datafile }

Then when you enter your field values interactively, they would be stored in the manner you suggest.
However I don't recommend it as the read-output would then look like this:
NAME = NAME,DaveSmith
NUMBER = NUMBER,094123545
ADDRESS = ADDRESS,5WilsonStDunedin
INVOICE = INVOICE,457143
VARIABLE1 = VARIABLE1,Holden
VARIABLE2 = VARIABLE2,Thursday
VARIABLE3 = VARIABLE3,whatever

An easier way to produce that human-readable file is to do a read function on your data file, and simply hit ENTER at the search prompt so it spits out the entire database in the form
==> Matching record 1 :
NAME = DaveSmith
NUMBER = 094123545
ADDRESS = 5WilsonStDunedin
INVOICE = 457143
VARIABLE1 = Holden
VARIABLE2 = Thursday
VARIABLE3 = whatever
*******************************************

==> Matching record 2 : etc etc

__________________________________________________________________

As for multiple entries in a single field, that can be done using "\n\t" as separators between the invoice numbers for example. The read output command:
printf "`cat $OP`\n" | fold -s >> $SEARCH_DIR/$SAVEAS.txt

uses the shell command "printf" which interprets the \n\t sequence as newline+tab, resulting in a very readable list of multiple entries in a single field.
In your "invoice" example, let's say you want to enter three invoice numbers
457143\n\t457980\n\t458231
in the INVOICE field when prompted.
Then your output for the INVOICE field would look like this:

INVOICE = 457143
457980
458231

I use this for multiple lines in mailing addresses etc.
If you find "\n\t" cumbersome to type, you can use copy/paste, or if you find they look confusing in the raw file you can add white-space on either side of them like so:
457143 \n\t 457980 \n\t 458231::
and it won't unduly affect the printout.

Hope some of this helps.