cli relational database

For discussions about programming, programming questions/advice, and projects that don't really have anything to do with Puppy.
Post Reply
Message
Author
User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

cli relational database

#1 Post by rufwoof »

[size=75]( ͡° ͜ʖ ͡°) :wq[/size]
[url=http://murga-linux.com/puppy/viewtopic.php?p=1028256#1028256][size=75]Fatdog multi-session usb[/url][/size]
[size=75][url=https://hashbang.sh]echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh[/url][/size]

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#2 Post by musher0 »

In the same line of thought, all Puppies also have sqlite3 and awk.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

s243a
Posts: 2580
Joined: Tue 02 Sep 2014, 04:48
Contact:

#3 Post by s243a »

I'll give this join function a try to see how fast it is:

https://linux.die.net/man/1/join

I want to add md5 sums to the puppy repo data bases.

https://gitlab.com/sc0ttj/Pkg/issues/75#note_246129899

If the join function is fast, then maybe putting this info in a separate table will work well :)
Find me on [url=https://www.minds.com/ns_tidder]minds[/url] and on [url=https://www.pearltrees.com/s243a/puppy-linux/id12399810]pearltrees[/url].

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#4 Post by technosaurus »

Reminded me of this old thread
http://www.murga-linux.com/puppy/viewtopic.php?t=86146
Where I demonstrated 1 way to use your filesystem as a database using only a shell script... just a proof of concept
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

s243a
Posts: 2580
Joined: Tue 02 Sep 2014, 04:48
Contact:

#5 Post by s243a »

s243a wrote:I'll give this join function a try to see how fast it is:

https://linux.die.net/man/1/join

I want to add md5 sums to the puppy repo data bases.

https://gitlab.com/sc0ttj/Pkg/issues/75#note_246129899

If the join function is fast, then maybe putting this info in a separate table will work well :)
I was thinking about this and in theory a join command should be fast if the input data is sorted.
Find me on [url=https://www.minds.com/ns_tidder]minds[/url] and on [url=https://www.pearltrees.com/s243a/puppy-linux/id12399810]pearltrees[/url].

s243a
Posts: 2580
Joined: Tue 02 Sep 2014, 04:48
Contact:

#6 Post by s243a »

technosaurus wrote:Reminded me of this old thread
http://www.murga-linux.com/puppy/viewtopic.php?t=86146
Where I demonstrated 1 way to use your filesystem as a database using only a shell script... just a proof of concept
I thought about doing this but to know where near the same degree as you have implemented. I'll have to check this out later :)
Find me on [url=https://www.minds.com/ns_tidder]minds[/url] and on [url=https://www.pearltrees.com/s243a/puppy-linux/id12399810]pearltrees[/url].

s243a
Posts: 2580
Joined: Tue 02 Sep 2014, 04:48
Contact:

#7 Post by s243a »

As a further thought, I bet you can replace the input files to join with process substitution:

Code: Select all

<<(....)
http://tldp.org/LDP/abs/html/process-sub.html

and if you do this then the input files don't necessarily need to be text files. The input files to the join utility could actually be the output from different databases queries or alternatively we could have a custom binary file that has similar properties to a database such as fast searching and filtering capabilities.
Find me on [url=https://www.minds.com/ns_tidder]minds[/url] and on [url=https://www.pearltrees.com/s243a/puppy-linux/id12399810]pearltrees[/url].

User avatar
gcav
Posts: 104
Joined: Fri 25 May 2012, 04:12
Location: Ontario

One of the first NoSQL's...

#8 Post by gcav »

NoSQL RDBMS

NoSQL is a shell-level relational database management system
for UNIX. Unlike most DBMS'es, NoSQL is not a single large program,
rather it is a set of small programs that are run from the shell.
This allows the user to utilize the full power of the UNIX shell
in coordination with NoSQL.

It uses the Operator/Stream DBMS paradigm described in "Unix
Review", March, 1991, page 24, entitled "A 4GL Language".
There are a number of "operators" that each perform a unique
function on the data. The "stream" is suplied by the UNIX
Input/Output redirection mechanism. Therefore each operator
processes some data and then passes it along to the next
operator via the UNIX pipe function. This tends to be rather
efficient as UNIX pipes are implemented in memory. NoSQL is
compliant with the "Relational Model". Read the document
"4gl.ps", included with the distribution, for further details.

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#9 Post by musher0 »

musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

s243a
Posts: 2580
Joined: Tue 02 Sep 2014, 04:48
Contact:

#10 Post by s243a »

Thankyou for this gcav and musher0. I'm going to have to give it a try :)
Find me on [url=https://www.minds.com/ns_tidder]minds[/url] and on [url=https://www.pearltrees.com/s243a/puppy-linux/id12399810]pearltrees[/url].

User avatar
sc0ttman
Posts: 2812
Joined: Wed 16 Sep 2009, 05:44
Location: UK

#11 Post by sc0ttman »

NoSql tutorial: http://www.troubleshooters.com/lpm/200704/200704.htm
---

Also see this post about csvsql from the csvkit package: https://stackoverflow.com/a/44177446/5479837

And there is Miller: https://github.com/johnkerl/miller

Ans also something really cool called VisiData: https://visidata.org/

(I went to a workshop with the guy who made VisiData, he showed it off, it's great!)

Helpful link of theory of joins, unions etc, in shell: http://matt.might.net/articles/sql-in-the-shell/
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#12 Post by jamesbond »

Thanks for sharing the good find. Really appreciate it. :D
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

step
Posts: 1349
Joined: Fri 04 May 2012, 11:20

#13 Post by step »

https://rbql.org/
...provides SQL-like language for data-transformation and data-analysis queries for structured data (like CSV or TSV files). RBQL query is executed using one of the available general-purpose "backend" languages (currently Python or JavaScript). RBQL allows to use non-SQL expressions inside SQL statements, e.g for JavaScript backend...
[url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Fatdog64-810[/url]|[url=http://goo.gl/hqZtiB]+Packages[/url]|[url=http://goo.gl/6dbEzT]Kodi[/url]|[url=http://goo.gl/JQC4Vz]gtkmenuplus[/url]

User avatar
tallboy
Posts: 1760
Joined: Tue 21 Sep 2010, 21:56
Location: Drøbak, Norway

#14 Post by tallboy »

Thank you sc0ttman, VisiData seems like a real gem!
VisiData's author wrote:I live in Seattle but the terminal is my home.
And that says a lot!
True freedom is a live Puppy on a multisession CD/DVD.

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#15 Post by musher0 »

Hello all.

I've been reading the VisiData docs. What's a "panda" ?
Not the black and white vegetarian bear, I suppose!
First time I've seen this word used in a data context.

TIA
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

User avatar
6502coder
Posts: 677
Joined: Mon 23 Mar 2009, 18:07
Location: Western United States

#16 Post by 6502coder »


musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#17 Post by musher0 »

Thanks, 6502coder.
I.e. a regular grid or matrix, but with 2 axes only and
possibly containing heterogeneous data.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

Post Reply