Puppy Database

Under development: PCMCIA, wireless, etc.
Message
Author
User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#31 Post by BarryK »

Komodo, it doesn't have to be made into a single PET package. When Pup 2.17 is released, the TkSqlite PET package will be in the selection list and the information on dependencies is also there, so all those dependent packages will automatically get installed as well.
Or even if someone downloaded 'tksqlite-xxx.pet', the dependency info is in Puppy and also they will get auto-installed.

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#32 Post by BarryK »

How about this? Hop is a language for creating gui applications viewable in a web browser. It uses sqlite as a backend, so is effectively a tool for creating databases that can be served over the web. Hop is also intended as a replacement for the traditional gui widget tools (like GTK, Qt) for local applications.

Main page:
http://hop.inria.fr/#home-notepad=np:0

Technical details:
http://hop.inria.fr/usr/local/share/hop ... ticle.html

...it's not a gui database-creator-manager thingy though, which is what we have been discussing.

IvanF
Posts: 15
Joined: Mon 12 Dec 2005, 14:11
Location: W. Australia

PupBase

#33 Post by IvanF »

I can across NoSQL the other day.

Home page
http://www.scriptaworks.com/cgi-bin/wiki.cgi/NoSQL

A Great Tutorial
http://www.troubleshooters.com/lpm/200704/200704.htm

Another older tutorial
http://www2.linuxjournal.com/article/3294

NoSQL looks like it is built for Puppy. Somone with better unix skills than me may be able to create PupBase with it.

regards Ivan F

raffy
Posts: 4798
Joined: Wed 25 May 2005, 12:20
Location: Manila

interesting

#34 Post by raffy »

Both are very interesting, especially for database use in the future.

User avatar
JohnMurga
Site Admin
Posts: 555
Joined: Wed 04 May 2005, 04:26
Location: Far to the east
Contact:

#35 Post by JohnMurga »

Dunno if it'll be of interest, but I knocked up a quick tool for personal use ...

http://www.murga-projects.com/forum/sho ... hp?tid=118



Cheers
JohnM
Attachments
sqLiteMngr.PNG
(11.57 KiB) Downloaded 2037 times

User avatar
Springer
Posts: 52
Joined: Tue 22 Aug 2006, 16:25
Location: Austin, TX

#36 Post by Springer »

I second IvanF's recommendation to consider NoSQL as a potential solution. It's worth a look because of the way it fundamentally changes the way databases are built to leverage the power of common Unix text processing tools and scripting. Tables are regular ASCII files SQL is replaced by shell commnads that perform operations on the data through regular unix pipes, tees, etc. This lets you use *anything* to perform special manipulations: For example, there is one implementation (Starbase) of a stream/operator database (the technical name for "/rdb-like" databases) of this idea widely used by astronomers that includes special astronomical data operators. Speed is not the problem you expect, especially with RAMdisks (and mawk)!

To really understand this, you need to read the Wolf and Schaeffer paper on "The Unix Shell as a 4GL" (http://www.rsw.com/lib/4gl.pdf). For more info, check out the Jorgensen, Schaeffer, and Manis book on /rdb.

I designed a state-of-the-art configuration management system based on this technology for a Fortune 20 company in 1993, and the idea has only become more practical and powerful with time. The power of this idea isn't apparent until you use it - the leverage is awesome, sometimes allowing several lines of script code to replace hundreds of lines of SQL or other alternatives. It might be a good idea to plumb this concept into tcl, but I haven't really thought much about how to best do that...

In addition, I think there could be some interesting synergies between NoSQL or a similar stream/operator database and ROX, which embraces the Unix file metaphor. Imagine a set of operators in ROX that could be strung together to build apps.... RSW tried this in the early 1990's with a product they never finished called /rdbX (or /Xrdb) - it was a graphical pipeline builder that ran under OpenLook that basically built scripts to perform database operations. The nice thing about this approach is that scripting expertise is considerably more common than database expertise, which might make it a doubly good option for a distro like Puppy.

Oh, and BTW, Carlo Strozzi, the author of NoSQL, also has written CSA, the CGI Services Architecture to allow stream/operator or other command-line processors to be exposed as CGI services. (see http://www.scriptaworks.com/cgi-bin/wiki.cgi/CSA/CSA)

User avatar
WhoDo
Posts: 4428
Joined: Wed 12 Jul 2006, 01:58
Location: Lake Macquarie NSW Australia

#37 Post by WhoDo »

Springer wrote:I second IvanF's recommendation to consider NoSQL as a potential solution. It's worth a look because of the way it fundamentally changes the way databases are built to leverage the power of common Unix text processing tools and scripting.
Wow! That does sound interesting. Wouldn't it be a real coup for Puppy if it included this capacity by default? Imagine the capability it would give this minimal distro! Please, Barry, someone, anyone, consider this option.... 8)
[i]Actions speak louder than words ... and they usually work when words don't![/i]
SIP:whodo@proxy01.sipphone.com; whodo@realsip.com

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#38 Post by BarryK »

I've downloaded nosql, I'll check it out.

Yes, I agree, Unix/Linux shell scripts can be very powerful, and I have had occasion to rewrite a shell script in C and it was really awkward to do... I didn't bother to complete it.

User avatar
JohnMurga
Site Admin
Posts: 555
Joined: Wed 04 May 2005, 04:26
Location: Far to the east
Contact:

#39 Post by JohnMurga »

Springer wrote:I designed a state-of-the-art configuration management system based on this technology for a Fortune 20 company in 1993, and the idea has only become more practical and powerful with time. The power of this idea isn't apparent until you use it - the leverage is awesome, sometimes allowing several lines of script code to replace hundreds of lines of SQL or other alternatives. It might be a good idea to plumb this concept into tcl, but I haven't really thought much about how to best do that...
It's is funny, I messed around with noSql and it too reminded me of a system I wrote around 1993 (on SCO Xenix), however, it is a fundamentally different paradigm ...
As far as I am concerned it doesn't compare or compete - If you want a relational database, you use one.

If you want an infinitely flexible meta-data system ... Then ... Err ...

Maybe that's the reason that as much as noSql has been around for A LONG time I haven't yet seen benchmarks comparing it to anything else ... I have seen a lot of claims and jingo-like statements, but nothing else.

Either way I would have thought this whole thing is a no brainer as even many unix tools and languages are turning to sqLite to store their structured data. So having some cross over might be nice ...
Springer wrote:Oh, and BTW, Carlo Strozzi, the author of NoSQL, also has written CSA, the CGI Services Architecture to allow stream/operator or other command-line processors to be exposed as CGI services. (see http://www.scriptaworks.com/cgi-bin/wiki.cgi/CSA/CSA)
I would have thought it's obvious why that's a bad idea (a bit like CGI itself). ;-)

Execution out of a sandbox, forking for web-requests, etc, etc ...

Cheers
JohnM

User avatar
Lobster
Official Crustacean
Posts: 15522
Joined: Wed 04 May 2005, 06:06
Location: Paradox Realm
Contact:

#40 Post by Lobster »

20k lo-tech flat file console database
http://www.murga-linux.com/puppy/viewto ... 053&t=1247

I think that is it here . . .
http://www.ajwells.uklinux.net/idexdb.html

m m m we might have room for two databases . . .

8)
Last edited by Lobster on Fri 10 Apr 2009, 11:30, edited 1 time in total.
Puppy Raspup 8.2Final 8)
Puppy Links Page http://www.smokey01.com/bruceb/puppy.html :D

User avatar
Springer
Posts: 52
Joined: Tue 22 Aug 2006, 16:25
Location: Austin, TX

#41 Post by Springer »

JohnM:

Just to clear up a misconception, NoSQL and its relatives (rdb, /rdb, starbase, etc.) ARE real relational databases. That's the idea - each of their application environments provides relational operators that work out-of-the-box on the tables, which happen to be (usually) TSV text files, sometimes with header lines.

IMHO, every well-read Unix hacker should at least have read this paper to see how Unix (the toolset, more than the OS proper) can be leveraged as an app platform, not just for underlying OS support: The Unix Shell as a Fourth Generation Language.

Stream/Operator databases will never flat outperform "real database engines", but they offer very real advantages, and in today's world, where the penalty for interpreted code is trivial and huge, fast harddisks and RAMdisks are dirt-cheap, the approach makes more sense than ever.

FWIW, here's a (slightly edited) post I made to the NoSQL mailing list a while back, when one of the new guys was wondering how to present NoSQL to his LUG as an alternative to "regular" SQL databases:
Well, Evan Schaffer's paper "The Unix shell as a 4GL" is a great place to start. Mainly, the advantage is not so much the database itself (DBs are just a way to hold and find stuff, after all) - it's the "stream-operator" paradigm that a database like /rdb, Starbase, or NoSQL makes possible.

This approach gives you awesome modularity and leverage, and lets non-DB people write serious DB apps without ever having to know all the arcane crap about how SQL works.

Further, in today's world where Gigabytes of RAM and RAMdisks are everywhere, speed isn't really the concern most people think it is. 4GB is actually a really LARGE database, and I'll stack up a RAMdisk against the latest snazzy FibreChannel SAN array any day! Another nice thing about using a filesystem-based table store is that you don't even need BLOBs: BLOBs are represented as the files they started out as - just drop them into the regular disk-based filesystem! If you like to work, you can keep a pointer to the BLOB's binary file in your tables, but even that's often not necessary if the directory structure and file naming scheme has been thought about very much.

Generally, people who say they need speed in databases need it to try to make up for extremely poor platform and/or application architectures - unfortunately, many of today's web apps fall into that category. Screaming DB speed isn't usually necessary, though - *good* design won't beat on the DB that hard in the first place. (A couple of years ago, I designed a high-performance deeply embedded web application server finely tuned for real-world I/O. Even though it ran on an *8-bit* microcontroller, with only about 3 KILObytes of RAM, it easily outperforms its Pentium-based competitors. That's not BS, it's true - good architecture really does make that much difference, and most modern, popular architectures are just flat *awful* from a performance point-of-view.)

Another reason this is an interesting idea: a number of years ago (literally around the dawn of Linux), RSW (Evan Schaffer's company, which wrote the commercial /rdb database that pretty much originated this whole idea) sold a product called Xrdb - a graphical (X/OpenWindows) interface to stream/operator programming. We bought a copy along with our /rdb license, and even though they kinda ripped us off becasue they never really finished the product, it was heading in a really cool direction - think of Yahoo's new Pipes, but building unix pipelines based on rdb (and other) operators, and you've got the idea. Of course, you could have separate scripts to perform certain data transformations (stored procedures, anyone?) that you could plug into the stream wherever you needed them, for stripping out phone numbers, formatting a report, whatever. Like I said, really cool, and it's about time somebody tried to build something like that again...

Really, I see stream-operator databases like NoSQL as mostly orthogonal to SQL-based RDBMSes, as both are orthogonal to specialized hybrid transactional systems such as Intersystems' M/Cache - they have different strengths, and there are good reasons for choosing each. Sadly, despite great potential, the rdb approach is the one that's still waiting for the world to notice - so go ahead and give a presentation or three...

(BTW, Carlo Strozzi, the author of NoSQL, replyed in enthusiastic agreement. He also recommended the following link on the effectiveness over efficiency approach of using text-based data: http://aplawrence.com/Opinion/textvsbinary.html)

User avatar
WhoDo
Posts: 4428
Joined: Wed 12 Jul 2006, 01:58
Location: Lake Macquarie NSW Australia

#42 Post by WhoDo »

JohnMurga wrote:As far as I am concerned it doesn't compare or compete - If you want a relational database, you use one.
Earlier in this thread I expressed what I thought we needed for Puppy, long and short term. Sqlite3, TkSqlite, your GUI frontend ... all offer a good short term solution for those familiar with SQL or not needing anything more powerful than an address book creator.

OTOH, long term we need an easy-to-use, fully relational database that can compete with 80% or more of what Microsoft Access can do and yet can be used by Window$ refugees who might not be able to use even Access but still need to create medium to large datasets. Something that does what FileMaker Pro does in Window$ and Mac environments. I believe GLOM is the answer for that, but compiling it for Puppy is a major task that is presently beyond my meager capabilities. What it offers is the ability to create SQL tables, forms and reports without knowing a single SQL command.

If someone of your stature and experience were to attempt it, we may stand a chance of success. Komodo stands willing to help. So do I. Maybe we could make this a new Puppy project. True relational database capability that is easy to use is all that stands between many Window$ users and a complete break from the Micro$oft stranglehold. What do you think?
[i]Actions speak louder than words ... and they usually work when words don't![/i]
SIP:whodo@proxy01.sipphone.com; whodo@realsip.com

DavidBell
Posts: 132
Joined: Fri 24 Nov 2006, 21:44

#43 Post by DavidBell »

WhoDo wrote:OTOH, long term we need an easy-to-use, fully relational database that can compete with 80% or more of what Microsoft Access...
I have never found an easy to use GUI based relational database for any OS. They all seem really easy to use until you try and do something not directly supported in the wizards. But I readily admit to being no expert on the subject and have very limited experience.... maybe Filemaker is the holy grail that I haven't tried. At the moment I use Access97 because it works and seems pretty sable, however trying to do some things in it can require a hell of a lot of hoop jumping. It's not intuitive, but in the end it works.

Higher up in this thread, I bagged OOo Base a bit because for me at least it, uuhhmm ... didn't work (and JRE is too big for basic puppy IMO). So the difference is Access is not that great or incredibly usable, it's just that it works.

So as I see it, sqllite engine is (already?) in puppy, it's the UI that's missing. Something that makes tables, queries, and reports just a little bit simpler is all that's needed. It doesn't have to be a complete solution, just a few integrated tools that make life a bit easier.

I'm not sure which of the packages mentioned higher up best suits this, but I think stability is more important than features with databases, because people with ???K records get spooked by even the hint of a failure. If you are targeting windows refugeees, then Access drivers are good, or something that can be used in both OSs and has data that can be manipulated in Access.

My 2c x 10

DB

User avatar
JohnMurga
Site Admin
Posts: 555
Joined: Wed 04 May 2005, 04:26
Location: Far to the east
Contact:

#44 Post by JohnMurga »

Springer wrote:JohnM:

Just to clear up a misconception, NoSQL and its relatives (rdb, /rdb, starbase, etc.) ARE real relational databases. That's the idea - each of their application environments provides relational operators that work out-of-the-box on the tables, which happen to be (usually) TSV text files, sometimes with header lines.
NO MISCONCEPTION my friend ;-)

I did read the docs and I am acutely familiar with your argument ...
I even worked on a few systems using this paradigm in the early 90s.

And ... Incidentally I found the following article very useful too :

http://www.troubleshooters.com/lpm/200704/200704.htm

BUT ...

It's a bit like calling my calculator a "Personal Computer" ...

In a way it is :-)

Just like you may be able to "pretend" noSql is a relational database by using the "sorttable/jointable" commands and the like, but at the end of the day it's missing fundamental relational database concepts like integrity constraints (most basically referential integrity) ...

As an example ... Foreign keys, the commonly accepted definition, is that these are integrity constraints enforcing that the value of the attribute set is drawn from a candidate key in another relation. This one of many things that noSql doesn't bother with, because to enforce you have to control ... And that's exactly what it DOESN'T WANT to do (well, as it happens you can't even define, but anyway). It breaks the relational paradigm, sorry ... The database itself is incapable of representing the relationships, you have to do it with code - The database itself is not relational.

And you can do it ... In code ... And its ugly ...

Furthermore in my tests it was pretty expensive to join data anyway.

To anyone that says noSql is relational I'd say :
  • * Referential integrity is explicitly not supported.
    * Nulls aren't treated uniformly.
    * Views can exist, but only as programs, not as 'virtual tables'.
    * Forget about data integrity, "data independence" and integrity constraints.
    * You have to relate data in code.
I guess you could call it quasi-relational ... :-)

And you can do neat stuff with it ...

I have nothing against noSql, but if you pitch it as something it isn't it'll fall flat on it's face as anything else would under the circumstances.

Cheers
John de Murga

User avatar
WhoDo
Posts: 4428
Joined: Wed 12 Jul 2006, 01:58
Location: Lake Macquarie NSW Australia

#45 Post by WhoDo »

DavidBell wrote:.... maybe Filemaker is the holy grail that I haven't tried.
Oh, it definitely is, and it beats Access 97 hands down, but it isn't available for Linux. I've used it for years and I've had it running under Wine but that's not the same thing.

I don't know how much like FileMaker is GLOM, but it looks interesting. I'd like to see someone with more knowledge than me compile it for Puppy. Then we might have a serious contender. Until then, we'll never know.
[i]Actions speak louder than words ... and they usually work when words don't![/i]
SIP:whodo@proxy01.sipphone.com; whodo@realsip.com

User avatar
JohnMurga
Site Admin
Posts: 555
Joined: Wed 04 May 2005, 04:26
Location: Far to the east
Contact:

#46 Post by JohnMurga »

WhoDo wrote:
JohnMurga wrote:As far as I am concerned it doesn't compare or compete - If you want a relational database, you use one.
Earlier in this thread I expressed what I thought we needed for Puppy, long and short term. Sqlite3, TkSqlite, your GUI frontend ... all offer a good short term solution for those familiar with SQL or not needing anything more powerful than an address book creator.

OTOH, long term we need an easy-to-use, fully relational database that can compete with 80% or more of what Microsoft Access can do and yet can be used by Window$ refugees who might not be able to use even Access but still need to create medium to large datasets. Something that does what FileMaker Pro does in Window$ and Mac environments. I believe GLOM is the answer for that, but compiling it for Puppy is a major task that is presently beyond my meager capabilities. What it offers is the ability to create SQL tables, forms and reports without knowing a single SQL command.

If someone of your stature and experience were to attempt it, we may stand a chance of success. Komodo stands willing to help. So do I. Maybe we could make this a new Puppy project. True relational database capability that is easy to use is all that stands between many Window$ users and a complete break from the Micro$oft stranglehold. What do you think?
Flattery will get you nowhere :-)

GLOM looks pretty amazing, but it's a bit heavy for Puppy (very), but the way Puppy's going you never know ...

My GUI front end is a demo for murgaLua that is constantly evolving ... And more gui like features are on the way. However, it is only ever going to support sqLite and it is a tiny part of a larger project with priorities that come from my main users.

As as far as my involvement with anything other than murgaLua at this point ... The fact remains that I've been homeless for a LONG TIME. I have found a house and will be moving in a couple of weeks, so once I have regular net access and don't have to do things from the hostels I've been staying in you might see a bit of more of me around here.

There are a LOT of things I want to do to Puppy :-)

Cheers
JohnM

User avatar
WhoDo
Posts: 4428
Joined: Wed 12 Jul 2006, 01:58
Location: Lake Macquarie NSW Australia

#47 Post by WhoDo »

JohnMurga wrote:Flattery will get you nowhere :-)
Ar, rats! :P
JohnMurga wrote:GLOM looks pretty amazing, but it's a bit heavy for Puppy (very), but the way Puppy's going you never know ...
Yep. That PostgreSQL backend is the killer. Something similar for sqLite would be nice though .... (hint, hint, hint) :wink:
JohnMurga wrote:My GUI front end is a demo for murgaLua that is constantly evolving ... And more gui like features are on the way. However, it is only ever going to support sqLite and it is a tiny part of a larger project with priorities that come from my main users.
Well it wouldn't take much to bring up to a similar standard to TksqLite, would it? At this point I'm for anything that offers an excuse for murgaLua to find its way into Puppy. That would mean we can have the u beaut Control Panel that dvw86 put together some time back:
http://www.murga-linux.com/puppy/viewto ... 53&t=15931
JohnMurga wrote:As as far as my involvement with anything other than murgaLua at this point ... The fact remains that I've been homeless for a LONG TIME. I have found a house and will be moving in a couple of weeks, so once I have regular net access and don't have to do things from the hostels I've been staying in you might see a bit of more of me around here.

There are a LOT of things I want to do to Puppy :-)
Ooohh! Good news! I'm really glad you've found a place to call home. Are you sure I can't sweet talk you into writing a complete GUI frontend for sqLite that a newbie could use? :wink: :roll:
[i]Actions speak louder than words ... and they usually work when words don't![/i]
SIP:whodo@proxy01.sipphone.com; whodo@realsip.com

User avatar
JohnMurga
Site Admin
Posts: 555
Joined: Wed 04 May 2005, 04:26
Location: Far to the east
Contact:

#48 Post by JohnMurga »

WhoDo wrote:
JohnMurga wrote:GLOM looks pretty amazing, but it's a bit heavy for Puppy (very), but the way Puppy's going you never know ...
Yep. That PostgreSQL backend is the killer. Something similar for sqLite would be nice though .... (hint, hint, hint) :wink:
And Python, otherwise you "could" try to replace the database backend.
WhoDo wrote:
JohnMurga wrote:My GUI front end is a demo for murgaLua that is constantly evolving ... And more gui like features are on the way. However, it is only ever going to support sqLite and it is a tiny part of a larger project with priorities that come from my main users.
Well it wouldn't take much to bring up to a similar standard to TksqLite, would it? At this point I'm for anything that offers an excuse for murgaLua to find its way into Puppy. That would mean we can have the u beaut Control Panel that dvw86 put together some time back:
http://www.murga-linux.com/puppy/viewto ... 53&t=15931
It has surprised me that murgaLua is a core part of DSL but completely unused in Puppy, but it doesn't bother me - Once I hit my 1.0 functionality I'll concentrate on apps, and them maybe someone here will be interested.

As for TksqLite ... I am the wrong person as I have been biased against anything TCL based since before Puppy (as I had to some professional work with it some time back and didn't enjoy it).
WhoDo wrote:
JohnMurga wrote:As as far as my involvement with anything other than murgaLua at this point ... The fact remains that I've been homeless for a LONG TIME. I have found a house and will be moving in a couple of weeks, so once I have regular net access and don't have to do things from the hostels I've been staying in you might see a bit of more of me around here.

There are a LOT of things I want to do to Puppy :-)
Ooohh! Good news! I'm really glad you've found a place to call home. Are you sure I can't sweet talk you into writing a complete GUI frontend for sqLite that a newbie could use? :wink: :roll:
That's something that will probably happen at some point, just don't expect it too soon as it is a small priority in a big world ;-)

Cheers
JohnM

User avatar
WhoDo
Posts: 4428
Joined: Wed 12 Jul 2006, 01:58
Location: Lake Macquarie NSW Australia

#49 Post by WhoDo »

JohnMurga wrote:It has surprised me that murgaLua is a core part of DSL but completely unused in Puppy, but it doesn't bother me - Once I hit my 1.0 functionality I'll concentrate on apps, and them maybe someone here will be interested.
I already am. How far away is that 1.0 functionality? We have guys here that are itching to get their teeth into programming applications for Puppy, but the breadth of development environments available seems to put off all but the most hardy.

Once I saw what dvw86 was able to quickly achieve with murgaLua, I was sold. It is only a matter of enthusing the "market". If what was done was designed to be compatible with IceWM, then I could introduce it through EZpup without impacting what happens in the mainstream version.

Frankly, I wish I knew more of programming ... especially in murgaLua. I'd have a lash at it myself. Unfortunately, my limited past experience with C and Basic barely qualifies me to write Hello World these days.

Keep plugging away John. Anything good is worth waiting for... just ask my wife! :wink:
[i]Actions speak louder than words ... and they usually work when words don't![/i]
SIP:whodo@proxy01.sipphone.com; whodo@realsip.com

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#50 Post by BarryK »

At this point I'm for anything that offers an excuse for murgaLua to find its way into Puppy.
MurgaLua is in Puppy, but as John says, it's not actually being used for anything ...yet. Looking at it from a fairly tough admin point of view, it would have to ultimately justify its inclusion, like some irresistable application that needs it.

Post Reply