Argument list too long .. [SOLVED]

[sc0ttman]

Background: I wanna import all the IceCast radio stations into an existing list of favourite streams, in VLC-GTK...

I posted previously in this Programming thread about formatting the IceCast xml list.. Well, that's done (thanks guys), and the list is ready to be imported/added to an existing combobox list, in VLC-GTK...

But, cos the IceCast list is about 3MB (10,000 lines), I cant add the new stuff to the 'fave streams' list in VLC-GTK, without getting the 'argument list too long' error...

The list of streams I wanna add to, is the file ${HOME}/.vlc-gtk/vlc-gtk-faves

It contains lines all of the format

NAME|URL

(Anything with a pipe in the middle would do to test the code below... )

Once this list gets to 1,700 lines long, then VLC-GTK starts to bug out, giving 'Argument list too long' all over the place... Under 1,700 and it just takes a while to load, but no errors...

I've tried to condense down the relevant code into an example below:

(the code was originally for a list of about 12 mms:// TV stations... needs updating me thinks)

Code: Select all

FAVES=${HOME}/.vlc-gtk/vlc-gtk-faves #the list of streams ... each line in format NAME|URL

# build the list for a gtkdialog combobox ..  <item>$LINE</item>
while read LINE
do  
	if [ "${LINE%%|*}" != "$FAVE_STREAM" ];then # dont include the last played stream in the list yet, add it top top later
	    FAVE_ITEMS="$FAVE_ITEMS
<item>${LINE%%|*}</item>" #get only fave names for combobox list
	fi
done <"$FAVES"

FAVE_ITEMS="$(echo "$FAVE_ITEMS" | sort)" #sort favourite streams

# add last played stream to top of list
if [ "$FAVE_STREAM" != "" ];then
	[ "$(cat $FAVES | grep "${FAVE_STREAM}|")" != "" ] && FAVE_ITEMS="<item>$FAVE_STREAM</item>
${FAVE_ITEMS}"  
fi

and later in the gtkdialog code of the main VLC-GTK GUI:

Code: Select all

<combobox width-request="295">
	<variable>FAVE_STREAM_URL</variable>
	'$FAVE_ITEMS'
</combobox>

..and lastly, here's the (very crappy, inefficient) code that finds the URL later, once the user has chosen from the list.. I've not even got to this bit yet, to test it with over 1,700 lines, but it'll be relevant..

Code: Select all

#FAVE_STREAM_URL is the name of the stream chosen by the user
while read LINE; do
	case $LINE in
		"${FAVE_STREAM_URL}|"*) FAVE_STREAM_URL="${LINE##*|}"; NAME="${LINE%%|*}" # get name and URL #040812 fix case selection
	esac
done<${HOME}/.vlc-gtk/vlc-gtk-faves

How can I rid myself of these 'Argument list too long' errors when the stream list is over 2,000 lines long? And ideally, speed up the code above a little bit at the same time..

[SFR]

Hey Sc0ttman

If it comes for large amount of data, It's better to use temp files instead of variables, both for building <item> index as well as Gtkdialog GUI.

I'd do it like this:

First, I created 10,000 items using:

Code: Select all

for i in {1..10000}; do
  echo "Some_Name"$RANDOM"|www.some_url"$i >> $HOME/.vlc-gtk/vlc-gtk-faves
done

and came up with this (though I'm not saying it's the best way; I'm sure there are better):

Code: Select all

#! /bin/bash

# Get name and URL (it's a function called via <button>, just for testing purposes)
user_choice () {
  NAME="`grep "^$FAVE_STREAM_URL|" $HOME/.vlc-gtk/vlc-gtk-faves`"
  xmessage "$NAME"
}
export -f user_choice

# -----------------------------------------------------------------------------

FAVES=${HOME}/.vlc-gtk/vlc-gtk-faves #the list of streams ... each line in format NAME|URL 

# Build index for <combobox> but exclude $FAVE_STREAM (if exists)
cat "$FAVES" | awk 'BEGIN {FS="|"}; {print "<item>"$1"</item>"}' | grep -v "^<item>$FAVE_STREAM</item>$" | sort > /tmp/templist

# add last played stream to top of list 
if [ "$FAVE_STREAM" != "" ];then 
   echo "<item>$FAVE_STREAM</item>" > /tmp/templist_final
else
   echo -n > /tmp/templist_final
fi

cat /tmp/templist >> /tmp/templist_final

# Build GUI as a file, not a variable
echo '<vbox>
  <combobox width-request="295"> 
    <variable>FAVE_STREAM_URL</variable> 
    '"$(cat /tmp/templist_final)"'
  </combobox>
  <button><label>Show chosen item and its URL</label>
    <action>user_choice</action>
  </button>
</vbox>
' > /tmp/temp_GUI

gtkdialog -f /tmp/temp_GUI

It's hard to say if/how this will work ok for you, since I haven't seen the original list, but I hope it'll help you a bit, somehow.

BTW: So many items makes the GUI really unresponsive (at least initially) - I've learned the lesson while writing IconFinder.

____________

EDIT: Even simpler, much more simpler:

Code: Select all

FAVES=${HOME}/.vlc-gtk/vlc-gtk-faves #the list of streams ... each line in format NAME|URL 

# Build GUI as a file, not a variable
echo '<vbox>
  <combobox width-request="295"> 
    <variable>FAVE_STREAM_URL</variable>
    
'"$([ "$FAVE_STREAM" != "" ] && echo "<item>$FAVE_STREAM</item>")"'
'"$(awk 'BEGIN {FS="|"}; {print "<item>"$1"</item>"}' $FAVES | grep -v "^<item>$FAVE_STREAM</item>$" | sort)"'

  </combobox>
  <button><label>Show chosen item + URL</label>
    <action>xmessage "`grep "^$FAVE_STREAM_URL|" $HOME/.vlc-gtk/vlc-gtk-faves`"</action>
  </button>
</vbox>
' > /tmp/temp_GUI

gtkdialog -f /tmp/temp_GUI

Greetings!

[sc0ttman]

Hi SFR,

I've yet to get round to testing all this in VLC-GTK fully, but your update looks great, I just ran it with 8,000+ lines.. Much simpler than what I came up with! I see the large delay in first opening the combobox, but that's expected..

Thanks for the help, that's great.