Slow Searching - Revision history

Dave: /* Why is searching and sorting (in the 32bit version) slow? */

2024-06-20T22:29:06Z

Why is searching and sorting (in the 32bit version) slow?

← Older revision		Revision as of 22:29, 20 June 2024
Line 1:		Line 1:
	==Why is searching and sorting (in the 32bit version) ~~slow~~?==		==Why is searching and sorting slow (in the 32bit version)?==

	Well, frankly it's cuz I am not using a super smart database in the back end. You see, when I started this project, i just whipped something together as fast as I could, so I could see results quickly. In hindsight, it would have been worth my time to invest in understanding how to program SQLite, rather than rolling my own. SQL has these blazing fast indexes that make sorting and searching fast enough to update as you type, even for a library with 100,000 songs in it. My implementation, however, is only fast enough if you have say 10,000 songs. But even this bogs down when the meta data is not cached, for example the "Label" field is stored only in xml files next to the song files, that data is not stored in the database proper, so when you go to sort by that, kJams must load every single xml file for every song.		Well, frankly it's cuz I am not using a super smart database in the back end. You see, when I started this project, i just whipped something together as fast as I could, so I could see results quickly. In hindsight, it would have been worth my time to invest in understanding how to program SQLite, rather than rolling my own. SQL has these blazing fast indexes that make sorting and searching fast enough to update as you type, even for a library with 100,000 songs in it. My implementation, however, is only fast enough if you have say 10,000 songs. But even this bogs down when the meta data is not cached, for example the "Label" field is stored only in xml files next to the song files, that data is not stored in the database proper, so when you go to sort by that, kJams must load every single xml file for every song.

Dave: /* How to Speed Up Searches */

2024-06-20T22:28:40Z

How to Speed Up Searches

← Older revision		Revision as of 22:28, 20 June 2024
Line 5:		Line 5:
	So, in the 64bit version, i've removed my old, creaky, written-from-the-seat-of-my-pants database, and replaced it with one based on SQLite. In the 64bit version, all the data is in one place, and all those progress bars have just "gone away". Keep in mind however that this was is a MAJOR re-write of a huge portion of my code, so there ARE new bugs because of this.		So, in the 64bit version, i've removed my old, creaky, written-from-the-seat-of-my-pants database, and replaced it with one based on SQLite. In the 64bit version, all the data is in one place, and all those progress bars have just "gone away". Keep in mind however that this was is a MAJOR re-write of a huge portion of my code, so there ARE new bugs because of this.

	==How to Speed Up Searches==		==How to Speed Up Searches (in the 32bit version)==
	A thing you can do in the mean time to speed up your searches is to hide any columns you don't want to search. The more columns you have showing, the slower the search will go. To show or hide the search columns, just [[Context Click]] in the [[Sorting\|column header]] and pick the one you want to show or hide.		A thing you can do in the mean time to speed up your searches is to hide any columns you don't want to search. The more columns you have showing, the slower the search will go. To show or hide the search columns, just [[Context Click]] in the [[Sorting\|column header]] and pick the one you want to show or hide.

Dave: /* Why is searching and sorting (in the 32bit version) slow? */

2024-06-20T22:28:03Z

Why is searching and sorting (in the 32bit version) slow?

← Older revision		Revision as of 22:28, 20 June 2024
Line 3:		Line 3:
	Well, frankly it's cuz I am not using a super smart database in the back end. You see, when I started this project, i just whipped something together as fast as I could, so I could see results quickly. In hindsight, it would have been worth my time to invest in understanding how to program SQLite, rather than rolling my own. SQL has these blazing fast indexes that make sorting and searching fast enough to update as you type, even for a library with 100,000 songs in it. My implementation, however, is only fast enough if you have say 10,000 songs. But even this bogs down when the meta data is not cached, for example the "Label" field is stored only in xml files next to the song files, that data is not stored in the database proper, so when you go to sort by that, kJams must load every single xml file for every song.		Well, frankly it's cuz I am not using a super smart database in the back end. You see, when I started this project, i just whipped something together as fast as I could, so I could see results quickly. In hindsight, it would have been worth my time to invest in understanding how to program SQLite, rather than rolling my own. SQL has these blazing fast indexes that make sorting and searching fast enough to update as you type, even for a library with 100,000 songs in it. My implementation, however, is only fast enough if you have say 10,000 songs. But even this bogs down when the meta data is not cached, for example the "Label" field is stored only in xml files next to the song files, that data is not stored in the database proper, so when you go to sort by that, kJams must load every single xml file for every song.

	So, ~~eventually~~ i'~~ll be removing~~ my old, creaky, written-from-the-seat-of-my-pants database, and ~~replacing~~ it with one based on SQLite. ~~Once that's done~~, all the data ~~will be~~ in one place, and all those progress bars ~~will~~ just "go away". Keep in mind however that this is a MAJOR re-write of a huge portion of my code, so ~~when i do this it likely will introduce~~ new bugs.		So, in the 64bit version, i've removed my old, creaky, written-from-the-seat-of-my-pants database, and replaced it with one based on SQLite. In the 64bit version, all the data is in one place, and all those progress bars have just "gone away". Keep in mind however that this was is a MAJOR re-write of a huge portion of my code, so there ARE new bugs because of this.

	==How to Speed Up Searches==		==How to Speed Up Searches==

Dave: /* Why is searching and sorting slow? */

2024-06-20T22:26:28Z

Why is searching and sorting slow?

← Older revision		Revision as of 22:26, 20 June 2024
Line 1:		Line 1:
	==Why is searching and sorting slow?==		==Why is searching and sorting (in the 32bit version) slow?==

	Well, frankly it's cuz I am not using a super smart database in the back end. You see, when I started this project, i just whipped something together as fast as I could, so I could see results quickly. In hindsight, it would have been worth my time to invest in understanding how to program SQLite, rather than rolling my own. SQL has these blazing fast indexes that make sorting and searching fast enough to update as you type, even for a library with 100,000 songs in it. My implementation, however, is only fast enough if you have say 10,000 songs. But even this bogs down when the meta data is not cached, for example the "Label" field is stored only in xml files next to the song files, that data is not stored in the database proper, so when you go to sort by that, kJams must load every single xml file for every song.		Well, frankly it's cuz I am not using a super smart database in the back end. You see, when I started this project, i just whipped something together as fast as I could, so I could see results quickly. In hindsight, it would have been worth my time to invest in understanding how to program SQLite, rather than rolling my own. SQL has these blazing fast indexes that make sorting and searching fast enough to update as you type, even for a library with 100,000 songs in it. My implementation, however, is only fast enough if you have say 10,000 songs. But even this bogs down when the meta data is not cached, for example the "Label" field is stored only in xml files next to the song files, that data is not stored in the database proper, so when you go to sort by that, kJams must load every single xml file for every song.

	So, eventually i'll be removing my old, creaky, written-from-the-seat-of-my-pants database, and replacing it with one based on SQLite. Once that's done, all the data will be in one place, and all those progress bars will just "go away". Keep in mind however that this is a MAJOR re-write of a huge portion of my code, so when i do this it likely will introduce new bugs.		So, eventually i'll be removing my old, creaky, written-from-the-seat-of-my-pants database, and replacing it with one based on SQLite. Once that's done, all the data will be in one place, and all those progress bars will just "go away". Keep in mind however that this is a MAJOR re-write of a huge portion of my code, so when i do this it likely will introduce new bugs.

	==How to Speed Up Searches==		==How to Speed Up Searches==
	A thing you can do in the mean time to speed up your searches is to hide any columns you don't want to search. The more columns you have showing, the slower the search will go. To show or hide the search columns, just [[Context Click]] in the [[Sorting\|column header]] and pick the one you want to show or hide.		A thing you can do in the mean time to speed up your searches is to hide any columns you don't want to search. The more columns you have showing, the slower the search will go. To show or hide the search columns, just [[Context Click]] in the [[Sorting\|column header]] and pick the one you want to show or hide.

Dave at 07:56, 10 February 2016

2016-02-10T07:56:20Z

← Older revision		Revision as of 07:56, 10 February 2016
Line 4:		Line 4:

	So, eventually i'll be removing my old, creaky, written-from-the-seat-of-my-pants database, and replacing it with one based on SQLite. Once that's done, all the data will be in one place, and all those progress bars will just "go away". Keep in mind however that this is a MAJOR re-write of a huge portion of my code, so when i do this it likely will introduce new bugs.		So, eventually i'll be removing my old, creaky, written-from-the-seat-of-my-pants database, and replacing it with one based on SQLite. Once that's done, all the data will be in one place, and all those progress bars will just "go away". Keep in mind however that this is a MAJOR re-write of a huge portion of my code, so when i do this it likely will introduce new bugs.
			==How to Speed Up Searches==
	A thing you can do in the mean time to speed up your searches is to hide any columns you don't want to search. The more columns you have showing, the slower the search will go. To show or hide the search columns, just [[Context Click]] in the [[Sorting\|column header]] and pick the one you want to show or hide.		A thing you can do in the mean time to speed up your searches is to hide any columns you don't want to search. The more columns you have showing, the slower the search will go. To show or hide the search columns, just [[Context Click]] in the [[Sorting\|column header]] and pick the one you want to show or hide.

Dave: /* Why is searching and sorting slow? */

2014-12-08T17:40:41Z

Why is searching and sorting slow?

← Older revision		Revision as of 17:40, 8 December 2014
Line 7:		Line 7:
	A thing you can do in the mean time to speed up your searches is to hide any columns you don't want to search. The more columns you have showing, the slower the search will go. To show or hide the search columns, just [[Context Click]] in the [[Sorting\|column header]] and pick the one you want to show or hide.		A thing you can do in the mean time to speed up your searches is to hide any columns you don't want to search. The more columns you have showing, the slower the search will go. To show or hide the search columns, just [[Context Click]] in the [[Sorting\|column header]] and pick the one you want to show or hide.

	Another thing you can do is [[Searching\|target your searches]] to one sort column. eg: to search for "Beatles" only in the "Artist" column, type this: "artist:beatles", that will go much faster as it will not waste time looking in other sort columns.		Another thing you can do is [[Searching#Targeted_Searches\|target your searches]] to one sort column. eg: to search for "Beatles" only in the "Artist" column, type this: "artist:beatles", that will go much faster as it will not waste time looking in other sort columns.

	Another, very effective thing you can do is pare down your library. Do you really need six versions of "Love Shack"? No, you probably don't. Pick the best one, trash the rest.		Another, very effective thing you can do is pare down your library. Do you really need six versions of "Love Shack"? No, you probably don't. Pick the best one, trash the rest.

Dave at 16:08, 28 August 2014

2014-08-28T16:08:14Z

← Older revision		Revision as of 16:08, 28 August 2014
Line 13:		Line 13:
	Another thing to do is turn off the Music Stores. They take up a lot of RAM.		Another thing to do is turn off the Music Stores. They take up a lot of RAM.

	Especially slow ~~are the following items (hiding~~ all of ~~them~~ will dramatically speed up searching):		==Especially slow==
			Hiding all of these '''Sort Columns''' will dramatically speed up searching:
	<pre>		<pre>
	Sub Genre		Sub Genre

Dave: /* Why is searching and sorting slow? */

2013-03-27T05:48:02Z

Why is searching and sorting slow?

← Older revision		Revision as of 05:48, 27 March 2013
Line 10:		Line 10:

	Another, very effective thing you can do is pare down your library. Do you really need six versions of "Love Shack"? No, you probably don't. Pick the best one, trash the rest.		Another, very effective thing you can do is pare down your library. Do you really need six versions of "Love Shack"? No, you probably don't. Pick the best one, trash the rest.

			Another thing to do is turn off the Music Stores. They take up a lot of RAM.

	Especially slow are the following items (hiding all of them will dramatically speed up searching):		Especially slow are the following items (hiding all of them will dramatically speed up searching):

Dave at 06:27, 24 March 2013

2013-03-24T06:27:52Z

@@ Line 1: / Line 1: @@
 ==Why is searching and sorting slow?==
@@ Line 18: / Line 10: @@
 Another, very effective thing you can do is pare down your library.  Do you really need six versions of "Love Shack"?  No, you probably don't.  Pick the best one, trash the rest.
 Especially slow are the following items (hiding all of them will dramatically speed up searching):

Dave: /* Why is searching and sorting slow? */

2013-03-23T18:33:25Z

Why is searching and sorting slow?

← Older revision		Revision as of 18:33, 23 March 2013
Line 16:		Line 16:

	Another thing you can do is [[Searching\|target your searches]] to one sort column. eg: to search for "Beatles" only in the "Artist" column, type this: "artist:beatles", that will go much faster as it will not waste time looking in other sort columns.		Another thing you can do is [[Searching\|target your searches]] to one sort column. eg: to search for "Beatles" only in the "Artist" column, type this: "artist:beatles", that will go much faster as it will not waste time looking in other sort columns.

			Another, very effective thing you can do is pare down your library. Do you really need six versions of "Love Shack"? No, you probably don't. Pick the best one, trash the rest.

	<strike>Meanwhile, if you have more than 10,000 songs, i recommend you turn OFF the "update search results as you type" preference.</strike> This feature has been removed because too many people [http://karaoke.kjams.com/forum/viewtopic.php?f=4&t=912 thought it sucked] because they didn't get it.		<strike>Meanwhile, if you have more than 10,000 songs, i recommend you turn OFF the "update search results as you type" preference.</strike> This feature has been removed because too many people [http://karaoke.kjams.com/forum/viewtopic.php?f=4&t=912 thought it sucked] because they didn't get it.