Database Tuning

• Tuning is the act of changing an application and DBMS environment to improve system performance

• Performance is usually measured in terms of response time

  • Time to get the first tuple

  • Time to get all the tuples to a query

• A workload consists of various operations performed by the system and their frequency

  • It is important to tune to improve workload, not just a single query

  • Some queries are bottlenecks: performed frequently and/or are very expensive

• Remember: when you improve the performance of one operation, other operations may slow down

  For example, creating an index to speed up a query will slow down insert/update operations

Database Tuning Steps

• Step 1: buy more or faster hardware

  memory is crucial for buffering query operations and caches for various operations

  hard disk speed is crucial, buy faster and more disks to improve the parallelism

• Step 2: tune the system installation

  databases provide a large number of tunable parameters, read database administration books

  learn about making better use of multiple CPUs

Disk caches

• A cache is a set of buffer pages maintained by the DBMS for a specific purpose

• Data cache for reading pages containing the index or the relation

• Procedure cache for storing previously constructed query plans

• Caches are usually shared between concurrent users

• Any requested item must be brought to cache from disk to read/modified

• If it is already in the cache, then the cache has a hit, otherwise the cache has a miss

• Since each hit is a savings in time, hit ratio must be maximized (some application designers seek 90% hit ratio)

Cache replacement algorithms

• If a new item has to be inserted into the cache, another item might need to be removed.

• Cache replacement algorithm decides what should be removed, LRU (least recently used), MRU (most recently used)

  • A recently used page may be used for an update in near future (LRU)

  • A page read in table scan is no longer needed (MRU)

  • Sophisticated caches may take the algorithm using the database into account

  • How would you use the cache for an index page?

• A dirty page is a page modified by an uncommitted transaction -if this page is moved out of cache, it must be written back to disk

Tuning the cache

• Divide the cache and bind a specific item to a cache (different tables may be cached in different caches)

• Divide the cache into pools of varied size, 2K, 4K, 8K, etc.

  • The query processor can choose the best available cache for a query (retrieve large sequences for table scans, even prefetch disk pages that are expected to be requested next)

• Procedure cache may use previously optimized query plans

  Hint: use program variables to increase possible reuse

  SELECT P.name FROM Professor P WHERE P.deptId = :deptid
  

  When you use prepared statements such as this, query and parameters are sent separately, allowing the same query for different parameters to use the same query plan.

Partitioning

• Step 3: partition your data

• Vertical partitioning divides the attributes in the relation and distributes them to different disks or tablespaces

  Frequently queried attributes could be separated from infrequently queried attributes.

• Horizontal partitioning divides the tuples in the relation to multiple disks

  Allows parallelism in reading data from disk

  Some optimizers are able to concentrate on a single partition given a specific query

Denormalization

• Step 4: change your data model

• Normalization reduces redundancy and null values

  • Lower storage requirements, simple queries and updates will be faster

  • Results in more tables, hence complex queries need more joins

    SELECT
        FAN.Title
    FROM
        Films F
        , FilmAlternateNames FAN
    WHERE
        F.filmid = FAN.filmid;
    

• Denormalization stores relations in a non-optimal manner

Examples of denormalization

• Store alternate names in a string and use application code to update and print the alternate names

• Add extra columns for frequently accessed information

  Example: number of movies per actor:

  SELECT
     a.name
     , a.surname
     , count(distinct mr.movieid)
  FROM
     actors a
     , movieroles mr
  WHERE
     a.id=mr.actorid
  GROUP BY
     a.id
     , a.name
     , a.surname ;
  

• Add a column “NumMovies” instead for each actor, no join or group by is needed.

  This column must be updated in the application anytime an update is made to the casts relation. If updates are not frequent, then this could be acceptable.

• Certain attributes might be duplicated if they are used often

  • Example: Store stagename attribute in the casts relation

    Queries involving this attribute are now fully answered from casts (avoiding a costly join)

    Attributes other than stagename might be queried rarely but take a lot of space causing the stagename to take up a lot of space

    Anytime a new actor is added or stagename is changed, the changes must be reflected to the casts relation by updating

    multiple tuples (this may be rare compared to the queries)

    The CASTS relation now stores redundant information and is larger in size

Query Restructuring

• Step 5: Rewrite queries to improve their speed, avoid nested queries, use joins whenever possible

• Nested queries are hard to optimize.

• Inner and outer expressions are optimized separately.

• For correlated expressions, inner query is executed many times for each tuple in the outer expression.

• Certain possible optimizations could be missed with a nested query (suppose an index for casts on (actorid, filmid) existed)

  SELECT DISTINCT
       m.title
  FROM
       movies m
       , movieroles mr
  WHERE
       m.id=mr.movieid
       and exists (
           select
             *
           from
             actors a
           where
             a.surname like '%Bacon'
             and  a.id = mr.actorid);
  

  • All queries below are equivalent to the above one

    -- uncorrelated query is better
    SELECT DISTINCT
       m.title
    FROM
       movies m
       , movieroles mr
    WHERE
       m.id=mr.movieid
       and mr.actorid in (
           select
             a.id
           from
             actors a
           where
             a.surname like '%Bacon');
    
    -- join query is even better
    SELECT DISTINCT
       m.title
    FROM
       movies m
       , movieroles mr
       , actors a
    WHERE
       m.id=mr.movieid
       and mr.actorid=a.id
       and a.surname like '%Bacon';
    

Drops of wisdom

• Avoid sorts (distinct, order by, group by, union, except) if possible, they are costly

• Some queries do not need a distinct or can be rewritten to avoid sorts

• Avoid full table scans

  • A search on a condition like A <> 3 or A like ‘%Bacon’ might result in a table scan

  • A search like A in (1,2,4) might be preferable (depending on the availability of statistics)

• Avoid retrieving tuples into application code, use stored procedures and even complicated queries might be preferable to the added communication cost

• Use views wisely

  • Even though views are useful in application development, use a view in an application that is useful for the given query

    CREATE VIEW together(actorid1, stagename1, actorid2, stagename2) AS
    SELECT a1.actorid, a1.stagename, a2.actorid, a2.stagename
    FROM
        actors a1
        , actors a2
        , movieroles mr1
        , movieroles mr2
    WHERE
        a1.id=mr1.actorid
        and mr1.movieid=mr2.movieid
        and a2.id=mr2.actorid
        and a1.id <> a2.id ;
    
    SELECT
        t.actorid1
    FROM
        together t
    WHERE
        t.name = 'Kevin'
        and t.surname = 'Bacon';
    

    • None of the joins are necessary to answer this query. The optimizer might miss some faster query plans

The use of indices

• Indices speed up query, but slow down insert/delete/update operations

• A clustered index allows fast access to a range query

  • There is only one clustered index per relation

  • Databases usually create one for the primary key by default

  • Reconstruction of clustered indices is costly

• Step 6: choose the most useful indices

  • Find the most useful clusters and use them if they are very useful for a range of queries and supported by the database

  • Next, find the most selective indices to add

    • Finally, find indices that might help with index only scans

Clustered Indices

• We can create clustered indices in Postgresql by using an index:

  create index mr_idx on movieroles(movieid,actorid) ;
  cluster movieroles using mr_idx ;
  

• Clusters are generated once and are not modified incrementally

• You need to recluster periodically if there are frequent updates:

  cluster movieroles;
  

• The reorganization may be too costly for very large tables.

• Create a clustered index for attributes frequently queried with a range or has multiple matching attributes for a value

  Above cluster is very useful for finding actors in a movie:

  select
     a.name
     , a.surname
  from
     movies m
     , movieroles mr
     , actors a
  where
     m.name = 'Harry Potter and the Goblet of Fire'
     and m.id = mr.movieid
     and mr.actorid = a.id;
  

• Clustered indices also provide a sorted order to the relation

• Create unclustered indices on attributes with high selectivity

  SELECT A.name, A.surname FROM Actors A WHERE A.gender = 'F';
  
  SELECT
      A.firstname
      , A.lastname
  FROM
      Actors A
  WHERE
      A.name = 'Kevin'
      and A.surname ='Bacon';
  

  • Gender is not a selective condition, but name and surname are.

• Index nested loop join is also beneficial when there is a highly selective index

  SELECT
     mr.movieid
  FROM
     actors a, movieroles mr
  WHERE
     a.name = 'Kevin'
     and a.surname = 'Bacon'
     and a.id = mr.actorid;
  

• For frequently asked queries, indices might be created to allow index only searches.

  For example, given (name, surname, id) for actors, answering a query like one below now requires only an index search for actors.

  ::

  SELECT

  mr.movieid

  FROM

  actors a , movieroles mr

  WHERE

  a.name = ‘Kevin’ and a.surname = ‘Bacon’ and a.id = mr.actorid;

  • This is in effect a type of vertical partitioning.

• For example, given a query like the one below:

  SELECT
      a2.name
      , a2.surname
  FROM
      actors a1
      , movieroles mr1
      , movieroles mr2
      , actors a2
  WHERE
      a1.name = 'Kevin'
      and a1.surname = 'Bacon'
      and a1.id = mr1.actorid
      and mr1.movieid = mr2.movieid
      and a2.id = mr2.actorid;
  

  for A1, the index is searched in the usual way.

  For A2, the index on(stagename, actorid) can be searched fully instead of the relation to speed up the query.

• Indices do not always help reduce the cost of queries:

  • they must be selective

  • they must be significantly smaller in size than the relation they are indexing

  • they must be used often in queries where they make a difference

• Foreign keys introduce hidden costs to updates since they must be checked for all updates that relate to them

• Count queries can be answered using indices on attributes with a “NOT NULL”” constraint (check if the index indices null values)

Other hints

• Partition data to multiple disks

  • Place data that is accessed sequentially on its own disk

• Invoke parallel query processing when multiple CPUs are available

• Create more detailed statistics (histograms)

• Recompute statistics periodically as needed

• Examine the query plans generated by the system and influence them as necessary

Postgresql Optimizer

• Postgresql’s CBO (cost-based-optimizer) relies heavily on table statistics being available for all tables used in a query.

  analyze;
  

  will recompute the statistics for all the tables in a database.

  Must be run periodically for updated statistics.

• You can ask the optimizer to give you the query plan for a query.

  EXPLAIN query ;
  
  EXPLAIN
  SELECT
     mr.movieid
  FROM
     actors a, movieroles mr
  WHERE
     a.name = 'Kevin'
     and a.surname = 'Bacon'
     and a.id = mr.actorid;
  
                                            QUERY PLAN
  ----------------------------------------------------------------------------------------------
   Hash Join  (cost=3302.62..8991.86 rows=2 width=4)
     Hash Cond: (mr.actorid = a.id)
     ->  Seq Scan on movieroles mr  (cost=0.00..4695.07 rows=265107 width=8)
     ->  Hash  (cost=3302.61..3302.61 rows=1 width=4)
           ->  Seq Scan on actors a  (cost=0.00..3302.61 rows=1 width=4)
                 Filter: (((name)::text = 'Kevin'::text) AND ((surname)::text = 'Bacon'::text))
  (6 rows)
  

Summary

• There are many methods from changing data model, queries and storage methods to improve performance.

• Performance must be improved for the whole workload. Compute the total cost of all queries multiplied by their frequency before and after tuning.

• Tuning is a complex optimization problem due to dependencies between different actions. Concentrate on queries that contribute most to the workload.