Analysis of the reasons why postgresql json value is slow

1. The origin of fate

Slow SQL analysis, with a total number of rows of 80w+, and a slow SQL analysis through monitoring and analysis, a query takes more than 1s.

What is more special is: there is a field info which is jsonb type, written info::json->'length' as length

The same query conditions are 3.3 times different from not checking this field.

It seems that json value pulls down the performance of the query.

There are many ways to take fields in jsonb (as follows). So what is the difference between them and what impact does it have on performance?

info::json->'length'
info::jsonb->'length'

info::json->>'length'
info::jsonb->>'length'

info->'length'
info->'length'

info->>'length'
info->>'length'

2. Comparison

2.1 Comparison of output types

Query different types of writing:

select 
info::json->'length'  AS "info::json->", pg_typeof(info::json->'length' ) ,
info::jsonb->'length' AS "info::jsonb->" , pg_typeof(info::jsonb->'length' ),
info::json->>'length'  AS "info::json->>" , pg_typeof(info::json->>'length' ),
info::jsonb->>'length' AS "info::jsonb->>"  , pg_typeof(info::jsonb->>'length'),
info->'length' AS "info->"  , pg_typeof(info->'length' ),
info->'length' AS "info->"  , pg_typeof(info->'length' ),
info->>'length' AS "info->>"  , pg_typeof(info->>'length' ),
info->>'length' AS "info->>"  , pg_typeof(info->>'length' )
from t_test_json limit 1;

result

info::json-> | pg_typeof | info::jsonb-> | pg_typeof | info::json->> | pg_typeof | info::jsonb->> | pg_typeof | info-> | pg_typeof | info-> | pg_typeof | info->> | pg_typeof | info->> | pg_typeof
--------------+-----------+---------------+-----------+---------------+-----------+----------------+-----------+--------+-----------+--------+-----------+---------+-----------+---------+-----------
123.9 | json | 123.9 | jsonb | 123.9 | text | 123.9 | text | 123.9 | jsonb | 123.9 | jsonb | 123.9 | text | 123.9 | textttui　

Analysis summary

->> Output type is text
->Why the output depends on the data type that calls it, for example: info type is jsonb, then info->'length' is jsonb type
::json and ::jsonb play a role in type conversion.
Info is originally a jsonb type. Info::jsonb is an invalid conversion. Whether it has an impact on performance will be verified later

2.2 Performance comparison

jihite=> EXPLAIN ANALYSE
jihite-> select
jihite-> info::json->'length'  AS "info::json->", pg_typeof(info::json->'length' ) 
jihite-> from t_test_json limit 1;
                                                  QUERY PLAN                                                  
---------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.00..0.04 rows=1 width=36) (actual time=0.028..0.028 rows=1 loops=1)
   ->  Seq Scan on t_test_json  (cost=0.00..30.62 rows=750 width=36) (actual time=0.027..0.027 rows=1 loops=1)
 Planning time: 0.056 ms
 Execution time: 0.047 ms
(4 rows)
jihite=> EXPLAIN ANALYSE
jihite-> select
jihite-> info::jsonb->'length' AS "info::jsonb->" , pg_typeof(info::jsonb->'length' )
jihite-> from t_test_json limit 1
jihite-> ;
                                                  QUERY PLAN                                                  
---------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.00..0.03 rows=1 width=36) (actual time=0.017..0.017 rows=1 loops=1)
   ->  Seq Scan on t_test_json  (cost=0.00..23.12 rows=750 width=36) (actual time=0.015..0.015 rows=1 loops=1)
 Planning time: 0.053 ms
 Execution time: 0.031 ms
(4 rows)
jihite=> EXPLAIN ANALYSE
jihite-> select
jihite-> info::jsonb->'length' AS "info::jsonb->" , pg_typeof(info::jsonb->'length' )
jihite-> from t_test_json limit 1;
                                                  QUERY PLAN                                                  
---------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.00..0.03 rows=1 width=36) (actual time=0.010..0.010 rows=1 loops=1)
   ->  Seq Scan on t_test_json  (cost=0.00..23.12 rows=750 width=36) (actual time=0.009..0.009 rows=1 loops=1)
 Planning time: 0.037 ms
 Execution time: 0.022 ms
(4 rows)
jihite=>
jihite=> EXPLAIN ANALYSE
jihite-> select
jihite-> info::json->>'length'  AS "info::json->>" , pg_typeof(info::json->>'length' )
jihite-> from t_test_json limit 1;
                                                  QUERY PLAN                                                  
---------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.00..0.04 rows=1 width=36) (actual time=0.026..0.027 rows=1 loops=1)
   ->  Seq Scan on t_test_json  (cost=0.00..30.62 rows=750 width=36) (actual time=0.025..0.025 rows=1 loops=1)
 Planning time: 0.056 ms
 Execution time: 0.046 ms
(4 rows)
jihite=>
jihite=> EXPLAIN ANALYSE
jihite-> select
jihite-> info::jsonb->>'length' AS "info::jsonb->>"  , pg_typeof(info::jsonb->>'length')
jihite-> from t_test_json limit 1;
                                                  QUERY PLAN                                                  
---------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.00..0.03 rows=1 width=36) (actual time=0.012..0.012 rows=1 loops=1)
   ->  Seq Scan on t_test_json  (cost=0.00..23.12 rows=750 width=36) (actual time=0.011..0.011 rows=1 loops=1)
 Planning time: 0.053 ms
 Execution time: 0.029 ms
(4 rows)
jihite=>
jihite=> EXPLAIN ANALYSE
jihite-> select
jihite-> info->'length' AS "info->"  , pg_typeof(info->'length' )
jihite-> from t_test_json limit 1;
                                                  QUERY PLAN                                                  
---------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.00..0.03 rows=1 width=36) (actual time=0.014..0.014 rows=1 loops=1)
   ->  Seq Scan on t_test_json  (cost=0.00..23.12 rows=750 width=36) (actual time=0.013..0.013 rows=1 loops=1)
 Planning time: 0.052 ms
 Execution time: 0.030 ms
(4 rows)
jihite=>
jihite=> EXPLAIN ANALYSE
jihite-> select
jihite-> info->'length' AS "info->"  , pg_typeof(info->'length' )
jihite-> from t_test_json limit 1;
                                                  QUERY PLAN                                                  
---------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.00..0.03 rows=1 width=36) (actual time=0.013..0.013 rows=1 loops=1)
   ->  Seq Scan on t_test_json  (cost=0.00..23.12 rows=750 width=36) (actual time=0.012..0.012 rows=1 loops=1)
 Planning time: 0.051 ms
 Execution time: 0.029 ms
(4 rows)
jihite=>
jihite=> EXPLAIN ANALYSE
jihite-> select
jihite-> info->>'length' AS "info->>"  , pg_typeof(info->>'length' )
jihite-> from t_test_json limit 1;
                                                  QUERY PLAN                                                  
---------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.00..0.03 rows=1 width=36) (actual time=0.012..0.013 rows=1 loops=1)
   ->  Seq Scan on t_test_json  (cost=0.00..23.12 rows=750 width=36) (actual time=0.011..0.011 rows=1 loops=1)
 Planning time: 0.053 ms
 Execution time: 0.030 ms
(4 rows)
jihite=>
jihite=> EXPLAIN ANALYSE
jihite-> select
jihite-> info->>'length' AS "info->>"  , pg_typeof(info->>'length' )
jihite-> from t_test_json limit 1;
                                                  QUERY PLAN                                                  
---------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.00..0.03 rows=1 width=36) (actual time=0.012..0.013 rows=1 loops=1)
   ->  Seq Scan on t_test_json  (cost=0.00..23.12 rows=750 width=36) (actual time=0.011..0.011 rows=1 loops=1)
 Planning time: 0.053 ms
 Execution time: 0.029 ms
(4 rows)

Execution time analysis summary

Type conversion jsonb->json, conversion performance (0.46ms) is obviouslyLow outNo conversion (0.3ms)

3. Optimization

Change the query field: info::json->'length' to info->>'length' to reduce the performance loss caused by type conversion.

4. To be investigated

4.1 Whether the conversion of the same type affects performance

The field itself is jsonb. Is it possible to force the ::jsonb affecting performance or is it optimized when precompiling

Judging from the pressure measurement of large amounts of data, the conversion will have an impact on performance, but it is not very important

4.2 How to analyze the time-consuming function

When explaining analyze, the impact of index on performance is mainly analyzed. How to view the specific impact of the function?

V. Attached

5.1 The difference between json and jsonb

jsonb performance is better than json
jsonb supports indexing
【Maximum difference: Efficiency】jsonb will process the write data when writing, and writing is relatively slow. Json will retain the original data (including useless spaces)

It is recommended to store JSON data as jsonb

5.2 Postgresql viewing field type function

pg_typeof()

5.3 Performance Analysis Instructions

If you have a SQL statement that executes very slowly, you want to know what is going on and how to optimize it.
EXPLAIN ANALYSE can obtain the process through which the database executes sql statements, and the time it takes to help optimize performance.

Key parameters:

Execution time: *** ms indicates the actual SQL execution time, which does not include the query plan generation time

5.4 Table creation statement in the example

# Table creation statement

create table t_test_json
(
    id          bigserial         not null PRIMARY KEY,
    task        character varying not null,
    info        jsonb             not null,
    create_time timestamp         not null default current_timestamp
);

# Pressure test data

insert into t_test_json(task, info) values('1', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('2', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('3', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('4', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('5', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('6', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('7', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('8', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('9', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('10', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('11', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('12', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('13', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('14', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('15', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('16', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('17', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('18', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('19', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');
insert into t_test_json(task, info) values('20', '{"length": 123.9, "avatar": "avatar_url", "tags": ["python", "golang", "db"]}');

5.5 Pressure test script in the example

import time
import psycopg
dbname, user, pwd, ip, port = '', '', '', '', '5432'
connection = "dbname=%s user=%s password=%s host=%s port=%s" % (dbname, user, pwd, ip, port)
db = (connection)
cur = ()
ss = 0
lens = 20
for i in range(lens):
    s = ()
    sql = ''' select
        id,
        info::json->'length' as length
        from
        t_test_json
        order by id
        offset %s limit 1000 ''' % (i * 1000)
    #print("sql:", sql)
    (sql)
    rev = ()
    e = ()
    print("scan:", i, e - s)
    ss += (e - s)
print('avg', ss / lens)

This is the end of this article about the reason why postgresql json is slow to obtain. For more related postgresql json value, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!