Django rest framework поиск
Началось всё с того, что мне предложили в рамках предмета "Основы веб-программирования" поучаствовать в проекте, вместо проделывания лабораторных работ и курсовой, поскольку я заявил о том, что хотел быть делать нечто отдалённое от общего курса (и так уже достаточно знаний было по связке DRF + Vue, хотелось чего-то нового). И вот в одном из своих PR на github я решил использовать полнотекстовый поиск (задание намекало на это) для фильтрации контента, что заставило меня обратиться к документации Django в поисках того, каким же образом лучше это дело реализовать. Думаю, вы знаете большую часть из тех методов, что были там предложены (contains, icontains, trigram_similar). Все они подходят для каких-то конкретных задач, но не слишком хороши в, именно, полнотекстовом поиске. Пролистав чуть ниже, я наткнулся на раздел, в котором говорилось о взаимодействии Django и Pgsql для реализации document-based поиска, что меня привлекло, поскольку в постгре встроен инструмент для реализации этого самого [полнотекстового] поиска. И я решил, что скорее всего, django просто предоставляет API к этому поиску, исходя из чего такое решение должно работать и точнее и быстрее, чем любые другие варианты. Преподаватель мне не слишком поверил, мы с ним поспорили, и он предложил провести исследование на эту тему. И вот я здесь.
DjangoFilterBackend
The django-filter library includes a DjangoFilterBackend class which supports highly customizable field filtering for REST framework.
To use DjangoFilterBackend , first install django-filter .
Then add 'django_filters' to Django's INSTALLED_APPS :
You should now either add the filter backend to your settings:
Or add the filter backend to an individual View or ViewSet.
If all you need is simple equality-based filtering, you can set a filterset_fields attribute on the view, or viewset, listing the set of fields you wish to filter against.
This will automatically create a FilterSet class for the given fields, and will allow you to make requests such as:
For more advanced filtering requirements you can specify a FilterSet class that should be used by the view. You can read more about FilterSet s in the django-filter documentation. It's also recommended that you read the section on DRF integration.
drf-url-filters
drf-url-filter is a simple Django app to apply filters on drf ModelViewSet 's Queryset in a clean, simple and configurable way. It also supports validations on incoming query params and their values. A beautiful python package Voluptuous is being used for validations on the incoming query parameters. The best part about voluptuous is you can define your own validations as per your query params requirements.
If you're doing REST-based web service stuff . you should ignore request.POST.
— Malcom Tredinnick, Django developers group
REST framework's Request objects provide flexible request parsing that allows you to treat requests with JSON data or other media types in the same way that you would normally deal with form data.
request.data returns the parsed content of the request body. This is similar to the standard request.POST and request.FILES attributes except that:
For more details see the parsers documentation.
Filtering against the current user
You might want to filter the queryset to ensure that only results relevant to the currently authenticated user making the request are returned.
Filtering & schemas
The method should return a list of coreapi.Field instances.
The following third party packages provide additional filter implementations.
.accepted_renderer
The renderer instance that was selected by the content negotiation stage.
The web framework for perfectionists with deadlines.
In this article, to reduce time consuming on writing, I did not show you how to build a project from scratch e.g. creating virtual environment, installing Django, etc. As mentioned above, I assumed that you should get familiar with Django beforehand.
The first step is to start a project (Assuming you all have created a directory to contain a project, virtual environment, and installed Django already)
then start an app
Now we already have our first project and app
models.py
Note: we use auto_now_add=True to automatically update time after creating a post, this will show only one time, and auto_now=True to automatically update after editing a post, and it changes every time we edit a post
Register the app
settings.py
Note: In this post, I use SQLite as a default database, if you want to chose the most recommended DB for Django “PostgreSQL” you can follow my previous article How to Start Django Project with a Database(PostgreSQL)
The next step is to migrate our model into the database
Now we have all tables containing in a single file called db.sqlite3
Type the following command to create a superuser
Go to admin.py file
admin.py
Now we just finish registering our model, let’s get into Django admin site to add some posts.
Then use previous username and password we just created
Finally, we have 3 simple posts
from your app directory, the path will look like this
→ myapp/templates/myapp/index.html
index.html
Now we already created an HTML file to display our posts
It’s time to render an HTML file to show on a web browser.
We just created a simple function only rendering an HTML file to display to a client side, so now we have to map the function with urls we are designing now. Keep in mind that urls.py does not come default with Django, so we should create a new one.
myapp/urls.py
We have one step left, to register myapp.urls module into myproject’s urls
myproject/urls.py
Now we have a simple web page, but it does not look good enought, and not responsive as well. So Bootstrap can help us with 2 very easy steps by adding the following block code into index.html
- Add Bootstrap CDN
- Add Bootstrap Navbar
Browse to Bootstrap official website to get Bootstrap CDN
Add Bootstrap CDN and Navbar to our page
index.html
Refresh a web page, we will see
The previous step We just rendered a template(HTML file) to display on a web browser but did not send any data back. Now it’s time to do that.
views.py
Loop to show all posts
index.html
Note: > used to count posts ascending order. You don’t have to use it in this post but you can integrate it with a table that used to represent a number of items. You can read more here
Security
Security issues are handled under the supervision of the Django security team.
The project maintainers will then work with you to resolve any issues where required, prior to any public disclosure.
Filtering and object lookups
Note that if a filter backend is configured for a view, then as well as being used to filter list views, it will also be used to filter the querysets used for returning a single object.
For instance, given the previous example, and a product with an id of 4675 , the following URL would either return the corresponding object, or return a 404 response, depending on if the filtering conditions were met by the given product instance:
Начало работы
Первая проблема, которая передо мной встала — поиск мокапа БД, чтобы не придумать каких-нибудь непонятных штук самому и я отправился гуглить и читать вики постгреса. В итоге остановился на их демо базе о полётах по России.
Хорошо, база найдена. Теперь нужно определиться в том, какие способы фильтрации будут использоваться для сравнения. Первое, что я бы хотел использовать, разумеется, стандартный метод search из django.contrib.postgres.search. Второе — contains (ищет слово в строке) и icontains (предоставляет данные, игнорируя акценты, например: по запросу "Helen" будет результат: , , ), которые предоставляет сам django. Все эти способы фильтрации я так же хочу сравнить со встроенным поиском внутри postgresql. Искать я решил по таблице tickets в версии small она содержит 366733 записей. Поиск будет происходить по полю passenger_name, где, как нетрудно догадаться, содержится имя пассажира. Написано оно транслитом.
Setting filter backends
The default filter backends may be set globally, using the DEFAULT_FILTER_BACKENDS setting. For example.
You can also set the filter backends on a per-view, or per-viewset basis, using the GenericAPIView class-based views.
SearchFilter
The SearchFilter class supports simple single query parameter based searching, and is based on the Django admin's search functionality.
When in use, the browsable API will include a SearchFilter control:
The SearchFilter class will only be applied if the view has a search_fields attribute set. The search_fields attribute should be a list of names of text type fields on the model, such as CharField or TextField .
This will allow the client to filter the items in the list by making queries such as:
You can also perform a related lookup on a ForeignKey or ManyToManyField with the lookup API double-underscore notation:
For JSONField and HStoreField fields you can filter based on nested values within the data structure using the same double-underscore notation:
By default, searches will use case-insensitive partial matches. The search parameter may contain multiple search terms, which should be whitespace and/or comma separated. If multiple search terms are used then objects will be returned in the list only if all the provided terms are matched.
The search behavior may be restricted by prepending various characters to the search_fields .
- '^' Starts-with search.
- '=' Exact matches.
- '@' Full-text search. (Currently only supported Django's PostgreSQL backend.)
- '$' Regex search.
By default, the search parameter is named 'search' , but this may be overridden with the SEARCH_PARAM setting.
To dynamically change search fields based on request content, it's possible to subclass the SearchFilter and override the get_search_fields() function. For example, the following subclass will only search on title if the query parameter title_only is in the request:
For more details, see the Django documentation.
OrderingFilter
The OrderingFilter class supports simple query parameter controlled ordering of results.
By default, the query parameter is named 'ordering' , but this may by overridden with the ORDERING_PARAM setting.
For example, to order users by username:
The client may also specify reverse orderings by prefixing the field name with '-', like so:
Multiple orderings may also be specified:
4 Steps closer
- We typed “admin” on this form to get desired admin results related to the word admin
- q (String field) is sent to a Django server with a value “admin”
- Django server receives q variable, and navigate to a database for seaching any posts related to, or contained with “admin”
- Then send posts back to a client showing all posts related to admin on a web page
search: Location of search form indicated that search swill start executing after this url
? : A Separator or Indicator used to indicate that after this sign is a query string part
=: Equals sign used to encapsulate value with q(string parametr)
admin: Value we typed to search for the result of “admin”
Note: This is a query string q=admin composed with field and value pair
Example
Let's take a look at a quick example of using REST framework to build a simple model-backed API.
We'll create a read-write API for accessing information on the users of our project.
Any global settings for a REST framework API are kept in a single configuration dictionary named REST_FRAMEWORK . Start off by adding the following to your settings.py module:
Don't forget to make sure you've also added rest_framework to your INSTALLED_APPS .
We're ready to create our API now. Here's our project's root urls.py module:
.stream
request.stream returns a stream representing the content of the request body.
You won't typically need to directly access the request's content, as you'll normally rely on REST framework's default request parsing behavior.
Assuming you are implementing your django blog post with search form(Bootstrap) or any HTML forms, but do not know how make it functional. This post is created for you.
- Get familiar with basic Django, and its structures
- Get familiar with Bootstrap, and basic frontend programming languages , HTML, CSS.
We type any word e.g. “django” to our search form from a navbar section, then a client(Web browser) sends request to a server, a server makes query to a database. Based on a query, if found, response a post we previously made query to show on a web page
For instance, look at the images below captured from Django official website’s documentation
It has a search form
Дать django возможность работать с уже существующей БД
Вторая проблема — разрешить django только чтение данных из нашей демонстрационной БД. Покопавшись ещё в документации django я нашёл каким же образом, можно составить модельки по существующей БД, чтобы не перепечатывать ручками всё:
Example
For example, you might need to restrict users to only being able to see objects they created.
We could achieve the same behavior by overriding get_queryset() on the views, but using a filter backend allows you to more easily add this restriction to multiple views, or to apply it across the entire API.
Full text search (через rest_framework.filters, точнее — SearchFilter)
Если не использвоать именно FTS, то результаты получаются сравнимыми с FTS внутри постгре, но хуже, чем contains и icontains. От 200 до 1710 мс.
А с использованием FTS эффективность повышается, отклонение сводится к минимальному. В среднем, это займёт от 800 до 1120 мс.
Adding search string parameter and url endpoint to search for
- adding search string parameter called “search”
- Add url endpoint to an action attribute of form after submitting a form
index.html
Now, the form will look like this
Test searching by typing a desired word
This is the core section of this article, we use Q objects to make more complex queries following
- Use & (AND) operator to search for more than 2 fields. For example, when searching title & content, both should be true, when searching for a word such as “python”, so it(python) should be contained in both title and content fields
- Use | (OR) operator to search for only one field. For example, when searching title| content, both don’t have to be true, only one is okay, when searching for a word such as “python”, so it(python) doesn’t have to be contained in both title and content fields
Do not forget to import Q objects
For more details about Q objects, you can read from the document here
views.py
In index function, create a new variable called “search_post” to get url parameter from a form submitting from a web page
We use GET method in this context because we did not make change to our resource(database) only show search result, so GET method is used.
Afterwards, add if-else conditional statements to the function
if getting a search param from client side, then do something, else does a default return data to a web page showing all posts
It founds because I typed “flask” and of course, Flask lives both title and content fields
Not get a result because “python” lives only a title field, but we use & (AND) for our searching condition
Not get a result because “django” lives only a title field, but we use & (AND) for our searching condition
Congratulations !! you just finished implementing a search form to search posts from a database
What we’ve learned in this article are
- How search works
- Basic Django MTV
- Understanding Q objects
- How to implement our existing search form with Django Q objects
If this article is helpful, please clap to support me continuing my good work, and do not hesitate to drop your comment below sharing me your opinion or problem. See you again next article.
Descending order of our posts
We can arrange our posts by using order_by(“-date_create”) with negative sign “-” so that the lastest post will be showed on the top of a web page
Specifying which fields may be ordered against
It's recommended that you explicitly specify which fields the API should allowing in the ordering filter. You can do this by setting an ordering_fields attribute on the view, like so:
This helps prevent unexpected data leakage, such as allowing users to order against a password hash field or other sensitive data.
If you don't specify an ordering_fields attribute on the view, the filter class will default to allowing the user to filter on any readable fields on the serializer specified by the serializer_class attribute.
If you are confident that the queryset being used by the view doesn't contain any sensitive data, you can also explicitly specify that a view should allow ordering on any model field or queryset aggregate, by using the special value '__all__' .
Django REST framework full word search filter
The djangorestframework-word-filter developed as alternative to filters.SearchFilter which will search full word in text, or exact match.
.query_params
request.query_params is a more correctly named synonym for request.GET .
Django REST framework filters package
The django-rest-framework-filters package works together with the DjangoFilterBackend class, and allows you to easily create filters across relationships, or create multiple filter lookup types for a given field.
Contains
Начнём с contains, по сути, он работает как WHERE LIKE.
Для того, чтобы получить результат из curl я выполнял запрос следующим образом (считается в секундах):
Свёл всё в таблице, на соответствующем листе.
Но если резюмировать — отклонение от скорости фильтрации внутри самого постгреса достаточно большое, и по факту исполнение такого запроса к серверу займёт от 140 до 1400 мс. Не претендую на истину, но работает всё приблизительно так. А время самой фильтрации через ORM займёт от 73 до 600 мс, в то время как такая же фильтрация внутри постгреса выполняется за промежуток от 55 до 100 мс.
Search | Django documentation | Django
Django URL Filter
django-url-filter provides a safe way to filter data via human-friendly URLs. It works very similar to DRF serializers and fields in a sense that they can be nested except they are called filtersets and filters. That provides easy way to filter related data. Also this library is generic-purpose so it can be used to filter other sources of data and not only Django QuerySet s.
Customizing the interface
to_html(self, request, queryset, view)
The method should return a rendered HTML string.
Использование фильтров через модуль django-filter
Результаты почти совпали со стандартными contains и icontains, поэтому смысла детально это рассматривать не вижу. Да и в целом, модуль django-filter не показал какого-то ощутимого преимущества перед стандартными средствами фильтрации Django ORM.
Filtering against the URL
Another style of filtering might involve restricting the queryset based on some part of the URL.
For example if your URL config contained an entry like this:
You could then write a view that returned a purchase queryset filtered by the username portion of the URL:
Так что в итоге?
Если у вас есть большой объём данных — нужно прописывать нормальные индексы и использовать полнотекстовый поиск (разумеется, только в том случае, когда соответствует вашим целям) с радостью и счастьем, потому что он решает довольно широкий круг проблем. Но вот всегда ли в нём есть необходимость — уже решать вам. Я усвоил для себя, что в некоторых случаях (когда не стоит задачи именно полнотекстового поиска, а есть поиск по подстроке, который реализуется с помощью contains/icontains) лучше вовсе не использовать полнотекстовый поиск, потому что индексы в определённый момент начинают кушать всё больше и больше памяти вашего сервера, что, скорее всего, негативно скажется на работе вашего сервера.
В целом, моё понимание некоторых внутренних механизмов работы django благодаря этому исследованию устаканилось. И пришло, наконец, осознание разницы между поиском по подстроке и полнотекстовым поиском. Разнице в их реализации через Django ORM.
Django REST framework is a powerful and flexible toolkit for building Web APIs.
Some reasons you might want to use REST framework:
- The Web browsable API is a huge usability win for your developers. including packages for OAuth1a and OAuth2. that supports both ORM and non-ORM data sources.
- Customizable all the way down - just use regular function-based views if you don't need the morepowerfulfeatures.
- Extensive documentation, and great community support.
- Used and trusted by internationally recognised companies including Mozilla, Red Hat, Heroku, and Eventbrite.
.parsers
The APIView class or @api_view decorator will ensure that this property is automatically set to a list of Parser instances, based on the parser_classes set on the view or based on the DEFAULT_PARSER_CLASSES setting.
You won't typically need to access this property.
Note: If a client sends malformed content, then accessing request.data may raise a ParseError . By default REST framework's APIView class or @api_view decorator will catch the error and return a 400 Bad Request response.
If a client sends a request with a content-type that cannot be parsed then a UnsupportedMediaType exception will be raised, which by default will be caught and return a 415 Unsupported Media Type response.
The request exposes some properties that allow you to determine the result of the content negotiation stage. This allows you to implement behaviour such as selecting a different serialization schemes for different media types.
Funding
REST framework is a collaboratively funded project. If you use REST framework commercially we strongly encourage you to invest in its continued development by signing up for a paid plan.
Every single sign-up helps us make REST framework long-term financially sustainable.
Overriding the initial queryset
Note that you can use both an overridden .get_queryset() and generic filtering together, and everything will work as expected. For example, if Product had a many-to-many relationship with User , named purchase , you might want to write a view like this:
.accepted_media_type
A string representing the media type that was accepted by the content negotiation stage.
REST framework provides flexible, per-request authentication, that gives you the ability to:
- Use different authentication policies for different parts of your API.
- Support the use of multiple authentication policies.
- Provide both user and token information associated with the incoming request.
request.auth returns any additional authentication context. The exact behavior of request.auth depends on the authentication policy being used, but it may typically be an instance of the token that the request was authenticated against.
If the request is unauthenticated, or if no additional context is present, the default value of request.auth is None .
License
Copyright © 2011-present, Encode OSS Ltd. All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
The root QuerySet provided by the Manager describes all objects in the database table. Usually, though, you'll need to select only a subset of the complete set of objects.
— Django documentation
The default behavior of REST framework's generic list views is to return the entire queryset for a model manager. Often you will want your API to restrict the items that are returned by the queryset.
The simplest way to filter the queryset of any view that subclasses GenericAPIView is to override the .get_queryset() method.
Overriding this method allows you to customize the queryset returned by the view in a number of different ways.
Installation
Install using pip , including any optional packages you want.
. or clone the project from github.
Add 'rest_framework' to your INSTALLED_APPS setting.
If you're intending to use the browsable API you'll probably also want to add REST framework's login and logout views. Add the following to your root urls.py file.
Note that the URL path can be whatever you want.
Full text search (через django.contrib.postgres)
Поскольку индексов никаких создано не было, full text search довольно сильно и вполне ощутимо проигрывает прошлым вариантам. Время исполнения запроса в постгресе колеблется около 1300 мс, а запрос к серверу занимает от 1000 до 1700 мс. При этом, фильтрация через ORM — укладывается в промежуток от 1000 до 1450 мс.
.method
Browser-based PUT , PATCH and DELETE forms are transparently supported.
Support
For priority support please sign up for a professional or premium sponsorship plan.
Quickstart
Can't wait to get started? The quickstart guide is the fastest way to get up and running, and building APIs with REST framework.
Filtering against query parameters
A final example of filtering the initial queryset would be to determine the initial queryset based on query parameters in the url.
As well as being able to override the default queryset, REST framework also includes support for generic filtering backends that allow you to easily construct complex searches and filters.
Generic filters can also present themselves as HTML controls in the browsable API and admin API.
Фильтруем в django
Предвосхищая вопросы, касательно кода из спойлера — время исполнения скрипта является обоснованной метрикой в том и только том случае, если с queryset были проведены какие-либо операции. Поэтому я вывожу длину получившегося ответа.
A QuerySet is iterable, and it executes its database query the first time you iterate over it. For example, this will print the headline of all entries in the database:
.content_type
You won't typically need to directly access the request's content type, as you'll normally rely on REST framework's default request parsing behavior.
Development
See the Contribution guidelines for information on how to clone the repository, run the test suite and contribute changes back to REST Framework.
.authenticators
The APIView class or @api_view decorator will ensure that this property is automatically set to a list of Authentication instances, based on the authentication_classes set on the view or based on the DEFAULT_AUTHENTICATORS setting.
You won't typically need to access this property.
REST framework supports a few browser enhancements such as browser-based PUT , PATCH and DELETE forms.
Icontains
Icontains работает несколько по-другому (он приводит всё к одному виду, чтобы сравнение было более близким). Код для вьюшки использовался почти аналогичный, только вместо contains — icontains. Вот и вся разница.
По итогу, отклонение в данном случае уже меньше, поскольку и сам постгрес начал тратить намного большее времени на исполнение запроса (порядка 300 мс), а исполнение такого запроса к серверу займёт у клиента от 200 до 1500 мс. Фильтрация через ORM — от до 200 до 700 мс.
Specifying a default ordering
If an ordering attribute is set on the view, this will be used as the default ordering.
Typically you'd instead control this by setting order_by on the initial queryset, but using the ordering parameter on the view allows you to specify the ordering in a way that it can then be passed automatically as context to a rendered template. This makes it possible to automatically render column headers differently if they are being used to order the results.
The ordering attribute may be either a string or a list/tuple of strings.
You can also provide your own generic filtering backend, or write an installable app for other developers to use.
To do so override BaseFilterBackend , and override the .filter_queryset(self, request, queryset, view) method. The method should return a new, filtered queryset.
As well as allowing clients to perform searches and filtering, generic filter backends can be useful for restricting which objects should be visible to any given request or user.
Выбор метрик
Изначально я подумал, что считать время фильтрации в питоне получится, используя таймер для получения времени исполнения скрипта, и дополнительной метрикой должно было стать время исполнения запроса через curl, поскольку это показывает приблизительное время, за которое отфильтрованные данные дойдут до конечного пользователя. Кроме этого, следует сравнивать это время с эталонным (прямым исполнением соответствующих запросов в БД).
Requirements
REST framework requires the following:
- Python (3.6, 3.7, 3.8, 3.9, 3.10)
- Django (2.2, 3.0, 3.1, 3.2, 4.0)
We highly recommend and only officially support the latest patch release of each Python and Django series.
The following packages are optional:
-
, uritemplate (5.1+, 3.0.0+) - Schema generation support. (3.0.0+) - Markdown support for the browsable API. (2.4.0+) - Add syntax highlighting to Markdown processing. (1.0.1+) - Filtering support. (1.1.1+) - Object level permissions support.
Читайте также: