Ошибка broken pipe 32 яндекс диск
At age 30, Python has been considered a mature language. The programming language is hugely popular among data scientist and AI engineers thanks to its simplicity and easy syntax. Despite that, its vague errors usually makes new users pull their hair out to debug.
In this article, we will discuss about [Errno 32] Broken pipe - a popular error message you often see when interacting with the file system. By the end of the article, you will understand why it happens and how to avoid it as well as how to fix your code. You might also be interested in finding out more about other common error messages of Python, such as locale.Error: unsupported locale setting or unindent does not match any outer indentation level.
Ошибка “сломанный канал” при подключении к терминалу Linux
Всякий раз, когда мы сталкиваемся с ошибкой [Errno 32] Broken pipe при попытке передать вывод скрипта Python другой программе, например:
Вышеупомянутый синтаксис конвейера создаст процесс, отправляющий данные в восходящем направлении, и процесс, читающий данные в нисходящем направлении. Когда нисходящему потоку не нужно читать данные восходящего потока, он отправит сигнал SIGPIPE процессу восходящего потока.
Когда нисходящий поток не должен читать данные восходящего потока? Давайте разберемся в этом на примере. Команда head в этом примере должна прочитать достаточно строк, чтобы сообщить восходящему потоку, что нам больше не нужно его читать, и она отправит сигнал SIGPIPE процессу восходящего потока.
Всякий раз, когда восходящий процесс является программой Python, возникает ошибка типа IOError: [Errno 32] Broken pipe.
[Errno 32] Broken pipe when pipe outputs in Linux terminal
If you encounter [Errno 32] Broken pipe when trying to pipe output of a Python script to another program such as the below example, read on.
This pipeline syntax will create a process that sends data upstream, and a process that reads data downstream. When the downstream does not need to read upstream data, it will send a SIGPIPE signal to the upstream process.
When downstream no longer needs to read upstream data? For example, the head command in the example only needs to read enough lines to tell the upstream that I no longer need to read it, and it will send the SIGPIPE signal to the upstream process.
When the upstream process is a Python program, an error such as IOError: [Errno 32] Broken pipe will occur.
5 Answers 5
Your server process has received a SIGPIPE writing to a socket. This usually happens when you write to a socket fully closed on the other (client) side. This might be happening when a client program doesn't wait till all the data from the server is received and simply closes a socket (using close function).
In a C program you would normally try setting to ignore SIGPIPE signal or setting a dummy signal handler for it. In this case a simple error will be returned when writing to a closed socket. In your case a python seems to throw an exception that can be handled as a premature disconnect of the client.
The broken pipe error usually occurs if your request is blocked or takes too long and after request-side timeout, it'll close the connection and then, when the respond-side (server) tries to write to the socket, it will throw a pipe broken error.
It depends on how you tested it, and possibly on differences in the TCP stack implementation of the personal computer and the server.
For example, if your sendall always completes immediately (or very quickly) on the personal computer, the connection may simply never have broken during sending. This is very likely if your browser is running on the same machine (since there is no real network latency).
In general, you just need to handle the case where a client disconnects before you're finished, by handling the exception.
Remember that TCP communications are asynchronous, but this is much more obvious on physically remote connections than on local ones, so conditions like this can be hard to reproduce on a local workstation. Specifically, loopback connections on a single machine are often almost synchronous.
I saw on the internet all the complicated ways to fix this, but I copied this code directly, so I think that there is something wrong with the code and not Python's SIGPIPE.
I am redirecting the output, so if the above script was named "open.py", then my command to run would be:
You've got two IO operations occurring on line 2: a read from a.txt and a write to stdout . Perhaps try splitting those onto separate lines so you can see which operation triggers the exception. If stdout is a pipe and the read end has been closed, then that could account for the EPIPE error.
I can reproduce this error on output (given the right conditions), so I suspect the print call is the culprit. @JOHANNES_NYÅTT, can you clarify how you're launching your Python script? Are you redirecting standard output somewhere?
Что вызывает “[Errno 32] Broken pipe” в Python?
«Сломанный канал» обычно считается ошибкой IOError (сокращение от «Ошибка ввода-вывода»), которая произошла на уровне системы Linux. Обычно она возникает при чтении и записи файлов или, другими словами, при выполнении ввода / вывода файлов или сетевого ввода / вывода (через сокеты).
Эквивалентная системная ошибка Linux – EPIPE, взятая из кодов ошибок GNU libc.
Possible solution for [Errno 32] Broken pipe in multi-process program.
In programs that uses worker processes to speed up processing and make use of multi-core CPUs, you can try reducing the number of the worker processes to see whether the error disappear or not.
Conclusion
Properly fixing [Errno 32] Broken pipe requires time and a close look at your code. Most of the time, you can safely ignore the problem if the Python program is relatively simple. Some other times, the solution involves in reducing worker processes in multi-process programs. But we hope that this article offer useful information and help you solve your problem.
Работаю с gmail api. Ошибка происходит конкретно в этом месте:
Как я понял у gmail api долгий ответ, и сервер его закрывает и поэтому выскакивает ошибка Broken pipe, возможно ошибаюсь.
BrokenPipeError at /ru/api/v1/my_url/
[Errno 32] Broken pipe
File "/my_project/.venv/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
35. response = get_response(request)
File "/my_project/.venv/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
128. response = self.process_exception_by_middleware(e, request)
File "/my_project/.venv/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
126. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/my_project/.venv/lib/python3.6/site-packages/django/views/decorators/csrf.py" in wrapped_view
54. return view_func(*args, **kwargs)
File "/my_project/.venv/lib/python3.6/site-packages/rest_framework/viewsets.py" in view
95. return self.dispatch(request, *args, **kwargs)
File "/my_project/.venv/lib/python3.6/site-packages/rest_framework/views.py" in dispatch
494. response = self.handle_exception(exc)
File "/my_project/.venv/lib/python3.6/site-packages/rest_framework/views.py" in handle_exception
454. self.raise_uncaught_exception(exc)
File "/my_project/.venv/lib/python3.6/site-packages/rest_framework/views.py" in dispatch
491. response = handler(request, *args, **kwargs)
File "/my_project/.venv/lib/python3.6/site-packages/rest_framework/mixins.py" in create
21. self.perform_create(serializer)
File "/my_project/.venv/lib/python3.6/site-packages/rest_framework/mixins.py" in perform_create
26. serializer.save()
File "/my_project/.venv/lib/python3.6/site-packages/rest_framework/serializers.py" in save
214. self.instance = self.create(validated_data)
File "/usr/lib64/python3.6/contextlib.py" in inner
53. return func(*args, **kwds)
File "/my_project/apps/applications/serializers/application_action.py" in create
83. self._submit_to_client(application, validated_data.get('comment', ""))
File "/my_project/apps/applications/serializers/application_action.py" in _submit_to_client
113. send = gmail.messages.send_message(message=message)
File "/my_project/.venv/lib/python3.6/site-packages/googleapiclient/_helpers.py" in positional_wrapper
130. return wrapped(*args, **kwargs)
File "/my_project/.venv/lib/python3.6/site-packages/googleapiclient/http.py" in execute
835. method=str(self.method), body=self.body, headers=self.headers)
File "/my_project/.venv/lib/python3.6/site-packages/googleapiclient/http.py" in _retry_request
179. raise exception
File "/my_project/.venv/lib/python3.6/site-packages/oauth2client/transport.py" in new_request
175. redirections, connection_type)
File "/my_project/.venv/lib/python3.6/site-packages/oauth2client/transport.py" in request
282. connection_type=connection_type)
File "/my_project/.venv/lib/python3.6/site-packages/httplib2/__init__.py" in request
1322. (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/my_project/.venv/lib/python3.6/site-packages/httplib2/__init__.py" in _request
1072. (response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/my_project/.venv/lib/python3.6/site-packages/httplib2/__init__.py" in _conn_request
996. conn.request(method, request_uri, body, headers)
File "/usr/lib64/python3.6/http/client.py" in request
1239. self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib64/python3.6/http/client.py" in _send_request
1285. self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.6/http/client.py" in endheaders
1234. self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.6/http/client.py" in _send_output
1065. self.send(chunk)
File "/usr/lib64/python3.6/http/client.py" in send
986. self.sock.sendall(data)
File "/usr/lib64/python3.6/ssl.py" in sendall
965. v = self.send(data[count:])
File "/usr/lib64/python3.6/ssl.py" in send
935. return self._sslobj.write(data)
File "/usr/lib64/python3.6/ssl.py" in write
636. return self._sslobj.write(data)
Exception Type: BrokenPipeError at /ru/api/v1/my_url/
Exception Value: [Errno 32] Broken pipe
Request information:
USER: admin2
В настоящее время Python считается зрелым языком программирования, который широко используется специалистами по обработке данных и инженерами по искусственному интеллекту (ИИ) из-за его простоты и легкочитаемого синтаксиса.
Перехват IOError во избежание ошибки Broken pipe
Поскольку ошибка Broken pipe является ошибкой IOError, мы можем разместить блок try / catch, чтобы ее перехватить, как показано в следующем фрагменте кода:
В приведенном выше фрагменте кода мы импортировали модуль sys и errno и разместили блок try / catch, чтобы перехватить возникшее исключение и обработать его.
Возможное решение проблемы в многопроцессорной программе
В программах, которые используют рабочие процессы для ускорения обработки и многоядерные процессоры, мы можем попытаться уменьшить количество рабочих процессов, чтобы проверить, сохраняется ли ошибка или нет.
Большое количество рабочих процессов может конфликтовать друг с другом при попытке взять под контроль ресурсы системы или разрешение на запись на диск.
Currently I am using an app built in python. When I run it in personal computer, it works without problems.
However, when I move it into a production server. It keeps showing me the error attached as below:.
I've done some research and I got the reason that the end user browser stops the connection while the server is still busy sending data.
I wonder why did it happen and what is the root cause that prevents it from running properly in production server, while it works on my personal computer. Any advice is appreciated
What causes "[Errno 32] Broken pipe" in Python?
"Broken pipe" is essentially an IOError error (short for input/output error), which happened at the Linux system level. It usually occurs when reading and writing files, or in other words, doing file input/output or network input/output (via sockets).
The corresponding Linux system error is EPIPE, excerpted from GNU libc error codes:
Macro: int EPIPE
“Broken pipe.” There is no process reading from the other end of a pipe. Every library function that returns this error code also generates a SIGPIPE signal; this signal terminates the program if not handled or blocked. Thus, your program will never actually see EPIPE unless it has handled or blocked SIGPIPE .
From what we've just read, we know that [Errno 32] Broken pipe is caused by the system sending SIGPIPE signal, which is an inter-process communication mechanism of Linux.
For example, SIGINT is another signal used internally by Linux system. In Linux, Ctrl+C will send a SIGINT signal to end the process, or we can use the kill command to achieve the same effect.
Python does not ignore SIGPIPE by default. Instead, it translate the signal into an exception and raises IOError: [Errno 32] Broken pipe every time it receives a SIGPIPE.
Макрос: int EPIPE
“Broken pipe.” означает, что на другом конце конвейера нет считывания процесса. Каждая функция библиотеки, вызывающая код ошибки, также выдает сигнал SIGPIPE; этот сигнал завершает программу, если не обрабатывается или не блокируется. Следовательно, программа никогда не отобразит EPIPE до тех пор, пока она не обработает или не заблокирует SIGPIPE.
Из приведенного выше утверждения мы можем сделать вывод, что система, отправляющая сигнал SIGPIPE, вызывает ошибку [Errno 32] Broken pipe в механизме межпроцессного взаимодействия Linux.
Например, система Linux внутренне использует другой сигнал, называемый SIGINT. В Linux команда Ctrl + C отправит сигнал SIGINT, чтобы завершить процесс, или мы можем использовать команду kill для достижения того же эффекта.
Python по умолчанию не игнорирует SIGPIPE. Однако он преобразует сигнал в исключение и вызывает ошибку – IOError: [Errno 32] Сломанный канал каждый раз, когда он получает SIGPIPE.
Avoid [Errno 32] Broken pipe by ignoring SIGPIPE
If you don't care too much about properly catching SIGPIPE and just need to get things running quickly, add the code snippet below to the top of your Python program.
What the code does is redirecting SIGPIPE signals to the default SIG_DFL, which the system usually ignore.
But beware, the Python manual on signal library warn against this type of handling SIGPIPE
Do not set SIGPIPE ’s disposition to SIG_DFL in order to avoid BrokenPipeError . Doing that would cause your program to exit unexpectedly also whenever any socket connection is interrupted while your program is still writing to it.
Properly catch IOError to avoid [Errno 32] Broken pipe
Since [Errno 32] Broken pipe is actually a IOError, you can place a try/catch block to catch it like the code snippet below :
Как избежать ошибки “сломанный канал”?
Если мы не заботимся о правильном перехвате SIGPIPE и нам нужно быстро запустить процесс, вставьте следующий фрагмент кода в начало программы Python.
В приведенном выше фрагменте кода мы перенаправили сигналы SIGPIPE на стандартный SIG_DFL, который система обычно игнорирует.
Однако рекомендуется остерегаться руководства Python по библиотеке сигналов, чтобы предостеречь от такой обработки SIGPIPE.
10 Answers 10
The problem is due to SIGPIPE handling. You can solve this problem using the following code:
Update: As pointed out in the comments, python docs already have a good answer.
See here for background on this solution. Better answer here.
@DavidBennett, I am sure its application dependent and for your purposes the accepted answer is the right one. There is a much thorough Q&A here for people to go through and make an informed decision. IMO, for command line tools, its probably best to ignore the pipe signal in most cases.
Could someone answer me why people are considering a blogspot article as a better source of truth than official documentation (hint: open the link to see how to fix the broken pipe error properly)? :)
To bring information from the many helpful answers together, with some additional information:
Standard Unix signal SIGPIPE is sent to a process writing to a pipe when there's no process reading from the pipe (anymore).
- This is not necessarily an error condition; some Unix utilities such as head by design stop reading prematurely from a pipe, once they've received enough data.
- Therefore, an easy way to provoke this error is to pipe to head [1] ; e.g.:
- python -c 'for x in range(10000): print(x)' | head -n 1
By default - i.e., if the writing process does not explicitly trap SIGPIPE - the writing process is simply terminated, and its exit code is set to 141 , which is calculated as 128 (to signal termination by signal in general) + 13 ( SIGPIPE 's specific signal number).
However, by design Python itself traps SIGPIPE and translates it into a Python BrokenPipeError (Python 3) / IOError (Python 2) instance with errno value errno.EPIPE .
- Note: If you use a Unix emulation environment on Windows, the error may surface differently - see this answer.
If a Python script does not catch the exception, Python outputs error message BrokenPipeError: [Errno 32] Broken pipe (Python 3, possibly twice, with Exception ignored in: ' mode='w' encoding='utf-8'> sandwiched in between) / IOError: [Errno 32] Broken pipe (Python 2) and terminates the script with exit code 1 [2] - this is the symptom Johannes (the OP) saw.
Windows considerations ( SIGPIPE is a Unix-only signal)
If your script needs to run directly on Windows too, you may have to conditionally bypass code that references SIGPIPE , as shown in this answer.
If your script runs in a Unix subsystem on Windows, the SIGPIPE signal may surface differently than on Unix - see this answer.
There are two ways to solve this problem:
Generally, it is not advisable to silence this exception, as it may signal a severe error condition, depending on your script's purpose, such as the receiving end of a network socket unexpectedly closing.
- However, if your script is a command-line utility, where quiet termination may not only be acceptable but preferred so as to play nicely with the standard head utility, for instance, you can abort quietly as follows, using signal.signal() to install the platform's default signal handler (which behaves as described above), as also shown in akhan's answer (works in both Python 3 and 2):
- Otherwise, if you want to handle the SIGPIPE-triggered exception yourself (works in both Python 3 and 2, adapted from the docs):
[1] Note that in bash you will by default only see head 's exit code - which is 0 - reflected in $? afterwards. Use echo $ to see Python's exit code.
[2] Curiously, on macOS 10.15.7 (Catalina), with Python 3.9.2 (but not 2.x), I see exit code 120 , but the docs say 1 , and that's what I also see on Linux.
Читайте также: