Questions tagged [celery]

Celery is a distributed task queue framework for Python, used for asynchronous and parallel execution.

0
votes
0answers
14 views

scikit-learn multiprocessing from Celery worker

I want to run some scikit-learn things as a job using celery. The problem is scikit-learn refuses to/can't use multiprocessing from a celery worker process. Typically it will give a warning like: [...
0
votes
0answers
7 views

Logstash filter to parse python cellery logs as json

I have a cellery app, let's call it generic myapp. The logs are configured to be in JSON format. We receive some entries from some operators we work with and unfortunatelly, those entries, in fact a ...
0
votes
0answers
7 views

Is it possible to have session-like parameters on Celery?

I have an API that send messages to an application running on Celery. Is it possible to have session-like arguments on the Celery application? For example: I send a message from the API and that ...
0
votes
0answers
13 views

Celery-Progress-Bar Not Working in Django

I am trying to use celery-progress module to show the user the progress of the task, since it takes lot of time to complete the task. However, after following the instructions on https://github.com/...
0
votes
0answers
18 views

Flask request context and celery

So, I am developing a multi-tenant Flask application. I am currently using sqlalchemy and the multi-schema pattern. To define the current schema I use this function: def on_pool_checkout(dbapi_conn,...
-1
votes
1answer
19 views

Why does Celery only recognize tasks from single Django app

So we have this Django project with multiple apps, and we use celery for tasks. The issue we are running into is that only the tasks inside the tasks.py of a single app will run, other tasks.py tasks ...
0
votes
0answers
10 views

How can I start celery flower as daemon after restart server?

I made init.d file to run the celery flower with daemon, and it works. But when I restart the EC2 server, I have to run this file manually. I hope this file will restart automatically when the server ...
0
votes
1answer
11 views

Celery is connecting to amqp://guest**@localhost instead of Redis. Heroku deploy

I have read simillar topics and done everything there was being suggested, but the problem remains. I am deploying my app to Heroku. Locally everything was working fine, but during deploy after ...
1
vote
0answers
21 views

unicode in celery (python 2.7)

I'm using celery 4.1 in flask. (python 2.7) print(u'glätteisen') the code is working in flask. but when I try this code inside celery, that doesn't print anything. print('something') when I try ...
1
vote
2answers
54 views

Celery can not find tasks nor settings with Django 2.1.1

I am trying to do Asynchronous task using Celery and RabbitMQ server. I have installed Celery and RabbitMQ on my system. Now when I'm running celery worker -l info, the celery starts working using ...
0
votes
0answers
20 views

django: how to update the html once the celery asynchronous task is completed

I have a django view without using celery background task. It does some long calculation and then it renders a template. Now i want to use celery to make the long calculation in background. But the ...
0
votes
0answers
16 views

Unexpected errors happened in Celery+RabbitMQ for periodic tasks

I set up the periodic task in Flask+Celery+RabbitMQ+Flower, what the task do is to synchronize the record from MySQL to MongoDB for every 15 minutes. My environment is shown as follows: celery 4.2.1 ...
0
votes
0answers
15 views

Non blocking celery task chaining

I have 30k+ RSS feed urls in my database. I am using feedparser to parse and extract new entries from each RSS feed.After I get a list of new entries I need to validate them and save them in DB. ...
0
votes
1answer
9 views

disable Celery ForkPoolWorker logs

I'm new in Celery. I want to disable worker logs on Celery Screen. So, I run celery like this. But after I send task Celery Server... Still output worker logs I tried get_task_logger but not ...
1
vote
0answers
47 views

Installing Apache Airflow 1.10.0 in kubernetes - Flask errors

I am currently working on deploying Apache Airflow 1.10.0 to a Kubernetes cluster. I am running into some dependency issues with Flask. If I use the current version of flask-login (0.4.1), I receive ...
0
votes
0answers
7 views

What does '*reply.celery.pidbox' stand for

We are using Celery with RabbitMQ as Broker and Redis as Backend. We are running into a problem where we see growth in the number of queues inside RabbitMQ with the format: 000d7a5b-f554-3817-bff4-...
0
votes
0answers
35 views

RabbitMQ consumer is slow

I'm running RabbitMQ in a private EC2 on Amazon and I'm encountering some issues. I've many consumers which read from the very same queue, and I experienced an unusual behavior in which, apparently, ...
0
votes
1answer
17 views

Celery - Permission Problem - Create folder

I use celery (jobs manager) on prod mode for a website (Django) on a centos7 server. My problem is that in a celery task my function did not create folder (see my_function). the function def ...
0
votes
0answers
15 views

Monit process permission issues

I have a celery consumer which is written in python and it executes a Perl script for every task in the queue. I have added the celery to init.d configuration, everything works as expected but when ...
0
votes
1answer
53 views

Python Celery: Update django model after state change

I managed to find 2 similar topics with discussion about this issue, but unfortunately I couldn't get what is the best solution from it: Update Django Model Field Based On Celery Task Status Update ...
2
votes
2answers
30 views

django rest framework update method in serializer, instance is not saved immediately

The instance to be updated has instance.email=abc@mail.com. email to be updated or changed to xyz@mail.com UserUpdateSerializer's update method. def update(self, instance, validated_data): ...
0
votes
0answers
35 views

Error on Celery with Pickle Serialization

I'm doing a flask system, and I'm need to use pickle serialization, since it's not JSON serializated. Error Message: kombu.exceptions.EncodeError: maximum recursion depth exceeded while calling a ...
0
votes
1answer
25 views

Celery + RabbitMQ becomes slow

I'm here with a performance issue that I can't seem to figure it out. The problem is that executing tasks is too slow. Based on the celery log most of the tasks are finished under 0.3 seconds. I ...
0
votes
0answers
7 views

RabbitMQ+celery periodically loses tasks

I have a problem: sometimes rabbitmq or celery loses tasks for example: @periodic_task(run_every=datetime.timedelta(minutes=2)) def gen_task(): print('start') add.apply_async((1,2), queue='A'...
1
vote
1answer
20 views

How to use result callback for celery AsyncTask

I have the following situatuion. My client sends following task to the worker: # client task = my_task.apply_async((some_params), queue='my_queue') # task.get() # This blocks My worker executes ...
0
votes
0answers
24 views

How to trigger a periodic calculation function based on local timezones?

I have a function that is to be triggered midnight everyday for daily report calculation of different facilities of a factory. I have written this with the help of Celery and crontab so that it does ...
-1
votes
0answers
16 views

Celery and Flask on a Apache2 server

I created a Flask app using celery, redis and WSGI to run my app on a apache2 server. when I try my app on local host, it works perfectly, it do what I want. By when I try to start it on my server (...
0
votes
1answer
19 views

How to Disable celery tasks logs

i'm using celery to send task. i want to see only 'received task ..' and 'Task .. succeeded' but spider logs on celery server how to disable task logs?
0
votes
1answer
102 views

Celery django : launch worker - ImportError: No module named XXX

I have difficulties to laucnh celery worker. The exemple here works fine and is closed to documentation. My problem is that my real configuration folder construction is like this : djangosite/ |-- ...
0
votes
1answer
11 views

Struggling to save objects to Django database thorough Celery Beat on Digital Ocean

I'm struggling to save objects through Celery Beat to my Django app (showing OHLC data). This script works fine on local environment (saves 3M objects) but not on VPN like Digital Ocean. It saves a ...
0
votes
1answer
13 views

django celery beat arguments with imported class functions from another library

I'm trying to get a function to work in my django project with celerybeat that imports a class based function from a wrapper library. I've been reading that celery doesn't work with classes too ...
1
vote
0answers
26 views

Which is the best way to programatically terminate (cancel) a celery task

According to Celery's documentation, we should not use the terminate option in revoke() function to cancel an executing task: The terminate option is a last resort for administrators when a task ...
0
votes
0answers
23 views

How to configure pycharm debug for docker-compose celery command?

I successfully configure my remote interpreter with docker-compose at Pycharm and can successfully run my server in debug mode to put breakpoints and debug the problems. Unfortunately I can't do the ...
0
votes
1answer
24 views

Pytest django generic relation old object appeared

I have a model: class Product(models.Model): url = models.TextField(blank=True) content_type = models.ForeignKey(ContentType, on_delete=models.CASCADE) object_id = models....
1
vote
1answer
20 views

Celery worker not stopping

I've got a django app running on webfaction, and a couple of days ago I upgraded a package that also upgraded django to version 2.1 (from version 2.0). Shortly thereafter, celery stopped working, and ...
0
votes
1answer
23 views

How to push and pop tasks of the queue in celery

I want to save large amount of images to the image server. I need to queue all requests of images information for saving them using Celery.I use Django framework. I read the document of Celery and ...
0
votes
0answers
12 views

Hung up Rabbitmq server

I am on a mac OS High Sierra 10.13.6, running rabbitmq version 3.7.8 So I was using rabbitmq with celery on a django project. Everything was working nicely. But suddenly it failed with the following ...
0
votes
0answers
18 views

Rabbitmq does not recognize celery queue, exchange

I have an Django application using Rabbitmq as message broker. I defined some queues and exchanges and they worked well locally but not in stagging. Rabbitmq does not change/create new queue/exchanges ...
0
votes
0answers
11 views

Celery conditionally picking up task and setting worker availability

We have a scenario where the celery workers will be responsible for allocating resources for users from a central pool. Is there a way to make the workers smart enough to understand that when the ...
0
votes
0answers
31 views

Using SQLAlchemy scoped_session with Celery

I'm using SQLAlchemy with Celery NOT as a backend (the Celery backend I'm using is RabbitMQ). Reading this question and the SQLAlchemy documentation I come up with this solution: First I've ...
0
votes
1answer
51 views

Celery with continuous deployment

I have a service that exposes an API which is then feeding tasks, it is implemented with Falcon (API) and Celery (task management). Specifically, my workers take long time to load and their code ...
1
vote
0answers
43 views

Celery Task retry on Backend Failure

We are using celery with Redis as the backend and RabbitMQ as the broker. We ran into an interesting scenario which we don't clearly understand how to recover from. The order of task execution is as ...
0
votes
1answer
25 views

Celery tasks suddenly stop executing, nodes can't find each other

I have one master server and 6 worker servers (each with a couple workers running on them). All of a sudden, after many months of flawless execution, all of the tasks stopped getting executed, even ...
1
vote
1answer
36 views

Retrieving executed tasks from Celery Scheduler?

I'm using Python 3.6.6, with the latest versions of Redis, Celery, Celery Beat and Celery Redbeat. I'm currently using Celery redbeat to schedule periodic execution of tasks. Everything is working ...
0
votes
0answers
22 views

Celery Beat Elastic Beanstalk

I am trying to setup celery beat on Elastic Beanstalk. The normal tasks are working perfectly fine, but the beat scheduler seems to have some issues. It is a permission issue and seeing that I am new ...
0
votes
0answers
25 views

Celery is not auto-discovering shared tasks in other modules

I am trying to use celery w/ Django and not getting very far. I am also using django-celery-beat. I don't seem to be discovering the shared tasks in other modules. Here is some code: proj/proj/...
5
votes
0answers
50 views

Reflected SQLAlchemy metadata in celery tasks?

For better testability and other reasons, it is good to have SQLAlchemy database sessions configuration non-global as described very well in the following question: how to setup sqlalchemy session in ...
0
votes
1answer
29 views

At what point does celery store result?

I am using celery + redis for tasks in Django web app. The problem is that I don't know at what point does celery store it's task result as JSON object into DB. Afterwards an issue is raised because ...
0
votes
1answer
17 views

How can I asynchronously receive processed messages in Celery?

I'm writing a data processing pipeline using Celery because this speeds things up considerably. Consider the following pseudo-code: from celery.result import ResultSet from some_celery_app ...
0
votes
0answers
21 views

Python3 exception using shared_task with queue option

I am trying to migrate to python3 (3.6) and having a problem with shared_task from celery, it raises TypeError: an integer is required (got type NoneType). I've managed to narrow down the issue to ...