I have a job I am running in Iguazio. It starts and then the status is "Pending" and the icon is blue. It stays like this indefinitely and there is nothing in the logs that describes what is going on. How do I fix this?
Iguazio job is stuck on 'Pending' status
66 Views Asked by Brennan At
1
There are 1 best solutions below
Related Questions in MLOPS
- Extract current running stage from dvc
- How can I download data from just one of the DVC repositories?
- connection issues when MLFLow is hosted on remote server
- feast.errors.FeatureViewNotFoundException: Feature view driver_stats does not exist
- I have a prolem with feast[redis]
- how can save model by tensorflowlite
- Why MLFlow raising HTTP/2 stream 5 was not closed cleanly before end of the underlying stream?
- Model serving - tools and components
- Unable to properly register model and create Sagemaker Endpoint using Sagemaker Pipelines
- Can MLFlow be used without the `with mlflow.start_run()` block?
- Databricks DBX and Asset Bundles: Support for Storing config files in Container/Storage Account
- Manual Scaling of Nodes on a deployed Vertex AI endpoint
- How to deploy multiple instances PyTorch model API for inference on a single GPU?
- how to import ml model (python) into another programming language
- sagemaker batch transformer with my own pre-trained model
Related Questions in NUCLIO
- Scaling Nuclio With KEDA Based on Queue Length: Error ScaledObject Name is Not Specified
- Integrating nuclio with GCP
- Why is the event offset always 0 for my kafka triggered nuclio function
- How will a nuclio based kafka triggered service behave when it receives a serialized message
- issue with igztop, show mlrun/nuclio function for k8s pods
- MLRun deploy, 0/3 nodes are available: 3 Insufficient cpu
- Pull nuclio metrics into prometheus-operator
- k8s/MLRun, issue with scale to zero
- MLRun, Issue with slow response times
- MLRun, Issue with memory request setup (1B) for nuclio function
- Function cannot be deleted as it is being provisioned
- Facing Error while deploy the serving function in mlrun
- function serving deployment failed
- Nuclio Streaming Contents Support? (Docker setup - Python)
- How do I use secrets in Iguazio?
Related Questions in MLRUN
- Integrating nuclio with GCP
- Getting error while deploying the model in MLRUN
- How to read csv file stored as an artifact in MLrun
- protobuf installed but cannot be imported in Poetry environment
- Access key must be provided in Client() arguments or in the V3IO_ACCESS_KEY environment variable
- MLRun, Role issue - Read only mode for Events, Identity, Grafana, etc
- MLRun ingestion, ConnectionResetError 10054
- Iguazio, Errno 28
- Issue, Iguazio synch Active Directory groups
- issue with igztop, show mlrun/nuclio function for k8s pods
- Issue with create a user in MLRun/Iguazio via API
- MLRun deploy, 0/3 nodes are available: 3 Insufficient cpu
- k8s/MLRun, issue with scale to zero
- MLRun, Issue with slow response times
- MLRun, Issue with memory request setup (1B) for nuclio function
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
Popular # Hahtags
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
A job stuck in this status is usually a Kubernetes issue. The reason there is no logs in the Iguazio dashboard for the job is because the pod never started, which is where the logs come from. You can navigate to the web shell / Jupyter service in Iguazio and use kubectl commands to find out what is going on in Kubernetes. Usually, I see this when there is an issue with the docker image for the pod, it either can’t be found or has bugs.
In a terminal: doing
kubectl get podsand find your pod. It usually hasImagePullBackOff, orCrashLoopBackOffor some similar error. Check the docker image which is usually the culprit. You can kill the pod in Kubernetes, which in turn will error the job out. You can also “abort” the job from the menu in the dashboard under that specific job.