Gui & Celery can't start on docker compose up

Gui & celery can't start. I'm running the following steps on a Mac M1:

clone project
cp config_template.yml config.yml
add API keys
docker compose up --build

The following startup errors occur:
superagi-celery-1           | Usage: celery [OPTIONS] COMMAND [ARGS]...
superagi-celery-1           | Try 'celery --help' for help.
superagi-celery-1           |
superagi-celery-1           | Error: Invalid value for '-A' / '--app':
superagi-celery-1           | Unable to load celery application.
superagi-celery-1           | While trying to load the module superagi.worker the following error occurred:
superagi-celery-1           | Traceback (most recent call last):
superagi-celery-1           |   File "/usr/local/lib/python3.9/site-packages/celery/app/utils.py", line 384, in find_app
superagi-celery-1           |     sym = symbol_by_name(app, imp=imp)
superagi-celery-1           |   File "/usr/local/lib/python3.9/site-packages/kombu/utils/imports.py", line 61, in symbol_by_name
superagi-celery-1           |     return getattr(module, cls_name) if cls_name else module
superagi-celery-1           | AttributeError: module 'superagi' has no attribute 'worker'
superagi-celery-1           |
superagi-celery-1           | During handling of the above exception, another exception occurred:
superagi-celery-1           |
superagi-celery-1           | Traceback (most recent call last):
superagi-celery-1           |   File "/usr/local/lib/python3.9/site-packages/celery/bin/celery.py", line 57, in convert
superagi-celery-1           |     return find_app(value)
superagi-celery-1           |   File "/usr/local/lib/python3.9/site-packages/celery/app/utils.py", line 387, in find_app
superagi-celery-1           |     sym = imp(app)
superagi-celery-1           |   File "/usr/local/lib/python3.9/site-packages/celery/utils/imports.py", line 105, in import_from_cwd
superagi-celery-1           |     return imp(module, package=package)
superagi-celery-1           |   File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
superagi-celery-1           |     return _bootstrap._gcd_import(name[level:], package, level)
superagi-celery-1           |   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
superagi-celery-1           |   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
superagi-celery-1           |   File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
superagi-celery-1           |   File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
superagi-celery-1           |   File "<frozen importlib._bootstrap_external>", line 850, in exec_module
superagi-celery-1           |   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
superagi-celery-1           |   File "/app/superagi/worker.py", line 6, in <module>
superagi-celery-1           |   File "/app/superagi/jobs/agent_executor.py", line 13, in <module>
superagi-celery-1           | ModuleNotFoundError: No module named 'superagi.agent.super_agi'
superagi-celery-1           |
superagi-gui-1              | Error: > Couldn't find any `pages` or `app` directory. Please create one under the project root
superagi-gui-1              |     at findPagesDir (/app/node_modules/next/dist/lib/find-pages-dir.js:94:15)
superagi-gui-1              |     at DevServer.getRoutes (/app/node_modules/next/dist/server/dev/next-dev-server.js:193:71)
superagi-gui-1              |     at new Server (/app/node_modules/next/dist/server/base-server.js:192:47)
superagi-gui-1              |     at new NextNodeServer (/app/node_modules/next/dist/server/next-server.js:172:9)
superagi-gui-1              |     at new DevServer (/app/node_modules/next/dist/server/dev/next-dev-server.js:151:9)
superagi-gui-1              |     at NextServer.createServer (/app/node_modules/next/dist/server/next.js:171:24)
superagi-gui-1              |     at /app/node_modules/next/dist/server/next.js:196:42
superagi-gui-1              |     at async NextServer.prepare (/app/node_modules/next/dist/server/next.js:153:24)
superagi-gui-1              |     at async Server.<anonymous> (/app/node_modules/next/dist/server/lib/render-server.js:121:17) {
superagi-gui-1              |   type: 'Error'
superagi-gui-1              | }
superagi-celery-1 exited with code 2
superagi-gui-1 exited with code 1

The remaining processes superagi-super__redis-1, superagi-super__postgres-1, and superagi-backend-1 seem to be working fine.

@shoptacle can you try pulling the latest main and try docker compose up --build also can you ensure that you have copied config_template.yaml and filled all the keys correctly?

Step 1 guarantees latest main and step 2 was the copy. My API keys are also valid (Pinecone api key, Pinecone env, open AI key, google api key, and google search engine ID).
I'll try on another machine, but tbh those errors don't look related to either machine or api keys.

Had the same issue as well.

@shoptacle @Momof3disneyfans lets have a discussion over this in the discord voice channel
https://discord.gg/ErBU4Ba9

Sounds good, I'll drop by in 10 minutes

Same issue:
module 'superagi' has no attribute 'worker'

Solved with
git checkout v0.0.4

This is related to the naming of the config file. It caught me off guard; the Dockerfile and DockerfileCelery are looking for config.YAML, not config.YML.  Check your naming when you create your config file from the template, as I'm pretty sure this isn't standard (at least not with most projects I've seen).
You can probably mark this one as closed, and if someone wants to assign an issue to me to update the readme, I'll get on it in the morning.

That was a typo above - I did indeed name my files config.yaml (I've corrected the typo above), so that wasn't the cause of the issue for me.
That said, the error for superagi-celery-1 no longer occurs for me though - I've made zero changes in code or config.yaml file on my test repo for this issue. Nor is the the error "Couldn't find any pages or app directory" for superagi-gui-1 displayed.
However, superagi-gui-1 is still not running, it exits with exit code 1 with no visible error. Here is the complete log for that process:
superagi-gui-1              |
superagi-gui-1              | ><EMAIL_ADDRESS>dev
superagi-gui-1              | > next dev
superagi-gui-1              |
superagi-gui-1              | - ready started server on <IP_ADDRESS>:3000, url: http://localhost:3000
superagi-gui-1              | - warn Invalid next.config.js options detected:
superagi-gui-1              | - warn     The root value has an unexpected property, webpackDevMiddleware, which is not in the list of allowed properties (amp, analyticsId, assetPrefix, basePath, cleanDistDir, compiler, compress, configOrigin, crossOrigin, devIndicators, distDir, env, eslint, excludeDefaultMomentLocales, experimental, exportPathMap, generateBuildId, generateEtags, headers, httpAgentOptions, i18n, images, modularizeImports, onDemandEntries, optimizeFonts, output, outputFileTracing, pageExtensions, poweredByHeader, productionBrowserSourceMaps, publicRuntimeConfig, reactStrictMode, redirects, rewrites, sassOptions, serverRuntimeConfig, skipMiddlewareUrlNormalize, skipTrailingSlashRedirect, staticPageGenerationTimeout, swcMinify, target, trailingSlash, transpilePackages, typescript, useFileSystemPublicRoutes, webpack).
superagi-gui-1              | - warn See more info here: https://nextjs.org/docs/messages/invalid-next-config
superagi-gui-1              | Attention: Next.js now collects completely anonymous telemetry regarding usage.
superagi-gui-1              | This information is used to shape Next.js' roadmap and prioritize features.
superagi-gui-1              | You can learn more, including how to opt-out if you'd not like to participate in this anonymous program, by visiting the following URL:
superagi-gui-1              | https://nextjs.org/telemetry
superagi-gui-1              |
superagi-gui-1 exited with code 1

The exit takes about 20 seconds, so it looks like a timeout. I've tried both main & v.0.0.4.

Also if config.yml doesn't work, that really should be it's own issue that's open til there's time to fix that.

Found the problem! It's not related to this project, so I recommend closing this issue.
Issue was I use colima for docker (which is usually awesome).
The docker spins up fine when using docker desktop.

@shoptacle I'm using docker desktop, still getting this error.

@coder-rancho can you share your config and related logs?

@JPDucky I'm not getting any relevant logs
superagi-gui-1              |
superagi-gui-1              | ><EMAIL_ADDRESS>dev
superagi-gui-1              | > next dev
superagi-gui-1              |
superagi-gui-1              |
superagi-gui-1              | - ready started server on <IP_ADDRESS>:3000, url: http://localhost:3000
superagi-backend-1          | Requirement already satisfied: aiohttp==3.8.4 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 1)) (3.8.4)
superagi-backend-1          | Requirement already satisfied: aiosignal==1.3.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 2)) (1.3.1)
superagi-backend-1          | Requirement already satisfied: alembic==1.11.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 3)) (1.11.1)
superagi-backend-1          | Requirement already satisfied: amqp==5.1.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 4)) (5.1.1)
superagi-backend-1          | Requirement already satisfied: anyio==3.7.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 5)) (3.7.0)
superagi-backend-1          | Requirement already satisfied: apiclient==1.0.4 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 6)) (1.0.4)
superagi-backend-1          | Requirement already satisfied: appdirs==1.4.4 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 7)) (1.4.4)
superagi-backend-1          | Requirement already satisfied: async-timeout==4.0.2 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 8)) (4.0.2)
superagi-backend-1          | Requirement already satisfied: attrs==23.1.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 9)) (23.1.0)
superagi-backend-1          | Requirement already satisfied: beautifulsoup4==4.12.2 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 10)) (4.12.2)
superagi-backend-1          | Requirement already satisfied: billiard==<IP_ADDRESS> in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 11)) (<IP_ADDRESS>)
superagi-backend-1          | Requirement already satisfied: boto3==1.26.146 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 12)) (1.26.146)
superagi-backend-1          | Requirement already satisfied: botocore==1.29.146 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 13)) (1.29.146)
superagi-backend-1          | Requirement already satisfied: bs4==0.0.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 14)) (0.0.1)
superagi-backend-1          | Requirement already satisfied: celery==5.2.7 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 15)) (5.2.7)
superagi-backend-1          | Requirement already satisfied: certifi==2023.5.7 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 16)) (2023.5.7)
superagi-backend-1          | Requirement already satisfied: cffi==1.15.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 17)) (1.15.1)
superagi-backend-1          | Requirement already satisfied: charset-normalizer==3.1.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 18)) (3.1.0)
superagi-backend-1          | Requirement already satisfied: click==8.1.3 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 19)) (8.1.3)
superagi-backend-1          | Requirement already satisfied: click-didyoumean==0.3.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 20)) (0.3.0)
superagi-backend-1          | Requirement already satisfied: click-plugins==1.1.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 21)) (1.1.1)
superagi-backend-1          | Requirement already satisfied: click-repl==0.2.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 22)) (0.2.0)
superagi-backend-1          | Requirement already satisfied: colorama==0.4.6 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 23)) (0.4.6)
superagi-backend-1          | Requirement already satisfied: confluent-kafka==2.1.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 24)) (2.1.1)
superagi-backend-1          | Requirement already satisfied: cryptography==41.0.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 25)) (41.0.1)
superagi-backend-1          | Requirement already satisfied: cssselect==1.2.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 26)) (1.2.0)
superagi-backend-1          | Requirement already satisfied: chromadb==0.3.26 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 27)) (0.3.26)
superagi-backend-1          | Requirement already satisfied: dataclasses-json==0.5.7 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 28)) (0.5.7)
superagi-backend-1          | Requirement already satisfied: defusedxml==0.7.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 29)) (0.7.1)
superagi-backend-1          | Requirement already satisfied: docx2txt==0.8 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 30)) (0.8)
superagi-backend-1          | Requirement already satisfied: dnspython==2.3.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 31)) (2.3.0)
superagi-backend-1          | Requirement already satisfied: email-validator==2.0.0.post2 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 32)) (2.0.0.post2)
superagi-backend-1          | Requirement already satisfied: exceptiongroup==1.1.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 33)) (1.1.1)
superagi-backend-1          | Requirement already satisfied: fake-useragent==1.1.3 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 34)) (1.1.3)
superagi-backend-1          | Requirement already satisfied: fastapi==0.95.2 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 35)) (0.95.2)
superagi-backend-1          | Requirement already satisfied: fastapi-jwt-auth==0.5.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 36)) (0.5.0)
superagi-backend-1          | Requirement already satisfied: FastAPI-SQLAlchemy==0.2.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 37)) (0.2.1)
superagi-backend-1          | Requirement already satisfied: feedfinder2==0.0.4 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 38)) (0.0.4)
superagi-backend-1          | Requirement already satisfied: feedparser==6.0.10 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 39)) (6.0.10)
superagi-backend-1          | Requirement already satisfied: filelock==3.12.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 40)) (3.12.0)
superagi-backend-1          | Requirement already satisfied: frozenlist==1.3.3 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 41)) (1.3.3)
superagi-backend-1          | Requirement already satisfied: google-search-results==2.4.2 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 42)) (2.4.2)
superagi-backend-1          | Requirement already satisfied: google-serp-api==1.0.3 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 43)) (1.0.3)
superagi-backend-1          | Requirement already satisfied: google-api-core==2.11.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 44)) (2.11.0)
superagi-backend-1          | Requirement already satisfied: google-api-python-client==2.88.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 45)) (2.88.0)
superagi-backend-1          | Requirement already satisfied: google-auth==2.19.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 46)) (2.19.1)
superagi-backend-1          | Requirement already satisfied: google-auth-httplib2==0.1.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 47)) (0.1.0)
superagi-backend-1          | Requirement already satisfied: google-auth-oauthlib==1.0.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 48)) (1.0.0)
superagi-backend-1          | Requirement already satisfied: greenlet==2.0.2 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 49)) (2.0.2)
superagi-backend-1          | Requirement already satisfied: h11==0.14.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 50)) (0.14.0)
superagi-backend-1          | Requirement already satisfied: halo==0.0.31 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 51)) (0.0.31)
superagi-backend-1          | Requirement already satisfied: httpcore==0.17.2 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 52)) (0.17.2)
superagi-backend-1          | Requirement already satisfied: httptools==0.5.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 53)) (0.5.0)
superagi-backend-1          | Requirement already satisfied: httpx==0.24.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 54)) (0.24.1)
superagi-backend-1          | Requirement already satisfied: idna==3.4 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 55)) (3.4)
superagi-backend-1          | Requirement already satisfied: importlib-metadata==6.6.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 56)) (6.6.0)
superagi-backend-1          | Requirement already satisfied: importlib-resources==5.12.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 57)) (5.12.0)
superagi-backend-1          | Requirement already satisfied: itsdangerous==2.1.2 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 58)) (2.1.2)
superagi-backend-1          | Requirement already satisfied: jieba3k==0.35.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 59)) (0.35.1)
superagi-backend-1          | Requirement already satisfied: Jinja2==3.1.2 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 60)) (3.1.2)
superagi-backend-1          | Requirement already satisfied: jira==3.5.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 61)) (3.5.0)
superagi-backend-1          | Requirement already satisfied: jmespath==1.0.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 62)) (1.0.1)
superagi-backend-1          | Requirement already satisfied: joblib==1.2.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 63)) (1.2.0)
superagi-backend-1          | Requirement already satisfied: json5==0.9.14 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 64)) (0.9.14)
superagi-backend-1          | Requirement already satisfied: jsonmerge==1.9.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 65)) (1.9.0)
superagi-backend-1          | Requirement already satisfied: jsonschema==4.17.3 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 66)) (4.17.3)
superagi-backend-1          | Requirement already satisfied: kombu==5.2.4 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 67)) (5.2.4)
superagi-backend-1          | Requirement already satisfied: llama-index==0.6.35 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 68)) (0.6.35)
superagi-backend-1          | Requirement already satisfied: log-symbols==0.0.14 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 69)) (0.0.14)
superagi-backend-1          | Requirement already satisfied: loguru==0.7.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 70)) (0.7.0)
superagi-backend-1          | Requirement already satisfied: lxml==4.9.2 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 71)) (4.9.2)
superagi-backend-1          | Requirement already satisfied: Mako==1.2.4 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 72)) (1.2.4)
superagi-backend-1          | Requirement already satisfied: MarkupSafe==2.1.2 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 73)) (2.1.2)
superagi-backend-1          | Requirement already satisfied: marshmallow==3.19.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 74)) (3.19.0)
superagi-backend-1          | Requirement already satisfied: marshmallow-enum==1.5.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 75)) (1.5.1)
superagi-backend-1          | Requirement already satisfied: multidict==6.0.4 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 76)) (6.0.4)
superagi-backend-1          | Requirement already satisfied: mypy-extensions==1.0.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 77)) (1.0.0)
superagi-backend-1          | Requirement already satisfied: newspaper3k==0.2.8 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 78)) (0.2.8)
superagi-backend-1          | Requirement already satisfied: nltk==3.8.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 79)) (3.8.1)
superagi-backend-1          | Requirement already satisfied: numexpr==2.8.4 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 80)) (2.8.4)
superagi-backend-1          | Requirement already satisfied: numpy==1.24.3 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 81)) (1.24.3)
superagi-backend-1          | Requirement already satisfied: oauthlib==3.2.2 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 82)) (3.2.2)
superagi-backend-1          | Requirement already satisfied: oauth2client==4.1.3 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 83)) (4.1.3)
superagi-backend-1          | Requirement already satisfied: openai==0.27.7 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 84)) (0.27.7)
superagi-backend-1          | Requirement already satisfied: openapi-schema-pydantic==1.2.4 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 85)) (1.2.4)
superagi-backend-1          | Requirement already satisfied: orjson==3.8.14 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 86)) (3.8.14)
superagi-backend-1          | Requirement already satisfied: packaging==23.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 87)) (23.1)
superagi-backend-1          | Requirement already satisfied: parse==1.19.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 88)) (1.19.0)
superagi-backend-1          | Requirement already satisfied: Pillow==9.5.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 89)) (9.5.0)
superagi-backend-1          | Requirement already satisfied: pinecone-client==2.2.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 90)) (2.2.1)
superagi-backend-1          | Requirement already satisfied: prompt-toolkit==3.0.38 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 91)) (3.0.38)
superagi-backend-1          | Requirement already satisfied: psycopg2==2.9.6 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 92)) (2.9.6)
superagi-backend-1          | Requirement already satisfied: pycparser==2.21 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 93)) (2.21)
superagi-backend-1          | Requirement already satisfied: pydantic==1.10.8 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 94)) (1.10.8)
superagi-backend-1          | Requirement already satisfied: PyJWT==1.7.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 95)) (1.7.1)
superagi-backend-1          | Requirement already satisfied: PyPDF2==3.0.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 96)) (3.0.1)
superagi-backend-1          | Requirement already satisfied: pyquery==2.0.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 97)) (2.0.0)
superagi-backend-1          | Requirement already satisfied: pyrsistent==0.19.3 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 98)) (0.19.3)
superagi-backend-1          | Requirement already satisfied: pytest==7.3.2 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 99)) (7.3.2)
superagi-backend-1          | Requirement already satisfied: python-dateutil==2.8.2 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 100)) (2.8.2)
superagi-backend-1          | Requirement already satisfied: python-dotenv==1.0.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 101)) (1.0.0)
superagi-backend-1          | Requirement already satisfied: python-multipart==0.0.6 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 102)) (0.0.6)
superagi-backend-1          | Requirement already satisfied: pytz==2023.3 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 103)) (2023.3)
superagi-backend-1          | Requirement already satisfied: PyYAML==6.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 104)) (6.0)
superagi-backend-1          | Requirement already satisfied: qdrant-client==1.3.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 105)) (1.3.1)
superagi-backend-1          | Requirement already satisfied: redis==4.5.5 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 106)) (4.5.5)
superagi-backend-1          | Requirement already satisfied: regex==2023.5.5 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 107)) (2023.5.5)
superagi-backend-1          | Requirement already satisfied: requests==2.31.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 108)) (2.31.0)
superagi-backend-1          | Requirement already satisfied: requests-file==1.5.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 109)) (1.5.1)
superagi-backend-1          | Requirement already satisfied: requests-html==0.10.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 110)) (0.10.0)
superagi-backend-1          | Requirement already satisfied: requests-oauthlib==1.3.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 111)) (1.3.1)
superagi-backend-1          | Requirement already satisfied: requests-toolbelt==1.0.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 112)) (1.0.0)
superagi-backend-1          | Requirement already satisfied: s3transfer==0.6.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 113)) (0.6.1)
superagi-backend-1          | Requirement already satisfied: sgmllib3k==1.0.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 114)) (1.0.0)
superagi-backend-1          | Requirement already satisfied: six==1.16.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 115)) (1.16.0)
superagi-backend-1          | Requirement already satisfied: sniffio==1.3.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 116)) (1.3.0)
superagi-backend-1          | Requirement already satisfied: soupsieve==2.4.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 117)) (2.4.1)
superagi-backend-1          | Requirement already satisfied: spinners==0.0.24 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 118)) (0.0.24)
superagi-backend-1          | Requirement already satisfied: starlette==0.27.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 119)) (0.27.0)
superagi-backend-1          | Requirement already satisfied: SQLAlchemy==2.0.16 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 120)) (2.0.16)
superagi-backend-1          | Requirement already satisfied: tenacity==8.2.2 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 121)) (8.2.2)
superagi-backend-1          | Requirement already satisfied: termcolor==2.3.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 122)) (2.3.0)
superagi-backend-1          | Requirement already satisfied: tiktoken==0.4.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 123)) (0.4.0)
superagi-backend-1          | Requirement already satisfied: tinysegmenter==0.3 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 124)) (0.3)
superagi-backend-1          | Requirement already satisfied: tldextract==3.4.4 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 125)) (3.4.4)
superagi-backend-1          | Requirement already satisfied: tqdm==4.65.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 126)) (4.65.0)
superagi-backend-1          | Requirement already satisfied: tweepy==4.14.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 127)) (4.14.0)
superagi-backend-1          | Requirement already satisfied: typing-inspect==0.8.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 128)) (0.8.0)
superagi-backend-1          | Requirement already satisfied: ujson==5.7.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 129)) (5.7.0)
superagi-backend-1          | Requirement already satisfied: urllib3==1.26.16 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 130)) (1.26.16)
superagi-backend-1          | Requirement already satisfied: uvicorn==0.22.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 131)) (0.22.0)
superagi-backend-1          | Requirement already satisfied: vine==5.0.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 132)) (5.0.0)
superagi-backend-1          | Requirement already satisfied: w3lib==2.1.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 133)) (2.1.1)
superagi-backend-1          | Requirement already satisfied: watchfiles==0.19.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 134)) (0.19.0)
superagi-backend-1          | Requirement already satisfied: wcwidth==0.2.6 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 135)) (0.2.6)
superagi-backend-1          | Requirement already satisfied: weaviate-client==3.20.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 136)) (3.20.1)
superagi-backend-1          | Requirement already satisfied: websockets==10.4 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 137)) (10.4)
superagi-backend-1          | Requirement already satisfied: yarl==1.9.2 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 138)) (1.9.2)
superagi-backend-1          | Requirement already satisfied: zipp==3.15.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 139)) (3.15.0)
superagi-backend-1          | Requirement already satisfied: slack-sdk==3.21.3 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 142)) (3.21.3)
superagi-backend-1          | Requirement already satisfied: pylint==2.17.4 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 144)) (2.17.4)
superagi-backend-1          | Requirement already satisfied: pre-commit==3.3.3 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 145)) (3.3.3)
superagi-backend-1          | Requirement already satisfied: pytest-cov==4.1.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 146)) (4.1.0)
superagi-backend-1          | Requirement already satisfied: pytest-mock==3.11.1 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 147)) (3.11.1)
superagi-backend-1          | Requirement already satisfied: transformers==4.30.2 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 148)) (4.30.2)
superagi-backend-1          | Requirement already satisfied: pypdf==3.11.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 149)) (3.11.0)
superagi-backend-1          | Requirement already satisfied: python-pptx==0.6.21 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 150)) (0.6.21)
superagi-backend-1          | Requirement already satisfied: EbookLib==0.18 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 152)) (0.18)
superagi-backend-1          | Requirement already satisfied: html2text==2020.1.16 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 153)) (2020.1.16)
superagi-backend-1          | Requirement already satisfied: duckduckgo-search==3.8.3 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 154)) (3.8.3)
superagi-backend-1          | Requirement already satisfied: google-generativeai==0.1.0 in /usr/local/lib/python3.9/site-packages (from -r /app/requirements.txt (line 155)) (0.1.0)
superagi-backend-1          | Requirement already satisfied: typing-extensions>=4 in /usr/local/lib/python3.9/site-packages (from alembic==1.11.1->-r /app/requirements.txt (line 3)) (4.5.0)
superagi-celery-1           | 2023-07-30 09:43:03 UTC - Super AGI - INFO - [/app/superagi/models/db.py:43] - Connected to the database! @ postgresql://superagi:password@super__postgres/super_agi_main
superagi-celery-1           | /usr/local/lib/python3.9/site-packages/celery/platforms.py:840: SecurityWarning: You're running the worker with superuser privileges: this is
superagi-celery-1           | absolutely not recommended!
superagi-celery-1           |
superagi-celery-1           | Please specify a different user using the --uid option.
superagi-celery-1           |
superagi-celery-1           | User information: uid=0 euid=0 gid=0 egid=0
superagi-celery-1           |
superagi-celery-1           |   warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
superagi-celery-1           |
superagi-celery-1           |
superagi-celery-1           |  -------------- celery@5d17b3594bde v5.2.7 (dawn-chorus)
superagi-celery-1           | --- ***** -----
superagi-celery-1           | -- ******* ---- Linux-5.15.49-linuxkit-pr-x86_64-with-glibc2.36 2023-07-30 09:43:03
superagi-celery-1           | - *** --- * ---
superagi-celery-1           | - ** ---------- [config]
superagi-celery-1           | - ** ---------- .> app:         superagi:0x7f6f400294c0
superagi-celery-1           | - ** ---------- .> transport:   redis://super__redis:6379/0
superagi-celery-1           | - ** ---------- .> results:     redis://super__redis:6379/0
superagi-celery-1           | - *** --- * --- .> concurrency: 10 (prefork)
superagi-celery-1           | -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
superagi-celery-1           | --- ***** -----
superagi-celery-1           |  -------------- [queues]
superagi-celery-1           |                 .> celery           exchange=celery(direct) key=celery
superagi-celery-1           |
superagi-celery-1           |
superagi-celery-1           | [tasks]
superagi-celery-1           |   . execute_agent
superagi-celery-1           |   . initialize-schedule-agent
superagi-celery-1           |   . summarize_resource
superagi-celery-1           |
superagi-celery-1           |
superagi-backend-1          | Requirement already satisfied: pandas>=1.3 in /usr/local/lib/python3.9/site-packages (from chromadb==0.3.26->-r /app/requirements.txt (line 27)) (2.0.3)
superagi-backend-1          | Requirement already satisfied: hnswlib>=0.7 in /usr/local/lib/python3.9/site-packages (from chromadb==0.3.26->-r /app/requirements.txt (line 27)) (0.7.0)
superagi-backend-1          | Requirement already satisfied: clickhouse-connect>=0.5.7 in /usr/local/lib/python3.9/site-packages (from chromadb==0.3.26->-r /app/requirements.txt (line 27)) (0.6.8)
superagi-backend-1          | Requirement already satisfied: duckdb>=0.7.1 in /usr/local/lib/python3.9/site-packages (from chromadb==0.3.26->-r /app/requirements.txt (line 27)) (0.8.1)
superagi-backend-1          | Requirement already satisfied: posthog>=2.4.0 in /usr/local/lib/python3.9/site-packages (from chromadb==0.3.26->-r /app/requirements.txt (line 27)) (3.0.1)
superagi-backend-1          | Requirement already satisfied: pulsar-client>=3.1.0 in /usr/local/lib/python3.9/site-packages (from chromadb==0.3.26->-r /app/requirements.txt (line 27)) (3.2.0)
superagi-backend-1          | Requirement already satisfied: onnxruntime>=1.14.1 in /usr/local/lib/python3.9/site-packages (from chromadb==0.3.26->-r /app/requirements.txt (line 27)) (1.15.1)
superagi-backend-1          | Requirement already satisfied: tokenizers>=0.13.2 in /usr/local/lib/python3.9/site-packages (from chromadb==0.3.26->-r /app/requirements.txt (line 27)) (0.13.3)
superagi-backend-1          | Requirement already satisfied: overrides>=7.3.1 in /usr/local/lib/python3.9/site-packages (from chromadb==0.3.26->-r /app/requirements.txt (line 27)) (7.3.1)
superagi-backend-1          | Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.56.2 in /usr/local/lib/python3.9/site-packages (from google-api-core==2.11.0->-r /app/requirements.txt (line 44)) (1.59.1)
superagi-backend-1          | Requirement already satisfied: protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 in /usr/local/lib/python3.9/site-packages (from google-api-core==2.11.0->-r /app/requirements.txt (line 44)) (4.23.4)
superagi-backend-1          | Requirement already satisfied: httplib2<1dev,>=0.15.0 in /usr/local/lib/python3.9/site-packages (from google-api-python-client==2.88.0->-r /app/requirements.txt (line 45)) (0.22.0)
superagi-backend-1          | Requirement already satisfied: uritemplate<5,>=3.0.1 in /usr/local/lib/python3.9/site-packages (from google-api-python-client==2.88.0->-r /app/requirements.txt (line 45)) (4.1.1)
superagi-backend-1          | Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.9/site-packages (from google-auth==2.19.1->-r /app/requirements.txt (line 46)) (5.3.1)
superagi-backend-1          | Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.9/site-packages (from google-auth==2.19.1->-r /app/requirements.txt (line 46)) (0.3.0)
superagi-backend-1          | Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.9/site-packages (from google-auth==2.19.1->-r /app/requirements.txt (line 46)) (4.9)
superagi-celery-1           | [2023-07-30 09:43:04,384: INFO/MainProcess] Connected to redis://super__redis:6379/0
superagi-celery-1           | [2023-07-30 09:43:04,396: INFO/MainProcess] mingle: searching for neighbors
superagi-backend-1          | Requirement already satisfied: langchain>=0.0.154 in /usr/local/lib/python3.9/site-packages (from llama-index==0.6.35->-r /app/requirements.txt (line 68)) (0.0.247)
superagi-backend-1          | Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.9/site-packages (from llama-index==0.6.35->-r /app/requirements.txt (line 68)) (2023.6.0)
superagi-backend-1          | Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python3.9/site-packages (from oauth2client==4.1.3->-r /app/requirements.txt (line 83)) (0.5.0)
superagi-celery-1           | [2023-07-30 09:43:05,409: INFO/MainProcess] mingle: all alone
superagi-celery-1           | [2023-07-30 09:43:05,426: INFO/MainProcess] celery@5d17b3594bde ready.
superagi-backend-1          | Requirement already satisfied: iniconfig in /usr/local/lib/python3.9/site-packages (from pytest==7.3.2->-r /app/requirements.txt (line 99)) (2.0.0)
superagi-backend-1          | Requirement already satisfied: pluggy<2.0,>=0.12 in /usr/local/lib/python3.9/site-packages (from pytest==7.3.2->-r /app/requirements.txt (line 99)) (1.2.0)
superagi-backend-1          | Requirement already satisfied: tomli>=1.0.0 in /usr/local/lib/python3.9/site-packages (from pytest==7.3.2->-r /app/requirements.txt (line 99)) (2.0.1)
superagi-backend-1          | Requirement already satisfied: grpcio>=1.41.0 in /usr/local/lib/python3.9/site-packages (from qdrant-client==1.3.1->-r /app/requirements.txt (line 105)) (1.56.2)
superagi-backend-1          | Requirement already satisfied: grpcio-tools>=1.41.0 in /usr/local/lib/python3.9/site-packages (from qdrant-client==1.3.1->-r /app/requirements.txt (line 105)) (1.56.2)
superagi-backend-1          | Requirement already satisfied: portalocker<3.0.0,>=2.7.0 in /usr/local/lib/python3.9/site-packages (from qdrant-client==1.3.1->-r /app/requirements.txt (line 105)) (2.7.0)
superagi-celery-1           | [2023-07-30 09:43:05,821: INFO/Beat] beat: Starting...
superagi-backend-1          | Requirement already satisfied: pyppeteer>=0.0.14 in /usr/local/lib/python3.9/site-packages (from requests-html==0.10.0->-r /app/requirements.txt (line 110)) (1.0.2)
superagi-celery-1           | [2023-07-30 09:43:06,017: INFO/Beat] Scheduler: Sending due task initialize-schedule-agent (initialize-schedule-agent)
superagi-celery-1           | [2023-07-30 09:43:06,050: INFO/MainProcess] Task initialize-schedule-agent[759d69ac-c90c-41d3-949c-8620b98ddeee] received
superagi-celery-1           | [2023-07-30 09:43:06,114: INFO/ForkPoolWorker-8] Task initialize-schedule-agent[759d69ac-c90c-41d3-949c-8620b98ddeee] succeeded in 0.06132132899801945s: None
superagi-backend-1          | Requirement already satisfied: validators<=0.21.0,>=0.18.2 in /usr/local/lib/python3.9/site-packages (from weaviate-client==3.20.1->-r /app/requirements.txt (line 136)) (0.20.0)
superagi-backend-1          | Requirement already satisfied: authlib>=1.1.0 in /usr/local/lib/python3.9/site-packages (from weaviate-client==3.20.1->-r /app/requirements.txt (line 136)) (1.2.1)
superagi-backend-1          | Requirement already satisfied: platformdirs>=2.2.0 in /usr/local/lib/python3.9/site-packages (from pylint==2.17.4->-r /app/requirements.txt (line 144)) (3.10.0)
superagi-backend-1          | Requirement already satisfied: astroid<=2.17.0-dev0,>=2.15.4 in /usr/local/lib/python3.9/site-packages (from pylint==2.17.4->-r /app/requirements.txt (line 144)) (2.15.6)
superagi-backend-1          | Requirement already satisfied: isort<6,>=4.2.5 in /usr/local/lib/python3.9/site-packages (from pylint==2.17.4->-r /app/requirements.txt (line 144)) (5.12.0)
superagi-backend-1          | Requirement already satisfied: mccabe<0.8,>=0.6 in /usr/local/lib/python3.9/site-packages (from pylint==2.17.4->-r /app/requirements.txt (line 144)) (0.7.0)
superagi-backend-1          | Requirement already satisfied: tomlkit>=0.10.1 in /usr/local/lib/python3.9/site-packages (from pylint==2.17.4->-r /app/requirements.txt (line 144)) (0.12.1)
superagi-backend-1          | Requirement already satisfied: dill>=0.2 in /usr/local/lib/python3.9/site-packages (from pylint==2.17.4->-r /app/requirements.txt (line 144)) (0.3.7)
superagi-backend-1          | Requirement already satisfied: cfgv>=2.0.0 in /usr/local/lib/python3.9/site-packages (from pre-commit==3.3.3->-r /app/requirements.txt (line 145)) (3.3.1)
superagi-backend-1          | Requirement already satisfied: identify>=1.0.0 in /usr/local/lib/python3.9/site-packages (from pre-commit==3.3.3->-r /app/requirements.txt (line 145)) (2.5.26)
superagi-backend-1          | Requirement already satisfied: nodeenv>=0.11.1 in /usr/local/lib/python3.9/site-packages (from pre-commit==3.3.3->-r /app/requirements.txt (line 145)) (1.8.0)
superagi-backend-1          | Requirement already satisfied: virtualenv>=20.10.0 in /usr/local/lib/python3.9/site-packages (from pre-commit==3.3.3->-r /app/requirements.txt (line 145)) (20.24.1)
superagi-backend-1          | Requirement already satisfied: coverage[toml]>=5.2.1 in /usr/local/lib/python3.9/site-packages (from pytest-cov==4.1.0->-r /app/requirements.txt (line 146)) (7.2.7)
superagi-backend-1          | Requirement already satisfied: huggingface-hub<1.0,>=0.14.1 in /usr/local/lib/python3.9/site-packages (from transformers==4.30.2->-r /app/requirements.txt (line 148)) (0.16.4)
superagi-backend-1          | Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.9/site-packages (from transformers==4.30.2->-r /app/requirements.txt (line 148)) (0.3.1)
superagi-backend-1          | Requirement already satisfied: XlsxWriter>=0.5.7 in /usr/local/lib/python3.9/site-packages (from python-pptx==0.6.21->-r /app/requirements.txt (line 150)) (3.1.2)
superagi-backend-1          | Requirement already satisfied: aiofiles>=23.1.0 in /usr/local/lib/python3.9/site-packages (from duckduckgo-search==3.8.3->-r /app/requirements.txt (line 154)) (23.1.0)
superagi-backend-1          | Requirement already satisfied: google-ai-generativelanguage==0.2.0 in /usr/local/lib/python3.9/site-packages (from google-generativeai==0.1.0->-r /app/requirements.txt (line 155)) (0.2.0)
superagi-backend-1          | Requirement already satisfied: proto-plus<2.0.0dev,>=1.22.0 in /usr/local/lib/python3.9/site-packages (from google-ai-generativelanguage==0.2.0->google-generativeai==0.1.0->-r /app/requirements.txt (line 155)) (1.22.3)
superagi-backend-1          | Requirement already satisfied: lazy-object-proxy>=1.4.0 in /usr/local/lib/python3.9/site-packages (from astroid<=2.17.0-dev0,>=2.15.4->pylint==2.17.4->-r /app/requirements.txt (line 144)) (1.9.0)
superagi-backend-1          | Requirement already satisfied: wrapt<2,>=1.11 in /usr/local/lib/python3.9/site-packages (from astroid<=2.17.0-dev0,>=2.15.4->pylint==2.17.4->-r /app/requirements.txt (line 144)) (1.15.0)
superagi-backend-1          | Requirement already satisfied: zstandard in /usr/local/lib/python3.9/site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.26->-r /app/requirements.txt (line 27)) (0.21.0)
superagi-backend-1          | Requirement already satisfied: lz4 in /usr/local/lib/python3.9/site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.26->-r /app/requirements.txt (line 27)) (4.3.2)
superagi-backend-1          | Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from grpcio-tools>=1.41.0->qdrant-client==1.3.1->-r /app/requirements.txt (line 105)) (58.1.0)
superagi-backend-1          | Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in /usr/local/lib/python3.9/site-packages (from httplib2<1dev,>=0.15.0->google-api-python-client==2.88.0->-r /app/requirements.txt (line 45)) (3.1.0)
superagi-backend-1          | Requirement already satisfied: socksio==1.* in /usr/local/lib/python3.9/site-packages (from httpx==0.24.1->-r /app/requirements.txt (line 54)) (1.0.0)
superagi-backend-1          | Requirement already satisfied: brotli in /usr/local/lib/python3.9/site-packages (from httpx==0.24.1->-r /app/requirements.txt (line 54)) (1.0.9)
superagi-backend-1          | Requirement already satisfied: h2<5,>=3 in /usr/local/lib/python3.9/site-packages (from httpx==0.24.1->-r /app/requirements.txt (line 54)) (4.1.0)
superagi-backend-1          | Requirement already satisfied: langsmith<0.1.0,>=0.0.11 in /usr/local/lib/python3.9/site-packages (from langchain>=0.0.154->llama-index==0.6.35->-r /app/requirements.txt (line 68)) (0.0.15)
superagi-backend-1          | Requirement already satisfied: coloredlogs in /usr/local/lib/python3.9/site-packages (from onnxruntime>=1.14.1->chromadb==0.3.26->-r /app/requirements.txt (line 27)) (15.0.1)
superagi-backend-1          | Requirement already satisfied: flatbuffers in /usr/local/lib/python3.9/site-packages (from onnxruntime>=1.14.1->chromadb==0.3.26->-r /app/requirements.txt (line 27)) (23.5.26)
superagi-backend-1          | Requirement already satisfied: sympy in /usr/local/lib/python3.9/site-packages (from onnxruntime>=1.14.1->chromadb==0.3.26->-r /app/requirements.txt (line 27)) (1.12)
superagi-backend-1          | Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.9/site-packages (from pandas>=1.3->chromadb==0.3.26->-r /app/requirements.txt (line 27)) (2023.3)
superagi-backend-1          | Requirement already satisfied: monotonic>=1.5 in /usr/local/lib/python3.9/site-packages (from posthog>=2.4.0->chromadb==0.3.26->-r /app/requirements.txt (line 27)) (1.6)
superagi-backend-1          | Requirement already satisfied: backoff>=1.10.0 in /usr/local/lib/python3.9/site-packages (from posthog>=2.4.0->chromadb==0.3.26->-r /app/requirements.txt (line 27)) (2.2.1)
superagi-backend-1          | Requirement already satisfied: pyee<9.0.0,>=8.1.0 in /usr/local/lib/python3.9/site-packages (from pyppeteer>=0.0.14->requests-html==0.10.0->-r /app/requirements.txt (line 110)) (8.2.2)
superagi-backend-1          | Requirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in /usr/local/lib/python3.9/site-packages (from uvicorn==0.22.0->-r /app/requirements.txt (line 131)) (0.17.0)
superagi-backend-1          | Requirement already satisfied: decorator>=3.4.0 in /usr/local/lib/python3.9/site-packages (from validators<=0.21.0,>=0.18.2->weaviate-client==3.20.1->-r /app/requirements.txt (line 136)) (5.1.1)
superagi-backend-1          | Requirement already satisfied: distlib<1,>=0.3.6 in /usr/local/lib/python3.9/site-packages (from virtualenv>=20.10.0->pre-commit==3.3.3->-r /app/requirements.txt (line 145)) (0.3.7)
superagi-backend-1          | Requirement already satisfied: grpcio-status<2.0dev,>=1.33.2 in /usr/local/lib/python3.9/site-packages (from google-api-core==2.11.0->-r /app/requirements.txt (line 44)) (1.56.2)
superagi-backend-1          | Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.9/site-packages (from h2<5,>=3->httpx==0.24.1->-r /app/requirements.txt (line 54)) (6.0.1)
superagi-backend-1          | Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.9/site-packages (from h2<5,>=3->httpx==0.24.1->-r /app/requirements.txt (line 54)) (4.0.0)
superagi-backend-1          | Requirement already satisfied: humanfriendly>=9.1 in /usr/local/lib/python3.9/site-packages (from coloredlogs->onnxruntime>=1.14.1->chromadb==0.3.26->-r /app/requirements.txt (line 27)) (10.0)
superagi-backend-1          | Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.9/site-packages (from sympy->onnxruntime>=1.14.1->chromadb==0.3.26->-r /app/requirements.txt (line 27)) (1.3.0)
superagi-backend-1          | WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
superagi-backend-1          | INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
superagi-backend-1          | INFO  [alembic.runtime.migration] Will assume transactional DDL.
superagi-backend-1          | INFO:     Will watch for changes in these directories: ['/app']
superagi-backend-1          | INFO:     Uvicorn running on http://<IP_ADDRESS>:8001 (Press CTRL+C to quit)
superagi-backend-1          | INFO:     Started reloader process [1] using WatchFiles
superagi-backend-1          | 2023-07-30 09:43:18 UTC - Super AGI - INFO - [/app/superagi/models/db.py:43] - Connected to the database! @ postgresql://superagi:password@super__postgres/super_agi_main
superagi-gui-1              | - event compiled client and server successfully in 17s (1636 modules)
superagi-backend-1          | INFO:     Started server process [27]
superagi-backend-1          | INFO:     Waiting for application startup.
superagi-backend-1          | 2023-07-30 09:43:22 UTC - Super AGI - INFO - [/app/main.py:155] - Running Startup tasks
superagi-backend-1          | 2023-07-30 09:43:23 UTC - Super AGI - INFO - [/app/main.py:159] - None
superagi-backend-1          | 2023-07-30 09:43:23 UTC - Super AGI - INFO - [/app/main.py:321] - Successfully registered local toolkits for all Organisations!
superagi-backend-1          | INFO:     Application startup complete.
superagi-gui-1 exited with code 1


Add steps to import DB snapshots in Lisk docs

Closes #1751

Hi @chris529 Let's add the same section to the docker page as well because it is applicable for both docker and pm2.

@nagdahimanshu Many thanks. Could you confirm if we still use the environment variables, as it is mentioned in the Update Lisk Service Configuration Reference PR that we don't use these anymore?
https://github.com/LiskHQ/lisk-service/pull/1799/files/571209645824f8c8066e97fe0b5f7ba1e2a10ef0#diff-a500397cf9d9162bf399f1087d2299ac895e6f7cf9f64a915e641e7525b51208


Error: Cannot find module 'appium-flutter-finder'

Hello,
Im having a problem with importing appium-flutter-finder, from the appium-flutter-driver package.
Im trying:
const { byValueKey } = require('appium-flutter-finder');
i have also tried:
const { byValueKey } = require('appium-flutter-driver');
which does work, but i cannot use find module
I have tried to install globally and locally within a folder, but both approaches dont work
npm i -g appium-flutter-driver
npm i appium-flutter-driver

Error: Cannot find module 'appium-flutter-finder'
Require stack:

/Users/.../app/appium/test.js
at Function.Module._resolveFilename (node:internal/modules/cjs/loader:927:15)
at Function.Module._load (node:internal/modules/cjs/loader:772:27)
at Module.require (node:internal/modules/cjs/loader:999:19)
at require (node:internal/modules/cjs/helpers:93:18)
at Object. (/Users/.../app/appium/test.js:3:24)
at Module._compile (node:internal/modules/cjs/loader:1095:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1124:10)
at Module.load (node:internal/modules/cjs/loader:975:32)
at Function.Module._load (node:internal/modules/cjs/loader:816:12)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:79:12) {
code: 'MODULE_NOT_FOUND',
requireStack: [ '/Users/.../app/appium/test.js' ]
I cant figure out why it wont work. Any help would be appreciated. Thanks!

I think yes, it is a JS client side module. Not part of appium-flutter-driver.
https://github.com/appium-userland/appium-flutter-driver/tree/main/finder/nodejs


Is it possible to report p-value for mutational signature

Hi
I have obtained mutational signature from my samples like below

You are please seeing, signature 1 fits to COSMIC_2 with 0.89 similarity but you also see only one sample is underlying getting COSMIC_2 signature
I think it's Ok to report Cos similarity, but less important than the odds ratio of enrichment or the p-value
My question is, is it possible to report p-value or odd ratio for this similarity?
Thanks a lot for any information

Hello,
As far as I know, it is not possible to calculate OR or P-value for cosine similarity. Feel free to correct me if I am wrong or point me to a reference, I am more than happy to implement it.

Thank you so much
Actually I showed my output to the writer of this paper
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5957269/
And she pointed this to me as a co-author for the data we submitted to ISDE congress
However in this website they are likely using p-value to report the best solution
https://signal.mutationalsignatures.com/analyse/jobs/Example/sample/Sample
They are saying
The algorithm that was used to determine the signatures contributing to your sample generates multiple solutions via bootstrapping. The result given in the table and chart above (if present) corresponds to the median contribution count for each signature's distribution of potential counts across each bootstrap, or 0 for signatures whose count falls below the sparsity filter threshold (dashed green line) for a significant fraction (p-value) of the solutions.

I am not really aware of the tool you are referring and I cannot comment on its output. Regardless, I would also suggest you to try tools like deconstructSigs if your goal is to get signature proportion per sample.

The functionality provided by maftools focus on signatures identified from a group of samples. If you want to check if a signature exists in a sample, as @PoisonAlien suggested, you should find other tools, like the new paper used the bootstrapping.

Thank you both but maftools also provides signature proportion per sample as in this picture

This is signature per cluster

Thank you
I think it's great
Can we make a stack plot of that to show exposure proportion?

@beginner984 Use show_sig_exposure(sig)
If you have any problem, you can open an issue at https://github.com/ShixiangWang/sigminer/issues, hope this helps you close the issue here.

Thank you @ShixiangWang
Where I can find a tutorial on your package?
I found this
https://shixiangwang.github.io/sigminer-doc/references.html
But that is empty
I mean something to use all of the capability of your package like the same maftools offers
I can find individual R codes in your git but I can not follow that well
Thank you

Hello @beginner984 ,
Here is how those barplots are drawn just to clarify certain things.

Estimate de-novo signatures with extractSignatures. In your case there are to 3 signatures.
Above step also gives you proportion of each of these 3 signatures in every sample. You can visualise these with maftools::plotSignatures(contributions = TRUE)
With this matrix of proportions, next step is to divide your cohort into k parts using k-means (here k = 3).
The barplot shows average contribution of all 3 signatures among 3 clusters. It seems that samples in cluster 1 have high load of signature-2, and so on..

This is a crude method of assigning samples to signatures. Other methods such as deconstructSigs  (and probably @ShixiangWang 's sigminer) does it in much better way.
Hope this is clear.

@beginner984 See https://shixiangwang.github.io/sigminer/reference/index.html for now, this tool is still under development, and I am writing the doc. sigminer focuses on the signature analysis, currently, I directly use maftools code to generate SNV matrix for NMF. Thank you for your previous work, @PoisonAlien :), I hope this package is worth to be recommended in maftools in the future 😄

@PoisonAlien  and @ShixiangWang, hello
I know in maftools implemented files and the other way like
sp_url <- paste("https://cancer.sanger.ac.uk/cancergenome/assets/",
+                 "signatures_probabilities.txt", sep = "")

implemented in MutationalPattern R package
I can download the probabilities of 30 mutational signatures from the COSMIC website but version 2 (30 signatures)
Anyone knows where I can obtain these probabilities for version 3 (60 signatures)
Thanks in advance

Use the code to get them from Maftools
      sigs_db <- readRDS(file = system.file("extdata", "SBS_signatures.RDs",
        package = "maftools", mustWork = TRUE
      ))

Thank you @ShixiangWang
How I convert sigs_db to a matrix like this in COSMIC
> head(cancer_signatures)
        Signature.1  Signature.2 Signature.3 Signature.4 Signature.5 Signature.6
A[C>A]A 0.011098326 6.827082e-04  0.02217231      0.0365 0.014941548      0.0017
A[C>A]C 0.009149341 6.191072e-04  0.01787168      0.0309 0.008960918      0.0028
A[C>A]G 0.001490070 9.927896e-05  0.00213834      0.0183 0.002207846      0.0005
A[C>A]T 0.006233885 3.238914e-04  0.01626515      0.0243 0.009206905      0.0019
C[C>A]A 0.006595870 6.774450e-04  0.01878173      0.0461 0.009674904      0.0101
C[C>A]C 0.007342368 2.136810e-04  0.01576046      0.0614 0.004952301      0.0241

Hello @beginner984
These trivial things must be resolved by your own. Please restrict your questions to actual issues associated with the package.

@PoisonAlien I totally agree with you. @beginner984 If you checked the result, you should know that you can obtain what you want very easy, just call sigs_db$db.


raydp.init_spark fails

This line of code fails:
spark = raydp.init_spark(app_name="RayDP example",
num_executors=2,
executor_cores=2,
executor_memory="4GB")
I am getting the following errors (linux server with a standalone Spark cluster):
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.apache.spark.deploy.raydp.AppMasterEntryPoint.main(AppMasterEntryPoint.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.internal.Logging
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 13 more
Traceback (most recent call last):
File "/home/guryaniv/try_raydp_simple.py", line 4, in
spark = raydp.init_spark(app_name="RayDP example", num_executors=2, executor_cores=2, executor_memory="4GB")
File "/opt/anaconda3/envs/raydp/lib/python3.6/site-packages/raydp/context.py", line 122, in init_spark
return _global_spark_context.get_or_create_session()
File "/opt/anaconda3/envs/raydp/lib/python3.6/site-packages/raydp/context.py", line 68, in get_or_create_session
spark_cluster = self._get_or_create_spark_cluster()
File "/opt/anaconda3/envs/raydp/lib/python3.6/site-packages/raydp/context.py", line 62, in _get_or_create_spark_cluster
self._spark_cluster = SparkCluster()
File "/opt/anaconda3/envs/raydp/lib/python3.6/site-packages/raydp/spark/ray_cluster.py", line 31, in init
self._set_up_master(None, None)
File "/opt/anaconda3/envs/raydp/lib/python3.6/site-packages/raydp/spark/ray_cluster.py", line 37, in _set_up_master
self._app_master_bridge.start_up()
File "/opt/anaconda3/envs/raydp/lib/python3.6/site-packages/raydp/spark/ray_cluster_master.py", line 52, in start_up
self._gateway = self._launch_gateway(extra_classpath, popen_kwargs)
File "/opt/anaconda3/envs/raydp/lib/python3.6/site-packages/raydp/spark/ray_cluster_master.py", line 115, in _launch_gateway
raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number
To run the code I am using spark-submit with the spark master:
spark-submit --master spark://:7077 raydp_example.py

Hi @yanivg10. In RayDP, we running spark on ray which means we do not need another resource manager (eg: standalone, yarn).
# first step, connect or init local ray cluster
ray.init(...)
# second step, startup spark cluster on top of the ray cluster
spark = raydp.init_spark(...)
# third step, using spark as normal
spark....
# stop spark
raydp.stop_spark(...)

And also, you should keep your SPARK_HOME environments should be compatible with the raydp required. Actually, you do not need to set SPARK_HOME because we use pyspark.

I already tried these steps using the spark on ray setup and could not get init_spark to work. ray.init() works fine.
raydp.init_spark() throws the following exception (I am running the code within a conda "raydp" virtual environment I created):
Exception                                 Traceback (most recent call last)
 in
4 cores_per_executor = 1
5 memory_per_executor = "2GB"
----> 6 spark = raydp.init_spark(app_name, num_executors, cores_per_executor, memory_per_executor)
/opt/anaconda3/envs/raydp/lib/python3.6/site-packages/raydp/context.py in init_spark(app_name, num_executors, executor_cores, executor_memory, configs)
120                 _global_spark_context = _SparkContext(
121                     app_name, num_executors, executor_cores, executor_memory, configs)
--> 122                 return _global_spark_context.get_or_create_session()
123             except:
124                 _global_spark_context = None
/opt/anaconda3/envs/raydp/lib/python3.6/site-packages/raydp/context.py in get_or_create_session(self)
66         if self._spark_session is not None:
67             return self._spark_session
---> 68         spark_cluster = self._get_or_create_spark_cluster()
69         self._spark_session = spark_cluster.get_spark_session(
70             self._app_name,
/opt/anaconda3/envs/raydp/lib/python3.6/site-packages/raydp/context.py in _get_or_create_spark_cluster(self)
60         if self._spark_cluster is not None:
61             return self._spark_cluster
---> 62         self._spark_cluster = SparkCluster()
63         return self._spark_cluster
64
/opt/anaconda3/envs/raydp/lib/python3.6/site-packages/raydp/spark/ray_cluster.py in init(self)
29         super().init(None)
30         self._app_master_bridge = None
---> 31         self._set_up_master(None, None)
32         self._spark_session: SparkSession = None
33
/opt/anaconda3/envs/raydp/lib/python3.6/site-packages/raydp/spark/ray_cluster.py in _set_up_master(self, resources, kwargs)
35         # TODO: specify the app master resource
36         self._app_master_bridge = RayClusterMaster()
---> 37         self._app_master_bridge.start_up()
38
39     def _set_up_worker(self, resources: Dict[str, float], kwargs: Dict[str, str]):
/opt/anaconda3/envs/raydp/lib/python3.6/site-packages/raydp/spark/ray_cluster_master.py in start_up(self, popen_kwargs)
50             return
51         extra_classpath = os.pathsep.join(self._prepare_jvm_classpath())
---> 52         self._gateway = self._launch_gateway(extra_classpath, popen_kwargs)
53         self._app_master_java_bridge = self._gateway.entry_point.getAppMasterBridge()
54         self._set_properties()
/opt/anaconda3/envs/raydp/lib/python3.6/site-packages/raydp/spark/ray_cluster_master.py in _launch_gateway(self, class_path, popen_kwargs)
113
114             if not os.path.isfile(conn_info_file):
--> 115                 raise Exception("Java gateway process exited before sending its port number")
116
117             with open(conn_info_file, "rb") as info:
Exception: Java gateway process exited before sending its port number

@yanivg10 could you check whether there are any other pyspark programming? And check the port 25333 whether is occupied (with lsof -i:25333).

This port is not occupied and there are no pyspark programs running

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging

This seems like you have an incompatible spark with raydp. raydp requires spark 3.0.0 or 3.0.1.

I am using Pyspark 3.0.1

pyspark.version
'3.0.1'

Could you provide more logs or messages? I tested locally with ubuntu and works fine.
In [1]: import raydp

In [2]: raydp.__version__
Out[2]: '0.1.1'

In [3]: import pyspark

In [4]: pyspark.__version__
Out[4]: '3.0.1'

In [5]: spark = raydp.init_spark("test", 1, 1, "2g")
2021-02-20 11:22:03,084 INFO services.py:1195 -- View the Ray dashboard at http://<IP_ADDRESS>:8265
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/lxy/miniconda3/envs/ray/lib/python3.7/site-packages/ray-2.0.0.dev0-py3.7-linux-x86_64.egg/ray/jars/ray_dist.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/lxy/miniconda3/envs/ray/lib/python3.7/site-packages/pyspark/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2021-02-20 11:22:06 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/02/20 11:22:08 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
(raylet) SLF4J: Class path contains multiple SLF4J bindings.
(raylet) SLF4J: Found binding in [jar:file:/home/lxy/miniconda3/envs/ray/lib/python3.7/site-packages/ray-2.0.0.dev0-py3.7-linux-x86_64.egg/ray/jars/ray_dist.jar!/org/slf4j/impl/StaticLoggerBinder.class]
(raylet) SLF4J: Found binding in [jar:file:/home/lxy/miniconda3/envs/ray/lib/python3.7/site-packages/pyspark/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
(raylet) SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
(raylet) SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

In [6]: spark.range(0, 10).count()
Out[6]: 10

In [7]:

I tried your example and here is what I get:

spark = raydp.init_spark("test", 1, 1, "2g")
2021-02-19 21:34:36,619	INFO services.py:1174 -- View the Ray dashboard at http://<IP_ADDRESS>:8266
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/guryaniv/.conda/envs/raydp2/lib/python3.7/site-packages/ray/jars/ray_dist.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/guryaniv/.conda/envs/raydp2/lib/python3.7/site-packages/pyspark/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2021-02-19 21:34:42 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Error: Missing application resource.

Usage: spark-submit [options] <app jar | python file | R file> [app arguments]
Usage: spark-submit --kill [submission ID] --master [spark://...]
Usage: spark-submit --status [submission ID] --master [spark://...]
Usage: spark-submit run-example [options] example-class [example args]
Options:
--master MASTER_URL         spark://host:port, mesos://host:port, yarn,
k8s://https://host:port, or local (Default: local[*]).
--deploy-mode DEPLOY_MODE   Whether to launch the driver program locally ("client") or
on one of the worker machines inside the cluster ("cluster")
(Default: client).
--class CLASS_NAME          Your application's main class (for Java / Scala apps).
--name NAME                 A name of your application.
--jars JARS                 Comma-separated list of jars to include on the driver
and executor classpaths.
--packages                  Comma-separated list of maven coordinates of jars to include
on the driver and executor classpaths. Will search the local
maven repo, then maven central and any additional remote
repositories given by --repositories. The format for the
coordinates should be groupId:artifactId:version.
--exclude-packages          Comma-separated list of groupId:artifactId, to exclude while
resolving the dependencies provided in --packages to avoid
dependency conflicts.
--repositories              Comma-separated list of additional remote repositories to
search for the maven coordinates given with --packages.
--py-files PY_FILES         Comma-separated list of .zip, .egg, or .py files to place
on the PYTHONPATH for Python apps.
--files FILES               Comma-separated list of files to be placed in the working
directory of each executor. File paths of these files
in executors can be accessed via SparkFiles.get(fileName).
--conf, -c PROP=VALUE       Arbitrary Spark configuration property.
--properties-file FILE      Path to a file from which to load extra properties. If not
specified, this will look for conf/spark-defaults.conf.
--driver-memory MEM         Memory for driver (e.g. 1000M, 2G) (Default: 1024M).
--driver-java-options       Extra Java options to pass to the driver.
--driver-library-path       Extra library path entries to pass to the driver.
--driver-class-path         Extra class path entries to pass to the driver. Note that
jars added with --jars are automatically included in the
classpath.
--executor-memory MEM       Memory per executor (e.g. 1000M, 2G) (Default: 1G).
--proxy-user NAME           User to impersonate when submitting the application.
This argument does not work with --principal / --keytab.
--help, -h                  Show this help message and exit.
--verbose, -v               Print additional debug output.
--version,                  Print the version of current Spark.
Cluster deploy mode only:
--driver-cores NUM          Number of cores used by the driver, only in cluster mode
(Default: 1).
Spark standalone or Mesos with cluster deploy mode only:
--supervise                 If given, restarts the driver on failure.
--kill SUBMISSION_ID        If given, kills the driver specified.
--status SUBMISSION_ID      If given, requests the status of the driver specified.
Spark standalone and Mesos only:
--total-executor-cores NUM  Total cores for all executors.
Spark standalone and YARN only:
--executor-cores NUM        Number of cores per executor. (Default: 1 in YARN mode,
or all available cores on the worker in standalone mode)
YARN-only:
--queue QUEUE_NAME          The YARN queue to submit to (Default: "default").
--num-executors NUM         Number of executors to launch (Default: 2).
If dynamic allocation is enabled, the initial number of
executors will be at least NUM.
--archives ARCHIVES         Comma separated list of archives to be extracted into the
working directory of each executor.
--principal PRINCIPAL       Principal to be used to login to KDC, while running on
secure HDFS.
--keytab KEYTAB             The full path to the file that contains the keytab for the
principal specified above. This keytab will be copied to
the node running the Application Master via the Secure
Distributed Cache, for renewing the login tickets and the
delegation tokens periodically.
Traceback (most recent call last):
File "", line 1, in
File "/home/guryaniv/.conda/envs/raydp2/lib/python3.7/site-packages/raydp/context.py", line 122, in init_spark
return _global_spark_context.get_or_create_session()
File "/home/guryaniv/.conda/envs/raydp2/lib/python3.7/site-packages/raydp/context.py", line 74, in get_or_create_session
self._configs)
File "/home/guryaniv/.conda/envs/raydp2/lib/python3.7/site-packages/raydp/spark/ray_cluster.py", line 69, in get_spark_session
spark_builder.appName(app_name).master(self.get_cluster_url()).getOrCreate()
File "/home/guryaniv/.conda/envs/raydp2/lib/python3.7/site-packages/pyspark/sql/session.py", line 186, in getOrCreate
sc = SparkContext.getOrCreate(sparkConf)
File "/home/guryaniv/.conda/envs/raydp2/lib/python3.7/site-packages/pyspark/context.py", line 378, in getOrCreate
SparkContext(conf=conf or SparkConf())
File "/home/guryaniv/.conda/envs/raydp2/lib/python3.7/site-packages/pyspark/context.py", line 133, in init
SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
File "/home/guryaniv/.conda/envs/raydp2/lib/python3.7/site-packages/pyspark/context.py", line 327, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "/home/guryaniv/.conda/envs/raydp2/lib/python3.7/site-packages/pyspark/java_gateway.py", line 105, in launch_gateway
raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number

Oh, I see. You can not submit the job with spark-submit script. You just need to run the python file directly.

I am running the python script directly, not using spark-submit. Could it be related to the ray version? Which ray version do you recommend? I'm using 1.2.0, but it looks like you are using 2.0.0.dev0

Did you set SPARK_HOME  environments?

@yanivg10 , it seems there is a problem to start the java process in your environment. Do you have JDK installed and have the environment variable JAVA_HOME set properly? Please try to run "java -version" and then run "pyspark" to see if pyspark itself can work properly.

Thanks. This was indeed related to the JAVA_HOME variable. It should not be set to any path for the code to run.
I am able to run your taxi fare prediction example, but Torch is throwing errors while the training session runs. It seems related to issue #74 and I added a comment there with the output log. Can you take a look at that?

Closes this. You can reopen if has further problems @yanivg10.

@carsonwang
I have a similar error at: spark = raydp.init_spark("test", 1, 1, "2g").
Do you have any step-by-step tutorial page or link to reproduce your codes? Do you have any yaml example to share as well to launch raydp at AWS?
(1) If I run it at ipython3, I have the following error:
Py4JJavaError: An error occurred while calling o0.setProperties.
: java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:460)
at java.util.Properties.setProperty(Properties.java:166)
at java.lang.System.setProperty(System.java:798)
at org.apache.spark.deploy.raydp.AppMasterJavaBridge.$anonfun$setProperties$1(AppM
(2) If I run "python test_ray_init.py", I have the following error:
Traceback (most recent call last):
File "test_ray_init.py", line 14, in
spark = raydp.init_spark("test", 1, 1, "2g")
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/raydp/context.py", line 126, in init_spark
return _global_spark_context.get_or_create_session()
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/raydp/context.py", line 70, in get_or_create_session
spark_cluster = self._get_or_create_spark_cluster()
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/raydp/context.py", line 63, in _get_or_create_spark_cluster
self._spark_cluster = SparkCluster(self._configs)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/raydp/spark/ray_cluster.py", line 34, in init
self._set_up_master(None, None)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/raydp/spark/ray_cluster.py", line 40, in _set_up_master
self._app_master_bridge.start_up()
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/raydp/spark/ray_cluster_master.py", line 56, in start_up
self._set_properties()
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/raydp/spark/ray_cluster_master.py", line 158, in _set_properties
self._app_master_java_bridge.setProperties(jvm_properties)
...

@yanivg10 , it seems there is a problem to start the java process in your environment. Do you have JDK installed and have the environment variable JAVA_HOME set properly? Please try to run "java -version" and then run "pyspark" to see if pyspark itself can work properly.

@dalgual , we don't have yaml or docker image published right now. Did you create your own image based on Ray's docker image and have RayDP and java installed properly? There is an example RayDP docker file for reference https://github.com/oap-project/raydp/tree/master/docker

@carsonwang
I launch EC2 instances as follows. Do I miss anything in the following?:
ray up -y config.raydp.yaml
------ config.raydp.yaml ----------------
cluster_name: ray-dp-bigdai
max_workers: 2
upscaling_speed: 1.0
provider:
type: aws
region: us-west-2
availability_zone: us-west-2a,us-west-2b
Whether to allow node reuse. If set to False, nodes will be terminated
instead of stopped.
cache_stopped_nodes: True # If not present, the default is True.
autoscaling_mode: default
auth:
ssh_user: ubuntu
available_node_types:
ray.head.default:
resources: {}
node_config:
InstanceType: m5.xlarge
ImageId: ami-0a2363a9cff180a64 # Deep Learning AMI (Ubuntu) Version 30
# You can provision additional disk space with a conf as follows
BlockDeviceMappings:
- DeviceName: /dev/sda1
Ebs:
VolumeSize: 100
#g4dn.2xlarge, 1 GPU, 8 vCPUs, 32 GiB of memory, 225 NVMe SSD, up to 25 Gbps network performance
ray.worker.default:
min_workers: 2
max_workers: 2
resources: {}
node_config:
InstanceType: g4dn.2xlarge
ImageId: ami-0050625d58fa27b6d #ami-0a2363a9cff180a64 # Deep Learning AMI (Ubuntu) Version 30
head_node_type: ray.head.default
setup_commands:
- pip install "raydp"

Unfortunately, we don't have such yaml now.
Are you using raydp 0.4.1 and ray 1.11.0? Also can you provide complete error message when running python script? Is it the same as ipython3 case?

@kira-lin
yes, here it is:
ray, version 1.11.0
ubuntu@ip-172-30-0-170:~/examples$ cat test_ray_init.py
import raydp
import ray
raydp.version
import pyspark
pyspark.version
ray.init(address='auto', _redis_password='5241590000000000')
spark = raydp.init_spark("test", 1, 1, "2g")
ubuntu@ip-172-30-0-170:~/examples$ python test_ray_init.py
Traceback (most recent call last):
File "test_ray_init.py", line 14, in
spark = raydp.init_spark("test", 1, 1, "2g")
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/raydp/context.py", line 126, in init_spark
return _global_spark_context.get_or_create_session()
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/raydp/context.py", line 70, in get_or_create_session
spark_cluster = self._get_or_create_spark_cluster()
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/raydp/context.py", line 63, in _get_or_create_spark_cluster
self._spark_cluster = SparkCluster(self._configs)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/raydp/spark/ray_cluster.py", line 34, in init
self._set_up_master(None, None)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/raydp/spark/ray_cluster.py", line 40, in _set_up_master
self._app_master_bridge.start_up()
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/raydp/spark/ray_cluster_master.py", line 56, in start_up
self._set_properties()
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/raydp/spark/ray_cluster_master.py", line 158, in _set_properties
self._app_master_java_bridge.setProperties(jvm_properties)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py", line 1322, in call
answer, self.gateway_client, self.target_id, self.name)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/py4j/protocol.py", line 328, in get_return_value
format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o0.setProperties.
: java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:460)
at java.util.Properties.setProperty(Properties.java:166)
at java.lang.System.setProperty(System.java:798)
at org.apache.spark.deploy.raydp.AppMasterJavaBridge.$anonfun$setProperties$1(AppMasterJavaBridge.scala:32)
at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:400)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:728)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:728)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:728)
at org.apache.spark.deploy.raydp.AppMasterJavaBridge.setProperties(AppMasterJavaBridge.scala:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)

I've verified that indeed raydp-0.4.1 does work with ray 1.11.0. Can you please try to use previous versions of ray? Or you can use raydp-nightly, too.

@kira-lin Thanks it works ;-)

@kira-lin @carsonwang  I run XGBoost using raydp by reading a dataset at s3. Do you have any example code to read S3 using ray? I got the following error
Traceback (most recent call last):
File "xgboost_ray_nyctaxi.py", line 25, in
.load(NYC_TRAIN_CSV)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pyspark/sql/readwriter.py", line 158, in load
return self._df(self._jreader.load(path))
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py", line 1322, in call
answer, self.gateway_client, self.target_id, self.name)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pyspark/sql/utils.py", line 111, in deco
return f(*a, **kw)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/py4j/protocol.py", line 328, in get_return_value
format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o61.load.
: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3443)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$checkAndGlobPathIfNecessary$1(DataSource.scala:747)
at scala.collection.immutable.List.map(List.scala:293)
at org.apache.spark.sql.execution.datasources.DataSource$.checkAndGlobPathIfNecessary(DataSource.scala:745)
at org.apache.spark.sql.execution.datasources.DataSource.checkAndGlobPathIfNecessary(DataSource.scala:577)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:408)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:274)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:245)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:245)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:188)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.lang.Thread.run(Thread.java:748)
(RayDPSparkMaster pid=6477) 2022-03-25 01:52:02,284 INFO RayAppMaster [Thread-2]: Stopping RayAppMaster

Are you able to read it using pyspark? Seems like reading/writing s3 requires some config, and some extra package, like hadoop-aws. Maybe you can refer to this

Does ray and raydp access data at the same node/server?
Is there any way to read files from other data source such as s3, object storage of Azure & GCP?

raydp is spark on ray. Anything you can do on spark should also be possible with raydp. Reading s3 should be possible with the library I provided above. This is another compact tutorial, hope it helps.

@kira-lin Thanks. I found out that the following works:
data = ray.data.read_csv("s3://my_bucket/test.csv")

@kira-lin
I cannot open ray dashboard when I launch a cluster with raydp and xgboost_ray together using "pip install xgboost_ray raydp-nightly" - I can with raydp only.
Do you have any idea how I can open ray dashboard or make it run?

That's strange. Do you mean ray dashboard or spark dashboard? Ray dashboard should be available without raydp. It should start once you create the cluster

@kira-lin I can run PySpark xgboost-ray code now with CPU. However, even though I have the following code with "gpu_hist", I dont have any clue if the code runs using GPU - the code actually works well with CPU but not with GPU. Dashboard only shows the resouces of CPU. How I can tell if the code runs in GPU? Do you have any example code link with GPU?
I also list a result of ray status below:
#g4dn.2xlarge, 1 GPU, 8 vCPUs, 32 GiB of memory, 225 NVMe SSD, up to 25 Gbps network performance
app_name = "Spark with RayDP"
num_executors = 1 #1
cores_per_executor = 1 #4, 2, 1
memory_per_executor = "2GB" # "2GB" "1GB" "500M"
spark = raydp.init_spark(app_name, num_executors, cores_per_executor, memory_per_executor)
...
"tree_method": "gpu_hist",
...
ray_params=RayParams(
num_actors=3,
gpus_per_actor=1))
ubuntu@ip-172-30-0-247:~/tutorials$ ray status
======== Autoscaler status: 2022-03-28 06:19:15.800303 ========
Node status
Healthy:
1 ray.head.default
Pending:
<IP_ADDRESS>: ray.worker.default, uninitialized
<IP_ADDRESS>: ray.worker.default, uninitialized
<IP_ADDRESS>: ray.worker.default, uninitialized
Recent failures:
(no failures)
Resources
Usage:
1.0/4.0 CPU
2.00/8.886 GiB memory
0.00/4.443 GiB object_store_memory
Demands:
{'CPU': 1.0, 'GPU': 1.0} * 3 (SPREAD): 1+ pending placement groups

@dalgual
It seems like your cluster is not ready. Only the head node is up, others are pending. Is it the status when you run the code? If so, the resource you require (3xGPU) cannot be met. I guess you should specify gpu in resource when you config the cluster. We don't have much experience on using gpu in ray, maybe you should ask people in the xgboost_ray repo.

@kira-lin
I have a similar issue. Would you please check it?
Environment:
$java --version
openjdk 11.0.19 2023-04-18
OpenJDK Runtime Environment (build 11.0.19+7-post-Ubuntu-0ubuntu118.04.1)
OpenJDK 64-Bit Server VM (build 11.0.19+7-post-Ubuntu-0ubuntu118.04.1, mixed mode, sharing)

$echo $JAVA_HOME
/usr/lib/jvm/java-11-openjdk-amd64

$pip freeze | grep ray
ray==2.4.0
raydp==0.6.0

$pip freeze | grep spark
pyspark==3.1.3

Error messages:
when  run raydp.init_spark(app_name='RayDP Example',
num_executors=4,
executor_cores=2,
executor_memory='7GB')
File "python/ray/_raylet.pyx", line 870, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 921, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 877, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 881, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 821, in ray._raylet.execute_task.function_executor
  File "/usr/local/lib/python3.6/dist-packages/ray/_private/function_manager.py", line 670, in actor_method_executor
    return method(__ray_actor, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/ray/util/tracing/tracing_helper.py", line 460, in _resume_span
    return method(self, *_args, **_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/raydp/spark/ray_cluster_master.py", line 55, in start_up
    self._set_properties()
  File "/usr/local/lib/python3.6/dist-packages/ray/util/tracing/tracing_helper.py", line 460, in _resume_span
    return method(self, *_args, **_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/raydp/spark/ray_cluster_master.py", line 170, in _set_properties
    self._app_master_java_bridge.setProperties(jvm_properties)
  File "/usr/local/lib/python3.6/dist-packages/py4j/java_gateway.py", line 1305, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/usr/local/lib/python3.6/dist-packages/py4j/protocol.py", line 328, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o0.setProperties.
: java.lang.NullPointerException
	at java.base/java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
	at java.base/java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
	at java.base/java.util.Properties.put(Properties.java:1340)
	at java.base/java.util.Properties.setProperty(Properties.java:228)
	at java.base/java.lang.System.setProperty(System.java:898)
	at org.apache.spark.deploy.raydp.AppMasterJavaBridge.$anonfun$setProperties$1(AppMasterJavaBridge.scala:32)
	at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:234)
	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:468)
	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:468)
	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:468)
	at org.apache.spark.deploy.raydp.AppMasterJavaBridge.setProperties(AppMasterJavaBridge.scala:31)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.base/java.lang.Thread.run(Thread.java:829)

@kira-lin
I got a similar issue, I would appreciate it if you could check:
(I downgraded raydp and pyspark, should I also downgrade ray?)
Environment
$java --version
openjdk 11.0.19 2023-04-18
OpenJDK Runtime Environment (build 11.0.19+7-post-Ubuntu-0ubuntu118.04.1)
OpenJDK 64-Bit Server VM (build 11.0.19+7-post-Ubuntu-0ubuntu118.04.1, mixed mode, sharing)

$echo $JAVA_HOME
/usr/lib/jvm/java-11-openjdk-amd64

$pip freeze | grep ray
ray==2.4.0
raydp==0.6.0

$pip freeze | grep spark
pyspark==3.1.3

Error messages
when  run raydp.init_spark(app_name='RayDP Example',
num_executors=4,
executor_cores=2,
executor_memory='7GB')
File "python/ray/_raylet.pyx", line 870, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 921, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 877, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 881, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 821, in ray._raylet.execute_task.function_executor
  File "/usr/local/lib/python3.6/dist-packages/ray/_private/function_manager.py", line 670, in actor_method_executor
    return method(__ray_actor, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/ray/util/tracing/tracing_helper.py", line 460, in _resume_span
    return method(self, *_args, **_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/raydp/spark/ray_cluster_master.py", line 55, in start_up
    self._set_properties()
  File "/usr/local/lib/python3.6/dist-packages/ray/util/tracing/tracing_helper.py", line 460, in _resume_span
    return method(self, *_args, **_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/raydp/spark/ray_cluster_master.py", line 170, in _set_properties
    self._app_master_java_bridge.setProperties(jvm_properties)
  File "/usr/local/lib/python3.6/dist-packages/py4j/java_gateway.py", line 1305, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/usr/local/lib/python3.6/dist-packages/py4j/protocol.py", line 328, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o0.setProperties.
: java.lang.NullPointerException
	at java.base/java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
	at java.base/java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
	at java.base/java.util.Properties.put(Properties.java:1340)
	at java.base/java.util.Properties.setProperty(Properties.java:228)
	at java.base/java.lang.System.setProperty(System.java:898)
	at org.apache.spark.deploy.raydp.AppMasterJavaBridge.$anonfun$setProperties$1(AppMasterJavaBridge.scala:32)
	at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:234)
	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:468)
	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:468)
	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:468)
	at org.apache.spark.deploy.raydp.AppMasterJavaBridge.setProperties(AppMasterJavaBridge.scala:31)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.base/java.lang.Thread.run(Thread.java:829)

@yunju63 You should use raydp-nightly to work with Ray 2.4.0. Spark 3.1.3 is fine. Run pip install --pre -U raydp and try again

@kira-lin It works, thank you :)