Latest Update - Dies

Please follow the guide below

You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your issue (like that [x])
Use Preview tab to see how your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.01.29. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

[ X] I've verified and I assure that I'm running youtube-dl 2017.01.29

Before submitting an issue make sure you have:

[ X] At least skimmed through README and most notably FAQ and BUGS sections
[ X] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

[X ] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:
Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):
youtube-dl -f 'bestvideo+bestaudio/best' -o '%(title)s.f%(format_id)s.%(ext)s' --verbose https://www.youtube.com/watch?v=abZ0PXadpB0
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-f', u'bestvideo+bestaudio/best', u'-o', u'%(title)s.f%(format_id)s.%(ext)s', u'--verbose', u'https://www.youtube.com/watch?v=abZ0PXadpB0']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.01.29
[debug] Python version 2.7.12 - Linux-4.4.0-59-generic-x86_64-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 10.7, avprobe 3.2.2-1, ffmpeg 3.2.2-1, ffprobe 3.2.2-1, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] abZ0PXadpB0: Downloading webpage
[youtube] abZ0PXadpB0: Downloading embed webpage
[youtube] abZ0PXadpB0: Refetching age-gated info webpage
[youtube] abZ0PXadpB0: Extracting video information
[youtube] {22} signature length 42.43, html5 player en_US-vflV3n15C
[youtube] abZ0PXadpB0: Downloading player /yts/jsbin/player-en_US-vflV3n15C/base.js
ERROR: Signature extraction failed: Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 1057, in _decrypt_signature
    video_id, player_url, s
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 971, in _extract_signature_function
    errnote='Download of %s failed' % player_url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 529, in _download_webpage
    res = self._download_webpage_handle(url_or_request, video_id, note, errnote, fatal, encoding=encoding, data=data, headers=headers, query=query)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 436, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 416, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2001, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 421, in open
    protocol = req.get_type()
  File "/usr/lib/python2.7/urllib2.py", line 283, in get_type
    raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type: /yts/jsbin/player-en_US-vflV3n15C/base.js
 (caused by ValueError(u'unknown url type: /yts/jsbin/player-en_US-vflV3n15C/base.js',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 1057, in _decrypt_signature
    video_id, player_url, s
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 971, in _extract_signature_function
    errnote='Download of %s failed' % player_url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 529, in _download_webpage
    res = self._download_webpage_handle(url_or_request, video_id, note, errnote, fatal, encoding=encoding, data=data, headers=headers, query=query)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 436, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 416, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2001, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 421, in open
    protocol = req.get_type()
  File "/usr/lib/python2.7/urllib2.py", line 283, in get_type
    raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type: /yts/jsbin/player-en_US-vflV3n15C/base.js
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 694, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 369, in extract
    return self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 1646, in _real_extract
    encrypted_sig, video_id, player_url, age_gate)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 1067, in _decrypt_signature
    'Signature extraction failed: ' + tb, cause=e)
ExtractorError: Signature extraction failed: Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 1057, in _decrypt_signature
    video_id, player_url, s
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 971, in _extract_signature_function
    errnote='Download of %s failed' % player_url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 529, in _download_webpage
    res = self._download_webpage_handle(url_or_request, video_id, note, errnote, fatal, encoding=encoding, data=data, headers=headers, query=query)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 436, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 416, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2001, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 421, in open
    protocol = req.get_type()
  File "/usr/lib/python2.7/urllib2.py", line 283, in get_type
    raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type: /yts/jsbin/player-en_US-vflV3n15C/base.js
 (caused by ValueError(u'unknown url type: /yts/jsbin/player-en_US-vflV3n15C/base.js',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

youtube-dl --version
2017.01.29

$ youtube-dl -v <your command line>
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2017.01.29
[debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
[debug] Proxy map: {}
...
<end of log>

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

Single video: https://www.youtube.com/watch?v=BaW_jenozKc
Single video: https://youtu.be/BaW_jenozKc
Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc

Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.

Description of your issue, suggested solution and other information
Explanation of your issue in arbitrary form goes here. Please make sure the description is worded well enough to be understood. Provide as much context and examples as possible.
If work on your issue requires account credentials please provide them or explain how one can obtain them.
This is the command line I used which worked before the 2017-01-29 update:
youtube-dl -f 'bestvideo+bestaudio/best' -o '%(title)s.f%(format_id)s.%(ext)s' --verbose https://www.youtube.com/watch?v=abZ0PXadpB0
Thanks and I hope a solution will be easy.

Fix is here:
https://github.com/rg3/youtube-dl/pull/11892


updated PR npo_watchlist: Rewrite to support new npo.nl layout

updated PR of #1901 with some minor fixes/changes and updated tests
Closes #1892 and closes #1893 and closes #1901.

@jeroenl

does this replace #1901 then?

yes, this supersedes #1901

you guys should have used the same branch then :)
closing #1901

I can't commit to his personal branch ;-)

A few other requests that aren't wrapped in try except. Fix those and I think we're good

I now updated the branch with the try-except statements for the requests, can this please be merged so that the plugin works again? Thanks!

As I wrote in Gitter, I am stuck why the Travis build keeps failing (for me I get even the same error on flexget/develop branch), I had hoped that the exception stuff should now have been fixed. If one of the devs could take a look, that'd be a great help because I will be afk for awhile.

@mfonville as i wrote to you on gitter, build crashes were unrelated to your code. merge from dev and it should be fine


M4942 syncs failing without data

Fixes https://github.com/CartoDB/cartodb-management/issues/4942
This is a tricky one, so some context. The problem was that a table was being deleted just after being synced. It imported fine the first time, but after the first sync period, it was deleted from the dashboard (it remained in the database). This only happened with a big table (4M rows) without geometries.
The problem here, was that the ruby migrator, detected the geometry was of a wrong SRID (NULL != 4326) and tried to reproject the data. This would result in a timeout, and as a recovery mechanism, it would rename the_geom to invalid_the_geom. Then, when visiting the dashboard, ghost tables would delete the table, since it does not have a the_geom column.
To understand why this isn't a problem at import time, let's look at the order of this setup methods: the import process calls the migrator before cartodbfycation, and the synchronizer calls the cartodbfycation first and then the migrator.
I'm truly impressed by the cartodbfycation code at the extension, it is really nice. And in this case, maybe too nice, as it was hiding an error (not the fault of the cartodbfycation though). During the import, the CartoDB::Migrator would run and rename the geometry column as discussed. But later, cdb_cartodbfytable runs and detects this is incorrect, so it proceeds to reverse the change, leaving the table in a consistent state.
My solution to the problem is to completely remove this reprojection code, as it's already present in the cartodbfy code at the extension and it's done better there. This also improves import performance for datasets without geometry.
As a fun bonus, I removed a couple of clearly unused lines from the CartoDB::Migrator. There is a lot more that can be removed but for little gain, but I removed a COUNT(*) that can be quite slow for large datasets, as a nice performance bonus.
Acceptance

[ ] Import a large file (> 1M rows) without geometry as a sync
[ ] Force a sync of the dataset
[ ] Refresh the dashboard after the sync is done. Ghost tables should not run, the table should not be deleted from dashboard.
[ ] Import and sync a dataset with geometry in a different projection than srid 4326, it should be reprojected
[ ] Duplicate a dataset with and w/o geometry

🤔 let me take a second look at the tests. I forgot that of course, the extension is available in cartodb, so the tests should be able to pass with minor modifications (just ensuring that the extension is called).

Fixing that test so it runs was easy, but it opened the 🍈 of fun inconsistencies. This PR breaks import of datasets with multi-points geometries with no SRID. But it was already broken for syncs, and tables created via ghost tables (because cartodbfy runs first there, and that creates indexes on the geometry columns that break the multipoint->point code).
I'll move this back to dev and see if there is an easy solution for that without adding back the reprojection code, or if it even matters, as it was broken in half the cases anyway.

@rafatower 👁  on this, please :-)

:+1:
Just one minor and quick thing: I'd add a typical torque file sync + torque map to acceptance just for completeness. There was once upon a time a subtle bug that caused imports/syncs to apparently work but left the_geom with multipoints and torque wasn't able to render those.

👍 Thanks! I added it as an acceptance step, so we make sure we check this. 🙇

Master merged, tomorrow morning deploy.


[cli] Incomplete due to Failures -- Success: 486, Failure: 53

I have re-run this install once already, updated all my packages, updated pip as it seemed to be a problem but no luck. Any assistance would be helpful.
output_sift_update.txt
saltstack_errors.txt

The first log indicates updates aren’t allowed if an install hasn’t been successful.
The second log indicates there is something wrong with your python install. As I’m unable to reproduce this locally it’s hard for me to troubleshoot.
Can you seen if python is installed? Can you run python -v
Sent from my iPhone

On Aug 28, 2018, at 21:40, jacobiusg<EMAIL_ADDRESS>wrote:
I have re-run this install once already, updated all my packages, updated pip as it seemed to be a problem but no luck. Any assistance would be helpful.
output_sift_update.txt
saltstack_errors.txt
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

$ python --version
Python 2.7.12
silence@siftworkstation -> ~
$ python -v
installing zipimport hook
import zipimport # builtin
installed zipimport hook
/usr/lib/python2.7/site.pyc matches /usr/lib/python2.7/site.py
import site # precompiled from /usr/lib/python2.7/site.pyc
/usr/lib/python2.7/os.pyc matches /usr/lib/python2.7/os.py
import os # precompiled from /usr/lib/python2.7/os.pyc
import errno # builtin
import posix # builtin
/usr/lib/python2.7/posixpath.pyc matches /usr/lib/python2.7/posixpath.py
import posixpath # precompiled from /usr/lib/python2.7/posixpath.pyc
/usr/lib/python2.7/stat.pyc matches /usr/lib/python2.7/stat.py
import stat # precompiled from /usr/lib/python2.7/stat.pyc
/usr/lib/python2.7/genericpath.pyc matches /usr/lib/python2.7/genericpath.py
import genericpath # precompiled from /usr/lib/python2.7/genericpath.pyc
/usr/lib/python2.7/warnings.pyc matches /usr/lib/python2.7/warnings.py
import warnings # precompiled from /usr/lib/python2.7/warnings.pyc
/usr/lib/python2.7/linecache.pyc matches /usr/lib/python2.7/linecache.py
import linecache # precompiled from /usr/lib/python2.7/linecache.pyc
/usr/lib/python2.7/types.pyc matches /usr/lib/python2.7/types.py
import types # precompiled from /usr/lib/python2.7/types.pyc
/usr/lib/python2.7/UserDict.pyc matches /usr/lib/python2.7/UserDict.py
import UserDict # precompiled from /usr/lib/python2.7/UserDict.pyc
/usr/lib/python2.7/_abcoll.pyc matches /usr/lib/python2.7/_abcoll.py
import _abcoll # precompiled from /usr/lib/python2.7/_abcoll.pyc
/usr/lib/python2.7/abc.pyc matches /usr/lib/python2.7/abc.py
import abc # precompiled from /usr/lib/python2.7/abc.pyc
/usr/lib/python2.7/_weakrefset.pyc matches /usr/lib/python2.7/_weakrefset.py
import _weakrefset # precompiled from /usr/lib/python2.7/_weakrefset.pyc
import _weakref # builtin
/usr/lib/python2.7/copy_reg.pyc matches /usr/lib/python2.7/copy_reg.py
import copy_reg # precompiled from /usr/lib/python2.7/copy_reg.pyc
/usr/lib/python2.7/traceback.pyc matches /usr/lib/python2.7/traceback.py
import traceback # precompiled from /usr/lib/python2.7/traceback.pyc
/usr/lib/python2.7/sysconfig.pyc matches /usr/lib/python2.7/sysconfig.py
import sysconfig # precompiled from /usr/lib/python2.7/sysconfig.pyc
/usr/lib/python2.7/re.pyc matches /usr/lib/python2.7/re.py
import re # precompiled from /usr/lib/python2.7/re.pyc
/usr/lib/python2.7/sre_compile.pyc matches /usr/lib/python2.7/sre_compile.py
import sre_compile # precompiled from /usr/lib/python2.7/sre_compile.pyc
import _sre # builtin
/usr/lib/python2.7/sre_parse.pyc matches /usr/lib/python2.7/sre_parse.py
import sre_parse # precompiled from /usr/lib/python2.7/sre_parse.pyc
/usr/lib/python2.7/sre_constants.pyc matches /usr/lib/python2.7/sre_constants.py
import sre_constants # precompiled from /usr/lib/python2.7/sre_constants.pyc
import _locale # builtin
/usr/lib/python2.7/_sysconfigdata.pyc matches /usr/lib/python2.7/_sysconfigdata.py
import _sysconfigdata # precompiled from /usr/lib/python2.7/_sysconfigdata.pyc
/usr/lib/python2.7/plat-x86_64-linux-gnu/_sysconfigdata_nd.pyc matches /usr/lib/python2.7/plat-x86_64-linux-gnu/_sysconfigdata_nd.py
import _sysconfigdata_nd # precompiled from /usr/lib/python2.7/plat-x86_64-linux-gnu/_sysconfigdata_nd.pyc
/usr/lib/python2.7/sitecustomize.pyc matches /usr/lib/python2.7/sitecustomize.py
import sitecustomize # precompiled from /usr/lib/python2.7/sitecustomize.pyc
import encodings # directory /usr/lib/python2.7/encodings
/usr/lib/python2.7/encodings/init.pyc matches /usr/lib/python2.7/encodings/init.py
import encodings # precompiled from /usr/lib/python2.7/encodings/init.pyc
/usr/lib/python2.7/codecs.pyc matches /usr/lib/python2.7/codecs.py
import codecs # precompiled from /usr/lib/python2.7/codecs.pyc
import _codecs # builtin
/usr/lib/python2.7/encodings/aliases.pyc matches /usr/lib/python2.7/encodings/aliases.py
import encodings.aliases # precompiled from /usr/lib/python2.7/encodings/aliases.pyc
/usr/lib/python2.7/encodings/utf_8.pyc matches /usr/lib/python2.7/encodings/utf_8.py
import encodings.utf_8 # precompiled from /usr/lib/python2.7/encodings/utf_8.pyc
Python 2.7.12 (default, Dec  4 2017, 14:50:18)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
dlopen("/usr/lib/python2.7/lib-dynload/readline.x86_64-linux-gnu.so", 2);
import readline # dynamically loaded from /usr/lib/python2.7/lib-dynload/readline.x86_64-linux-gnu.so

python_output.txt

What version of pip is installed
Sent from my iPhone

On Aug 28, 2018, at 21:59, jacobiusg<EMAIL_ADDRESS>wrote:
python_output.txt
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

v 9.01

salt-call —versions-report
Sent from my iPhone

On Aug 29, 2018, at 06:59, jacobiusg<EMAIL_ADDRESS>wrote:
v 9.01
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

saltcall.txt

Try running, you might need to alter the file-root, I'm going from memory at the moment, but I think that path is right.
salt-call --local --file-root=/var/cache/sift/cli/v2018.28.0 state.sls sift.python-packages.colorama

path was right i double checked
state.txt

Try ..
salt-call --local --file-root=/var/cache/sift/cli/v2018.28.0/sift-saltstack-2018.28.0 state.sls sift.python-packages.colorama

If that doesn't work look for the subfolder that has the sift folder in it.

Still seems to be a problem with pip.. I may just attempt to re-install python and pip see if that works
salt_state.txt

Add -l debug to the command.

salt_debug.txt

Try installing pip 9.0.3 if that doesn't work let me know. It looks like this might have been a bug in salt-stack, but according to everything it shouldn't behaving this way, but we can try upgrading salt-stack next (without using the sift-cli) to see if we can't get it working.

unfortunately it's still not working.

You can try and update to 2017.7.6, apt-get install sift-minion=2017.7.6 and then retry re-running the command.

same issue here, @ekristen was the above supposed to be apt-get install **salt**-minion=2017.7.6  ?
if so, isn't salt-minion=2017.7.5+ds-1 the newest version?

I didn’t check the exact version syntax but 2017.7.6 was released end of May. Actually 2017.7.7 is our try installing that.
Sent from my iPhone

On Aug 30, 2018, at 09:18, Davide Paltrinieri<EMAIL_ADDRESS>wrote:
same issue here, @ekristen was the above supposed to be apt-get install salt-minion=2017.7.6 ?
if so, isn't salt-minion=2017.7.5+ds-1 the newest version?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Having the same issues.
Installing sift on a fresh VM through virtual box running Ubuntu Desktop 16.04.5. Installing it on a VM shouldn't effect anything?
I have been randomly selecting older versions and trying to install them.
When I installed v2018.26.0

Completed with Failures -- Success: 534, Failure: 5

Reinstalling v2018.28.0, but it does not list what failed below.

Completed with Failures -- Success: 534, Failure: 5

I had been previously getting Success: 486, Failure: 53 before trying older versions.

Logs would help to determine what’s broken.
Sent from my iPhone

On Aug 30, 2018, at 14:44, Shaun<EMAIL_ADDRESS>wrote:
Having the same issues.
Installing sift on a fresh VM through virtual box running Ubuntu Desktop 16.04.5. Installing it on a VM shouldn't effect anything?
I have been randomly selecting older versions and trying to install them.
When I installed v2018.26.0
Completed with Failures -- Success: 534, Failure: 5
Reinstalling v2018.28.0, but it does not list what failed below.
Completed with Failures -- Success: 534, Failure: 5
I had been previously getting Success: 486, Failure: 53 before trying older versions.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

my issue with salt-minion version was an hardcoded pointer to 2017.7.5 in source.list,
I changed that to latest, ran again sift upgrade, this time it ended with only 3 failed, (mainly only sift-python-package-machinae) "could not find a pip binary"
checking the /etc/apt/sources.list.d/saltstack.list now it is back to "deb http://repo.saltstack.com/apt/ubuntu/16.04/amd64/archive/2017.7.5 xenial main" so I guess that hardcoded version comes from the sift-cli script

@Folii you are 99% installed, I need to push an update to account for that poster as it's contents changed. SIFT doens't think it's installed because it wants 100% of everything to work, but you are there. Just so you know the CLI helper doesn't roll back anything in the face of failure so all the tools and such are installed.
@davidep welcome to the wonderful world of python breaking. As you can see @Folii managed to not get any python errors, but you are. The CLI is hardcoded but I will be adding an option to allow people to override it. If you got edit the source list and change the 2017.7.5 to 2017.7.7 and upgrade the salt-minion using apt, then run salt manually like
salt-call --local --file-root=/var/cache/sift/cli/v2018.28.0/sift-saltstack-2018.28.0 state.sls sift.python-packages.machinae

We can see if that fixes the issue.

I had similar issues with updating/upgrading on a fresh VM for FOR508 which an updated copy was provided on 7/16/18. I have the VM snapshotted so I can easily roll back.

<EMAIL_ADDRESS>sift-version: v2017.27.0
updating the current version (2017.27.0) presents errors as does upgrading to v2018.28.0
Attached is a log for

salt-call --local --file-root=/var/cache/sift/cli/v2018.28.0/sift-saltstack-2018.28.0 state.sls sift.packages
salt-call.log
when i issue and apt-get -f install a number of issues are resolved. After running the same salt-call command again the number of issues drops down to 5 with them revolving around python-dfvfs, python-plaso, and python3-pip

The following packages have unmet dependencies:
python3-pip : Depends: python-pip-whl (= 8.1.1-2) but 8.1.1-2ubuntu0.4 is to be installed
Recommends: python3-dev (>= 3.2) but it is not going to be installed
Recommends: python3-setuptools but it is not going to be installed
Recommends: python3-wheel but it is not going to be installed

[ERROR   ] Command '['systemd-run', '--scope', 'apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'python-dfvfs=latest']' failed with return code: 100
[ERROR   ] stdout: Reading package lists...
Building dependency tree...
Reading state information...
[ERROR   ] stderr: Running scope as unit run-r995c5fcb31a54a8e96302b552e948518.scope.
E: Version 'latest' for 'python-dfvfs' was not found
[ERROR   ] retcode: 100
[ERROR   ] Problem encountered installing package(s). Additional info follows:
errors:
- Running scope as unit run-r995c5fcb31a54a8e96302b552e948518.scope.
E: Version 'latest' for 'python-dfvfs' was not found

No apt-get update or upgrades have been run on this vm.

When I install with

--mode=packages-only

sudo sift install --mode=packages-only --user=${currentUser}
I am not getting any errors and the tools I need work just fine.


Add Soundcore icon

Description
https://play.google.com/store/apps/details?id=com.oceanwing.soundcore
Type of change

:x: Bug fix (non-breaking change which fixes an issue)
:white_check_mark: Icon addition (non-breaking change that adds/modifies Lawnicons's icons)
:x: General change (non-breaking change that doesn't fit the above categories like copyediting)

Icons addition information

Icons added

soundcore (com.oceanwing.soundcore)

Thanks!


Add tests for mesheryctl root config and flags

Signed-off-by: Nishant Mittal<EMAIL_ADDRESS>Description
This PR fixes n/a. These tests detect errors in config loading preventing bugs like: https://github.com/meshery/meshery/issues/5905
Notes for Reviewers
Signed commits

[X] Yes, I signed my commits.

Codecov Report

Merging #5930 (f67e2ed) into master (b0c509c) will decrease coverage by 4.38%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #5930      +/-   ##
==========================================
- Coverage   17.80%   13.41%   -4.39%
==========================================
  Files         107      110       +3
  Lines        9994    10161     +167
==========================================
- Hits         1779     1363     -416
- Misses       7779     8506     +727
+ Partials      436      292     -144

Flag
Coverage Δ

gointegrationtests
?

unittests
13.41% <100.00%> (-0.05%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

mesheryctl/internal/cli/root/root.go
23.43% <100.00%> (ø)

mesheryctl/internal/cli/root/system/update.go
1.05% <0.00%> (-58.95%)
:arrow_down:

mesheryctl/internal/cli/root/system/start.go
1.88% <0.00%> (-55.19%)
:arrow_down:

mesheryctl/internal/cli/root/system/status.go
1.14% <0.00%> (-55.18%)
:arrow_down:

mesheryctl/internal/cli/root/system/restart.go
1.92% <0.00%> (-48.08%)
:arrow_down:

mesheryctl/internal/cli/root/system/stop.go
2.00% <0.00%> (-46.67%)
:arrow_down:

mesheryctl/internal/cli/root/system/logs.go
3.26% <0.00%> (-36.96%)
:arrow_down:

mesheryctl/internal/cli/root/system/check.go
1.03% <0.00%> (-23.03%)
:arrow_down:

mesheryctl/internal/cli/root/system/dashboard.go
1.70% <0.00%> (-9.41%)
:arrow_down:

mesheryctl/pkg/utils/healthcheck.go
8.82% <0.00%> (-4.42%)
:arrow_down:

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b658c39...f67e2ed. Read the comment docs.

@alphaX86 updated the code to remove all files it generates while testing


iroh add command

iroh-ctl add [--recursive | -r]  [-q | --quiet] [--progress=true] [-w | --wrap-with-directory=true] [-n | --only-hash] [-H | --hidden] <path>
Upload to IPFS a file or directory given by path.

-r or --recursive is required if path is a directory. If you omit it, the command fails with an error. This is to prevent you from uploading a directory by accident.
--wrap=false can be supplied to prevent Iroh from wrapping what’s uploaded in a directory. This means that the filename information of the root path is lost.
--progress=false suppresses the progress bar. It should default to that in non-interactive scenarios too.
-q or --quiet makes all output go away, overriding any other options that control output.
-n, --only-hash: only chunk and hash, do not write to disk.
-H, --hidden: Include files that are hidden. Only takes effect on recursive add.

Differences with Kubo:

ipfs add does not wrap what’s uploaded into a directory by default. This means that the filename of the root path or directory is not known to the network.
ipfs add has a whole host of options to control the hashing algorithm and chunking.

Test cases

[x] Add a single file, success
[ ] Add a directory, success
[ ] Add a directory with subdirectories, success
[ ] Add directory without -r, failure
[ ] Adding a file or directory that does not exist, failure
[ ] Turn off wrapping
[ ] Only hash -- not sure how to test difference with other scenario, except to see whether an option is passed into unixfs. Not sure if supported yet
[ ] Add a directory with hidden files without -H enabled
[ ] Add a directory with hidden files and -H is enabled
[ ] Add a file or directory that does not exist

trycmd tests are a bit limited here as we can't really see that the correct underlying APIs are called. Instead we will have to rely on API-level tests where hopefully the right factoring can make us detect that the correct underlying code is called.
Maybe later

--column can be supplied multiple times to control the column names outputted. Pick from action, cid and path. If no --column entries exist, all columns are outputted.
-C or --root-cid shows no columns or progress bar, only the final root CID by which this content can be outputted. This should override any --column specifications.
adding from stdin

We want progress bars here first, not yet for iroh get. This needs to be multi progress bar, one per file, after an initial phase where it scans all the files you want to add.

Reconciling your flags with mine:

Instead of --no-wrap I'd prefer --wrap, which defaults to true. IMHO using negation in boolean flag labels is confusing: --wrap=false is clearer than --no-wrap
I don't think we need the --column flag quite yet, but I think it's well planned out and would not fight it in a future release.
do we need the -q flag? I'd drop it unless bash magicians want to overrule me
if we include --column, -C should override it.

in kubo <path> is variadic, allowing multiple paths to be ingested into the same DAG bound by a common root. We might need to support this in a later release, but we do need to do it at some point. I'm fine if v0.1.0 only takes a single path

Full help text:
$ iroh add --help
iroh-add
Add a file to iroh & make it available on IPFS

USAGE:
    iroh add [OPTIONS] <PATH>

ARGS:
    <PATH>    The path to a file or directory to be added

OPTIONS:
    -C, --root-cid               Output only the final Content Identifier (CID)
    -h, --help                   Print help information
    -H, --hidden                 Include files that are hidden. Only takes effect on recursive add.
    -n, --only-hash              Only chunk and hash. Do not write to disk.
    -p, --progress               Stream progress data. Default: true
    -r, --recursive              Add directory paths recursively. Default: false
    -w, --wrap-with-directory    Wrap files with a directory object. Default: true

NOTE: IROH CURRENTLY PROVIDES NO WAY TO REMOVE CONTENT ONCE ADDED. This will be
addressed in a future release.

Add copies the file or directory specified by of <path> into the iroh store,
splitting the input file into a tree of immutable blocks. Each block is labeled
by the hash of its content. The final output of the add command is the hash of
the root of the tree, which contains references to all other blocks:

  > iroh add example.jpg
  added [HASH] example.jpg
  added [ROOT_CID]

[ROOT_CID] is a Content IDentifier (or CID), which is a hash plus some extra
metadata. The opposite of the add command is the get command, which accepts a
CID and turns it back into files or directories:

  > iroh get [ROOT_CID]
  [ROOT_CID] written to working directory

The stored result of add is a 'MerkleDAG'. Merkle proofs (hashes) are a fast
method of proving and checking data inclusion, and the tree formed by chunking
the input into blocks is always a directed acyclic graph (DAG). These MerkleDAGs
can be provably checked for tamper resistance by anyone who fetches all blocks
in the tree, which means MerkleDAGs can be provided by anyone, without concern
for tampering.

Once content is added to iroh, it can be provided & rehosted by any other IPFS
node with iroh start:

  > iroh start

By default all content added to iroh is made available to the network, and the
default network is the public IPFS network. We can prove this by creating an
empty iroh instance. Run this in a separate terminal with iroh start running:

  > IROH_PATH=iroh_temp && iroh get [ROOT_CID]

We can use an HTTPS gateway hosted at https://gateway.lol to fetch the content
from the node running with iroh start:

  > curl https://gateway.lol/ipfs/[ROOT_CID]/example.jpg

In this case we're trusting whoever runs gateway.lol to verify the merkle proof
for us, because the last mile of our request is over HTTPS, whereas 'iroh get'
always performs this check locally.

The wrap option, '-w', wraps the file (or files, if using the recursive option)
in a directory and is on by default. This directory contains only the files
which have been added so the input retains its filename. set -w=false for
'raw' adds that lose their input name:

    > iroh add example.jpg
    added [CID] example.jpg
    added [ROOT_CID]
    > iroh add example.jpg -w=false
    added [ROOT_CID] example.jpg

Note that in both cases the CID of example.jpg is the same. Adding the same
file/directory with the same flags is deterministic, which means the same input
paired with the same configuration flags will give the same root hash.

I don't think we need the --column flag quite yet, but I think it's well planned out and would not fight it in a future release.

Given that we have a progress bar based user interface this is likely less relevant. I think we may be better off with a --format=json that just dumps all columns for the use case where people need data.

The whole
    added [CID] example.jpg
    added [ROOT_CID]

bit doesn't seem like we'll implement it in this form, given that we anticipate having a single progress bar with a status line that indicates the current file. #263

I imagine --only-hash does not actually upload or announce the chunks anymore and is really a kind of test run? Is that correct? "Does not write to disk" is confusing me -- why would iroh add ever write to disk? Does this mean do not write to the store"?

I'm not sure how to implement short forms of -w and -p. These flags are on by default, and in this they're not really flags at all, but values. Is -p=false a thing people do in CLIs? See #310 for a discussion about the technical limitations.

I imagine --only-hash does not actually upload or announce the chunks anymore and is really a kind of test run? Is that correct? "Does not write to disk" is confusing me -- why would iroh add ever write to disk? Does this mean do not write to the store"?

yes --only-hash should not write to the store, not announce, and drop all chunks created. It's basically used as a "what would iroh call this" function. Let's punt the --only-hash flag to a later release. Based on your confusion I think it merits more thought

I'm not sure how to implement short forms of -w and -p. These flags are on by default, and in this they're not really flags at all, but values. Is -p=false a thing people do in CLIs? See https://github.com/n0-computer/iroh/issues/310 for a discussion about the technical limitations.

Don't implement short forms in this case. -p=false destroys the legibility argument entirely. Users should have to explicitly supply --wrap=false.

If I understand it correctly then wee could call --only-hash --dry-run as that's what tools often name it. Once we implement the capability.


[FAQ] Minimal clear, using gles2 on ipad not clearing & causing GL_INVALID_ENUM

I'm just getting sokol_gfx integrated and replacing my old c++ bindings, so starting with a minimal implementation;
I have a GLView, which I'm manually repainting. In the view delegate drawInRect callback, (which is always on mainthread) I added
glClearColor(1, 0, 0, 1);
glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT|GL_STENCIL_BUFFER_BIT);

This works. (no flip, no make-current-thread-context etc). Checked glGetError() before & after, always 0
I've got a sokol desc setup
sg_desc desc={0};
sg_setup(&desc);

From my drawInRect callback, I call a simple default pass to clear (viewrect x/y from the cgrect)
sg_pass_action mPassAction = {0};
mPassAction.colors[0] = {.action = SG_ACTION_CLEAR, .val = {0, 1, 0, 1}};
sg_begin_default_pass( &mPassAction, ViewRect.x, ViewRect.y );
sg_end_pass();
sg_commit();

this;

Clears black (or, stays red if my raw gl code is still there)
No Error at end of pass/commit
On second frame, in sg_begin_default_pass has a glError of GL_INVALID_ENUM

I've tried

Ensuring thread is set/madecurrent. (setting to null at end of frame flushes error, but otherwise no diff as already bound to context in thread)
Adding setting GL_FRAMEBUFFER to 0 in my code (no diff)
Removing my gl code (then clears black)
Commenting out all the other gl calls in sokol (doing this has lead to clearing green, but still erroring!)

Just nailing down what else is being called, and which change allowed clear to correct colour...
Not seeing any memory corruption or anything odd (no odd params etc)
AFAIK no other gl code is called (unless deep inside ios, profiling calls hasn't revealed any). So I'm currently a bit stumped as to what is possibly causing the GL_INVALID_ENUM.
It could be erroneous somewhere in ios or how I've got my glview setup... but I only get this error if I call sokol code. (Plus I think that default pass should be clearing without changing any sokol code)

commenting out
glBindFramebuffer(GL_FRAMEBUFFER, _sg.gl.cur_context->default_framebuffer); in _sg_gl_begin_pass
Makes the green clear appear. Still errors, and glError() immediately after this call has no error.

If I switch to GLES3 so I can read the framebuffer bindings;
GLint drawFboId = 0;
GLint readFboId = 0;
glGetIntegerv(GL_DRAW_FRAMEBUFFER_BINDING, &drawFboId);
glGetIntegerv(GL_READ_FRAMEBUFFER_BINDING, &readFboId);

These are both 1 (GLKView is always rendering to it's own allocated buffer? a metal wrapper perhaps?)
Ipad Pro 24" 2020 on ios14...

Also, with GLES3 instead of GLES2, _sg_gl_begin_pass asserts at the end of the function with glGetError() == GL_INVALID_FRAMEBUFFER_OPERATION
Guessing that's the driver just being a little different (maybe GLES2 underneath is missing a check)

As soon as I started searching for default_framebuffer the answer revealed itself; (plus some documentation I missed - though the call order still wasn't super clear, so writing this out for anyone who finds the same problem)
I have to create my sokol context on the first render callback which sets up the default frame buffer :)
auto OnPaint = [this](CGRect Rect)
{
	[EAGLContext setCurrentContext:mOpenglContext];
	if ( mSokolContext.id == 0 )
	{
		sg_desc desc={0};
		sg_setup(&desc);
		mSokolContext = sg_setup_context();
	}
	// callbacks and render etc here
};

Notably, it seems like ios only sets up the frame buffer before rendering, NOT on context creation, this does NOT work
// on my create-stuff thread
mOpenglContext = [[EAGLContext alloc] initWithAPI:Api];
mView.context = mOpenglContext;
sg_desc desc={0};
sg_setup(&desc);
mSokolContext = sg_setup_context();

// on paint later will bind to 0

I think this is just PEBCAK, but... it wasn't obvious :) (just figuring out why my clear is h/2 and w/2, like the view rect and framebuffer size dont match)

Hmm, lot's of stuff to unpack :)
Unfortunately, sokol_gfx.h can't assume that the framebuffer of the default pass has the id 0 because this is not true on various platforms (not just iOS) so what it does is querying the currently bound framebuffer in sg_setup() (which in turn calls _sg_gl_create_context() for the "default context"):
https://github.com/floooh/sokol/blob/36b35207c7016ef42c43c7425e21ffd4c11bf01b/sokol_gfx.h#L5586
The expectation is basically that you create and make active the GL context before calling sg_setup().
This "default GL context" will then be bound in sg_begin_default_pass():
https://github.com/floooh/sokol/blob/36b35207c7016ef42c43c7425e21ffd4c11bf01b/sokol_gfx.h#L6286
GLKView seems to do this automatically (make the GLKView GL context active as soon as it is created), but on other platforms this must be done explicitly before sg_setup() is called.
On macOS (which uses NSOpenGLView) this must be done explicitly, I think the best place to do this is in the prepareOpenGL callback:
https://github.com/floooh/sokol/blob/36b35207c7016ef42c43c7425e21ffd4c11bf01b/sokol_app.h#L2798-L2804
On other platforms this is usually done right after the GL context has been created:
Win32/WGL:
https://github.com/floooh/sokol/blob/36b35207c7016ef42c43c7425e21ffd4c11bf01b/sokol_app.h#L4640-L4646
It seems that some platforms also need to set the GL context at the start of each frame, for instance on Linux/GLX I do this:
https://github.com/floooh/sokol/blob/36b35207c7016ef42c43c7425e21ffd4c11bf01b/sokol_app.h#L8099

As for the "clear only clears w/2, h/2": This is because sokol-gfx resets the viewport and scissor before clearing to the framebuffer size that's passed into sg_begin_default_pass() here:
https://github.com/floooh/sokol/blob/36b35207c7016ef42c43c7425e21ffd4c11bf01b/sokol_gfx.h#L6288-L6289
If it only clears a quarter of the screen this means that there's a HighDPI mismatch. The GL framebuffer is high resolution, but the size passed into sg_begin_default_pass() isn't.
I had a similar problem with NSOpenGLView on macOS because of the Catalina breakage (and this problem still wasn't fixed so I had to implement a workaround. The problem was that the reported framebuffer size was wrong right after NSOpenGLView was created, instead I had to move querying the framebuffer size into the reshape callback:
https://github.com/floooh/sokol/blob/36b35207c7016ef42c43c7425e21ffd4c11bf01b/sokol_app.h#L2781-L2793
However on iOS the initial framebuffer_width/height is taken from UIScreen.mainScreen.bounds, which I then multiply by 2 if HighDPI rendering is requested:
https://github.com/floooh/sokol/blob/36b35207c7016ef42c43c7425e21ffd4c11bf01b/sokol_app.h#L3113-L3124
...and during frame rendering, the size is taken from GLKView.drawableWidth:
https://github.com/floooh/sokol/blob/36b35207c7016ef42c43c7425e21ffd4c11bf01b/sokol_app.h#L3057-L3058
(I'm actually not sure why I'm not using drawableWidth/Height also for the initial size)

I guess The width/height is a seperate issue, I'll figure out if the drawable rect is definitely different (my rect/onpaint is coming from the GLKView delegate's draw callback)
My initial confusion I think all stemmed from the fact that I didn't realise I had to make my own context wrapper, as I could happily make a sg_context AND draw with no errors on frame0.
Also FYI (in case its some cause of problems) my GLKView is in a xib, with NO GLKViewController. (Allowing me to do metal & opengl in one app/view!)

I had to make my own context wrapper

Hmm, I'm a bit confused by this. You shouldn't need to call sg_setup_context() unless you're going to use more than one GL context. sg_setup() already does this to create an internal default sokol-gfx context" (this does not create a GL context, just create a local "resource environment" for sokol-gfx):
https://github.com/floooh/sokol/blob/2cd00ec6c80d4ef05d0522c7c0a5de0fa6daaadd/sokol_gfx.h#L13969
On the GL backend, sg_setup_context() calls _sg_gl_create_context(), which queries the currently bound framebuffer id (which will bound each time sg_begin_default_pass() is called):
https://github.com/floooh/sokol/blob/2cd00ec6c80d4ef05d0522c7c0a5de0fa6daaadd/sokol_gfx.h#L5586
...and a context also creates its own global VAO object:
https://github.com/floooh/sokol/blob/2cd00ec6c80d4ef05d0522c7c0a5de0fa6daaadd/sokol_gfx.h#L5591
...but that's it.

If I had called sg_setup() in the draw callback, I think it would have been fine. (Evidently, as that's what I do now)
Now I think about it, if I was calling sg_setup() outside the gl's [current] thread (even if not from the draw callback), shouldn't some of the initialisation GL calls have failed?...
In fact, perhaps, when I was calling it before, there was no gl context at all (but again, no errors...)
I totally understand what's happening underneath re: contexts/sg contexts (I'm managing render contexts, flipping etc and sg is just the generic rendering).
I just don't quite get why there weren't [more obvious] errors when I was doing it wrong :) (and wouldn't want others to have the same problem in future)

I do have it working, so don't spend too much time digging more into this, unless perhaps this crops up again for more users?

That's why I slapped a FAQ label to the issue :) But I'll close it anyway, otherwise it's too distracting.


planning scene ros api

Live version

@davetcoleman Ready for merge?

@davetcoleman added mvt and some other minor fixes. Ready for another review


JSON Schema conformity using Llama.cpp Grammar generation for Tool Calling

First thanks to the ollama Team, it's a pleasure to use!
I was looking into the Topic of Schema and Grammar from the tools' perspective.
I assume:

the tool arguments json schema, only inserted into the Prompt by ollama
no grammar is used to enforce the json schema for tool arguments
(only the json grammar is used)
the json-schema is not used for validation after parsing as well

Since Multiple pull request for the Topic of JSON schema and Grammars are open, I would like to address that by suggesting closing them.
For the few people that actually need Grammar support, using llama.cpp without ollama seem appropriate, since using grammars is already very low level.
ollama could focus to support tools calling with JSON Schema conformity.
One would need to combine all possible Tools into one JSON Schema.
A bit like in this https://github.com/ggerganov/llama.cpp/issues/7703 example.
Then the model can be constrained to comply to the correct tool use.
The problem is the exact tool support output depends on the model:
https://github.com/ollama/ollama/blob/f5e3939220e9cd3d7a636708bc9df031ebfd4854/server/testdata/tools/llama3-groq-tool-use.out#L14
vs
https://github.com/ollama/ollama/blob/f5e3939220e9cd3d7a636708bc9df031ebfd4854/server/testdata/tools/firefunction.out#L17
vs
https://github.com/meta-llama/llama-models/blob/e51c73ac639a38877da9bdfaecb4cb07dc8ba6d0/models/llama3_1/api/tool_utils.py#L16
How to deal with the:
--"functools" in firefunction
--"<tool_call>" token in llama3-groq
--"<function=****>" for llama 3.1
is not clear for me jet, since I think it would be best to leverage the JSON-Schema to grammar functionality in llama.cpp.
@mxyng What do think about that?
Should I spend time working or that, or is there no chance to be merged into ollama?
This should address:
https://github.com/ollama/ollama/issues/5976
https://github.com/ollama/ollama/issues/1507
https://github.com/ollama/ollama/pull/5348
https://github.com/ollama/ollama/pull/830
https://github.com/ollama/ollama/pull/565
https://github.com/ollama/ollama/pull/1606
https://github.com/ollama/ollama/pull/4525
https://github.com/ollama/ollama/pull/2404
on the consumer side:
https://github.com/thmsmlr/instructor_ex/issues/11

I'd love to see this feature added. I don't understand why it isn't just passed straight through to ggml via llama.cpp. We need arbitrary grammars in GBNF format.

Hey all, I know there's an automated ping here but just to better align everyone please check out and comment on my new call to the Ollama team for clarity here. As always please be civil and stay on topic! 😄 - https://github.com/ollama/ollama/issues/6237

@Kinglord Thanks for creating a place for that discussion.
I tried to get around that discussion here.
Because tool use with models that are trained specifically for that, like llama3.1,
comes with its own challenges for the grammar.
Because just using the JSON schema to grammar conversion in llama.cpp will not be enough as explained.
A way to template the grammar generation in the model file could be a solution that I am thinking about.

Hi! Going to close this out as we're supporting structured outputs through https://github.com/ollama/ollama/pull/7900
Left a comment with some background as well: https://github.com/ollama/ollama/issues/6237#issuecomment-2518836758

@ParthSareen  I interepted this request as actually using the provided tool calling schema for the grammar schema -- which i believe is slightly different than supporting structured outputs in the response.  Did this change actually support grammars for tool calls?
I want to make sure this was intentional and not a misunderstanding. Having tool calls following the schema would be a huge quality win for smaller models that don't always produce correct tool call outputs.

Thanks for the ping @allenporter. Seems like I misinterpreted on my first run through of this. Reopening the issue for now. Going to think a bit about how we can support this, what does extensibility look like, and if it makes sense for the stage of the project we're in. I'm also not sure what the exact interface would look like but am going to think through this one. I do think this could be really cool and improve accuracy. Just a bit worried about the interface as there are big updates to the engine incoming.
Will keep you all posted! Thanks!

Ok thanks, I'm happy to help contribute (I'm familiar with some of how the tool parsing code works) but also I know it's a bit tricky as described above since it also depends on model specific formats as described above by the reporter. And as you say if the tech direction is shifting here, adds a bit more to be requirements...
Happy to keep discussing.
Just to motivate this a bit: The primary use case I'm pushing is or to improve tool use quality for home assistant device control where llama3.1 sometimes gets the tool params wrong. (We have some benchmarks tracking this)

@allenporter If you'd like you can take a crack at it for fun and see how far you get. There are a couple things I need to get to in the meantime but can pick it up from wherever you leave off. We can coauthor it if you're interested 😄 For a starting point I'd dig into the ChatHandler around here: https://github.com/ollama/ollama/blob/da09488fbfc437c55a94bc5374b0850d935ea09f/server/routes.go#L1467-L1530
We're also currently using go templates for the tool parsing - something that I potentially want to refactor too. Would be out of scope for the PR but important to keep in mind with whatever you prototype. If you do choose to pick this up just open a draft PR and tag me!
Thanks!

We're also currently using go templates for the tool parsing - something that I potentially want to refactor too.

Yeah, when I was looking at this looked at, it seemed a little difficult to specify up front as a grammar since:
(1) Tool response format depends on the model / template
(2) Tool responses are optional
My impression is the flow is something like this to prepare for a tool call:

https://github.com/ollama/ollama/blob/900f64e6be859f52350c25032ff5b11f10509c7e/server/prompt.go#L25 appends any present tool calls into the request
The completion request point you cited will have the tool calls in the request
The response is parsed https://github.com/ollama/ollama/blob/900f64e6be859f52350c25032ff5b11f10509c7e/server/model.go#L303 using the template.
Response parsing is a little more creative. It assumes that the output format must contain json inside of what is specified in the template (works for most models, but not all).

It instantiates the template with a fake tool call with placeholder variable names
It then brute force parses the output as json https://github.com/ollama/ollama/blob/900f64e6be859f52350c25032ff5b11f10509c7e/server/model.go#L277 iterating through each character, parsing objects as it goes
Then it reverse engineers the fields that contain the tool names and arguments from the placeholder names
Then repeats the same process for the real output collecting the tool call objects

Proposed approach to get started:

Maybe we start simple with a model like llama3 which has a very straight forward format: Either it's responding with text, or it's responding with json, with no other wrapping characters.
Define a grammar where the structure is optional.  Unless it starts producing json: it has a schema where the parameters match the tool call in the request.
It looks possible to even try this out with the current request API by making assumptions about the model format, except for the part where it's optional

@allenporter I think this sounds like a good start - would be cool to get a prototype working! Keep me posted!

I made a simple notebook to call llama3.1 w/ tools while also passing  the json schema of the tools to the format parameter as an anyOf:
https://github.com/allenporter/ml-papers/blob/main/function-calling/ollama/json-schema.ipynb
It works well (it's easier to tell if its working by changing around the tools and mismatching the questions since you can see it force tool calls it would not otherwise).
This does makes tool calling mandatory when using this approach.  Llama does bias towards calling tools when provided generally, but when you apply the format it will always call tools, which is not always desired.

I've applied this technique to the Home Assistant assist-mini benchmark using the naive approach above and it appears to improve from 83% to 91% before and after confirming that using a tool schema is a worthwhile improvement to explore.

As a next step, i'll try:

get a grammar working forcing a json schema inferred from tool schema (always)
make tools optional in the grammar
see if its possible to create a grammar from the template output

+1, structured outputs from tool calls would be extremely helpful for adoption.


YCSB for Aerospike benchmark

Hi there,
I am trying to use YCSB to benchmark Aerospike(AS) database. I have successfully installed Aerospike and the installation seems to be good (verified as per AS site),  I have downloaded the YCSB via Git and compiled and built using Maven, and build is successsful,
then I am trying to load data for AS using the below command as suggested in the site / wiki:
./bin/ycsb load aerospike -s -P workloads/workloada >outputLoad.txt
but it fails with an error:
: No such file or directory
but infact all the directories and contents are present, permissions seems to be good for bin and ycsb both, am not sure where going wrong,
need help
thanks

@phanimv - can you post the entire output when you run your load command?
Also, are you running YCSB on the same machine that Aerospike is installed on?

I am running YCSB on the same machine on which AS installed,
here is what i get, am listing the pwd and contents of the same, which might help,
[root@pkhr YCSBbfc]# pwd
/root/myGit/YCSBbfc
[root@pkhr YCSBbfc]# ls
accumulo        cassandra2      dynamodb         hbase10      mapkeeper       pom.xml
aerospike       checkstyle.xml  elasticsearch    hypertable   mongodb         README.md
bin             core            gemfire          infinispan   nosqldb         redis
binding-parent  couchbase       googledatastore  jdbc         NOTICE.txt      tarantool
BUILD           distribution    hbase094         kudu         orientdb        voldemort
cassandra       doc             hbase098         LICENSE.txt  outputLoad.txt  workloads
[root@pkhr YCSBbfc]# ./bin/ycsb load aerospike -s -P workloads/workloada >outputLoad.txt
: No such file or directory
[root@pkhr YCSBbfc]#

Oh, right you're also directing the output to outputLoad.txt. Can you rerun the command without >outputLoad.txt and post the output?

I have re-run the command, which seems to be same, also i have tried cat of outputLoad.txt which is empty as well.
[root@pkhr YCSBbfc]# ./bin/ycsb load aerospike -s -P workloads/workloada
: No such file or directory
[root@pkhr YCSBbfc]# cat outputLoad.txt
[root@pkhr YCSBbfc]#
thanks

@phanimv - what OS and what type of shell are you using?
I can almost replicate your issue if I try and run something that does not exist, ex: ./bin/ycsb2 but I get a little better output from bash. This leads me to believe that there is something strange in the environment you are running in.
The base line test you can run is simply:
./bin/ycsb

This should print out a usage report. If it does not then you have an issue probably unrelated to YCSB.

A few other things you might check:

Is java on your path? which java
Have you tried running from a release of YCSB, rather than a git checkout?

RHEL 7
bash
java path is set properly,
using which am able to build packages usign Maven

even with simple ./bin/ycsb it is the same
./bin/ycsb
: No such file or directory

Are you missing python?

@cmccoy
Have you tried running from a release of YCSB
does this mean, I have to download the gz files from the below page ?
https://github.com/brianfrankcooper/YCSB/releases/tag/0.5.0
I should download ycsb-0.5.0.tar.gz ? does this suffice ?
ycsb-aerospike-binding ? also required ?
thanks

This should be all you need: https://github.com/brianfrankcooper/YCSB/releases/download/0.5.0/ycsb-0.5.0.tar.gz
Unpack it with something like tar -xvf, then run ./bin/ycsb from inside.
From trying some random things, I see similar No such file or directory errors (with a little better diagnostic attached) when things like this are happening:

no python
no /usr/bin/env
no bin/ycsb

@kruthar @cmccoy
just now I have downloaded ycsb-0.5.0.tar.gz, ,
unzipped and tried the basic command ./bin/ycsb, it gives the usage as below:
I believe it got downloaded properly.
[root@pkhr ycsb-0.5.0]# ./bin/ycsb
usage: ./bin/ycsb command database [options]
Commands:
load           Execute the load phase
run            Execute the transaction phase
shell          Interactive mode
Databases:
accumulo       https://github.com/brianfrankcooper/YCSB/tree/master/accumulo
aerospike      https://github.com/brianfrankcooper/YCSB/tree/master/aerospike
basic          https://github.com/brianfrankcooper/YCSB/tree/master/basic
cassandra-10   https://github.com/brianfrankcooper/YCSB/tree/master/cassandra
cassandra-7    https://github.com/brianfrankcooper/YCSB/tree/master/cassandra
cassandra-8    https://github.com/brianfrankcooper/YCSB/tree/master/cassandra
cassandra-cql

Great, so the usage report is expected.

I have tried the below command
./bin/ycsb load aerospike -s -P workloads/workloada
now the question / clarity I need is, how to make sure this is running proper and everything is clean
it executes as below:
/usr/java/jdk1.8.0_65/bin/java -cp /root/ycsb5/ycsb-0.5.0/aerospike-binding/conf:/root/ycsb5/ycsb-0.5.0/conf:/root/ycsb5/ycsb-0.5.0/lib/jackson-core-asl-1.9.4.jar:/root/ycsb5/ycsb-0.5.0/lib/core-0.5.0.jar:/root/ycsb5/ycsb-0.5.0/lib/HdrHistogram-2.1.4.jar:/root/ycsb5/ycsb-0.5.0/lib/jackson-mapper-asl-1.9.4.jar:/root/ycsb5/ycsb-0.5.0/aerospike-binding/lib/jbcrypt-0.3m.jar:/root/ycsb5/ycsb-0.5.0/aerospike-binding/lib/luaj-jse-3.0.jar:/root/ycsb5/ycsb-0.5.0/aerospike-binding/lib/aerospike-binding-0.5.0.jar:/root/ycsb5/ycsb-0.5.0/aerospike-binding/lib/aerospike-client-3.1.2.jar:/root/ycsb5/ycsb-0.5.0/aerospike-binding/lib/gnu-crypto-2.0.1.jar com.yahoo.ycsb.Client -db com.yahoo.ycsb.db.AerospikeClient -s -P workloads/workloada -load
YCSB Client 0.1
Command line: -db com.yahoo.ycsb.db.AerospikeClient -s -P workloads/workloada -load
Loading workload...
Starting test.
2015-12-08 01:15:07:231 0 sec: 0 operations; est completion in 0 seconds
2015-12-08 01:15:07:614 0 sec: 1000 operations; 1216.55 current ops/sec; [CLEANUP: Count=1, Max=1049, Min=1049, Avg=1049, 90=1049, 99=1049, 99.9=1049, 99.99=1049] [INSERT: Count=1000, Max=30303, Min=110, Avg=318.83, 90=392, 99=1896, 99.9=9711, 99.99=30303]
[OVERALL], RunTime(ms), 822.0
[OVERALL], Throughput(ops/sec), 1216.54501216545
[CLEANUP], Operations, 1.0
[CLEANUP], AverageLatency(us), 1049.0
[CLEANUP], MinLatency(us), 1049.0
[CLEANUP], MaxLatency(us), 1049.0
[CLEANUP], 95thPercentileLatency(us), 1049.0
[CLEANUP], 99thPercentileLatency(us), 1049.0
[INSERT], Operations, 1000.0
[INSERT], AverageLatency(us), 318.834
[INSERT], MinLatency(us), 110.0
[INSERT], MaxLatency(us), 30303.0
[INSERT], 95thPercentileLatency(us), 457.0
[INSERT], 99thPercentileLatency(us), 1896.0

The output here looks good. There should be an additional line at the bottom with return codes from the operations that isn't shown here. If you have 1000 entries for OK, then you shouldn't have had any issues.
You can verify that the records were properly inserted by some other means like looking directly in Aerospike.

Yeah I can see that, infact I didn't copy whole message in earlier message:
here is the tail of the message:
[INSERT], MaxLatency(us), 30303.0
[INSERT], 95thPercentileLatency(us), 457.0
[INSERT], 99thPercentileLatency(us), 1896.0
[INSERT], Return=OK, 1000
thanks

@kruthar based on your response, I believe ycsb installation is good,
I shall try to load large data and try to benchmark AS, shall get back to you about it went on,
otherwise,
@kruthar @cmccoy thank you very much for your quick turn around and responses, very much appreciate it and it is helpful.
Thanks and Cheers, Shall get back to you soon

Hi,
I am able to successfully load data ~40GB,
but when am trying to run the stress test using the below command,
./bin/ycsb run aerospike -s -P workloads/workloada
there are lot of errors of key not found, though at the sametime there are many reads, still ultimately it breaks out with the following error
Error while writing key user126681452320941688: com.aerospike.client.AerospikeException: Error Code 2: Key not found
Record key user4200679211667895822 not found (read)
Record key user9156865532434275658 not found (read)
Record key user2406830966585268249 not found (read)
Write failed: Connection reset by peer

So, it looks like there could possibly be two issues here.

Keys not found on reads and updates
Whatever is causing this: "Write failed: Connection reset by peer"

I'm going to focus on the first one for now.
Did you see any errors when you loaded your data, did you get all OK return codes?
Can you verify that you have the expected number of records in Aerospike?
Have you tried manually checking Aerospike for the missing records by key?
Have you tried loading and testing a smaller data set, if so, did you have the same issues?

the configuration is mostly defaults, except the record count:

recordcount=30000000
operationcount=30000000
workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=true
readproportion=0.5
updateproportion=0.5
scanproportion=0
insertproportion=0
requestdistribution=zipfian

there were no errors in data load:  tails of the same below:

[CLEANUP], 99thPercentileLatency(us), 2007.0
[INSERT], Operations, 3.0E7
[INSERT], AverageLatency(us),<PHONE_NUMBER>
[INSERT], MinLatency(us), 52.0
[INSERT], MaxLatency(us), 28319.0
[INSERT], 95thPercentileLatency(us), 238.0
[INSERT], 99thPercentileLatency(us), 269.0
[INSERT], Return=OK, 30000000
I have tried the default for the smaller which is 1000 records, both load and run were good,

@phanimv - any luck here? Were you able to do these two things:

Can you verify that you have the expected number of records in Aerospike?

Have you tried manually checking Aerospike for the missing records by key?

I would also suggest trying a few load values that start small and build up to your 30000000 and see at what point things go wrong.

Hi,
@kruthar - I was off from the work for couple of days, and couldnt try over weekend as well,
going to give a try today, I hope I should have good luck in this, get back to you tomorow.
thanks

@kruthar  Hi,
I have loaded, end of the load below is the message:
[CLEANUP], MaxLatency(us), 2005.0
[CLEANUP], 95thPercentileLatency(us), 2005.0
[CLEANUP], 99thPercentileLatency(us), 2005.0
[INSERT], Operations, 3.0E7
[INSERT], AverageLatency(us),<PHONE_NUMBER>
[INSERT], MinLatency(us), 53.0
[INSERT], MaxLatency(us), 31103.0
[INSERT], 95thPercentileLatency(us), 273.0
[INSERT], 99thPercentileLatency(us), 292.0
[INSERT], Return=OK, 30000000
which I believe should be good,
then I have run the stress test against the DB, end of which below is the message:
[CLEANUP], MaxLatency(us), 2909.0
[CLEANUP], 95thPercentileLatency(us), 2909.0
[CLEANUP], 99thPercentileLatency(us), 2909.0
[UPDATE], Operations, 1.5004515E7
[UPDATE], AverageLatency(us),<PHONE_NUMBER>3251933
[UPDATE], MinLatency(us), 47.0
[UPDATE], MaxLatency(us), 17535.0
[UPDATE], 95thPercentileLatency(us), 243.0
[UPDATE], 99thPercentileLatency(us), 272.0
[UPDATE], Return=OK, 15004515
I believe now the load is good and proper. the final OK should be the confirmation, right ?
thanks

Looks fine to me as long as you didn't get any errors reported.

We faced a similar issues where the reads failed. The problem was that the inserts during the loads failed if the  set wasn't empty which is weird. We had to clear the set and run the load everytime which made the test work

@GavinDmello if this was with a current version of YCSB, please open a new issue with details. You should only need to run the load phase a single time, presuming your test runs are all with the built-in core workload generator.


Make TokenBucket extendable

fixed class ref
modified private's to protected, in order the make TokenBucket extendable

Why do think it is a good idea to make that class extendable?

For a project I had different requirements for the consume method, so I wanted to extend the class and implement a custom consume method. But since the properties used in the consume method are private, it wasn't possible without making the properties protected.

Could you please be more specific, what requirement made it necessary to extends TokenBucket?
Making this class extendable means increasing the public API. That's a big impact on maintainability. So you need to convince me in doing so.


Trigger indexes change event even if they don't change

Related to: https://github.com/CartoDB/cartodb/issues/12661
The handlers where getting stuck in the middle of the buckets, debugging I found that if you moved the handler to the first half of the next bucket, it should move where it was previously, which means the same indexes, so the backbone model didn't trigger any changes which lead to the function which calculates the handler position not being called.
The proposed solution is to make the changes silent, and the trigger the event manually.

Works like a charm ✨


type 'Null' is not a subtype of type 'String' in type cast

The error is thrown when fetching app details

Should be fixed in peekanapp: ^0.0.2, let me know if you still have this problem.


It's not working

Hello, I'm using with ionic.
I think I'm doing all right however seems not working.
I'm link the files
<script src="js/ionic.bundle.min.js"></script>
<script src="js/imgcache.js"></script>
<script src="js/angular-imgcache.js"></script>
<script src="js/app.js"></script>
<script src="js/controllers.js"></script>
<script src="js/services.js"></script>

In the file app.js I init the imgcache
angular.module('starter', ['ionic', 'starter.controllers', 'starter.services', 'ImgCache'])

.run(function($ionicPlatform, ImgCache) {
	$ionicPlatform.ready(function() {
		if (window.cordova && window.cordova.plugins.Keyboard) {
			cordova.plugins.Keyboard.hideKeyboardAccessoryBar(true);
		}

		ImgCache.$init();
	});
})

.config(function($stateProvider, $urlRouterProvider, $ionicConfigProvider, ImgCacheProvider) {
  	ImgCacheProvider.manualInit = true;
...
});

And in the templates  I'm using
<img img-cache ic-src="http://site/{{ PhotoApp) }}" ng-src="http://site/{{ PhotoApp }}" alt="{{ Name }}">

Checking the cache size the images appear cached.
But after a while accessing the app offline the images don't work.
I forgot something?

Now I removed the ng-src in the  tag, and is working on chrome. But when I get the build with phonegap the images aren't displayed.
<img img-cache ic-src="http://site/{{ PhotoApp) }}" alt="{{ Name }}">

Were you able to solve this?
I'm having the same problem. It's working on the browser but not on a device.

Anyone know of a platform indepenent way?  this sucks cuz I got it all working only later to find that it doesn't work on cordova IOS (ionic app) or safari.   How is THAT a solution ?


Background not transparent/appearing

Background transparency not working as intended, and I am unable to see my custom background.

However, when I am in a menu, you can see it.

I'm having the same issue, can't seem to figure out how to fix it.

Until Zerthox uploads a fixed version there is a slight workaround.
Go to custom CSS, paste this in there:
`.container-2OU7Cz, .theme-dark .title-qAcLxz, .channels-3g2vYe, .theme-dark .chat form, .theme-dark .headerBar-cxbhPD {
background-color: transparent;
}
css.theme-dark .title-qAcLxz, .channels-3g2vYe {
Background-color: transparent !important;
}
.titleWrapper-3Vi_wz {
Background-color: rgba(0, 0, 0, 0.8) !important;
}
/* Main background and settings background */
.app,
.callout-backdrop {
background-size: cover !important;
}
.container-2OU7Cz, .channels-3g2vYe, .title-qAcLxz,
.theme-dark .chat form, .headerBar-cxbhPD
{ background-color: transparent; }
/* Channel List, Server Name */
.channels-3g2vYe, .title-qAcLxz {
background-color: rgba(0, 0, 0, 0.7) !important;
}
/* Channel Name Bar */
.theme-dark .title-qAcLxz {
background: rgba(0, 0, 0, 0.8) !important;
}
/* Friends Header */
.headerBar-cxbhPD {
background-color: rgba(0, 0, 0, 0.9) !important;
}`
Hit update hit save a couple times then type Ctrl + R.
If for some reason the CSS doesnt save across exiting discord/rebooting etc. You can paste the code at the bottom of the Theme file.

Fixed by now


Added ALT Tags for install/desktop-msi-installer-and-group-policy-ins…

/check-cla

Hello @jaskiratsingh2000,
Thanks for your pull request! A Core Committer will review your pull request soon. For code contributions, you can learn more about the review process here.

This PR has been automatically labelled "stale" because it hasn't had recent activity.
A core team member will check in on the status of the PR to help with questions.
Thank you for your contribution!

This PR has been automatically labelled "stale" because it hasn't had recent activity.
A core team member will check in on the status of the PR to help with questions.
Thank you for your contribution!


[Event Request] codeunit 432 Consolidate - OnBeforeRun

In codeunit 432 Consolidate please add the following event:
    trigger OnRun()
    var
        PreviousDate: Date;
        i: Integer;
    begin
        // New event
        OnBeforeRun(Rec);
        // New event
        BusUnit := Rec;
        if NormalDate(EndingDate) - NormalDate(StartingDate) + 1 > ArrayLen(RoundingResiduals) then

    [IntegrationEvent(true, false)]
    local procedure OnBeforeRun(var BusUnit: Record "Business Unit")
    begin
    end;

Thank you

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.


NextJS + TS example - Can't resolve 'react-transition-group'

Duplicates

[X] I have searched the existing issues

Latest version

[X] I have tested the latest version

Current behavior 😯
After cloning the nextjs with typescript example, installing and running the server, when accessing http://localhost:3000/about I'm getting the following error

./node_modules/@mui/material/ButtonBase/TouchRipple.js:13:0
Module not found: Can't resolve 'react-transition-group'
Import trace for requested module:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>./pages/about.tsx
https://nextjs.org/docs/messages/module-not-found

This is happening only when accessing /about page. Initially the home page is OK, but after attempting to access the about page, the home page too stops working. Seems like it's a problem with the Button component on the about page.
Expected behavior 🤔
Opening the about page (http://localhost:3001/about) works fine, and the home page continues to work
Steps to reproduce 🕹

Clone example like in the README.md

curl https://codeload.github.com/mui/material-ui/tar.gz/master | tar -xz --strip=2  material-ui-master/examples/nextjs-with-typescript
cd nextjs-with-typescript
npm install
npm run dev

Open http://localhost:3000/ (should work)
Click on Go to the about page link (or just visit http://localhost:3000/about)

Context 🔦
Attempting to setup a nextjs + material ui + typescript project.
Your environment 🌎

npx @mui/envinfo
  System:
    OS: macOS 12.4
  Binaries:
    Node: 16.14.0 - /usr/local/bin/node
    Yarn: Not Found
    npm: 8.15.1 - /usr/local/bin/npm
  Browsers:
    Chrome: 103.0.5060.134 (used)
    Edge: 103.0.1264.77
    Firefox: Not Found
    Safari: 15.5
  npmPackages:
    @emotion/react: latest => 11.9.3
    @emotion/styled: latest => 11.9.3
    @mui/base:  5.0.0-alpha.91
    @mui/icons-material: latest => 5.8.4
    @mui/material: latest => 5.9.2
    @mui/private-theming:  5.9.1
    @mui/styled-engine:  5.8.7
    @mui/system:  5.9.2
    @mui/types:  7.1.5
    @mui/utils:  5.9.1
    @types/react: latest => 18.0.15
    react: latest => 18.2.0
    react-dom: latest => 18.2.0
    typescript: latest => 4.7.4

Iam also expriencing the same issue

I was able to solve this issue installing the  "react-transition-group": "4.3.0" version in my package.json

Infact an issue with react-transition-group v4.4.3. They've published a fix to version v4.4.4 and now it works! Thanks to @jamolpe


Fix preparation of movement/senses ranges and units

I noticed in certain cases we currently end up with prepared null range/units instead of 0/"ft".

Units should probably be initialised to some default, but null is a valid value for these NumberFields. We don't prepare every single NumberField in the schema to 0 so it's unclear why doing so for these particular fields is necessary.

The units need to be initialized to for prepareMovement. The movement units are null, some of the movement reductions have no effect. No null ranges would be nice to have so that downstream code doesn't have to deal with that. In the context of detection modes a range of null is infinity: so there are chances that things go wrong with null ranges and a missing ?? 0 if the detection modes are prepared from the actors senses.

Yeah, definitely for units, but that makes sense for movement and senses as well


set offset.

Hi,
Current RandomState.seed() does not really reset all of random state.
Here is a small code to reproduce this issue:
from __future__ import print_function
from cupy.random import get_random_state, seed
from chainer import cuda

gpu = 0
device = cuda.get_device(gpu)
with device:
    seed(0)
    rs = get_random_state()

    for _ in range(10):
        r = rs.rand(1)
        print(r)

    print("====")

    seed(0)
    rs = get_random_state()

    for _ in range(10):
        r = rs.rand(1)
        print(r)

The results are here (different values are generated even though seed value is same):
[ 0.56154916]
[ 0.5396353]
[ 0.74978529]
[ 0.50525623]
[ 0.94698888]
[ 0.66230074]
[ 0.60323748]
[ 0.12558134]
[ 0.51783317]
[ 0.9571602]
====
[ 0.49158581]
[ 0.3454503]
[ 0.48739564]
[ 0.73569951]
[ 0.94801935]
[ 0.42100349]
[ 0.61444364]
[ 0.09178467]
[ 0.35837964]
[ 0.7166009]

This PR fixes this issue to generate same values.
[ 0.56154916]
[ 0.5396353]
[ 0.74978529]
[ 0.50525623]
[ 0.94698888]
[ 0.66230074]
[ 0.60323748]
[ 0.12558134]
[ 0.51783317]
[ 0.9571602]
====
[ 0.56154916]
[ 0.5396353]
[ 0.74978529]
[ 0.50525623]
[ 0.94698888]
[ 0.66230074]
[ 0.60323748]
[ 0.12558134]
[ 0.51783317]
[ 0.9571602]

Thank you for your PR!
LGTM.

We'll need to add a test @jnory wrote to tests directory.


Epic stack

Basic pyodide setup for Epic Stack

TODO resolve missing plaintext loader for .py files in Epic Stack.

[ ] I think we need to add ?raw to the end of the python import in javascript:
https://vitejs.dev/guide/assets.html#importing-asset-as-string

I found instructions about how to patch with a new file type here: https://github.com/remix-run/remix/discussions/2468#discussioncomment-2639271 and guessing it mostly works when I combined it with the other loader.ts file here

We got it working for me. I'd love if we can try it one more time on a fresh environment.


Release github and EcoCI plugins as npm packages

What
Release the new github and eco-ci plugins as npm packages under the GSF org
Why
for easier installation that does not rely on npm link
Context
These are not builtins, they are standalone plugins. They should follow best practises for distribution.
Prerequisites/resources
none
SoW (scope of work)

[x] publish github plugin to npm
[x] publish eco-ci plugin to npm

Acceptance criteria
Both plugins can be installed using npm i @grnsft/xxx`

@zanete this one is also covered
https://www.npmjs.com/package/@grnsft/if-github-plugin
https://www.npmjs.com/package/@grnsft/if-eco-ci-plugin


[Bug]: Not all requests show up when using page.on

Bug description
`Not all requests show up when using page.on, requests show up in network tab but not in console.
       const page = await browser.newPage()
        page.on('request', async (req) => {
            console.log(req.url())
        });
        await page.goto('https://www.twitch.tv/esl_csgo');
        await page.waitForTimeout(30000);`

Puppeteer version
17.1.3
Node.js version
18.1.0
npm version
8.8.0
What operating system are you seeing the problem on?
Windows
Relevant log output
No response

update: this does not happen in playwright

Note enough information to reproduce: which requests are missing? DevTools shows many requests that might not come from the page. Also, please retry using the latest Puppeteer version as there have been changes.


URI is not absolute: Referenced Schema contains self $refs in raml

This is the raml file i would like use for client generation but there is an exception during the code generation.
https://devportal.yaas.io/services/beta/loyaltyrule/v1/api.raml

Can you please describe the issue you have with more details?
Also, your raml is not valid - you defined rules schema twice.

Oh I see. :D but this s not my bug can remove it first local and send some feedback to the developers of the API.
I'm not so sure about the issues. There were these parts of Code producing the error:
"items": {                                       "$ref": "keyVal"                                   }

Can you please provide smaller raml file that reproduces the issue?
What is the actual error you are getting?

Hey,
I will try to provide a smaller file.
The actual error is: IllegalArgumentException: URI is not absolute.
It is failing at the provided sample location above.

Hi @stojsavljevic ,
I have a look at this issue.
First, I don't think it's a RAML 1.0 feature. From the RAML 1.0 sepcification, we can see that "schemas" and "schema" are deprecated and might be removed in a future RAML version.
Second, I don't think it's a bug of springmvc-raml-parser, but an error in raml file.
From the JSON Schema specification, we can refer a schema snippet of current document as following:
{ "$ref": "#/definitions/address" }
we can refer a a schema snippet of another document by relative URL as following:
{ "$ref": "definitions.json#/address" }
You can refer to the detail from [https://spacetelescope.github.io/understanding-json-schema/structuring.html].
But here from the raml file, we can see
` schemas:

addressbook: |
{
"$schema": "http://json-schema.org/draft-04/schema",
"title": "addressbook",
"description": "addressbook object",
"type": "object",
"properties": {
"address": {
"description": "address",
"type": "string",
"default":""
}
}
}
testinput1: |
{
"$schema": "http://json-schema.org/draft-04/schema",
"title": "Testinput1",
"description": "Testinput1 object",
"type": "object",
"properties": {
"name": {
"description": "name",
"type": "string",
"default":""
},
"addressbook": {
"$ref": "addressbook"
}
}
}`
Though here JSON schemas are defined in a raml file, but they are separated each other. We can see the "addressbook" and the "testinput1" as independent JSON schema files. So in the testinput1 schema file, we try to refer to another schema file addressbook, we have to include it, not ref it.

How do you think about it?
BR,
Xiangbin

Hi @champagne
I agree that it's not a 'real' 1.0 feature although still supported in the spec.
In general, I don't think we (in this project) should support schemas in 1.0 code flow.
But for 0.8 this looks like a missing feature.
Using include is just a workaround.
JSON schemas can reference each other using ref: http://json-schema.org/latest/json-schema-core.html#rfc.section.8
So, in the above example, if addressbook is already defined, testinput1 should be able to reference it using $ref.

Hi @stojsavljevic ,
Thanks for your reply.
Yes, for RAML 0.8, we should support JSON Schema.
But it not an issue of raml-parser, but an issue of json shema parser.
I have found that jsonschema2pojo has support $ref already. Please see https://github.com/joelittlejohn/jsonschema2pojo/wiki/Reference#ref.
For this issue, though addressbook and testinput1 are defined in a same raml file, but they are treated as two independent json schema file, but user try to refer an external json schema snippet by the way of refering to a internal json schema snippet. So I still believe this is a valid RAML file but with invalid JSON Schema. It's not a raml-parser bug.
The following links are helpful to understand how to using JSON Schema $ref:

http://json-schema.org/latest/json-schema-core.html#rfc.section.8
https://spacetelescope.github.io/understanding-json-schema/structuring.html

BR,
Xiangbin

It looks you're right @champagne
If somebody feels we're missing something - please reopen this one.

Just one more hint: you can try to use classpath: when defining $ref.
Something like this:
...
"payloadItems" : {
      "$schema": "http://json-schema.org/draft-04/schema",
      "type": "array",
      "items" :  {
         "$ref" : "classpath:payloadItem.schema"
     }
}
...


PVWatts-Battery "Adjust" Bug

There appears to be a bug in the PVWatts-Battery model with the "adjust" input:

Open a PVWatts-Battery Residential case
Click "simulate"- note that annual energy is 8023
Open the "adjust" factor on the System Design page, enter 25% in a constant loss
Click "simulate"- annual energy is still 8023

This does not appear to be happening in the normal PVWatts case. This bug is present in 2020.11.29 r2 as well as develop.

@sjanzou  do you have time to add this to your list? If not I can take a look next week

Addressed in ssc pull reqeust 672 https://github.com/NREL/ssc/pull/672

With test project file
SAM_727_test.zip

Closed with https://github.com/NREL/ssc/pull/672


Snippet Streamfield help text covers controls

In the snippet model:
body = StreamField([
        ('heading', blocks.CharBlock(classname="full title")),
        ('paragraph', blocks.RichTextBlock()),
        ('image', ImageChooserBlock()),
        ('table', TableBlock(icon="fa-table"))
    ], help_text='Enter the body content for the FAQ')

On the snippet page

Enter the body content for the FAQ
 blocks the delete icon for the first item.
Since the help text isn't displayed the trivial solution is to delete the help_text from the model, but that isn't ideal since the client needs the help text. Have navigated around this with custom CSS for this project, but it'd be good for help text to not break the controls :)
Note, the class being used on the snippet for help text is different to the class used on a normal page. On normal pages adding help text to the streamfield doesn't cover the controls (though of course the help text is still not visible).

A fairly straight forward fix is to add this css to the admin interface:
/* fix bug with streamfield help text: */
.object.stream-field .object-help {
  position: relative; /* counteracts the default absolute position */
}
.object.stream-field:hover .object-help {
  opacity: 1; /* makes the helptext show up when hovering the streamfield */
}

The only slight disadvantage with this fix is that it adds a bit extra space between the pink title bar and the stream field content.
Until this is fixed in wagtail you can fix it by overriding the wagtail admin base template by creating a template called <your-project-template-dir>/wagtailadmin/base.html like this:
{% extends "wagtailadmin/base.html" %}
{% block css %}
  {{ block.super }}

{% endblock %}

Fixed in #3168


Need to use a custom Snowpack config until a new @sveltejs/snowpack-config release

const path = require('path');
const pkg = require(path.join(process.cwd(), 'package.json'));

// Consult https://www.snowpack.dev to learn about these options
module.exports = {
    packageOptions: {
        // always include Svelte in your project
        knownEntrypoints: ['svelte'],
        // ignore `import fs from 'fs'` etc
        external: [...require('module').builtinModules, ...Object.keys(pkg.dependencies || {})]
    },
    plugins: [
        [
            '@snowpack/plugin-svelte',
            {
                compilerOptions: {
                    hydratable: true
                },
                configFilePath: 'svelte.config.cjs'
            }
        ],

        [
            '@snowpack/plugin-build-script',
            {
                cmd: "postcss",
                input: [".css", ".pcss"],
                output: [".css"],
            }
        ],

        '@snowpack/plugin-typescript'
    ],
    devOptions: {
        open: 'none',
        output: 'stream'
    },
    buildOptions: {
        sourcemap: true
    },
    mount: {
        '.svelte/assets': `/${process.env.SVELTE_KIT_APP_DIR}/assets`,
        'src/components': '/_components'
    },
    alias: {
        $app: './.svelte/assets/runtime/app',
        $components: './src/components'
    }
};

I'll try to get a new version released, so that you don't have to change anything here: https://github.com/sveltejs/kit/pull/370

Ok, it looks like 1.0.0-next.3 is out now. That should hopefully fix this<EMAIL_ADDRESS>


SwaggerUI shows optional enums as required

All versions of Swashbuckle.AspNetCore are affected including 5.2.0
public enum MyEnum {Val1, Val2}

[HttpGet("{myEnum?}")]
IActionResult Get([FromRoute]MyEnum? myEnum = null)
{
}

Swagger UI shows myEnum as required and there is no option to omit a value for it.
I have read all the issues related to nullable/optional enums but haven't found any working solution.

It's nothing to do with being an enum - it's because it's a path parameter.
This issue is actually a dup of #406 and #824. Long story short, the Swagger/OpenAPI specification DOES NOT allow optional route parameters and therefore they're NOT supported by Swashbuckle.

the Swagger/OpenAPI specification DOES NOT allow optional route parameters and therefore they're NOT supported by Swashbuckle.
Why it does not allow this when it's supported by webapi/aspnetcore?
Multiple actions with the same route aren't supported as well...
Too many limitations, which make usage of this lib in real applications useless...


Put the h2 first in .features > article > header

To make each article more semantic the h2 should be moved to the top in the header-element. This will help e.g people that uses screen readers to understand what each article is about.

Well it is true that the outline produced by a tool like HTML 5 Outliner is very confusing. See for yourself: http://gsnedders.html5.org/outliner/process.py?url=http%3A%2F%2Fhtml5please.us

closing this as a dupe of #105
thx!


Add lslaws command

About the PR

Adds the lslaws command for admins. The command lists the active laws of a borg.
Why / Balance

Currently the only ways to get borg laws from a borg as an admin is to either go VV diving, or use toolshed commands. The former is time consuming, and admins often lack the knowledge to do the latter while also being less convenient than a simple command.
This ideally would just be a verb that opens the borg's laws gui without allowing you to state them but this works for now. I'll probably make that in a time where I have more sleep. Even still I often find myself just typing out commands for verbs that give you a gui because it's usually faster so I think this will be useful even if a verb is added.
Media

[x] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Breaking changes

Changelog

:cl:
ADMIN:

add: The lslaws command was added and can be used to view the laws of a borg in the console.

Following maintainer feedback I'll just modify toolshed to just be able to take usernames and convert them to EntityUIDs instead of adding a new command.

Superseded by https://github.com/space-wizards/space-station-14/pull/26342 which allows the equivalent toolshed command to be conveniently used.
E.g.
> player:ent "Gamer" laws:get
If people want this I can just reopen the pr.


⚠ Spryr has degraded performance

In 6b52faf, Spryr (https://spryr.com) experienced degraded performance:

HTTP code: 200
Response time: 3266 ms

Resolved: Spryr performance has improved in a79cbfd after 10 minutes.


Use Check.name in repr

Previously Check.name could be set but was unused and the __repr__ fell back to the fn name, which isn't particularly useful for lambdas. This change uses the name argument in the repr, but keeps the same fallback, which has moved to __init__.

Codecov Report

Merging #265 into master will increase coverage by 0.15%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #265      +/-   ##
==========================================
+ Coverage   96.62%   96.78%   +0.15%
==========================================
  Files          15       15
  Lines        1332     1491     +159
==========================================
+ Hits         1287     1443     +156
- Misses         45       48       +3

Impacted Files
Coverage Δ

pandera/checks.py
98.59% <100.00%> (+0.20%)
:arrow_up:

pandera/decorators.py
90.80% <0.00%> (-1.74%)
:arrow_down:

pandera/errors.py
100.00% <0.00%> (ø)

pandera/schema_components.py
98.97% <0.00%> (+0.56%)
:arrow_up:

pandera/hypotheses.py
94.11% <0.00%> (+1.66%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c7dcef4...e5318b3. Read the comment docs.


Changed favicon, landing page image, navbar image

Description
Changed favicon, landing page image, navbar image
Fixes # (issue)
Type of change
Please delete options that are not relevant.

[x] New feature (non-breaking change which adds functionality)

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[x] Test A - chrome dev tools

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] My code has been reviewed by at least one peer
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

I like this PR.  It adds that nice "subtle" styling that a production site needs.  Great job.


[SPARK][CLIENT-JAVA] The Spark integration doesn't relocate all of its dependencies, this causes runtime errors when older versions (< 2.x) of org.yaml:snakeyaml are on the classpath

2024-03-26 11:22:04.767 ERROR org.apache.spark.util.Utils - throw uncaught fatal error in thread spark-listener-group-shared
java.lang.NoSuchMethodError: 'void org.yaml.snakeyaml.parser.ParserImpl.<init>(org.yaml.snakeyaml.reader.StreamReader, org.yaml.snakeyaml.LoaderOptions)'
	at io.openlineage.spark.shaded.com.fasterxml.jackson.dataformat.yaml.YAMLParser.<init>(YAMLParser.java:204) ~[openlineage-spark_2.13-1.10.2.jar:1.10.2]
	at io.openlineage.spark.shaded.com.fasterxml.jackson.dataformat.yaml.YAMLFactory._createParser(YAMLFactory.java:505) ~[openlineage-spark_2.13-1.10.2.jar:1.10.2]
	at io.openlineage.spark.shaded.com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:407) ~[openlineage-spark_2.13-1.10.2.jar:1.10.2]
	at io.openlineage.spark.shaded.com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:15) ~[openlineage-spark_2.13-1.10.2.jar:1.10.2]
	at io.openlineage.spark.shaded.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3809) ~[openlineage-spark_2.13-1.10.2.jar:1.10.2]
	at io.openlineage.client.OpenLineageClientUtils.deserializeInputStream(OpenLineageClientUtils.java:269) ~[openlineage-spark_2.13-1.10.2.jar:1.10.2]

I suggest that we produce a shaded JAR for the openlineage-java

I think our idea before was that as openlineage-java is used mostly in integrations, the particular integration would took care of shading itself.
The shading being done in client itself would make it harder to use some dependencies, which we actually transitively use but have to shade later - like Jackson


Feature Request - Fee in fiat

Like to see what my fee is in fiat when sending. Wonder if this could be incorporated, sats/vbyte etc don't really mean too much to me tbh.

Not sure what you mean - the fee is shown in fiat on the right of the fee field:

ahhh, hadn't put a receiving address in so didn't see it :)