Splitting "proxies/scanners" from BLE "trackers" and every other random BT device

When you go to configure>set devices, I get something like : Bermuda can currently see 31 active of 170 bluetooth devices, reported across 3 active of 4 bluetooth scanner devices. Life looks good.
With my Shelly BT nodes these were easy to find in the drop down list. Therefore easy to add (I have 3 BT capable Shellys) They were labelled as "Shelly xxxx.x.xx etc " in the list
But I just can't seem to see/find the 4th "scanner device" from all the other reported BT noise...
Is there any way in set up, to separate fixed BT scanner devices from all the other random BLE's that seem to come and go?
Thanks!

I can see how that would be a bit confusing. The drop-down list is for selecting "devices" that transmit advertisements, so you don't need to select your proxies/scanners in that list.
The Shelly devices are a bit different in that they are both proxies and "devices", because they also transmit advertisements. This is why Shellys show up in the "configure devices" drop-down.
Local bluetooth adaptors (hci0, hci1 etc) and esphome proxies won't show up in that list, because they don't usually transmit advertisements.
Bermuda will use all scanners by default, so there's no gui for choosing which scanners to use. This will change a bit shortly when we fix #45
So you don't need to specifically "enable" scanners, all you need to do is enable which "devices" you want Bermuda to set up sensors for. In the case of Shellys you can track them, but other scanners don't offer that (yet).
FYI, esphome just recently added the ability for sending advertisements while also acting as a proxy, so it will soon be able to do that too, but I haven't had a chance to do any testing yet to see if it works well (it might be tricky to get the balance right, and the esp is already pretty busy with proxy work), but we'll see. That shouldn't affect you for what you want to get going right now, though.
Does that make sense?

Sure that makes a lot of sense.
Ok how can I identify my 4th (hidden - ie non shelly) scanner? I have no idea what it is!
Second question are you developing a kind "of list" of compatible "devices"/device names? (Or can you point me to one?)

as mostly (other than ibeacon - which I assume are apple)  I have zero idea what they are...

Ok how can I identify my 4th (hidden - ie non shelly) scanner?

As a first step, you could use https://macvendors.com/ to look up the vendor of the chip.

Dude this is pretty darn cool!!!!!! Thanks for all the help setting it up!

How can I identify my 4th (hidden - ie non shelly) scanner?

On each device page, there's a show more button:

Each of the "distance to" entities point to a single scanner, which should help you identify what scanners bermuda is seeing. I'll think about exposing this somewhere else, too - since it does make it easier to work out what's going on if it just tells you :-)

a kind "of list" of compatible "devices"/device names? (Or can you point me to one?)

That's a really good question, partly because it's a difficult one to answer. So I've started a wiki page that tries to answer that :-) Thanks for the nudge!
https://github.com/agittins/bermuda/wiki/Supported-devices
Basically, the goal is to support tracking of any bluetooth device that sends advertisements, but this isn't always practical.

Dude this is pretty darn cool!!!!!! Thanks for all the help setting it up!

Awesome! There's more coolness on the way, too :-)

^ This is fantastic. It really is.  They are working really well with Bermuda and being able to be identified straight away once BT was turned on was the clinger for me. Being hardwired and fixed (I have switch relays and i4 buttons throughout) they become fixed point 'beacons". Testing now - the iphone "distance to Shelly" works really snappy- this is so so cool. So much potential  here. Congrats on the project. Can't wait to roll out some automations based off this.

Yeah, I didn't even realise they did that - I was writing Bermuda just thinking about esphome - but then all these people who'd bought Shelly devices essentially just magically got bluetooth trackers for free, it's awesome! :-)

Why do all the devices wander around different areas all the time

The Shelly devices will probably wander because even though they are "in their own area" so to speak, they are being read against the other areas, because the proxy doesn't read its "own" signal. If the other proxies are a reasonable distance away, then even small changes in signal strength can make a big difference in "estimated" distance, so it's easy for say, the "hallway" shelly to flip between "lounge" and "bedroom" if both are around 15 m away, since it doesn't take much signal variation for it to appear closer to one or the other.
Ultimately though, the "area" sensor of a shelly is a bit of a nonsense, since it is really always in it's own area, not in the area of some other proxy.
You'll probably see other devices flip around a bit between areas, this is because rssi is a pretty unreliable way to measure distances, but there are things you can do to limit it - check out this wiki page for a bit more info https://github.com/agittins/bermuda/wiki/Troubleshooting#my-device-switches-between-areas-every-few-seconds

^Ok that makes sense as my Shellys are 10-15M apart.
All good
I'll continue to test
Next I am going to try locate a casio BT watch
Thanks for the time answering the Q and setting up the wikis
Awesome project
Onwards and upwards!
Cheers

Hi again. Is there a lot of writing of files with Bermuda? Will this fill up my data? Will it wear out my SSD? thanks again


screen appears weird !!! xiaomi note 7

I have the Same issue .

@Abdelrhman22 do you also have a Xiaomi 7 or is this on another device?

I just picked up one of these devices myself, we'll see if I can reproduce this bug when it arrives.

Looks like this is a duplicate of #31 I'm going to close this one and consolidate the discussion there. Note: I'm still pretty new to Github, let me know if this isn't the typical protocol.

We are working on the same project , let me keep in contact with you


fixes #229 use Pytest for all tests

Pull Request
Description
Replace the unittest framework in the tests with pytest:

Removed the setUp method as it is not needed in pytest.

Replaced the self.assertEqual assertions with assert statements. In pytest, assertions can be directly written using the assert keyword.

Fixes #229

Thank you @jacobbieker. Yeap

@all-contributors add @norbline for code


Bug Fix regarding boundary attack

Signed-off-by: tagomaru<EMAIL_ADDRESS>Description
This PR is going to fix a kind of bug regarding boundary attack.
According to 2.2 of original paper, after first step, the image should be on the sphere. On the other hand, the image with current implementation is not there but on the on the tangent of current sample of the sphere. It means the image becomes far from the center of the sphere a little bit after first step.
This PR is going to let it on the sphere.
Type of change
Please check all relevant options.

[x] Bug fix (non-breaking)
[ ] New feature (non-breaking)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Testing
Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

[x] test_boundary.py

Test Configuration:

OS
Linux version 4.19.76-linuxkit (gcc version 8.3.0 (Alpine 8.3.0))
Python version
Python 3.6.9
ART version or commit number
1.2.0
TensorFlow

Checklist

[X] My code follows the style guidelines of this project
[X] I have performed a self-review of my own code
[ ] I have commented my code
[ ] I have made corresponding changes to the documentation
[X] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[X] New and existing unit tests pass locally with my changes

Hi @tagomaru Thank you very much for your pull request and fixing this, we will review it as soon as possible. I have changed the target branch to dev_1.3.0 which created a small merge conflict, could you please take a look at notebooks/adversarial_action_recognition.ipynb and accept the dev_1.3.0 branch version.

Hi @beat-buesser

could you please take a look at notebooks/adversarial_action_recognition.ipynb and accept the dev_1.3.0 branch version.

I checked the file and confirmed that the file was not related to my current PR. I think you can merge my PR after your review.
Do I have to do some more operation on GitHhub?

@tagomaru I think this could be caused by the current branch tagomaru:bug-fix-bounday-attack starting from branch master. I think the best way to resolve this is if you create a new local branch off an updated branch dev_1.3.0 and use git cherry-pick to transfer your commit da3f3f33c14fdcf424e3a4a950c0f7428538b226 to this new branch. From here you can either create a new pull request from this new branch and close this pull request or you can force push the new local branch to the remote branch used in this pull request using a command similar to git push origin local-name:remote-name

Hi @beat-buesser
Thank you for your advice.
I sent new PR with #426 and am going close this PR.


pipenv reports being unable to find any packages in PyPI

$ pipenv --version
pipenv, version 3.2.3

$ cat Pipfile
[[source]]
url = "https://pypi.org/"
verify_ssl = true

[requires]
os_name = "posix"
python_version = "3.5"

[dev-packages]
google-api-python-client = "*"
fakeredis = "==0.7"
mock = "==1.2.0"

[packages]
ua-parser = "==0.7.1"
fluent-logger = "==0.4.0"
gevent = "*"
isodate = "==0.5.4"
newrelic = "==<IP_ADDRESS>"
click = "==6.6"
redis = "==2.10.5"
numpy = "==1.11.1"
hiredis = "==0.2.0"
gunicorn = "==18.0"
simplejson = "==3.5.2"
Django = "==1.8"
maxminddb = "==1.2.0"
pytz = "==2016.6.1"
user-agents = "==1.0.1"
python-dateutil = "==2.4.2"
boto = "==2.42.0"
SPARQLWrapper = "==1.7.6"
geoip2 = "==2.2.0"
django-cors-headers = "==1.1.0"
nose = "==1.3.7"
httpagentparser = "==1.6.0"
requests = "==2.8.1"

$ pipenv install
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    load_entry_point('pipenv==3.2.3', 'console_scripts', 'pipenv')()
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/pipenv/cli.py", line 576, in install
    ensure_project(three=three)
  File "/usr/local/lib/python3.5/dist-packages/pipenv/cli.py", line 92, in ensure_project
    ensure_pipfile()
  File "/usr/local/lib/python3.5/dist-packages/pipenv/cli.py", line 69, in ensure_pipfile
    ensure_proper_casing()
  File "/usr/local/lib/python3.5/dist-packages/pipenv/cli.py", line 120, in ensure_proper_casing
    proper_case_section('packages')
  File "/usr/local/lib/python3.5/dist-packages/pipenv/cli.py", line 109, in proper_case_section
    new_casing = proper_case(dep)
  File "/usr/local/lib/python3.5/dist-packages/pipenv/cli.py", line 474, in proper_case
    raise IOError('Unable to find package {0} in PyPI repository.'.format(crayons.green(package_name)))
OSError: Unable to find package pytz in PyPI repository.

Running pipenv install again will report a different package not being found.
This worked fine before upgrading.  Did the Pipfile format change?

Hey @thirtyseven, it looks like the problem is actually in your Pipfile. https://pypi.org/ isn't a valid endpoint for retrieving packages, it needs to be something like https://pypi.org/simple or https://pypi.python.org/simple.
Is this a new Pipfile, or was this working prior to 3.2.3?

The Pipfile was created with an 0.2 release and definitely works if I test
with 0.2.8.
On Thu, Jan 26, 2017, 4:57 PM Nate Prewitt<EMAIL_ADDRESS>wrote:

Hey @thirtyseven https://github.com/thirtyseven, it looks like the
problem is actually in your Pipfile. https://pypi.org/ isn't a valid
endpoint for retrieving packages, it needs to be something like
https://pypi.org/simple or https://pypi.python.org/simple.
Is this a new Pipfile, or was this working prior to 3.2.3?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kennethreitz/pipenv/issues/116#issuecomment-275561416,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHjHmWTRt1WvdqY8kGBoOtm70Dtltwjks5rWUD0gaJpZM4LvU4i
.

Ok, I think I've got a handle on this. pipenv didn't use the source field until 0.2.6 which was after you generated your original file. When we started using the source field, we realized that the default configuration was wrong and updated it. The default since ~3.0.0 has been https://pypi.python.org/simple. It should be updated here and at the Pipfile repo.
If you update your URL to that, things should work as intended. I had a moment of hesitation because there is in fact a bug when using https://pypi.org/simple because the HTML returned is slightly different and tripping up our parser.
Let me know if you're still experiencing issues after updating the URL and running pipenv lock.

Confirming that updating the source fixes the issue.  I'm leaving the issue open though in case you want to handle this case for the users that started with the bad version.

@thirtyseven could you clarify what you'd like to see to address this?

I think the most helpful thing to do would be for pipenv to check if the source is exactly the string "https://pypi.org/", and if so display a warning message like: "Your Pipfile was created with a buggy version of pipenv, please change source.url to correct url".  Or it could just change the source itself.
I can write a PR implementing this if you'd like.  It's also quite possible that not enough people were affected by this to make it worth it.

I agree that adding a helpful message may be useful, as this bug was caused by this software.

I am not sure if this is related at all, but I am now unable to fetch packages from public git repos:
Traceback
pipenv install git+https://github.com/paradox41/heroprotocol.git#egg=heroprotocol
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pipenv/cli.py", line 594, in install
    ensure_project(three=three)
  File "/usr/local/lib/python2.7/site-packages/pipenv/cli.py", line 92, in ensure_project
    ensure_pipfile()
  File "/usr/local/lib/python2.7/site-packages/pipenv/cli.py", line 69, in ensure_pipfile
    ensure_proper_casing()
  File "/usr/local/lib/python2.7/site-packages/pipenv/cli.py", line 128, in ensure_proper_casing
    casing_changed = proper_case_section('packages')
  File "/usr/local/lib/python2.7/site-packages/pipenv/cli.py", line 111, in proper_case_section
    new_casing = proper_case(dep)
  File "/usr/local/lib/python2.7/site-packages/pipenv/cli.py", line 492, in proper_case
    raise IOError('Unable to find package {0} in PyPI repository.'.format(crayons.green(package_name)))
IOError: Unable to find package heroprotocol in PyPI repository.

Pipfile
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
flask-oauthlib = "==0.9.3"
py = "==1.4.32"
"delegator.py" = "==0.0.7"
configargparse = "==0.11.0"
pytest-cov = "==2.2.0"
graphene = "==1.1.3"
fixture = "==1.5.9"
dateutils = "==0.6.6"
ipython-genutils = "==0.1.0"
packaging = "==16.8"
ipython = "==5.1.0"
pydash = "==3.4.8"
pygments = "==2.2.0"
promise = "==1.0.1"
prompt-toolkit = "==1.0.9"
pyparsing = "==2.1.10"
click = "==6.7"
appdirs = "==1.4.0"
mako = "==1.0.6"
oauthlib = "==2.0.1"
pexpect = "==4.2.1"
amqp = "==2.1.4"
flask = "==0.12"
ipdb = "==0.10.1"
ujson = "==1.35"
flask-sqlalchemy = "==2.1"
redis = "==2.10.5"
ptyprocess = "==0.5.1"
kafka-python = "==1.3.2"
celery = "==4.0.2"
graphql-relay = "==0.4.5"
pytest = "==2.8.7"
flask-bcrypt = "==0.7.1"
kombu = "==4.0.2"
typing = "==<IP_ADDRESS>"
toml = "==0.9.2"
itsdangerous = "==0.24"
bcrypt = "==3.1.2"
uwsgi = "==2.0.14"
python-editor = "==1.0.3"
virtualenv = "==15.1.0"
crayons = "==0.1.2"
requirements-parser = "==0.1.0"
pathlib2 = "==2.2.1"
simplejson = "==3.10.0"
vine = "==1.1.3"
flask-login = "==0.4.0"
graphene-sqlalchemy = "==1.1.1"
markupsafe = "==0.23"
billiard = "==<IP_ADDRESS>"
colorama = "==0.3.7"
coverage = "==3.7.1"
flask-shell-ipython = "==0.2.2"
python-dateutil = "==2.6.0"
coveralls = "==1.1"
iso8601 = "==0.1.11"
pickleshare = "==0.7.4"
six = "==1.10.0"
decorator = "==4.0.11"
"backports.shutil-get-terminal-size" = "==1.0.0"
mpyq = "==0.2.5"
flask-script = "==2.0.5"
flask-graphql = "==1.4.0"
graphql-core = "==1.0.1"
pipenv = "==0.2.8"
enum34 = "==1.1.6"
pytz = "==2016.10"
sqlalchemy = "==1.1.5"
wcwidth = "==0.1.7"
jinja2 = "==2.9.4"
traitlets = "==4.3.1"
requests-oauthlib = "==0.7.0"
alembic = "==0.8.10"
pycparser = "==2.17"
psycopg2 = "==2.6.2"
singledispatch = "==<IP_ADDRESS>"
requests = "==2.13.0"
docopt = "==0.6.2"
simplegeneric = "==0.8.1"
cffi = "==1.9.1"
scandir = "==1.4"

[packages.heroprotocol]
git = "https://github.com/paradox41/heroprotocol.git"

[packages.resource_alchemy]
git = "https://github.com/tomneyland/resource-alchemy.git"

Pipefile.lock
{
    "default": {
        "Mako": "==1.0.6",
        "flask-oauthlib": "==0.9.3",
        "flask-sqlalchemy": "==2.1",
        "configargparse": "==0.11.0",
        "Flask": "==0.12",
        "cffi": "==1.9.1",
        "ipython-genutils": "==0.1.0",
        "packaging": "==16.8",
        "pygments": "==2.2.0",
        "promise": "==1.0.1",
        "Werkzeug": "==0.11.15",
        "prompt-toolkit": "==1.0.9",
        "pyparsing": "==2.1.10",
        "appdirs": "==1.4.0",
        "oauthlib": "==2.0.1",
        "Flask-Script": "==2.0.5",
        "pexpect": "==4.2.1",
        "flask": "==0.12",
        "ipdb": "==0.10.1",
        "py": "==1.4.32",
        "celery": "==4.0.2",
        "Jinja2": "==2.9.4",
        "python-editor": "==1.0.3",
        "typing": "==<IP_ADDRESS>",
        "itsdangerous": "==0.24",
        "appnope": "==0.1.0",
        "sqlalchemy": "==1.1.5",
        "pathlib2": "==2.2.1",
        "simplejson": "==3.10.0",
        "vine": "==1.1.3",
        "Flask-OAuthlib": "==0.9.3",
        "iso8601": "==0.1.11",
        "markupsafe": "==0.23",
        "billiard": "==<IP_ADDRESS>",
        "jinja2": "==2.9.4",
        "python-dateutil": "==2.6.0",
        "flask-graphql": "==1.4.0",
        "decorator": "==4.0.11",
        "graphql-core": "==1.0.1",
        "mpyq": "==0.2.5",
        "flask-script": "==2.0.5",
        "coveralls": "==1.1",
        "delegator.py": "==0.0.7",
        "Pygments": "==2.2.0",
        "requests-oauthlib": "==0.7.0",
        "uWSGI": "==2.0.14",
        "pycparser": "==2.17",
        "pydash": "==3.4.8",
        "resource-alchemy": "==0.1.0",
        "dateutils": "==0.6.6",
        "resource_alchemy": {
            "git": "https://github.com/tomneyland/resource-alchemy.git"
        },
        "graphene": "==1.1.3",
        "Flask-SQLAlchemy": "==2.1",
        "fixture": "==1.5.9",
        "SQLAlchemy": "==1.1.5",
        "singledispatch": "==<IP_ADDRESS>",
        "click": "==6.7",
        "mako": "==1.0.6",
        "amqp": "==2.1.4",
        "ujson": "==1.35",
        "six": "==1.10.0",
        "redis": "==2.10.5",
        "ptyprocess": "==0.5.1",
        "kafka-python": "==1.3.2",
        "graphql-relay": "==0.4.5",
        "pytest": "==2.8.7",
        "flask-bcrypt": "==0.7.1",
        "kombu": "==4.0.2",
        "Flask-Bcrypt": "==0.7.1",
        "bcrypt": "==3.1.2",
        "uwsgi": "==2.0.14",
        "heroprotocol": {
            "git": "https://github.com/paradox41/heroprotocol.git"
        },
        "virtualenv": "==15.1.0",
        "crayons": "==0.1.2",
        "requirements-parser": "==0.1.0",
        "traitlets": "==4.3.1",
        "flask-login": "==0.4.0",
        "graphene-sqlalchemy": "==1.1.1",
        "colorama": "==0.3.7",
        "coverage": "==3.7.1",
        "flask-shell-ipython": "==0.2.2",
        "ipython": "==5.1.0",
        "pickleshare": "==0.7.4",
        "Flask-Login": "==0.4.0",
        "Flask-GraphQL": "==1.4.0",
        "ConfigArgParse": "==0.11.0",
        "pipenv": "==0.2.8",
        "MarkupSafe": "==0.23",
        "enum34": "==1.1.6",
        "simplegeneric": "==0.8.1",
        "pytest-cov": "==2.2.0",
        "wcwidth": "==0.1.7",
        "pytz": "==2016.10",
        "toml": "==0.9.2",
        "psycopg2": "==2.6.2",
        "requests": "==2.13.0",
        "docopt": "==0.6.2",
        "backports.shutil-get-terminal-size": "==1.0.0",
        "alembic": "==0.8.10",
        "scandir": "==1.4"
    },
    "develop": {},
    "_meta": {
        "sources": [
            {
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ],
        "requires": {},
        "Pipfile-sha256": "16d57a0d06ac907d923c9b516310698779fab27ca9487b89de1c26d09fd1bd9e"
    }
}

This was working prior to v3.0.0 FWIW. I just tried installing again today, and it threw out this error.
If this isn't related, I don't mind opening another issue.
Thanks in advance.

@paradox41 this appears to be related to the way pipenv normalizes package names rather than being unable to find the repository. I would consider this a separate bug if you'd like to open a new ticket. Thanks!


Windows paths aren't handled correctly

aichat -r %functions% what time is it
Call get_current_time '{}'
Call get_current_time '{}':
    /bin/bash: C:UsersmeAppDataRoamingaichatfunctionsshget_current_time.sh: No such file or directory

It seems to try to run the file via the windows path, so the backslashes are replaced with nothing and the function doesn't work.

@lurebat
Did you setup accordingthe readme instructions?
Can you try running argc call get_current_time '{}'?

Running the command directly does work, so it seems that it's aichat who gets it wrong.
Maybe it's related to the symlinking?
Setup I made:
I cloned the repo, made functions.txt with the following content:
get_current_time.sh
get_current_weather.sh
get_web_page.sh
search_duckduckgo.sh
search_wolframalpha.sh
may_execute_py_code.py
may_execute_js_code.js

ran argc build and then argc install

aichat simply execute ./bin/get_current_time.cmd,  that's all.
It seems the bash path is incorrect,  #15  fixed it.
please sync the repo, run argc build, and try again.

Confirmed working


Wrong CARTO files

vtk files generated by the new version of MUSIC (since the new pipeline system) contain metaata.
CARTO Writer of the new MUSIC deals with this, CARTO Writer of MUSIC Viewer does not.
CARTO systems can't read the files containing metadata.

The solution I am working on right now aims at adding a new step to the VT pipeline in which the user does the export of the scene (it wasn't a step before) and an algo go through all the vtk files to remove their Field data.

Good idea. Can it be an Export button on the last displayResults step? You could add a button in medPipelineValuesToolBox maybe :)

By the way, it could be linked to this issue: https://github.com/Inria-Asclepios/music/issues/158

Trickier than expected.
An easy way to remove the FieldData is to use the VTK methods.
But you cannot access VTK methods in the core of MUSIC (the scene saving is in the core: medMainWindow, it's the small button at the right of the status bar).
So, my first idea (a prototype) was to duplicate the scene saving algo, put it in a step and remove the field data using VTK (I added a button in the Display Results step). It works well, but it's quite dirty, since code is duplicated.
If we want to avoid code duplication,
_ we can create a plugin dedicated to that, which could then use VTK methods, but then we say good bye to the button in the status bar.
_ we can keep the code in the core and use C++ methods to remove the lines between the key words FIELD Field Data and POINTS and hope that Points does not appear in the metaData (it's lame I know)
To sum up, the 3 options I propose are in bold.

I know it's easy, but it would mean that I would have to create new installers for MUSIC Viewers then ask the centres to update their MUSIC Viewer version, finally to deal with the few that didn't.
I talked with Maxime yesterday and we may have found a good way to deal with this, I'll tell you more about that at the meeting.


rust_xlsxwriter Roadmap

rust_xlsxwriter is a Rust library that can be used to write text, numbers, dates and formulas to multiple worksheets in an Excel 2007+ XLSX file.
It is a port of the XlsxWriter Python module by the same author.  I also actively maintain a C version libxlsxwriter and a Perl version Excel::Writer::XLSX. The Rust version will also try to address some limitations and frequently requested features of the previous versions, such as the separation of formatting and data writing.
The overall focus of rust_xlsxwriter is on performance, on testing, on documentation, and on fidelity with the file format created by Excel. Reading or modifying files is outside the scope of this library.
New features will be added in approximately this order:
Phase 1: Basic functionality

[x] mdBook documentation - Done: Working with rust_xlsxwriter
[x] Support for worksheets decoupled from workbook to avoid ownership issues with more than one worksheet. Done: Creating Worksheets
[x] Worksheet page setup functions. Done: Page Setup
[x] Freeze panes. Done: set_freeze_panes()
[x] Document properties/metadata. Done
[ ] Worksheet protection Example
[x] URL/Hyperlink support. Done: write_url()
[x] Merge Range support. Done: merge_range()
[x] Simulated autofit. Done: set_autofit()
[x] Tab manipulation. Working with worksheet tabs

Phase 2: Medium functionality

[x] Image support. Done: Image
[x]  Images in headers. Done: Headers and Footers
[ ] URLs in images.
[x] Autofit - simulated: Done
[x] Defined names. Done
[ ] Low memory mode. Example
[x] In memory mode. Done: workbook.save_to_buffer()
[ ] Tempfile directory support. Example
[x] Rich strings. Done
[x] Dynamic functions . Done
[ ] Features/Optional dependencies.
[ ] Format merging

Phase 3: Larger functionality

[x] Autofilters. Partially implemented
[ ] Charts. Example
[ ] Tables. Example
[ ] Conditional formatting. Example
[ ] Data Validation. Example
[ ] Other XlsxWriter features

Initial performance data:
$ hyperfine target/release/examples/app_perf_test ./c_perf_test "python py_perf_test.py"
Benchmark 1: target/release/examples/app_perf_test
  Time (mean ± σ):     447.2 ms ±   8.3 ms    [User: 402.8 ms, System: 39.4 ms]
  Range (min … max):   431.5 ms … 460.7 ms    10 runs

Benchmark 2: ./c_perf_test
  Time (mean ± σ):     362.5 ms ±   6.4 ms    [User: 305.2 ms, System: 53.0 ms]
  Range (min … max):   353.2 ms … 371.9 ms    10 runs

Benchmark 3: python py_perf_test.py
  Time (mean ± σ):      2.899 s ±  0.023 s    [User: 2.787 s, System: 0.088 s]
  Range (min … max):    2.868 s …  2.934 s    10 runs

Summary
  './c_perf_test' ran
    1.23 ± 0.03 times faster than 'target/release/examples/app_perf_test'
    8.00 ± 0.16 times faster than 'python py_perf_test.py'

Or in other words, the C version is the fastest and if we take that as 1 then the rust version is 1.2x (or 20%) slower and the Python version is 8x slower.
The Rust version is ~ 6.5 x faster than the Python version.

$ hyperfine target/release/examples/app_perf_test "python py_perf_test.py"
Benchmark 1: target/release/examples/app_perf_test
  Time (mean ± σ):     450.8 ms ±   5.5 ms    [User: 406.8 ms, System: 39.1 ms]
  Range (min … max):   443.5 ms … 459.1 ms    10 runs

Benchmark 2: python py_perf_test.py
  Time (mean ± σ):      2.942 s ±  0.040 s    [User: 2.821 s, System: 0.090 s]
  Range (min … max):    2.877 s …  3.014 s    10 runs

Summary
  'target/release/examples/app_perf_test' ran
    6.53 ± 0.12 times faster than 'python py_perf_test.py'

Is polars support planned? I saw the python xlsxwriter have pandas support, I wonder if the rust xlsxwriter have any plans to support polars.

I wonder if the rust xlsxwriter have any plans to support polars.

Good suggestion. That was something that I was thinking about. I wrote the initial xlsxwriter integration into Pandas. I'll have a look in their GitHub issues/requests and see if there is any planned work.

Chart support is the next major feature under development. This has a roadmap of it own since it is has a lot of properties and methods:

[ ] Initial support for Bar charts

[ ] Initial support for all other major chart types

[ ] Series option:

[ ] name: Set the name for the series.
[ ] line: Set the properties of the series line type such as color and width.
[ ] border: Set the border properties of the series such as color and style.
[ ] fill: Set the solid fill properties of the series such as color.
[ ] pattern: Set the pattern fill properties of the series.
[ ] gradient: Set the gradient fill properties of the series.
[ ] marker: Set the properties of the series marker such as style and color.
[ ] trendline: Set the properties of the series trendline.
[ ] smooth: Set the smooth property of a line series.
[ ] y_error_bars: Set vertical error bounds for a chart series.
[ ] x_error_bars: Set horizontal error bounds for a chart series.
[ ] data_labels: Set data labels for the series.
[ ] points: Set properties for individual points in a series.
[ ] invert_if_negative: Invert the fill color for negative values.
[ ] overlap: Set the overlap between series in a Bar/Column chart.
[ ] gap: Set the gap between series in a Bar/Column chart.
[ ] chart.set_y_axis()

[ ] chart.set_x2_axis()

[ ] chart.set_y2_axis()

[ ] chart.combine()

[ ] chart.set_size()

[ ] chart.set_title()

[ ] chart.set_legend()

[ ] chart.set_chartarea()

[ ] chart.set_plotarea()

[ ] chart.set_style()

[ ] chart.set_table()

[ ] chart.set_up_down_bars()

[ ] chart.set_drop_lines()

[ ] chart.set_high_low_lines()

[ ] chart.show_blanks_as()

[ ] chart.show_hidden_data()

[ ] chart.set_rotation()

[ ] chart.set_hole_size()

Is it possible to use rayon or std::thread::scope (parallelism) to do each worksheet and at the end add in the final workbook?

Is it possible to use rayon or std:🧵:scope (parallelism) to do each worksheet and at the end add in the final workbook?

It wouldn't be easy. I've thought a good bit about this in the past in relation to the other language version of the library. The main issue is that the xlsx file format has a lot of interlinked "relationships" stored in .rel files. Worksheet strings are also stored in a shared hash table and referenced by id. These, more or less, need to be worked out sequentially.
Not to say that it is impossible, nothing really is in software, just hard and potentially error prone.
Do you have a performance bottleneck or some use case that would benefit from parallelism or is it just a general question?

Just a general question to speed up printing excel files with parallelism.
Thanks!

@claudiofsr that is not a bad idea/implementation. The problem, as you probably figured out, is that threading would be most beneficial at the point where each worksheet is written but that is also the part where the interlocks happen. And again it wouldn't be impossible, just very tricky/not worth the effort.
The main use case where threading would be beneficial is with a medium to large number of large worksheets. However in most use cases people have a medium number of small worksheets or one large worksheet (although of course there are always exceptions). If the most common use case was large number of large worksheets then I would probably try to redesign to facilitate threading but I think the current design is probably the right balance of performance for most people and writability for me.
Finally, just on overall performance the latest version of rust_xlsxwriter is 10% faster that previous versions and also supports a new feature called zlib which uses the C zlib library to boost performance by about 1.6x and is even 1.3x faster than the native C version libxlsxwriter. See #23 (Ideas for performance) and the updated benchmarks at https://rustxlsxwriter.github.io/performance.html

Good morning, jmcnamara.
I just updated rust_xlsxwriter to version 0.34. I tested both libraries: rust_xlsxwriter and xlsxwriter. I noticed that xlsxwriter is twice as fast as rust_xlsxwriter.
Another question. Is it possible to change or add worksheet names without relying on the workbook?

Is it possible to change or add worksheet names without relying on the workbook?

Yes. There is a set_name() method .

I suggest something like this:
let worksheet = Worksheet::new().set_name("Name XYZ")?;
And then:
workbook.push_worksheet(worksheet);

@claudiofsr It is probably better to open a separate question for this.

Correcting my previous text.
I just updated rust_xlsxwriter to version 0.36.1.
I tested both libraries: rust_xlsxwriter and xlsxwriter.
Now, I noticed that xlsxwriter and rust_xlsxwriter are equivalent in terms of processing time!
Congratulations!

I would be really happy to see the tables functionality added

That will be the next major feature after I complete more of the chart feature.

@Christoph-AK see #41 for initial table support.

Hi.. Thanks for the library!! I was in need of an active excel writer library in rust for my new project and this project seems to be most active and promising.
My project may depend on a lot of existing excel templates. I understand from the readme that currently editing an existing excel file is not supported. Any chances that this could be added in the future?

Any chances that this could be added in the future?

Unfortunately no, I don’t plan to tackle reading or rewriting XLSX files.
The  XLSX structure is difficult to parse and rewrite for anything beyond basic data (and even that it can be hard for elements like dates).
Instead I’m going to concentrate my efforts to try give Rust a best in class XLSX writing library.
For additional context here is a reply that I gave to a similar request to the Python version of the library: https://github.com/jmcnamara/XlsxWriter/issues/653#issuecomment-607173901

I've added support for Conditional Formatting. See Working with Conditional Formats in the docs.

I'm looking for some usability feedback for an initial implementation of support for Serde serialization if anyone would like to try serializing data to an Excel worksheet using rust_xlsxwriter. Please comment in the other issue thread (#61).
The code is on main. You will need to enable it in a local project with the following or similar:
cargo add --git https://github.com/jmcnamara/rust_xlsxwriter.git -F serde

I have added support for Serde serialization in v0.57.0. See Working with Serde in the rust_xlsxwriter docs and the discussion thread #61.
Some additional serialisation features and helpers will be added in upcoming releases.

Do you have any plans regarding reaching version 1.0? You seem to make a new 0.x release every week or so. Many of them don't have any breaking changes, but still require me to manually bump the version in Cargo.toml to make sure that I don't miss any new bug fixes. It would be nice to actually utilize semantic versioning and indicate non-breaking releases with 1.x bumps. This way, my app would be able to depend on the major version and automatically receive library updates. Note that I'm not asking for any new stability commitments, you can release 2.0 as soon as you want to make a change. This model would already be more comfortable than a breaking bump on every release.
In theory, I assume that major numbers would also make the life easier for library authors who depend on rust_xlsxwriter. Currently, they have to either pin it to a very narrow minor version or define a range with an explicit upper bound and bump it every week.

Obligatory mention of https://github.com/obi1kenobi/cargo-semver-checks - might be worth including.

Do you have any plans regarding reaching version 1.0?

I plan to release a 1.x.x version once the feature set is ~ 100% of the Python feature set. Based on the task list above the current feature set is 25/33 features and based on ported test cases it is ~ 700/900. I would hope to get to 1.0.0 by the end of the year.

You seem to make a new 0.x release every week or so.

That will probably continue although through this year (with an upcoming pause of 1-2 months while I work on some of the other language libraries/features).

Many of them don't have any breaking changes, but still require me to manually bump the version in Cargo.toml to make sure that I don't miss any new bug fixes.

Yes. Some, or many, of those could be path level releases but most contain a reasonable level of new functionality.
The semver docs say:

Minor version Y (x.Y.z | x > 0) MUST be incremented if new, backward compatible functionality is introduced to the public API. It MUST be incremented if any public API functionality is marked as deprecated. It MAY be incremented if substantial new functionality or improvements are introduced within the private code. It MAY include patch level changes. Patch version MUST be reset to 0 when minor version is incremented.

I am usually in the "MAY" category and sometimes in the "MUST".

Note that I'm not asking for any new stability commitments, you can release 2.0 as soon as you want to make a change. This model would already be more comfortable than a breaking bump on every release.

I think I would end up incrementing a large number of major versions as well. I don't know if that would be better or worse for the end user.
Anyway, overall I think you (and others) will just need to bear with me for the next year or so. The downside is that there will be several more bumps in minor versions but the (hopefully) upside is that there will be new substantive features added on a regular basis.

Obligatory mention of https://github.com/obi1kenobi/cargo-semver-checks - might be worth including.

Thanks. I have tried it in the past and find it a bit hard to use/parse. No disrespect to the devs, it is clearly a good tool. I (personally) just haven't managed to put it to good use. I might try again to at least work out if some releases could be patch releases.

I have released rust_xlsxwriter v0.63.0 with support for embedding images into worksheets. See the Embedded Images example in the docs.
This can be useful if you are building up a spreadsheet of products with a column of images of each product. Embedded images move with the cell so they can be used in worksheet tables or data ranges that will be sorted or filtered.
This functionality is the equivalent of Excel's menu option to insert an image using the option to "Place in Cell" which is available in Excel 365 versions from 2023 onwards. I was a frequently requested feature for Excel and for the xlsxwriter variants.

I have released rust_xlsxwriter v0.64.0 with support for sparklines. See the Working with Sparklines section in the docs.
This is a somewhat niche feature and as far as I can tell not widely used in the Python version. However, it is needed for compatibility with Polars which is currently one of my priorities.

A +1 for low memory mode, in case such feedback is useful :)

A +1 for low memory mode, in case such feedback is useful :)

It is useful. :-)

I've added support for adding Excel data validations to worksheet cells in rust_xlsxwriter v0.70.0.
Data validation is a feature of Excel that allows you to restrict the data that a user enters in a cell and to display associated help and warning messages. It also allows you to restrict input to values in a dropdown list.
Here is an example:
use rust_xlsxwriter::{DataValidation, DataValidationRule, Workbook, XlsxError};

fn main() -> Result<(), XlsxError> {
    // Create a new Excel file object.
    let mut workbook = Workbook::new();
    let worksheet = workbook.add_worksheet();

    worksheet.write(1, 0, "Enter rating in cell D2:")?;

    let data_validation = DataValidation::new()
        .allow_whole_number(DataValidationRule::Between(1, 5))
        .set_input_title("Enter a star rating!")?
        .set_input_message("Enter rating 1-5.\nWhole numbers only.")?
        .set_error_title("Value outside allowed range")?
        .set_error_message("The input value must be an integer in the range 1-5.")?;

    worksheet.add_data_validation(1, 3, 1, 3, &data_validation)?;

    // Save the file.
    workbook.save("data_validation.xlsx")?;

    Ok(())
}

Output:

See DataValidation for details.

I've added support for adding VBA Macros to rust_xlsxwriter using files extracted from Excel files. This isn't very useful and it is also a little kludgy but it is a reasonably popular feature of the Python library and it has utility in some circumstance.
Also, this lays some of the groundwork for adding cell comments (now called Notes by Excel).
Explanation
An Excel xlsm file is structurally the same as an xlsx file except that it contains an additional vbaProject.bin binary file containing VBA functions and/or macros.
Unlike other components of an xlsx/xlsm file this data isn't stored in an XML format. Instead the functions and macros as stored as a pre-parsed binary format. As such it wouldn't be feasible to programmatically define macros and create a vbaProject.bin file from scratch (at least not in the remaining lifespan and interest levels of the author).
Instead, as a workaround, the Rust vba_extract utility is used to extract vbaProject.bin files from existing xlsm files which can then be added to rust_xlsxwriter files.
See Working with VBA Macros.
Here is an example:
use rust_xlsxwriter::{Button, Workbook, XlsxError};

fn main() -> Result<(), XlsxError> {
    // Create a new Excel file object.
    let mut workbook = Workbook::new();

    // Add the VBA macro file.
    workbook.add_vba_project("examples/vbaProject.bin")?;

    // Add a worksheet and some text.
    let worksheet = workbook.add_worksheet();

    // Widen the first column for clarity.
    worksheet.set_column_width(0, 30)?;

    worksheet.write(2, 0, "Press the button to say hello:")?;

    // Add a button tied to a macro in the VBA project.
    let button = Button::new()
        .set_caption("Press Me")
        .set_macro("say_hello")
        .set_width(80)
        .set_height(30);

    worksheet.insert_button(2, 1, &button)?;

    // Save the file to disk. Note the `.xlsm` extension. This is required by
    // Excel or it raise a warning.
    workbook.save("macros.xlsm")?;

    Ok(())
}

Output:

I've dded support for cell Notes (previously called Comments) in v0.72.0.
See https://docs.rs/rust_xlsxwriter/latest/rust_xlsxwriter/struct.Note.html
Here is an example:
use rust_xlsxwriter::{Note, Workbook, XlsxError};

fn main() -> Result<(), XlsxError> {
    // Create a new Excel file object.
    let mut workbook = Workbook::new();

    // Add a worksheet to the workbook.
    let worksheet = workbook.add_worksheet();

    // Widen the first column for clarity.
    worksheet.set_column_width(0, 16)?;

    // Write some data.
    let party_items = [
        "Invitations",
        "Doors",
        "Flowers",
        "Champagne",
        "Menu",
        "Peter",
    ];
    worksheet.write_column(0, 0, party_items)?;

    // Create a new worksheet Note.
    let note = Note::new("I will get the flowers myself").set_author("Clarissa Dalloway");

    // Add the note to a cell.
    worksheet.insert_note(2, 0, &note)?;

    // Save the file to disk.
    workbook.save("notes.xlsx")?;

    Ok(())
}

And the output:

I didn't port some of the Python Note/Comment features such as note positioning since they weren't widely used and Excel's implementation tends to surprise people. If people ask of them I'll add them. The infrastructure is already in place.
Note, in versions of Excel prior to Office 365 Notes were referred to as "Comments". The name Comment is now used for a newer style threaded comment and Note is used for the older non threaded version. The newer Threaded Comments are unlikely to be added to rust_xlsxwriter due to fact that it relies on company specific metadata to identify the comment author.
As an aside the internal traits that I had put in place for other worksheet objects (images, charts, buttons) made this feature relatively easy to add. I really like this aspect of Rust where some of the abstractions can give very clean and easy to maintain/refactor code. Overall I really enjoy using Rust as a language.

I've uploaded version v0.74.0 of rust_xlsxwriter which adds methods to format cells separately from the data writing functions.
In Excel the data in a worksheet cell is comprised of a type, a value and a format. When using rust_xlsxwriter the type is inferred and the value and format are generally written at the same time using methods like Worksheet::write_with_format().
However, if required you can now write the data separately and then add the format using the new methods like Worksheet::set_cell_format(),Worksheet::set_range_format() and Worksheet::set_range_format_with_border().
For example you can now create border formatting like this with a single method call:

This has always been a heavily requested feature in the Python version but due to some different design decisions it was never easy to implement.

I've released v0.75.0 of rust_xlsxwriter which removes  the dependency on the regex.rs crate for smaller binary sizes. The only non-optional dependency is now zip.rs.
An example of the size difference is shown below for one of the sample apps:

app_hello_world
v0.74.0
v0.75.0

Debug
9.2M
4.2M

Release
3.4M
1.6M

See the discussion at #108

Version v0.76.0 of rust_xlsxwriter is out with added support for adding Textbox shapes to worksheets.

See the documentation for Shape.
This is the last of the "Larger functionality" features. We are probably on track to be feature complete with the Python version by the end of the year.

Folks, I am looking for some input on "constant_memory" mode for rust_xlsxwriter: https://github.com/jmcnamara/rust_xlsxwriter/issues/111

Released version v0.77.0 of rust_xlsxwriter with support for Chartsheets.
A Chartsheet in Excel is a specialized type of worksheet that doesn't have cells but instead is used to display a single chart. It supports worksheet display options such as headers and footers, margins, tab selection and print properties.

Chartsheets aren't widely used these days (as far as I can see) but end users sometimes request this feature.

I have an initial working version of the "constant memory" mode on the constant_memory branch (see #111 ). It currently has limited functionality but there is enough to allow me to benchmark the potential savings. I'm reposting the results here for a wider audience and hopefully some feedback/testing.
The memory usage profile is effectively flat (as designed):

Cells
Standard - Size (MB)
Constant Memory - Size (MB)
Standard - Time (s)
Constant Memory - Time (s)

100,000
16.213
0.021
0.101
0.088

200,000
32.405
0.021
0.214
0.179

300,000
52.794
0.021
0.335
0.276

400,000
64.793
0.021
0.443
0.369

500,000
76.792
0.021
0.564
0.468

600,000
105.569
0.021
0.673
0.564

700,000
117.567
0.021
0.768
0.669

800,000
129.567
0.021
0.874
0.799

900,000
141.566
0.021
1.002
0.862

1,000,000
153.565
0.021
1.081
1.022

Which looks like this:

Similarly to the Python version the performance is also slightly better (5-15%) in this mode. Lower time is better. However for numeric only/heavy data (which is the case in practice) the performance is more of less the same.

The tests were run like this:
./target/release/examples/app_memory_test 4000
./target/release/examples/app_memory_test 4000 --constant-memory

So the initial results are good. I'll continue with the functionality.
It would be good to have a few other eyes on this in #111.

How about the serde sub-struct flattening support?

How about the serde sub-struct flattening support?

@hackers267 It is probably worth opening a feature request for that.
The main problem is how would a sub-structure be flattened in the 2D cell matrix of a worksheet. The obvious approach would be to flatten the sub-structure into a a string in a cell, but that might not be a great solution for some (most?) users.
So it is probably best to kick off a Feature Request issue with an example of your data structure and also an Excel example of what you think the output should look like. Anyone else who is facing this issue can then add their opinion/suggestion.

I have an initial working version of the "constant memory" mode on the constant_memory branch (see #111 ).

Wow, this looks incredibly promising. Are there reasons not to use this as default, or maybe switch to it intelligently if only supported features are used?

How about the serde sub-struct flattening support?

@hackers267 It is probably worth opening a feature request for that.
The main problem is how would a sub-structure be flattened in the 2D cell matrix of a worksheet. The obvious approach would be to flatten the sub-structure into a a string in a cell, but that might not be a great solution for some (most?) users.
So it is probably best to kick off a Feature Request issue with an example of your data structure and also an Excel example of what you think the output should look like. Anyone else who is facing this issue can then add their opinion/suggestion.

@jmcnamara  I'm dealing with a scenario where in the generated excel, there are some fixed columns like name, age, gender, etc. and some unfixed columns like 2022 results, 2023 results, January 2024 results... Until results of the month. In this, the column that is fixed can be represented by the field of Struct, while the column that changes can be represented by the sub-struct of the flatten of serde, which can be of HashMap type. I think this is one of the application scenarios for sub-structure.

Wow, this looks incredibly promising. Are there reasons not to use this as default, or maybe switch to it intelligently if only supported features are used?

@Christoph-AK There are a number of restrictions around "constant memory" mode that don't make it suitable for all use case. Such as:

Data needs to be written in strict row/column order. Once row n has been written it is no longer possible to write to any row that is n - 1. This is generally okay when you are converting large datasets that are structured in that way but even then they are often structured based on columns (like Pandas or Polars).
Some Excel functionality like merged ranges, tables and array formulas write across multiple rows in one go and therefore they move the row number forward and don't allow the user to update previous rows. I will fix this in the upstreamed version using a lookahead buffer for the merge ranges and array formula cases. Tables may take a bit longer to resolve since it contains more edges cases.
Embedded (but not inserted) images need to adhere to the row by row order.
The "constant memory" mode works by writing data to disk instead of keeping it in memory and this means that each worksheet in constant memory mode consumes a filehandle. This has been been an issue with users of the Python/C versions who wanted to create workbooks with thousands of worksheets and ran into the ulimit for open files. Not a common occurrence but still one that happened enough for it to be reported.
On systems where the disk is loaded in memory there isn't any memory saving. :-) This has also been reported a few times.

So in general there are a few too many gotchas to turn it on as the default. In fact it will probably be behind a feature flag since it will require an additional dependency on the tempfile crate.
However, for use cases where it makes sense it will be a big efficiency gain. Also, it is possible to turn it on on a worksheet by worksheet basis so users can mix and match worksheets are required.

I'm dealing with a scenario where in the generated excel, there are some fixed columns like name, age, gender, etc. and some unfixed columns like 2022 results, 2023 results, January 2024 results... Until results of the month. In this, the column that is fixed can be represented by the field of Struct, while the column that changes can be represented by the sub-struct of the flatten of serde, which can be of HashMap type

@hackers267 Thanks for the description. The HashMap type at least may be feasible to support. Open a feature request and put in a sample struct and some example data and I will look at what can be done.

Released v0.78.0 with support for "constant memory" mode to reduce memory usage when writing large worksheets.
The constant_memory mode works by flushing the current row of data to disk when the user writes to a new row of data. This limits the overhead to one row of data stored in memory. Once this happens it is no longer possible to write to a previous row since the data in the Excel file must be in row order. As such this imposes the limitation of having to structure your code to write in row by row order. The benefit is that the required memory usage is very low,and effectively constant, regardless of the amount of data written.

@jpmckinney You asked about this a while back. If you are still interested you can check it out.

I've released v0.79.0 of rust_xlsxwriter with support for files larger than 4GB.
The rust_xlsxwriter library uses the zip.rs crate to provide the zip container for the xlsx file that it generates. The size limit for a standard zip file is 4GB for the overall container or for any of the uncompressed files within it.  Anything greater than that requires ZIP64 support. In practice this would apply to worksheets with approximately 150 million cells, or more.
See Workbook::use_zip_large_file().

For anyone who is interested I wanted to highlight a new wasm wrapper for rust_xlsxwriter that looks fairly complete even in the initial stages: https://github.com/estie-inc/wasm-xlsxwriter

Hi @jmcnamara,
I currently use nodejs/exceljs to write very large xlsx files. I'm looking into potentially faster alternatives, including this lib.
I have created a quick benchmark comparing different language/lib combinations  https://github.com/brunoargolo/excel-writing-multilanguage-benchmark
I'm a 100% beginner in Rust, GO and Python. So I might have royally fudged something.
I was hoping Rust would prove significantly faster than node but its pretty much a tie.
If you find any mistakes in my approach please let me know, I tried to make the repo as easy to reproduce as possible so anyone can test it.
Node and Go are using the streaming function of their respective libs, on python I used the constant memory mode on rust I tried both low_memory and constant memory with similar results.

Language + Lib
1 sheet
9 sheets

Nodejs with exceljs
5s
41s

Go with excelize
5s
30s

Python3 with xlsxwriter
26s
236s

Rust with rust_xlsxwriter
5s
40s

I tried to make the repo as easy to reproduce as possible so anyone can test it.

Thanks. That is helpful.
Overall, I don't see anything wrong with the benchmarks although the rust version does a bit too much type conversion. A real world example would probably use serde_json, or similar, and do the type conversion as part of the read step. However, this adds at most a 10% overhead to the write phase. Also, in this particular case the add_worksheet_with_constant_memory() method should be 5-10% faster.
I also crosschecked with the examples/app_perf_test.rs program for a similar number of cells and the runtime was approximately the same.
So overall, with some fine tuning and you could get 1.1-1.2 times better performance but that would still be in the same ballpark of the nodejs/exceljs solution.

I really appreciate you taking the time to go over this, thank you!
My guess is we're probably hitting the limits of what is viable in any language... I imagine we're not going to see 2x or 3x faster no matter the language/lib combination.
The one thing that might be significantly faster could be writing to XLSB format, but it is probably a ridiculous amount of work...
Do you still have any major performance related items on the roadmap or do you think this is mostly as good as it gets plus or minus a few percentage points?

My guess is we're probably hitting the limits of what is viable in any language... I imagine we're not going to see 2x or 3x faster no matter the language/lib combination.

Agreed. The bottlenecks come down to IO/writing the worksheet.xml files and zipping them and other supporting files into a zip container. So most implementations will probably hit the same performance floor. Notwithstanding that I'm impressed with the ExcelJS performance.

Do you still have any major performance related items on the roadmap or do you think this is mostly as good as it gets plus or minus a few percentage points?

There may be some small (< 10%) IO/writing gains still to be made but most of the low hanging fruit have  been picked. A potentially bigger gain would probably come from a threaded/parallelized zip library.
The C library gets a 20-30% performance improvement from using the Milo Yip DTOA library library for float to string conversion but I didn't get a similar speed increase from the ryu library except in very large files.
It is also possible that a properly parallelized backend worksheet assembly using rayon or similar might improve multiple worksheet writing. I have thread::scope parallelization in the backend but someone with more multithreaded rust experience might be able to do better:
https://github.com/jmcnamara/rust_xlsxwriter/blob/main/src/packager.rs#L122-L135
But to take a step back it is worth noting that the Rust version is at almost 90% of the C library performance so any 2-3x gain is unlikely without a radically different approach.

The one thing that might be significantly faster could be writing to XLSB format, but it is probably a ridiculous amount of work...

Yes. I maintained a library for the similar XLS format and it wasn't fun to maintain/extend. So for me that would be a non-runner. :-)

I wonder if a different allocator can give more performance in this benchmark.
Combining all approaches (allocator, faster serialisation and floats, better parallelization) would probably approach the go time.
I think garbage collected languages like go and js have an advantage vs default rust here as the gc can easier reuse allocated memory, and go does have an advantage when it comes to easily throwing multithreading at everything.
Of cause all of those characteristics can be programmed in rust with some care, so there shouldn't really be a language other then c thats inherently faster.

I've released v0.80.0 or rust_xlsxwriter with an improvement/fix for heap memory usage in constant_memory mode. This is a recommended upgrade for anyone using that mode/feature. See #120 for more information.


HVI Inspection per Vehicle

When submitting a connector, please make sure that you follow the requirements below, otherwise your PR might be rejected. We want to make you have a well-built connector, a smooth certification experience, and your users are happy :)
If this is your first time submitting to GitHub and you need some help, please sign up for this session.

[x] I attest that the connector doesn't exist on the Power Platform today. I've verified by checking the pull requests in GitHub and by searching for the connector on the platform or in the documentation.
[x] I attest that the connector works and I verified by deploying and testing all the operations.
[x] I attest that I have added detailed descriptions for all operations and parameters in the swagger file.
[x] I attest that I have added response schemas to my actions, unless the response schema is dynamic.
[x] I validated the swagger file, apiDefinition.swagger.json, by running paconn validate command.
[x] If this is a certified connector, I confirm that apiProperties.json has a valid brand color and doesn't use an invalid brand color, #007ee5 or #ffffff. If this is an independent publisher connector, I confirm that I am not submitting a connector icon.

If you are an Independent Publisher, you must also attest to the following to ensure a smooth publishing process:

[x] I have named this PR after the pattern of "Connector Name (Independent Publisher)" ex: HubSpot Marketing (Independent Publisher)
[ ] Within this PR markdown file, I have pasted screenshots that show: 3 unique operations (actions/triggers) working within a Flow. This can be in one flow or part of multiple flows. For each one of those flows, I have pasted in screenshots of the Flow succeeding.
[ ] Within this PR markdown file, I have pasted in a screenshot from the Test operations section within the Custom Connector UI.
[ ] If the connector uses OAuth, I have provided detailed steps on how to create an app in the readme.md.

Please validate the connector

@microsoft-github-policy-service agree company="JRS Innovation"

Hy Amjed,
please review it i have added a new folder with name HVI Inspection per Vehicle

@Amjed-Ayoub  Please verify it

Hi, could you please provide an update. I have to report back the reason for delay in publishing the connector. Let me know if a meeting/ teams meeting can speedup this process?

@Amjed-Ayoub you can check i have validated the swagger file then also it is showing failing in git.
Please review it and reply back.

@Amjed-Ayoub  can you please let me know why the validation is failing in git as i have validated it through paconn validate command and there it was successfully validated. Please Check and let me know
paconn validate Image Link:-
https://user-images.githubusercontent.com/130588212/255330457-5aeb133f-d8f1-449b-80ba-9bdd32d142f0.png


OpenBLAS dev is not working on E7 Haswell & Broadwell CPUs

Dear Xianyi,
Do you know why I can not complie OpenBLAS dev"gmake TARGET=HASWELL", but worked with "gamke"? On OpenBLAS 0.2.18 show different message.
Thanks.
John Centeno
gmake TARGET=HASWELL(OpenBLAS dev)
../kernel/x86_64/dtrmm_kernel_4x8_haswell.c:77: Error: no such instruction: vpe                                                                                rmpd $0x1b,%ymm3,%ymm3' ../kernel/x86_64/dtrmm_kernel_4x8_haswell.c:78: Error: no such instruction: vpe                                                                                rmpd $0xb1,%ymm2,%ymm2'
../kernel/x86_64/dtrmm_kernel_4x8_haswell.c:79: Error: no such instruction: vpe                                                                                rmpd $0xb1,%ymm3,%ymm3' gmake[1]: *** [dtrmm_kernel_LN.o] Error 1 gmake[1]: Leaving directory /root/OpenBLAS/kernel'
gmake: *** [libs] Error 1
gmake TARGET=HASWELL(OpenBLAS 0.2.18)
gmake
edhat-linux/4.4.7/../../..  -lgfortran -lm -lgfortran -lm -lc   && echo OK.
OK.
rm -f linktest
gmake[1]: Leaving directory `/root/OpenBLAS/exports'
OpenBLAS build complete. (BLAS CBLAS LAPACK LAPACKE)
OS               ... Linux
Architecture     ... x86_64
BINARY           ... 64bit
C compiler       ... GCC  (command line : gcc)
Fortran compiler ... GFORTRAN  (command line : gfortran)
Library Name     ... libopenblas_sandybridgep-r0.2.19.dev.a (Multi threaded; Max num-threads is 60)
To install the library, you can run "make PREFIX=/path/to/your/installation install".

Please read FAQ item about enterprise linux 6. Your assembler does not support AVX2.
If you cannot acquire functional binutils you can disable AVX2 for build

Dear Andrews,
Thanks, but GCC in RHEL 6.6 is 4.7 and different parameters show different result.  (BLAS CBLAS LAPACK LAPACKE)
OS               ... Linux
Architecture     ... x86_64
BINARY           ... 64bit
C compiler       ... GCC  (command line : gcc)
Fortran compiler ... GFORTRAN  (command line : gfortran)
Library Name     ... libopenblas_nehalemp-r0.2.18.a (Multi threaded; Max num-t                                                                                hreads is 60)
gmake(Sandybridge)

Direct link to fix your system:
https://github.com/xianyi/OpenBLAS/wiki/faq#binutils

Dear Andrew,
Thanks. I tried both methods, but I can not install devtools-3 or devtools -4 64bit on my Centos 7 or RHE6 64bit. I only install devtools-2 successfully before on Scientific Linux when I want to find a way to upgrade GCC 4.4.7.
Thanks.
John Centos
1. Install a package with repository for your system:
On CentOS, install package centos-release-scl available in CentOS repository:
$ sudo yum install centos-release-scl
On RHEL, enable RHSCL repository for you system:
$ sudo yum-config-manager --enable rhel-server-rhscl-7-rpms
2. Install the collection:
$ sudo yum install devtoolset-3
3. Start using software collections:
$ scl enable devtoolset-3 bash

You are expected to follow instructions on linked page to the last letter
# yum install centos-release-scl->>>rh<<<

Dear Andrew,
Thanks. I tried, but it sill not working.
John Centeno
No package devtoolset-4 available.
Error: Nothing to do
[root@localhost ~]# sudo yum install devtoolset-3
Loaded plugins: product-id, refresh-packagekit, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use su                                                                                bscription-manager to register.
Setting up Install Process
Repository rhscl-centos-release-scl is listed more than once in the configuratio                                                                                n
No package devtoolset-3 available.
Error: Nothing to do
[root@localhost ~]# sudo yum install devtoolset-2
Loaded plugins: product-id, refresh-packagekit, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use su                                                                                bscription-manager to register.
Setting up Install Process
Repository rhscl-centos-release-scl is listed more than once in the configuratio                                                                                n
No package devtoolset-2 available.
Error: Nothing to do
[root@localhost ~]# sudo yum-config-manager --enable rhel-server-rhscl-7-rpms
Loaded plugins: product-id, refresh-packagekit
Repository rhscl-centos-release-scl is listed more than once in the configuratio                                                                                n
[root@localhost ~]# sudo yum-config-manager --enable rhel-server-rhscl-7-rpms
Loaded plugins: product-id, refresh-packagekit
Repository rhscl-centos-release-scl is listed more than once in the configuratio                                                                                n
[root@localhost ~]# sudo yum install devtoolset-3
Loaded plugins: product-id, refresh-packagekit, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use su                                                                                bscription-manager to register.
Setting up Install Process
Repository rhscl-centos-release-scl is listed more than once in the configuratio                                                                                n
No package devtoolset-3 available.

First refresh repo files
yum upgrade centos-release
Install brand new scl repo
yum install centos-release-scl-rh
Install just binutils (will not hurt if you install entire gcc5 but it is not reqired for AVX2 support)
yum install devtoolset-4-binutils

Dear Andrew,
Thanks, but I still got error messages.
With Best Regards.
John Centeno
Installation
To set up the repository, install the webtatic-release RPM:
Webtatic EL7 for CentOS/RHEL 7:
rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
warning: /var/tmp/rpm-tmp.pVO7yd: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
Preparing...                          ################################# [100%]
package epel-release-7-6.noarch is already installed
[root@localhost ~]#
rpm -Uvh https://mirror.webtatic.com/yum/el7/webtatic-release.rpm
https://access.redhat.com/articles/1320623
If above article doesn't help to resolve this issue please open a ticket with Re                                                                                d Hat Support.
One of the configured repositories failed (testing 2 devtools for CentOS 7Serve                                                                                r),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:
 1. Contact the upstream for the repository and get them to fix the problem.

 2. Reconfigure the baseurl/etc. for the repository, to point to a working
    upstream. This is most often useful if you are using a newer
    distribution release than is supported by the repository (and the
    packages for the previous distribution release still work).

 3. Disable the repository, so yum won't use it by default. Yum will then
    just ignore the repository until you permanently enable it again or use
    --enablerepo for temporary usage:

        yum-config-manager --disable testing-devtools-2-centos-7Server

 4. Configure the failing repository to be skipped, if it is unavailable.
    Note that yum will try to contact the repo. when it runs most commands,
    so will have to try and fail each time (and thus. yum will be be much
    slower). If it is a very temporary problem though, this is often a nice
    compromise:

        yum-config-manager --save --setopt=testing-devtools-2-centos-7Server                                                                                .skip_if_unavailable=true

failure: repodata/repomd.xml from testing-devtools-2-centos-7Server: [Errno 256]                                                                                No more mirrors to try.
http://people.centos.org/tru/devtools-2/7Server/x86_64/RPMS/repodata/repomd.xml:                                                                                [Errno 14] HTTP Error 404 - Not Found

CentOS 7 will compile OpenBLAS Haswell kernels out of the box.
You need ONE SINGLE additional package to build them on CentOS 6
I dont understand where you dig up all those broken repositories. Please clean them from your system and try to repat FAQ entry to the letter on a system where it applies.

Dear Andrew,
Thanks. On Centos 7, I have to install GCC 4.8 first and it showed Haswell CPUs. But I can not run "Yum Install development tool". I have to run something to execute "Yum install". By the way, when it will support Broadwell CPUs. My CPUs on this system is E7-4880 V2, but I have the other system is E7-8855 V4.
With Best Regards.
John Centono
251  yum-config-manager --disable RHEL_7_Disc
252  yum install "Development Tools"
253  yum-config-manager --disable testing-1.1-devtools-7Server
254  yum install "Development Tools"
255  yum-config-manager --disable testing-devtools-2-centos-7Server
256  yum install "Development Tools"
257  yum install centos-release-scl
258  sudo yum-config-manager --enable rhel-server-rhscl-7-rpms
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
OpenBLAS build complete. (BLAS CBLAS)
OS               ... Linux
Architecture     ... x86_64
BINARY           ... 64bit
C compiler       ... GCC  (command line : gcc)
Library Name     ... libopenblas_haswellp-r0.2.18.a (Multi threaded; Max num-threads is 80)
To install the library, you can run "make PREFIX=/path/to/your/installation install".

Dear Andrew,
I still got error message when I compile hpl-2.2. Below are my procedure. Thanks again for your patience and time.
With Best Regards.
John Centos
Partil content of MAKE_LINUX_PII_CBLAS
LAdir        = $(HOME)/OpenBLAS
LAinc        =
LAlib        = $(LAdir)/libopenblas.a
OpenBLAS
gmake NO_AVX=1
make PREFIX=/rootOpenBLAS install
MPICH
./configure --prefix=/root/root/mpich
HPL
make arch=Linux_PII_CBLAS
make[2]: Entering directory `/root/hpl-2.2/testing/ptest/Linux_PII_CBLAS'
/root/mpich/bin/mpif77  -DHPL_CALL_CBLAS -I/root/hpl-2.2/include -I/root/hpl-2.2                                                                                /include/Linux_PII_CBLAS  -I/root/mpich/include -fomit-frame-pointer -O3 -funrol                                                                                l-loops -o /root/hpl-2.2/bin/Linux_PII_CBLAS/xhpl HPL_pddriver.o         HPL_pdi                                                                                nfo.o           HPL_pdtest.o /root/hpl-2.2/lib/Linux_PII_CBLAS/libhpl.a  /root/O                                                                                penBLAS/libopenblas.a /root/mpich/lib/libmpich.so
/usr/bin/ld: /root/OpenBLAS/libopenblas.a(blas_server.o): undefined reference to                                                                                symbol 'pthread_create@@GLIBC_2.2.5'
/usr/bin/ld: note: 'pthread_create@@GLIBC_2.2.5' is defined in DSO /lib64
/libpth                                                                                read.so.0 so try adding it to the linker command line
/lib64/libpthread.so.0: could not read symbols: Invalid operation
collect2: error: ld returned 1 exit status
make[2]: *** [dexe.grd] Error 1
make[2]: Leaving directory /root/hpl-2.2/testing/ptest/Linux_PII_CBLAS' make[1]: *** [build_tst] Error 2 make[1]: Leaving directory /root/hpl-2.2'
make: *** [build] Error 2

I am astonished how you managed to skip supported Red Hat 6 devtoolset 4.1 which is included even in their 50$/year subscription. I am sorry but your haswell build works like charm, there is nothing to add to this non-issue.
OpenBLAS FAQ deals with static linking issues too. It should not take more than 5 minutes to read to the point where pthreads are mentioned.

Dear Andrew,
Thanks a lot. Due to the system is for test only for few days(I did not know if I keep reinstall RHEL how to register it), I will format it to reinstall the other OS like SuSe or Windows
I did read, but it kind of difficult for me to understand.
With Best Regards.
John Centeno

Red Hat is subscription service. If you dont subscribe ($$$) it does not work. Install CentOS7(Or Scientific Linux 7 if that flatters you better) or OpenSUSE 42 (or fedora or mageia for that sake) and get OpenBLAS 0.2.18 (prebuilt well-followed latest release with all haswell support) from provided repositories.  Windows or Debian or Ubuntu will force you back into painful recompilation track if you are not fine with old releases.
And there is no need for HPL cluster scheduler on a single system ever.

Dear Andrew,
My co-workes told me he fixed the problem by modifying HPL compiler file "Make.Linux_PIII_CBLAS". He did not change my configure on OpenBLAS
below 3 items in Make.Linux_PII_CBLAS
HPL_INCLUDES =
HPL_LIBS     =
LINKFLAGS    =
With Best Regards.
John Centeno

Congrats. You can close this non-issue about your broken system that has nothing to do with hpl or openblas.


Problem to show file content when using  more than one file and custom layout

Hi there!
I'm using a custom layout based on this example https://github.com/google/playground-elements#custom-layouts.
Since I need to use more than one file I added playground-tab-bar component, which is rendered ok. The problem arise when I want to select another file: the new content is not rendered/updated into playground-file-editor.
Here is my code.
<playground-project id="project1"  >

</playground-project>

    <playground-tab-bar project="project1"></playground-tab-bar>
    <playground-file-editor  project="project1"></playground-file-editor>
    <playground-preview project="project1"> </playground-preview>

And  css style:
.example {
  width: 100%;
  border: 1px solid #ccc;
  overflow: hidden;
  border-radius: 10px;
}
.example > playground-file-editor {
  height: 300px;
}
.example > playground-preview {
  height: max-content;
  border-top: 1px solid #ccc;
  --playground-preview-toolbar-background: #eaeaea;
}

Hi @tinchox5 , I have the same issue as well, but looking at the source code for <playground-ide> , it seems like the docs missed out on documenting the  editor attribute for <playground-tab-bar>
See source code
https://github.com/google/playground-elements/blob/2712837d118a35e038cac04b20734962593fd70f/src/playground-ide.ts#L310
<playground-tab-bar project="project1" editor="editor"></playground-tab-bar>
<playground-file-editor project="project1" filename="index.ts" id="editor"> </playground-file-editor>

it works well for me after i made that change

Thanks, I'll look into it.
On Mon, 6 May 2024 at 16:17 LuKhei @.***> wrote:

Hi @tinchox5 https://github.com/tinchox5 , I have the same issue as
well, but looking at the source code for  , it seems like
the docs missed out on documenting the editor attribute for

See source code
https://github.com/google/playground-elements/blob/2712837d118a35e038cac04b20734962593fd70f/src/playground-ide.ts#L310

it works well for me after i made that change
—
Reply to this email directly, view it on GitHub
https://github.com/google/playground-elements/issues/384#issuecomment-2096132417,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACYFYDJ6CQNKMAJGIN62AADZA6GHHAVCNFSM6AAAAAA6K6LNECVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJWGEZTENBRG4
.
You are receiving this because you were mentioned.Message ID:
@.***>


Make ProgramError & CairoRunError implement PartialEq

This PR aims to make the CairoRunError more user-friendly by making sure it implements PartialEq

Hey,
Just saw this one and it will conflict with #751
The reason is Box<dyn Error> does not implement PartialEq and we cannot use Box<dyn Error + PartialEq> because lot of error types don't implement PartialEq.
You will find multiple online threads about this specific issue and how most projects opt for not implementing PartialEq on their error types. Instead, they use assert_matches for tests.
@hurrikaanig  is coming up with a PR that will remove PartialEq on the different Errors types in the project and migrate all tests from assert! to assert_matches!
So please don't merge  :)

@tdelabro this sounds like a sensible request. @fmoletta do you think we can close this one?


adds Public Permalinks

I included tests too!
Includes the routing work in Rustland, along with the React stuff.

good call; I adjusted that to use propTypes. Also added tests for AdList (the factored-out bit of Frontend.js).


Browserify usage

Take a look on browserify and its grunt implementation. Grunt-rowserify vs grunt-concat + Bower

There is alternative workflow based on webpack. See other branch of this repo


Constants are not 'virtual' when defined in a derived class but accessed from a static function in a base class

Godot version:
c633b770cb648613ca88fc9e007718acfc219317
OS/device including version:
Linux
Issue description:
I'm not sure if this is a bug or not.
It appears that constants defined in a base class are not 'virtual' when it comes to using them in a derived class via a static function. The static function uses the const definition of its own class.
Example:
res://objects/a/a.gd
extends Node2D

const SCENE=preload("a.tscn")
const WHOAMI="a"

static func instance():
    print(WHOAMI)
    return SCENE.instance()

res://objects/b/b.gd
extends "res://objects/a/a.gd"

const SCENE=preload("b.tscn")
const WHOAMI="b"

res://screens/test.gd
const B = preload("res://objects/b/b.gd")

func _ready():
    print("Should be: ", B.WHOAMI) # Prints 'b'
    var test = b.instance() # Prints 'a'

Workaround
Defining the static function again in b.gd will fix the issue as long as it doesn't call its superclass.

Closest equivalent I can think of in python:
class A(object):
  WHOAMI="a"

  @classmethod
  def test(cls):
    print(cls.WHOAMI)

class B(A):
  WHOAMI="b"

a=A()
b=B()

a.test() # prints 'a'
b.test() # prints 'b'

82e769f
Indeed I confirm there's something strange going on. That example confused the hell out of me tho' so I made my own version. See the project.
godot-dbg.zip

You can no longer re-assign constants in inheriting classes since 3.1, so this is basically resolved...


Idea: watch option

Create an option to watch file changes and rerun the tests.

Right now you can just use the watch option of your test runner. When #32 gets implemented, we might wanna implement a watch option for the new shutter runner 👍


ngAnimate and Bootstrap. Select input does not bind correctly when using class "form-control"

This bug can be reproduce with chrome on mobile device.
When I try to select an option value in  input, it does not work the first time. Also, the control is flickering on first open.
https://jsfiddle.net/vj28awk7/
In this jsfiddle, on first section, try to select a value in each select control and you will see that it is not working on first time.
But like you can see, on second section, it is always working.
The only difference between these two section is the "class='form-control'" on the  input.
This bug occured on AngularJs 1.5.5
Android device with Chrome v50.0.2661.89

In jsfiddle, if you remove the "ngAnimate" module, this bug disappeared

I can't reproduce it on Android 5.0.2/Chrome v50.0.2661.89. What android version are you trying it on ?
If it appears on specific devices/browsers only, it sounds more like non-Angular bug.

I reproduce it on 2 differentes device. One on Android 6.0 and the other on Android 5.1.1
This is a screen recording that I have done to show you the behavior. in this video, I click on each drop-down, and select a value.
https://drive.google.com/file/d/0B_NwJiTIdBPtaFdFZFAyekpwb0U/view?usp=drivesdk

I can see it now. Interestingly it does not happen the very first time I click on a select. From the second time onwards, it does happen.

BTW, it only happens when ngAnimate is present and seems to be related with the fact that form-control has a transition property.

After some investigation, I have found this post about an issue with Chrome itself.
Maybe its a bug with chrome and not about angular. But it seem critical for me. A lot of site may have an issue with this.
https://productforums.google.com/forum/#!topic/chrome/-JLlY3eKQ-E;context-place=topicsearchin/chrome/category$3Aandroid|sort:relevance|spell:false

It seems to be a Chrome bug indeed (here is the [bug report] for future reference).
I don't think it makes sense to try to incorporate a temporary work-around into the core (until Chrome is fixed).
In the meantime, users can work around the issue, by overwriting/removing the transition style on <select> elements (e.g. the style added via Bootstrap's form-control class). E.g.:
select.form-control {
  transition: none;
}

I will close this, but feel free to continue the discussion below.

Thank you for your help.


BNF uniformity issue and unknown sequence in list of contributors

While performing a search of the word "almanac", instead of getting this output:
Performing query: almanac
I get the following output:
Performing query: bib.anywhere all (almanac)
However, I still get results.
Another issue is that the list of contributors shows this:
- Contributors:
  - 19970701d1945    m  y0frey50      ba

I checked the original entry in the BNF database, and that sequence doesn't appear anywhere. Different entries show the same final sequence "m  y0frey50      ba", with the first part of the sequence changing.

Thanks for spotting the odd lists of contributors! I don't have a fix for this immediately, because it seems that BnF puts the contributors across different fields, but I will look further into this.


kde-frameworks: 5.49 -> 5.50

Motivation for this change
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Fits CONTRIBUTING.md.

Could you add an expression for the new Framework, syndication?

Yep. Will do that.

@ttuegel done :)


New conference: ConFrontJS 2018

Hi @watson ! 👐
Maybe you will interests our conference! Please welcome:

Thank you 💯

I unfortunately don't have time to go my self this year :(

Thanks for merge 🍺


fix: increase import filesize limit from 32KB to 1MB

I believe that this will address https://github.com/cnwangjie/better-onetab/issues/132 for most users. I've tested this on my computer on Firefox with the following onetab exports (~1KB, ~250KB, ~750KB, respectively):

https://gist.github.com/adamreis/e27fc2a953eaa318d8c24c054fa9ca42
https://gist.github.com/adamreis/4db38221dc4f12f123334a9ea93ddaa2
https://gist.github.com/adamreis/a431176e61c3462b1d318227c408585a

All work smoothly and have no noticeable latency. I've not tested on Chrome.
Regardless of whether this is merged, there should probably be a visible error message if the file size limit is exceeded. I spent ~10 minutes trying to add this (with a <v-alert>), but couldn't figure out how to import that component (first time in a Vue codebase). @cnwangjie think this is worth adding as a small TODO on the roadmap?

ping @cnwangjie :)


Change str to string

str seems needlessly terse.

Does anyone else like str? I like how terse it is. int is a terse version of integer, which is only 1 character longer than string.

I like str, it is consistent with int

@naegelejd Valid point

@MovingtoMars @bread-man @kiljacken
You guys agree or disagree?

@felixangell I don't really mind.

Sure? It wouldn't hurt IMO

str seems alright

If you want to keep consistent syntax with C and Java, keep 'string', and else use 'str'.

@AdiSai we're not exactly looking to keep our syntax consistent with another language. We're trying to make it such that we're not too non-conventional. That said, str isn't really a change for the worse, since its actually shorter to type and (somehow) minimal.


Ways for improvement

Found some ways for improvement:
General:

use rem instead of mod because its more efficient
use (.) to get rid of parameters and allow pipelining
always add types of parameters and result to increase readability

isPrime:

$ infront of list is not necessary
instead of filter odd use [3,5..(ceiling $ sqrt $ fromIntegral t)]
instead of t | even t and case t of do _ | even t or something like

isPrime t
  | t < 2     = False
  | even t    = t == 2
  | otherwise = all ((/= 0) . rem t) [3,5.. (ceiling $ sqrt $ fromIntegral t)]

getNPrimes:

instead of foldr and take simply use filter
(2 : [3,5..]) instead of [1..] because we know that 1 and even numbers are not prime

getNPrimes n = take n $ filter isPrime $ 2 : [3,5..]
-- or
getNPrimes = flip take (filter isPrime $ 2 : [3,5..])

primeProducts:

l.36: change map (\y -> (fst y) * (snd y)) to map (uncurry (*))

Efficiency:

isPrime 999983

before: (0.01 secs, 224,648 bytes)
after: (0.01 secs, 185,552 bytes)

getNPrimes 10000

before: (1.00 secs, 525,875,992 bytes)
after: (0.50 secs, 375,398,952 bytes)

Hope i can help.

Hey, this is great! This was just written as a quick job interview test but this feedback is educational 👍


uwsm: init at 0.17.0

Description of changes
Homepage:- https://github.com/Vladimir-csp/uwsm
Description:-  Wraps standalone Wayland compositors into a set of Systemd units on the fly. This provides robust session management including environment, XDG autostart support, bi-directional binding with login session, and clean shutdown.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Tested uuctl and it seems fine.


Larger models (70B, 405B)

Thanks a lot for open sourcing this amazing library!
I was wondering whether you tried/are planning to prepare some larger models too, like Llama-3.1-70B/405B. As it seems, there is an actual chance to be able to run Llama-3.1-405B on a single Mac.
Also, would you mind opening up the discussions section?

Running a 405B model on a single PC is super cool! And BitNet can definitely help that.
Unfortunately, there is no plan for now due to the constrain of resources. Hopefully will make it in the near future.
We have opened up the discussion section. Enjoy!

In the benchmark's graph, there is a 70b model. So, is it supported or not?

Good! Please start it
I very happy

I would like a 120B model, I can run 120B models at 2 bits with normal methods but I obviously losing a lot of quality but around same size as 70B model. In theory, this will be best of both worlds 120B at 70B size and FP16 120B quality :)

no

now model is not workd

If you good model, please push a library

very thanks

hi guys, so the 70B model should work with BitNet? is there any other potentisl risks?

What model? I am not aware of a 70B model, please don't leave useless comments like this. This project won't magicly make your models use less memory, you need new models trained from scratch that are bigger. As is it seems no one has really done this, for what reason, likely goes over my head but could be technical or just no one wanting to take the risk


[staging & production] figshare revoked access not handled properly

Related to #332 but a special case
Steps

Link and configure figshare for at least one project/component
Follow the steps listed in https://trello.com/c/vvQwHOdE/63-staging-handle-revoked-api-access-through-figshare to revoke API access through figshare
Visit user settings page
Visit the node settings page for the project/component linked to figshare
Open the dev console and then click 'Change' button to change the folder associated with figshare
If able, change the linked folder to a private folder
Go to the files tab for node
If the files are showing in the files tab, click on a file to render it

Expected
I expected that there would be an error message similar to that seen for dropbox when API access is revoked, explaining what has occurred:

I also expected that when access is revoked, current figshare folders should not be able to load and, furthermore, that files/filenames should not be able to load in the files tab.
Actual
There is no error message displayed to the user. Additionally, here are other unexpected actions related to the step numbers listed above:
3. No error message on user settings page
4. No error message on project settings page
5. Figshare folders load most of the time, but sometimes the console shows a '400 Bad Request' and it appears to the user as though the folders are just taking forever to load
6. When the folders load, linked folders can be changed
7. Files some of the time. The three situations I have encountered are:

Files load and all filenames can be seen
Files do not load but the figshare icon is present but appears collapsed, and when the plus sign is pressed to expand it, an error occurs in the grid saying that 'Files couldn't load, please try again later' and the console shows a '401 Unauthorized' error
The figshare icon does not appear in the grid at all, even though the settings page says that a folder is linked

.8 When the files load in the files grid, a file can be clicked and the user is redirected to the files view page. The file does not render though, and a '401 Unauthorized' error is shown in the console and the user sees a message on the page saying that the file could not be rendered.

As the new folder which is https://github.com/CenterForOpenScience/osf.io/pull/2014 by @samchrisinger just go on staging, the pr is just under testing and modifying a lot of parts of the current addon backend. It's better to address addon issues after this is quite settled. Besides revoking api process itself is quite an edge case and it will be solved in the process when we refactor the old addons into the new oauth system. Put this to defer and change the priority to low.


Don't accidentally swallow script errors

Pull Request Test Coverage Report for Build 74fdeae3b33f51ef019a211e04a9231a97901648-PR-1052

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 65.977%

Totals

Change from base Build 2720200b68dcab36fa0485c97a289b285127f464:
0.0%

Covered Lines:
1148

Relevant Lines:
1740

💛  - Coveralls


Swagger.json has an old version

After doing metadata docker --start
It shows 0.3.0 instead of 0.6.0

@amiorin lets make this parameterized with ${project.version} https://github.com/open-metadata/OpenMetadata/blob/main/catalog-rest-service/pom.xml#L364


Can we switch to another email provider other than Sendgrid?

Sendgrid doesn't have a free plan monthly plan.   Maybe use Amazon SES (here's a larger list with free services - https://www.ventureharbour.com/transactional-email-service-best-mandrill-vs-sendgrid-vs-mailjet/)

blocker is that there isn't an ansible module.

Pull requests accepted on having an SES module. ;)
We discussed getting rid of sending any sort of email message as part of #141

@alberttwong as an FYI, Sendgrid offers 100 free emails a day as part of their free trial. So if you sign up, after the trial you can continue to send 100 emails a day without charge.
https://sendgrid.com/pricing/

@kyleabenson I signed up just for the month but the plan is to have more ansible workshops.   It would be great have more a pay as you use as opposed to a subscription.

What I found easier was to create generic accounts, for example:
student1 - 20 for example...
And either have that up in gist.github for example or have that available during the workshop and assign students the approrpiate student(x) node.  Often saw students not having received e-mails, walk-ins, etc. so it makes it easier to control.
So, in roles/email/defaults/main.yml
email: no
The sendgrid username and password can be w/e at this point.


is this useful in real-time detection?

hello, deep learning newbie here.
can this be used in real time detection? and if so, is it API rewriting? which function do I use to try and get it to work?

see #6

thanks, the rewrite api files are in the include folder right?

Yes, but you may not need to rewrite it, just use it for frame by frame processing.

frame by frame processing works for real-time detection as well? woah, I had no idea, so I can get it to detect on a webcam and such?

Well, if you want real-time, you need a gpu at least.
refer to https://github.com/zhreshold/mxnet-ssd

I got a pretty good gpu at the moment, object detection on videos work well, how do I begin with real-time detection? Thanks for being so helpful 👍

You need to enable GPU in mxnet, but it's not written in makefile right now.

oh alright, so all I have to do is enable GPU in the makefile? I'd gladly do so, but the only thing is I can't see the makefile in the directory

mxnet/amagamation

the amalgamation folder is not the windows branch, could it still work if I use the makefile from the master branch?

the win branch is what I pre-built with visual studio, you can use amalgamation or simply build the entire lib using VS, either one is ok. The latter have an official tutorial on dmlc/mxnet


Fullscreen resets the server sreen size

Thank you for a great program. I am using its Windows build to connect in such setup:

server -- Linux running VBox
client -- Windows

I try to connect with fullscreen turned on ("/f") but it does not work -- instead fullscreen VBox running on server is reset to small size window, then wfreerdp uses this size as its own fullscreen resulting in pretty small window on client computer.
For comparison I tried other rdp clients and they do not reset server screen size and thus they are able to use the size set on the server.
Tested today on nightly build and also using version 1.1.

@macias just to clarify you are connecting with wfreerdp to virtualbox (vrdp)?

@bmiklautz, yes, exactly, this is the case. However small correction, the other client I tested (2X RDP) "cheated" -- I rechecked it and it also resets the screen size on server, but on client it runs fullscreen by scaling the display. So it is not true fullscreen really.

@macias what do you have installed inside VBox? - Are the guest utils installed or not? Also the VBox version would be interesting.

@bmiklautz , I have Windows 7 inside VBox. The guest additions are installed. I tested older VBox 4.3.x version, and then I installed the newest 5.0.10. Both versions behave the same. Just in case, host system for VBox is openSUSE 13.2.

I cannot tell because for some time I don't use this setup anymore.


fix: enable source desc and allowing editing source name and desc

Please describe the purpose of this pull request.
This enables storing the source description and allows users to edit an existing source name and description.
How to test
How can we test your PR during review? What commands should we run? What outcomes should we expect?
You can load a simple data source, run the memgpt server, and then edit the name and/or description of the existing data source without the source ID changing.
Have you tested this PR?
Yes, will attach screenshots
Related issues or PRs
https://github.com/cpacker/MemGPT/issues/1588